Optimizing Data Layouts for Evolving Cloud Table Storage

Abstract

Modern data analytics platforms increasingly adopt disaggregated architectures, storing data in cost-effective cloud object stores. While this approach enables a clean separation of concerns, allowing each layer to be independently managed and scaled, it introduces significant performance bottlenecks due to expensive data movement. Effective data layouts, which organize data to minimize unnecessary data reads, are thus critical to achieving high query performance. However, existing techniques typically rely on manually specified layouts, collect limited metadata, or lack mechanisms to dynamically adapt to changing data and workloads.

This thesis investigates adaptive, metadata-rich, expressive data layouts for cloud table storage. First, we introduce Pando, a correlation-aware layout technique that leverages rich metadata on query predicates to significantly improve data skipping. Next, we propose CopyRight, a partial replication strategy that selectively replicates subsets of data and optimizes each replica differently, efficiently serving heterogeneous query patterns. Finally, we describe Self-Organizing Data Containers (SDCs), a practical table storage layer for the cloud that incrementally reorganizes complex data layouts based on changes in data and workload distributions.