Create Benchmarks for Data Lakes

Abstract: Data lakes have emerged as a flexible and scalable solution for storing and analyzing large volumes of heterogeneous data, including structured, semi-structured, and unstructured formats. Despite their growing adoption in both industry and academia, there is a lack of standardized and comprehensive benchmarks for evaluating the performance of data lake systems. Existing benchmarks primarily target traditional data warehouses and focus on structured SQL workloads, making them insufficient for capturing the diverse workloads and access patterns typical of data lakes. In this work, we propose a new benchmarking framework for data lakes that aims to provide an objective and comparative evaluation of different data lake implementations. Our benchmark covers multiple data types and workload models, including data retrieval, aggregation, querying, and similarity search, which is a common yet underexplored operation in existing benchmarks. We measure key performance metrics such as query execution time, metadata generation time, and metadata size across different scale factors. The benchmark is designed to be extensible and reproducible, enabling users to generate datasets and evaluate data lake systems under realistic and diverse scenarios. We conduct our experiments on CloudLab and demonstrate how the proposed benchmark can be used to compare both commercial and open-source data lake platforms.