LSM-based Storage Techniques: A Survey (1812.07527v3)

Abstract: Recently, the Log-Structured Merge-tree (LSM-tree) has been widely adopted for use in the storage layer of modern NoSQL systems. Because of this, there have been a large number of research efforts, from both the database community and the operating systems community, that try to improve various aspects of LSM-trees. In this paper, we provide a survey of recent research efforts on LSM-trees so that readers can learn the state-of-the-art in LSM-based storage techniques. We provide a general taxonomy to classify the literature of LSM-trees, survey the efforts in detail, and discuss their strengths and trade-offs. We further survey several representative LSM-based open-source NoSQL systems and discuss some potential future research directions resulting from the survey.

• The paper systematically categorizes LSM-tree improvements into domains like reducing write amplification, optimizing merge operations, and exploiting hardware capabilities.
• It details innovative merge techniques, such as tiering and pipelined compaction, that enhance throughput and resource utilization in storage systems.
• The survey emphasizes auto-tuning strategies and advanced secondary indexing to address traditional LSM-tree limitations for diverse workloads.

A Survey of LSM-based Storage Techniques

The paper "LSM-based Storage Techniques: A Survey" by Chen Luo and Michael J. Carey presents a comprehensive survey of advancements and methodologies associated with the Log-Structured Merge-tree (LSM-tree) in modern NoSQL systems. LSM-trees have been pivotal in optimizing the storage layer through approaches that capitalize on their structural advantages such as improved write performance, high space utilization, and simplified concurrency control. This paper categorizes and examines the extensive range of improvements made to LSM-trees, underscoring the developing patterns and research trends in the field.

Core Contributions and Organization

The survey categorizes LSM-tree advancements into several key domains: reducing write amplification, optimizing merge operations, harnessing hardware opportunities, addressing special workloads, auto-tuning, and enhancing secondary indexing. These categories are systematically analyzed, detailing how each endeavor tackles the inherent limitations of traditional LSM-tree implementations.

1. Reducing Write Amplification: The paper discusses several strategies aimed at minimizing write costs. Notably, techniques such as tiering merge policies (e.g., WriteBuffer (WB) Tree, LWC-tree) showcase methods to improve write throughput. These approaches often trade-off query performance and space efficiency to some degree, indicating a conscious decision in system optimization.
2. Optimizing Merge Operations: Improvements like the VT-tree's stitching merge and Zhang et al.'s pipelined compaction are highlighted for their ability to improve merge efficiency and better utilize system resources, thus addressing bottlenecks associated with traditional merge practices.
3. Exploiting Hardware Opportunities: The survey emphasizes how modern hardware (such as SSDs and NVMs) influences LSM-tree design. Variants like the FD-tree that employs fractional cascading, and WiscKey's separation of keys and values, showcase the potential to exploit hardware capabilities for optimized performance.
4. Special Workloads: Efforts like LHAM for temporal workloads and SlimDB for semi-sorted data are discussed to illustrate how specialized applications can benefit from custom LSM-tree enhancements.
5. Auto-Tuning: The evolution towards self-tuning LSM systems is evaluated through efforts like Monkey and Dostoevsky, which use cost models to automatically calibrate system parameters, thereby optimizing performance and reducing configuration overhead.
6. Secondary Indexing: The development of LSM-based secondary indexes is increasingly crucial in database contexts. The research highlights techniques to handle efficient secondary index maintenance and query processing through methods that address traditional limitations like update overheads and space complexity.

Implications and Future Directions

The paper outlines potential future research areas, stressing the need for thorough performance evaluations that consider LSM-tree tunability and the implications of different merge policies. Particularly, the hybrid merge policies show promise for offering improved write performance without severely impacting query efficiency or space utilization.

In essence, this survey acts as a foundational reference for researchers and practitioners seeking to innovate on LSM-trees within storage systems. It consolidates the state of the art, providing clarity on existing techniques while motivating further advancements in the optimization of LSM-trees for varied application landscapes and emerging hardware technologies. The growing adoption and adaptation of LSM-trees within DBMS engines necessitates continued innovation, particularly towards integrating adaptive structures and optimization frameworks that accommodate diverse and evolving workload demands.