CUBIT: Concurrent Updatable Bitmap Indexing (Extended Version) (2410.16929v2)

Abstract: Bitmap indexes are widely used for read-intensive analytical workloads because they are clustered and offer efficient reads with a small memory footprint. However, they are notoriously inefficient to update. As analytical applications are increasingly fused with transactional applications, leading to the emergence of hybrid transactional/analytical processing (HTAP), it is desirable that bitmap indexes support efficient concurrent real-time updates. In this paper, we propose Concurrent Updatable Bitmap indexing (CUBIT) that offers efficient real-time updates that scale with the number of CPU cores used and do not interfere with queries. Our design relies on three principles. First, we employ a horizontal bitwise representation of updated bits, which enables efficient atomic updates without locking entire bitvectors. Second, we propose a lightweight snapshotting mechanism that allows queries (including range queries) to run on separate snapshots and provides a wait-free progress guarantee. Third, we consolidate updates in a latch-free manner, providing a strong progress guarantee. Our evaluation shows that CUBIT offers 3x - 16x higher throughput and 3x - 220x lower latency than state-of-the-art updatable bitmap indexes. CUBIT's update-friendly nature widens the applicability of bitmap indexing. Experimenting with OLAP workloads with standard, batched updates shows that CUBIT overcomes the maintenance downtime and outperforms DuckDB by 1.2x - 2.7x on TPC-H. For HTAP workloads with real-time updates, CUBIT achieves 2x - 11x performance improvement over the state-of-the-art approaches.

• The paper introduces a novel concurrent updatable bitmap index employing horizontal update deltas (HUD) to enable efficient, atomic updates without full bitvector locks.
• The paper demonstrates a lightweight wait-free snapshotting mechanism that separates query execution from ongoing updates, ensuring high availability and consistency.
• The paper shows that CUBIT outperforms existing solutions with 3–16× throughput improvements and significant latency reductions in HTAP scenarios.

CUBIT: Enabling Efficient Real-Time Updates in Bitmap Indexes for HTAP Workloads

The research paper addresses a fundamental challenge in the field of bitmap indexing: the inefficiency of updates. Bitmap indexes, known for their efficiency in read-heavy analytical workloads, have historically struggled with update operations, limiting their applicability in hybrid transactional/analytical processing (HTAP) environments. The work introduces CUBIT, an innovative concurrent updatable bitmap indexing mechanism designed to offer scalable real-time updates without query interference, leveraging modern multi-core architectures.

Key Contributions

CUBIT offers a robust solution built around several core principles:

1. Horizontal Update Deltas (HUD): By employing a horizontal bitwise representation of update deltas, CUBIT eliminates the need for locking entire bitvectors during updates. This strategy enables efficient, atomic updates, thereby addressing a significant bottleneck in traditional bitmap indexes.
2. Lightweight Snapshotting Mechanism: CUBIT maintains a snapshotting mechanism allowing queries to execute on separate index snapshots, exhibiting wait-free behavior and ensuring query completion without blocking ongoing updates. This ensures high availability and consistency in HTAP scenarios.
3. Latch-Free Design: The work introduces a sophisticated latch-free synchronization method for consolidating updates, which reduces contention and improves scalability even under skewed data distributions.

Performance Evaluation

The paper provides comprehensive empirical results demonstrating CUBIT's superior performance. CUBIT achieves a throughput improvement of 3–16× and latency reduction of 3–220× compared to the state-of-the-art updatable bitmap indexes. Noteworthy is its performance in OLAP workloads, outperforming DuckDB by 1.2–2.7× on TPC-H benchmarks. In HTAP settings, CUBIT reaches a 2–11× improvement over existing solutions.

Implications and Future Directions

The implications of CUBIT are substantial for the database community. By enhancing the update efficiency of bitmap indexes, CUBIT widens their applicability to real-time analytics and hybrid workloads where update frequency is substantial. This work bridges a critical gap, enabling bitmap indexes to handle frequent updates without sacrificing query performance, a necessity in modern data-driven applications.

Theoretically, the adoption of HUD and latch-free mechanisms in indexing hints at broader applications. Researchers could explore these concepts in other types of indexes, potentially revolutionizing their update mechanisms similarly.

As for practical applications, CUBIT can be integrated into existing DBMSs, particularly those operating with HTAP workloads, further pushing the boundaries of performance and efficiency.

Conclusion

The paper on CUBIT marks a significant step in bitmap indexing, effectively addressing long-standing update inefficiencies. The integration of novel data structures and concurrency mechanisms sets a precedent, promising to enhance both transactional and analytical processing in data-centric environments. For future explorations, expanding the latch-free techniques and HUD implementations into other indexing paradigms could yield additional advancements in database technology.