Partition-Based Indices

• Partition-Based Indices are defined as index structures that divide a global dataset into independent, non-overlapping segments, enabling localized query processing and improved cache/I/O performance.
• They employ various partitioning strategies—including range-based, hash, and hierarchical splits—to optimize update costs, load balancing, and duplicate elimination across both single-node and distributed systems.
• Applications span MV-PBT for version control, spatial and interval queries, string similarity joins, and high-dimensional vector searches, demonstrating significant gains in throughput and efficiency.

A partition-based index is any index structure in which the global dataset is divided—explicitly or implicitly—into multiple non-overlapping segments ("partitions"), each of which constitutes a localized subindex. Partitions can be defined by key ranges, spatial regions, one-dimensional or multi-dimensional attributes, or application-level criteria, and may serve system scalability, update performance, or query efficiency objectives. Implementation techniques encompass both single-node and distributed/parallel architectures, and span traditional B-tree and hash tables, spatial indices, temporal/interval indices, string edit similarity joins, and high-dimensional vector/feature search. Partition-based indices can target base data, versioned/temporal data, spatial/geometric objects, or coarsened representations such as MBRs, clusters, or quotient graphs.

1. Partitioning Principles and Indexing Architectures

Partition-based indices instantiate the general principle of dividing a global dataset $D$ into $P_1, \ldots, P_p$, with each partition mapped to an independent sub-index structure. Partitioning may be static (grid/tiling, range slices, hash buckets), dynamic (adaptive splits under load/size/temporal constraints), or application-driven (spatial Voronoi partitions, speed bands for moving objects, graph quotienting) (Riegger et al., 2019, Jr, 2013, Akdogan, 2016, Christodoulou et al., 2021, Xu et al., 2014, Tratnik, 2019, Thomasian, 2024, Tsitsigkos et al., 2020, Tsitsigkos et al., 2023).

Partitioned indices are essential to modern data management, providing:

• Locality of reference and improved cache/I/O performance via spatial/temporal clustering.
• Scalability by limiting per-query or per-update costs to a partition subset.
• Parallelism and high throughput for distributed execution.
• Partition-isolated maintenance/GC, enabling workload adaptation.

Variants include:

• Logically monolithic B$\!^+$-trees or R-trees over composite keys with prepended partition fields (cf. Partitioned B-Tree, MV-PBT) (Riegger et al., 2019, Riegger et al., 2022).
• Physically independent per-partition indices, coordinated by a global directory or dynamic partition map (Jr, 2013, Xu et al., 2014, Akdogan, 2016).
• Hierarchical partitioning schemes (e.g. HINT, two-layer grids), where primary indices are recursively refined by secondary per-partition indices (Christodoulou et al., 2021, Tsitsigkos et al., 2020, Tsitsigkos et al., 2023).

2. Partition Strategies: Schemes, Adaptivity, and Optimization

The partitioning scheme directly influences index performance, replication, and maintenance overhead. Techniques include:

• Range-based partitioning: Fixed or adaptive, based on key, value, time, spatial extent (Jr, 2013, Riegger et al., 2019, Christodoulou et al., 2021). Adaptive partitioning (e.g., in STOP's geo-spatial index) uses both row-count and max-age thresholds to trigger splits/merges, dynamically aligning partition boundaries to workload hotspots and coldspots.
• Hash partitioning: Uniformly maps records to partitions using a hash function, effective for load balancing but inhibits locality-based query pruning.
• Hierarchical/recursive partitioning: Domains or keyspaces are recursively partitioned (as in quadtrees, recursive interval splits in HINT, or multi-level PCA spaces in SDI indices) to bound candidate set sizes and support logarithmic query complexity (Christodoulou et al., 2021, Thomasian, 2024).
• Attribute-driven partitioning: Partitioned by velocity (for moving-object indices (Xu et al., 2014)), cluster membership (for high-dimensional feature indices (Thomasian, 2024)), or edge equivalence classes in graphs (for quotient-graph Szeged/PI indices (Tratnik, 2019)).

A central analytic theme is cost-driven partitioning. For instance, the moving-object speed-partitioning framework seeks partitions that minimize total search-space expansion $V(t_h)$, solved via dynamic programming over the sorted speed domain (Xu et al., 2014). Workload-driven adaptive partitioning uses per-partition growth rates, access frequencies, or query hotness metrics to trigger reclustering or splitting (Jr, 2013, Riegger et al., 2019).

3. Partitioned Index Structures: Exemplars and Workflows

Multi-Version Partitioned B-Tree (MV-PBT)

MV-PBT (Riegger et al., 2019, Riegger et al., 2022) serves as a canonical example for version-aware, partition-based indexing:

• The search key is augmented to $$(\text{partition number},\, \text{indexed columns},\, \text{version timestamp})$$.
• The structure consists of $p+1$ partitions: $p$ immutable, on-disk partitions; 1 hot, in-memory mutable partition absorbing all updates.
• All index records embed version metadata (create/invalidate timestamps), allowing index-only MVCC visibility checks.

Partition growth is governed analytically, balancing update rate, retention window, and partition size. Search proceeds by visiting successively older partitions until a visible version is found. Range queries merge scans from each relevant partition, with per-partition Bloom/range filters optimizing probe elimination.

Garbage collection is partition-local: obsolete versions are pruned upon memory eviction or opportunistically within the mutable partition. Partition merging, size splitting, and coalescing maintain bounded partition counts and control space amplification. MV-PBT achieves up to $2\times$ higher analytical throughput and $40\%$ higher transactional throughput than LSM-trees in practical HTAP workloads.

Spatial/Interval Partitioning: Two-Layer, Hierarchical, and Grid Methods

Spatial and interval indices employ both primary (space/domain) partitioning and secondary per-partition groupings to optimize query cost and avoid duplicate results:

• Two-layer schemes (e.g., in-memory grid or SOP grid (Tsitsigkos et al., 2020, Tsitsigkos et al., 2023)) assign each spatial object to all overlapping cells (primary), but further classify cell-level entries by origin (e.g., which tile the lower-left corner lands in) (secondary). This allows queries to examine only one replica per object, eliminating de-duplication cost.
• Hierarchical interval indices (Christodoulou et al., 2021): Partition the 1D interval domain at multiple granularities, assigning each interval to at most two cells per level, with further subdivisions within each cell (e.g., original/replica, end-inside/end-after). This achieves $O(\log \lambda)$ replication and expected $P_1, \ldots, P_p$0 query comparisons, outperforming tree and flat-grid baselines by up to $P_1, \ldots, P_p$1 in query throughput.

String Similarity Joins by Partitioned Segment Indexing

PASS-JOIN (Li et al., 2011) leverages partitioning for edit-distance joins:

• Each string is partitioned into $P_1, \ldots, P_p$2 nearly equal-length segments (for threshold $P_1, \ldots, P_p$3), and each segment is indexed (per length, per segment slot) in a set of inverted lists.
• Minimal substring selection per string and per segment guarantees completeness while minimizing probes.
• Partitioned indexing yields order-of-magnitude lower candidate and verification cost than prior $P_1, \ldots, P_p$4-gram or trie-based methods, with O($P_1, \ldots, P_p$5) per-string substring extraction and inverted-list lookups.

High-Dimensional Vector Search: Clustered and Dimensional Partitioning

Order Partition (OP) and Stepwise Dimensionality Increasing (SDI) indices (Thomasian, 2024) partition the high-dimensional data space by clustering (CSVD) and cluster-local dimension reduction. Each cluster's low-dimensional space is indexed by an OP-tree (multiway, round-robin split), supporting branch-and-bound $P_1, \ldots, P_p$6-NN search. SDI refines this, using a multi-level PCA-driven hierarchy, at each level indexing only a progressively larger subset of variance-capturing dimensions. Analytical results provide cost models and optimal dimension selection: $P_1, \ldots, P_p$7 where $P_1, \ldots, P_p$8 and $P_1, \ldots, P_p$9 are projection and search cost coefficients.

Empirical results: up to $\!^+$0 faster than VA-file or SR-tree at $\!^+$1, $\!^+$2, and robustness to scaling in both $\!^+$3 and $\!^+$4.

4. Query Processing, Maintenance, and Parallelism

Partitioned indices support diverse query and maintenance patterns:

• Point/range search: Locate the partition(s) covering query keys or spatial windows, issue sub-queries, and merge/union results. Partition-local per-filtering (Bloom, range, spatial) reduces candidate scans.
• Join/overlap queries: In the two-layer SOP scheme, only a subset of all mini-join combinations produces non-duplicate results, enabling efficient distributed or parallel joins (Tsitsigkos et al., 2023).
• Insert/update/delete: Hot (“mutable") partitions absorb updates (append-only or in-memory), flushed to immutable partitions upon reaching thresholds. This design amortizes I/O, minimizes write amplification, and enables asynchronous GC (Riegger et al., 2019, Riegger et al., 2022).
• Parallel/distributed processing: Statistically independent partitions can be distributed across servers or threads, supporting query atomic (per-query) or tile atomic (per-tile) execution (Tsitsigkos et al., 2020, Tsitsigkos et al., 2023, Akdogan, 2016).

A summary table of selected partition-based index variants:

Index Type	Partition Basis	Indexing Model	Key Strength
MV-PBT	Logical partition #	single B$\!^+$5-tree/leaf grouping	Version-aware, append-only, flash-optimal
SOP 2-layer	Grid x secondary	per-cell class lists	No duplicate results, parallelism
HINT	Hierarchical cells	per-level interval lists	$\!^+$6 replication, fast scan
OP/SDI	Cluster + PCA dims	cluster OP-tree / SDI-tree	High-dim $\!^+$7-NN, exact NMSE control
PASS-JOIN	Segment (edit pos.)	per-segment inverted index	Minimal substring probing, all string lengths

5. Performance Analysis and Trade-Offs

Empirical and analytical studies demonstrate the following key trade-offs:

• Space amplification: Partition replication increases storage, but hierarchical/secondary partitioning bounds the factor (e.g. $\!^+$8 in HINT (Christodoulou et al., 2021)).
• Write amplification and update cost: Partition-local append or flush amortizes random write I/O, decreasing WA compared to classic update-in-place trees (MV-PBT WA: $\!^+$9 vs B$V(t_h)$0-tree $V(t_h)$1) (Riegger et al., 2022).
• Read/query cost: Finer partitioning can increase the number of partitions to scan for some queries, trading off with update/write efficiency. Optimal partition count is workload- and buffer-dependent (Riegger et al., 2019).
• Load balancing/scalability: Partitioning by value/random hash can hurt compression (inverted index) or spatial locality but achieves superior parallelism and query load distribution (Feldman et al., 2011).
• Duplicate elimination: Two-layer and hierarchical designs directly avoid duplicate generation, in contrast to classic grid or R-tree approaches that require explicit deduplication (Tsitsigkos et al., 2023, Tsitsigkos et al., 2020).

6. Extensions and Theoretical Unification

Partition-based index design unifies a spectrum of index families, including:

• Comparison-based indices (sprawl/ambits, (Hetland, 2019)): Any region-based or metric space partitioning can be mapped to a sprawl index defined by positive/negative regions (ambits) and query-centric reachability.
• Graph- and quotient-based indices (Tratnik, 2019): Topological indices such as (weighted) Szeged and PI can be decomposed to sums over partition-induced quotient graphs—often trees in the case of chemical graphs—yielding sublinear evaluation cost.
• High-dimensional and time-evolving datasets: Partitioning via clustering and dimension reduction (OP/SDI), or by externalities (partition function games, (Alonso-Meijide et al., 2024)), demonstrates the generality of the partition-based principle.

Systematic partitioning via analytical cost modeling and dynamic programming (cf. speed-partitioning (Xu et al., 2014) or partitioning for cloud spatial index construction (Akdogan, 2016)) provides optimality guarantees absent in heuristic partitioning.

7. Limitations, Open Questions, and Future Work

Despite broad effectiveness, partition-based indices face intrinsic limitations:

• Partition proliferation: Excessively fine partitions can increase indexing overhead, scan cost, and complicate maintenance.
• Dimensionality curse: In very high dimensions, naive partitioning may induce fragmentation, and advanced dimensionality reduction or clustering is required (Thomasian, 2024).
• Complexity of dynamic partitioning: Adaptive schemes depend on accurate statistics, timely maintenance, and can be costly at very large scales without careful engineering (Jr, 2013).
• Compression vs. parallelism trade-off: As shown in the context of inverted index partitioning, random partitioning loses doc-order compressibility but gains dramatic improvements in distributed query balance (Feldman et al., 2011).
• Combinatorial explosion for partition-based external validity or power indices: e.g., in partition function games, the superexponential growth of partitions limits the practical computation of certain indices to modest $V(t_h)$2 (Alonso-Meijide et al., 2024).

Ongoing research directions include automated analytical partition boundary selection, adaptive index reorganization under shifting workloads, generalization to mixed data modalities, and principled integration of partition-based logic with learned indexes or workload-aware query optimization strategies.