Scaled Gray–Hilbert Index in High-Dimensional Data

• The scaled Gray–Hilbert Index is a method that adapts depth based on local data density for high-dimensional indexing.
• It minimizes storage and boosts efficiency by using flexible, local subtree adaptations instead of uniform global subdivision.
• Applications include data queries in complex spatial and spatio-temporal datasets with demonstrated storage reductions.

The scaled Gray–Hilbert index is a data-driven space-filling curve indexing method that extends the classical Hilbert (Gray–Hilbert) curve framework to efficiently support storage and query operations in high-dimensional spatial or spatio-temporal data. Unlike static Hilbert curve indices, which impose a uniform depth across the index structure, the scaled Gray–Hilbert index adapts its subdivision depth locally to the underlying data distribution, significantly reducing storage requirements and improving efficiency while preserving the desirable locality properties of space-filling curves (Jahn et al., 2019, Bradley et al., 2019).

1. Theoretical Foundations and Motivation

The static n-dimensional Hilbert curve is defined for a fixed “order” $k$ as a continuous mapping $H_k : [0,1] \to [0,1]^n$, achieved by recursively subdividing the unit cube into $2^n$ subcubes at each of $k$ levels and visiting subcubes in the binary-reflected Gray code order. Each point $x \in [0,1]^n$ with binary expansions $x_j = 0.x_{j,1}x_{j,2}\ldots x_{j,k}$ for $j=1,\ldots, n$ is mapped to a scalar $H_k(x) = 0.y_1 y_2 \ldots y_{kn}$, where each $n$-bit vector of coordinate digits at level $i$ is mapped through the Gray code with suitable coordinate permutations.

A key motivation for moving to a scaled, data-driven approach originates from the exponential blow-up in the number of leaf nodes, $H_k : [0,1] \to [0,1]^n$0, required to enforce a maximum bucket capacity $H_k : [0,1] \to [0,1]^n$1 in high-dimensional data. This leads to impractical index sizes for moderate to large $H_k : [0,1] \to [0,1]^n$2 or $H_k : [0,1] \to [0,1]^n$3. The scaled Gray–Hilbert index replaces this rigid global subdivision by growing an adaptive subtree whose leaves satisfy the bucket constraint locally—deepening the tree only in dense regions and pruning it early in sparse regions. This achieves substantial storage and cache behavior improvements without degrading spatial locality (Jahn et al., 2019).

Generalization to p-adic systems (with $H_k : [0,1] \to [0,1]^n$4 prime) provides further flexibility, constructing high-dimensional analogs using p-adic reflected Gray codes and affine transformations, and retaining the key scaling properties in larger ambient spaces (Bradley et al., 2019).

2. Construction Algorithms and Mathematical Structure

Let $H_k : [0,1] \to [0,1]^n$5 be a finite point cloud. The scaled Gray–Hilbert index is constructed by recursively partitioning the space using a Gray–Hilbert tree, where each node represents an $H_k : [0,1] \to [0,1]^n$6-dimensional dyadic subcube (or $H_k : [0,1] \to [0,1]^n$7-adic in generalizations). At each step, the set $H_k : [0,1] \to [0,1]^n$8 of points within a current subcube $H_k : [0,1] \to [0,1]^n$9 is tested: if $2^n$0 (bucket capacity), $2^n$1 is marked as a leaf; otherwise, it is subdivided into $2^n$2 (or $2^n$3) children in Gray code order, and the process recurses. Permutations are chosen locally to ensure continuity of the curve.

The traversal of the leaves (pre-order) defines the space-filling curve index. In the binary (Hilbert) case, the Gray code $2^n$4 is used at each subdivision level, applied to the current $2^n$5-bit vector derived from coordinate digits at that level (with local coordinate permutations). For the p-adic version, the p-adic Gray code $2^n$6 and its affine transformations generate the ordering, ensuring that consecutive subcubes differ in only one digit (Bradley et al., 2019).

Pseudocode for the binary case can be summarized as: $i$8 For the p-adic generalization, the index is constructed by: at each level $2^n$7, compute the local code $2^n$8 (with the entry corner recursively updated), and assemble the final index as $2^n$9 (Bradley et al., 2019).

The static index executes the same recursive subdivision but stops globally at $k$0, resulting in $k$1 leaves regardless of data distribution.

3. Capacity, Local Sparsity, and Theoretical Properties

Let $k$2 denote the resulting Gray–Hilbert subtree. Several key quantities are introduced:

• $k$3leaves of $k$4.
• $k$5overfilled leaves with $k$6.
• $k$7nonempty leaves with $k$8.

Define the overfill ratio $k$9 and the capacity $x \in [0,1]^n$0.

The index size ratio $x \in [0,1]^n$1 quantifies relative storage efficiency, with $x \in [0,1]^n$2 indicating an advantage for the scaled index in both binary and p-adic settings. For fixed $x \in [0,1]^n$3 and increasing dimension $x \in [0,1]^n$4 (or $x \in [0,1]^n$5 in the p-adic case), $x \in [0,1]^n$6, so the scaled index becomes substantially more efficient (Bradley et al., 2019, Jahn et al., 2019).

A local sparsity measure $x \in [0,1]^n$7 is defined via

$x \in [0,1]^n$8

with $x \in [0,1]^n$9. Values near $x_j = 0.x_{j,1}x_{j,2}\ldots x_{j,k}$0 indicate near-uniform distributions, while $x_j = 0.x_{j,1}x_{j,2}\ldots x_{j,k}$1 signals heavy-tailed or strongly clustered datasets. The measure can distinguish differences in distribution (uniform, normal, or highly clustered) beyond what log-log tail plots can capture (Jahn et al., 2019). In the p-adic case, the analogous $x_j = 0.x_{j,1}x_{j,2}\ldots x_{j,k}$2 and $x_j = 0.x_{j,1}x_{j,2}\ldots x_{j,k}$3 are defined and interpreted similarly (Bradley et al., 2019).

4. Empirical Evaluation and Storage Efficiency

A key experimental application involves a dataset of 2.5 million tropical rainforest tree records in 18 dimensions, with projection to 8 attributes (including spatial, temporal, and categorical fields) (Jahn et al., 2019). Storage is assessed using the capacity $x_j = 0.x_{j,1}x_{j,2}\ldots x_{j,k}$4 and the index size ratio $x_j = 0.x_{j,1}x_{j,2}\ldots x_{j,k}$5. Empirical results show that for bucket capacities $x_j = 0.x_{j,1}x_{j,2}\ldots x_{j,k}$6, $x_j = 0.x_{j,1}x_{j,2}\ldots x_{j,k}$7, the storage ratio $x_j = 0.x_{j,1}x_{j,2}\ldots x_{j,k}$8 ranges from approximately $x_j = 0.x_{j,1}x_{j,2}\ldots x_{j,k}$9 (for $j=1,\ldots, n$0) to $j=1,\ldots, n$1 (for $j=1,\ldots, n$2), corresponding to 7–80% of the storage used by the best static index. As dimension increases, $j=1,\ldots, n$3, indicating significant storage savings.

For varying projections:

• $j=1,\ldots, n$4 only: $j=1,\ldots, n$5 (indicative of uniformity in the spatial footprint),
• $j=1,\ldots, n$6: $j=1,\ldots, n$7,
• Full 8D: $j=1,\ldots, n$8 (indicative of strong clustering and heavy tails).

Further p-adic experiments show that for the Iris dataset ($j=1,\ldots, n$9 and $H_k(x) = 0.y_1 y_2 \ldots y_{kn}$0 with disjunctive encodings), the scaled index uses 30–40% of the static index space at moderate dimensions and virtually 0% at very high dimensions. For random point clouds in $H_k(x) = 0.y_1 y_2 \ldots y_{kn}$1 of size $H_k(x) = 0.y_1 y_2 \ldots y_{kn}$2, the scaled index remains more efficient for both uniform and normal densities, with $H_k(x) = 0.y_1 y_2 \ldots y_{kn}$3 values (scaled/static storage) rising with bucket size but always lower for uniform data (Bradley et al., 2019).

Table: Storage Ratio $H_k(x) = 0.y_1 y_2 \ldots y_{kn}$4 and Local Sparsity $H_k(x) = 0.y_1 y_2 \ldots y_{kn}$5 for Example Data (Jahn et al., 2019, Bradley et al., 2019)

Data/projection	$H_k(x) = 0.y_1 y_2 \ldots y_{kn}$6 (scaled)	$H_k(x) = 0.y_1 y_2 \ldots y_{kn}$7 (scaled/static)
$H_k(x) = 0.y_1 y_2 \ldots y_{kn}$8	$H_k(x) = 0.y_1 y_2 \ldots y_{kn}$9	$n$0–$n$1
$n$2	$n$3	–
Full 8D	$n$4	–
Iris $n$5	$n$6	$n$7–$n$8
Iris $n$9	$i$0	$i$1

A plausible implication is that the index's adaptability yields practical order-of-magnitude storage savings on typical real-world data with non-uniform density.

5. Locality, Query Performance, and Structural Visualization

The scaled Gray–Hilbert index retains the spatial locality preservation of Hilbert curves, critical for efficient range and k-nearest-neighbor queries. Query time correlates with tree height in dense regions; however, the reduced overall size frequently leads to improved query performance by shortening traversal paths. Binary-tree structures differ markedly: the static index yields perfectly balanced trees with uniform leaf depth, while the scaled index displays deep branches ("tentacles") in dense regions and shallow branches in sparse areas. Visualization of these binary trees, with color encoding depth and glyph size indicating occupancy, elucidates the adaptation of the index to various tail distributions in the data (Jahn et al., 2019).

6. Extensions and p-adic Generalizations

Beyond the binary ($i$2) case, p-adic scaled Gray–Hilbert indices exploit p-adic reflected Gray codes and affine transformations to define higher-arity, higher-dimensional space-filling curves. The scaling principle and analytical results transfer directly: for fixed bucket size, the space consumption of the scaled index vanishes exponentially compared to the static index as dimension increases. Full algorithms, complexity analysis (time $i$3 for $i$4-level depth, $i$5 dimensions), and storage efficiency bounds are provided, confirming the generality and efficacy of this approach for high-dimensional indexing tasks (Bradley et al., 2019).

7. Interpretation, Limitations, and Research Significance

The scaled Gray–Hilbert index represents a straightforward generalization of the classical Hilbert-curve approach, replacing global depth control with local, data-driven subdivision governed by a tunable parameter, the bucket capacity $i$6. Empirical and theoretical results consistently demonstrate substantial storage and, often, performance gains for real and synthetic high-dimensional datasets with heterogeneous density profiles (Jahn et al., 2019, Bradley et al., 2019). Since locality preservation is not compromised, and the method yields a fine-grained measure ($i$7) for data sparsity and tail structure, it is well-suited to modern spatial data workloads in both academic and applied contexts. The core limitations relate to the complexity of local permutation management in the p-adic case and the potential space–query trade-off at extreme bucket-size values, but these remain well-controlled within the framework established by current research.