Hyperbolic Residual Quantization

• Hyperbolic Residual Quantization is a method that replaces Euclidean operations with hyperbolic counterparts to align discrete representations with latent hierarchies.
• It employs gyrovector operations like Möbius addition and subtraction to iteratively quantize residuals within the Poincaré ball model.
• Empirical evaluations show up to 20% improvements in hierarchy modeling, enhancing code selection and reconstruction in tasks such as WordNet and recommendation systems.

Hyperbolic Residual Quantization (HRQ) is a residual quantization method for learning discrete multitoken representations of data with latent hierarchies by replacing Euclidean operations with hyperbolic counterparts in the quantization pipeline. Introduced as an extension of residual quantization (RQ), HRQ embeds data in a hyperbolic manifold, performs nearest-neighbor codeword selection with a hyperbolic distance, and updates residuals and partial reconstructions with manifold-native operations. The resulting discrete identifiers are intended to inherit an inductive bias aligned with hierarchical branching, addressing the mismatch between Euclidean geometry and tree-like structure that arises in standard RQ (Piękos et al., 18 May 2025).

1. Conceptual basis and motivation

Residual quantization in Euclidean space constructs a multitoken representation by iteratively quantizing the residual of a vector relative to a partial reconstruction. With codebooks $C = [C_0, \dots, C_{k-1}]$, each $C_i$ containing $s$ vectors in $\mathbb{R}^h$, Euclidean RQ selects codewords greedily so as to minimize reconstruction error, producing a reconstruction $y = e_0 + \dots + e_{k-1}$ and a token sequence $[t_0, \dots, t_{k-1}]$ (Piękos et al., 18 May 2025).

In the formulation reported for RQ, the greedy objective at level $k$ selects $c_k \in C_k$ to minimize

$\|x - (c_1 + \dots + c_k)\|_2,$

with residual

$r_k = x - (c_1 + \dots + c_k).$

In RQ-VAE, codebooks are learned end-to-end with an autoencoder and the residual-commitment loss

$C_i$0

where $C_i$1 denotes stop-gradient (Piękos et al., 18 May 2025).

The motivation for HRQ is geometric. Euclidean volume grows polynomially with radius, whereas hierarchical branching grows exponentially with depth. The paper argues that this mismatch hinders faithful representation of hierarchical data, even though Euclidean RQ induces a hierarchical structure in the code selection sequence. Hyperbolic geometry, by contrast, has constant negative curvature and exponential volume expansion, and is therefore more naturally aligned with tree-like organization (Piękos et al., 18 May 2025).

This leads to the central claim of HRQ: by adapting the embedding network, residual computation, and distance metric to hyperbolic geometry, the learned discrete tokens can better capture latent hierarchies and become more useful for downstream tasks than Euclidean RQ tokens (Piękos et al., 18 May 2025).

2. Hyperbolic geometry and manifold structure

HRQ uses the $C_i$2-dimensional Poincaré ball model with curvature $C_i$3. The manifold is

$C_i$4

and its Riemannian metric is conformal to the Euclidean metric with conformal factor

$C_i$5

For nearest-neighbor search in codebooks, HRQ uses the hyperbolic distance

$C_i$6

and equivalently

$C_i$7

These definitions supply the metric structure underlying codeword selection and manifold-native residual updates (Piękos et al., 18 May 2025).

HRQ replaces Euclidean arithmetic with gyrovector operations. The Möbius addition used in the method is

$C_i$8

and Möbius subtraction is defined as

$C_i$9

The exponential and logarithmic maps at base point $s$0 are

$s$1

and

$s$2

These maps transfer vectors between the manifold and the tangent space $s$3, making it possible to implement hyperbolic neural network layers and stable optimization in the Poincaré ball (Piękos et al., 18 May 2025).

The geometric intuition emphasized in the work is that negative curvature induces an inductive bias matching tree-like branching. Two effects are highlighted. First, radial organization places cluster centers, interpreted as common ancestors, closer to the origin while pushing leaves outward. Second, branch separation increases because metric expansion away from the center enlarges distances across distinct branches while preserving relative proximity within a branch. In Appendix D, the paper connects the Fréchet mean under hyperbolic distance to an analogue of the nearest common ancestor and reports that hierarchical clustering in hyperbolic space places cluster centers nearer the origin than their leaves (Piękos et al., 18 May 2025).

3. Residual quantization in the hyperbolic manifold

HRQ performs residual quantization directly in the manifold rather than in Euclidean latent space. The input is first represented in the Poincaré ball, and the quantizer then iteratively selects codewords from level-specific codebooks using hyperbolic nearest-neighbor search. The procedure initializes the residual as $s$4 in $s$5 and the partial reconstruction as $s$6, the origin. At level $s$7, it selects $s$8 and token $s$9 by nearest neighbor under $\mathbb{R}^h$0, updates the residual via Möbius subtraction,

$\mathbb{R}^h$1

and updates the partial reconstruction via Möbius addition,

$\mathbb{R}^h$2

After $\mathbb{R}^h$3 levels, the method returns the multitoken $\mathbb{R}^h$4 and the hyperbolic reconstruction $\mathbb{R}^h$5 (Piękos et al., 18 May 2025).

This construction mirrors Euclidean RQ structurally while changing the ambient geometry. The novelty claimed for HRQ is not merely the use of hyperbolic embeddings before quantization, but the full residual quantization procedure in hyperbolic space: manifold-native residuals, hyperbolic nearest neighbors, and hyperbolic autoencoding (Piękos et al., 18 May 2025).

The encoder and decoder are also adapted to the manifold. Inputs $\mathbb{R}^h$6 are first mapped onto the manifold through the exponential map at the origin, $\mathbb{R}^h$7, then processed by hyperbolic layers relying on combinations of logarithmic and exponential maps with gyrovector operations. Decoding back to Euclidean space uses $\mathbb{R}^h$8. In the notation reported for HRQ-VAE, the encoder $\mathbb{R}^h$9 maps in-manifold embeddings, the decoder $y = e_0 + \dots + e_{k-1}$0 maps hyperbolic latents back, and the quantizer operates within $y = e_0 + \dots + e_{k-1}$1 (Piękos et al., 18 May 2025).

A practical implication is that the entire representation pathway is geometry-consistent: embedding, quantization, residual computation, and reconstruction all use operations compatible with the same negatively curved manifold. The paper presents this consistency as the mechanism by which hierarchical structure is better preserved in the learned discrete codes (Piękos et al., 18 May 2025).

4. Objective, optimization, and implementation

HRQ adopts the RQ-VAE residual-commitment loss unchanged in form,

$y = e_0 + \dots + e_{k-1}$2

but the residuals and codebook vectors live in $y = e_0 + \dots + e_{k-1}$3 and nearest neighbors are computed with the hyperbolic distance $y = e_0 + \dots + e_{k-1}$4 (Piękos et al., 18 May 2025).

The end-to-end HRQ-VAE training procedure reported in the paper proceeds in five stages. First, given Euclidean input $y = e_0 + \dots + e_{k-1}$5, the model computes $y = e_0 + \dots + e_{k-1}$6 and encodes it hyperbolically as $y = e_0 + \dots + e_{k-1}$7, which lies either in $y = e_0 + \dots + e_{k-1}$8 or $y = e_0 + \dots + e_{k-1}$9 depending on the design. Second, it performs level-wise codeword selection by initializing $[t_0, \dots, t_{k-1}]$0 and $[t_0, \dots, t_{k-1}]$1, then selecting codewords by hyperbolic nearest neighbor. Third, it updates residuals and partial reconstructions with Möbius subtraction and addition. Fourth, it decodes via $[t_0, \dots, t_{k-1}]$2, converts back to Euclidean space by $[t_0, \dots, t_{k-1}]$3, computes reconstruction loss $[t_0, \dots, t_{k-1}]$4, and optimizes the total loss

$[t_0, \dots, t_{k-1}]$5

Finally, it returns the multitoken representation (Piękos et al., 18 May 2025).

The main training-time difference from Euclidean RQ-VAE lies in optimization. HRQ uses Riemannian optimizers, such as Riemannian SGD or Riemannian Adam, for hyperbolic parameters including embeddings and codebooks, while Euclidean parameters use standard optimizers. The paper states that the curvature $[t_0, \dots, t_{k-1}]$6 is treated as a hyperparameter in experiments and is not learned in the reported runs (Piękos et al., 18 May 2025).

Several implementation constraints are highlighted. Operations are executed within the ball using exponential and logarithmic maps and Riemannian retractions; in practice, one keeps $[t_0, \dots, t_{k-1}]$7 and clips norms near the boundary to avoid numerical issues. The reported complexity for quantization is $[t_0, \dots, t_{k-1}]$8 per datapoint, since each level performs nearest-neighbor search over $[t_0, \dots, t_{k-1}]$9 codewords with hyperbolic distance of the same asymptotic cost as Euclidean distance, albeit with additional manifold-map overhead (Piękos et al., 18 May 2025).

The paper also provides practical guidance for reproducibility. It recommends normalizing vectors before the exponential map, using batched $k$0 or $k$1 formulations for distance, and considering approximate nearest-neighbor search adapted to hyperbolic metrics when codebook size is large. These recommendations reflect numerical and computational considerations rather than changes to the underlying algorithm (Piękos et al., 18 May 2025).

5. Empirical evaluation

The reported experiments evaluate HRQ against Euclidean RQ in two distinct scenarios: supervised hierarchy modeling using WordNet hypernym trees, and hierarchy discovery without explicit hierarchical supervision using recommendation datasets (Piękos et al., 18 May 2025).

For hierarchy modeling, the dataset is the WordNet nouns subnetwork with 82,115 nouns and 743,241 hypernymy edges, using an 85% train and 15% test split of relations. Training learns multitokens by embedding nouns and optimizing a contrastive objective on $k$2 together with the quantization loss:

$k$3

where $k$4 is Euclidean distance for RQ and hyperbolic distance for HRQ, and $k$5 is a sampled set of negatives not in hypernymy with $k$6. The downstream task fixes the learned multitokens and trains a sequence-to-sequence transformer to map a noun’s multitoken to its hypernym’s multitoken, evaluated with Recall@10 on the held-out 15% edges (Piękos et al., 18 May 2025).

For this WordNet setting, the reported hyperparameters are token lengths $k$7, codebook sizes $k$8, and embedding dimensions $k$9. Encoders are optimized with SGD or Riemannian SGD, learning rate $c_k \in C_k$0, for 1500 epochs with a 20-epoch warmup at $c_k \in C_k$1. The downstream transformer has 4 encoder and 4 decoder layers, hidden size 256, FFN size 1024, 8 heads, tied embeddings, and is trained for 100 epochs with Adam at learning rate $c_k \in C_k$2 (Piękos et al., 18 May 2025).

The principal result is that HRQ multitokens consistently outperform RQ multitokens in hierarchy modeling, with gains of up to about 20% in Recall@10. The paper states that, for example, with $c_k \in C_k$3 and $c_k \in C_k$4, HRQ sustains higher Recall@10 across embedding dimensions and remains more stable across dimensionality than RQ. Table 1 reports improvements such as $c_k \in C_k$5, $c_k \in C_k$6, and $c_k \in C_k$7, with typical gains between $c_k \in C_k$8 and $c_k \in C_k$9 depending on $\|x - (c_1 + \dots + c_k)\|_2,$0 and $\|x - (c_1 + \dots + c_k)\|_2,$1 (Piękos et al., 18 May 2025).

For hierarchy discovery, the datasets are Amazon Reviews 2014 subsets and MovieLens 10M. The reported sizes are Beauty with 22,363 users and 12,101 items, Toys & Games with 35,598 users and 18,357 items, Sports & Outdoors with 19,412 users and 11,924 items, and MovieLens 10M with 71,567 users and 10,681 items. For MovieLens, item descriptions are generated with Claude 3.5 Sonnet and all item descriptions are embedded by MPNet (768-D). HRQ-VAE and RQ-VAE are trained on item embeddings without direct hierarchical supervision. The encoder has 3 layers of sizes 512, 256, and 128 with hyperbolic ReLU activations, output latent size 32, and a symmetric decoder. Codebook size is 256 per token and codebooks are not shared across levels. Training uses 5000 epochs, batch size 128, and learning rates in $\|x - (c_1 + \dots + c_k)\|_2,$2 chosen by validation performance on the downstream task. The downstream recommender is a T5-like transformer encoder-decoder that predicts the next item multitoken from previous history, with histories limited to 20, filtered to at least 5 interactions, and evaluated using leave-one-out splitting. Metrics are Recall@$\|x - (c_1 + \dots + c_k)\|_2,$3 and NDCG@$\|x - (c_1 + \dots + c_k)\|_2,$4 for $\|x - (c_1 + \dots + c_k)\|_2,$5 (Piękos et al., 18 May 2025).

The reported recommendation results show consistent advantages for HRQ-VAE over Euclidean RQ-VAE and random tokens. On Amazon Beauty, NDCG@5 is 2.41% versus 2.29%, and Recall@10 is 5.01% versus 4.83%. On Amazon Toys & Games, Recall@10 is 4.53% versus 4.22%, and NDCG@5 is 1.94% versus 1.91%. On Amazon Sports & Outdoors, Recall@10 is 2.85% versus 2.78%, and Recall@5 is 1.62% versus 1.58%. On MovieLens 10M, NDCG@10 is 14.27% versus 13.89%, and Recall@10 is 25.49% versus 25.11% (Piękos et al., 18 May 2025).

6. Interpretation, related methods, and limitations

The paper attributes HRQ’s gains to geometry-aligned code selection. In supervised hierarchy modeling, the hyperbolic metric is reported to align codebook selection with tree-like structure by placing ancestors closer to the origin and leaves farther out, thereby yielding multitokens that reflect ancestor-descendant relations more faithfully than Euclidean sums. In hierarchy discovery tasks, HRQ is said to encourage consistent radial stratification and clearer branch separation, improving the semantic coherence of discrete codes and making them better labels for sequence prediction (Piękos et al., 18 May 2025).

The work also reports a geometric comparison of learned latent spaces by examining the variability of norms of low-dimensional vectors before quantization. After mapping hyperbolic vectors to the tangent space via the logarithmic map, the mean norm is reported as 0.7213 for Euclidean RQ-VAE and 0.3251 for HRQ-VAE; the standard deviation is 0.2696 versus 0.0664; and the coefficient of variation, defined as $\|x - (c_1 + \dots + c_k)\|_2,$6, is 0.3738 versus 0.2042. The interpretation offered is that lower variability under HRQ-VAE supports a more regular, radially structured space in which norms are less exploited for cluster selection, which the authors argue benefits gradient-based learning and generalization (Piękos et al., 18 May 2025).

HRQ is positioned at the intersection of three research lines: hyperbolic embeddings and hyperbolic neural networks for hierarchical data, discrete quantization models such as VQ-VAE and RQ-VAE, and vector quantization in hyperbolic space such as HyperVQ. The paper describes HyperVQ as complementary to HRQ’s residual approach and states that the two can be combined. HRQ’s specific novelty is the full residual quantization procedure carried out in hyperbolic space rather than a Euclidean residual scheme applied to hyperbolic embeddings only (Piękos et al., 18 May 2025).

Several limitations are stated explicitly. The reported claims focus on data with latent hierarchies and a need for discrete tokens; broader modalities such as images and audio are left for future exploration. Numerical stability remains a concern near the ball boundary, requiring robust implementations of exponential and logarithmic maps and retractions, along with norm clipping or $\|x - (c_1 + \dots + c_k)\|_2,$7-buffers. Riemannian optimization introduces overhead, and efficient batched distance computations or approximate nearest-neighbor search in hyperbolic space may be useful. The impact on purely non-hierarchical domains is described as unclear, and the paper identifies learnable curvature, hybrid manifolds, mixed-manifold designs, and adaptive level depth as promising future directions (Piękos et al., 18 May 2025).

A common misunderstanding would be to treat HRQ as a universally superior replacement for Euclidean RQ. The paper does not make that claim. Its empirical and conceptual arguments are specific to data with latent hierarchies, where negative curvature provides an inductive bias that Euclidean geometry lacks. This suggests that the significance of HRQ lies less in quantization alone than in the alignment between discrete coding and the geometry of the latent structure it is intended to represent (Piękos et al., 18 May 2025).