Learning Variable-Length Tokenization for Generative Recommendation

Abstract: Generative recommendation reformulates recommendation as next-token prediction over discrete semantic identifiers (IDs). A fundamental yet unexplored design choice is that existing methods employ fixed-length tokenization for all items, implicitly assuming uniform encoding capacity regardless of item characteristics. Through systematic experiments across four datasets, we discover the Popularity-Length Paradox: popular items achieve optimal performance with short IDs, while tail items require substantially longer codes to capture discriminative semantics. This reveals a critical mismatch where popular items benefit from abundant collaborative signals and require minimal semantic detail, whereas tail items must rely on fine-grained content features due to sparse interaction data. To address this, we propose VarLenRec, a framework for learning variable-length tokenization. We develop Popularity-Weighted Information Budget Allocation (PIBA), an information-theoretic framework proving that optimal ID length should scale as a negative power of popularity. Directly implementing variable-length allocation faces two technical challenges: standard Euclidean residual quantization lacks geometric capacity to support diverse code lengths without distortion, and discrete length decisions are non-differentiable. We address these through Hyperbolic Residual Quantization, which leverages the exponential volume growth of the Poincaré ball to naturally stratify encoding capacity, and a Soft Length Controller, which enables differentiable length prediction via continuous layer retention probabilities regularized by PIBA-derived priors. Extensive experiments demonstrate that VarLenRec achieves significant improvements over state-of-the-art methods in recommendation accuracy and training/inference efficiency, revealing the importance of adaptive encoding capacity in generative recommendation.

• The paper reveals the Popularity-Length Paradox, showing that optimal token lengths vary with item popularity to ensure discriminative semantics.
• It introduces VarLenRec, an architecture combining hyperbolic residual quantization and a soft length controller for adaptive code allocation.
• Empirical results across datasets demonstrate reduced semantic collisions and improved NDCG@10 accuracy, outperforming fixed-length baselines.

Learning Variable-Length Tokenization for Generative Recommendation

Introduction and Motivation

Generative recommendation systems formulate recommendation as next-item prediction over discrete semantic item identifiers (IDs), aligning interaction modeling with advances in autoregressive sequence modeling and LLMs. Existing generative methods universally employ fixed-length tokenization: every item is assigned the same-length ID, irrespective of its popularity or semantic complexity. The paper "Learning Variable-Length Tokenization for Generative Recommendation" (2605.17779) demonstrates, through systematic evaluation, the inadequacy of this uniform-length design and identifies what it calls the Popularity-Length Paradox: popular (head) items attain best accuracy with minimal-length IDs, while rare (tail) items require longer codes to encode sufficient discriminative semantics due to the sparseness of collaborative signals.

This paradox is substantiated via empirical studies, which indicate that the optimal ID length is anti-correlated with item popularity—short for frequently interacted items, longer for rare ones—rendering fixed-length tokenization fundamentally suboptimal. The results motivate a paradigm shift: semantic ID length should be adaptively matched to item-specific information needs.

Theoretical Foundations: Popularity-Weighted Information Budget Allocation (PIBA)

The authors formalize length allocation from an information-theoretic viewpoint. They introduce the Popularity-Weighted Information Budget Allocation (PIBA), which theoretically derives the optimal ID length for an item as a negative power function of its popularity,

$L_i = C \cdot p_i^{-a/y},$

where $p_i$ is normalized popularity, $C$ and $a/y$ are constants determined by semantic capacity and redundancy rates, and $L_i$ is the optimal length for the $i$-th item.

This result emerges from a model in which the representation of an item must fill a minimum information budget through two complementary components: (1) collaborative information, which grows submodularly (logarithmic in frequency), and (2) semantic ID bits, which scale asymptotically as the logarithm of token length (consistent with rate-distortion and hierarchical quantization principles). The explicit anti-correlation between $L_i$ and $p_i$ underpins the main claim: encoding capacity should decrease monotonically with popularity. This quantifiable prescription provides a rigorous prior for downstream learning and enables the design of an adaptive tokenization architecture.

VarLenRec: Architecture and Learning Algorithm

To operationalize variable-length ID assignment within generative recommenders, the paper presents VarLenRec, a modular framework with four components:

1. Hyperbolic Residual Quantization (HARQ): Rather than relying on Euclidean quantization—which cannot accommodate the exponentially increasing cardinality and semantic granularity required by rare items—the authors employ hyperbolic geometry, specifically the Poincaré ball model. Hyperbolic space exhibits exponential volume growth with increasing radius, enabling a natural stratification: popular items cluster compactly near the origin with short codes; tail items are embedded near the boundary, receiving longer, more distinctive codes. The architecture supports multi-layer residual quantization in hyperbolic space, preserving hierarchical structure and semantic granularity.
2. Soft Length Controller: Discrete (integer) length selection is non-differentiable and hinders end-to-end learning. The soft controller predicts, at each codebook layer, the gate (retention) probability, regularized against the PIBA-derived prior, and implemented as cumulative gating masks over quantization layers. This mechanism enables gradient-based optimization of variable-length assignment and allows content-adaptive early stopping when sufficient information has been encoded.
3. ID Collision and Length Bias Mitigation: Variable-length IDs introduce issues absent in fixed-length settings, including ID collisions (sequence duplicates), hallucinations during autoregressive generation, and systematic length bias in scoring (shorter sequences are disproportionately favored). The framework integrates post-hoc collision disambiguation (using an auxiliary codebook), Trie-constrained decoding (to guarantee valid ID generation), and length-invariant rescoring (using an odds ratio criterion) to ensure retrieval accuracy and fair ranking across items with diverse token lengths.
4. Practical Integration and Efficiency: The system allows for efficient cold-start handling (assigning maximum-length codes to new items) and periodic reallocation for evolving catalogs, decoupling the lightweight HARQ tokenizer from the main recommendation model for scalable deployment.

Empirical Results

Experiments span four well-established recommendation datasets (Amazon Beauty, Sports, Toys, and Yelp). Integration of VarLenRec with state-of-the-art generative backbones (TIGER, LCRec) yields consistent improvements in NDCG@10, up to 4.7% over strong fixed-length baselines, with recall gains throughout. Notably, the improvement is uniform across head, body, and tail segments—a direct consequence of resolving the Popularity-Length Paradox.

VarLenRec dramatically reduces semantic ID collision rates (from 12.7% for fixed-length to 3.2% with variable-length on average) and accelerates both training (by 15.8%) and inference (by 19.3%) relative to prior state-of-the-art, owed to the fact that popular items dominate interaction frequency and are encoded with much shorter sequences.

Detailed ablation studies underscore the importance of each architectural component:

• Ablating length cost or replacing the soft controller with hard assignment harms performance, indicating that content-adaptive, differentiable length selection is crucial.
• Replacing hyperbolic quantization with Euclidean degrades accuracy by 6–7%, confirming the geometric argument for hyperbolic capacity.
• Removing downstream inference strategies leads to substantial retrieval errors and further loss in accuracy, verifying that practical adoption necessitates comprehensive integration support.

Implications and Future Directions

This work provides compelling evidence that encoding capacity heterogeneity is a critical, previously neglected axis in generative recommendation. The demonstration of the Popularity-Length Paradox and the associated theoretical solution via PIBA can be expected to generalize to other discrete representation learning tasks exhibiting long-tail effects.

From a practical standpoint, the reduction in ID length for popular items brings significant computational and latency gains, directly addressing efficiency bottlenecks for large-scale recommender systems without sacrificing—and, indeed, improving—accuracy on tail items.

On the theoretical side, the use of hyperbolic geometry for adaptive-capacity semantic tokenization sets a precedent for combining geometric prior structure with information-theoretic foundations in representation learning.

Potential future extensions include:

• Multimodal or cross-domain variable-length tokenization, capturing even richer semantic diversity.
• Adaptive allocation strategies informed by domain, category, temporal dynamics, or user group characteristics, not just popularity.
• Synergetic integration with foundation models and emerging LLM architectures, where compatibility and compositionality of semantic tokens may bring further performance and generalization gains.

Conclusion

The paper convincingly shows that variable-length semantic tokenization—guided by theoretical information budget allocation and operationalized in hyperbolic space—substantially improves the accuracy, uniqueness, and efficiency of generative recommender systems. The VarLenRec framework systematically addresses the challenges introduced by heterogeneity in item popularity and lays groundwork for future research on adaptive, information-efficient representation learning in large-scale AI systems.