VarLenRec: Adaptive Tokenization for Recommendations

• The paper presents a framework that allocates variable-length IDs to items, addressing the inefficiencies of fixed-length tokenization in recommendation systems.
• It combines PIBA to weight encoding by popularity, HRQ to utilize hyperbolic geometry for residual quantization, and SLC for differentiable, content-adaptive length prediction.
• Experiments demonstrate improved performance, reduced collisions, and faster inference compared to fixed-length methods, especially for diverse head–tail item distributions.

Searching arXiv for the VarLenRec paper and closely related generative recommendation/tokenization work to ground the article. VarLenRec is a framework for learning variable-length tokenization for generative recommendation. It is motivated by a basic design choice in generative recommendation: existing methods employ fixed-length tokenization for all items, implicitly assuming uniform encoding capacity regardless of item characteristics. The framework combines Popularity-Weighted Information Budget Allocation (PIBA), Hyperbolic Residual Quantization (HRQ), and a Soft Length Controller (SLC), and is integrated with a Transformer-based generative recommender through collision-safe decoding and length-bias correction (Wang et al., 18 May 2026).

1. Problem setting and motivating paradox

Generative recommendation reformulates recommendation as autoregressive next-token prediction over discrete item semantic identifiers (IDs). Each item $i$ is mapped to a sequence of discrete tokens $z_i$ drawn from learned codebooks, and the model predicts the next item by generating its ID given the user’s past ID sequence. In prior quantization-based tokenizers, including RQ-VAE variants, the number of layers $L$ is fixed for all items, so every item receives the same encoding capacity (Wang et al., 18 May 2026).

VarLenRec is built around the observation that this assumption is suboptimal for long-tail catalogs. Popular items accumulate abundant collaborative signals and need little semantic detail, while tail items require richer content semantics to compensate for sparse interactions. The paper terms the resulting empirical pattern the Popularity-Length Paradox. When recommendation performance is evaluated by partitioning target items into Head (top 20%), Body (middle 60%), and Tail (bottom 20%) popularity groups and sweeping fixed ID lengths $L \in \{4, 6, 8, 10\}$, performance for head items decreases as $L$ grows, while performance for tail items increases with $L$. The overall optimal fixed $L$ is a compromise that underperforms for every individual popularity group. This pattern is reported across Beauty, Sports, Toys, and Yelp (Wang et al., 18 May 2026).

A concise view of the framework is as follows.

Component	Core mechanism	Function
PIBA	popularity-weighted information allocation	induces a prior over ID lengths
HRQ	hyperbolic residual quantization	affords nonuniform capacity across code lengths
SLC	differentiable length prediction	trains variable-length IDs end-to-end

This decomposition reflects the paper’s central claim: adaptive encoding capacity is necessary because the information required to discriminate items is not uniform across the catalog. This suggests that fixed-length semantic IDs are not merely inefficient; they are structurally misaligned with the head–tail asymmetry in collaborative and content signals.

2. Popularity-Weighted Information Budget Allocation

PIBA formalizes how much information each item’s ID should carry as a function of its popularity, under a global budget constraint (Wang et al., 18 May 2026). Each item $i$ has normalized popularity $p_i \ge 0$, with $\sum_i p_i = 1$. The framework assumes that collaborative information grows submodularly with popularity,

$z_i$0

with $z_i$1 and $z_i$2.

Semantic information is modeled through residual quantization. The $z_i$3-th layer contributes diminishing returns of approximately $z_i$4 bits, with $z_i$5 for codebook size $z_i$6. The total semantic information is

$z_i$7

If $z_i$8 denotes the minimum information required to make an item discriminable, then the semantic ID must fill the gap

$z_i$9

Under a total length budget $L$0,

$L$1

a practical PIBA formulation maximizes total semantic information minus a linear length penalty while respecting both feasibility and budget constraints:

$L$2

For items that bind the constraint $L$3, the minimum feasible $L$4 is optimal. The resulting KKT solution is

$L$5

For $L$6, this becomes the closed-form power law

$L$7

with

$L$8

The paper presents this as the principal theoretical result of PIBA: optimal ID length should scale as a negative power of popularity. Intuitively, the exponent $L$9 emerges from balancing collaborative information, which is logarithmic in popularity, against semantic capacity, which is logarithmic in length.

PIBA also provides a prior over lengths. Popularity is measured from interaction counts $L \in \{4, 6, 8, 10\}$0 with smoothing,

$L \in \{4, 6, 8, 10\}$1

where $L \in \{4, 6, 8, 10\}$2 prevents zero-popularity instability. In practice, the paper reports that rank-based quantile mapping improves robustness:

$L \in \{4, 6, 8, 10\}$3

where $L \in \{4, 6, 8, 10\}$4 is the descending popularity rank and $L \in \{4, 6, 8, 10\}$5. This discrete prior is then used as supervision for SLC.

3. Hyperbolic Residual Quantization

Directly implementing variable-length allocation introduces a geometric difficulty: standard Euclidean residual quantization lacks geometric capacity to support diverse code lengths without distortion. In Euclidean space, ball volume grows polynomially, $L \in \{4, 6, 8, 10\}$6, whereas hierarchical item semantics exhibit exponential branch growth. As a result, longer residual stacks in Euclidean space must pack exponentially many fine-grained distinctions into polynomial capacity, leading to higher distortion and collisions for tail items (Wang et al., 18 May 2026).

HRQ addresses this by using hyperbolic geometry, specifically the Poincaré ball. For curvature $L \in \{4, 6, 8, 10\}$7 with $L \in \{4, 6, 8, 10\}$8, the $L \in \{4, 6, 8, 10\}$9-dimensional ball is

$L$0

The framework uses hyperbolic distance, Möbius addition, and the exponential and logarithmic maps at the origin:

$L$1

$L$2

$L$3

The geometric motivation is the exponential volume growth of hyperbolic space:

$L$4

This provides natural stratification across radii: short codes are placed near the origin and correspond to coarse semantics, while long codes can expand toward the boundary and represent fine semantics.

The HRQ pipeline uses $L$5 residual layers. An encoder maps content features $L$6 to Euclidean $L$7 via $L$8, then projects them to the manifold:

$L$9

At layer $L$0, the nearest code in codebook $L$1 is selected using hyperbolic distance,

$L$2

and the residual is updated via Möbius subtraction,

$L$3

The paper’s theoretical and empirical comparison is explicit. Hyperbolic embeddings preserve tree-like semantics with $L$4 distortion, while Euclidean embeddings suffer $L$5 distortion. Empirically, replacing hyperbolic HRQ with Euclidean RQ decreases Recall@10 by about $L$6–$L$7 on Beauty and Toys; collision rates increase. HRQ reduces collisions, for example from $L$8 to $L$9 on average, improves tail discrimination, and works synergistically with variable-length allocation (Wang et al., 18 May 2026).

In computational terms, tokenization per item is $L$0, because each of $L$1 layers performs nearest-code selection over $L$2 codes in $L$3 dimensions. Hyperbolic distance adds constant overhead via $L$4 and $L$5, and approximate nearest neighbor search is optional.

4. Soft Length Controller and optimization objective

SLC makes length prediction differentiable through continuous layer retention probabilities and regularizes them with PIBA-derived priors (Wang et al., 18 May 2026). For each layer $L$6,

$L$7

Prefix constraints are enforced by cumulative masks

$L$8

so $L$9 loosely represents the probability that layer $i$0 is kept given all prior gates are open.

The gate is content-adaptive. At layer $i$1, the current residual and selected code are projected to tangent space and passed to a small $i$2 with sigmoid:

$i$3

The gate closes early when residual semantics are already captured, and stays open for complex items.

PIBA supplies the supervisory prior. From popularity $i$4, a discrete target length $i$5 is computed via quantile mapping, and a binary target mask is formed as

$i$6

The cumulative masks are regularized toward this target with cross-entropy:

$i$7

HRQ and SLC are trained jointly with four losses. The reconstruction loss is

$i$8

The quantization/commitment loss, using stop-gradient $i$9 as in VQ-VAE, is

$p_i \ge 0$0

with $p_i \ge 0$1. The length cost is

$p_i \ge 0$2

The combined tokenizer objective is

$p_i \ge 0$3

At inference time, SLC uses $p_i \ge 0$4-thresholding, for example $p_i \ge 0$5, and the cumulative mask to select the truncation point:

$p_i \ge 0$6

The paper reports that calibration is minimal and $p_i \ge 0$7 is tuned on validation. A notable ablation is that using hard PIBA lengths without SLC underperforms by $p_i \ge 0$8–$p_i \ge 0$9, which indicates that the popularity prior alone is insufficient and that content-aware gating materially contributes to performance.

5. End-to-end architecture, training, and decoding

VarLenRec is an end-to-end pipeline from item features to a generative recommender (Wang et al., 18 May 2026). First, a content encoder maps item features, such as Sentence-T5 text embeddings, to the hyperbolic manifold. Second, HRQ with $\sum_i p_i = 1$0 residual layers and codebooks of size $\sum_i p_i = 1$1 performs layerwise hyperbolic quantization, while SLC predicts soft retention. Third, the tokenizer outputs variable-length discrete IDs as the selected code indices up to the learned truncation $\sum_i p_i = 1$2. Fourth, a Transformer/T5 decoder is trained on sequences of item IDs to predict next-item IDs.

The downstream generative objective is standard next-token loss:

$\sum_i p_i = 1$3

Training proceeds in two stages. In the HRQ+SLC stage, item features are encoded, quantized layer by layer, masked through cumulative gates, softly reconstructed, and optimized with Riemannian Adam. In the GR stage, token sequences are built from HRQ IDs, with collision resolution and EOS, and the LLM is trained with cross-entropy.

The representative settings reported are: codebook size $\sum_i p_i = 1$4, maximum length $\sum_i p_i = 1$5, curvature $\sum_i p_i = 1$6, and gate threshold $\sum_i p_i = 1$7. Riemannian Adam is used for HRQ with learning rate $\sum_i p_i = 1$8 and batch size $\sum_i p_i = 1$9; Adam is used for GR with learning rate $z_i$00 and batch size $z_i$01; beam search size is $z_i$02 at inference.

Variable-length IDs create specific decoding issues, and the paper therefore emphasizes three mechanisms. First, collision handling reserves an auxiliary disambiguation range, for example IDs in $z_i$03, to append to colliding short IDs and avoid semantic interference. Second, trie-constrained decoding builds a prefix tree of all valid IDs to avoid hallucinations. Third, length bias correction rescales decoded candidates with the length-invariant odds ratio

$z_i$04

The inference procedure is therefore: run constrained beam search over a prefix trie of valid IDs, decode until EOS, rescore candidates to correct length bias, and map decoded IDs to items.

6. Experimental results, practical considerations, and limitations

The experiments are conducted on Amazon Beauty, Sports and Outdoors, Toys and Games, and Yelp, with users and items having fewer than $z_i$05 interactions dropped. The splits are leave-one-out: last item as test, second last as validation, and the rest as train. Evaluation uses Recall@K and NDCG@K for $z_i$06 under a full ranking protocol (Wang et al., 18 May 2026).

Baselines include sequential recommenders—HGN, GRU4Rec, BERT4Rec, SASRec, FMLP, $z_i$07-Rec, and HSTU—as well as fixed-length GR tokenizers such as TIGER and ETEGRec, and enhanced GR tokenizers such as LETTER-TIGER and LETTER-LCRec. Across all datasets, VarLenRec consistently improves over fixed-length GR baselines and traditional sequential recommenders. With LCRec backbones, VarLenRec achieves top performance on Beauty, Sports, and Toys; with TIGER backbones, it tops Yelp. The reported gains reach up to about $z_i$08 in NDCG@10, and are strongest on semantically diverse catalogs such as Beauty and Toys.

The ablation results isolate the role of each component. No single fixed $z_i$09 is optimal; adaptive lengths outperform all fixed choices $z_i$10. Replacing hyperbolic HRQ with Euclidean RQ reduces Recall@10 by about $z_i$11–$z_i$12 on Beauty and Toys. Removing $z_i$13 or $z_i$14 degrades results, and $z_i$15 prevents unnecessarily long IDs. Removing collision resolution, trie decoding, and length-normalized scoring leads to about an $z_i$16 Recall@10 drop. These findings indicate that VarLenRec is not only a variable-length tokenizer, but a coupled system whose theoretical allocation rule, geometry, gating, and decoding strategy interact.

The headline result for the motivating paradox is reported on Beauty at $z_i$17: VarLenRec simultaneously improves head $z_i$18, body $z_i$19, and tail $z_i$20 groups versus fixed-length TIGER. In efficiency terms, VarLenRec reduces training time by approximately $z_i$21 and inference time by approximately $z_i$22 versus TIGER, due to shorter IDs for popular items and early termination during decoding.

Several implementation details are emphasized. Stability on the Poincaré ball is maintained by setting $z_i$23 and enforcing a safety margin $z_i$24 from the boundary. Popularity smoothing $z_i$25 stabilizes $z_i$26, and rank-based mapping reduces sensitivity to outliers. The paper recommends tuning $z_i$27 and $z_i$28 per dataset. Riemannian Adam is used to keep codebook vectors on the manifold, with updates applied through the exponential map and projection back into the ball interior if parameters approach the boundary.

The stated limitations are tied to the assumptions of the framework. PIBA assumes collaborative information grows as $z_i$29; if popularity dynamics or interaction quality violate this submodular trend, the exponent may mis-specify length. The quality of the content encoder matters for tail items, because poor content features reduce the benefit of longer IDs. Curvature choice $z_i$30 and codebook size $z_i$31 interact with capacity, so mis-tuning may cause underfitting near the origin or crowding near the boundary. Proposed extensions include adaptive budgets over time, user-conditioned lengths, multimodal content, and category-aware priors (Wang et al., 18 May 2026).

The paper does not provide a public repository link at the time of writing. Its recommended setup is Sentence-T5 embeddings for item features; HRQ with $z_i$32, $z_i$33, and $z_i$34; a T5 decoder trained with Adam at learning rate $z_i$35; beam size $z_i$36; trie-constrained decoding with odds-rescoring; and popularity estimated from normalized interaction counts with small-$z_i$37 smoothing and rank-based quantile mapping. Within that formulation, VarLenRec presents variable-length tokenization as a principled allocation problem, a hyperbolic quantization problem, and a differentiable control problem, unified within generative recommendation.