Popularity-Weighted Information Budget Allocation (PIBA)

• PIBA is an information-theoretic approach that assigns variable-length semantic IDs based on item popularity, ensuring popular items use shorter IDs while tail items receive longer codes.
• It derives a closed-form scaling law, L_i* ∝ p_i^(-α/γ), balancing collaborative signals and semantic fidelity to address the Popularity-Length Paradox.
• By integrating PIBA as a prior in the Soft Length Controller of VarLenRec, the method significantly improves recommendation performance compared to fixed-length tokenization.

Popularity-Weighted Information Budget Allocation (PIBA) is the information-theoretic core of VarLenRec, a framework introduced in "Learning Variable-Length Tokenization for Generative Recommendation" for learning variable-length tokenization in generative recommendation (Wang et al., 18 May 2026). In this setting, recommendation is reformulated as next-token prediction over discrete semantic identifiers, and PIBA specifies how semantic ID length should be allocated across items with different popularity. The central claim is that existing fixed-length tokenization schemes impose uniform encoding capacity regardless of item characteristics, whereas the optimal per-item length $L_i^*$ should scale as a negative power of item popularity $p_i$. PIBA formalizes this through a trade-off between semantic fidelity and code-length cost, derives a closed-form scaling law for optimal length allocation, and supplies the prior that guides the Soft Length Controller in VarLenRec (Wang et al., 18 May 2026).

1. Position within generative recommendation

Generative recommendation reformulates recommendation as next-token prediction over discrete semantic identifiers. Within that formulation, a fundamental yet unexplored design choice is the use of fixed-length tokenization for all items, which implicitly assumes uniform encoding capacity regardless of item characteristics (Wang et al., 18 May 2026). Systematic experiments across four datasets identify a "Popularity-Length Paradox": popular items achieve optimal performance with short IDs, while tail items require substantially longer codes to capture discriminative semantics.

The stated interpretation is that popular items benefit from abundant collaborative signals and require minimal semantic detail, whereas tail items must rely on fine-grained content features due to sparse interaction data. PIBA is designed to resolve this mismatch. Rather than treating ID length as globally fixed, it formulates length allocation as an information-budgeting problem in which collaborative signals and semantic bits jointly determine how much representation capacity an item requires.

This framing suggests that token length in generative recommendation is not merely an architectural hyperparameter but a variable tied to the statistical regime of each item. In the terminology of the source paper, PIBA provides the principled prior for variable-length ID assignment, and VarLenRec uses that prior to learn adaptive encoding capacity.

2. Information-theoretic formulation

PIBA begins with an item $i$ having content feature $X$ and popularity $p_i \in (0,1)$, with $\sum_i p_i = 1$. The item is assigned a variable-length semantic ID

$Z_{1:L_i} = (z^{(1)}, \dots, z^{(L_i)}).$

Two sources contribute to the total information needed to distinguish the item: collaborative signals $I_{\collab}(p_i)$, arising from how often the item appears in user histories, and semantic bits $I_{\semantic}(L_i)$, provided by a length-$L_i$ quantized code (Wang et al., 18 May 2026).

The construction posits a minimal required information budget $p_i$0 in bits needed to identify an item. Collaborative information grows submodularly in popularity: $p_i$1 Here, $p_i$2 controls the rate of information gain per log-popularity and $p_i$3 scales the baseline.

Semantic information from a residual quantization ID of length $p_i$4 follows the harmonic-sum model: $p_i$5 where $p_i$6 is codebook size and layers exhibit diminishing returns. The semantic gap that remains after accounting for collaborative information is then

$p_i$7

PIBA formulates a global objective that trades off semantic fidelity against expected code-length: $p_i$8 Under the hard constraint that semantic bits exactly cover the gap, $p_i$9. This defines the optimal allocation problem that PIBA solves.

3. Closed-form scaling law and interpretation

Substituting the harmonic approximation into the exact-cover condition yields

$i$0

and therefore

$i$1

For sufficiently popular items satisfying $i$2, the approximation $i$3 gives

$i$4

This is stated as Theorem 1: optimal length scales as a negative power of popularity (Wang et al., 18 May 2026).

The exact LaTeX expression is

$i$5

Its interpretation is explicit. The factor $i$6 implies that tail items, for which $i$7 is small, increase length, while head items, for which $i$8 is large, decrease length. The ratio $i$9 determines the sharpness of adaptation: larger $X$0, corresponding to more collaborative information, or smaller $X$1, corresponding to slower semantic-bit growth, steepens the curve. The constant $X$2 sets the overall scale, for example through the minimal budget $X$3 and codebook granularity $X$4.

The underlying intuition follows directly from the semantic gap: $X$5 Head items with large $X$6 accumulate substantial collaborative information, so their semantic gap is small and only a few quantization layers are necessary. Tail items with tiny $X$7 receive little collaborative information, so the gap is large and many quantization layers are required to furnish the missing bits. In the paper’s wording, abundant user–item interactions make popular items easily recognizable with coarse codes, while rare items must rely on fine-grained content semantics.

4. Injection into the Soft Length Controller

PIBA is not only a theoretical statement about optimal length; it is also used as a training prior for the Soft Length Controller in VarLenRec. During training, the continuous target $X$8 is converted into a discrete binary mask: $X$9 where $p_i \in (0,1)$0 is the maximum allowed length (Wang et al., 18 May 2026).

At each layer $p_i \in (0,1)$1, the Soft Length Controller produces a retention probability $p_i \in (0,1)$2. The cumulative mask is defined as

$p_i \in (0,1)$3

A PIBA-based regularizer is then added: $p_i \in (0,1)$4 This binary cross-entropy loss steers the controller’s soft gating decisions to match the PIBA prior $p_i \in (0,1)$5, while still allowing content-adaptive deviations when beneficial.

The source identifies two technical challenges in directly implementing variable-length allocation. First, standard Euclidean residual quantization lacks geometric capacity to support diverse code lengths without distortion. Second, discrete length decisions are non-differentiable. VarLenRec addresses these through Hyperbolic Residual Quantization, which leverages the exponential volume growth of the Poincaré ball to naturally stratify encoding capacity, and through the Soft Length Controller, which enables differentiable length prediction via continuous layer retention probabilities regularized by PIBA-derived priors. This places PIBA at the interface between theory and architecture: it supplies the target length distribution, while the controller implements a differentiable approximation to that target.

5. Empirical ablations and observed effects

The empirical role of PIBA is isolated through ablations on the Beauty and Toys datasets. The reported results show that removing the length-alignment loss $p_i \in (0,1)$6 while keeping the rest of VarLenRec leads to a 3–7 % absolute drop in NDCG@10. Replacing the Soft Controller with a fixed truncation at $p_i \in (0,1)$7, that is, direct PIBA assignment, loses another 3–5 % compared to the learned gating. Comparing against any single fixed length $p_i \in (0,1)$8 under-performs by up to 8 % in recall or NDCG (Wang et al., 18 May 2026).

Ablation or comparison	Reported effect
Remove $p_i \in (0,1)$9	3–7 % absolute drop in NDCG@10
Fixed truncation at $\sum_i p_i = 1$0 instead of Soft Controller	Another 3–5 % loss versus learned gating
Any single fixed length $\sum_i p_i = 1$1	Under-performed by up to 8 % in recall or NDCG

Beyond aggregate metrics, Figure 1 in Appendix C.3 stratifies performance by popularity tier. Fixed-length IDs exhibit the "Popularity-Length Paradox": head items peak with very short $\sum_i p_i = 1$2, while tail items require very long $\sum_i p_i = 1$3, with no single choice satisfying both. In contrast, PIBA-guided VarLenRec simultaneously improves head by +41.3 % and tail by +6.8 % in NDCG@10 relative to the best fixed length.

These ablations support two distinct conclusions. First, the PIBA prior itself contributes materially, since removing the alignment loss degrades performance. Second, learning soft gating around that prior is preferable to direct assignment, since fixed truncation at the PIBA-derived length underperforms the controller-based implementation. The paper therefore presents PIBA not as a standalone heuristic but as a quantitatively accurate prior whose effectiveness depends on differentiable architectural integration.

6. Conceptual implications and relation to fixed-length tokenization

PIBA directly challenges the assumption that semantic IDs should have fixed length across items. The paper’s central empirical finding is that fixed-length tokenization creates a mismatch between the encoding needs of head and tail items: head items already possess abundant collaborative information and therefore require minimal semantic detail, whereas tail items depend on fine-grained content features due to sparse interaction data (Wang et al., 18 May 2026). The resulting "Popularity-Length Paradox" shows that no single fixed length can satisfy both regimes simultaneously.

A common misconception would be to treat longer IDs as uniformly better because they carry more semantic detail. PIBA rejects that view through its explicit code-length penalty and through the observation that popular items can achieve optimal performance with short IDs. Conversely, another misconception would be to regard collaborative signals as sufficient for all items. The semantic-gap formulation makes clear that low-popularity items receive little collaborative information and therefore need longer codes.

The broader implication is that encoding capacity should be adapted to the information profile of each item. This suggests a division of labor between collaborative information and semantic information: the former grows submodularly with popularity, while the latter is supplied through a residual quantization code with diminishing returns across layers. PIBA balances these terms and thereby provides the criterion for allocating token length under an information budget.

Within VarLenRec, this logic also explains why Hyperbolic Residual Quantization and the Soft Length Controller are paired with PIBA. A plausible implication is that the variable-length principle requires both a geometric substrate capable of supporting diverse code lengths and a differentiable mechanism for matching item-specific targets. The source paper states this in operational terms by identifying geometric capacity and non-differentiability as the two technical barriers to direct implementation.

7. Role within VarLenRec and stated significance

PIBA is presented as the information-theoretic core of VarLenRec. The paper’s sequence of argument is explicit: it first shows how PIBA formulates length allocation as a trade-off between semantic fidelity and code-length cost, then derives in closed form that the optimal per-item length $\sum_i p_i = 1$4 scales as a negative power of item popularity $\sum_i p_i = 1$5, then explains how these theoretical priors are injected into the Soft Length Controller, and finally summarizes ablations isolating PIBA’s impact on recommendation performance (Wang et al., 18 May 2026).

The work reports that VarLenRec achieves significant improvements over state-of-the-art methods in recommendation accuracy and training/inference efficiency, revealing the importance of adaptive encoding capacity in generative recommendation. At the level of PIBA specifically, the combined theoretical derivation, controller regularization, and ablation evidence are used to argue that variable-length ID assignment should be understood as a principled consequence of item popularity rather than as an ad hoc design choice.

In that sense, PIBA occupies a narrowly defined but foundational role. It does not replace the broader architecture of VarLenRec; rather, it specifies the quantitative prior governing how much semantic capacity each item should receive. The resulting principle can be stated succinctly in the form of Theorem 1: $\sum_i p_i = 1$6 Within the scope of the source paper, this scaling law is the formal resolution of the Popularity-Length Paradox and the mechanism by which adaptive tokenization is justified in generative recommendation.