Variable-Length Semantic Identifiers

• Variable-length semantic identifiers are dynamic representations that adjust token counts to match content complexity, ambiguity, and task demand.
• They employ diverse methods—from discrete code sequences and continuous embeddings to natural language tokens—to enhance generative recommendation and cross-modal retrieval.
• Adaptive mechanisms like confidence-driven stopping and residual refinement efficiently allocate tokens, optimizing performance in search, IoT, and semantic communication.

Searching arXiv for the cited papers to ground the article and citations. arXiv search query: [Variable-Length Semantic IDs](https://www.emergentmind.com/topics/variable-length-semantic-ids) [semantic identifiers](https://www.emergentmind.com/topics/semantic-identifiers-sids) recommender systems Variable-length semantic identifiers are item, document, image, message, or device representations whose semantic description length is not fixed a priori, but varies with content, ambiguity, popularity, task demand, or confidence. Across the recent literature, the term covers several distinct but related constructions: discrete semantic ID sequences for generative recommendation, structured natural-language concept strings for multimodal retrieval, continuous latent token sequences for images, sequential feature packets for semantic communication, and compact context-coded identifiers for DNS-based discovery. What unifies them is that the identifier is intended to preserve semantic structure while allocating more or fewer tokens, symbols, or fields as needed, rather than assigning every entity the same flat identifier budget (Li et al., 22 Sep 2025, Chiu et al., 2023, Mao et al., 4 Jun 2025, Khrylchenko, 18 Feb 2026, Wang et al., 18 May 2026).

1. From fixed-length semantic IDs to variable-length formulations

Semantic IDs emerged as an alternative to arbitrary numeric IDs in generative retrieval and recommendation. In this line of work, an item is represented by a short sequence of discrete tokens derived from embeddings, so that similar items share parts of their identifiers and a LLM can both consume and generate item references as tokens. Two influential fixed-length formulations make the distinction clear. In "Semantic IDs for Joint Generative Search and Recommendation," an item is represented by a fixed-length multi-token code produced by residual quantization; the default configuration uses two codebooks of size 256 each, yielding exactly 2 tokens per item, and the paper explicitly states that there is no per-item variable-length semantic ID (Penha et al., 14 Aug 2025). In "LLMs As Semantic Indexers," semantic IDs are sequential and hierarchical, but the implementation fixes the length at $T=3$, even though the autoregressive formulation could in principle support variable stopping (Jin et al., 2023).

This distinction matters because a common misconception is to equate multi-token semantic IDs with variable-length semantic IDs. The former only require more than one token per entity; the latter require that the number of tokens itself be adaptive. The recommendation literature initially emphasized fixed-length hierarchical or residual codes, while later work imported explicit length adaptation from emergent communication, hyperbolic quantization, capsule routing, and multimodal generation (Penha et al., 14 Aug 2025, Jin et al., 2023).

A second conceptual shift concerns what “semantic” means. In the fixed-length recommender setting, semantics usually come from content or behavior embeddings discretized into codewords. In later variable-length work, semantics may instead be expressed directly in natural-language concept tokens, or in progressively refined continuous latent tokens, or in selected coordinates of a learned feature vector. This broadens the notion of semantic identifiers from discrete codebooks to any structured representation whose units are semantically grounded and whose description length can vary.

2. Main representation families

The literature now contains several distinct families of variable-length semantic identifiers, differing in whether the tokens are discrete or continuous, natural-language or learned codewords, and whether length is determined by EOS, confidence, residual error, or an explicit probabilistic latent variable (Li et al., 22 Sep 2025, Chiu et al., 2023, Mariani et al., 26 Mar 2025, Mao et al., 4 Jun 2025, Fernandez et al., 2021).

Family	Identifier form	Length mechanism
Structured semantic identifiers	Hyphen-separated concept tokens or phrases	Autoregressive stopping or output-format boundary
Variable-length discrete semantic IDs	Codebook token prefixes	Latent length variable, confidence stopping, or soft layer retention
Variable-length continuous embeddings	Continuous latent token sequence	Residual iterations or EOS prediction
Feature-identification messages	Sequence of feature index/value packets	Stop when semantic confidence exceeds threshold
Context-coded DNS/IoT identifiers	Context bits plus semantic fields	Number of used fields or prefix precision

The natural-language formulation is exemplified by "MLLM-Driven Semantic Identifier Generation for Generative Cross-Modal Retrieval," where a Structured Semantic Identifier is a short sequence $s_i=(c_{i,1},\dots,c_{i,L_i})$ of concept-level tokens or short phrases such as cheerleaders - chairs - spectators or woman - kitten kiss; $L_i$ is variable across images and is governed by autoregressive stopping and prompt instructions favoring a “short and unique identifier” (Li et al., 22 Sep 2025). This family is vocabulary-efficient because it uses only existing MLLM tokens rather than adding one token per item.

A continuous-latent formulation appears in "Variable Length Embeddings," where an image is represented as a sequence $z=(z_1,\dots,z_T)$, with each $z_n\in\mathbb{R}^d$, generated autoregressively from residuals. The number of tokens is bounded during training but variable in principle, and later tokens refine what earlier tokens have not reconstructed. In the masked variant, each token is associated with a learned mask, so the sequence behaves like a structured identifier over distinct semantic regions rather than a single monolithic embedding (Chiu et al., 2023).

"Images are Worth Variable Length of Representations" extends this idea with DOVE, where a dynamic vision encoder generates continuous visual tokens $d_1,\dots,d_T$ and predicts EOS, so each image receives an effective representation length $T$ that correlates with complexity. Q-DOVE further conditions tokenization on a text query, producing query-relevant variable-length token sequences (Mao et al., 4 Jun 2025).

A different formulation appears in semantic communication. "Semantic Communications via Features Identification" represents a message by a fixed-length identity vector $\mathbf{i}=\{f(1),\dots,f(N)\}$, but the transmitted semantic identifier is a variable-length sequence of packets $(\text{index},\text{value})$, sent until posterior confidence over the semantic element exceeds a threshold $\lambda$. Here the internal identity is fixed, while the communicated semantic description is variable-length (Mariani et al., 26 Mar 2025).

Finally, "Semantic Identifiers and DNS Names for IoT" defines compact semantic identifiers as bit strings composed of a Context and several semantic fields, then base32-encodes them into DNS labels. Length varies with the number of fields or the precision of the geographic or hierarchical prefix, so shorter prefixes denote coarser semantics and longer ones denote finer ones (Fernandez et al., 2021).

3. Mechanisms for constructing and length-controlling identifiers

The central technical problem is to decide how semantic content is discretized or tokenized and when the identifier should stop. Different communities solve this with distinct mechanisms, but most can be read as variants of rate allocation, residual refinement, or adaptive stopping (Khrylchenko, 18 Feb 2026, Cheng et al., 6 May 2026, Wang et al., 18 May 2026, Zhang et al., 2023).

One explicit probabilistic treatment is "Variable-Length Semantic IDs for Recommender Systems." For each item embedding $s_i=(c_{i,1},\dots,c_{i,L_i})$0, the model introduces a latent length $s_i=(c_{i,1},\dots,c_{i,L_i})$1 and a discrete message $s_i=(c_{i,1},\dots,c_{i,L_i})$2, with only the prefix $s_i=(c_{i,1},\dots,c_{i,L_i})$3 meaningful. The generative model is

$s_i=(c_{i,1},\dots,c_{i,L_i})$4

with a truncated geometric prior over $s_i=(c_{i,1},\dots,c_{i,L_i})$5, a uniform prior over active vocabulary positions, and a reconstruction term aggregated over prefixes. The training objective decomposes into reconstruction, vocabulary regularization, and length regularization, the latter taking the form $s_i=(c_{i,1},\dots,c_{i,L_i})$6. The use of Gumbel-Softmax makes the discrete token decisions differentiable and avoids REINFORCE instability (Khrylchenko, 18 Feb 2026).

VarLenRec reaches a similar goal with a different mechanism. It introduces Hyperbolic Residual Quantization inside a Poincaré ball, motivated by exponential volume growth in hyperbolic space, and a Soft Length Controller with per-layer retention probabilities $s_i=(c_{i,1},\dots,c_{i,L_i})$7. The cumulative mask

$s_i=(c_{i,1},\dots,c_{i,L_i})$8

enforces prefix-closed lengths, while a PIBA-derived prior regularizes the controller toward popularity-dependent target lengths. The information-theoretic result $s_i=(c_{i,1},\dots,c_{i,L_i})$9 makes popularity an explicit driver of identifier length (Wang et al., 18 May 2026).

CapsID replaces hard residual quantization with capsule routing. At each layer, an item residual is softly assigned to several semantic capsules, the residual is updated by the routed reconstruction, and the SID terminates when confidence is high enough or the residual norm is sufficiently small. The stopping rule is

$L_i$0

with $L_i$1. This makes length a function of semantic certainty rather than a fixed hyperparameter (Cheng et al., 6 May 2026).

In multimodal reconstruction, DOVE uses EOS prediction guided by a reconstruction threshold, so variable length emerges from a competition between reconstruction fidelity and EOS encouragement. In semantic XR communications, VL-SCC uses a Rate Allocation Network to predict a scalar $L_i$2 or a spatial rate map $L_i$3, quantizes that rate, and uses mask generation plus straight-through estimators so that the number of transmitted semantic-channel symbols becomes differentiable and trainable end to end (Zhang et al., 2023).

These mechanisms differ in implementation, but they share a common principle: length is treated as an allocatable resource. More layers, tokens, or symbols are used when reconstruction error remains large, when residual semantics remain unexplained, or when the content prior predicts that more semantic budget is warranted.

4. Generative retrieval, recommendation, and multimodal use

Variable-length semantic identifiers matter because generative systems decode sequences, not items. A semantic ID is therefore both an internal representation and an output target. This makes sequence length directly relevant to token budget, beam search, trie constraints, and latency (Penha et al., 14 Aug 2025, Li et al., 22 Sep 2025, Khrylchenko, 18 Feb 2026, Chen et al., 29 May 2026).

The unified search-and-recommendation setting illustrates the precursor problem. In the fixed-length Flan-T5 system of "Semantic IDs for Joint Generative Search and Recommendation," search and recommendation prompts are mapped to shared or task-specific 2-token semantic IDs, and diversified beam search generates item IDs that are then mapped back via a dictionary. The paper explicitly notes that residual quantization naturally allows more stages and that the architecture is directly compatible with variable-length code sequences, even though the reported experiments remain fixed-length (Penha et al., 14 Aug 2025).

In cross-modal retrieval, the SSID pipeline is already natively variable-length. At indexing time, an MLLM generates caption: ... identifier: ... reason: ...; at retrieval time, a text query is mapped to an SSID and exact matching against pre-indexed IDs retrieves the image. A Trie over valid identifiers constrains the next token at each step, so the generated identifier must correspond to a real item. Because the IDs are phrase-like and semantically interpretable, the generation task is closer to ordinary language modeling than atomic item-token generation (Li et al., 22 Sep 2025).

Recommendation-focused variable-length systems exploit the same autoregressive interface. In the dVAE-based recommender framework, each item is replaced by a variable-length prefix $L_i$4, user histories become token streams over semantic IDs, and the downstream model is trained as a decoder-only transformer on those streams. VarLenRec similarly inserts variable-length token sequences into T5-style generative recommendation, but adds Trie-constrained decoding, collision resolution, and a length-bias correction score to avoid penalizing longer tail-item IDs (Khrylchenko, 18 Feb 2026).

A further refinement is that hierarchical semantic IDs themselves may require a dedicated encoder before they are given to an LLM. "LLMs Need Encoders for Semantic IDs Too" argues that a SID level token’s meaning depends on its prefix context and proposes PrefixMem, a prefix-conditioned SID encoder based on prefix n-gram memory tables. The work uses 5-level SIDs with 2048 codes per level and shows that flat token embeddings are poorly matched to this prefix-dependent modality, especially at deeper levels (Chen et al., 29 May 2026). Although the paper studies fixed-depth SIDs, the argument generalizes: variable-length hierarchical codes also require models that can represent context-dependent meaning at each level.

5. Empirical regularities and trade-offs

The empirical literature converges on several regularities. First, semantic structure matters more than raw vocabulary expansion. In the unified search-and-recommendation study, task-specific fixed-length IDs were strong only on their own task: Search-based IDs achieved Search R@30 $L_i$5 and Rec. R@30 $L_i$6, whereas Rec.-based IDs achieved Search R@30 $L_i$7 and Rec. R@30 $L_i$8. The best balanced fixed-length solution used a shared multi-task embedding space with RQ-KMeans, yielding Search R@30 $L_i$9 and Rec. R@30 $z=(z_1,\dots,z_T)$0; RQ-KMeans also outperformed dictionary encoding, ResidualLFQ, and RQ-VAE in the tokenizer ablation (Penha et al., 14 Aug 2025).

Second, natural-language variable-length semantic IDs can be highly effective when constrained decoding and rationale supervision are added. On Flickr30K, SSID achieved R@1/R@5/R@10 $z=(z_1,\dots,z_T)$1, and on MS-COCO it achieved $z=(z_1,\dots,z_T)$2. The rationale channel improved performance, with w/ Reason (Training + Inference) at $z=(z_1,\dots,z_T)$3, w/ Reason (Training Only) at $z=(z_1,\dots,z_T)$4, and w/o Reason (Training + Inference) at $z=(z_1,\dots,z_T)$5. Using Qwen2.5-VL, SSID reached Flickr30K R@1 $z=(z_1,\dots,z_T)$6, surpassing image tokenization at $z=(z_1,\dots,z_T)$7 while remaining vocabulary-efficient (Li et al., 22 Sep 2025).

Third, adaptive-length continuous representations track content complexity. DOVE reports a Pearson correlation of $z=(z_1,\dots,z_T)$8 between token length and Laplacian variance, and in VQA settings DOVE maintained strong performance with an average of $z=(z_1,\dots,z_T)$9 tokens, while Q-DOVE reduced this further to an average of $z_n\in\mathbb{R}^d$0 tokens by query conditioning. On ImageNet100 at 32 tokens, DOVE achieved FID $z_n\in\mathbb{R}^d$1 versus ALIT’s $z_n\in\mathbb{R}^d$2, and in linear probing on CIFAR-100 it reached $z_n\in\mathbb{R}^d$3 versus $z_n\in\mathbb{R}^d$4 for VQGAN and $z_n\in\mathbb{R}^d$5 for ALIT (Mao et al., 4 Jun 2025).

Fourth, recommendation repeatedly shows that head and tail items need different identifier lengths. The dVAE-based recommender study reports that on Yambda the average length under the catalog distribution was $z_n\in\mathbb{R}^d$6, while under the data distribution it dropped to $z_n\in\mathbb{R}^d$7, indicating that popular items receive shorter codes. Under a fixed 512-token history budget, recall improved from $z_n\in\mathbb{R}^d$8 to $z_n\in\mathbb{R}^d$9 on Yambda and from $d_1,\dots,d_T$0 to $d_1,\dots,d_T$1 on VK-LSVD as length penalties were introduced, because shorter IDs for frequent items freed budget for more events per sequence (Khrylchenko, 18 Feb 2026). VarLenRec names this phenomenon the Popularity-Length Paradox: head items prefer short IDs, tail items prefer long IDs, and a single fixed length is suboptimal for both. Its full system reduces collision rates to about $d_1,\dots,d_T$2 on average, reports average lengths of $d_1,\dots,d_T$3 on Toys, $d_1,\dots,d_T$4 on Beauty, $d_1,\dots,d_T$5 on Sports, and $d_1,\dots,d_T$6 on Yelp, and yields average training and inference time reductions of about $d_1,\dots,d_T$7 and $d_1,\dots,d_T$8 relative to fixed-length GR (Wang et al., 18 May 2026).

Fifth, soft routing and confidence-driven termination improve both accuracy and cost in generative recommendation. CapsID+SemanticBPE improves Recall@10 by $d_1,\dots,d_T$9 on average over ReSID and, on a 35M-item industrial catalog, reaches $T$0 of COBRA’s per-query latency while matching or exceeding its public-benchmark accuracy. On Beauty, ablating soft residual updates reduces R@10 from $T$1 to $T$2, using only one routing iteration reduces it to $T$3, and forcing fixed length $T$4 reduces it to $T$5. The largest gains occur on tail items, where the paper reports a $T$6 Recall@10 improvement over TIGER on Beauty (Cheng et al., 6 May 2026).

Taken together, these results support a common interpretation: variable-length identifiers are most beneficial when token budget is scarce, the entity distribution is heavy-tailed, and semantics are either compositional or multi-faceted. They do not merely compress sequences; they reallocate representational capacity toward difficult or weakly supervised cases.

6. Constraints, misconceptions, and open questions

The literature also defines clear limits. One misconception is that allowing variable length automatically increases namespace size or expressivity without cost. Coding theory shows otherwise. "On the maximum size of variable-length non-overlapping codes" proves that the size of a $T$7-ary variable-length non-overlapping code of length up to $T$8 is upper bounded by $T$9, the maximum size of a fixed-length non-overlapping code of length $\mathbf{i}=\{f(1),\dots,f(N)\}$0. It also shows that the average length is lower bounded by $\mathbf{i}=\{f(1),\dots,f(N)\}$1, and, when $\mathbf{i}=\{f(1),\dots,f(N)\}$2, is asymptotically no shorter than $\mathbf{i}=\{f(1),\dots,f(N)\}$3 as $\mathbf{i}=\{f(1),\dots,f(N)\}$4 (Wang et al., 2024). In other words, near-capacity code systems with strong non-overlap constraints remain almost fixed-length on average.

A second misconception is that variable length has already replaced fixed-length semantic IDs in mainstream retrieval. It has not. Several influential systems remain fixed-length, and some of the most compelling papers study variable-length identifiers precisely because prior work did not. The 2025 unified search-and-recommendation paper states that there is no per-item variable-length Semantic ID in its design, even though residual quantization could be extended in that direction (Penha et al., 14 Aug 2025). The same is true of LMIndexer’s fixed $\mathbf{i}=\{f(1),\dots,f(N)\}$5 setup (Jin et al., 2023).

A third issue concerns normalization and consistency. SSIDs can drift into full sentences, and synonyms such as “police officer” versus “cop” can yield multiple identifiers for similar items; the paper explicitly lists consistency and normalization as open questions (Li et al., 22 Sep 2025). In recommender systems, collisions, invalid generated codes, and prefix ambiguity remain deployment concerns, motivating Trie constraints, auxiliary disambiguation tokens, PrefixMem-style prefix encoders, and length-bias corrections (Wang et al., 18 May 2026, Chen et al., 29 May 2026).

Scalability is another persistent tension. Trie-based constrained decoding works well at tens of thousands of identifiers, but the cross-modal retrieval work notes that efficient data structures and pruning become critical for millions of IDs (Li et al., 22 Sep 2025). CapsID identifies dynamic capsule expansion and codebook refresh as future work under catalog growth (Cheng et al., 6 May 2026). PrefixMem shows that semantic IDs behave as a distinct modality with large prefix-conditioned memory requirements, suggesting that length adaptation alone is insufficient if the model cannot represent prefix-dependent semantics at depth (Chen et al., 29 May 2026).

A final practical perspective comes from IoT and DNS. Semantic identifiers can be made variable-length by construction through a Context field plus a variable number of semantic fields or a variable geo-prefix, and shorter prefixes deliberately encode coarser semantics. This design supports discovery and delegation with standard DNS infrastructure, but it also makes explicit that variable length is always a trade-off between compactness, precision, and query granularity (Fernandez et al., 2021).

These threads suggest a broad synthesis. Variable-length semantic identifiers are not a single algorithmic object but a design space. Some approaches vary the number of learned discrete codes, some vary natural-language concept sequences, some vary continuous latent tokens, and some vary transmitted feature packets or field prefixes. Across all of them, the unresolved questions are largely the same: how to allocate semantic budget efficiently, how to preserve interpretability and validity at scale, how to control stopping behavior without harming fidelity, and how to encode hierarchical or prefix-dependent semantics so that the downstream generator does not have to rediscover them from scratch.