LoST: Level-of-Semantics Tokenization

• LoST is a tokenization paradigm that orders tokens by semantic salience, encoding global to local details for various data modalities.
• It utilizes prefix-decodable sequences and semantic alignment techniques to enable coherent, coarse-to-fine generative and retrieval tasks.
• Empirical results in 3D shapes, time series, molecules, images, and recommendation systems demonstrate significant performance gains and compression over traditional methods.

Level-of-Semantics Tokenization (LoST) refers to tokenization paradigms that encode and order discrete representations according to their semantic salience, such that each token or prefix of tokens encapsulates a coherent, interpretable level of meaning relevant to the target modality. Unlike conventional tokenizers that prioritize spatial locality (in images, 3D, or time series) or atomic granularity (e.g., atom-level for molecules), LoST schemes explicitly integrate and structure semantic information—ranging from global, high-level concepts to fine, local or instance-level details—directly into the sequence of tokens. LoST approaches have demonstrated substantial improvements across generative modeling, masked modeling, sequence-to-sequence tasks, retrieval, and autoregressive generation by aligning tokenization with human-interpretable or domain-driven semantic hierarchies (Dutt et al., 18 Mar 2026, Wei et al., 27 Nov 2025, Kim et al., 30 Aug 2025, Qu et al., 17 Mar 2026, Zhuang et al., 27 Oct 2025, Mathisen et al., 2024).

1. Key Principles of Level-of-Semantics Tokenization

LoST is predicated on the following core principles:

1. Semantic Hierarchy: Tokens are ordered such that early tokens capture broad, high-level semantics (e.g., object category, principal motif, domain function), and subsequent tokens refine instance-level or local detail. This is in contrast to spatial or atomic sequencing which lacks explicit semantic ordering (Dutt et al., 18 Mar 2026, Wei et al., 27 Nov 2025).
2. Prefix-Decodability: Any leading prefix of the token sequence should decode to a coherent, plausible object or representation that preserves the main semantics, with further tokens incrementally adding refinement. This property enables early-stopped decoding, coarse-to-fine editing, semantic truncation, and efficient inference (Dutt et al., 18 Mar 2026, Wei et al., 27 Nov 2025, Qu et al., 17 Mar 2026).
3. Semantic Alignment: LoST schemes are trained with explicit constraints to ensure that token representations align with external or learned semantic feature spaces. These may be derived from expert ontologies, multimodal encoders, or domain-specific SSL objectives, ensuring meaningful token clusters (Dutt et al., 18 Mar 2026, Kim et al., 30 Aug 2025, Qu et al., 17 Mar 2026, Zhuang et al., 27 Oct 2025).
4. Structured Token Generation: In autoregressive or masked generation, the LLM (LM) is guided to decode tokens in semantic order, often from coarse to fine, enhancing the model’s ability to capture user intent or preserve semantic consistency during generation (Wei et al., 27 Nov 2025, Mathisen et al., 2024).
5. Code Efficiency: LoST yields highly compact code streams—requiring orders of magnitude fewer tokens than spatial or atomic tokenizers for similar or better fidelity, due to semantic front-loading and efficient coding (Dutt et al., 18 Mar 2026, Qu et al., 17 Mar 2026, Kim et al., 30 Aug 2025).

2. Architectures and Mathematical Formulation Across Modalities

2.1. 3D Shapes

The LoST framework for 3D shapes (Dutt et al., 18 Mar 2026) uses "register tokens" to absorb information from a VAE triplane latent via ViT encoding. Nested dropout with causal masking ensures that earlier tokens capture maximal global semantic content. Prefixes as short as 1–4 tokens recover principal category information, while 512 tokens allow for high-fidelity instance-level reconstruction.

Semantic alignment is enforced via Relational Inter-Distance Alignment (RIDA), aligning the structure of 3D token features with a DINO v2 image feature space:

• Loss:

$$\mathcal{L}_{\mathrm{RIDA}} = \lambda_g \mathcal{L}_{\mathrm{global}} + \lambda_r \mathcal{L}_{\mathrm{rank}} + \lambda_s \mathcal{L}_{\mathrm{spatial}}$$

which combines global InfoNCE, inter-instance rank distillation, and spatial structure distillation.

2.2. Time Series

NC-VQVAE implements LoST for time series (Mathisen et al., 2024), merging a VQ-VAE encoder–decoder pipeline with a Siamese branch for non-contrastive SSL. The quantized codes z_q are aligned with continuous representations of augmented views z′ using Barlow Twins or VIbCReg losses:

• SSL objective (Barlow Twins):

$$L_{BT} = \sum_{i=1}^D (1 - C_{ii})^2 + \lambda \sum_{i=1}^D \sum_{j \neq i} C_{ij}^2$$

ensuring that codebook entries capture both local structure and global, augmentation-invariant dynamics.

2.3. Molecules

CAMT5 motifs tokenizer segments molecular graphs into substructures—cycles, non-single-bond fragments, and singleton atoms—using graph-based motif extraction and depth-first linearization (Kim et al., 30 Aug 2025). Each motif maps to a unique token, replacing traditional atom-level tokenization. Importance weights λ(M) for each motif, based on number of atoms, guide a weighted MLM pre-training objective:

$$L_{CAMT5} = \sum_{i=1}^n \lambda(M_i) \cdot L_{MLM}(M_i)$$

2.4. Images

SemTok tokenizes 2D images into a compact 1D sequence of "semantic tokens" using a dual-stream MMDiT encoder with semantic alignment to SigLIP features (Qu et al., 17 Mar 2026). Tokens are quantized using a Binary Spherical Quantizer to maximize discrete codebook utilization, with distillation and contrastive losses enforcing high-level semantics.

2.5. Web Items and Recommendation

CoFiRec decomposes items into semantic levels (category, title, description, collaborative filtering) (Wei et al., 27 Nov 2025). Each level is quantized independently with separate codebooks, and decoding is explicitly ordered:

$$P\left(s^{(1)}, ..., s^{(L)} \mid \mathcal{H}_u\right) = \prod_{k=1}^L P\left(s^{(k)} \mid s^{(<k)}, \mathcal{H}_u\right)$$

with per-level ranking-guided cross-entropy losses and theoretical analysis of expected dissimilarity reduction vs. flat tokenization.

3. Semantic Hierarchy and Information Fusion

A defining property of LoST is the integration of multiple semantic levels in a structured, often hierarchical, order:

• Low-level/local semantics: Captured by original VQ-VAE codebooks (e.g., short waveform fragments, atomic bonds, spatial grids).
• High-level/global semantics: Injected via alignment to domain-specific features (e.g., DINO v2, SigLIP, expert-annotated motifs, functional domains).
• Prefix semantics: Any prefix, by construction, decodes into valid, principal semantic content (object category, core motif, coarse geometry), enabling any-prefix synthesis and semantic-controlled refinement (Dutt et al., 18 Mar 2026, Wei et al., 27 Nov 2025).

In many implementations, losses and architectural constraints ensure that tokens encoding earlier in the sequence control broad meaning, while later tokens refine details without overriding global structure.

The fusion of local and global representations yields discrete tokens that are robust to deformations, augmentations, or occlusions, and are less sensitive to input noise or idiosyncratic features compared to atomic or locality-based codes (Qu et al., 17 Mar 2026, Mathisen et al., 2024, Zhuang et al., 27 Oct 2025).

4. Training and Inference Methodologies

LoST models typically employ multi-stage training procedures corresponding to (a) semantic tokenizer training and (b) prior/autoregressive modeling:

• Stage 1 (Tokenizer): Learn semantic representations using a combination of reconstruction, commitment, semantic alignment (e.g., with RIDA or SSL losses), and importance weighting.
• Stage 2 (Prior or AR Model): Train masked or autoregressive transformers (e.g., MaskGIT, bidirectional AR) on the semantic token streams using order-preserving objectives.
• Inference: Enables masked inpainting, autoregressive sampling, and prefix decoding for flexible granularity in downstream synthesis or retrieval tasks (Dutt et al., 18 Mar 2026, Wei et al., 27 Nov 2025, Mathisen et al., 2024).

In biomolecular settings, LoST systems may bypass raw sequence embedding entirely, instead mapping curated context (Pfam, GO terms, motif descriptions) directly into high-density semantic tokens, avoiding the noise and misalignment induced by residue-level sequences (Zhuang et al., 27 Oct 2025).

Representative Table: Multi-Modality LoST Tokenizer Characteristics

Modality	Semantic Granularity	Token Ordering
3D Shapes	Category → Subtype → Instance	Prefix (nested dropout)
Molecules	Motif (cycles/fragments)	DFS over motif tree
Time Series	Shape fragment → Dynamics	Sequence, fused by SSL
Images	Scene/global → Texture/local	1D, semantics-aligned
Recommendation	Category → Metadata → CF signal	Coarse-to-fine levels
Biomolecules	Domain/GO/Function	Domain-driven list

5. Quantitative Impact and Empirical Results

LoST tokenization achieves substantial gains over spatially or atomically structured baselines, often with significant compression:

• 3D Shapes: At 512 tokens, LoST achieves a Chamfer Distance of 0.234e-2, FID of 13.59, DINO score of 0.921—outperforming baselines by large margins with orders-of-magnitude fewer tokens (Dutt et al., 18 Mar 2026).
• Time Series: NC-VQVAE lifts SVM accuracy on UCR datasets (e.g., FordA: 0.74 → 0.89), Inception Score (FordA: 1.16 → 1.45), and reduces FID (FordA: 5.15 → 2.28), with improved t-SNE clustering (Mathisen et al., 2024).
• Molecules: CAMT5 motif tokens improve exact match and RDK fingerprint similarity by 20–30% over atom-level models, using only ∼2% as many training tokens (Kim et al., 30 Aug 2025).
• Images: SemTok achieves reconstruction rFID of 0.88 vs. 1.45 (baseline); AR generation gFID=2.34 and IS=310.5 with only 256 tokens, surpassing FlexTok, FlowMo-Lo, and MaskGIT (Qu et al., 17 Mar 2026).
• Recommendation: CoFiRec’s LoST tokenization yields +6–10% Recall@10/NDCG@10 and reduces code collision rate by an order of magnitude compared to flat quantization (Wei et al., 27 Nov 2025).
• Biomolecules: High-level context-only input outperforms sequence-only by ∼40 LLM-Score points; ARI versus UniClust50 clusters is maximized (0.958) when using semantic context vs. sequence tokens (Zhuang et al., 27 Oct 2025).

6. Domain-Specific Considerations and Extensions

• Biological Sequences: Direct sequence tokenization is eclipsed by high-level semantic annotation, such as Pfam/GO/ProTrek tokens, leading to superior performance on protein reasoning and EC number prediction (Zhuang et al., 27 Oct 2025).
• Chemistry: Importance-weighted motif tokenization is superior to frequency or size-based alternatives, and DFS over the motif tree produces better representations than BFS (Kim et al., 30 Aug 2025).
• Recommendation: Explicitly preserving the semantic hierarchy in both tokenization and decoding order empirically and theoretically lowers dissimilarity and amplifies cold-start performance (Wei et al., 27 Nov 2025).
• Vision/3D: Alignment to off-the-shelf transformer-based multimodal features (DINO, SigLIP) is critical for ensuring that tokens are clusterable and human-interpretable (Dutt et al., 18 Mar 2026, Qu et al., 17 Mar 2026).

Potential future directions include multigranular or nested LoST hierarchies, cross-modal fusion with large multimodal backbones, and in-context adaptation to evolving expert ontologies or open vocabulary expansions.

7. Theoretical Analysis and Guarantees

Structured LoST tokenization is theoretically justified to reduce expected prediction error versus unstructured (flat) tokenization. In hierarchical decoding for a tree with $L$ levels and $V$ tokens per level, the expected dissimilarity under LoST

$\mathbb{E}_\text{hier}[d] = L - \sum_{k=1}^L p^k$

is strictly less than under independent prediction

$$\mathbb{E}_\text{indep}[d] = \left(L - \sum_{k=1}^L \frac{1}{V^k}\right) \frac{1 - p^L}{1 - 1/V^L}$$

whenever per-level accuracy $p > 1/V$, reflecting the compounding advantages of hierarchical organization (Wei et al., 27 Nov 2025).

The above results are robust across tasks, architectures, and domains, indicating that semantic-level ordering and alignment directly encode the statistical structure of real-world data, ultimately yielding more interpretable, efficient, and performant generative and reasoning models.