MM-Tokenizer: Advances in Multimodal Tokenization

Updated 23 December 2025

MM-Tokenizer is a multimodal tokenization technique that transforms raw modalities, such as images and structured data, into discrete or continuous tokens while preserving semantic fidelity.
It employs adaptive-length tokenization, hierarchical codebooks, continuous latent spaces, and mixture-of-experts quantization to overcome limitations of fixed-length tokenizers.
Empirical evaluations demonstrate improved reconstruction fidelity, efficient token utilization, and state-of-the-art performance in visual understanding, generation, and cross-modal integration.

A multimodal tokenizer (MM-Tokenizer) is a key primitive in multimodal LLMs (MLLMs): it transforms raw modalities (such as images or structured data) into sequences of discrete or continuous tokens suitable for consumption or generation by autoregressive text-oriented models. Efficient MM-tokenization balances semantic fidelity, computational efficiency, and integration into autoregressive next-token prediction protocols. Recent research introduces techniques such as adaptive-length tokenization, hierarchical codebooks, continuous token spaces, and multi-expert quantization to resolve the unique challenges of representing visual, multimodal, and structured data as model tokens.

1. Motivation for Multimodal Tokenization and Limitations of Rigid Tokenizers

MM-tokenizers address the bottleneck that arises when mapping high-dimensional, variable-content modalities (images, masks, EHR codes, multimodal metadata) into regularized token sequences for MLLMs. Traditional vision tokenizers (e.g., VQGAN, VQVAE, TiTok, HiMTok) emit a fixed number of tokens per input segment, regardless of complexity or spatial content. This leads to inefficient over-tokenization of simple structures and insufficient representation for complex or fine-detailed regions, negatively impacting downstream autoregressive visual-textual modeling in both comprehension and generation tasks.

The ALTo approach demonstrates that by adaptively allocating token count per instance (e.g., mask), one can achieve superior trade-offs between accuracy (measured by metrics like IoU) and compute cost, matching or exceeding state-of-the-art mask segmentation with ~50% fewer tokens compared to fixed-length baselines. This paradigm shift mirrors human strategies: more “attention” is allocated for detailed subregions, while simple ones require less representation (Wang et al., 22 May 2025).

2. Architectures and Core Technical Innovations

2.1 Adaptive-Length Tokenization (ALTo)

ALTo introduces three core modules:

Mask Tokenizer (MT): Transformer encoder yields up to 32 latent tokens for a mask.
Token Length Predictor (TLP): Predicts the required token count $L \in [1,32]$ per input via expectation over a stopping-point distribution computed from the mask’s global context.
Mask De-Tokenizer (MD): Reconstructs the mask from only the first $L$ tokens.

A length regularization term and a straight-through differentiable soft-chunking operation enable backpropagation through the adaptive slicing of token streams. The chunking is accomplished by computing cumulative keep-probabilities $P_i$ over the predicted stopping-point probabilities $p_i$ , creating a mixture of “hard” token selection and differentiable masking (Wang et al., 22 May 2025).

2.2 Semantic-Guided Hierarchical Codebooks (SemHiTok)

SemHiTok proposes a two-level codebook:

Semantic-Priority Codebook (SPC): A global codebook, learned via quantization of features from a frozen semantic encoder, associating image patches primarily with semantic identity.
Pixel Sub-Codebooks: Each semantic code-index triggers its bespoke sub-codebook for pixel-level variations, decoupling high-level and low-level token learning.

This separation addresses the tradeoff between semantic degeneracy (ineffective for understanding) and pixel detail collapse (ineffective for generation) inherent in monolithic codebooks (Chen et al., 9 Mar 2025).

2.3 Continuous Latent Token Spaces (MingTok)

The MingTok tokenizer employs a three-stage architecture:

Low-Level Encoder: Compresses image patches into compact continuous vectors ( $\mathbb{R}^{d_z}$ ).
Semantic Expansion: Causally or non-causally maps low-level tokens to high-dimensional, semantic tokens.
Pixel Decoder: Upsamples and reconstructs high-fidelity pixels from expanded semantic tokens.

All understanding and generation operate within the same continuous latent space, letting the LLM both read and write image representations as continuous vectors, eliminating discrete quantization artifacts (Huang et al., 8 Oct 2025).

2.4 Mixture-of-Experts and Behavior Adaptation (MMQ)

MMQ splits tokenization into shared modality and modality-specific expert branches, each outputting latent vectors for quantization via a common or separate codebook. Soft indexing allows staircase gradient flow for post-training behavioral fine-tuning, enabling semantic IDs to track actual user behaviors while maintaining reconstruction fidelity and coverage—critical for recommendation or retrieval in multimodal item spaces (Xu et al., 21 Aug 2025).

2.5 Multimodal Code Tokenization (MedTok)

MedTok integrates both textual description and graph-relational context per code, projecting each to a shared embedding space and performing vector quantization. Cross-modal projections and alignment losses enforce preservation of orthogonal modality information and semantic linkage, crucial for extremely large structured sets (e.g., medical codes) (Su et al., 6 Feb 2025).

2.6 End-to-End Tokenizer Tuning (ETT)

ETT couples the vision tokenizer and downstream LLM into a single optimization, allowing visual codebook embeddings to directly receive gradients from language-driven objectives. This resolves the bottleneck where a frozen tokenizer’s representation is agnostic to downstream semantics, especially impactful for tasks that hinge on fine-grained visual discrimination (Wang et al., 15 May 2025).

3. Tokenization Workflows and Integration Paradigms

MM-tokenizers have matured to accommodate two main operational paradigms:

Discrete Scheme: Tokenization yields integer indices which are fed (or autoregressively generated) via LLMs or fusion transformers. This is typical for VQ-VAE- and codebook-based approaches (ALTo, SemHiTok, MedTok, MMQ). Hierarchical codebooks, token chunking, or multi-expert selection further extend this baseline.
Continuous Scheme: Tokenization produces continuous vectors which are projected (sometimes via MLP/projectors) into the LLM’s token embedding space. MingTok and ETT exemplify this, enabling end-to-end differentiability and seamless semantic adaptation.

Integration typically involves concatenating vision and/or textual tokens, importing special sentinel tokens to signal cross-modal transitions, and plugging into standard next-token prediction with no required changes to the LLM’s core architecture (though codebook size, tokenization stride, and fusion mechanism may vary) (Wang et al., 15 May 2025, Wang et al., 22 May 2025, Chen et al., 9 Mar 2025, Huang et al., 8 Oct 2025).

4. Empirical Evaluation and Impact

MM-tokenizers demonstrate marked improvements on both fine-grained visual understanding and flexible visual/textual generation.

Model	Image Recon (rFID↓)	Understanding (SEED-B↑)	Generation (GenEval↑)	Avg Token Usage
ALTo	n/a	n/a	n/a	17.5 (avg)
SemHiTok	1.24	84.2	0.66	256
MingTok	0.38	53.57	0.465	256
MMQ	n/a	n/a	n/a	6 (per item)
MedTok	n/a	n/a	n/a	~1-2 per code
ETT	1.65 (α=0.25)	60.0 (SEED)	0.43	—

ALTo achieves state-of-the-art performance with ~50% fewer mask tokens, matching or exceeding fixed-length baselines in IoU (Wang et al., 22 May 2025). SemHiTok delivers an rFID of 1.24 (best among fixed-length, unified tokenizers) and competitive results in both understanding and autoregressive image generation (Chen et al., 9 Mar 2025). MingTok’s continuous approach aligns generation and understanding, optimizing both semantic coherence and reconstruction accuracy under a single latent protocol (Huang et al., 8 Oct 2025). MMQ in recommender systems yields +32.7% Recall@5, +20.6% Recall@10 in industrial settings, and MedTok improves AUPRC on EHR prediction tasks by 4–11%, with +63% on inpatient drug recommendation (Xu et al., 21 Aug 2025, Su et al., 6 Feb 2025). ETT achieves consistent 2–6% performance increases in multimodal understanding and generation by aligning vision tokenizer semantics with downstream loss (Wang et al., 15 May 2025).

5. Optimization Techniques and Trade-Offs

Key innovations include:

Adaptive-length regularization: Explicit penalty on token count allows trading generation cost for reconstruction fidelity (Wang et al., 22 May 2025).
Hierarchical codebook training: Two-stage procedures decouple semantic and pixel codeword learning, preventing codebook collapse (Chen et al., 9 Mar 2025).
Soft/hard index interpolation: Straight-through soft index selection (as in MMQ) allows gradient signal propagation through quantization during behavioral tuning, reducing the semantic–behavior gap (Xu et al., 21 Aug 2025).
End-to-end gradient flow: Direct optimization of tokenizers with autoregressive model loss recovers semantic fidelity lost in frozen-tokenizer pipelines (ETT paradigm) (Wang et al., 15 May 2025).
Multi-expert and orthogonalization: Expert diversity and codebook utilization coverage are promoted by explicit orthogonality penalties and modular expert architectures (Xu et al., 21 Aug 2025).

Dominant trade-offs include balancing reconstruction (rFID, perceptual loss), semantic fidelity (for downstream tasks), and efficiency (token sequence length and GPU compute). Over-tuning for one metric (e.g., only semantic alignment) may degrade others if not regulated (as shown in α ablation in ETT) (Wang et al., 15 May 2025).

6. Application Domains and Generalization

Recent MM-tokenizers span a variety of domains:

Autoregressive mask and segmentation (ALTo) (Wang et al., 22 May 2025)
Unified image understanding and generation (SemHiTok, MingTok) (Chen et al., 9 Mar 2025, Huang et al., 8 Oct 2025)
Personalized recommender systems (MMQ, yielding high-quality semantic IDs) (Xu et al., 21 Aug 2025)
Clinical code tokenization for EHR prediction (MedTok) (Su et al., 6 Feb 2025)
End-to-end optimized visual LLMs (ETT) (Wang et al., 15 May 2025)
Medical radiology report generation with multi-scale, multi-modal fusion (μ²Tokenizer) (Li et al., 30 Jun 2025)

Most solutions generalize to both understanding and generation, with some (notably ALTo) demonstrating transfer to new segmentation and open-vocabulary tasks with minimal or zero-shot adaptation. Domain extension (e.g., RGB image adaptation for ALTo, higher resolution for SemHiTok, behavior-in-the-loop tuning for MMQ) is feasible with retraining or modest architectural adaptation.

7. Limitations and Future Directions

While MM-tokenizers provide substantial advances, key limitations persist:

Training complexity: Multi-stage, multi-loss regimes increase cost and tuning difficulty (ALTo, SemHiTok, MMQ).
Modality/domain specificity: Tokenizers (e.g., ALTo, SemHiTok) are often validated on specific modalities (masks, 256×256 images) and require retraining to adapt to color images, video, or point clouds.
Interpretability and token semantics: Continuous tokenizers and adaptive-length schemes may complicate interpretability in safety-critical settings.
Memory/computation: Some variants (e.g., those retaining dense pixel-encoder features at inference) increase resource requirements.

Potential research avenues include adaptive-length image tokenizers beyond masks, dynamic codebook scaling for higher resolutions, federated or content-driven sub-codebook sharing, and integration into in-context multi-modal reasoning with next-token prediction (Wang et al., 22 May 2025, Chen et al., 9 Mar 2025, Huang et al., 8 Oct 2025).

In summary, MM-tokenizer design is a rapidly advancing subfield resolving the inefficiencies and semantic bottlenecks of fixed rigid tokenization. Innovations such as adaptive-length prediction, hierarchical semantic–pixel decoupling, continuous latent codes, mixture-of-experts quantization, and end-to-end downstream-aligned optimization have collectively enabled MLLMs to better balance quality, efficiency, and domain transfer, unlocking new performance regimes and application verticals (Wang et al., 22 May 2025, Chen et al., 9 Mar 2025, Huang et al., 8 Oct 2025, Xu et al., 21 Aug 2025, Su et al., 6 Feb 2025, Wang et al., 15 May 2025).