InfoTok: Info-Theoretic Tokenization Methods

• InfoTok comprises two distinct information-theoretic frameworks: one for shared visual tokenization in unified MLLMs and another for adaptive discrete video tokenization.
• The unified MLLM variant employs the Information Bottleneck principle to compress high-frequency noise and enhance cross-modal alignment, improving both image understanding and generation.
• The video variant dynamically allocates tokens based on per-sample ELBO estimates, optimizing compression by matching token lengths to the inherent information density of video frames.

Searching arXiv for the InfoTok papers and closely related context. arxiv_search(query="InfoTok", max_results=10, sort_by="submittedDate") arxiv_search(query="shared visual tokenization unified MLLMs Information Bottleneck", max_results=5, sort_by="relevance") InfoTok is the name of two information-theoretic tokenization frameworks introduced in recent arXiv literature. In "InfoTok: Regulating Information Flow for Capacity-Constrained Shared Visual Tokenization in Unified MLLMs" (Tang et al., 2 Feb 2026), InfoTok is an information-regularized visual tokenization mechanism for unified multimodal LLMs, grounded in the Information Bottleneck (IB) principle and designed for a shared token space that supports both image understanding and image generation. In "InfoTok: Adaptive Discrete Video Tokenizer via Information-Theoretic Compression" (Ye et al., 18 Dec 2025), InfoTok is an adaptive discrete video tokenizer that allocates token length according to informational richness using Shannon-style source-coding arguments and an ELBO-based router. The shared name reflects a common concern with regulating information flow under finite token budgets, but the two methods address distinct settings and should not be conflated.

1. Terminological scope and research setting

A recurring source of confusion is that "InfoTok" does not denote a single canonical method. The name has been used for two separate systems with different objects of optimization, different downstream tasks, and different mathematical machinery. One addresses shared visual tokens in unified MLLMs; the other addresses adaptive compression for discrete video tokenization.

InfoTok variant	Setting	Core mechanism
InfoTok (Tang et al., 2 Feb 2026)	Unified MLLMs with shared visual tokens for I2T and T2I	IB-based mutual-information regularization
InfoTok (Ye et al., 18 Dec 2025)	Discrete video tokenization for long video sequences	ELBO-based adaptive routing and compression

The unified-MLLM InfoTok is motivated by the claim that existing shared-token designs are mostly architecture-driven and lack an explicit criterion for what information tokens should preserve. The paper frames the visual tokenizer as a compute-bounded learner whose finite token budget should prioritize reusable structure, such as object entities, compositional layout, and generalizable style cues, rather than hard-to-exploit high-entropy variations and redundancy (Tang et al., 2 Feb 2026).

The video-tokenizer InfoTok begins from a different bottleneck: current tokenizers rigidly compress all content at a fixed rate despite variable information density across videos. Its central claim is that fixed-rate and data-agnostic adaptive tokenizers are theoretically suboptimal in representation length, and that adaptive token allocation should approximate the source entropy of each sample (Ye et al., 18 Dec 2025).

2. InfoTok for shared visual tokenization in unified MLLMs

The unified-MLLM formulation introduces the random variables $\mathcal{I}$ for the input image, $\mathcal{Z}$ for shared visual tokens produced by the tokenizer $V_{enc}$, $\tilde{\mathcal{Z}}$ for lightweight task-specific projections of $\mathcal{Z}$, $\mathcal{T}$ for textual tokens, and $\mathcal{Y}^{\mathcal{GT}}$ for ground-truth outputs. The method first recalls the single-task IB objective

$$\mathcal{L}_{\mathrm{IB}} = \inf_{p(\mathcal{Z}\mid \mathcal{I})} I(\mathcal{Z};\mathcal{I}) - \beta\,I(\mathcal{Z};\mathcal{Y}^{\mathcal{GT}}),$$

where $\beta>0$ balances compression against predictive sufficiency.

InfoTok extends this to dual tasks: image understanding and image generation. The resulting objectives are

$$\begin{aligned} \mathcal{L}_{\mathrm{IB}}^{(u)} &= I(\tilde{\mathcal{Z}}_u;\mathcal{I}_u) - \beta_u I(\tilde{\mathcal{Z}}_u;\mathcal{Y}_u^{\mathcal{GT}}) - \alpha_u I(\tilde{\mathcal{Z}}_u;\mathcal{T}_u),\ \mathcal{L}_{\mathrm{IB}}^{(g)} &= I(\tilde{\mathcal{Z}}_g;\mathcal{I}_g) - \beta_g I(\tilde{\mathcal{Z}}_g;\mathcal{Y}_g^{\mathcal{GT}}) - \alpha_g I(\tilde{\mathcal{Z}}_g;\mathcal{T}_g), \end{aligned}$$

with total regularizer

$\mathcal{Z}$0

The third mutual-information term, $\mathcal{Z}$1, explicitly promotes visual-text alignment through cross-modal consistency.

The practical estimators are variational. Compactness is controlled through the upper bound

$\mathcal{Z}$2

with $\mathcal{Z}$3 and $\mathcal{Z}$4. Sufficiency is lower-bounded by a variational decoder:

$\mathcal{Z}$5

Alignment is estimated with an InfoNCE objective,

$\mathcal{Z}$6

where $\mathcal{Z}$7 is the posterior mean, $\mathcal{Z}$8 is cosine similarity, and $\mathcal{Z}$9 is a temperature hyperparameter (Tang et al., 2 Feb 2026).

3. Training mechanism, architectural integration, and reported performance in unified MLLMs

During fine-tuning, the original multimodal objective is augmented as

$V_{enc}$0

where

$V_{enc}$1

The paper describes this as a plug-and-play fine-tuning procedure: InfoTok is applied atop existing shared-token unified MLLMs such as Harmon, OpenUni, and Show-o2 by adding two small projection heads $V_{enc}$2 during training, with no change to the core LLM or visual encoder at inference. Both continuous latent tokens and discrete codebook tokens are supported, and the bottleneck is realized by Gaussian sampling via reparameterization (Tang et al., 2 Feb 2026).

The evaluation suite spans understanding benchmarks GQA, SEED, POPE, MME, MMV2, MMMU, and UniBench; generation benchmarks GenEval, GenEval++, and WISE; and a shared-evaluation setting in which UniBench jointly tests image understanding and generation. Reported metrics include accuracy or F1, cross-modal normalized CKA, FID, GenEval overall score, and WISE world-knowledge alignment.

The quantitative results are reported as consistent improvements on both understanding and generation. For token quality, Harmon improves from FID 14.4 to 12.0 and CKA 0.24 to 0.27, while Show-o2 improves from FID 13.2 to 11.4 and CKA 0.23 to 0.28. For understanding, Harmon improves on MME from 1411 to 1548, POPE from 0.76 to 0.84, GQA from 0.51 to 0.59, and UniBench from 0.64 to 0.66; Show-o2 improves on MME from 1714 to 1823, SEED from 0.66 to 0.69, GQA from 0.58 to 0.61, and UniBench from 0.39 to 0.49. For generation, Harmon’s GenEval overall rises from 0.74 to 0.85 and WISE from 0.45 to 0.60; on GenEval++, Harmon improves from 0.19 to 0.46 and Show-o2 from 0.17 to 0.27. The paper further states that InfoTok-fine-tuned 1.5B models often match or exceed larger unified MLLMs (Tang et al., 2 Feb 2026).

Ablations compare full InfoTok, IB only (Compactness + Sufficiency), and vanilla fine-tuning. On Harmon, IB only yields large gains, including MME +92 and GenEval++ overall +0.43, while adding the Alignment term improves stability and peak metrics, including MME +137 and overall +0.27. Show-o2 follows a similar pattern, with IB alone boosting MME by 87 and GenEval++ overall by 0.20, and the alignment term adding another 22 points on GenEval++. The summary notes that exhaustive sweeps are not reported; it therefore presents the ranges $V_{enc}$3, $V_{enc}$4, and $V_{enc}$5 as practical guidance rather than as an exhaustive characterization.

4. InfoTok for adaptive discrete video tokenization

The video-tokenizer InfoTok is derived from classical source coding. For a discrete source with distribution $V_{enc}$6 and code alphabet of size $V_{enc}$7, the reported bound is

$V_{enc}$8

where $V_{enc}$9 is the codeword length. Fixed-rate tokenizers instead assign

$\tilde{\mathcal{Z}}$0

which is far from the entropy bound when $\tilde{\mathcal{Z}}$1 is nonuniform. Adaptive compression seeks $\tilde{\mathcal{Z}}$2.

The paper formalizes this through two theoretical claims. Theorem 2.1 states that there exists an adaptive scheme achieving

$\tilde{\mathcal{Z}}$3

Theorem 2.2 states that if an adaptive tokenizer is trained with a data-agnostic uniform router over $\tilde{\mathcal{Z}}$4, then for any constant $\tilde{\mathcal{Z}}$5 one can construct a distribution $\tilde{\mathcal{Z}}$6 such that

$\tilde{\mathcal{Z}}$7

so the expected token length can be arbitrarily larger than the information-theoretic optimum. The proof sketch models the compression pipeline as a $\tilde{\mathcal{Z}}$8-ary tree and argues that a uniform router biases the solution toward a tall tree with large average depth.

Because $\tilde{\mathcal{Z}}$9 is intractable in realistic video models, the method replaces it with a VAE evidence lower bound:

$\mathcal{Z}$0

Theorem 3.1 then defines the router

$\mathcal{Z}$1

for $\mathcal{Z}$2, and states that minimizing reconstruction loss under this router yields an expected length bounded by the entropy term plus an ELBO approximation error. In effect, token budget becomes proportional to per-sample negative ELBO, which functions as a proxy for informational richness (Ye et al., 18 Dec 2025).

5. ELBO-based algorithm, architecture, and systems details for video InfoTok

The training objective jointly optimizes VQ-VAE reconstruction and adaptive compression:

$\mathcal{Z}$3

The implementation sets $\mathcal{Z}$4 because the router is already information-theoretic. The algorithm computes $\mathcal{Z}$5, assigns

$\mathcal{Z}$6

selects the top-$\mathcal{Z}$7 frames or tokens by per-token ELBO, quantizes the retained representation, reconstructs the video, and backpropagates the negative log-likelihood plus KL term.

The architecture is explicitly specified. The encoder $\mathcal{Z}$8 is a 3D causal CNN using a Cosmos-DV backbone with temporal-spatial downsampling $\mathcal{Z}$9, output dimension $\mathcal{T}$0, and sequence length $\mathcal{T}$1. The router $\mathcal{T}$2 is an MLP that takes $\mathcal{T}$3 and maintains an EMA for $\mathcal{T}$4. The compressor $\mathcal{T}$5 is an 8-layer Transformer encoder with patch-based input, learnable linear projection to $\mathcal{T}$6 dimensions, block-causal self-attention, feed-forward MLP hidden size $\mathcal{T}$7, GeLU activations, and 2D RoPE. Token pruning ranks per-token reconstruction error as a proxy for negative ELBO and masks out the $\mathcal{T}$8 lowest-information tokens. Quantization uses Finite Scalar Quantizer with codebook size $\mathcal{T}$9. The decompressor is a symmetric 8-layer Transformer decoder with cross-attention to preserved token positions, followed by a 3D upsampling CNN decoder (Ye et al., 18 Dec 2025).

The reported training setup uses TokenBench, with 500 videos drawn from BDD100K, EgoExo-4D, BridgeData V2, and Panda-70M, and DAVIS Test-Dev2019 with 30 videos. Preprocessing includes random crop to $\mathcal{Y}^{\mathcal{GT}}$0 or original-aspect-ratio generalization tests, temporal clipping to 33 frames, and pixel normalization to $\mathcal{Y}^{\mathcal{GT}}$1. Optimization uses AdamW with initial learning rate $\mathcal{Y}^{\mathcal{GT}}$2, cosine decay to $\mathcal{Y}^{\mathcal{GT}}$3 over $\mathcal{Y}^{\mathcal{GT}}$4 steps, batch size 1 clip per GPU, 32 NVIDIA H100 GPUs, training for $\mathcal{Y}^{\mathcal{GT}}$5 steps, router EMA smoothing 0.99, compression factors $\mathcal{Y}^{\mathcal{GT}}$6 for InfoTok-Flex, a stability floor $\mathcal{Y}^{\mathcal{GT}}$7, and mixed-precision FP16 training (Ye et al., 18 Dec 2025).

6. Empirical behavior, limitations, and broader interpretation

The video-tokenizer paper evaluates reconstruction quality with PSNR, SSIM, LPIPS, and FVD; compression with $\mathcal{Y}^{\mathcal{GT}}$8; and inference overhead via number of forward passes. On TokenBench-256 and DAVIS-256 at average $\mathcal{Y}^{\mathcal{GT}}$9, InfoTok reports PSNR 30.08, SSIM 0.881, LPIPS 0.145, and FVD 25.79, while InfoTok-Flex reports PSNR 29.86, SSIM 0.878, LPIPS 0.148, and FVD 25.69. At $$\mathcal{L}_{\mathrm{IB}} = \inf_{p(\mathcal{Z}\mid \mathcal{I})} I(\mathcal{Z};\mathcal{I}) - \beta\,I(\mathcal{Z};\mathcal{Y}^{\mathcal{GT}}),$$0, InfoTok reports PSNR 29.27, SSIM 0.854, LPIPS 0.176, and FVD 24.52, while InfoTok-Flex reports PSNR 29.30, SSIM 0.857, LPIPS 0.179, and FVD 24.84. The paper summarizes these results by stating that at $$\mathcal{L}_{\mathrm{IB}} = \inf_{p(\mathcal{Z}\mid \mathcal{I})} I(\mathcal{Z};\mathcal{I}) - \beta\,I(\mathcal{Z};\mathcal{Y}^{\mathcal{GT}}),$$1, InfoTok matches or exceeds Cosmos-DV while saving 20% tokens, and that against ElasticTok the InfoTok variants improve PSNR by approximately 2 dB, reduce LPIPS by 40%, and reduce FVD by 40–60% (Ye et al., 18 Dec 2025).

Inference efficiency is also emphasized. ElasticTok is reported to require binary search over mask lengths, incurring 11 extra passes, whereas InfoTok uses one extra decoder pass for ELBO, incurring 1 extra NFE. On an A5000 GPU for 33 frames at $$\mathcal{L}_{\mathrm{IB}} = \inf_{p(\mathcal{Z}\mid \mathcal{I})} I(\mathcal{Z};\mathcal{I}) - \beta\,I(\mathcal{Z};\mathcal{Y}^{\mathcal{GT}}),$$2, the measured latency is 0.61 s for Cosmos, 1.23 s for Cosmos+InfoTok, 13.45 s for Cosmos+ElasticTok, and 42.75 s for ElasticTok end-to-end. Ablations further report marginal difference between the ELBO router and exhaustive optimal length search at $$\mathcal{L}_{\mathrm{IB}} = \inf_{p(\mathcal{Z}\mid \mathcal{I})} I(\mathcal{Z};\mathcal{I}) - \beta\,I(\mathcal{Z};\mathcal{Y}^{\mathcal{GT}}),$$3 (PSNR 29.86 versus 29.92), improved masking performance for ELBO-rank over random or periodic strategies, and gains from ELBO routing across both Cosmos-DV and VisionTransformer backbones.

The limitations differ across the two InfoTok lines. In unified MLLMs, the paper notes limited effectiveness in models with frozen visual backbones, exemplified by OpenUni, where gradients cannot fully reshape the encoder. It also points to hierarchical or multi-scale tokenizers, tighter mutual-information estimators, and extension to video or audio under joint information budgets as future directions. In adaptive video tokenization, the open questions include lighter routers that estimate per-token complexity without a full decoder pass, effects on downstream tasks such as generative modeling, action recognition, and video-language understanding, and application to modalities beyond video (Tang et al., 2 Feb 2026).

A common interpretive thread is that both methods reject indiscriminate token preservation. The unified-MLLM InfoTok explicitly seeks to compress high-frequency, instance-specific pixel noise and redundancy while preserving semantic entities, spatial composition, object attributes, and perceptually salient cues essential for faithful generation. The video InfoTok allocates discrete tokens proportional to negative ELBO and uses a Transformer-based adaptive compressor to reshape information from discarded to retained tokens. This suggests a broader research pattern in which information-theoretic control is used not merely for compression, but for deciding which parts of a latent representation are worth spending token budget on under architectural and computational constraints.