Unified Tokenization & Single-Stage Architecture

• Unified Tokenization and Single-Stage Architecture is a paradigm that integrates diverse data types (vision, language, audio) into a common token space for streamlined processing.
• It employs techniques like vector quantization and dual-codebooks to balance semantic alignment and reconstruction fidelity, enhancing efficiency and performance.
• Single-stage architectures eliminate multi-phase processing by using end-to-end transformer models, reducing latency and improving interpretability.

Unified tokenization and single-stage architectures constitute a paradigm shift in multimodal, sequential, and multi-domain learning systems, enabling different input modalities (vision, language, audio, actions, items) to be represented within a unified, often discrete, token space and processed through a single, end-to-end model. This design delivers seamless integration of understanding and generative tasks, supports multi-skill unification, mitigates modality-specific bottlenecks, and can yield strong improvements in efficiency, interpretability, and performance across a broad range of domains.

1. Principles of Unified Tokenization

Unified tokenization seeks to map heterogeneous input (e.g., images, speech, item IDs, sensor trajectories) into a common discrete or hybrid token representation, making them accessible to architectures—usually transformers—originally built for language sequence modeling. The dominant approaches leverage vector quantization (VQ), residual quantization (RQ), codebook-based discretization, or mixture-of-experts (MoE) to produce semantically aligned, modality-agnostic token streams. Designs may incorporate multiple codebooks for increasing representational capacity or allocate token budget adaptively according to input complexity.

For example, TokenFlow adopts a dual-codebook scheme, where both a semantic encoder and a pixel-level encoder output features that are quantized via a shared index sequence to enforce alignment of high-level semantics and low-level reconstruction cues (Qu et al., 4 Dec 2024). UniTok extends the concept to multi-domain item spaces by combining shared and expert-specific RQ codebooks, and calibrating mutual information across domains to preserve domain-specific semantics and ensure balanced informativeness (Hou et al., 17 Nov 2025). In the continuous-discrete dualistic setting, CDD-VT adaptively allocates a variable number of codebook primitives per patch, so as to blend the efficiency of discrete tokens with the expressivity of continuous codes (Chen et al., 3 Nov 2025).

2. Architectural Patterns and Mechanisms

Unified single-stage architectures eliminate the need for multi-phase processing (separate encoders, decoders, diffusion steps) by constructing a flat, end-to-end network that directly consumes the tokenized input. Most commonly, this consists of a single transformer model (or CNN variant, e.g., UDC in (Tian et al., 24 Oct 2025)), which receives the entire tokenized stream—potentially interleaved across modalities—and produces the next-token prediction, reconstructed output, or control signal in one forward pass.

In TokenFlow, visual tokens are embedded and appended or prepended to textual tokens for LLM-based multimodal tasks, and the same visual token stream feeds an autoregressive transformer for image synthesis (Qu et al., 4 Dec 2024). In TokenHSI, action control is recast as token prediction by encoding proprioception and task cues into tokens, arranged in a variable-length sequence, and processed in a single transformer with task-conditioned masking (Pan et al., 25 Mar 2025). OmniJARVIS represents vision, language and action as discrete token streams and models them with a single autoregressive transformer, enabling holistic reasoning and action in open-world scenarios (Wang et al., 27 Jun 2024).

In recommendation, UniTok reconstructs items from their tokenized representation via a shared autoencoder, with routing to domain-specific and shared RQ-expert codebooks; the resulting token sequences can be consumed by LLMs for recommendation in any domain in a plug-and-play fashion (Hou et al., 17 Nov 2025).

3. Loss Objectives and Training Protocols

Unified tokenization frameworks typically integrate multiple objectives, often balancing token-level reconstruction losses, perceptual or adversarial criteria, and semantic-alignment or contrastive losses. The challenge is to ensure these objectives do not compete, which is addressed by expanding bottleneck capacity (with multi-codebook, hierarchical, or adaptive quantization), dynamic weighting of losses, or architecture-level separation of representation spaces.

A canonical formulation in the visual domain is:

$$L_\mathrm{total} = L_\mathrm{recon} + L_\mathrm{VQ} + L_\mathrm{semantic~align} + L_\mathrm{adv/perceptual}$$

For TokenFlow, the loss combines per-patch pixel reconstruction, semantic decoder–teacher alignment, and standard VQ-VAE loss, with a distance-balance hyperparameter controlling the semantic/pixel trade-off (Qu et al., 4 Dec 2024). In QLIP, losses from binary-spherical quantization and image-text contrastive alignment are balanced dynamically, with a two-stage training pipeline to separate large-batch contrastive pretraining from high-fidelity reconstruction (Zhao et al., 7 Feb 2025). Audio models incorporate semantic knowledge distillation (SKD) to embed high-level semantic guidance into a unified transformer TTS system with joint masked prediction (Gállego et al., 17 Sep 2024).

In multi-domain contexts, mutual information calibration across domains, as in UniTok for recommender systems, reduces informativeness imbalance and cross-domain performance variance (Hou et al., 17 Nov 2025).

4. Applications and Empirical Performance

Unified tokenization and single-stage architectures have enabled leading results across a wide spectrum of domains, summarized in the following table:

Domain	Model/Paper	Key Metrics (Summary)
Vision	TokenFlow (Qu et al., 4 Dec 2024)	+7.2% average VQA uplift vs. LLaVA-1.5, rFID=0.63, GenEval=0.55 (SOTA)
Vision	UniTok (Ma et al., 27 Feb 2025)	rFID=0.38, 78.6% zeroshot acc. (ImageNet), GenEval=0.67, CLIP-aligned tokens
Multimodal	QLIP (Zhao et al., 7 Feb 2025)	Drop-in VLM: matches LLaVA, T2I: gFID=15.29 (SOTA in class)
RecSys	UniTok (Hou et al., 17 Nov 2025)	+51.89% NDCG@10 vs. LLM baselines, 10x fewer params., strong zero-shot gen.
Audio/TTS	SKD-TTS (Gállego et al., 17 Sep 2024)	WER=5.9% (single-stage SKD), narrows 2-stage gap, 50% faster
RL (Ctrl)	UTR (Tian et al., 24 Oct 2025)	9x lower attention cost, matches DT/DC on D4RL, Hopper–medium: −67% GFLOPs
Embodied	TokenHSI (Pan et al., 25 Mar 2025)	99.2% composite skill success, 1–5% improvement over specialists
VLA agent	OmniJARVIS (Wang et al., 27 Jun 2024)	+20% over plan/controller split, best open-ended reward, lowest video FSD
Tracking	USTrack (Xia et al., 2023)	+11% precision/success vs. prior, 84 FPS (real-time SOTA)

Single-stage architectures demonstrably simplify system engineering (eliminating ad-hoc fusion modules or separate encoders), reduce inference latency and memory, support genuine multi-modal reasoning, and retain or surpass the performance of multi-stage alternatives.

5. Theoretical Analysis and Constraints

The unified token approach presents significant theoretical and practical advantages in sample efficiency, regularization, generalization, and scalability. UTR for sequential decision models, for instance, proves via Rademacher complexity analysis that fusing return, state, and action per time-step yields tighter generalization bounds than treating each as a separate token (Tian et al., 24 Oct 2025). Similarly, the use of expert-specialized codebooks in multi-domain tokenization is shown (Theorem 1–3 in (Hou et al., 17 Nov 2025)) to strictly increase entropy, reduce quantization error, and limit cross-domain loss variability.

However, representational bottlenecks remain a challenge: when the discrete code space is under-capacitated, the competing demands of precision (for reconstruction/generation) and abstraction (for understanding/alignment) can induce loss conflicts or degraded performance. Solutions include exponential scaling of codebook vocabulary and bottleneck width via multi-codebook or hierarchical quantization (Ma et al., 27 Feb 2025), or adaptive token budgets as in CDD-VT (Chen et al., 3 Nov 2025).

6. Extensions, Limitations, and Future Directions

Despite their demonstrated efficacy, unified tokenization and single-stage architectures face limitations. The current analyses often presuppose linear predictors or uniform block covariances, and do not fully address the integration of nonlinear modules or specialized long-horizon tasks (Tian et al., 24 Oct 2025). Scaling unified schemes to very high-dimensional, highly compositional, or streaming data (e.g., real-time embodied decision-making, internet-scale retrieval) presents unresolved engineering and algorithmic questions.

A plausible implication is that further advances may emerge by fusing unified tokenization with emerging efficient transformer or SSM backbones, hybrid continuous-discrete logistics (e.g., wave–particle schemes (Chen et al., 3 Nov 2025)), and advanced mixture-of-experts routing for broader cross-domain generalization. Cross-modal supervision, joint codebook evolution, and curriculum-driven token budget allocation also appear promising open avenues for robustly scaling these approaches to new modalities and tasks.

7. Representative Models and Comparative Schematics

Unified tokenization and single-stage models vary in their granularity and operational targets, as illustrated below:

Model	Tokenization Scheme	Single-stage Consumer	Loss Balancing
TokenFlow	Dual codebook, aligned index	LLM for VQA/gen	Pix/sem VQ+distillation+adv+perc.
UniTok (Vision)	Multi-codebook quantization	AR transformer, CLIP	VQ+LPIPS+CLIP contrastive
QLIP	BSQ, late-fusion alignment	Unified AR model	Dynamically weighted L2/NCE
UniTok (RecSys)	Shared+expert MoE, RQ codebooks	LLM for ranking	Recon+RQ+MI calibration
UTR	Fused return–state–action	Transformer/CNN	Unified trajectory embedding
USTrack	Dual-modality patch tokens (RGB–T)	ViT backbone	Unified attention, reliability
TokenHSI	Shared+task tokens, masking	Transformer policy	PPO, adversarial, multi-task
OmniJARVIS	Discrete text/image/action tokens	AR transformer	Joint behavior + language LM

This crystallizes a prevailing trend: the collapse of modality and task bottlenecks into a singular, learnable token stream, universally ingestible by high-capacity, single-stage neural models.