OneCAT: Unified Multimodal Transformer

• OneCAT is a unified multimodal model that integrates language and visual tasks using a pure decoder-only transformer architecture with modality-specific Mixture-of-Experts.
• It employs a single autoregressive objective for language modeling, image understanding, text-to-image synthesis, and editing, eliminating the need for external vision components.
• OneCAT achieves state-of-the-art efficiency with dynamic resolution adaptation and multi-scale autoregression, significantly reducing generation latency on high-resolution images.

OneCAT is a unified multimodal model that integrates understanding, generation, and editing within a pure decoder-only transformer architecture. Distinct from prior approaches, OneCAT eliminates the need for external components such as Vision Transformers (ViT) or vision tokenizers during inference. The framework leverages a modality-specific Mixture-of-Experts (MoE) structure and trains under a single autoregressive objective, enabling dynamic resolution support and efficient high-resolution image processing. OneCAT employs a multi-scale visual autoregressive mechanism, yielding substantially faster and more efficient generation compared to diffusion-based systems, and achieves state-of-the-art performance across standard multimodal benchmarks (Li et al., 3 Sep 2025).

1. Architectural Features

1.1 Pure Decoder-Only Transformer

OneCAT is constructed as a decoder-only transformer that processes both visual and textual modalities within a single stack. Raw images are patch-embedded at inference by a sequential process consisting of a small convolution, pixel-unshuffle, and MLP projection. This produces continuous visual token vectors matched in hidden size to WordPiece text embeddings. Both token types are concatenated and input directly to the decoder stack; no separate encoder or VAE is active during inference. This approach reduces prefill latency and avoids the lossy quantization inherent in traditional tokenizer-based designs.

1.2 Modality-Specific Mixture-of-Experts (MoE)

In each transformer block (except for Q, K, V and attention mechanisms), three parallel feed-forward “experts” are implemented:

• Text FFN (language modeling)
• Und. FFN (continuous visual tokens for understanding/editing)
• Gen. FFN (discrete multi-scale visual tokens for generation)

A deterministic (hard) routing mechanism based on modality and task activates only one expert per token. The abstract soft gating function is defined as:

$g(x) = \text{softmax}(W_g x)$

with OneCAT’s operational routing simplified to:

$k = \arg\max_m g_m(x)$

where only expert $k$ is executed. Attention weights and QKV projections are fully shared, providing cross-modal integration, while FFN layers remain modality-specialized.

2. Autoregressive Training Objective

All tasks—language modeling, vision understanding, text-to-image synthesis, and editing—are unified by a single autoregressive next-token/next-scale prediction loss:

$$\mathcal{L}(\theta) = -\sum_{t=1}^T \log p_\theta(x_t \mid x_{<t})$$

The mixed-token sequence $x_1, ..., x_T$ includes text tokens, continuous visual tokens (for understanding), and discrete visual tokens at various scales (for generation). This single-objective setting omits any secondary objectives or distillation, enabling the architecture to learn sequence prediction across all modalities using cross-entropy loss exclusively.

3. Multi-Scale Visual Autoregression

OneCAT employs multi-scale autoregressive visual synthesis within its decoder. A pretrained multi-scale VAE ("Infinity") defines $S$ spatial resolution scales, each producing $N_s$ discrete tokens $G_s^1, ..., G_s^{N_s}$. The generation process follows a coarse-to-fine, next-scale strategy:

• For each scale $s = 1 \dots S$, generate all $N_s$ tokens, each conditioned on already generated tokens (text $k = \arg\max_m g_m(x)$0, earlier scales $k = \arg\max_m g_m(x)$1, and intra-scale $k = \arg\max_m g_m(x)$2 prefix).
• Causal/block-causal attention enables parallel decoding within blocks.

Pseudo-code summary: $k = \arg\max_m g_m(x)$6 The sum $k = \arg\max_m g_m(x)$3 is orders of magnitude smaller than the total pixels, and blockwise generation parallelizes inference, resulting in approximately one-tenth the generation steps of a diffusion chain.

4. Dynamic Resolution Adaptation

The patch-embedding layer, being a sequence of convolution, pixel-unshuffle, and MLP, is inherently fully convolutional and natively supports arbitrary $k = \arg\max_m g_m(x)$4 input shapes. During training, image side lengths are sampled dynamically in $k = \arg\max_m g_m(x)$5, exposing the model to diverse aspect ratios and resolutions. Position and patch coordinates enter only via their roles in causal masking; no fixed 1D positional embeddings are used. This design supports immediate generalization to unseen resolutions at inference. Optionally, a 448×448 “thumbnail” can be prepended to the token sequence to provide global context for large images.

5. Computational Efficiency and Benchmark Results

5.1 Vision Prefill Latency

OneCAT reduces prefill (Vision→Text) latency as measured by Time-To-First-Token (TTFT) on A100/H800 hardware. For instance: | Input Size | Qwen2.5-VL-3B TTFT | OneCAT TTFT | Reduction | |------------|---------------------|-------------|-----------| | 768² | 135 ms | 67 ms | 50% | | 1024² | 216 ms | 92 ms | 57% | | 1792² | 583 ms | 225 ms | 61% |

5.2 Generation and Editing Speed

On a single H800 GPU:

• 512² text-to-image: 1.40 s (vs. 8.76 s for BAGEL-7B)
• 512² editing: 2.03 s (vs. 13.45 s)
• 1024² text-to-image: 2.85 s (≈10× faster)
• 1024² editing: 4.61 s

5.3 Multimodal Benchmark Performance

OneCAT-3B achieves the following on open-source benchmarks, outperforming competing encoder-free unified models: | Task / Benchmark | OneCAT-3B | Strongest Baseline(s) | |----------------------------|-----------|---------------------------| | TextVQA | 73.9% | HoVLE 70.9, EvEv2 71.1 | | ChartQA | 81.2% | 78.6, 73.9 | | InfoVQA | 64.8% | 62.6, 60.9 | | DocVQA | 91.2% | 86.9, 88.7 | | MME-P/ MMBench | 1630 / 78.8% | 1628 / 71.9 | | GenEval (T2I overall) | 0.90 | Mogao-7B 0.89, BAGEL-7B 0.88 | | DPG-Bench (compositional) | 84.53 | Janus-Pro-7B 84.19 | | ImgEdit-Bench (overall) | 3.43 | 2nd overall, best in Adjust/Extract/Background |

6. Design Insights and Ablations

Ablation studies demonstrate that:

• Hard MoE routing is critical for the specialization of visual versus textual processing.
• Multi-scale block autoregression delivers over 2× speedup compared to single-scale next-token generation.
• Dynamic resolution sampling removes the necessity for retraining at new, unseen resolutions.

These findings support the view that pure autoregressive modeling of raw tokens, with minimal architectural modality priors, suffices for unified multimodal intelligence encompassing language, visual understanding, high-fidelity generation, and flexible image editing (Li et al., 3 Sep 2025).