Lumina-mGPT 2.0: Unified AR Transformer

• Lumina-mGPT 2.0 is a unified autoregressive Transformer designed for high-quality image generation and diverse multimodal tasks without pre-trained components.
• It leverages a unified tokenization scheme and advanced inference techniques such as speculative Jacobi decoding and LineAR cache compression to boost speed and reduce memory usage.
• Benchmark results show competitive and often superior performance over diffusion models, with scalable configurations from 2B to 7B parameters and robust improvements in key metrics.

Lumina-mGPT 2.0 is a stand-alone, decoder-only autoregressive Transformer model designed to deliver state-of-the-art performance in high-quality image generation and multimodal tasks. Developed as a GPT-style model trained entirely from scratch, it foregoes pretrained vision or language components to eliminate licensing constraints and to maximize architectural flexibility. Lumina-mGPT 2.0 achieves competitive or superior results relative to leading diffusion models such as DALL-E 3 and SANA, with a unified tokenization scheme and efficient inference techniques that enable broad applicability, compositional generalization, and high throughput (Xin et al., 23 Jul 2025).

1. Model Architecture and Unified Tokenization

Lumina-mGPT 2.0 employs a decoder-only Transformer architecture built from the ground up, available in two principal configurations:

Model	Parameter Count	Hidden Dim. (d)	Layers (L)	Attn. Heads (H)	Vocabulary Size (V)
2B	2B	2,048	32	32	171,385
7B	7B	4,096	32	32	171,385

Key architectural features include standard causal self-attention, static pre-allocated key-value (KV) caches, and positional encodings (learned or rotary). The objective is maximization of the autoregressive likelihood $P(x_{1:T}) = \prod_{t=1}^T P(x_t | x_{<t})$, optimized via cross-entropy loss.

Tokenization is unified across modalities: image tokens derive from the SBER-MoVQGAN codebook (8×8 downsampling, selected via empirical PSNR/SSIM/LPIPS results), while text tokens employ the QwenTokenizer. No external pretrained encoders are used; all model parameters are trained from scratch (Xin et al., 23 Jul 2025).

2. Decoding Techniques and Efficient Inference

To mitigate the inherent sequential inefficiency of autoregressive models and to bolster image generation quality, Lumina-mGPT 2.0 integrates several inference-time strategies:

• Inference-Time Scaling ("best-of-N" and prompt refinement): Prompts are optionally refined using LLMs (e.g., GPT-4o), and the model samples $N$ candidate images, selecting the one with the highest aggregate verifier score (combining VQAScore, LAION-AestheticScore, PickScore, etc.). Increasing $N$ to 16 elevates GenEval accuracy from 0.69 to 0.80 (Xin et al., 23 Jul 2025).
• Speculative Jacobi Decoding (SJD): This approach builds on Jacobi decoding by conducting multiple speculative token-prediction steps in parallel. A token is committed if the ratio of the target to draft logit exceeds a preset threshold. This synergizes with quantization (TorchAo 4-bit) and PyTorch 2.0 static compilation to yield a 72% reduction in sampling time and a 47% reduction in GPU memory usage (Xin et al., 23 Jul 2025).

3. Multi-Task and Compositional Capabilities

Lumina-mGPT 2.0 exploits the raster scan paradigm, generating image tokens line by line (top to bottom), which naturally supports multi-task workflows by contextually partitioning token sequences. The model can conduct:

• Text-to-image synthesis: Lower segment is generated conditioned on text.
• Subject-driven and reference-based generation: Upper half encodes subject reference.
• Image editing: Upper panel as source, lower as edited output.
• Controllable generation: Lower half conditioned on control maps (depth, canny, pose, etc.).
• Dense prediction: Lower half as prediction map, upper as input.

Prompts encode task structure via special tokens (<upper half>, <lower half>) and descriptors, allowing all tasks to operate under a single cross-entropy objective. Only image tokens incur loss during training (Xin et al., 23 Jul 2025).

4. Integration of LineAR Cache Compression

Autoregressive image generation models typically suffer high memory and throughput costs due to cumulative KV caching over extended visual sequences. Lumina-mGPT 2.0 directly adopts the training-free, mathematically principled 2D line-level cache compression pipeline, LineAR (Qin et al., 4 Dec 2025). The method operates as follows for an $n \times n$ image:

• The sequence is partitioned into $n$ raster lines.
• The cache budget is set by $\rho = B/N$ ($B$ = retained KV tokens; $N = n^2$).
• For the first $\lceil \rho n \rceil$ lines, all cache is preserved to establish global context.
• Subsequent steps partition cached tokens into: (i) initial anchors (global style), (ii) a recent window (local context), and (iii) a mid-region (candidates for eviction).
• Inter-line attention is computed: the average query-key attention between current line queries and mid-region keys yields a saliency score vector. The $n$ least salient mid-region tokens are evicted.
• This process strictly ensures that crucial global and local contexts are maintained.

On Lumina-mGPT-768 (precursor to 2.0), LineAR achieved a 2.13× throughput boost and 64.69% memory reduction at 1/8 cache (294 tokens of 2352), with DPG overall improving from 76.96 (full cache) to 77.19 (1/8 cache). The pipeline generalizes directly to Lumina-mGPT 2.0 without retraining or architectural adaptation; typical throughput and memory reductions (2–3× and 60–65%, respectively) remain robust (Qin et al., 4 Dec 2025).

5. Benchmark Results and Comparison

Evaluation of Lumina-mGPT 2.0 on GenEval and DPG benchmarks demonstrates parity or superiority to competing autoregressive and diffusion models:

Model	GenEval Overall	DPG Overall	Notes
SANA 1.6B (diffusion)	0.66	84.80	Diffusion baseline
DALL·E 3 (closed)	0.67	83.50	Diffusion baseline
Janus-Pro 7B (AR)	0.80	84.19	AR baseline
Lumina-mGPT 2.0 (7B)	0.80	84.30	Best AR/diffusion
Lumina-mGPT 2.0 (2B)	0.68	82.05	Smaller config

Lumina-mGPT 2.0 further demonstrates strong generalization in dense prediction (Graph200K: F1 up to 0.49, RMSE down to 17.4), subject-driven generation (CLIP-I up to 87.4, DINOv2 up to 76.6), and controllable generation (Canny F1=0.49, FID=30.9, SSIM=0.54) (Xin et al., 23 Jul 2025).

6. Ablations, Licensing, and Limitations

Lumina-mGPT 2.0 is trained without reliance on external models or weights, affording unrestricted use in any research or commercial context. Key ablation studies demonstrate:

• Model scaling: 2B→7B brings GenEval up from 0.68 to 0.80 and DPG from 82.05 to 84.30.
• Prompt refinement (“thinking before generation”): GenEval increases from 0.69 to 0.73.
• Best-of-16 decoding: GenEval further increases to 0.80.
• Quantization + SJD: Sampling time reduction by 72%, GPU memory cut by 47%.

Current limitations include slower decoding relative to single-step diffusion models, and the persistent overhead from static KV cache allocation. Future work includes scaling to ≥20B parameters, integration of continuous visual encoders, and enhanced multimodal reasoning (Xin et al., 23 Jul 2025).

7. Impact and Future Directions

Lumina-mGPT 2.0 establishes that stand-alone, from-scratch decoder-only autoregressive models can match or surpass state-of-the-art diffusion systems in image quality, compositionality, and flexibility, while offering multi-task generation in a unified next-token framework. The adoption of progressive line-level cache compression (LineAR) supplies a rigorous solution to inference bottlenecks, enabling large-scale usage without retraining. Prospective avenues involve model scaling, tighter visual encoder integration, and exploration of more adaptive, compile-friendly memory management (Xin et al., 23 Jul 2025, Qin et al., 4 Dec 2025).