ARM: An AutoRegressive Large Multimodal Model with Unified Discrete Representations

Abstract: This paper introduces ARM, a discrete representation-based AutoRegressive Model that unifies image understanding, generation, and editing within a next-token prediction framework. ARM is built on three efforts: first, we train a discrete semantic visual tokenizer that maps images into compact token sequences. Our tokenizer is supervised with multiple objectives that jointly promote semantic discriminability, language alignment and faithful reconstruction, thereby supporting diverse tasks in a shared latent space. With this, we train a 7B autoregressive model over large-scale text and image token sequences, seamlessly developing vision-language perception and generation capabilities. Finally, to further improve preference-aligned behavior for text-to-image generation and instruction-guided editing, ARM applies reinforcement learning (RL) to optimize task-level objectives such as visual quality, instruction adherence, and edit consistency. Surprisingly, the results show that RL not only substantially improves performance on the target tasks (e.g., raising WISE overall from 0.50 to 0.56, GEdit-Bench-EN G_O from 5.75 to 6.68), but also induces cross-task synergy between text-to-image generation and editing. Collectively, these findings highlight autoregressive modeling, when paired with strong representations and preference optimization, as a scalable foundation for multimodal intelligence. Code: https://github.com/wdrink/ARM.

• The paper presents a unified autoregressive model that integrates vision-language understanding, image generation, and editing through a novel discrete visual tokenizer.
• It employs complementary supervision via caption, pixel reconstruction, contrastive, and distillation losses to align semantic features with precise pixel-level details.
• Experiments show that preference-based reinforcement learning boosts performance across multimodal benchmarks, achieving competitive results in tasks like VQA, image synthesis, and editing.

Authoritative Summary of "ARM: An AutoRegressive Large Multimodal Model with Unified Discrete Representations" (2606.11188)

Introduction and Motivation

ARM (\system) introduces a unified autoregressive large multimodal model (LMM) that employs a novel unified discrete visual tokenizer to enable seamless integration of vision-language understanding, high-fidelity image generation, and instruction-guided image editing with a single transformer backbone. Historically, LMM systems have relied on distinct visual encoders for understanding versus generation due to incompatible requirements on visual representations. This fragmentation hinders both efficiency and performance. ARM directly addresses this by constructing and leveraging a visual tokenizer designed—via complementary supervision—for both strong semantic alignment and pixel-accurate reconstruction, optimizing a single architecture for both recognition and synthesis.

Figure 1: High-resolution images of various aspect ratios generated by \system.

Unified Discrete Visual Tokenizer: Architecture and Training

The core enabler of ARM’s capabilities is its unified discrete visual tokenizer. Built atop a frozen SigLIP2 encoder, the system projects high-dimensional semantic features into a compact latent space, then applies Finite Scalar Quantization (FSQ) to discretize these representations—eschewing the need for classical codebooks. This process is augmented with loss functions targeting both language-vision alignment and low-level pixel reconstruction, as described:

• Caption Loss: Cross-entropy objective aligns discrete visual embeddings with downstream LLMs for optimal recognition and text-driven tasks.
• Pixel Reconstruction Loss: Optimizes a DiT-style diffusion decoder to supervise fine details via pixel-space reconstruction, circumventing lossy VAE bottlenecks.
• Sigmoid Contrastive Loss: Ensures discrete embeddings are semantically matched to SigLIP2 text features.
• Feature Distillation Loss: Minimizes cosine distance between quantized and original visual embeddings.

  Figure 2: Architecture of our unified discrete visual tokenizer.

Ablative experiments confirm all loss components are necessary to maintain high codebook utilization and balanced image/text performance. Pixel-level supervision promotes fidelity, but semantic regularization is indispensable for expressive, language-aligned tokens.

Autoregressive Sequence Modeling and Unified Generation

With all modalities (text and images) now represented as discrete tokens, ARM trains a 7B parameter autoregressive transformer on massive interleaved multimodal textual and visual token sequences (2.5T tokens). The same next-token prediction paradigm is applied to model standard language tasks, image captioning, text-to-image, and interleaved referencing.

For image synthesis and editing, detokenization utilizes a high-capacity diffusion model (FLUX, operating in latent space), conditioned solely on the predicted visual tokens. ARM demonstrates that the transformer backbone predicts tokens specifying global semantics and structure, while the diffusion renderer recovers high-frequency visual details. Qualitative and quantitative analyses confirm decoder selection affects final fidelity but not overall semantic correctness.

Figure 3: Comparison between diffusion decoders. Left: reconstruction comparison between SANA1.5 and Flux with shared visual tokens. Right: text-to-image comparison between diffusion decoders.

Preference Alignment via Group Relative Policy Optimization

ARM incorporates human-preference alignment using Group Relative Policy Optimization (GRPO), leveraging GPT-based reward models to provide feedback on both text-to-image and image editing outputs. Because ARM’s outputs are discrete tokens, preference alignment maps precisely onto the autoregressive optimization objective, producing strong synergy between editing and T2I tasks. Empirically, RL on a single task (editing or generation) consistently improves the reciprocal task’s performance; joint RL achieves the highest overall benchmark scores.

Empirical Results and Analysis

• Multimodal Understanding: On POPE and MMMU benchmarks, ARM’s unified approach attains 87.3 and 40.2, outperforming previous discrete-token systems and reaching parity with several continuous-representation LLMs. ARM narrows the gap in VQA and compositional reasoning, confirming the semantic quality of its discrete visual tokens.
• Image Generation: ARM achieves strong results on GenEval (0.91 base, 0.93 with RL), WISE (0.50 base, 0.56 with RL), and DPG. Notably, ARM-RL’s performance is competitive with or superior to models using far larger backbones or hybrid architectures.
• Image Editing: On GEdit-Bench, ARM reaches a G_O score of 6.68 following RL, substantially improving over prior autoregressive and diffusion-based editing approaches.

  Figure 4: Image generation w/ and w/o CFG. Prompt: 'Book cover, A surreal double exposure portrait that blends a woman’s face with a beautiful seascape.'

ARM showcases reduced reliance on classifier-free guidance (CFG) due to the semantically rich tokenizer; disabling CFG has only marginal effects on image faithfulness and visual quality.

Qualitative Performance

ARM demonstrates robust and diverse high-resolution image synthesis and precise instruction-guided editing across a variety of prompt styles and content configurations.

Figure 5: Image generation and editing results by \system.

Theoretical and Practical Implications

ARM’s design, in centering a discrete tokenizer with strong semantic alignment and high perceptual fidelity, marks a concretely viable path toward unified token-based autoregressive modeling for LMMs. It demonstrates that:

• Discrete tokenization—when supervised by both semantic and pixel-level losses—does not entail forfeiting multimodal recognition performance nor synthesis/editing fidelity, a central concern in previous VQ-based approaches.
• Preference-based RL on sequence-level outputs for visual tasks scales efficiently and induces bidirectional task improvement, reinforcing the value of unified modeling.
• AR architectures with discrete visual tokens approach or match the generation and editing fidelity of larger, diffusion-based or modular systems but with simplified infrastructure (single model, shared vocabulary).

Future Directions

ARM suggests a number of avenues for further research:

• Expansion to video and other modalities by extending semantic-tokenizer supervision and interleaved AR training.
• Investigation into further efficiency and scalability improvements via optimized quantization, codebook learning, and transformer specialization.
• Deeper exploration of preference alignment and RL with richer reward models and broader human/AI evaluative signals.
• Application of ARM’s unified approach to real-time interactive multimedia agents or open-ended creative tools requiring multimodal reasoning and control.

Conclusion

ARM presents empirical evidence that a single AR transformer, enabled by a unified discrete visual tokenizer with complementary supervision, can deliver state-of-the-art or competitive results across multimodal understanding, high-fidelity generation, and editable image synthesis. The architectural simplification, direct RL alignment, and demonstrated cross-task synergy position ARM as an instructive baseline for further progress in large-scale, general-purpose multimodal AI.