Understand Before You Generate: Self-Guided Training for Autoregressive Image Generation (2509.15185v1)

Abstract: Recent studies have demonstrated the importance of high-quality visual representations in image generation and have highlighted the limitations of generative models in image understanding. As a generative paradigm originally designed for natural language, autoregressive models face similar challenges. In this work, we present the first systematic investigation into the mechanisms of applying the next-token prediction paradigm to the visual domain. We identify three key properties that hinder the learning of high-level visual semantics: local and conditional dependence, inter-step semantic inconsistency, and spatial invariance deficiency. We show that these issues can be effectively addressed by introducing self-supervised objectives during training, leading to a novel training framework, Self-guided Training for AutoRegressive models (ST-AR). Without relying on pre-trained representation models, ST-AR significantly enhances the image understanding ability of autoregressive models and leads to improved generation quality. Specifically, ST-AR brings approximately 42% FID improvement for LlamaGen-L and 49% FID improvement for LlamaGen-XL, while maintaining the same sampling strategy.

• The paper demonstrates that ST-AR overcomes AR limitations by integrating masked attention and contrastive losses, boosting linear probing accuracy from 21% to 55.23%.
• It identifies three bottlenecks—local dependence, inter-step semantic inconsistency, and spatial invariance deficiency—that hinder global coherence in image generation.
• Empirical results show that ST-AR improves FID by 49% in LlamaGen-XL, matching larger models with fewer parameters and reduced training epochs.

Self-Guided Training for Autoregressive Image Generation: Mechanisms, Methods, and Empirical Analysis

Introduction and Motivation

Autoregressive (AR) models have demonstrated strong generative capabilities in both language and vision domains, yet their adaptation to image generation exposes fundamental limitations in visual understanding. The paper "Understand Before You Generate: Self-Guided Training for Autoregressive Image Generation" (2509.15185) presents a systematic analysis of these limitations and introduces Self-guided Training for AutoRegressive models (ST-AR), a training paradigm that integrates self-supervised learning objectives to enhance the semantic modeling capacity of AR image generators.

The authors identify three key properties that hinder high-level visual representation learning in AR models: (1) local and conditional dependence, (2) inter-step semantic inconsistency, and (3) spatial invariance deficiency. These properties are empirically demonstrated using the LlamaGen-B model, as illustrated below.

Figure 1: Three properties of LlamaGen-B: (a) local attention bias, (b) semantic inconsistency across steps, (c) spatial invariance deficiency in tokenization.

Analysis of Autoregressive Image Generation Bottlenecks

Local and Conditional Dependence

AR models, when applied to images, exhibit a strong bias towards local and conditional information. Attention maps from LlamaGen-B reveal that the model predominantly attends to the initial conditioning token and spatially adjacent tokens, limiting its ability to capture global context.

Figure 2: Attention maps across layers and steps in LlamaGen-B, showing dominant weights on conditional and adjacent tokens.

Inter-Step Semantic Inconsistency

Linear probing accuracy across timesteps indicates that semantic information is not consistently maintained throughout the generation process. Early steps show low accuracy due to limited context, while later steps suffer from semantic drift, undermining global coherence.

Spatial Invariance Deficiency

Visual tokenizers such as VQ-GAN, used to discretize images for AR modeling, lack invariance constraints. Minor spatial perturbations in the input image can result in drastically different token sequences, complicating semantic alignment and leading to redundant learning.

The ST-AR Training Paradigm

To address these bottlenecks, ST-AR augments the standard next-token prediction objective with three self-supervised losses:

1. Masked Image Modeling (MIM) Loss: Random masking is applied directly to the attention maps within transformer layers, forcing the model to expand its receptive field and attend to broader image regions. Unlike conventional MIM, masking is not performed on input tokens but on attention weights, preserving the AR sampling mechanism.
2. Inter-Step Contrastive Loss: Enforces semantic consistency across different timesteps within the same image by aligning feature representations from different positions.
3. Inter-View Contrastive Loss: Promotes invariance by aligning representations of different augmented views of the same image, mitigating the effects of spatial perturbations in tokenization.

The overall training pipeline is depicted below.

Figure 3: ST-AR pipeline: masked attention, inter-step and inter-view contrastive learning, with a non-trainable EMA teacher for feature alignment.

The final loss function is:

$$\mathcal{L}_{ST\text{-}AR} = \mathcal{L}_{AR} + \alpha \mathcal{L}_{\text{MIM}} + \beta \frac{1}{2} (\mathcal{L}_{step} + \mathcal{L}_{view})$$

where $\mathcal{L}_{AR}$ is the standard token prediction loss, $\mathcal{L}_{\text{MIM}}$ is the masked attention alignment loss, and $\mathcal{L}_{step}$, $\mathcal{L}_{view}$ are contrastive losses.

Empirical Results

Image Understanding

ST-AR substantially improves the linear probing accuracy of LlamaGen-B, from 21.00% to 55.23%, indicating enhanced semantic representation learning. The accuracy remains stable across timesteps, demonstrating effective mitigation of semantic drift.

Figure 4: Linear probing accuracy for LlamaGen-B and ST-AR, showing consistent improvements in image understanding.

Attention maps from ST-AR-trained models exhibit expanded receptive fields and focus on semantically relevant regions, in contrast to the local bias observed in vanilla AR models.

Image Generation Quality

On class-conditional ImageNet, ST-AR delivers significant improvements in FID scores across all LlamaGen variants. For example, LlamaGen-XL trained with ST-AR for 50 epochs achieves a 49% FID reduction compared to the baseline, and matches the performance of LlamaGen-3B trained for 300 epochs, despite having 4x fewer parameters. These results are robust across different model sizes and training budgets.

Implementation Considerations

• Computational Overhead: ST-AR introduces additional training costs due to the teacher network and contrastive objectives. However, inference remains unchanged, preserving compatibility with standard AR sampling.
• Hyperparameters: Masking ratio, contrastive loss depth, and number of contrastive steps are critical for optimal performance. Ablation studies indicate that moderate masking (ratio 0.25–0.35) and contrastive losses applied at mid-network depth yield the best results.
• Scalability: The paradigm is applicable to various AR architectures and does not require pre-trained representation models, facilitating integration into unified multimodal generative frameworks.

Theoretical and Practical Implications

The findings demonstrate that AR models, when equipped with self-supervised objectives, can overcome modality-specific limitations and achieve competitive performance in image generation. The approach bridges the gap between generative modeling and representation learning, suggesting that future AR models should incorporate explicit semantic alignment mechanisms.

Practically, ST-AR enables more robust and semantically coherent image synthesis, with potential applications in multimodal generation, conditional synthesis, and unified generative modeling. The paradigm may also inform the design of AR models for other modalities where local dependence and invariance are critical.

Conclusion

This work provides a rigorous analysis of the limitations of autoregressive image generation and introduces ST-AR, a self-guided training framework that integrates masked attention and contrastive learning to enhance visual understanding. Empirical results substantiate substantial improvements in both representation quality and generative performance. While increased training costs remain a limitation, the paradigm sets a precedent for future research in unified, semantically-aware generative modeling. Further exploration of efficient training strategies and extension to other modalities are promising directions for advancing the field.