Self-Guided Training for AR Models

• Self-guided Training for Autoregressive Models (ST-AR) is a paradigm that integrates self-supervised objectives with next-token prediction to improve semantic abstraction, temporal stability, and spatial invariance in image generators.
• The framework augments the standard cross-entropy loss with masked image modeling on attention maps and both inter-step and inter-view contrastive losses to address local dependency, semantic inconsistency, and spatial invariance deficiencies.
• Empirical results demonstrate up to a 49% reduction in FID and enhanced linear probing accuracy, indicating significant improvements in image synthesis quality without altering the autoregressive inference process.

Self-guided Training for Autoregressive Models (ST-AR) denotes a set of methodologies that enhance the capability of autoregressive generative models by integrating self-supervised objectives into the standard next-token prediction framework. This paradigm is motivated by the need to overcome structural limitations in traditional autoregressive image generators, especially those operating in the visual domain, and aims to foster improved semantic understanding and synthesis quality without requiring pre-trained representation models or changes to the generation procedure (Yue et al., 18 Sep 2025).

1. Autoregressive Image Generation: Sequence Modeling in the Visual Domain

Autoregressive models for image generation first quantize images $I$ into discrete sequences of visual tokens:

$x = q(I) = [x_1, x_2, \ldots, x_T]$

where the quantizer $q(\cdot)$ may use codebook indices from VQ-GAN or similar discrete representation schemes. The generative task is cast as predicting these tokens sequentially given some conditioning $c$ (e.g., class label or textual prompt):

$$p_\theta(x) = \prod_{t=1}^T p_\theta(x_t \mid c, x_{<t})$$

The training loss for the standard autoregressive model is the cross-entropy:

$$\mathcal{L}_{AR} = -\frac{1}{T} \sum_{t=1}^{T} \log p_\theta(x_t \mid c, x_{<t})$$

This framework directly transposes methodologies from autoregressive language generation to visual generation, leading to issues unique to spatial and semantic characteristics of images.

2. Limitations of Next-Token Prediction for Visual Semantics

Three major deficiencies emerge when directly applying autoregressive training to images as outlined in (Yue et al., 18 Sep 2025):

1. Local and Conditional Dependence
   • The model allocates excessive attention to initial conditional tokens (e.g., the class token) and spatially adjacent tokens. This manifests as strong local bias, leading to overdependence on local cues, neglect of global structure, and accumulation of regional errors.
   • Attention analysis shows that weights are heavily concentrated near the conditional and previously generated tokens, impeding the integration of semantic context.
2. Inter-step Semantic Inconsistency
   • Representations learned for different generation steps are semantically misaligned. Earlier predictions yield low linear probing accuracy indicative of poor high-level semantic capture; as the context grows, inconsistency across steps persists.
   • Semantic information is not stably propagated throughout the generation sequence, which limits the coherence of large context regions.
3. Spatial Invariance Deficiency
   • Visual tokenizers such as VQ-GAN, trained solely for reconstruction/compression objectives, fail to enforce spatial invariance. Minor input perturbations can generate completely different token sequences even if the semantics are unchanged—a property not exhibited in language tokenization.
   • The result is fragmented, redundant learning of similar semantics across spatial neighborhoods, diluting semantic abstraction.

3. Self-Supervised Objectives in ST-AR

The ST-AR framework mitigates the above limitations by augmenting autoregressive training with the following self-supervised objectives:

a. Masked Image Modeling (MIM) on Attention Maps

• Instead of masking input tokens (which is incompatible with the sequential AR process), random masking is applied to transformer attention maps.
• Specifically, in each attention matrix, a fraction $r$ of entries are set to $-\infty$ before the softmax, effectively dropping attention links.
• A teacher network (updated via exponential moving average, EMA) provides target features for reconstruction. The MIM loss is:

$$\mathcal{L}_{MIM} = \frac{1}{T} \sum_{t=1}^{T} \mathcal{D}(h_t, \hat{h}_t)$$

where $h_t$ is the student’s hidden feature at step $t$, $\hat{h}_t$ is the teacher’s, and $\mathcal{D}$ is a distance metric (such as cosine distance).

b. Inter-step Contrastive Loss ($\mathcal{L}_{step}$)

• To align semantic representations across timesteps, features from multiple steps within the same view are used as positive samples, while negatives come from other images.
• This contrastive mechanism stabilizes semantic drift during generation and encourages consistent abstraction across the sequence.

c. Inter-view Contrastive Loss ($\mathcal{L}_{view}$)

• Features from different augmented views (crops, transformations) of the same image are pulled together.
• The loss enforces spatial invariance by aligning locations corresponding to the same semantics in different views, countering deficiencies in initial tokenization.

Combined Objective

The overall loss function for ST-AR is:

$$\mathcal{L}_{ST-AR} = \mathcal{L}_{AR} + \alpha \mathcal{L}_{MIM} + \frac{\beta}{2} (\mathcal{L}_{step} + \mathcal{L}_{view})$$

where hyperparameters $\alpha$ and $\beta$ determine the relative importance of the self-supervised terms.

4. Architectural and Training Implications

ST-AR does not rely on pre-trained visual representation models. All self-supervised objectives are internal to the generative training. The use of attention-masked MIM differentiates ST-AR from traditional masking approaches (which mask tokens) and from diffusion or masked autoencoder frameworks. The self-supervised teacher features are maintained throughout training using EMA, ensuring stability and robust feature targets.

Contrastive objectives both across time (steps) and space (views) enable the autoregressive generator to learn representations that are invariant to minor changes in the input and stable across the course of the generation. This improves semantic abstraction without compromising fine-grained local modeling.

5. Empirical Performance and Semantic Effects

ST-AR yields substantial improvements in both image understanding and final image synthesis quality:

• LlamaGen-L models trained with ST-AR achieve approximately 42% reduction in Fréchet Inception Distance (FID).
• LlamaGen-XL models observe a 49% reduction in FID.
• These gains are achieved without any modification to the sampling or generation strategy: the model still generates sequentially, token by token, as in the original AR approach.

Additionally, linear probing accuracy analyses suggest marked improvement in semantic understanding and invariance in intermediate representations. This effect is not contingent on external pre-trained models but arises purely from the joint optimization of next-token prediction and self-supervised learning.

6. Sampling and Inference

The autoregressive sampling procedure remains untouched in ST-AR. At inference, images are synthesized by sequentially sampling tokens in a left-to-right fashion:

$x_{t} \sim p_\theta(x_t | c, x_{<t})$

Self-supervised objectives affect only training. Thus, ST-AR improves semantic and generative quality without altering the inference pipeline, maintaining full compatibility with downstream applications or further modalities (e.g., conditional generation with text).

7. Theoretical and Practical Significance

ST-AR demonstrates that combining next-token prediction with self-supervised training objectives leads to autoregressive models capable of learning more robust and invariant semantic features in the visual domain. This strategic integration of masked attention modeling and contrastive regularization addresses key deficiencies—local dependency, semantic inconsistency, and spatial non-invariance—unique to tokenized image synthesis.

The ability to substantially improve generation quality (as measured by FID) and representation quality (as measured by semantic probing) without impacting decoding speed or sampling complexity suggests ST-AR is an attractive paradigm for advanced autoregressive image generators. The generality of the framework, eschewing reliance on external pretraining, allows its principles to be extended to other generative modalities where autoregressive modeling is the foundation.

---

In summary, the Self-Guided Training for AutoRegressive models paradigm (ST-AR) provides principled and empirically validated objectives that equip autoregressive image generators with improved semantic abstraction, temporal stability, and spatial invariance. This advancement is achieved exclusively through enhanced training strategies, preserving the original autoregressive architecture and inference dynamics (Yue et al., 18 Sep 2025).