End-to-End Autoregressive Image Generation with 1D Semantic Tokenizer

Abstract: Autoregressive image modeling relies on visual tokenizers to compress images into compact latent representations. We design an end-to-end training pipeline that jointly optimizes reconstruction and generation, enabling direct supervision from generation results to the tokenizer. This contrasts with prior two-stage approaches that train tokenizers and generative models separately. We further investigate leveraging vision foundation models to improve 1D tokenizers for autoregressive modeling. Our autoregressive generative model achieves strong empirical results, including a state-of-the-art FID score of 1.48 without guidance on ImageNet 256x256 generation.

• The paper presents EOSTok, which integrates a 1D ViT tokenizer and an autoregressive model with APR loss, eliminating 2D spatial priors for unified training.
• It leverages semantic supervision from vision foundation models via implicit alignment, effectively bridging the gap between reconstruction and generation.
• The framework achieves a state-of-the-art gFID of 1.48 on ImageNet, offering orders-of-magnitude faster synthesis compared to diffusion-based methods.

End-to-End Autoregressive Image Generation with 1D Semantic Tokenizer: An Expert Analysis

Introduction and Motivation

Contemporary autoregressive (AR) image generation predominantly leverages visual tokenizers to map high-dimensional images to compact discrete latent codes. The prevailing two-stage paradigm separates tokenizer training (focused on reconstruction) from AR model training (focused on generation given the learned codes). This separation fundamentally misaligns the representation learning objectives, especially when 2D tokenization is employed, as the induced spatial dependencies conflict with the sequential unidirectional structure optimal for AR modeling. "End-to-End Autoregressive Image Generation with 1D Semantic Tokenizer" (2605.00503) directly addresses these misalignments and optimization inefficiencies.

The paper introduces EOSTok, a framework for single-stage, end-to-end joint training of a 1D vision transformer (ViT) tokenizer and an AR generative model. This system incorporates global semantic supervision from vision foundation models (VFMs), substantially enhancing the alignment between reconstruction and generation objectives. By removing explicit 2D spatial priors and infusing protocol-level generative supervision, EOSTok achieves strong empirical results on ImageNet 256$\times$256, establishing a new state-of-the-art in FID (gFID 1.48 without guidance).

Methodology

1D ViT Tokenizer Architecture

EOSTok employs a ViT-based tokenizer which processes images as a sequence comprising both patch embeddings and learnable 1D query tokens. This design abolishes the 2D prior inherent in most tokenizers, producing a sequential discrete latent space compatible with standard AR sequence modeling. Tokenization is performed via IBQ—a vector quantization approach with robust codebook utilization and effective gradient propagation via straight-through estimation.

Figure 1: Overall training pipeline of EOSTok, exhibiting joint optimization of the 1D ViT tokenizer and autoregressive model with intertwined reconstruction and AR objectives.

End-to-End Joint Training with Generative Feedback

The paper critiques the classical separation of tokenizer and AR model training, highlighting how it prohibits the latent space from adapting to generative demands. EOSTok’s innovation is a unified loss:

• Reconstruction loss: Ensures the quantized latent tokens preserve necessary image information.
• Next Token Prediction (NTP) loss: Supervises the AR model to predict sequential latent codes.
• Autoregressive Prediction Reconstruction (APR) loss: Bridges the optimization gap between token prediction and actual pixel-level synthesis. The decoded AR teacher-forcing predictions are matched in the image space with ground-truths, closing the loop from generative modeling to encoder adjustment.

The APR loss regulates the latent space and avoids codebook collapse, a failure mode where the tokenizer exploits the NTP loss by mapping most samples to a limited set of codes, harming sample diversity and final image quality.

Figure 2: (a) NTP loss in isolation causes codebook collapse as the tokenizer overfits to a small subset of codes. (b) APR loss restores the latent space structure by providing end-to-end pixel-level supervision.

Incorporating Semantic Supervision from VFMs

While prior works used VFM features for 2D latent space guidance, such approaches are suboptimal for 1D tokenization. EOSTok explores three injection strategies:

• Direct alignment: Aligning 1D latents to interpolated 2D VFM embeddings, which introduces undesirable 2D priors.
• Direct substitution: Replacing image patches with VFM features, avoiding spatial priors but risking loss of granular local information.
• Implicit alignment: Aligning only the internal patch embeddings (not the final latents) to VFM features, distilling semantic content without enforcing specific spatial structure.

Experiments show implicit alignment and decoder-level alignment maximize generative quality benefits without reintroducing detrimental alignment with 2D spatial layouts.

Figure 3: Schematic of different strategies for injecting VFM representations into the 1D ViT tokenizer: (a) direct alignment, (b) direct substitution, and (c) implicit alignment.

Empirical Results

State-of-the-Art ImageNet Generation

EOSTok exhibits robust scaling behavior and SOTA performance over a large suite of models across both 1D and 2D tokenization spaces. The EOSTok-H configuration (388M tokenizer + 644M AR model) achieves gFID 1.48 (no classifier guidance) and is orders-of-magnitude faster than diffusion-based competitors during sampling. These improvements are realized without the instability typical of aggressive compression settings or large codebooks, given the APR loss and semantic alignment regularization.

Analysis of Training and Model Components

Scaling analysis reveals monotonically improved gFID with increased AR model and tokenizer size, with diminishing sensitivity to codebook size in larger models, further validating the design's scalability.

Figure 4: Training-time evolution of gFID highlights consistent improvements in generative quality with model scaling and effective semantic alignment.

Qualitative samples confirm that the latent space remains diverse and the decoded images achieve high semantic fidelity.

Figure 5: Representative images generated by EOSTok on ImageNet 256$\times$256, demonstrating photorealistic quality and semantic consistency.

Ablation and Robustness

• Token sequence length: Increasing sequence length benefits reconstruction but impedes AR generation beyond optimal lengths due to sequence modeling difficulties.
• Codebook size: Larger codebooks favor reconstruction at the cost of harder token prediction, but this tradeoff softens with AR model scaling.
• Training strategies: Only the one-stage joint protocol with APR loss achieves alignment between latent structure and AR modeling, outperforming two-stage and pure NTP-based training.

Theoretical and Practical Implications

EOSTok’s joint training strategy, APR loss, and VFM-guided semantic alignment clarify the representational and optimization structure required for effective AR image synthesis. The study undercuts the notion that 2D spatial priors are necessary for visual tokenizers and demonstrates that AR-friendly 1D latent spaces trained with pixel-domain supervision can yield superior sample quality. The design cleanly decouples the traditional reconstruction–generation tradeoff encountered in compressed latent space models.

Practically, EOSTok enables rapid high-resolution synthesis—orders-of-magnitude faster than diffusion models—while being compatible with further extensions (e.g., to text-conditional synthesis, video). The model also circumvents inefficiencies and mode collapse typical in poorly regularized AR training of discrete codes.

On a theoretical level, the results call for reevaluation of standard two-stage paradigms and the role of spatial priors in AR visual generative modeling. Additionally, the robust integration with VFM representations via implicit alignment points toward future work in unifying foundation models and generative pipelines for semantically guided synthesis, retrieval, and editing.

Conclusion

"End-to-End Autoregressive Image Generation with 1D Semantic Tokenizer" delineates key architectural and optimization principles necessary to bridge the gap between visual tokenization and AR generation. By integrating joint end-to-end training, APR loss, and VFM-driven semantic alignment, the EOSTok framework achieves high-quality and efficient unconditional image generation, setting a new benchmark in AR-based visual synthesis. These findings have direct implications for the design of next-generation generative models—maximizing quality, efficiency, and extensibility of their visual latent spaces.