BLIP3o-NEXT: Next Frontier of Native Image Generation (2510.15857v1)

Abstract: We present BLIP3o-NEXT, a fully open-source foundation model in the BLIP3 series that advances the next frontier of native image generation. BLIP3o-NEXT unifies text-to-image generation and image editing within a single architecture, demonstrating strong image generation and image editing capabilities. In developing the state-of-the-art native image generation model, we identify four key insights: (1) Most architectural choices yield comparable performance; an architecture can be deemed effective provided it scales efficiently and supports fast inference; (2) The successful application of reinforcement learning can further push the frontier of native image generation; (3) Image editing still remains a challenging task, yet instruction following and the consistency between generated and reference images can be significantly enhanced through post-training and data engine; (4) Data quality and scale continue to be decisive factors that determine the upper bound of model performance. Building upon these insights, BLIP3o-NEXT leverages an Autoregressive + Diffusion architecture in which an autoregressive model first generates discrete image tokens conditioned on multimodal inputs, whose hidden states are then used as conditioning signals for a diffusion model to generate high-fidelity images. This architecture integrates the reasoning strength and instruction following of autoregressive models with the fine-detail rendering ability of diffusion models, achieving a new level of coherence and realism. Extensive evaluations of various text-to-image and image-editing benchmarks show that BLIP3o-NEXT achieves superior performance over existing models.

• The paper introduces a unified AR + Diffusion model that combines text-to-image synthesis with image editing, integrating reinforcement learning for improved instruction alignment.
• The methodology employs a hybrid pipeline where an AR module generates quantized image tokens and a diffusion transformer refines them into high-fidelity visuals.
• Empirical results demonstrate robust improvements in object composition and text rendering, validated on benchmarks like GenEval using RL-driven optimization.

BLIP3o-NEXT: A Unified Autoregressive + Diffusion Model for Native Image Generation

Introduction and Motivation

BLIP3o-NEXT introduces a fully open-source foundation model for native image generation, unifying text-to-image synthesis and image editing within a single scalable architecture. The model leverages an Autoregressive (AR) + Diffusion design, integrating the semantic reasoning and instruction-following capabilities of AR models with the fine-detail rendering of diffusion models. The architecture is motivated by the need for high-fidelity, instruction-aligned image generation and editing, with a focus on scalability, inference efficiency, and extensibility to reinforcement learning (RL) paradigms.

Figure 1: The architecture of BLIP3o-NEXT (left) and its reinforcement learning pipeline (right). BLIP3o-NEXT adopts an AR + Diffusion design, with joint optimization and RL rollouts rendered from the diffusion transformer.

Model Architecture and Training Paradigm

BLIP3o-NEXT employs a hybrid pipeline: the AR module generates discrete image tokens conditioned on multimodal inputs (text and reference images), and the hidden states of these tokens serve as conditioning signals for a diffusion transformer that synthesizes the final image. The AR model operates on quantized visual tokens (729 per image at 384x384 resolution, using SigLIP2 encoding), trained via next-token prediction. The diffusion model refines these tokens into high-fidelity images, conditioned on the AR hidden states.

Training is performed with a composite objective:

$$\mathcal{L} = \mathcal{L}_{\text{CE}} + \lambda \mathcal{L}_{\text{diff}}$$

where $\mathcal{L}_{\text{CE}}$ is the cross-entropy loss over text and image tokens, and $\mathcal{L}_{\text{diff}}$ is the diffusion loss. The AR backbone is initialized from Qwen3, and the diffusion transformer from SANA1.5, resulting in a 3B parameter model. Pretraining utilizes BLIP3o-Pretrain and BLIP3o-60K datasets, with extensive data cleaning, captioning, and synthetic data augmentation.

Reinforcement Learning for Image Generation

BLIP3o-NEXT integrates RL to enhance text rendering and instruction following. RL is applied to the AR model using Group Relative Policy Optimization (GRPO), leveraging the compatibility of discrete token generation with existing RL infrastructures for LLMs. For each prompt, the AR model samples multiple trajectories of image tokens, which are decoded by a frozen diffusion model. Rewards are assigned via verifiable metrics (e.g., GenEval for object composition, OCR for text rendering), and policy optimization is performed on the AR model.

Figure 2: Training reward for GenEval and visual text rendering.

Figure 3: Qualitative results of multiple object composition before and after GRPO.

Figure 4: Qualitative results of text rendering before and after GRPO.

Empirical results demonstrate that RL substantially improves both object composition and text rendering quality. The central challenge in RL for image generation is the design of reward models that balance image quality, instruction following, and human preference alignment.

Image Editing and VAE Feature Integration

Image editing in BLIP3o-NEXT is achieved by conditioning on reference images, represented as quantized tokens concatenated with text prompt tokens. The primary challenge is maintaining consistency between generated and reference images. BLIP3o-NEXT addresses this via:

• Image Reconstruction Task: The model is trained to reconstruct the reference image when prompted to "keep the image unchanged," enforcing pixel-level fidelity.
• VAE Latent Conditioning: Low-level VAE latents are injected as additional conditioning signals for the diffusion model, using two complementary strategies:
  1. Cross-attention conditioning: Flattened VAE tokens are appended to multimodal context tokens for the diffusion transformer's cross-attention layers.
  2. Noise-space injection: VAE features are concatenated with the diffusion noise tensor, augmenting the denoising process.

     Figure 5: Comparison of VAE feature integration strategies in BLIP3o-NEXT. Combining cross-attention and noise-space injection yields optimal visual consistency.

     

     Figure 6: Qualitative results for image editing comparing models with and without VAE latent condition.

Multitask training on image reconstruction and editing objectives, using a 10M sample corpus, yields competitive results on the ImgEdit benchmark. Incorporating VAE latents improves consistency, though downsampling in the VAE introduces minor artifacts.

Design Choices and Ablation Studies

BLIP3o-NEXT systematically explores design choices for image generation in unified multimodal models:

• Image Representation: CLIP embeddings provide compact, semantically rich representations, yielding higher training efficiency and prompt alignment than VAE features.
• Training Objective: Flow matching loss better models the image distribution, enabling sample diversity and improved visual quality compared to MSE loss.
• Training Strategy: Sequential training (freezing the AR backbone and training the generation module) offers flexibility and avoids inter-task interference.

  Figure 7: Three design choices for image generation in unified multimodal model.

  

  Figure 8: Comparison of different design choices.

  

  Figure 9: Joint Training vs. Sequential Training: Joint training mixes image-understanding and generation data, while sequential training freezes the AR backbone for dedicated image generation.

Benchmark Results and Human Evaluation

BLIP3o-NEXT achieves strong performance across image understanding and generation benchmarks, outperforming or matching larger models in most tasks. On GenEval, DPG-Bench, and WISE, BLIP3o-NEXT 8B attains scores of 0.84, 81.60, and 0.62, respectively. Human studies on DPG-Bench confirm superior visual quality and prompt alignment compared to Janus Pro 7B, with statistically significant $p$-values.

Figure 10: Human paper results for DPG-Bench between Janus Pro and BLIP3o-NEXT, showing preference for BLIP3o-NEXT in both visual quality and prompt alignment.

Practical and Theoretical Implications

BLIP3o-NEXT demonstrates that architectural simplicity, scalability, and RL integration are sufficient for state-of-the-art native image generation. The AR + Diffusion paradigm enables unified multimodal models to inherit reasoning and instruction-following capabilities from LLMs, while maintaining high-fidelity image synthesis. RL further enhances controllability and alignment, though reward model design remains an open problem. Data quality and scale are decisive for performance ceilings.

The model's open-source release, including weights, datasets, and code, ensures reproducibility and fosters further research. The findings suggest future directions in unified architectures, scalable RL for editing, and specialized benchmarks for instruction alignment and consistency.

Conclusion

BLIP3o-NEXT advances the frontier of native image generation by unifying text-to-image synthesis and image editing in a scalable AR + Diffusion framework. Systematic exploration of architectural, training, and RL strategies yields a model that excels in both generation and editing tasks. The work highlights the importance of reward model design, data quality, and scalable post-training, pointing toward future developments in controllable, instruction-aligned multimodal foundation models.