BLIP3-o: Unified Open Multimodal Model

• The paper introduces BLIP3-o, a unified architecture combining autoregressive and diffusion transformers to generate semantically rich CLIP image features.
• It employs a two-stage sequential pretraining strategy with a custom BLIP3o-60k instruction tuning dataset to enhance both image understanding and generation metrics.
• Efficiency gains are achieved by reducing sequence length via CLIP embeddings and stabilizing training with grouped-query attention and sandwich normalization.

BLIP3-o denotes a family of fully open unified multimodal models designed to bridge high-level image understanding and high-fidelity image generation within a single, scalable system. BLIP3-o integrates autoregressive transformer and diffusion transformer architectures, introducing a method for generating semantically rich CLIP image features rather than employing conventional VAE-based representations. Through a two-stage sequential pretraining strategy and a bespoke instruction-tuning dataset (BLIP3o-60k), BLIP3-o establishes new state-of-the-art benchmarks across both image understanding and image generation tasks, while offering transparency and reproducibility via full open-source release (Chen et al., 14 May 2025).

1. Unified Model Architecture

BLIP3-o employs a composite architecture in which an autoregressive (AR) transformer backbone is combined with a compact diffusion transformer head. The AR transformer (e.g., Qwen 2.5‐VL-Instruct) encodes text prompts into a sequence of token embeddings $C$ and appends a single learnable query vector $Q$. This yields a compact intermediate visual feature after self-attention. The diffusion transformer (DiT), based on Lumina-Next (Next-DiT), consumes $(X_t, Q, t)$ to denoise towards ground-truth CLIP image features $X$ using a flow-matching objective. The predicted CLIP features condition a subsequent diffusion-based decoder for pixel synthesis, implemented via an off-the-shelf SDXL decoder.

Key innovations in BLIP3-o include:

• Use of a single learnable query vector $Q$, replacing typical long discrete token sequences for cross-modal alignment.
• Adoption of grouped-query attention and sandwich normalization (RMSNorm pre/post each block) to stabilize and accelerate training.
• Application of 3D rotary embeddings for positional encoding in the diffusion head.

A high-level pseudocode sketch for the text-to-image generation pipeline is as follows:

$(X_t, Q, t)$2

2. Training Objectives and Sequential Curriculum

BLIP3-o employs clearly separated objectives for image understanding and generation, realized through a two-stage pretraining pipeline:

• Stage 1 (Image Understanding): The model is trained for cross-modal text generation using a cross-entropy objective:

$$L_{CE} = -\mathbb{E}_{(I,y)\sim D}\left[\sum_{i=1}^L \log p(y_i\,|\,y_{<i}, \operatorname{CLIP}(I))\right].$$

• Stage 2 (Image Generation): After freezing the AR backbone, only the DiT diffusion head is trained for CLIP feature regression via a flow-matching objective. Two options for AR-to-feature alignment are considered:
  • (a) $$\mathcal{L}_{MSE} = \mathbb{E}\left[\|X_{gt} - W Q_{pred}\|_2^2\right]$$
  • (b) Flow-matching (Rectified Flow):

  $$X_1 = X_{gt},\quad X_0 \sim \mathcal{N}(0, I),\quad t \sim \mathrm{Uniform}(0,1),\ X_t = t X_1 + (1 - t) X_0,\ V_t = X_1 - X_0,\ L_{flow} = \mathbb{E}\left[\|V_\theta(X_t, Q_{pred}, t) - V_t\|_2^2\right].$$

The final curriculum is strictly sequential (Stage 1, then Stage 2) rather than joint multitask training, with backbone parameters shared but frozen in Stage 2.

3. Dataset Construction and Instruction Tuning

A custom instruction-tuning dataset, BLIP3o-60k, is curated to address prominent weaknesses in scene diversity, human gestures, object representation, landmarks, and textual content in images. For each category, GPT-4o is prompted to generate approximately 10k high-quality prompt–image pairs, with extra aesthetic diversity via inputs from JourneyDB and DALL·E 3-inspired prompts. The dataset is manually reviewed for quality and ambiguous outputs are removed, resulting in around 60k high-quality samples. This data supplements ~25M open-source captioned images (CC12M, SA-1B, JourneyDB), and, for the larger model variant, ~30M proprietary generation samples.

4. Training Strategies and Hyperparameters

BLIP3-o adopts the following training practices for stability and efficiency:

• Optimizer: AdamW with $\beta = (0.9, 0.999)$, weight decay 0.1.

• Learning Rates:

  • Understanding: $5 \times 10^{-5}$ (linear decay, 20k steps, 1k warmup)
  • Generation: $Q$0 (linear decay, 100k steps, 5k warmup)
  • Instruction tuning: $Q$1 (10k steps, 1k warmup)
• Batch Sizes: 256 (text), 512 (diffusion), distributed over 64 A100 GPUs.
• Mixed Precision: FP16 with dynamic loss scaling.
• Memory and Stability: Gradient checkpointing on DiT, grouped-query attention, and sandwich normalization (RMSNorm before/after each DiT block).

5. Quantitative Benchmarks and Ablations

BLIP3-o establishes superior results across prominent multimodal benchmarks, as summarized below:

Task	BLIP3-o 8B	Prior Best (approx)
VQAv2	83.1 %	79.4 %
MMBench	83.5 %	79.2 %
SEED	77.5 %	72.6 %
MMMU	50.6 %	43.2 %

• On TEXTVQA and RealWorldQA, BLIP3-o also leads (83.1% and 69.0%, respectively).
• Image generation evaluated via GenEval (0.84 vs. Janus Pro 7B: 0.80), DPG-Bench (81.6; Janus Pro: 84.2), and WISE (0.62; prior: 0.35). Human studies indicate BLIP3-o is strongly preferred on prompt alignment and aesthetics ($Q$2).
• Ablations demonstrate:
  • CLIP + Flow surpasses CLIP + MSE on alignment.
  • VAE + Flow yields best FID ($Q$315) but inferior semantic alignment.
  • Flow matching promotes %%%%14$(X_t, Q, t)$15%%%% greater image diversity compared to MSE alignment.

6. Efficiency, Scalability, and Open Source Release

BLIP3-o provides parameter-efficient scaling:

• 4B: 3B AR backbone + 0.4B DiT head
• 8B: 7B AR backbone + 1.4B DiT head

Sequence length is reduced by employing CLIP embeddings (64 vectors) versus typical VAE latents (256–1024 tokens), conferring a %%%%16$Q$17%%%% acceleration in DiT. Inference cost clocks at $Q$82 TFLOPs per 1024$Q$9 resolution image; pretraining (8B) is feasible within 5 days on 64 A100s ($(X_t, Q, t)$040k GPU-hr).

Open access to model code, weights (4B & 8B), all pretraining and instruction datasets, and evaluation pipelines is provided via https://github.com/JiuhaiChen/BLIP3o and Hugging Face repositories, enabling broad reproducibility and extensibility (Chen et al., 14 May 2025).

7. Empirical Insights and Design Implications

Systematic study within BLIP3-o quantitatively establishes the superiority of CLIP-based representations and flow matching objectives for unified multimodal modeling. Sequential curriculum is empirically shown to maintain peak image understanding performance while enabling strong generative capabilities, as opposed to joint multitask approaches. Expanding DiT head capacity proportionally improves alignment and generative metrics, with doubling DiT width lowering $(X_t, Q, t)$1 by 12% and GenEval increasing by 0.03. Fixed-sequence CLIP features further enhance efficiency.

A plausible implication is that, for unified multimodal systems, discrete and low-dimensional semantic features (CLIP) paired with gradient-stable flow-based diffusion offer an effective compromise between sample diversity, semantic fidelity, and runtime cost.

---

For source code, model checkpoints, datasets, and evaluation details, see the official project repository and paper (Chen et al., 14 May 2025).