STARFlow2: Unified Multimodal Generation

• STARFlow2 is a unified multimodal generative model that integrates large language models with autoregressive normalizing flows to synthesize interleaved text and image sequences in a single pass.
• It employs a vertical interleaving of a frozen visual-language model and a trainable TARFlow stream, enabling efficient cross-modal fusion and cache-optimized generation.
• Empirical results confirm high-fidelity image synthesis and robust multimodal reasoning, setting a new standard for unified text-to-image generation architectures.

STARFlow2 is a unified multimodal generative model that integrates LLMs and autoregressive normalizing flows to enable single-pass, causal modeling of interleaved text and image sequences. It addresses fundamental limitations in prior vision-language architectures by harmonizing the structure of both linguistic and visual generation under a shared left-to-right Transformer paradigm, achieving both high-fidelity image synthesis and robust visual-language understanding without resorting to re-encoding or hybrid decoding mechanisms (Shen et al., 8 May 2026).

1. Unified Multimodal Generation: Motivation and Problem Formulation

STARFlow2 is designed to model interleaved multimodal sequences $C = (x_1, …, x_T)$, where each $x_t$ may represent a discrete text token or a continuous image latent. The objective is to support both multimodal understanding (e.g., image-to-text) and generation (e.g., text-to-image) within a single-pass, autoregressive, and cache-efficient model.

Prior modalities for unification suffered from the following deficiencies:

• Discrete visual tokenization (e.g., VQ-VAE) leads to loss of visual fidelity.
• Diffusion-based image generation plus autoregressive fusion results in mismatches between the iterative visual denoising and causal text generation, leading to the need for re-encoding generated images for reuse in the sequence.
• Mixture-of-Transformers (MoT) approaches split Transformer feedforward parameters by modality, breaking parameter sharing and introducing a trade-off: freezing VLM hurts image generation, while fine-tuning VLM degrades multimodal reasoning.

STARFlow2’s core insight is to deploy autoregressive normalizing flows, parameterized by a causal Transformer (TARFlow), which inherently match the left-to-right causal structure and key-value cache mechanism of LLMs. This enables continuous latent modeling of images with exact likelihood and a strict unification of generation protocols for both modalities.

2. Architecture: Pretzel Design and Vertical Interleaving

The STARFlow2 architecture adopts the "Pretzel" configuration, implementing two vertically interleaved streams across shared Transformer layers, with common causal masking and KV-cache:

• VLM stream: A frozen Qwen2.5-VL-7B visual-LLM for text and multimodal understanding.
• TARFlow stream: A trainable autoregressive normalizing flow over visual latents; also models text as categorical.

The vertical interleaving is realized via residual skip connections:

• At a visual position $t$: TARFlow receives

$$\hat{h}_t = x_t + W_{\text{vlm}} \cdot h_{\text{vlm},t}$$

ensuring access to both the low-level image latent $x_t \in \mathbb{R}^D$ and the high-level VLM feature $h_{\text{vlm},t}$.

• At a text position $t$: the final language modeling logits are formed as

$h_{\text{vlm},t} + W_D \cdot h_{D,t}$

which allows TARFlow to lightweightly refine the VLM’s text representations.

Both $W_{\text{vlm}}$ and $W_D$ are zero-initialized, meaning the model’s initial behavior strictly mirrors the frozen VLM and pretrained flows, tuning towards cross-modal fusion during training.

3. Deep–Shallow Flow Design and Unified FAE Latent Space

Images are mapped into a grid of continuous latents $x_t$0 by a frozen Feature Autoencoder (FAE). The flow framework is composed of:

• Shallow AF blocks $x_t$1: Remove short-range spatial correlations through visually-only flows with alternating scan directions.
• Deep TARFlow block $x_t$2: Implements a causal autoregressive normalizing flow, where for $x_t$3 positions:

$x_t$4

and sampling is done via $x_t$5 with $x_t$6.

The composite change of variables yields the exact log-likelihood:

$x_t$7

where $x_t$8 and $x_t$9 is the standard Gaussian.

All modalities, including discrete tokens, are processed by the same autoregressive Transformer, allowing direct caching of their representations in the shared key-value cache with no need for re-encoding.

Pseudocode Sketch for Multimodal Decoding

$x_t \in \mathbb{R}^D$0

4. Training Procedure and Objectives

STARFlow2 is trained in three stages:

1. Stage 1 (Text→Image Generation):

Train the TARFlow stream $t$0 and shallow flows $t$1 with VLM frozen. The loss objective is negative log-likelihood for flows:

$t$2

2. Stage 2 (Image→Text Understanding): Freeze $t$3 and VLM, training only a small adapter from $t$4 to VLM-space using standard next-token-prediction loss:

$t$5

3. Stage 3 (Interleaved Joint Training): All vertical skips are activated with zero-init. The loss is:

$t$6

Tasks in this stage include pure understanding, text-to-image generation, editing, and interleaved generation.

Optimization specifics include AdamW ($t$7, $t$8, $t$9e$$\hat{h}_t = x_t + W_{\text{vlm}} \cdot h_{\text{vlm},t}$$0), weight decay $$\hat{h}_t = x_t + W_{\text{vlm}} \cdot h_{\text{vlm},t}$$1e$$\hat{h}_t = x_t + W_{\text{vlm}} \cdot h_{\text{vlm},t}$$2, and learning rates scaling from $$\hat{h}_t = x_t + W_{\text{vlm}} \cdot h_{\text{vlm},t}$$3e$$\hat{h}_t = x_t + W_{\text{vlm}} \cdot h_{\text{vlm},t}$$4 (stages 1/2) to $$\hat{h}_t = x_t + W_{\text{vlm}} \cdot h_{\text{vlm},t}$$5e$$\hat{h}_t = x_t + W_{\text{vlm}} \cdot h_{\text{vlm},t}$$6 (stage 3). Data comprises approximately 800M captioned images (including CC12M and JourneyDB), 200M image→text pairs, and 80M mixed multimodal sequences (from BLIP3, Cambrian, CoMM, Pico-Banana, OmniEdit, Zebra-CoT, among others).

5. Empirical Results

STARFlow2 (10.6B parameters) achieves strong performance across both multimodal understanding and text-to-image generation tasks:

• Multimodal understanding:
  • MME-P: 1 528.8 vs. Qwen-VL’s 1 677.9
  • GQA: 55.8 (Qwen-VL: 60.7)
  • SEED: 71.1 (Qwen-VL: 75.5)
• Text-to-image generation:
  • GenEval: 0.82 (final), improving from 0.51 post-Stage 1 (+60.8%)
  • DPG-Bench: 84.94 (final), up from 82.02 (+3.6%)

Qualitative results demonstrate faithful attribute-based edits and multi-turn consistency for interleaved editing and generation scenarios.

6. Comparative Analyses and Ablations

STARFlow2’s Pretzel vertical skip architecture outperforms horizontal parameter splits (MoT). Specifically:

• MoT-style fusion (horizontal split):
  • Freezing VLM: image quality degrades due to insufficient cross-modal flow signal.
  • Fine-tuning VLM: leads to a collapse in multimodal reasoning (MME scores drop by ≈50%).
• Vertical Skip Analysis:
  • Visual skip contribution ($$\hat{h}_t = x_t + W_{\text{vlm}} \cdot h_{\text{vlm},t}$$7): The VLM projection magnitude is nearly half the total, and the directional cosine is $$\hat{h}_t = x_t + W_{\text{vlm}} \cdot h_{\text{vlm},t}$$8, indicating complementary feature fusion.
  • Text skip contribution ($$\hat{h}_t = x_t + W_{\text{vlm}} \cdot h_{\text{vlm},t}$$9): TARFlow introduces minor corrections to text, largely preserving the pretrained LM.

Ablations of interleaved training demonstrate that Stage 3 joint fusion improves generation metrics while maintaining or improving understanding capability.

7. Significance and Implications

STARFlow2 demonstrates that autoregressive normalizing flows, interleaved with frozen visual-LLMs using the Pretzel vertical skip mechanism and a unified causal Transformer stack, provide an effective solution for unified multimodal modeling. This architecture achieves simultaneous high-fidelity continuous image synthesis, strong multimodal understanding, and efficient interleaved text-image generation—all under a single, left-to-right, cache-optimized generative mechanism (Shen et al., 8 May 2026). A plausible implication is that architectures following this paradigm may set a new standard for backbone models in interleaved multimodal reasoning and generation tasks.