---
title: 'MADFormer: Hybrid AR-Diffusion Transformer'
url: https://www.emergentmind.com/topics/madformer
type: topic
---

# MADFormer: Hybrid AR-Diffusion Transformer

MADFormer is a hybrid generative Transformer model that systematically integrates autoregressive (AR) and diffusion-based generation mechanisms within a unified architecture for continuous image generation. It addresses the allocation of model capacity between AR and diffusion modules, using a spatial block partitioning of image representations and mixing AR and diffusion objectives across the depth of the Transformer network. Empirical studies on FFHQ-1024 and ImageNet datasets reveal that careful spatial partitioning and vertical (layer-wise) mixing deliver substantial improvements in sample quality under corresponding computational budgets [2506.07999].

## 1. Model Architecture and Data Representation

MADFormer encodes multimodal inputs via discrete and continuous channels. Input text is tokenized using Llama 3, while images are encoded by a frozen VAE (e.g., Stable Diffusion VAE) into continuous latent representations $z_{\rm image} \in \mathbb{R}^d$, which are subsequently linearized in raster order. These latents are partitioned into $L$ contiguous spatial blocks, e.g., $L=16$ blocks of $256 \times 256$ for FFHQ-1024.

All modalities propagate through a single stack of $N$ Transformer layers, but utilize separate FFN and QKV parameters by modality. The early layers $1 \dots (N\!-\!D)$ implement an AR, causally-masked modeling objective, ingesting tokenized text plus previously generated image blocks to produce per-block AR conditioning vectors $c^{(i)}$. The remaining layers $(N\!-\!D+1)\dots N$ operate under a diffusion denoising objective on noisy target block $z^{(i)}$ at each diffusion step $t$, conditioning on the AR-computed $c^{(i)}$.

Blockwise, the generative process comprises:

1. AR pass: Compute $c^{(i)}=f_{\rm AR}(z^{(<i)})$ via causal attention.
2. Diffusion pass: Iteratively denoise $x_t^{(i)} \sim q(x_t^{(i)}|z^{(i)})$ to $\hat{z}^{(i)}_0$ conditioned on $c^{(i)}$.

The architecture offers a mid-level decomposition of global structure (via AR) and local refinement (via diffusion).

## 2. Mathematical Formulation

### 2.1 Autoregressive Prior

Given latent blocks $z=(z^{(1)},\dots,z^{(L)})$, the AR prior factorizes as:
\[
p_{\rm AR}(z) = \prod_{i=1}^L p_{\rm AR}(z^{(i)}|z^{(<i)})
\]
Hidden states evolve by
\[
h^{(i)} = f_{\rm AR}(h^{(i-1)}, z^{(i-1)}), \quad p_{\rm AR}(z^{(i)}|z^{(<i)}) = \mathcal{N}(z^{(i)};\, \mu_{\rm AR}(h^{(i)}), \Sigma_{\rm AR})
\]

### 2.2 Diffusion-Based Denoising

The forward (noising) process for block $z^{(i)}$:
\[
q(x_t|x_{t-1}) = \mathcal{N}(x_t; \sqrt{1-\beta_t} x_{t-1}, \beta_t I), \quad x_t = \sqrt{\bar{\alpha}_t} z^{(i)} + \sqrt{1-\bar{\alpha}_t} \epsilon
\]
where $\epsilon \sim \mathcal{N}(0, I)$ and $\bar{\alpha}_t = \prod_{s=1}^t (1-\beta_s)$.

Reverse modeling:
\[
p_\theta(x_{t-1}|x_t, c) = \mathcal{N}\left(x_{t-1}; \mu_\theta(x_t, t, c), \sigma_t^2 I\right)
\]
with diffusion loss:
\[
\mathcal{L}_{\rm diff} = \mathbb{E}_{t,z,\epsilon}\| \epsilon_\theta(x_t, t, c) - \epsilon \|^2
\]

### 2.3 Training Objective

The total loss is a weighted sum:
\[
\mathcal{L} = \lambda_{\rm text} \, \mathcal{L}_{\rm text} + \lambda_{\rm image} \, \mathcal{L}_{\rm image} + \lambda_{\rm hidden} \, \mathcal{L}_{\rm hidden} + \lambda_{\rm tower} \, \mathcal{L}_{\rm tower}
\]
with typical weights $\lambda_{\rm text}=1$, $\lambda_{\rm image}=5$, $\lambda_{\rm hidden}=0.1$, $\lambda_{\rm tower} \approx 0$.

## 3. Inference Procedure

Image generation proceeds blockwise, combining AR conditioning with iterative diffusion. The organism of the process is:

```python
# Pseudocode for inference in MADFormer
Initialize Z_out = []
for i in 1…L:  # for each block
    c = AR_layers_forward(z^{(<i)}, text)
    x_T ~ Normal(0, I)
    for t = T…1:
        h0 = embed(x_t) + c
        hD = Diffusion_layers_forward(h0)
        epsilon_pred = epsilon_theta(hD, t)
        mu_t = (x_t - sqrt(1-alpha_bar_t) * epsilon_pred) / sqrt(alpha_bar_t)
        x_{t-1} = mu_t + sigma_t * Normal(0, I)
    z^{(i)} = x0
    append z^{(i)} to Z_out
# VAE decoder decodes Z_out to image
```

This staged approach divides the generative burden: AR produces core spatial layouts, while diffusion executes high-fidelity refinement.

## 4. Computational Complexity and Quality-Efficiency Trade-offs

Let $N$ be the total Transformer layers ($N\!-\!D$ AR, $D$ diffusion), $T$ the number of diffusion steps, $L$ the number of blocks, and $C$ the per-layer evaluation cost. The regimes compare as follows:

| Model Type         | Function Evals (NFE)            | Computational Cost (per image)           |
|--------------------|---------------------------------|------------------------------------------|
| Pure AR            | $1$                             | $L (N C)$                               |
| Pure Diffusion     | $T$                             | $L T (N C)$                             |
| Mixed MADFormer    | $\frac{(N\!-\!D) + T D}{N}$     | $L C [(N\!-\!D) + T D]$                 |

Empirical analysis demonstrates:
- For small NFE, AR-heavy splits (large $N\!-\!D$, small $D$) improve FID by 60–75% over pure diffusion.
- As NFE increases, diffusion-heavy splits (larger $D$) surpass in final image fidelity.

This relationship can be fitted by:
\[
\Delta \mathrm{FID}(B) = \gamma \exp(-\alpha B)
\]
with $\alpha, \gamma$ parameters estimated from observed curves.

## 5. Experimental Results

Experiments on FFHQ-1024 and ImageNet 256 clarify the modeling regimes:

### FFHQ-1024 ($1024\times1024$ resolution, $N=28$)
- **Diffusion-depth ablation**: $d=7 \Rightarrow$ FID=20.2; $d=14 \Rightarrow$ 17.8; $d=21 \Rightarrow$ 16.6; $d=28 \Rightarrow$ 15.9.
- **Optimal block partition**: $L=16$ yields FID=17.8 versus 18.9 ($L=4$) or 21.9 ($L=64$).
- **Under only 9 diffusion steps (NFE$\simeq$280)**: AR:Diff$=21$:7, FID$\approx$20 (global facial coherence with low step count).

### ImageNet 256
- **Diffusion-depth ablation**: $d=7 \Rightarrow$ FID=34.0; $d=14 \Rightarrow$ 30.0; $d=21 \Rightarrow$ 28.1; $d=28 \Rightarrow$ 27.4.
- **Best AR-length**: $L=1$ (single block), FID=28.4. Larger $L$ fragments context, degrading quality.

In all tested settings, low NFE favors AR-heavy mixing, while increased compute allows diffusion to ultimately minimize FID further.

## 6. Design Principles and Best Practices

Several practical design principles are established:

- **Blockwise Partitioning**: For high-res images, increase AR blocks ($L=16$ at $1024\times1024$). For mid-res images ($256\times256$), a single block suffices.
- **Vertical Mixing**: Early AR layers (ratio $\sim$3:1 for AR:Diff) are critical for quality at low compute; later diffusion layers target local refinement. Adjust the AR/diffusion layer count to fit available NFE.
- **Auxiliary Losses**: Hidden loss on AR conditioning ($\|c^{(i)}-z^{(i)}\|^2$) yields $\sim$1.6 FID gain; clean tower provides $\sim$2 FID gain; causal attention is necessary (removal increases FFHQ FID from 17.8 to 21.2).
- **Parameter Sharing**: Distinct FFN/QKV sets for text, clean, and noisy blocks show negligible benefit; parameter sharing suffices.

The systematic fusion of AR and diffusion in MADFormer, via spatial partitioning and vertical task mixing, enables principled speed–quality tradeoffs. For constrained inference, AR-heavy hybrids dominate, while aggressive diffusion allocation produces the lowest FID with sufficient compute. These principles inform the architecture of future hybrid generative-vision models [2506.07999].

Source: https://www.emergentmind.com/topics/madformer