---
title: Pyramidal Patchification Flow (PPFlow)
url: https://www.emergentmind.com/topics/pyramidal-patchification-flow-ppflow
type: topic
---

# Pyramidal Patchification Flow (PPFlow)

Pyramidal Patchification Flow (PPFlow) is a framework for accelerating diffusion transformers (DiTs) in visual generation by dynamically adjusting patchification granularity throughout the denoising timeline. Rather than using a fixed patch size for the full reverse diffusion process, PPFlow segments time into pyramidal intervals, applying larger patches (lower token count) at high-noise (early) timesteps and smaller patches (higher fidelity) at low-noise (late) steps. This approach preserves full-resolution latent representations and leverages distinct, learnable input/output projections for each patch size, integrating with standard DDPM/DDIM samplers while requiring no core changes to DiT blocks or auxiliary renoising mechanisms. Empirically, PPFlow delivers up to 2.0× inference speedup with equivalent or improved image generation metrics relative to baseline DiTs [2506.23543].

## 1. Formal Definition and Patchification Schedule

PPFlow partitions the normalized diffusion timeline $t \in [0,1]$, or discrete timesteps $t = 0, \ldots, T$, into $L$ contiguous intervals $[\tau_0, \tau_1), [\tau_1, \tau_2), \ldots, [\tau_{L-1}, \tau_L]$ with $\tau_0 = 0$, $\tau_L = 1$. Each interval $\mathcal{I}_l = [\tau_{l-1}, \tau_l)$ is assigned a patch size $p_l \in \mathbb{N}$ (in pixels). High-noise (early) stages use larger $p_l$ to minimize token count; low-noise (late) stages use smaller $p_l$ to maximize detail.

The patch size as a function of (continuous) time is
$$
s(t) = \sum_{l=1}^L p_l \cdot \mathbb{1}[t \in [\tau_{l-1}, \tau_l))
$$
At discrete step $t$, $s_t = s(t/T)$. Example schedules include:
- 2-level: $\tau_1 = 0.5$, $p_1 = 4$, $p_2 = 2$.
- 3-level: $\tau_1 = 0.5$, $\tau_2 = 0.75$, $p_1 = 4$, $p_2 = 2\sqrt{2}$, $p_3 = 2$ (with $p_2$ realized by grouping 2×2 patches).

## 2. Mathematical Formulation and Operators

Given latent $X \in \mathbb{R}^{C \times H \times W}$ and patch size $s$:
- Number of patches: $N_s = (H/s) \cdot (W/s)$.
- Per-patch dimension: $D_s = C \cdot s^2$.

### Patchify Operator
$\text{Patchify}_s \colon \mathbb{R}^{C \times H \times W} \to \mathbb{R}^{N_s \times d}$
1. Extract non-overlapping patches $\{ P_i \}_{i=1 \ldots N_s}$, each $P_i \in \mathbb{R}^{C \times s \times s}$.
2. Flatten $P_i \to p_i \in \mathbb{R}^{D_s}$.
3. Project: $T_i = p_i W_s$, where $W_s \in \mathbb{R}^{D_s \times d}$. Output $T \in \mathbb{R}^{N_s \times d}$.

### Unpatchify Operator
$\text{Unpatchify}_s \colon \mathbb{R}^{N_s \times d} \to \mathbb{R}^{C \times H \times W}$
1. For each token $T_i \in \mathbb{R}^d$, recover $p'_i = T_i W'_s$ with $W'_s \in \mathbb{R}^{d \times D_s}$.
2. Reshape $p'_i$ to $\mathbb{R}^{C \times s \times s}$ and re-tile to reconstruct $\hat{X} \in \mathbb{R}^{C \times H \times W}$.

### Integration with Diffusion Sampling
Noise prediction $\hat{\varepsilon} = \epsilon_\theta(\text{Patchify}_{s_t}(x_t), t)$; DDPM update:
$$
x_{t-1} = \frac{1}{\sqrt{\alpha_t}}\left( x_t - \frac{1-\alpha_t}{\sqrt{1-\bar{\alpha}_t}} \cdot \text{Unpatchify}_{s_t}(\hat{\varepsilon}) \right) + \sigma_t z,\quad z \sim \mathcal{N}(0, I)
$$
Standard DDIM formula similarly applies, replacing with the respective re-parameterizations.

## 3. Architectural and Implementation Considerations

PPFlow contributes $L$ distinct pairs of projection matrices $\{ W_s, W'_s \}$ (one per patch size), added outside the intact DiT/SiT transformer blocks. All core model parameters (self-attention, MLPs) are shared, and the internal sequence length dynamically matches the number of patches determined by $s_t$. During training, packing techniques such as "patch n' pack" are used to efficiently batch samples with varying token counts.

Positional embeddings are recalculated per sequence length (i.e., per-timestep patch size), using either 2D sinusoidal or learned positional codes. Optionally, a stage-dependent patch-level embedding $p_l \in \mathbb{R}^d$ is injected, which improves FID. No masking or specialized scheduling (such as renoising tricks) is required.

## 4. Training Protocols and Empirical Performance

Two principal regimes are established:
- **Training from Scratch:** With SiT-B/2 backbone on 256×256 ImageNet z-latents. PPF-B-2 (4→2 patch size) and PPF-B-3 (4→2√2→2) are trained for 7M steps. PPF-B-2 uses 62.5% of SiT-B/2 training FLOPs; PPF-B-3 uses 50.0%. Final FID-50K (250 steps): SiT-B/2: 4.46; PPF-B-2: 4.12 (62.5% FLOPs) → 3.83 (98.2% FLOPs, 11M steps); PPF-B-3: 4.71 (50.0%) → 4.43 (78.5%, 11M steps). Inference speedup (A100): 1.61× for PPF-B-2, 2.04× for PPF-B-3.
- **Finetuning from Pretrained DiT:** PPF-B-2 and PPF-B-3 from SiT-B/2 and PPF-XL-2, PPF-XL-3 from SiT-XL/2, with only 1M additional steps (8–9% FLOPs). Inference FLOPs: PPF-B-2 at 62.0% (1.6×), PPF-B-3 at 49.1% (2.0×), PPF-XL-2 at 62.6% (2.02×), PPF-XL-3 at 49.4%. Best FID: PPF-XL-2 1.99 vs SiT-XL/2 2.15. Tables 1–3 and Figures 1, 4, 6 present detailed benchmarking [2506.23543].

## 5. Sampling Algorithm

The PPFlow sampling process is as follows:

```pseudo
Require: pretrained model $f_\theta$, breakpoints $\{\tau_l\}$, patch sizes $\{p_l\}$, noise schedule $\{\alpha_t\}$
$x_T \sim \mathcal{N}(0, I)$
for $t = T, \ldots, 1$ do
    $u \leftarrow t / T$
    find $l$ such that $u \in [\tau_{l-1}, \tau_l)$
    $s \leftarrow p_l$
    $Z \leftarrow \text{Patchify}_s(x_t)$
    $\hat{\varepsilon} \leftarrow f_\theta(Z, t)$
    $\tilde{X} \leftarrow \text{Unpatchify}_s(\hat{\varepsilon})$
    update $x_{t-1}$ by DDPM rule with $\tilde{X}$
return $x_0$
```

## 6. Ablation Findings and Comparative Analyses

Experiments demonstrate:
- The number of patchification levels ($L$) trades speed for fidelity: 2-level yields 1.6× speedup (FID ≈ 3.8–4.1); 3-level yields 2.0× (FID ≈ 4.4–4.7).
- Adding patch-level embeddings improves FID by 0.8 (22.56→21.73 at 400k steps).
- Applying stage-wise classifier-free guidance (CFG) further reduces FID (21.73→13.13 at 400k steps).
- Compared to pyramid-representation flow [23], PPFlow achieves lower FID and crisper images without the need for renoising scheduling (Table 3, Figure 8).

## 7. Advantages, Constraints, and Prospects

PPFlow yields up to 2.0× inference acceleration while matching or improving metrics such as FID, IS, sFID, and precision/recall against conventional DiTs. Because it maintains full-resolution latents and modifies only the input/output projections, DiT internals remain unaltered, facilitating application in both from-scratch and finetuning workflows. No jump points or auxiliary renoising are necessary.

Current limitations include restriction to class-conditional ImageNet; extension to text-to-image or other modalities is uninvestigated. Patchification schedules are fixed; learned or adaptive scheduling may offer further benefits. Additional stages ($L>3$) or non-uniform grouping could push speedups but introduce challenges in token sequence management.

Potential extensions include combining PPFlow with step-reduction procedures (DDIM, distillation), end-to-end learned stage breakpoints, or extending to video and 3D diffusion transformers where token explosion is prevalent [2506.23543].

Source: https://www.emergentmind.com/topics/pyramidal-patchification-flow-ppflow