Sequential Diffusion Tokenizer

• The paper establishes a framework where diffusion processes guide token extraction, aligning token generation with a coarse-to-fine temporal structure.
• It details architectural variants like DiTo, FlowMo, and D-AR that use convolutional and transformer encoders to achieve efficient, progressive image reconstruction.
• Empirical results show competitive reconstruction and generative performance, supporting applications in streaming previews and conditional image synthesis.

A Sequential Diffusion Tokenizer is an architectural and algorithmic framework for mapping visual data, typically images, to sequences of tokens via a process tightly coupled to diffusion, and optionally, autoregressive or sequential generative modeling. The defining characteristic is that token generation and utilization are coherently aligned with the temporal (coarse to fine) structure of diffusion processes, allowing for natural integration with next-token autoregressive models, streaming preview mechanisms, latent-variable image synthesis, and powerful generative models. This entry surveys the central methodologies, architectures, theoretical underpinnings, and empirical properties of state-of-the-art sequential diffusion tokenizers, with reference to recent advances including DiTo, FlowMo, and D-AR (Chen et al., 30 Jan 2025, Sargent et al., 14 Mar 2025, Gao et al., 29 May 2025).

1. Architectural Principles

Sequential diffusion tokenizers appear in multiple instantiations unified by a shared pipeline: an encoder projects the input image to a 1D (optionally quantized) sequence of latent tokens, and a diffusion decoder reconstructs or generates images conditioned on subsets or complete sequences of these tokens, often following a time-aligned (coarse-to-fine) schedule.

Key architectural variants are summarized below:

Model	Encoder Type	Token Type	Decoder Type	Spatial Structure
DiTo	ConvNet	1024×4d (continuous)	U-Net (diffusion)	Grid (flattened)
FlowMo	Transformer (MMDiT)	S×D (quantized ±1)	Transformer (MMDiT, ODE flow)	Flat 1D sequence
D-AR	Transformer	N (VQ, discrete idx)	Transformer (diffusion)	Flat 1D sequence

• In DiTo (Chen et al., 30 Jan 2025), the encoder is convolutional, outputting a $4 \times 32 \times 32$ grid flattened as 1024 latent vectors.
• FlowMo (Sargent et al., 14 Mar 2025) employs a transformer encoder and decoder (MMDiT), producing quantized ±1 latent codes as a 1D sequence. Patches are projected and tied to learned tokens, with full transformer-style attention and AdaLN conditioning on diffusion time.
• D-AR (Gao et al., 29 May 2025) utilizes a transformer encoder with learnable queries for token extraction, vector quantization using a large codebook, and a diffusion transformer decoder for pixel-level denoising.

Notably, all contemporary designs eliminate explicit 2D alignment or codebook routing and favor pure 1D sequences suitable for integration with transformer-based AR generative models.

2. Diffusion Tokenization and Decoding

The sequential diffusion tokenizer couples the mapping from images to tokens to the structure of diffusion processes, either as conditional autoencoding or full autoregressive generation.

• Forward Process: For time $t \in [0,1]$, the image $x$ is mapped to a noisy intermediate $x_t = \alpha_t x_0 + \sigma_t \epsilon$ with schedule parameters (e.g., $\alpha_t = 1-t$).
• Token Extraction: The encoder produces a sequence of tokens, either as continuous latents (DiTo), quantized binary codes (FlowMo), or VQ indices (D-AR).
• Decoding/Generation:
  • In DiTo and FlowMo, a diffusion decoder, given a noisy $x_t$, time step $t$, and full token sequence $z$ (or $c$), predicts the velocity $v$ to enable denoising or generation under a learned inverse diffusion trajectory.
  • D-AR uniquely supports partial conditioning: any prefix of the token sequence can be used to condition the decoder to produce progressively more refined previews, aligning token indices to diffusion time steps in a group-wise (coarse-to-fine) schedule (Gao et al., 29 May 2025).

The use of flow-matching continuous ODEs (rather than discrete DDPM steps) is common, allowing efficient backward integration with a small number of solver steps (e.g., 25).

3. Sequentiality: Token Ordering and Coarse-to-Fine Structure

Sequential diffusion tokenizers implement a strict or implicit ordering of tokens:

• Token Grouping: In D-AR, $N$ tokens are divided into $K$ groups; the decoder at each diffusion step $t_k$ conditions on the first $k$ groups, mirroring inverse diffusion by progressively "revealing" finer details (Gao et al., 29 May 2025).
• Autoregressive Modeling: The entire sequence is autoregressively generated using standard next-token prediction (causal masking, e.g., with a Llama backbone), with no modifications to the core attention architecture. This approach ensures compatibility with high-throughput KV-caching and stream generation.
• Coarse-to-Fine Property: Early tokens specify global structure in noisy contexts; later tokens encode high-frequency details as the diffusion process approaches denoising (low-noise) limits.
• Streaming Previews: Because each token group corresponds to a reverse diffusion step, the pipeline naturally supports streaming image previews, where partial token sequences and corresponding reverse steps yield lower-fidelity previews (Gao et al., 29 May 2025).

This joint alignment of token sequence, diffusion time, and autoregressive generation distinguishes sequential diffusion tokenizers from earlier VQ- or GAN-based tokenizers.

4. Training Objectives and Theoretical Underpinnings

The diffusion autoencoder and AR training objectives are grounded in variational ELBO maximization and flow-matching theory.

• Diffusion Reconstruction Loss: The decoder is trained to minimize the mean-squared deviation from the true denoising velocity, matching the flow-matching loss or ELBO:

$$L = \mathbb{E}_{t, \epsilon} \left\| D(x_t, t, z) - \left[(1-\sigma_{\min})\epsilon - x_0\right] \right\|_2^2$$

(DiTo (Chen et al., 30 Jan 2025)), or analogous for FlowMo.

• Token Quantization Losses: For models with discrete tokens (FlowMo, D-AR), vector quantization commitment and entropy regularization terms are used, e.g., VQ-VAE-style objective (Gao et al., 29 May 2025).
• Perceptual & Mode-Seeking Losses: FlowMo adds an LPIPS-based one-step denoising perceptual loss during joint training, and perceptual LPIPS-resnet reward sampled along full ODE chains in a post-training phase that encourages mode-seeking behavior (Sargent et al., 14 Mar 2025).
• Autoregressive Cross-Entropy: D-AR minimizes standard AR cross-entropy over the token sequence for next-token prediction:

$$\mathcal{L}_{\mathrm{AR}} = -\sum_{i=1}^N \log p_\theta(z_i \mid z_{<i})$$

(Gao et al., 29 May 2025).

Recent theoretical analyses (Kingma & Gao, NeurIPS 2024; Lipman et al, ICLR 2023) establish that the diffusion ELBO and flow-matching losses optimize a lower bound on the marginal likelihood, providing principled justification for using a single L2 loss (with no need for GAN or perceptual losses) (Chen et al., 30 Jan 2025).

5. Empirical Properties and Benchmark Results

Sequential diffusion tokenizers provide competitive or superior performance on standard datasets and metrics.

• Reconstruction Fidelity:
  • DiTo: rFID (lower is better). DiTo-XL, with LPIPS, achieves rFID = 3.53, surpassing GLPTo-XL (rFID = 4.14) on 5K images. Human A/B testing shows that DiTo-XL reconstructions are preferred over GLPTo-XL in ~52% of cases (Chen et al., 30 Jan 2025).
  • FlowMo: FlowMo-Hi at 0.219 BPP attains rFID = 0.56, PSNR = 24.93, SSIM = 0.785, and LPIPS = 0.073 (Sargent et al., 14 Mar 2025).
  • D-AR: Tokenizer reconstructions with 256 tokens, 8 steps, Adams solver reach rFID = 1.52 (Gao et al., 29 May 2025).
• Generative Performance:
  • DiTo tokens enable downstream latent diffusion models to achieve gFID = 7.41 (with noise synchronization), competitive with GLPTo-XL (Chen et al., 30 Jan 2025).
  • D-AR achieves FID = 2.09 on ImageNet 256×256 with a 775M Llama backbone, matching or exceeding non-diffusion tokenizers (Gao et al., 29 May 2025).
• Ablations: D-AR demonstrates that increasing codebook size improves rFID; increasing $K$ (number of token groups) raises rFID, suggesting trade-offs between sequence length, detail, and fidelity (Gao et al., 29 May 2025).
• Compression Efficiency: FlowMo operates in regimes as low as 0.070 BPP with strong perceptual metrics, far outperforming prior MagViT-V2 and LlamaGen-32 baselines (Sargent et al., 14 Mar 2025).

6. Applications and Functional Capabilities

Sequential diffusion tokenizers enable a suite of advanced generative modeling capabilities:

• Streaming Generation and Previews: Generation may be conducted in a streaming manner, revealing successively higher-quality images as token groups and corresponding diffusion steps are completed (Gao et al., 29 May 2025).
• Zero-Shot Layout-Controlled Synthesis: By fixing a prefix of tokens (e.g., derived from a semantic mask or layout code) and sampling only the remainder, D-AR allows for layout or content control without modification or retraining (Gao et al., 29 May 2025).
• Downstream Conditioning: Token sequences can be fed directly to large transformers (e.g., DiT-XL/2), serving as text-like input or via cross-attention to enable conditional generation or manipulation (Chen et al., 30 Jan 2025).
• Compression and Perceptual Coding: State-of-the-art reconstructions at controlled BPPs permit deployment in visual compression and storage settings (Sargent et al., 14 Mar 2025).

7. Limitations and Open Directions

A principal limitation of sequential diffusion tokenizers is decoding cost, with diffusion reconstruction typically requiring 8–25 ODE solver steps per image, significantly higher than single-pass GAN-based or non-diffusive tokenizers (Sargent et al., 14 Mar 2025). The use of continuous rather than discrete codes (DiTo) may restrict compatibility with AR language modeling architectures, while quantization can impact fidelity. A plausible implication is that future work on diffusion-distillation, reduced-step ODE solvers, and hybrid quantization will further close the gap in efficiency and broaden application scope.

Recent advances suggest that the sequential alignment between tokens, diffusion time, and autoregressive generation facilitates new capabilities in streaming vision models and multimodal transformers, offering a fertile ground for further research in unified generative modeling (Gao et al., 29 May 2025).