Fast Diffusion Transformer: Acceleration Methods

• Fast-DiT is a family of acceleration strategies for Diffusion Transformers that reduce training costs through methods like masked training and SNR-guided recipes.
• It enhances inference efficiency by employing caching, dynamic token routing, and selective execution to minimize redundant computations while preserving image quality.
• Deployment efficiency is achieved through quantization, pruning, and backbone replacements, enabling faster execution on resource-constrained devices.

Fast Diffusion Transformer (fast-DiT) denotes a line of research that seeks to reduce the cost of Diffusion Transformers (DiTs), which have become the de facto choice for generating high-quality images and videos, while preserving the generative quality that motivated the DiT family in the first place. In the literature, the term is used for faster training, faster inference, and deployment-oriented efficiency, including masked training, timestep- and region-adaptive execution, temporal activation caching, quantization, pruning, and highly compressed latent representations. This suggests that fast-DiT is best understood as a family of acceleration strategies rather than a single canonical model (Zheng et al., 2023, Yao et al., 2024, Selvaraju et al., 2024, Yin et al., 28 Jun 2026).

1. Scope and research program

Across the literature, fast-DiT work targets three recurring bottlenecks: convergence during training, repeated full-stack execution during sampling, and deployment costs from model size, activation footprint, and attention-kernel behavior. Some methods preserve the original DiT architecture and alter only the training recipe; others introduce dynamic routing, masking, or caching; still others pursue low-precision inference or replace the DiT backbone with a more efficient substitute (Yao et al., 2024, Zhao et al., 9 Apr 2025, Ding et al., 10 Mar 2025, Ai et al., 16 May 2025).

Objective	Representative methods	Core mechanism
Faster training	MaskDiT, FasterDiT, FastDiT-3D	masked training, SNR/PDF-guided recipes, extreme voxel masking
Faster inference	FORA, $\Delta$-DiT, ICC, FastCache, SAFE-DiT, DyDiT++	caching, dynamic execution, selective updates, fast-path mask elision
Deployment efficiency	HTG PTQ, mobile video DiT, FSVideo, DiCo	quantization, pruning, compressed latents, ConvNet replacement

A central organizing distinction is whether a method reduces training cost, per-step inference cost, or the number of effective computations at deployment. This distinction is not cosmetic. Several of the most influential fast-DiT papers explicitly accelerate training but do not materially reduce inference cost, whereas others are entirely training-free and modify only the sampling-time execution path (Zheng et al., 2023, Yao et al., 2024, Selvaraju et al., 2024).

2. Training-time acceleration

The earliest fast-DiT work in this collection is MaskDiT, introduced in "Fast Training of Diffusion Models with Masked Transformers" (Zheng et al., 2023). Its main idea is to exploit patch redundancy by randomly masking out a high proportion of patches in diffused input images during training, typically 50%, so that the expensive encoder processes only visible tokens. The method uses an asymmetric encoder-decoder transformer, applies the denoising score matching objective only on unmasked tokens, adds an auxiliary reconstruction loss on masked noisy patches, and reports competitive or better ImageNet-256x256 and ImageNet-512x512 performance using only around 30% of DiT’s original training time. The paper is explicit that the gain is primarily in training: “At inference, all the patches are processed for sampling” (Zheng et al., 2023).

"FasterDiT: Towards Faster Diffusion Transformers Training without Architecture Modification" (Yao et al., 2024) takes a different route. Rather than modifying the backbone, it combines flow matching with $v$-prediction, target-standard-deviation modulation, logit-normal timestep sampling $\mathrm{lognorm}(0,1)$, and a velocity direction loss. The paper interprets training through the probability density function of a data-dependent SNR and argues that faster convergence comes from shifting and concentrating supervision into a better SNR region while strengthening single-step supervision. On ImageNet 256×256, FasterDiT reaches FID 2.30 after 1,000k training iterations, compared with DiT’s 2.27 at 7,000k iterations, which it summarizes as roughly 7× faster convergence in training iterations (Yao et al., 2024).

Three-dimensional generation sharpened the training-efficiency question because voxel token counts grow cubically. "Fast Training of Diffusion Transformer with Extreme Masking for 3D Point Clouds Generation" (Mo et al., 2023) therefore extends masked diffusion to voxelized point clouds. It introduces foreground/background-aware masking, runs the encoder only on visible tokens, and reports state-of-the-art 128-resolution voxel point cloud generation using only 6.5% of the original training cost. The key quantitative intuition is that ShapeNet voxel occupancy is extremely sparse, with average occupied voxels at 2.34% and non-occupied voxels at 97.66%, so extreme masking ratios near 99% can be effective if the masking policy is voxel-aware rather than random (Mo et al., 2023).

A broader implication is that fast-DiT training research is less about one universal recipe than about matching the acceleration mechanism to the structure of the data: patch redundancy for image latents, SNR imbalance for flow-based image training, and occupancy sparsity for 3D voxels (Zheng et al., 2023, Yao et al., 2024, Mo et al., 2023).

3. Inference acceleration through caching and selective execution

Inference-side fast-DiT methods usually begin from the observation that adjacent denoising steps are highly similar. FORA operationalizes this with a static periodic cache. At timestep $t$, it refreshes attention and MLP outputs only when $t \bmod N = 0$; otherwise it reuses cached submodule outputs. On ImageNet 256×256 with class-conditional DiT-XL/2-G and 250 DDIM steps, the paper reports a preferred operating point at $N=3$, giving $2.80\times$ speed-up with FID 2.82 versus a reproduced baseline FID 2.30, and shows that quality degrades sharply for larger cache intervals such as $N=10$ (Selvaraju et al., 2024).

A more corrective version of cache reuse appears in "Accelerating Diffusion Transformer via Increment-Calibrated Caching with Channel-Aware Singular Value Decomposition" (Chen et al., 9 May 2025). ICC does not reuse stale features directly; instead it writes the current layer output as

$$y_{m,l} = \mathcal{C}_l(y) + W(x_{m,l} - \mathcal{C}_l(x)),$$

and approximates the increment with a low-rank factorization derived from the pretrained weights. Channel-aware SVD further weights input and output channels to improve robustness to outlier activations. The paper states that, when compared with 35-step DDIM, ICC eliminates more than 45% computation and improves IS by 12 at the cost of less than 0.06 FID increase (Chen et al., 9 May 2025).

$\Delta$-DiT adds a second insight: different DiT depths matter differently at different denoising stages. Its analysis associates front blocks with outlines and rear blocks with details, and its training-free framework therefore accelerates rear blocks in early sampling stages and front blocks in later stages. On PIXART-$v$0, it reports a $v$1 speedup on 20-step generation and, notably, stronger robustness than naive caching in a 4-step consistency-model regime (Chen et al., 2024).

Other work shifts from explicit caching to dynamic execution. DyDiT++ introduces Timestep-wise Dynamic Width (TDW), which activates only a subset of attention heads and MLP channel groups as a function of timestep, and Spatial-wise Dynamic Token (SDT), which bypasses unnecessary token computation in MLP blocks. On ImageNet 256×256, the paper highlights 51% FLOPs reduction, 1.73× real speedup, and FID 2.07 on ImageNet 256×256 (Zhao et al., 9 Apr 2025). FastCache moves the control variable to hidden states themselves: it combines tokenwise saliency

$v$2

with a blockwise statistical reuse criterion

$v$3

and replaces skipped blocks with learned affine surrogates. Across DiT-S/2 through DiT-XL/2, it reports the best FID and t-FID among the compared cache methods, while remaining competitive in runtime and memory (Liu et al., 26 May 2025).

4. Fast-path execution, quantization, and deployment efficiency

A distinct branch of fast-DiT work emphasizes deployment rather than sampling algorithmics. SAFE-DiT is the clearest systems example. It identifies Mask-Induced Dispatch Tax (MIDT), the slowdown caused when semantically redundant self-attention masks push PyTorch SDPA away from FlashAttention fast paths, and proves that row-constant additive masks are exactly removable because

$v$4

SAFE-Core combines exact mask elision with prompt-conditioned token partitioning, selective state updates, and periodic dense refreshes, but keeps semantics-bearing masks such as text-padding masks. On Lumina-Next, SAFE-DiT achieves $v$5 end-to-end acceleration at $v$6 resolution and $v$7 at $v$8, reduces peak memory at $v$9 from 94.1 to 27.9 GB, and enables $\mathrm{lognorm}(0,1)$0 generation when dense inference runs out of memory (Yin et al., 28 Jun 2026).

Quantization addresses the same deployment problem from a different angle. "Post-Training Quantization for Diffusion Transformer via Hierarchical Timestep Grouping" (Ding et al., 10 Mar 2025) attributes DiT quantization difficulty to time-dependent, channel-specific activation outliers. It introduces timestep-aware shift-and-scale smoothing, groups denoising timesteps by hierarchical clustering, and absorbs quantization parameters into nearby modules such as AdaLN to avoid extra runtime arithmetic. On ImageNet 256×256 at 100 steps, HTG reports W8A8 FID 5.14 versus full-precision 5.02, and W4A8 FID 6.73, while reducing checkpoint size from about 2575 MB to about 652 MB for W8A8 and about 330 MB for W4A8 including HTG overhead (Ding et al., 10 Mar 2025).

Mobile and large-scale video systems push deployment efficiency further by attacking token count, parameter count, and denoising steps simultaneously. "Taming Diffusion Transformer for Real-Time Mobile Video Generation" (Wu et al., 17 Jul 2025) uses an $\mathrm{lognorm}(0,1)$1 video VAE, KD-guided tri-level pruning, and adversarial step distillation to 4 denoising steps without classifier-free guidance. Its final mobile model has 915M parameters and reaches 12.2 FPS on iPhone 16 Pro Max for 49-frame $\mathrm{lognorm}(0,1)$2 generation. FSVideo pursues a different compromise: a highly compressed $\mathrm{lognorm}(0,1)$3 latent space, a layer-memory DiT, and a few-step latent upsampler. With a 14B base model and a 14B DiT upsampler, it reports competitive performance while being an order of magnitude faster, including a $\mathrm{lognorm}(0,1)$4 speedup over Wan2.1-I2V-14B-720P on two H100 GPUs for 5-second $\mathrm{lognorm}(0,1)$5 image-to-video generation (Team et al., 2 Feb 2026).

A final deployment-oriented response is to replace the Transformer backbone entirely. DiCo is not a DiT acceleration patch but a fully convolutional substitute motivated by the claim that pre-trained DiT attention often predominantly captures local patterns. On ImageNet, DiCo-XL reports FID 2.05 at 256×256 and 2.53 at 512×512, with a 2.7× and 3.1× speedup over DiT-XL/2, respectively (Ai et al., 16 May 2025).

5. Domain-specific reinterpretations of the fast-DiT idea

The fast-DiT agenda has expanded well beyond class-conditional image synthesis. In 3D point generation, DiT-3D made plain transformer denoising feasible with 3D window attention and transfer from 2D DiT checkpoints; its ablation reports training cost dropping from 91.85 hours to 41.67 hours when 3D window attention is used (Mo et al., 2023). FastDiT-3D then pushed this logic to extreme masking, reaching state-of-the-art high-fidelity and diverse 3D point cloud generation performance at 128 resolution while using only 6.5% of the original training cost (Mo et al., 2023).

In speech synthesis, DPI-TTS argues that Mel spectrograms should not be treated as generic images. It replaces broad 2D interactions with Directional Patch Interaction, uses low-to-high frequency and previous-frame structure, and reports training speed increased by nearly 2 times without compromising accuracy (Qi et al., 2024). In genomics, "Continuous Diffusion Transformers for Designing Synthetic Regulatory Elements" (Liu et al., 11 Mar 2026) combines a shallow DiT with a locality-preserving 2D CNN input encoder. The paper states that the model matches the U-Net’s best validation loss in 13 epochs, 60× fewer, converges 39% lower, and reduces memorization from 5.3% to 1.7% of generated sequences aligning to training data via BLAT (Liu et al., 11 Mar 2026).

Video editing has produced yet another interpretation. EraserDiT is framed as a practical fast video inpainting DiT: it uses LTX-Video, a 3D VAE with $\mathrm{lognorm}(0,1)$6 compression, 40 denoising steps, and a Circular Position-Shift strategy for long-range consistency. The paper reports 180 seconds on one NVIDIA A100 GPU to complete a 1080 × 1920 video with 121 frames without any acceleration method, a result presented as making high-resolution DiT-based video object removal operationally feasible (Liu et al., 15 Jun 2025).

These cases suggest that fast-DiT research is increasingly domain-conditional. In some areas the dominant lever is structured sparsity; in others it is better tokenization, temporal scheduling, or a locality-preserving front end (Mo et al., 2023, Qi et al., 2024, Liu et al., 11 Mar 2026).

6. Limitations, trade-offs, and common misconceptions

A persistent misconception is to treat all fast-DiT work as faster sampling. Some influential methods are explicitly not inference accelerators. MaskDiT reduces training FLOPs, memory, and wall-clock convergence time, but “At inference, all the patches are processed for sampling” (Zheng et al., 2023). FasterDiT likewise accelerates convergence without changing the DiT architecture, so its main gain is fewer training iterations to a target FID rather than a cheaper sampling graph (Yao et al., 2024).

A second misconception is that all cache-based methods are equally robust. FORA uses a fixed cache interval and acknowledges that quality degrades significantly for aggressive caching, especially beyond $\mathrm{lognorm}(0,1)$7 to $\mathrm{lognorm}(0,1)$8, while FastCache’s theory relies on weak stationarity and Gaussian hidden-state assumptions and leaves the training or calibration of its learnable linear surrogates underspecified in the main text (Selvaraju et al., 2024, Liu et al., 26 May 2025). SAFE-DiT is more restrictive but also more explicit: its exactness guarantee applies only to provenance-certified removable masks, while its spatial scheduling remains approximate; the paper therefore separates SAFE-Core from SAFE-DiT+SW and treats the latter as a distinct prompt-alignment operating point rather than part of the exact acceleration claim (Yin et al., 28 Jun 2026).

A third misconception is to conflate neighboring research themes. DiCo is best understood as a practical replacement backbone for DiT, not a DiT-internal speed hack (Ai et al., 16 May 2025). DiTF, despite being highly relevant to DiT internals, is not a systems acceleration method; it is a training-free feature-extraction framework for visual correspondence that localizes and normalizes massive activations and then discards weakly massive channels (Gan et al., 24 May 2025). This suggests that fast-DiT research increasingly overlaps with representation analysis, kernel dispatch, quantization, and architecture search, but these remain distinct problem formulations.

The broad historical pattern is therefore not a march toward one dominant acceleration technique. Instead, the literature supports a layered view: masked objectives accelerate training, dynamic width and token routing reduce unnecessary execution, calibrated caching exploits temporal smoothness, mask-elision restores systems fast paths, and compression or quantization turns DiTs into deployable models. Under that view, fast-DiT is less a single model class than an evolving optimization program for keeping DiT-scale generative quality compatible with practical compute budgets (Zheng et al., 2023, Zhao et al., 9 Apr 2025, Yin et al., 28 Jun 2026, Ding et al., 10 Mar 2025).