---
title: Per-Timestep Conditioned Diffusion Transformer
url: https://www.emergentmind.com/topics/per-timestep-conditioned-diffusion-transformer
type: topic
---

# Per-Timestep Conditioned Diffusion Transformer

Searching arXiv for the cited papers and closely related work on timestep-conditioned diffusion transformers.
Per-timestep conditioned diffusion transformers are diffusion-based Transformer denoisers in which the diffusion timestep does not merely index the reverse process, but actively modulates the model’s internal computation, conditioning pathway, or even the forward trajectory itself. In the standard formulation, a diffusion Transformer receives a noisy sample $x_t$ and a timestep representation, typically through Adaptive LayerNorm or FiLM-style modulation; recent work expands this design space so that timestep information can determine active attention heads and MLP channel groups, token selection, spectral residual correction, feature aggregation weights, condition-dependent trajectory shifts, or segment-wise noise levels in multimodal data [2504.06803], [2302.02373], [2405.13762].

## 1. Formal basis in diffusion Transformers

A conventional diffusion Transformer augments the noisy input with an explicit time embedding. One formulation computes
\[
\tau = \mathrm{pe}(t)\in\mathbb{R}^d,\qquad e_t=\mathrm{ffn}(\tau)\in\mathbb{R}^{h},
\]
and injects it at each Transformer block through an Adaptive LayerNorm or FiLM-style modulation,
\[
\mathrm{AdaIN}(h;e_t)=\gamma(e_t)\,\frac{h-\mu(h)}{\sigma(h)}+\beta(e_t),
\]
so that the denoiser can be written as
\[
\epsilon_\theta(x_t,t)=\mathrm{DiT}\bigl(x_t;\{\gamma(e_t),\beta(e_t)\}\bigr)=f_\theta(x_t,t).
\]
This is the baseline temporal-conditioning interface in DiT-like models [2606.20416].

Within that baseline, the timestep embedding $E_t$ is already shared across the network. Dynamic Diffusion Transformer (DyDiT) makes this explicit by reusing the same timestep embedding employed by adaLN to drive additional per-block routing decisions. Each block inserts two lightweight routers that map $E_t\in\mathbb{R}^{C}$ to head-wise and channel-group-wise gating scores, thereby turning the timestep from a purely modulatory signal into a computation-allocation signal [2410.03456].

A central conceptual distinction follows. In standard DiT, the timestep conditions feature normalization while the computational graph remains static. In per-timestep conditioned variants, the timestep may instead determine width, token participation, spectral emphasis, condition aggregation, or noise allocation. This suggests that “timestep conditioning” is not a single mechanism but a family of architectural interventions that share the property that denoising behavior changes systematically with diffusion time.

## 2. Architectural patterns of per-timestep conditioning

Several recurrent patterns have emerged for making diffusion Transformers timestep-adaptive.

| Mechanism | Timestep-conditioned quantity | Representative paper |
|---|---|---|
| Timestep-wise Dynamic Width (TDW) | Active MHSA heads and MLP channel groups | [2504.06803] |
| Spatial-wise Dynamic Token (SDT) | Tokens routed through the expensive MLP | [2504.06803] |
| Image-Timestep Adaptive Feature Aggregator (ITAFA) | Layer-wise weights over image-encoder features | [2503.20418] |
| Spectral residual correction | Low- and high-frequency residual amplification coefficients | [2606.18765] |
| Mixture of Noise Levels (MoNL) | Per-modality, per-time-segment diffusion indices | [2405.13762] |
| Shifted conditional trajectories | Per-timestep forward-process shift $s_t$ | [2302.02373] |

In DyDiT and DyDiT++, TDW partitions each Transformer layer’s Multi-Head Self-Attention into $H$ heads of dimension $C_H$ and its MLP into $H$ parallel channel groups of size $D_H$. Two router MLPs, $R_{\text{head}}$ and $R_{\text{chan}}$, read the timestep embedding and output continuous scores
\[
S_{\text{head}}=R_{\text{head}}(E_t)\in[0,1]^H,\qquad S_{\text{chan}}=R_{\text{chan}}(E_t)\in[0,1]^H,
\]
which are binarized into masks selecting active heads and channel groups. MHSA and MLP computation then sum only over the activated subsets. Because the masks depend only on $t$, the active indices can be pre-computed for each timestep [2504.06803].

DyDiT further couples TDW with Spatial-wise Dynamic Token. A lightweight token router $R_{\text{tok}}$ maps token features $X\in\mathbb{R}^{N\times C}$ to scores $S_{\text{tok}}\in[0,1]^N$, which are binarized so that only selected tokens are processed by the expensive MLP, while the rest bypass it. The result is a joint timestep-and-space adaptive computation policy rather than a purely temporal one [2504.06803].

ITA-MDT implements a different form of timestep conditioning. Its Image-Timestep Adaptive Feature Aggregator takes hidden-layer outputs from an image encoder,
\[
\{f^{(l)}\in\mathbb{R}^{s\times d}\}_{l=1}^{H},
\]
projects the diffusion-timestep embedding and a three-component image-complexity descriptor into a shared space, combines them through a learned balance parameter $\alpha$, and produces softmax-normalized layer weights $W$. The aggregated feature is
\[
F=\sum_{l=1}^{H}W_l f^{(l)}.
\]
At each timestep, ITAFA recomputes $W^T$ and hence $W$, shifting emphasis among encoder layers as denoising progresses [2503.20418].

SpectralDiT localizes timestep conditioning even more narrowly. It leaves the standard flow-matching DiT structure intact and intervenes only in the MLP residual branch. The MLP residual is reshaped onto the patch-token grid, decomposed into low- and high-frequency components by a fixed depthwise low-pass filter, and corrected by timestep-conditioned coefficients $(\alpha^{\ell}_{\mathrm{low}},\alpha^{\ell}_{\mathrm{high}})$ predicted by a zero-initialized two-layer MLP that consumes only the timestep embedding $e_t$. The corrected residual is
\[
R'_{\mathrm{mlp}}
=(1+\alpha_{\mathrm{low}})R_{\mathrm{low}}+(1+\alpha_{\mathrm{high}})R_{\mathrm{high}}.
\]
This makes spectral emphasis an explicit function of diffusion time [2606.18765].

## 3. Conditioning the diffusion process rather than only the denoiser

A broader interpretation of per-timestep conditioning alters the diffusion trajectory itself. ShiftDDPMs introduce conditions into the forward process by allocating an exclusive diffusion trajectory for each condition $c$. Given a shift predictor $E(c)$ and a schedule $k_t$, the per-timestep shift is
\[
s_t=k_t\cdot E(c),
\]
and the conditional marginal becomes
\[
q(x_t\mid x_0,c)=\mathrm{Normal}\bigl(\sqrt{\bar{\alpha}_t}\,x_0+s_t,\,(1-\bar{\alpha}_t)I\bigr).
\]
The one-step kernel accordingly changes to
\[
q(x_t\mid x_{t-1},c)=\mathrm{Normal}\bigl(\sqrt{\alpha_t}\,x_{t-1}+(s_t-\sqrt{\alpha_t}s_{t-1}),\,\beta_t I\bigr).
\]
In the accompanying Transformer design, both the timestep embedding $\tau_t$ and the shift $s_t$ are injected into attention and MLP sublayers through FiLM [2302.02373].

A different generalization appears in audiovisual latent diffusion with Mixture of Noise Levels. Instead of a single scalar timestep, the forward process uses a matrix
\[
t\in\{1\ldots T\}^{M\times N},
\]
with one diffusion index for each modality $m$ and time-segment $n$. Each segment is noised at its own step, and the denoiser receives the entire noise-level pattern. Four sampling schemes are defined—Vanilla, Per-modality, Per-time, and Per-time-per-modality—and training draws from this mixture so that one model can cover joint generation, cross-modal generation, and multimodal interpolation tasks [2405.13762].

ITA-MDT occupies an intermediate position. Its forward diffusion remains standard,
\[
q(z_t\mid z_{t-1})=\mathcal{N}\bigl(\sqrt{1-\beta_t}\,z_{t-1},\,\beta_tI\bigr),
\]
but the conditioning feature $c(t)$ changes at every timestep because global garment features $F_g(t)$ and salient-region features $F_s(t)$ are recomputed through ITAFA and then combined with the time embedding before cross-attention. The timestep therefore controls not only denoising phase but also which conditional evidence is emphasized [2503.20418].

These formulations show that per-timestep conditioning need not be confined to a scalar embedding entering the reverse model. It can instead define the geometry of the forward trajectory, the granularity of the noise schedule, or the composition of the conditioning context itself.

## 4. Training objectives and parameter-efficient adaptation

The dominant training objective remains the standard noise-prediction mean-squared error. In DDPM form,
\[
\mathcal{L}(\theta)=\mathbb{E}_{x_0,\epsilon,t}\Bigl\lVert \epsilon-\epsilon_\theta(x_t,t)\Bigr\rVert^2,
\]
and several of the timestep-conditioned variants preserve this basic criterion even while changing what the network does with $t$ [2606.20416].

DyDiT++ augments the base diffusion loss with a FLOPs-aware penalty. For a batch of timesteps $\{t_b\}$, it computes dynamic FLOPs relative to the static DiT budget and imposes
\[
L_{\mathrm{FLOPs}}
=
\left(
\frac{1}{B}\sum_b \frac{F^{t_b}_{\mathrm{dynamic}}}{F_{\mathrm{static}}}
-\lambda
\right)^2,
\]
with overall loss
\[
L=L_{\mathrm{DiT}}+w\cdot L_{\mathrm{FLOPs}},
\]
typically with $w=1.0$. Optimization uses Gumbel-Sigmoid plus Straight-Through for discrete mask gradients, warm-up “complete” DiT supervision for first few steps, and a safeguard that at least one head and one group remain always active by magnitude-based ranking [2504.06803].

ITA-MDT is trained with three objectives: denoising, mask reconstruction, and inpainting. The denoiser $\epsilon_\theta(L,c,t)$ follows MDT-v2, while a random binary mask in latent token space supports side-interpolator reconstruction and a garment-region-focused inpainting objective emphasizes the encoded garment mask. The reported total loss is
\[
L_{\mathrm{total}}=L_{\mathrm{denoise}}+L_{\mathrm{mask}}+L_{\mathrm{inpaint}}.
\]
Within this setup, timestep-adaptive conditioning is embedded in ITAFA rather than in a specialized temporal auxiliary loss [2503.20418].

SpectralDiT is trained with a continuous-time linear flow-matching objective,
\[
\mathcal{L}
=\mathbb{E}_{t,x_0,z_t}\Bigl\|v_\theta(z_t,t)-\tfrac{d}{dt}z_t\Bigr\|^2.
\]
Its gate MLPs are zero-initialized, so early training matches the baseline DiT exactly. This design makes the timestep-conditioned spectral correction a residual refinement rather than a disruptive reparameterization [2606.18765].

DyDiT++ also introduces timestep-based dynamic LoRA (TD-LoRA) for parameter-efficient fine-tuning. Standard LoRA adapts a frozen weight $W$ through low-rank matrices $A$ and $B$:
\[
X(W+AB)=XW+(XA)B.
\]
TD-LoRA replaces $B$ by an MoE of experts $\{B_i\}$ and computes timestep-dependent expert weights
\[
\mu=\mathrm{Softmax}(R_{\mathrm{expert}}(E_t))\in\Delta^M,
\]
so that the effective LoRA injection depends on the timestep. The fine-tuning schedule fully tunes the few router MLPs, LoRA-adapts AdaLN layers using standard LoRA, and applies TD-LoRA only on core Transformer weights $Q,K,V,O,W_1,W_2$ [2504.06803].

## 5. Inference-time realization, complexity, and reported efficiency

Per-timestep conditioning becomes practically significant only when it produces real inference-time gains. DyDiT and DyDiT++ emphasize that head and channel masks depend only on $t$, not on the sample, so the active indices can be pre-computed for all $t=1\ldots T$. At inference, the system simply looks up the masks rather than performing per-sample graph surgery; this is identified as critical for real-world batched speedups [2504.06803], [2410.03456].

In DyDiT++, static DiT-XL per-step FLOPs for MHSA+MLP are reported as approximately $118\,\mathrm{G}$. With $\lambda=0.7$, DyDiT-XL uses approximately $84\,\mathrm{G}$, corresponding to an approximately $29\%$ reduction and a $1.32\times$ end-to-end speedup on V100. With $\lambda=0.5$, the budget is approximately $58\,\mathrm{G}$, corresponding to a $51\%$ FLOPs cut, a $1.73\times$ wall-clock speedup, and FID $2.07$ versus $2.27$ baseline. The per-timestep savings vary, with early steps using as few as $20$–$30\%$ of heads and channels, and SDT often skipping $40$–$60\%$ of tokens on easy regions [2504.06803].

The earlier DyDiT report gives closely aligned ImageNet $256\times256$ results for $250$ DDPM steps and guidance $1.5$: DiT-XL at $118.7\,\mathrm{G}$, $10.22\,\mathrm{s/image}$, FID $2.27$; DyDiT-XL $(\lambda=0.7)$ at $84.3\,\mathrm{G}$, $7.76\,\mathrm{s/image}$, FID $2.12$; and DyDiT-XL $(\lambda=0.5)$ at $57.9\,\mathrm{G}$, $5.91\,\mathrm{s/image}$, FID $2.07$. On $512\times512$ ImageNet, DyDiT-XL $(\lambda=0.7)$ gives FID $3.61$ versus $3.75$ at $375\,\mathrm{G}$ versus $515\,\mathrm{G}$ [2410.03456].

SpectralDiT illustrates a different efficiency regime. It adds only the gate MLPs, for a parameter overhead of $\delta P=74{,}896$, or approximately $1.36\%$ relative to the DiT-S/2 base model, and approximately $0.6\%$ more MACs in theory. The intent is not compute pruning but more effective timestep-conditioned residual correction with minimal overhead [2606.18765].

ITA-MDT frames efficiency differently again. It uses a lightweight, scalable transformer-based denoising diffusion model with a mask latent modeling scheme, and the Salient Region Extractor provides high-resolution local information as an additional condition while avoiding unnecessarily processing the entire garment image. Quantitatively, ITAFA cuts FID by approximately $2$ points while keeping model size approximately $671\,\mathrm{M}$ parameters and inference time approximately $4.6\,\mathrm{s}$ per image [2503.20418].

## 6. Empirical scope, applications, and points of contention

The empirical scope of per-timestep conditioned diffusion Transformers now spans class-to-image generation, flow-matching, video generation, text-to-image generation, image-based virtual try-on, and audiovisual generation. DyDiT++ reports effectiveness on diverse visual generation models including DiT, SiT, Latte, and FLUX, and states that video generation and text-to-image generation show similar trade-offs and $1.6\times$–$2.5\times$ speedups with minimal quality loss [2504.06803].

On ImageNet $256\times256$, DyDiT++ reports a FLOPs–FID trade-off in which DyDiT-XL $(\lambda=0.7)$ at $84\,\mathrm{G}$ reaches FID $2.12$ and DyDiT-XL $(\lambda=0.5)$ at $58\,\mathrm{G}$ reaches FID $2.07$, compared with static DiT-XL at $118\,\mathrm{G}$ and FID $2.27$. For flow matching, DySiT-XL $(\lambda=0.7)$ is reported at $85\,\mathrm{G}$ and FID $1.95$ versus $2.11$ static. In parameter-efficient fine-tuning, DyDiT-XL\_PEFT uses only approximately $10\,\mathrm{M}$ tunable parameters, approximately $1.5\%$ total, yet reaches FID $2.23$ versus $2.07$ full fine-tune, while standard LoRA yields FID $2.41$ [2504.06803].

ITA-MDT situates timestep conditioning in a task-specific setting. On VITON-HD, the MDT-IVTON baseline without ITAFA reports FID(pair/unp.) approximately $7.77/9.84$; adding ITAFA yields $5.72/8.88$; adding ITAFA and SRE yields $5.46/8.68$. The paper also reports that ITAFA reduces LPIPS from $0.099$ to $0.086$, improves SSIM marginally, and that Figure 7(a) shows early timesteps weighting low-level layers while late timesteps weight high-level layers [2503.20418].

In audiovisual generation, Mixture of Noise Levels enables a single transformer-based latent diffusion model to cover multiple conditional tasks. On the Monologues dataset, trained for $400\,\mathrm{K}$ steps and sampled with $250$ DDIM steps, MoNL reports joint generation FAD $6.4$ versus task-specific $7.1$ and FVD $77.6$ versus $63.6$; Audio$\rightarrow$Video FVD $40.2$ versus vanilla $53.3$; Video$\rightarrow$Audio FAD $5.3$ versus $8.1/11.5$; AV-inpaint FAD $4.6$, FVD $11.8$; and AV-continue FAD $3.1$, FVD $8.8$. Averaged across five tasks, MoNL reports FAD $4.9$, FVD $34.6$ versus conditional $(7.8,35.3)$, per-modality $(5.6,106.1)$, and vanilla $(7.4,121.1)$ [2405.13762].

A major point of contention concerns the necessity of explicit timestep embeddings. “On the Redundancy of Timestep Embeddings in Diffusion Models” argues that, under stated assumptions, the global minimizer of the standard diffusion loss can be achieved without explicit timestep conditioning, because the network can implicitly infer noise scales from the corrupted input. Empirically, on CelebA $128\times128$, removing $t$ hurts DiT slightly—FID $70.04$ for DiT versus $70.48$ for DiT*—but improves U-Net FID from $69.82$ to $63.63$. On CIFAR-10 $32\times32$, class-conditional DiT* reports FID $34.74$ versus $51.32$ for DiT, with higher precision and recall, and U-Net* reports FID $41.18$ versus $52.67$ [2606.20416].

That result does not invalidate per-timestep conditioned diffusion Transformers, but it narrows the claim that explicit temporal embeddings are universally required. A plausible implication is that there are two separable questions: whether the denoiser must be told the timestep explicitly, and whether a model benefits from timestep-dependent computation or conditioning policies. DyDiT, ITAFA, SpectralDiT, MoNL, and ShiftDDPMs address the second question in distinct ways, while the time-agnostic results primarily challenge the first [2606.20416].

Source: https://www.emergentmind.com/topics/per-timestep-conditioned-diffusion-transformer