Timestep-Dependent Expert Capacity in Neural Models

• Timestep-dependent expert capacity is a principle that modulates expert resource allocation based on temporal indices in models like MoEs, diffusion language models, and LoRA systems.
• It adapts key variables such as token budgets, active parameters, and routing strategies to respond to changing temporal regimes during training and inference.
• Empirical results demonstrate that dynamic scheduling methods, including linear-reverse and specialized expert activations, improve model performance and computational efficiency.

Searching arXiv for the cited papers to ground the article and verify metadata. arxiv_search(query="(Cong et al., 2024)", max_results=5) Timestep-dependent expert capacity denotes a family of mechanisms in which the effective capacity assigned to experts varies with a discrete temporal index $t$. The index may be the training iteration of a sparse Mixture-of-Experts (MoE) model, the denoising step of a diffusion or diffusion-LLM, or the current time step in online forecasting. Correspondingly, “capacity” may mean the token budget of an MoE expert, the number of token–expert pairs executed at a denoising step, the rank or contribution of a timestep-specialized LoRA expert, the fraction of layers activated for a timestep interval, or the amount of quantization precision preserved for a specific expert at a specific stage. Across these formulations, the unifying premise is that temporal regimes are not homogeneous: routing statistics, noise structure, learning efficiency, and sensitivity to approximation all change over time, so uniform expert allocation is often a mismatch to the underlying dynamics (Cong et al., 2024).

1. Conceptual scope and formal meanings

The literature does not use a single canonical definition of timestep-dependent expert capacity. Instead, closely related papers instantiate the idea through different operational variables.

In sparse MoE training, expert load at iteration $t$ is the number or proportion of tokens processed by expert $i$. If the total number of tokens entering an MoE layer is $N_{\text{token}}$ and expert $i$ processes $n_{i,t}$ tokens, then its load proportion is

$L_{i,t}=\frac{n_{i,t}}{N_{\text{token}}}.$

Expert capacity is constrained by a capacity factor $CF$ through

$$\mathrm{capacity}_i = CF\cdot \left(\frac{N_{\text{token}}}{E}\right),$$

with overflow tokens spilled via the residual connection (Cong et al., 2024).

In diffusion LLMs with expert-choice routing, capacity is an explicit timestep function

$$C(t)=k(\gamma(t))=\mathrm{clamp}\Bigl(k_{\min}+(k_{\max}-k_{\min})\,s(\gamma(t)),k_{\min},k_{\max}\Bigr),$$

where $t$0 is the masking ratio. This directly sets the per-step MoE computation to $t$1 token–expert pairs while preserving deterministic load balance (Zhang et al., 2 Apr 2026).

In timestep-specialized LoRA systems for diffusion models, capacity is tied to the low-rank update $t$2 attached to a timestep interval. The rank $t$3 controls expressivity because storing $t$4 costs $t$5 parameters per weight matrix, and larger $t$6 gives higher capacity (Zhuang et al., 10 Mar 2025). Related systems keep the rank fixed but vary expert influence by activating different LoRAs on different timestep segments or by assigning timestep- and layer-dependent mixture weights to multiple LoRA experts (Liang et al., 11 Jun 2025).

In layer-partitioned flow models, capacity is the active parameter budget per denoising step. LaTtE-Flow partitions a 28-layer Transformer into $t$7 groups of $t$8 layers and invokes exactly one group per step, so only $t$9 of the parameters are active at a given timestep; concretely, the full model has approximately $i$0B parameters, while the per-step active parameters are approximately $i$1B (Shen et al., 8 Jun 2025).

A compact way to organize these meanings is the following.

Setting	Capacity variable	Representative formulation
Sparse MoE training	Per-expert token budget	$i$2 or predicted load $i$3 (Cong et al., 2024)
Diffusion language MoE	Per-step expert quota	$i$4 under EC routing (Zhang et al., 2 Apr 2026)
Diffusion LoRA experts	Rank or gated update strength	$i$5 from timestep experts (Zhuang et al., 10 Mar 2025)
Segmented LoRA alignment	Active specialist by interval	Motion/fidelity expert switching (Liang et al., 11 Jun 2025)
Layerwise flow experts	Active layers per timestep	One group $i$6 per step (Shen et al., 8 Jun 2025)
Quantized MoE DiT	Preserved precision/calibration budget	Expert- and timestep-wise clipping and low-rank branch (Wu et al., 26 May 2026)

This suggests that the phrase is best understood as a structural principle rather than a single algorithm: expert capacity is treated as a function of temporal context, rather than as a fixed architectural constant.

2. Iteration-indexed capacity in sparse MoE training

The most direct formulation appears in “Prediction Is All MoE Needs: Expert Load Distribution Goes from Fluctuating to Stabilizing” (Cong et al., 2024). That work studies the evolution of expert loads over training iterations in GPT-3 125M and 350M MoE models and distinguishes two temporal regimes: a transient state with “obvious load fluctuation” and a stable state with “temporal locality.”

The transition is quantified by sliding-window statistics on the load-proportion time series. For a window of size $i$7,

$i$8

and

$i$9

The model is declared transient at time $N_{\text{token}}$0 if there exists an expert with variance or range above threshold, and stable if all experts stay below threshold. In the reported GPT-3 125M and 350M experiments, after roughly 5,000 iterations both variance and range fall below small empirical thresholds such as $N_{\text{token}}$1 and $N_{\text{token}}$2, marking the onset of temporal locality.

This temporal split has an immediate capacity interpretation. In the transient phase, expert loads are described as swinging unpredictably, so the recommended policy is conservative over-provisioning: choose a larger $N_{\text{token}}$3 so that $N_{\text{token}}$4 covers the worst early load. In the stable phase, load proportions become predictable enough to support dynamic per-expert capacity tuning, with

$N_{\text{token}}$5

plus a safety margin.

The paper evaluates three forecasting methods for future load proportions: Sliding-Window Average (SW_Avg), ARIMA$N_{\text{token}}$6, and an LSTM-based sequence model trained with mean absolute percentage error. On the GPT-3 350M setup—24 layers total, 12 MoE layers, 128 experts per MoE layer, global batch 256, and 4$N_{\text{token}}$7A800 GPUs—the stable-state results show that SW_Avg predicts the next 1,000 steps with average error approximately $N_{\text{token}}$8, and the next 2,000 steps with average error approximately $N_{\text{token}}$9; ARIMA$i$0 reaches approximately $i$1 error for 1,000-step prediction, while the LSTM has higher and less stable errors, reported as around $i$2 for each 1,000-iteration block early on before improving later (Cong et al., 2024).

The importance of this result is not merely predictive accuracy. It reframes MoE capacity from a static safety parameter into a time-indexed control variable. In this view, the onset of temporal locality is the point at which expert provisioning can switch from robustness-oriented over-allocation to prediction-driven allocation, with implications for expert placement, GPU-to-expert mapping, memory allocation, and parallel utilization.

3. Explicit timestep scheduling in diffusion language MoEs

“Expert-Choice Routing Enables Adaptive Computation in Diffusion LLMs” makes timestep-dependent capacity an explicit architectural primitive (Zhang et al., 2 Apr 2026). The paper argues that token-choice routing, inherited from autoregressive systems, is a poor fit for diffusion LLMs, and replaces it with expert-choice routing. Because expert-choice routing has externally controllable capacity, the model can vary expert allocation across denoising steps.

The central object is the per-step capacity function $i$3 above. Since each diffusion step has a masking ratio $i$4, capacity can be scheduled as a normalized function of that ratio. The best-performing policy in the experiments is the “linear-reverse” schedule,

$i$5

Under this policy, low-mask-ratio steps receive more capacity than high-mask-ratio steps.

The rationale is empirical and mechanistic. The paper buckets validation loss by mask-ratio bin $i$6 and defines a per-bin convergence rate

$i$7

At 8B-A1B scale, low-mask-ratio bins such as $i$8 exhibit $i$9 up to $n_{i,t}$0 higher than high-mask-ratio bins such as $n_{i,t}$1. Because marginal return on extra compute scales with $n_{i,t}$2, allocating more expert capacity to low-mask-ratio steps yields larger loss reduction per FLOP. The paper accordingly interprets MoE computation in DLMs as an adaptive policy rather than a fixed constant.

The matched-FLOP results on OpenWebText make the point concrete. With $n_{i,t}$3, $n_{i,t}$4, and $n_{i,t}$5, static capacity $n_{i,t}$6 gives validation perplexity $n_{i,t}$7, whereas linear-reverse scheduling gives $n_{i,t}$8; cosine-reverse gives $n_{i,t}$9, Gaussian $L_{i,t}=\frac{n_{i,t}}{N_{\text{token}}}.$0, linear $L_{i,t}=\frac{n_{i,t}}{N_{\text{token}}}.$1, and cosine $L_{i,t}=\frac{n_{i,t}}{N_{\text{token}}}.$2. At 8B-A1B scale, dynamic expert-choice with linear-reverse scheduling and $L_{i,t}=\frac{n_{i,t}}{N_{\text{token}}}.$3, $L_{i,t}=\frac{n_{i,t}}{N_{\text{token}}}.$4, outperforms static expert-choice with $L_{i,t}=\frac{n_{i,t}}{N_{\text{token}}}.$5 in validation perplexity, 5-shot MMLU, and 25-shot ARC (Zhang et al., 2 Apr 2026).

A common misconception is that timestep-dependent capacity necessarily sacrifices load balancing. In this design, the opposite is asserted: expert-choice routing preserves deterministic load balance by construction, and the adaptive part is the externally scheduled capacity $L_{i,t}=\frac{n_{i,t}}{N_{\text{token}}}.$6, not an uncontrolled fluctuation of router behavior.

4. Timestep-specialized experts in diffusion adaptation and generation

A large diffusion literature implements timestep-dependent capacity through expert specialization over denoising intervals rather than through a single scalar quota.

“TimeStep Master” introduces TimeStep LoRA experts trained on different timestep intervals and assembled asymmetrically through a core-context mixture (Zhuang et al., 10 Mar 2025). The finest-interval expert acts as an ungated core expert, while coarser experts contribute through time-dependent gates

$L_{i,t}=\frac{n_{i,t}}{N_{\text{token}}}.$7

yielding

$L_{i,t}=\frac{n_{i,t}}{N_{\text{token}}}.$8

The paper explicitly ties expert capacity to interval difficulty: early high-noise steps can be served by lower-rank or coarser experts, middle steps benefit from medium capacity, and very late steps require higher-capacity experts for fine-detail recovery. On T2I-CompBench, SD1.5 with Vanilla LoRA $L_{i,t}=\frac{n_{i,t}}{N_{\text{token}}}.$9 gives Color $CF$0, Shape $CF$1, Spatial $CF$2; TSM 1-stage $CF$3 raises Color to $CF$4 and Spatial to $CF$5; TSM 2-stage reaches Color $CF$6, Shape $CF$7, Spatial $CF$8. In model distillation, DMD2 + Vanilla LoRA has FID $CF$9, whereas DMD2 + TSM reaches FID $$\mathrm{capacity}_i = CF\cdot \left(\frac{N_{\text{token}}}{E}\right),$$0 while adding less than $$\mathrm{capacity}_i = CF\cdot \left(\frac{N_{\text{token}}}{E}\right),$$1M extra parameters and only $$\mathrm{capacity}_i = CF\cdot \left(\frac{N_{\text{token}}}{E}\right),$$2 A100-day cost (Zhuang et al., 10 Mar 2025).

“AlignHuman” uses a stricter segmentation of denoising time into a motion segment and a fidelity segment (Liang et al., 11 Jun 2025). With total denoising steps $$\mathrm{capacity}_i = CF\cdot \left(\frac{N_{\text{token}}}{E}\right),$$3 and switch point $$\mathrm{capacity}_i = CF\cdot \left(\frac{N_{\text{token}}}{E}\right),$$4, the motion expert is active on

$$\mathrm{capacity}_i = CF\cdot \left(\frac{N_{\text{token}}}{E}\right),$$5

and the fidelity expert on

$$\mathrm{capacity}_i = CF\cdot \left(\frac{N_{\text{token}}}{E}\right),$$6

with $$\mathrm{capacity}_i = CF\cdot \left(\frac{N_{\text{token}}}{E}\right),$$7. Both experts are rank-$$\mathrm{capacity}_i = CF\cdot \left(\frac{N_{\text{token}}}{E}\right),$$8 LoRA adapters, but exactly one is active at a given timestep. The paper reports that removing Motion-LoRA drops HKV by about $$\mathrm{capacity}_i = CF\cdot \left(\frac{N_{\text{token}}}{E}\right),$$9 points, removing Fidelity-LoRA worsens FID by about $$C(t)=k(\gamma(t))=\mathrm{clamp}\Bigl(k_{\min}+(k_{\max}-k_{\min})\,s(\gamma(t)),k_{\min},k_{\max}\Bigr),$$0 and drops HKC by about $$C(t)=k(\gamma(t))=\mathrm{clamp}\Bigl(k_{\min}+(k_{\max}-k_{\min})\,s(\gamma(t)),k_{\min},k_{\max}\Bigr),$$1, and the complete TPO system improves FID from $$C(t)=k(\gamma(t))=\mathrm{clamp}\Bigl(k_{\min}+(k_{\max}-k_{\min})\,s(\gamma(t)),k_{\min},k_{\max}\Bigr),$$2 to $$C(t)=k(\gamma(t))=\mathrm{clamp}\Bigl(k_{\min}+(k_{\max}-k_{\min})\,s(\gamma(t)),k_{\min},k_{\max}\Bigr),$$3, FVD from $$C(t)=k(\gamma(t))=\mathrm{clamp}\Bigl(k_{\min}+(k_{\max}-k_{\min})\,s(\gamma(t)),k_{\min},k_{\max}\Bigr),$$4 to $$C(t)=k(\gamma(t))=\mathrm{clamp}\Bigl(k_{\min}+(k_{\max}-k_{\min})\,s(\gamma(t)),k_{\min},k_{\max}\Bigr),$$5, HKV from $$C(t)=k(\gamma(t))=\mathrm{clamp}\Bigl(k_{\min}+(k_{\max}-k_{\min})\,s(\gamma(t)),k_{\min},k_{\max}\Bigr),$$6 to $$C(t)=k(\gamma(t))=\mathrm{clamp}\Bigl(k_{\min}+(k_{\max}-k_{\min})\,s(\gamma(t)),k_{\min},k_{\max}\Bigr),$$7, and HKC from $$C(t)=k(\gamma(t))=\mathrm{clamp}\Bigl(k_{\min}+(k_{\max}-k_{\min})\,s(\gamma(t)),k_{\min},k_{\max}\Bigr),$$8 to $$C(t)=k(\gamma(t))=\mathrm{clamp}\Bigl(k_{\min}+(k_{\max}-k_{\min})\,s(\gamma(t)),k_{\min},k_{\max}\Bigr),$$9. It also supports inference with $t$00 NFEs instead of $t$01, a reported $t$02 speedup (Liang et al., 11 Jun 2025).

“FantasyTalking2” generalizes the same principle to three dimensions of preference: motion naturalness, lip-sync, and visual quality (Wang et al., 15 Aug 2025). The framework uses $t$03 rank-$t$04 LoRA experts and a layer-specific timestep gate

$t$05

with fused activation

$t$06

Empirically, early timesteps favor the motion expert, mid-range timesteps favor the lip-sync expert, and late timesteps favor the visual-quality expert. Removing timestep-wise gating reduces Sync-C from $t$07 to $t$08 and worsens FID from $t$09 to $t$10 (Wang et al., 15 Aug 2025).

“LaTtE-Flow” moves from expert adapters to expert layer groups (Shen et al., 8 Jun 2025). With $t$11 layers and $t$12 groups, the timestep interval is partitioned into $t$13, $t$14, $t$15, and $t$16, and routing is hard:

$t$17

Only one group executes at each sampling step. The paper reports $t$18 s/img and FID $t$19 for the non-expert “Vanilla Couple,” versus $t$20 s/img and FID $t$21 for LaTtE-Flow Couple, and describes the resulting speedup relative to recent unified models as approximately $t$22 (Shen et al., 8 Jun 2025).

A related but distinct line appears in “A-SelecT,” which does not define timestep-dependent expert capacity explicitly but demonstrates sharp timestep-wise variation in representational usefulness (Liu et al., 25 Mar 2026). It introduces the High-Frequency Ratio

$t$23

and selects the timestep $t$24 maximizing dataset-averaged $t$25. Exhaustive traversal over all timesteps is reported as approximately $t$26 hours on CUB, whereas A-SelecT takes approximately $t$27 hours, a reduction of about $t$28–$t$29, while preserving final accuracy (Liu et al., 25 Mar 2026). This does not implement expert capacity directly, but it supports the broader premise that denoising timesteps have highly nonuniform functional roles.

5. Capacity preservation under quantization and non-stationarity

Another branch of the literature studies timestep dependence not by redistributing compute, but by preserving expert capacity where approximation is most damaging.

Wu et al., in “Timestep-Aware SVDQuant-GPTQ for W4A4 Quantization of Wan2.2-I2V,” analyze a two-expert MoE video diffusion Transformer with distinct high-noise and low-noise experts (Wu et al., 26 May 2026). For each expert, layer, and timestep, they compute channelwise activation maxima

$t$30

The measured statistics show that early timesteps have large, heavy-tailed activations and late timesteps fall by $t$31–$t$32 orders of magnitude; the low-noise expert has systematically lower dynamic range but is more sensitive to clipping. The framework combines SVDQuant-based low-rank outlier compensation, GPTQ on the residual branch, and timestep-bin-wise per-layer activation clipping-ratio search for each expert. On OpenS2V-Eval, the BF16 baseline has peak memory $t$33 GB, VBench average $t$34, and Imaging Quality $t$35, whereas the full method reaches peak memory $t$36 GB, VBench average $t$37, and Imaging Quality $t$38, corresponding to a $t$39 memory reduction, a $t$40 drop in VBench average score, and a $t$41 drop in Imaging Quality (Wu et al., 26 May 2026). In this setting, timestep-aware calibration is presented as essential because a global policy cannot simultaneously handle early outliers and late fine-detail sensitivity.

Online forecasting provides a different temporal interpretation. “Dynamic Multi-period Experts for Online Time Series Forecasting” selects specialized experts at each time step by FFT-based identification of dominant periods and blends them with a stable general expert (Hong et al., 10 Mar 2026). The specialized experts are fitted on the fly by dual ridge regression, while an EWMA-based danger signal

$t$42

increases reliance on the general expert under emergent drift. An ablation on ETTh2 and ETTm1 shows MSE improving from approximately $t$43 at $t$44 to $t$45 at $t$46 on ETTh2 $t$47, then plateauing at approximately $t$48 for $t$49 (Hong et al., 10 Mar 2026). Here capacity is time-dependent because the set of relevant periodic specialists and their weights change at each forecasting step.

DynaMoE pushes this further by making the number of active experts token-dependent through percentile-threshold routing (Gülmez, 2 Mar 2026). For token representation $t$50, the selected set is

$t$51

The paper interprets this as timestep-dependent in the sense of token-dependent activation and combines it with layer-wise schedules such as descending, ascending, pyramid, and wave patterns. It reports, for example, $t$52 on CIFAR-10 for descending MoE versus $t$53 for the MLP baseline, and states that descending DynaMoE reaches $t$54 of its final MNIST accuracy in $t$55 epochs versus $t$56 for uniform MoE and the MLP (Gülmez, 2 Mar 2026).

A historical antecedent appears in “Adapting to Non-stationarity with Growing Expert Ensembles,” which introduces a new expert every $t$57 rounds so that

$t$58

and adapts the fixed-shares algorithm to a growing expert set (Shalizi et al., 2011). This is not framed in terms of denoising timesteps or MoE capacity factors, but it establishes a closely related principle: when the environment is temporally non-stationary, the effective expert ensemble itself may need to vary over time.

6. Recurring patterns, misconceptions, and open directions

Several recurring patterns are visible across these otherwise heterogeneous formulations.

First, timestep-dependent expert capacity is not synonymous with increasing the number of experts. The number of experts may remain fixed while the token budget varies by denoising step, as in expert-choice diffusion LLMs; the LoRA experts may be fixed while their contributions are reweighted by timestep gates, as in TimeStep Master and TLPO; or exactly one expert may be active per interval, as in AlignHuman and LaTtE-Flow (Zhang et al., 2 Apr 2026).

Second, “capacity” is not a single quantity. It can mean token throughput, active parameter count, LoRA rank, mixture weight, or quantization headroom. This explains why seemingly different methods fall under the same conceptual umbrella. A model may preserve capacity by allocating more token slots, by assigning a higher-rank expert to late denoising, by activating a larger fraction of layers, or by reserving higher-precision branches for timestep- and expert-specific outliers.

Third, a recurrent empirical pattern is temporal asymmetry. Several papers report that early stages favor coarse structure, robustness, or motion guidance, whereas later stages favor detail recovery, fidelity, or richer contextual exploitation. TimeStep Master associates early noisy steps with coarse stable representations and very late steps with higher-capacity detail recovery; AlignHuman assigns motion to early denoising and fidelity to late denoising; TLPO reports early motion dominance, mid lip-sync dominance, and late visual-quality dominance; Wu et al. distinguish high-noise and low-noise experts with different activation ranges and clipping sensitivities; and expert-choice DLMs report substantially higher learning efficiency at low mask ratios (Zhuang et al., 10 Mar 2025).

A further misconception is that timestep dependence must be learned by a complex controller. The literature includes hard interval routing, classical time-series predictors such as SW_Avg and ARIMA, small fully connected gates, and externally specified schedules. This suggests that the main challenge is often not controller complexity but identifying a temporal variable whose heterogeneity is stable enough to exploit.

Open directions are already explicit in the source papers. The MoE load-prediction work states that an expert placement scheme for transient and stable states will be proposed in subsequent work (Cong et al., 2024). The diffusion-language study proposes learning the scheduler $t$59 end-to-end, for example via a lightweight neural predictor or reinforcement-learning policy, and suggests extending adaptive capacity scheduling to other iterative generative paradigms (Zhang et al., 2 Apr 2026). A plausible implication is that future systems will combine several levels of temporal adaptation simultaneously: routing capacity, expert specialization, placement, calibration, and approximation strategy may all become functions of timestep.

Taken together, the literature presents timestep-dependent expert capacity as a general response to temporal heterogeneity in modular models. Whether expressed as predicted per-expert token capacity in sparse LLM training, reverse-scheduled expert quotas in diffusion LLMs, interval-specialized LoRA experts in diffusion, or expert- and timestep-wise quantization calibration, the underlying claim is consistent: when temporal regimes differ systematically, expert capacity is most effective when it is itself time-indexed rather than uniform.