Heterogeneous Step Allocation (HSA) Overview

• HSA is a nonuniform allocation framework that assigns different step budgets, update frequencies, or work fractions based on token dynamics or task demands.
• In diffusion transformers, HSA tailors denoising steps per token using dynamic selection and KV-cache synchronization to balance quality and runtime.
• HSA is also applied in heterogeneous computing and multi-agent planning, optimizing resource allocation and ensuring safety via explicit synchronization mechanisms.

Searching arXiv for papers relevant to “Heterogeneous Step Allocation (HSA)” and closely related constructs. Heterogeneous Step Allocation (HSA) denotes a class of nonuniform allocation schemes in which different computational or planning entities receive different step budgets, update frequencies, or work fractions rather than being processed under a uniform schedule. In its explicit contemporary usage, HSA is a training-free inference algorithm for Diffusion Transformers (DiTs) that assigns different denoising step counts to different spatiotemporal tokens according to their velocity dynamics. Closely related formulations appear in heterogeneous computing, where Monte Carlo work is fractionally split across CPUs, GPUs, and FPGAs under latency–accuracy models, and in heterogeneous multi-agent planning, where asynchronous spatial-temporal allocation assigns agent-specific replanning schedules and pairwise safe regions without a global clock (Chu et al., 7 May 2026, Inggs et al., 2015, Chen et al., 2023).

1. Conceptual structure

Taken together, the cited formulations indicate a recurring structure: a workload is partitioned into heterogeneous units; each unit is assigned a nonuniform update or execution budget; and the resulting mismatch in timing or workload granularity is repaired by an explicit synchronization mechanism. In DiT inference, the units are spatiotemporal tokens grouped into sets $G_1,\dots,G_K$ with different denoising budgets $s_k$. In heterogeneous computing, the units are pricing tasks whose Monte Carlo paths are split across platforms through fractions $a_{i,p}$. In asynchronous multi-agent planning, the units are agent-specific replanning steps and pairwise time-stamped half-space renewals.

Setting	Allocated object	Coordination mechanism
DiT video generation	Token groups $G_k$ with budgets $s_k$	Modular activation, KV-cache synchronization, cached Euler update
Heterogeneous computing	Task fractions $a_{i,p}$ and binaries $b_{i,p}$	MILP minimizing makespan $Z$
Multi-agent planning	Step schedules $t_i^{k+1}=t_i^k+\Delta t_i^k+w_i^k$ and TSHS renewals	Asynchronous spatial-temporal allocation

A useful distinction follows from the sources. In (Chu et al., 7 May 2026), HSA is the name of a specific algorithm. In (Inggs et al., 2015) and (Chen et al., 2023), the term is used interpretively: the underlying methods realize the same broad idea of heterogeneous allocation, but under different domain objectives and constraints.

2. HSA in Diffusion Transformers

In video generation with DiTs, standard inference applies the same number of reverse denoising steps to every token. HSA replaces this uniform policy with a grouped budget allocation. The token sequence $S=\{1,\dots,N\}$ is partitioned into disjoint groups $s_k$0, each group receives a step budget $s_k$1, and each $s_k$2 divides the total number of diffusion steps $s_k$3. The baseline group satisfies $s_k$4. The effective average step count per token is

$s_k$5

so the ideal speedup relative to the uniform baseline is $s_k$6 (Chu et al., 7 May 2026).

The active-set rule is modular. At global iteration $s_k$7, group $s_k$8 is active if and only if $s_k$9, giving

$a_{i,p}$0

This preserves aligned scheduling across groups despite heterogeneous budgets. The construction is defined on top of flow matching, where the DiT predicts a continuous velocity field $a_{i,p}$1 over latents $a_{i,p}$2, and the uniform baseline uses the Euler reverse integrator

$a_{i,p}$3

Budget assignment is driven by token velocity dynamics observed online. During an initial bootstrap phase, the method computes each token’s relative velocity change,

$a_{i,p}$4

averages this quantity over several early steps, and uses the result to rank tokens. Tokens with smaller average relative change are assigned to lower-budget groups, while tokens with larger change remain in higher-budget groups up to the full baseline. Group sizes are chosen to meet a runtime target, and membership is determined per sample.

HSA applies to both text-to-video and image-to-video generation. The method is explicitly described as training-free and model-agnostic, and it uses the same $a_{i,p}$5 schedule as the pretrained model.

3. Synchronization, caching, and numerical update

Nonuniform activation creates an immediate systems problem: at a given iteration, only a subset of tokens is active, but active tokens still require global context. HSA resolves this through KV-cache synchronization. For each transformer block $a_{i,p}$6, cached keys and values $a_{i,p}$7 and $a_{i,p}$8 are maintained over the full sequence. At iteration $a_{i,p}$9, fresh $G_k$0 are computed only for active tokens, the cache is updated at active indices, and attention is evaluated only for active queries against the full cached key–value state:

$G_k$1

This reduces per-step attention complexity from $G_k$2 to $G_k$3, because only $G_k$4 queries attend to $G_k$5 keys while inactive tokens skip query projection, attention, and feed-forward computation (Chu et al., 7 May 2026).

A second mechanism handles skipped latent updates. Each token $G_k$6 stores a cached velocity $G_k$7 from its last active iteration. HSA then advances all tokens with a unified Euler step,

$G_k$8

where $G_k$9 if $s_k$0, and $s_k$1 otherwise. For skipped tokens, repeated application telescopes to a single Euler step from the last active noise level with constant velocity. The paper emphasizes that this is a single tensor operation across all $s_k$2 tokens and is therefore GPU-friendly.

Because Q/K/V features and velocities exhibit higher relative change early and late in denoising, HSA restricts caching to a central window. If the central caching fraction is $s_k$3, with margin $s_k$4, then the first $s_k$5 and last $s_k$6 steps use full computation for all tokens, while heterogeneous activation is applied only to the middle interval. The local truncation error of Euler is $s_k$7, so this phase-aware design shields structure formation and detail refinement from excessive cache staleness.

The implementation further reorders tokens so non-baseline tokens precede the always-active baseline $s_k$8. Since $s_k$9 is refreshed every iteration, only the non-baseline prefix requires persistent KV storage.

4. Presets, empirical behavior, and quality–runtime trade-offs

The reported presets are tied to runtime targets. For text-to-video and image-to-video, the paper uses the following examples: HSA-75A with $a_{i,p}$0 tokens at $a_{i,p}$1 steps, $a_{i,p}$2 at $a_{i,p}$3 steps, and $a_{i,p}$4 at $a_{i,p}$5 steps; HSA-75B with $a_{i,p}$6 at $a_{i,p}$7 steps and $a_{i,p}$8 at $a_{i,p}$9 steps; HSA-50 with $b_{i,p}$0 at $b_{i,p}$1 steps and $b_{i,p}$2 at $b_{i,p}$3 steps; and HSA-25 with $b_{i,p}$4 at $b_{i,p}$5 steps and $b_{i,p}$6 at $b_{i,p}$7 steps. The more aggressive presets use a larger central caching window and, at $b_{i,p}$8 runtime, random allocation because there are insufficient early steps to bootstrap token dynamics (Chu et al., 7 May 2026).

On Wan-2.1-1.3B text-to-video, HSA improves the quality-runtime Pareto frontier over uniform Flow Matching and TeaCache, especially at aggressive acceleration. At $b_{i,p}$9 runtime, HSA-75A reports $Z$0 VBench and HSA-75B $Z$1, versus $Z$2 for FM($Z$3) and $Z$4 for TC($Z$5). The distortion metrics relative to FM($Z$6) are markedly better for HSA-75A/B, with PSNR approximately $Z$7 and $Z$8, and LPIPS approximately $Z$9–$t_i^{k+1}=t_i^k+\Delta t_i^k+w_i^k$0, compared with PSNR $t_i^{k+1}=t_i^k+\Delta t_i^k+w_i^k$1 and LPIPS $t_i^{k+1}=t_i^k+\Delta t_i^k+w_i^k$2 for FM($t_i^{k+1}=t_i^k+\Delta t_i^k+w_i^k$3). At $t_i^{k+1}=t_i^k+\Delta t_i^k+w_i^k$4 runtime, HSA-50 reports $t_i^{k+1}=t_i^k+\Delta t_i^k+w_i^k$5 versus $t_i^{k+1}=t_i^k+\Delta t_i^k+w_i^k$6 for both FM($t_i^{k+1}=t_i^k+\Delta t_i^k+w_i^k$7) and TC($t_i^{k+1}=t_i^k+\Delta t_i^k+w_i^k$8), with PSNR $t_i^{k+1}=t_i^k+\Delta t_i^k+w_i^k$9 versus $S=\{1,\dots,N\}$0 and LPIPS $S=\{1,\dots,N\}$1 versus $S=\{1,\dots,N\}$2. At $S=\{1,\dots,N\}$3 runtime, HSA-25 reports $S=\{1,\dots,N\}$4 versus $S=\{1,\dots,N\}$5 for FM($S=\{1,\dots,N\}$6) and $S=\{1,\dots,N\}$7 for TC($S=\{1,\dots,N\}$8); LPIPS is comparable at approximately $S=\{1,\dots,N\}$9–$s_k$00, but VBench remains higher.

On Wan-2.1-1.3B image-to-video, HSA remains competitive within the narrow quality band imposed by the input image and preserves the reference trajectory more effectively than uniform Flow Matching at the same runtime. At $s_k$01 runtime, HSA-75A/B report VBench-I2V of approximately $s_k$02–$s_k$03, PSNR approximately $s_k$04, and LPIPS approximately $s_k$05, compared with PSNR $s_k$06 and LPIPS $s_k$07 for FM($s_k$08). At $s_k$09 runtime, HSA-50 reports PSNR $s_k$10 and LPIPS $s_k$11, versus $s_k$12 and $s_k$13 for FM($s_k$14). Qualitative results on Wan-2.2-A14B and LTX-2 are reported to preserve image alignment, structure, and motion fidelity better than FM($s_k$15) at $s_k$16 runtime.

Ablation studies attribute the gains primarily to dynamic token selection and the phase-aware cache window. Dynamic selection outperforms Uniform, Random, and Random-with-first-frame reservation at approximately $s_k$17 and $s_k$18 runtimes. Per-dimension VBench radar plots show that HSA tracks the full-budget reference across all sixteen dimensions more closely than the baselines, particularly when runtime is reduced to $s_k$19 and $s_k$20.

5. HSA as fractional work allocation in heterogeneous computing

A domain-specific heterogeneous computing formulation realizes HSA by splitting Monte Carlo work across distributed CPUs, GPUs, and FPGAs. Tasks are indexed by $s_k$21, platforms by $s_k$22, and the relaxed HSA variable $s_k$23 denotes the fraction of task $s_k$24’s Monte Carlo paths executed on platform $s_k$25, with $s_k$26. To account for fixed setup and communication overheads once per task–platform pair, binary variables $s_k$27 are introduced with $s_k$28. The platform completion time is

$s_k$29

and makespan minimization is expressed through the standard auxiliary variable $s_k$30 (Inggs et al., 2015).

The model is built from two domain metrics. Latency is linear in the number of Monte Carlo paths,

$s_k$31

while the $s_k$32 confidence interval width obeys

$s_k$33

For a required accuracy $s_k$34, the number of paths is

$s_k$35

and substitution yields the latency–accuracy model

$s_k$36

with $s_k$37. The allocator therefore optimizes latency directly as a function of accuracy.

The empirical setting contains $s_k$38 option-pricing tasks drawn from Black–Scholes and Heston underlyings and a heterogeneous pool of $s_k$39 platforms comprising multicore CPUs, GPUs, and FPGAs. Model calibration uses online micro-benchmarking and weighted least squares for $s_k$40 and $s_k$41, while $s_k$42 is estimated from initial Monte Carlo samples. Across the $s_k$43-task, $s_k$44-platform workload, latency and accuracy predictions are generally within about $s_k$45 of runtime observations. When these models are used by the metaheuristic allocator and the MILP allocator, reported makespan improvements reach up to $s_k$46 and $s_k$47, respectively, relative to the proportional heuristic.

Within this formulation, HSA is not simple load balancing. The split variable $s_k$48 captures proportional work, the binary $s_k$49 isolates one-time per-platform overheads, and the combined model exposes when redundant setup costs dominate and therefore when a naive proportional split becomes poor.

6. HSA as asynchronous step scheduling in heterogeneous multi-agent systems

In heterogeneous multi-agent trajectory planning, the paper does not introduce the term Heterogeneous Step Allocation. Its central mechanism is Asynchronous Spatial-Temporal Allocation (ASTA). The provided concept mapping identifies ASTA as a concrete realization of HSA because it allocates heterogeneous replanning steps and pairwise spatiotemporal safe regions across agents with different dynamics, compute times, and waiting times (Chen et al., 2023).

Each agent $s_k$50 evolves under agent-specific dynamics

$s_k$51

with heterogeneity arising from different models such as double integrator, unicycle, and bicycle; different bounds such as $s_k$52, $s_k$53, $s_k$54, and $s_k$55; different convex polygon footprints $s_k$56; and different onboard compute times $s_k$57 and waiting times $s_k$58. The asynchronous step schedule is

$s_k$59

with the design condition $s_k$60.

Safety is enforced through pairwise time-stamped half-spaces,

$s_k$61

assembled into the time-stamped allocation

$s_k$62

The pairwise half-spaces satisfy $s_k$63 and $s_k$64, hence $s_k$65. The derivable time-stamped space for agent $s_k$66 is the intersection over current neighbors,

$s_k$67

The key asynchronous mechanism is the renewal update. When one agent finishes replanning and a neighbor is in its waiting window, a renewal $s_k$68 is computed and only the future portion of the pairwise allocation is replaced: $s_k$69 By freezing the allocation before $s_k$70, the method preserves safety even if neighboring agents temporarily use different allocation versions.

The paper states two formal guarantees. The safety theorem shows that, given collision-free initialization and adherence to the current allocations, inter-agent collisions are avoided for all future time. The update-frequency theorem states that if $s_k$71 for all agents, then the pairwise update frequency satisfies

$s_k$72

Reported results include an $s_k$73-agent heterogeneous crossing task in a $s_k$74 m diameter circle, with minimum inter-agent distance approximately $s_k$75 m for $s_k$76 m and completion within $s_k$77 s. In the $s_k$78-agent antipodal comparison against Ego-swarm, IMPC-DR, DMPC, LSC, and MADER, the proposed asynchronous method reports moving time $s_k$79 s min/max, replanning runtime $s_k$80 ms min/max, and transition length $s_k$81 m min/max. Hardware experiments with $s_k$82 heterogeneous UGVs, onboard Raspberry Pi 4B computation, WiFi/ROS communication, and low-level MPC tracking report no collisions in a cluttered arena and safe behavior at an unsignalized intersection.

7. Distinctions, limitations, and future directions

The available literature suggests that HSA is best treated as a family of nonuniform allocation mechanisms rather than as a single standardized formalism. The explicit name refers to token-level denoising allocation in DiTs, whereas the heterogeneous computing and multi-agent papers instantiate analogous structures under different mathematical objects and objectives (Chu et al., 7 May 2026, Inggs et al., 2015, Chen et al., 2023).

Several distinctions are technically important. In DiTs, HSA is training-free and avoids expensive offline profiling, but its speedup diminishes in rare cases of globally high motion or complex scenes where most tokens exhibit high velocity. Practical gains can also be limited if KV caches must be offloaded to CPU on constrained hardware. In heterogeneous computing, the main formulation assumes independent tasks and sequential execution per platform; communication overlaps, batching effects, and concurrent task execution are not explicitly modeled, although the details note that such effects can be approximated by adjusted $s_k$83 and $s_k$84 or by additional constraints. In ASTA, safety proofs do not model communication delays or packet loss, and feasibility of the trajectory optimization problem is not guaranteed; when infeasible, the agent holds the previous trajectory.

The future directions stated in the sources remain domain-specific. For video generation, proposed extensions include learning or optimizing the budget mapping function $s_k$85, adaptive per-iteration budget reallocation, richer dynamics estimators such as velocity curvature or second-order terms, and joint optimization with other caching axes while preserving full-context attention. For multi-agent planning, proposed extensions include robust treatment of delays, slack or penalty structures for recursive feasibility, scaling mechanisms for denser neighborhoods, 3D and aerial robots, dynamic obstacles, and asynchronous distributed optimization such as ADMM layered under the same allocations. For heterogeneous computing, the generalizable guidance is to define domain metrics, derive simple predictive models linking algorithmic knobs to those metrics, calibrate coefficients online, and encode the resulting allocation as ILP or MILP with explicit variables for fractional work and overhead activation.

In all three settings, the central technical idea is the same: heterogeneous systems need not spend the same number of steps, updates, or execution fractions on every unit of work. The formal contribution of HSA-style methods is to make that asymmetry explicit, model it quantitatively, and preserve correctness or quality through a synchronization mechanism appropriate to the domain.