Cluster-Driven Feature Caching (ClusCa)

• The paper introduces ClusCa, an inference-time method that exploits spatial token clustering alongside temporal caching to reduce token computations by over 90%.
• ClusCa uses a cycle-based pipeline where full calculation steps are followed by partial steps that recompute only cluster representatives using K-means and feature propagation.
• The method achieves 4–6× acceleration on models like FLUX, DiT, and HunyuanVideo without retraining, while preserving or even enhancing generation quality.

Searching arXiv for the cited papers to ground the article and confirm metadata. Cluster-Driven Feature Caching (ClusCa) is an inference-time acceleration method for diffusion transformers introduced in “Compute Only 16 Tokens in One Timestep: Accelerating Diffusion Transformers with Cluster-Driven Feature Caching” (Zheng et al., 12 Sep 2025). It is defined as a lightweight, training-free wrapper that can be dropped into any diffusion Transformer at inference time to exploit both spatial and temporal redundancy among its patch tokens. In contrast to earlier feature caching schemes that reuse features across denoising timesteps while ignoring spatial similarity, ClusCa adds a spatial clustering mechanism within each timestep: it computes only one token in each cluster and propagates their information to the remaining tokens. The method is presented as orthogonal and complementary to previous feature caching, and is evaluated on DiT, FLUX, and HunyuanVideo for both text-to-image and text-to-video generation (Zheng et al., 12 Sep 2025).

1. Conceptual basis and problem setting

Diffusion transformers generate images and videos through an iterative denoising process, and their computational cost is dominated by repeated Transformer evaluations across timesteps (Zheng et al., 12 Sep 2025). Feature caching was introduced to reduce this cost by caching feature computation in previous timesteps and reusing it in following timesteps, leveraging temporal similarity. The central limitation identified by ClusCa is that such methods ignore similarity in the spatial dimension.

ClusCa addresses that omission by combining temporal reuse with spatial grouping. The core claim is that tokens within a timestep exhibit redundancy that can be exploited by clustering patch-token features and recomputing only a representative token from each cluster. The remaining tokens receive approximated updates through propagation from the representative computation and the cached history. In the formulation given for partial steps, this reduces the number of computed tokens by over 90% for typical settings such as $H \cdot W = 256$ and $K = 16$, where the token reduction ratio is $(256-16)/256 = 93.75\%$ (Zheng et al., 12 Sep 2025).

The term “feature caching” in this context refers to maintaining per-token, per-block features across diffusion timesteps. The term “cluster-driven” designates the addition of a spatial partition over tokens, obtained by K-means on last-layer features, to guide which tokens are recomputed and how updates are propagated. This framing places ClusCa among inference-time approximation methods rather than training-time model compression or architectural redesign.

2. Cache-cycle pipeline and operational stages

ClusCa partitions the $T$ denoising steps into disjoint cache cycles of length $N$ (Zheng et al., 12 Sep 2025). The first step of each cycle is a full-calculation step, and the remaining $N-1$ steps are partial-calculation steps. This cyclic structure is the organizing principle of the method.

At the first step of a cycle, denoted $t_0 = 1 \pmod N$, ClusCa computes all $H \times W$ tokens through every Transformer block $g_l$, stores their output features $C_i^l$ in a cache, and runs K-means on the last-layer features to obtain $K = 16$0 spatial clusters $K = 16$1 with assignments $K = 16$2 (Zheng et al., 12 Sep 2025). This stage establishes both the temporal cache and the spatial partition used during later partial steps.

In each subsequent timestep within the same cycle, ClusCa selects exactly one token per cluster to recompute through the Transformer blocks. The set of selected indices is written as $K = 16$3, and representative selection may be random or based on simple heuristics (Zheng et al., 12 Sep 2025). For each selected token, the full self-/cross-attention plus MLP stack is executed, and the corresponding cache entries are updated.

For non-selected tokens, ClusCa does not execute the full block stack. Instead, it approximates the new feature through a weighted mixture of the old cache and the newly computed co-cluster centroid:

$K = 16$4

where $K = 16$5 is the mean of the computed features in cluster $K = 16$6, and $K = 16$7 is the propagation ratio (Zheng et al., 12 Sep 2025). Since exactly one token per cluster is selected in the described pipeline, the centroid is determined from the computed representative set.

This four-stage organization—full calculation and clustering, representative token selection, feature computation for cluster centers, and propagation to other tokens—defines ClusCa as a spatiotemporal caching scheme rather than a purely temporal cache reuse mechanism (Zheng et al., 12 Sep 2025).

3. Formalization and update equations

The formal notation in (Zheng et al., 12 Sep 2025) defines $K = 16$8 as the feature of patch token $K = 16$9 at Transformer block $(256-16)/256 = 93.75\%$0 and diffusion timestep $(256-16)/256 = 93.75\%$1. The layer-$(256-16)/256 = 93.75\%$2 update is denoted $(256-16)/256 = 93.75\%$3.

During a full-calculation step, the final features $(256-16)/256 = 93.75\%$4 are collected and clustered by K-means. The clustering objective is written in two equivalent forms. The first is a partition form minimizing intra-cluster variance:

$(256-16)/256 = 93.75\%$5

The second is a centroid form:

$(256-16)/256 = 93.75\%$6

After convergence, assignments $(256-16)/256 = 93.75\%$7 and centroids $(256-16)/256 = 93.75\%$8 are recorded (Zheng et al., 12 Sep 2025).

The cache structure maintains $(256-16)/256 = 93.75\%$9 for every block $T$0 and token $T$1, storing the latest computed feature. This allows a direct contrast with temporal-caching-only baselines such as FORA or ToCa, in which one would simply reuse $T$2 for $T$3 (Zheng et al., 12 Sep 2025). ClusCa instead applies a spatiotemporal update during partial-calculation steps. For each selected representative $T$4,

$T$5

while for each non-selected token $T$6,

$T$7

The paper characterizes $T$8 as the parameter that balances new spatial information from the recomputed cluster representatives against the preserved temporal cache (Zheng et al., 12 Sep 2025).

A plausible implication is that ClusCa relies on the stability of both cluster structure and cached activations over short denoising intervals. The paper’s cycle-based design and small recommended values of $T$9 are consistent with that interpretation, though the explicit empirical claims remain those reported in the ablations.

4. Complexity reduction and token skipping

The computational analysis in (Zheng et al., 12 Sep 2025) expresses the original cost per cycle of $N$0 denoising steps, without caching, as approximately $N$1, where $N$2 times per-token overhead. Under ClusCa, the cost per cycle is approximated as

$N$3

because a full Transformer pass is performed once per cycle and only $N$4 tokens are recomputed in each remaining partial step.

This yields an effective speed-up:

$N$5

For the typical example $N$6, $N$7, and $N$8, the reported computation is

$N$9

The same section notes that empirically on FLUX this becomes $N-1$0 end-to-end (Zheng et al., 12 Sep 2025).

The token reduction ratio in partial steps is given as

$N-1$1

which, for $N-1$2 tokens and $N-1$3 representatives, becomes $N-1$4 (Zheng et al., 12 Sep 2025). This is the basis of the paper’s statement that ClusCa is able to reduce the number of tokens by over 90%.

The method therefore shifts the dominant approximation from omitting timesteps to omitting most token computations within selected timesteps. This suggests a different trade-off surface from purely temporal caching methods: quality degradation is controlled by cluster count $N-1$5, cycle length $N-1$6, and propagation ratio $N-1$7, rather than by temporal reuse alone.

5. Hyper-parameters, implementation, and inference integration

The implementation described in (Zheng et al., 12 Sep 2025) is fully inference-time and does not modify model weights or require fine-tuning. The stated inputs to the pseudocode are a Transformer $N-1$8, cache cycle length $N-1$9, cluster count $t_0 = 1 \pmod N$0, and propagation ratio $t_0 = 1 \pmod N$1. Cache initialization is performed for $t_0 = 1 \pmod N$2 over all tokens, for example with zeros.

Several hyper-parameters are singled out. The cluster count $t_0 = 1 \pmod N$3 is typically 16 for DiT and FLUX image models, and 32 for HunyuanVideo. The cycle length $t_0 = 1 \pmod N$4 is 5–7 steps; larger $t_0 = 1 \pmod N$5 yields more speedup but accumulates more error. The propagation ratio $t_0 = 1 \pmod N$6 is small, 0.002–0.005, to smoothly blend spatial updates, and ablations show an optimum around $t_0 = 1 \pmod N$7–$t_0 = 1 \pmod N$8 (Zheng et al., 12 Sep 2025). Clustering is performed once per cycle with at most 5% overhead, and the previous cycle’s centroids are used as initialization to accelerate convergence. Representative selection may be random or fixed one per cluster, and empirically one suffices.

The method is described as interposing a small caching and cluster-propagation layer around Transformer blocks while treating the Transformer as a black-box feature extractor (Zheng et al., 12 Sep 2025). No retraining or access to model internals beyond the block outputs is necessary. This is the operational basis for the paper’s generality claim across DiT, FLUX, HunyuanVideo, and future architectures.

A common misconception would be to interpret ClusCa as a learned token-pruning module. The reported design does not require training, does not alter weights, and does not introduce a learned routing network. Its approximation mechanism is cache-based and clustering-driven, with K-means and deterministic or random representative selection at inference time (Zheng et al., 12 Sep 2025).

6. Empirical results on image and video generation

The empirical evaluation in (Zheng et al., 12 Sep 2025) covers FLUX.1-dev, DiT-XL/2, and HunyuanVideo. The following table reproduces the main reported results.

Model / Setting	Speed-up or latency	Quality metrics
Original (FlashAttention), FLUX.1-dev	latency = 25.82 s, 1.00×	ImageReward = 0.9898, CLIP = 19.761
ClusCa $t_0 = 1 \pmod N$9, FLUX.1-dev	latency = 7.12 s, 3.62×, FLOPs↓ = 748.48 T, end-to-end speed = 4.96×	ImageReward = 0.9949, CLIP = 19.453
ToCa $H \times W$0, DiT-XL/2	2.44×	FID = 2.87, sFID = 4.76
ClusCa $H \times W$1, DiT-XL/2	3.23×	FID = 2.51, sFID = 5.03
ClusCa $H \times W$2, DiT-XL/2	5.91×	FID = 3.59, sFID = 6.28
TaylorSeer $H \times W$3, HunyuanVideo	5.56×	VBench Score = 79.78%
ClusCa $H \times W$4, HunyuanVideo	5.54×	VBench Score = 79.96%
ClusCa $H \times W$5, HunyuanVideo	6.21×	VBench Score = 79.60%

For FLUX.1-dev at 50 steps and $H \times W$6 resolution, ClusCa with $H \times W$7 reduces latency from 25.82 s to 7.12 s, with 3.62× acceleration and 4.96× end-to-end speed, while increasing ImageReward from 0.9898 to 0.9949 and slightly decreasing CLIP from 19.761 to 19.453 (Zheng et al., 12 Sep 2025). The abstract summarizes this as 4.96× acceleration on FLUX with an ImageReward of 99.49%, surpassing the original model by 0.51% (Zheng et al., 12 Sep 2025).

For DiT-XL/2 on ImageNet 50k samples, ClusCa $H \times W$8 achieves 3.23× speed-up with FID 2.51 and sFID 5.03, while ClusCa $H \times W$9 reaches 5.91× with FID 3.59 and sFID 6.28 (Zheng et al., 12 Sep 2025). The comparison point listed is ToCa $g_l$0 at 2.44×, FID 2.87, sFID 4.76.

For HunyuanVideo on 946 prompts using VBench, ClusCa $g_l$1 attains 5.54× speed-up with 79.96% VBench Score, and ClusCa $g_l$2 attains 6.21× with 79.60% (Zheng et al., 12 Sep 2025). The listed comparison is TaylorSeer $g_l$3 at 5.56× and 79.78%.

The ablations report that $g_l$4 yields a U-shaped FID curve with optimum around $g_l$5–$g_l$6; $g_l$7 trades compute against fidelity, with $g_l$8 or $g_l$9 giving the best quality/speed balance; and $C_i^l$0 yields up to 5–6× speedup at modest FID/sFID degradation (Zheng et al., 12 Sep 2025).

7. Generality, related notions of “ClusCa,” and broader context

Within diffusion transformers, ClusCa is presented as a plug-and-play strategy applicable to any diffusion Transformer, including DiT, FLUX, HunyuanVideo, or future architectures, simply by wrapping the inference loop (Zheng et al., 12 Sep 2025). The reported justification is that it does not modify model weights, require fine-tuning, or demand access to internals beyond block outputs. Its generality is therefore architectural rather than task-specific.

The name “ClusCa” also appears in a distinct and unrelated context in graph analytics, where a summary of Cagra’s “frequency-based clustering” is labeled “ClusCa” in the provided material and linked to “Making Caches Work for Graph Analytics” (Zhang et al., 2016). There, the technique reorders vertices so that high-frequency vertices are laid out contiguously, improving cache-line utilization and L2/L3 hit rates during pull-style graph algorithms. The mechanism, objective, and computational substrate differ fundamentally from diffusion-transformer feature caching: graph-analytic clustering is a one-time data-layout transformation on vertices and CSR structures, whereas Cluster-Driven Feature Caching in diffusion transformers is an inference-time token clustering and feature-propagation method (Zhang et al., 2016).

The shared label can create terminological ambiguity. In the graph setting, “frequency-based clustering” groups hot vertices using out-degree as a proxy for pull frequency and yields zero extra instructions at steady state after preprocessing (Zhang et al., 2016). In the diffusion-transformer setting, ClusCa groups patch tokens by K-means over last-layer features within a cache cycle and updates non-recomputed tokens through a mixture of cached state and cluster-level propagated information (Zheng et al., 12 Sep 2025). The overlap is thus nominal rather than methodological.

Taken on its own terms, Cluster-Driven Feature Caching defines a specific spatiotemporal approximation regime for diffusion transformers: one full step per cache cycle, clustered token representatives in subsequent steps, and propagation controlled by $C_i^l$1. The reported outcome is 90%+ token skipping per partial step, 4–6× measured end-to-end acceleration, and preservation or improvement of selected quality metrics in the tested settings (Zheng et al., 12 Sep 2025). A plausible implication is that the method is most attractive where denoising trajectories exhibit both short-horizon temporal smoothness and spatial redundancy in token features; this interpretation is consistent with the design and the reported evaluations, though the empirical claims remain those explicitly enumerated in the paper.