HiCache: Dual Mechanisms in ML Systems

• HiCache is a dual-purpose ML mechanism that uses Hermite polynomial forecasting for caching intermediate features in diffusion models and multi-tier KV caching in LLM serving.
• In diffusion models, HiCache predicts skipped activations during iterative denoising, achieving significant FLOPs and wall-clock speedups while preserving image quality.
• In LLM serving, HiCache manages a multi-tier cache spanning GPU, CPU, and external storage, improving throughput and reducing latency for large-scale token retrieval.

Searching arXiv for papers on “HiCache” and closely related uses of the term. HiCache is a name used in contemporary machine-learning systems literature for two unrelated caching mechanisms. In diffusion-model research, HiCache denotes a training-free acceleration framework that predicts intermediate features with Hermite polynomials during iterative denoising, thereby reducing the cost of text-to-image, video generation, and super-resolution inference (Feng et al., 23 Aug 2025). In LLM-serving research, “SGLang/HiCache” denotes a multi-tier KV-cache subsystem that extends RadixAttention into a hierarchy spanning GPU, CPU, and external storage, and is studied as a substrate for data movement and formal cache-obligation semantics rather than as a Hermite-based predictor (Ren et al., 1 Apr 2026, Stepanek, 31 May 2026).

1. Terminological scope and research contexts

The two usages differ in object, granularity, and optimization target. The diffusion-model usage caches and forecasts internal activations across denoising timesteps; the LLM-serving usage manages resident KV blocks across memory tiers and network paths. Confusing them obscures both the mathematical contribution of the diffusion paper and the systems contribution of the serving papers.

Usage of “HiCache”	Domain	Core role
HiCache	Diffusion models	Hermite polynomial-based feature caching
SGLang/HiCache	LLM serving	Multi-tier KV hierarchy spanning GPU, CPU, and external storage

In the diffusion lineage, HiCache belongs to the broader family of feature-caching methods that exploit the smooth evolution of activations along the reverse diffusion trajectory. The ReCache work situates it specifically as a feature-forecasting mechanism, distinct from feature-reuse methods such as FORA and $\Delta$-DiT and from Taylor-based forecasting such as TaylorSeer (Aliev et al., 4 Jun 2026). In the serving lineage, HiCache appears as a sophisticated KV substrate with radix priority, lock/reference state, protected and evictable accounting, GPU/host/storage tiers, prefetch, write-back, load-back, storage success/failure flags, cached-token details, and block events, but with semantics that are analyzed independently of any diffusion-model usage (Stepanek, 31 May 2026).

2. HiCache as a diffusion-model feature-forecasting mechanism

The diffusion-model HiCache addresses the cost of iterative sampling in large Diffusion Transformers. During denoising, a model evaluates a large network at each timestep, and the resulting compute is substantial for systems such as FLUX.1-dev, HunyuanVideo, DiT-XL/2, and Inf-DiT. Feature caching accelerates this process by storing intermediate features at sparse activation timesteps and predicting features at skipped timesteps instead of recomputing them from scratch (Feng et al., 23 Aug 2025).

HiCache is positioned against two prior families. Cache-then-reuse methods such as DeepCache, FORA, and ToCa simply reuse nearby features; they work only for small skips and can induce distortions, artifacts, and style drift at larger intervals. Cache-then-forecast methods such as TaylorSeer extrapolate feature trajectories with truncated Taylor series, which substantially improves over naive reuse but suffers when the underlying trajectories are non-monotonic and when high-order terms become noisy and numerically unstable (Feng et al., 23 Aug 2025).

HiCache’s defining claim is that the basis used for feature forecasting should match the empirical statistics of feature evolution. It therefore replaces the Taylor monomial basis with a Hermite polynomial basis. In the ReCache framing, HiCache caches attention and FFN outputs at cache steps and forecasts them at reuse steps using Hermite polynomial extrapolation; “uniform HiCache” denotes this mechanism coupled with a fixed, uniformly spaced subset of timesteps of size $k$ (Aliev et al., 4 Jun 2026).

3. Mathematical basis and inference workflow

The mathematical starting point is the observation that derivative approximations of diffusion features behave like samples from a multivariate Gaussian. Using backward finite differences,

$$\Delta^{(0)} F_t = F_t, \quad \Delta^{(i)}F_{t} = \frac{\Delta^{(i-1)}F_{t} - \Delta^{(i-1)}F_{t-1}}{\Delta t},\quad i\ge 1,$$

the authors report that, for each order $k$, $\Delta^{(k)} F_t \sim \mathcal{N}(\mu_k, \Sigma_k)$. They validate this across multiple DiT modules in FLUX.1-dev using the Energy test for multivariate normality and report a p-value of 1.0 for every configuration examined (Feng et al., 23 Aug 2025).

This motivates the use of the physicist’s Hermite polynomials,

$H_n(x) = (-1)^n e^{x^2}\frac{d^n}{dx^n} e^{-x^2},$

with recurrence

$H_{n+1}(x) = 2x H_n(x) - 2n H_{n-1}(x),$

because these polynomials are orthogonal with respect to a Gaussian weight and, in the authors’ framing, are optimal for representing Gaussian-correlated processes. HiCache further introduces scaled Hermite polynomials,

$$\tilde{H}_n(x) = \sigma^n H_n(\sigma x), \quad \sigma \in (0,1),$$

which implement a dual-scaling mechanism: input scaling keeps $|\sigma x|$ in a numerically stable regime, while coefficient scaling suppresses high-order terms (Feng et al., 23 Aug 2025).

Operationally, HiCache alternates between cache-update steps and prediction steps. At activation timesteps $t_i$, the full model is run, the feature $k$0 is computed, and derivative approximations $k$1 up to order $k$2 are stored. At skipped timesteps, the model predicts features from the most recent activation step $k$3 and skip distance $k$4 via

$k$5

The method is entirely training-free, does not modify model weights, and integrates by rewiring the inference-time forward pass to record $k$6, compute finite differences, and predict selected block outputs at skipped steps (Feng et al., 23 Aug 2025).

4. Empirical performance across modalities

The diffusion-model paper reports substantial acceleration with limited quality degradation and, in some settings, quality gains relative to the unaccelerated baseline. On FLUX.1-dev at $k$7, interval $k$8 with $k$9 yields a FLOPs speedup of about $$\Delta^{(0)} F_t = F_t, \quad \Delta^{(i)}F_{t} = \frac{\Delta^{(i-1)}F_{t} - \Delta^{(i-1)}F_{t-1}}{\Delta t},\quad i\ge 1,$$0 and a wall-clock speedup of about $$\Delta^{(0)} F_t = F_t, \quad \Delta^{(i)}F_{t} = \frac{\Delta^{(i-1)}F_{t} - \Delta^{(i-1)}F_{t-1}}{\Delta t},\quad i\ge 1,$$1; at $$\Delta^{(0)} F_t = F_t, \quad \Delta^{(i)}F_{t} = \frac{\Delta^{(i-1)}F_{t} - \Delta^{(i-1)}F_{t-1}}{\Delta t},\quad i\ge 1,$$2, the reported FLOPs speedup is $$\Delta^{(0)} F_t = F_t, \quad \Delta^{(i)}F_{t} = \frac{\Delta^{(i-1)}F_{t} - \Delta^{(i-1)}F_{t-1}}{\Delta t},\quad i\ge 1,$$3 and wall-clock speedup is about $$\Delta^{(0)} F_t = F_t, \quad \Delta^{(i)}F_{t} = \frac{\Delta^{(i-1)}F_{t} - \Delta^{(i-1)}F_{t-1}}{\Delta t},\quad i\ge 1,$$4. Against the DDIM-50 baseline with ImageReward $$\Delta^{(0)} F_t = F_t, \quad \Delta^{(i)}F_{t} = \frac{\Delta^{(i-1)}F_{t} - \Delta^{(i-1)}F_{t-1}}{\Delta t},\quad i\ge 1,$$5, TaylorSeer at $$\Delta^{(0)} F_t = F_t, \quad \Delta^{(i)}F_{t} = \frac{\Delta^{(i-1)}F_{t} - \Delta^{(i-1)}F_{t-1}}{\Delta t},\quad i\ge 1,$$6 obtains $$\Delta^{(0)} F_t = F_t, \quad \Delta^{(i)}F_{t} = \frac{\Delta^{(i-1)}F_{t} - \Delta^{(i-1)}F_{t-1}}{\Delta t},\quad i\ge 1,$$7, whereas HiCache reports $$\Delta^{(0)} F_t = F_t, \quad \Delta^{(i)}F_{t} = \frac{\Delta^{(i-1)}F_{t} - \Delta^{(i-1)}F_{t-1}}{\Delta t},\quad i\ge 1,$$8; at $$\Delta^{(0)} F_t = F_t, \quad \Delta^{(i)}F_{t} = \frac{\Delta^{(i-1)}F_{t} - \Delta^{(i-1)}F_{t-1}}{\Delta t},\quad i\ge 1,$$9, TaylorSeer reports $k$0 and HiCache $k$1 (Feng et al., 23 Aug 2025).

The same pattern appears outside text-to-image. On HunyuanVideo, with $k$2 and $k$3, the paper reports FLOPs speedup $k$4, time speedup about $k$5, and slightly better VBench than TaylorSeer. On DiT-XL/2, $k$6 and $k$7 yield around $k$8 speedup with near-baseline FID, while $k$9 yields about $\Delta^{(k)} F_t \sim \mathcal{N}(\mu_k, \Sigma_k)$0 FLOPs speedup with HiCache remaining substantially better than FORA and ToCa. On Inf-DiT super-resolution, interval 8 yields theoretical FLOPs speedup about $\Delta^{(k)} F_t \sim \mathcal{N}(\mu_k, \Sigma_k)$1 and wall-clock speedup about $\Delta^{(k)} F_t \sim \mathcal{N}(\mu_k, \Sigma_k)$2; at that interval, HiCache reports PSNR $\Delta^{(k)} F_t \sim \mathcal{N}(\mu_k, \Sigma_k)$3 and SSIM $\Delta^{(k)} F_t \sim \mathcal{N}(\mu_k, \Sigma_k)$4, compared with TaylorSeer’s PSNR $\Delta^{(k)} F_t \sim \mathcal{N}(\mu_k, \Sigma_k)$5 and SSIM $\Delta^{(k)} F_t \sim \mathcal{N}(\mu_k, \Sigma_k)$6 (Feng et al., 23 Aug 2025).

Ablation results identify the contraction factor $\Delta^{(k)} F_t \sim \mathcal{N}(\mu_k, \Sigma_k)$7 as critical. On FLUX with $\Delta^{(k)} F_t \sim \mathcal{N}(\mu_k, \Sigma_k)$8, $\Delta^{(k)} F_t \sim \mathcal{N}(\mu_k, \Sigma_k)$9 yields ImageReward $H_n(x) = (-1)^n e^{x^2}\frac{d^n}{dx^n} e^{-x^2},$0, while $H_n(x) = (-1)^n e^{x^2}\frac{d^n}{dx^n} e^{-x^2},$1 reduces ImageReward to $H_n(x) = (-1)^n e^{x^2}\frac{d^n}{dx^n} e^{-x^2},$2, worse than TaylorSeer’s $H_n(x) = (-1)^n e^{x^2}\frac{d^n}{dx^n} e^{-x^2},$3. The paper also reports a relative error ratio

$H_n(x) = (-1)^n e^{x^2}\frac{d^n}{dx^n} e^{-x^2},$4

with $H_n(x) = (-1)^n e^{x^2}\frac{d^n}{dx^n} e^{-x^2},$5 across 25 configurations, and observes that the advantage of Hermite grows with larger prediction steps and higher orders (Feng et al., 23 Aug 2025).

5. Budget-aware scheduling and the relation to ReCache

ReCache does not alter HiCache’s prediction mechanism. It treats HiCache as the caching mechanism $H_n(x) = (-1)^n e^{x^2}\frac{d^n}{dx^n} e^{-x^2},$6 and learns only the recomputation schedule, replacing HiCache’s hand-crafted scheduling logic with a learned, budget-aware policy conditioned on a target number of full evaluations $H_n(x) = (-1)^n e^{x^2}\frac{d^n}{dx^n} e^{-x^2},$7 (Aliev et al., 4 Jun 2026).

The ReCache formulation defines, for latent $H_n(x) = (-1)^n e^{x^2}\frac{d^n}{dx^n} e^{-x^2},$8, a full-inference output $H_n(x) = (-1)^n e^{x^2}\frac{d^n}{dx^n} e^{-x^2},$9 and a cached output $H_{n+1}(x) = 2x H_n(x) - 2n H_{n-1}(x),$0, with per-schedule loss

$H_{n+1}(x) = 2x H_n(x) - 2n H_{n-1}(x),$1

where $H_{n+1}(x) = 2x H_n(x) - 2n H_{n-1}(x),$2 is instantiated as patch-wise LPIPS and $H_{n+1}(x) = 2x H_n(x) - 2n H_{n-1}(x),$3 uses HPSv2 for images and HPSv2 + VBench for videos. To optimize a distribution over schedules $H_{n+1}(x) = 2x H_n(x) - 2n H_{n-1}(x),$4, ReCache minimizes the expected loss plus an entropy regularizer and uses REINFORCE rather than backpropagation through full diffusion inference. The schedule selects exactly $H_{n+1}(x) = 2x H_n(x) - 2n H_{n-1}(x),$5 timesteps via a Plackett–Luce distribution with Gumbel-Top-$H_{n+1}(x) = 2x H_n(x) - 2n H_{n-1}(x),$6 sampling, and the policy predictor is a small MLP conditioned on $H_{n+1}(x) = 2x H_n(x) - 2n H_{n-1}(x),$7 (Aliev et al., 4 Jun 2026).

For HiCache specifically, ReCache reports that learned schedules dominate uniform schedules at fixed budgets. On FLUX with $H_{n+1}(x) = 2x H_n(x) - 2n H_{n-1}(x),$8 and $H_{n+1}(x) = 2x H_n(x) - 2n H_{n-1}(x),$9, uniform HiCache $$\tilde{H}_n(x) = \sigma^n H_n(\sigma x), \quad \sigma \in (0,1),$$0 reports LPIPS $$\tilde{H}_n(x) = \sigma^n H_n(\sigma x), \quad \sigma \in (0,1),$$1, HPS $$\tilde{H}_n(x) = \sigma^n H_n(\sigma x), \quad \sigma \in (0,1),$$2, and IR DB $$\tilde{H}_n(x) = \sigma^n H_n(\sigma x), \quad \sigma \in (0,1),$$3, whereas ReCache HiCache $$\tilde{H}_n(x) = \sigma^n H_n(\sigma x), \quad \sigma \in (0,1),$$4 reports LPIPS $$\tilde{H}_n(x) = \sigma^n H_n(\sigma x), \quad \sigma \in (0,1),$$5, HPS $$\tilde{H}_n(x) = \sigma^n H_n(\sigma x), \quad \sigma \in (0,1),$$6, and IR DB $$\tilde{H}_n(x) = \sigma^n H_n(\sigma x), \quad \sigma \in (0,1),$$7. On Wan 2.1 with $$\tilde{H}_n(x) = \sigma^n H_n(\sigma x), \quad \sigma \in (0,1),$$8 and $$\tilde{H}_n(x) = \sigma^n H_n(\sigma x), \quad \sigma \in (0,1),$$9, uniform HiCache $|\sigma x|$0 reports LPIPS $|\sigma x|$1, HPS $|\sigma x|$2, and VBench $|\sigma x|$3, whereas ReCache HiCache $|\sigma x|$4 reports LPIPS $|\sigma x|$5, HPS $|\sigma x|$6, and VBench $|\sigma x|$7. The abstract further summarizes that, on Wan 2.1 at a $|\sigma x|$8 speedup, ReCache drops LPIPS by 65% from $|\sigma x|$9 to $t_i$0 and boosts the VBench score by 7% from $t_i$1 to $t_i$2 over uniform HiCache (Aliev et al., 4 Jun 2026).

This suggests a useful conceptual separation. HiCache addresses what to cache and how to forecast features; ReCache addresses when to recompute under an explicit compute budget. A plausible implication is that the quality–compute frontier depends materially on schedule learning even when the underlying forecasting rule is unchanged.

6. SGLang/HiCache in LLM serving

In the serving literature, HiCache designates a different system: SGLang’s multi-tier KV-cache subsystem. TENT describes it as extending RadixAttention into a multi-tier KV hierarchy spanning GPU, CPU, and external storage, with TENT integrated as a new backend for SGLang HiCache (Ren et al., 1 Apr 2026). In this setting, HiCache is not a feature predictor; it is a KV management layer that issues large, latency-critical transfers such as remote KVCache block retrieval.

TENT evaluates this stack on SGLang with HiCache, Qwen3-235B-A22B-Instruct-2507, a single 8×H800 node, TP=8, 60 clients, concurrency 4, a multi-turn conversation workload with 2048 input tokens and 10 turns, and a KVCache budget of 600 GB. Relative to HiCache with Mooncake TE, HiCache with TENT improves input throughput from $t_i$3 to $t_i$4 tokens/s, reduces average TTFT from $t_i$5 s to $t_i$6 s, and reduces P90 TTFT from $t_i$7 s to $t_i$8 s. The paper summarizes this as up to $t_i$9 higher throughput and 26% lower P90 TTFT, attributing the gain to faster retrieval of global KVCache blocks (Ren et al., 1 Apr 2026).

A separate systems paper analyzes SGLang/HiCache under a formal ResidentClaim framework. There, HiCache is characterized as “the strongest storage/offload comparator,” with source inspection at commit bbe9c7e showing radix priority, lock/reference state, protected and evictable accounting, GPU/host/storage tiers, prefetch, write-back, load-back, storage success/failure flags, cached-token details, and block events. Yet the same paper concludes that these are strong mechanisms but mostly request-, page-, hash-, token-, or cache-node-scoped rather than claim-outcome-scoped. Its checker yields one positive row—best_effort / telemetry_join / litmus_trace—satisfying claim identity, materialization predicate, claim materialized event, and claim-scoped telemetry via an external adapter, while offloadable remains only an approximation substrate rather than native ResidentClaim conformance (Stepanek, 31 May 2026).

The most common misconception in this serving context is to treat HiCache’s storage tiers and load-back behavior as if they already constituted a formal contract for future KV reuse. The ResidentClaim analysis rejects that reading: storage tiers, priority fields, block events, and telemetry do not by themselves supply claim identity, explicit acceptance, restore-before-reuse semantics, or claim-scoped restoration-failure outcomes (Stepanek, 31 May 2026).

7. Limitations, misconceptions, and research directions

For the diffusion-model HiCache, the paper states several limitations. Its Hermite basis is motivated by empirical Gaussianity of feature derivatives, so a future architecture with strongly non-Gaussian feature derivatives may render Hermite suboptimal. Its practical behavior depends on tuning $k$00, $k$01, and $k$02; overly aggressive intervals or high orders can still degrade quality. The reported evaluations focus on DiT-like models and one patch-based DiT for super-resolution, while classical U-Net latent diffusion models are not extensively studied. The default implementation also uses fixed activation intervals, leaving adaptive scheduling as future work (Feng et al., 23 Aug 2025).

For the SGLang/HiCache usage, the principal limitation is semantic rather than algorithmic. The ResidentClaim analysis states that HiCache provides strong substrate mechanisms and selected adapter positives, not native ResidentClaim conformance. In that framework, it supports adapter-observational best_effort evidence, but not offloadable, hard_protected, expiring, or routed_reuse semantics as formal, claim-scoped guarantees (Stepanek, 31 May 2026).

Across both literatures, terminological ambiguity is itself a recurring source of error. ReCache improves HiCache in the diffusion setting only by replacing hand-crafted scheduling with a learned policy; it does not change how HiCache predicts features (Aliev et al., 4 Jun 2026). TENT improves HiCache in the serving setting only by changing the data-movement backend underneath identical cache policies; it does not redefine HiCache’s cache semantics (Ren et al., 1 Apr 2026). The shared label therefore names distinct objects: a Hermite-based feature-forecasting mechanism in diffusion inference, and a multi-tier KV-cache subsystem in LLM serving.