Patch-wise Feature Cache: Accelerating Neural Inference

Updated 5 April 2026

Patch-wise Feature Cache is a computational strategy that caches spatial patch-level features in deep networks to accelerate inference by reusing stable activations.
It employs adaptive token selection, dynamic schedule search, and polynomial extrapolation to strike a balance between efficiency and output fidelity.
Applications span image synthesis, text-to-image serving, and video diffusion, achieving 2–6× speedups with minimal degradation in quality metrics like FID and PSNR.

A patch-wise feature cache is a computational mechanism designed for the acceleration of neural inference—primarily in diffusion models and vision transformers—by storing and selectively reusing intermediate representations at a spatial patch (or token) level, rather than at the whole-layer or block level. This approach exploits spatial-temporal redundancy in high-dimensional data by avoiding redundant computation for regions where features evolve slowly or remain invariant across model timesteps. Patch-wise feature caching is motivated by empirical findings that local patches often exhibit heterogeneous importance and sensitivity to caching, and that fine-grained reuse can yield more favorable trade-offs between efficiency and output fidelity than coarse-grained strategies.

1. Conceptual Foundations and Motivation

Patch-wise feature caching extends the general principle of feature caching in deep neural inference by focusing on groups of spatially contiguous activations—patches or transformer tokens—within intermediate feature maps. In diffusion transformers (DiTs), each inference step or block typically processes an image as a set of tokens corresponding to image patches. Reuse is realized not for entire layers or blocks, but for particular patches whose feature trajectories are deemed stable or redundant over time (Zou et al., 2024, Cao et al., 19 Dec 2025, Sun et al., 16 Jan 2025, Feng et al., 23 Aug 2025).

This fine-grained paradigm is empirically motivated by observations that errors and temporal drift from feature reuse vary widely across patches, with some regions exhibiting far greater quality degradation (up to 10 $\times$ more) when cached naively (Zou et al., 2024). Identifying less-sensitive or low-motion patches and enabling their reuse—while selectively updating others—achieves higher acceleration and output quality than uniform or block-wise caching.

2. Core Algorithmic Approaches

Patch-wise caching can be implemented with either training-free or learning-based policies, with a variety of selection and update strategies:

Adaptive Token Selection: Methods such as ToCa assign each patch a score based on self-attention influence, cross-attention entropy, caching frequency, and local maxima, then cache those with the lowest scores. This selection is performed at each layer and timestep, with per-layer and per-type ratios controlling granularity (Zou et al., 2024).
Dynamic Schedule Search: ProCache employs a constraint-aware, offline search for optimal per-step binary compute/reuse schedules, then mitigates error drift within cached intervals by selectively recomputing deep blocks and high-importance tokens—identified, for example, by attention-value norms (Cao et al., 19 Dec 2025).
Dual-Threshold and Drift Controllers: X-Slim applies a drift monitor to each patch, with early-warning and critical thresholds. When cumulative drift crosses a threshold, only patches with high relative change are recomputed, while others continue to be reused (Wen et al., 14 Dec 2025).
Mathematical Extrapolation: HiCache applies Hermite polynomials as an optimal basis for feature extrapolation in patch-wise fashion, assuming approximate Gaussianity of finite differences. Patch features are predicted for skipped steps with error-controlled polynomial expansions (Feng et al., 23 Aug 2025).
Selective Caching Guided by Attention and Motion: In video models, approaches such as ProfilingDiT and WorldCache first segment blocks or patches into foreground- vs. background- or saliency-focused regions. Foreground is recomputed in all steps, whereas background regions are cached and periodically refreshed based on inter-step similarity, predicted motion, patch saliency, or phase-aware schedules (Ma et al., 4 Apr 2025, Nawaz et al., 23 Mar 2026).
Learning-Based Caching: In HarmoniCa, a patch-level router is trained via step-wise denoising and an image error proxy, assigning learnable scores for recomputation or reuse at each patch and timestep. The system is trained to minimize final image error for a target cache utilization (Huang et al., 2024).

Representative algorithms maintain per-patch caches (often as hash maps or buffers indexed by request, block, patch), with progressive cache updates, eviction tied to active pipeline status, and inference policies that strategically refresh only the most dynamic or relevant patches.

3. Application Domains and System Architectures

Patch-wise feature caching has been deployed and evaluated in a range of domains:

Diffusion Transformers for Images: Patch-level external caches accelerate DiT sampling, with demonstrated 2–3× speedups and sub-5% degradation in FID or other perceptual metrics (Zou et al., 2024, Cao et al., 19 Dec 2025).
Text-to-Image Serving Systems: PATCHEDSERVE implements patch-wise caching for hybrid-resolution batching, delivering up to 50% per-block skips and 30% higher SLO (Service Level Objective) satisfaction without harming output quality. Caches are realized as per-block GPU-resident hash maps, supporting efficient insertion, lookup, and eviction per patch (Sun et al., 16 Jan 2025).
Video Diffusion and World Models: Methods like ProfilingDiT and WorldCache exploit spatio-temporal redundancy at the patch level and couple caching to saliency, motion, and diffusion phase, preserving visual fidelity during aggressive acceleration (Ma et al., 4 Apr 2025, Nawaz et al., 23 Mar 2026).
Few-Shot Classification and Adaptation: In classification adapters, patch-wise caches store refined intermediate representations, boosting adaptation and accuracy on diverse datasets at zero additional inference cost (Ahmad et al., 13 Dec 2025).
Super-Resolution and Multi-Task Generative Models: HiCache integrates Hermite-based patch-wise extrapolation in multiple domains, with speedups as high as 6.24× and minimal quality loss (Feng et al., 23 Aug 2025).

Architecturally, patch-wise caches typically require a small, spatially organized cache per block or layer, managed with assignment policies (heuristic or learned), drift trackers, and occasional flushing or recomputation upon error threshold crossings.

4. Performance Trade-offs and Error Control

Patch-wise strategies carefully balance distortion accumulation against computational savings. Key findings include:

Heterogeneous Sensitivity: Certain patches, when mis-cached, induce order-of-magnitude increases in output error. Methods avoid caching patches with low temporal redundancy or high propagation error via specific scoring functions or sensitivity estimations (Zou et al., 2024).
Selective Refresh and Partial Compute: Error drift is constrained by periodic selective recomputation (e.g., “refresh” steps in X-Slim and ProCache), inflection-aware gating (Qiu et al., 7 Mar 2025), or by mathematical prediction error bounds (HiCache) (Feng et al., 23 Aug 2025).
Quality/Latency Pareto Frontier: Experimental benchmarks show that, at moderate cache ratios (e.g., 50–70% patches cached), patch-wise methods match or closely approach full-compute baselines in FID, PSNR, and perceptual metrics—while doubling or tripling speed. More aggressive reuse yields larger speedups but sharper degradation, often controlled via learned or adaptive mechanism tuning (Sun et al., 16 Jan 2025, Huang et al., 2024, Feng et al., 23 Aug 2025).

Ablation studies confirm that hybrid schemes (combining block- and patch-level strategies) usually outperform patch- or block-only policies, and that the inclusion of attention, motion, or saliency weighting mechanisms is essential for minimizing quality loss at high acceleration (Wen et al., 14 Dec 2025, Nawaz et al., 23 Mar 2026, Ma et al., 4 Apr 2025).

5. Implementation Details and System Integration

Patch-wise caches are implemented as GPU-resident hash maps or contiguous buffers, with per-patch keys and efficient batch operations (query, fill, insert, evict) (Sun et al., 16 Jan 2025). Indexing is commonly by spatial raster order or unique patch ID, with batch-level operations to reconcile expired, new, and reusable patches across requests and steps.

Scheduling policies can be integrated with latency prediction (e.g., SVR-based throughput models) for request batching and SLO awareness in real-world image and video serving systems (Sun et al., 16 Jan 2025). In large-scale deployments, patch-wise policies naturally support hybrid-resolution batching, memory-aware scheduling, and rapid eviction tied to pipeline activity. Memory overhead is generally modest due to per-patch granularity and on-demand eviction.

6. Empirical Results and Comparative Analysis

A selection of reported results illustrates the trade-offs achieved:

Method/Domain	Speedup×	FID/Quality Δ	Notable Features
PATCHEDSERVE (Sun et al., 16 Jan 2025)	~2.0	Maintains quality	Patch hash map for SLO-optimized serving
ToCa (Zou et al., 2024) (DiT, PixArt)	1.93–2.36	Near-baseline FID	Per-token adaptive caching ratios, per-depth/type
ProCache (Cao et al., 19 Dec 2025)	2.90	FID as baseline	Constraint-aware schedule + selective refresh
X-Slim (Wen et al., 14 Dec 2025) (FLUX.1-dev)	3.38	PSNR drop ~0.7 dB	Dual-threshold push-then-polish pipeline
HiCache (Feng et al., 23 Aug 2025) (FLUX.1-dev)	6.24	+1% to –7.6% ImageReward	Hermite polynomial extrapolation
WorldCache (Nawaz et al., 23 Mar 2026) (Video)	2.1	~0.4% quality loss	Saliency-, motion-, phase-aware thresholds
ProfilingDiT (Ma et al., 4 Apr 2025) (Video)	~2.0	LPIPS/PSNR improved	Foreground-background-aware caching

Patch-wise caches are shown to outperform naïve or coarse-grained layer/block-level caching, especially regarding preservation of spatial detail and fine structure in generated images or videos (Zou et al., 2024, Ma et al., 4 Apr 2025).

7. Limitations, Open Challenges, and Extensions

While patch-wise feature caching demonstrates substantial empirical benefits, several challenges and limitations are noted:

Complexity of Sensitivity Scoring: Accurate quantification of patchwise sensitivity and cacheability may require extra computation or profiling.
Controller and Hyperparameter Tuning: Thresholds, cache ratios, and refresh parameters often require tuning per-domain or model, and may introduce instability if chosen suboptimally (Wen et al., 14 Dec 2025).
Scalability to Arbitrary Architectures: Not all systems expose explicit patch or token decompositions (e.g., mixed convolutional/transformer backbones), complicating generic integration.
Lack of Closed-form Error Bounds in Most Schemes: Extrapolative methods such as HiCache provide error bounds, but most policies use empirical or heuristic drift monitors without formal guarantees (Feng et al., 23 Aug 2025).
Learning Overhead: Learning-based caching (e.g., HarmoniCa) can introduce additional training cost due to teacher-student and image error proxy objectives, though inference remains efficient (Huang et al., 2024).

Possible extensions include integration with multi-modal generative models, adaptive granularity-aware caching (patch/block/token), and further automation of patch sensitivity profiling.

Patch-wise feature cache techniques constitute a critical advance for accelerating deep generative inference with minimal loss of output fidelity, especially as models and deployment settings scale in size and diversity. Their design continues to evolve through principled algorithmic, statistical, and system innovations across domains such as diffusion-based image, video, and few-shot recognition models.