BudCache: Caching with Explicit Budget Control

Updated 6 July 2026

BudCache is a domain-dependent caching paradigm that couples explicit resource constraints (e.g., depth, bandwidth, evaluations) with caching strategies for improved efficiency.
It is applied across diverse areas—from KV-Aware LLM routing and GPU DRAM caching to diffusion model acceleration and managed runtime optimizations—tailoring mechanisms to specific system budgets.
Empirical studies report notable gains in performance, energy savings, and resource management through techniques like decode-time rerouting, adaptive tag caching, and offline policy search.

Searching arXiv for the provided BudCache-related papers and terminology. arxiv_search(query="BudCache", max_results=10) BudCache is used in recent arXiv literature for several distinct mechanisms that couple caching with explicit resource constraints. In LLM inference it corresponds to the KV-Aware Planner within BUDDY, which reuses the first-layer KV cache to provide a lightweight, global context signal for layer-routing decisions under a strict budget (Zhou et al., 8 Jun 2026). In GPU memory systems it denotes a bandwidth-effective DRAM cache for Storage-Class Memory with SCM-aware bypass, a Configurable Tag Cache, and an Aggregated Metadata-In-Last-column organization (Hong et al., 2024). In diffusion inference it denotes a budget-constrained step-level caching method that fixes the number of fresh evaluations in advance and searches offline for a cache policy that best preserves final output quality (Lei et al., 11 Jun 2026). In managed runtimes, a BudCache interpretation is a Sache-inspired, memory-bounded software cache built from prioritized garbage collection primitives (Nunez et al., 2016). A related cache-centric interpretation aligns BudCache with CCache-style buffered commutative updates (Balaji et al., 2017).

1. Terminological scope

The term is used for several distinct mechanisms in the supplied literature. The commonality is not a single implementation, but the coupling of caching with an explicit control variable such as depth budget, bandwidth efficiency, heap bound, or number of function evaluations.

Domain	Meaning of BudCache	Source
LLM inference	KV-Aware Planner in BUDDY; reuse of the first-layer KV cache for budget-driven dynamic depth routing	(Zhou et al., 8 Jun 2026)
GPU memory hierarchy	Bandwidth-effective DRAM cache for GPUs with SCM, integrating bypass, CTC, and AMIL	(Hong et al., 2024)
Diffusion inference	Budget-Constrained Step-Level Diffusion Caching with fixed NFE budget and offline search	(Lei et al., 11 Jun 2026)
Managed runtimes	Sache-inspired, memory-bounded software cache using prioritized garbage collection	(Nunez et al., 2016)
Parallel shared-memory systems	Related interpretation based on CCache-style buffered commutative updates	(Balaji et al., 2017)

This distribution of meanings indicates that “BudCache” is a domain-dependent label rather than a standardized architecture. The term is most tightly defined in the GPU and diffusion papers, while the LLM and managed-runtime interpretations are explicit mappings supplied in the source material.

2. BudCache as the KV-Aware Planner in budget-driven LLM depth routing

In "BUDDY: BUdget-Driven DYnamic Depth Routing for Adaptive LLM Inference" (Zhou et al., 8 Jun 2026), the paper does not use the name “BudCache.” The mechanism aligned with that name is explicitly called the “KV-Aware Planner”: reuse of the first-layer KV cache to provide a lightweight, global context signal for the routing decision at each decode step. Its role is to inject global information into the Decision Module at every decoding step without substantial overhead by reusing the first-layer KV cache, pooling that global context, and concatenating it with the newest token representation before scoring candidate layers.

The Decision Module is a lightweight two-layer MLP, Linear–SiLU–Linear, operating over candidate middle layers $B_{\mathrm{mid}}=\{B_2,\dots,B_{L-1}\}$ , with the first and last layers always executed. Let $G_t \in \{K_t, V_t, K_t+V_t\}$ denote the first-layer cache-derived context features at decoding step $t$ . The router pools token representations across the prefix to obtain $c_t = \mathrm{Pool}(G_{1:t})$ , concatenates this with the newest token features $h_t^{\mathrm{new}}$ to form $z_t = \mathrm{Concat}(c_t, h_t^{\mathrm{new}})$ , and scores layers with an MLP. Optional prior fusion uses

$s' = s + \omega p,$

with $\omega=0.1$ as the default balance coefficient and normalized offline priors. Budget enforcement is deterministic:

$P(s', b)=\mathrm{Top}\text{-}K(\mathrm{Softmax}(s'),\, k=\mathrm{round}(b\cdot L)-2),$

yielding a binary mask over middle layers, while residual alignment is preserved by executing $H_l = F_l(H_{l-1}) + H_{l-1}$ if $G_t \in \{K_t, V_t, K_t+V_t\}$ 0 and skipping with $G_t \in \{K_t, V_t, K_t+V_t\}$ 1 if $G_t \in \{K_t, V_t, K_t+V_t\}$ 2.

The distinctive feature of this BudCache interpretation is decode-time rerouting. During prefill, the Decision Module computes layer scores using pooled first-layer cache features of the full prompt and instantiates an initial mask. During decode, the first-layer cache is refreshed with the newest token, the pooled global context changes, scores are recomputed, and the layer-execution mask is updated. The paper reports that path recomputation improves ROUGE over reusing the prefill path consistently on SAMSum, and that the number of distinct paths grows with sparsity. The same framework supports strict user budgets, since $G_t \in \{K_t, V_t, K_t+V_t\}$ 3 exactly fixes the number of middle layers executed at each step, and it also supports the no-explicit-budget case through a Budget Predictor trained with GRPO over the discrete action space $G_t \in \{K_t, V_t, K_t+V_t\}$ 4.

Empirically, the method is competitive with static pruning baselines and often improves the accuracy–compute trade-off while uniquely supporting strict budget control, decode-time rerouting, and multiple budgets within a single trained model. On Llama3-8B, BUDDY reaches average 67.40 at 12.5% removal versus dense 67.44, 61.24 at 25% removal, 54.40 at 37.5% removal, and 48.11 at 50% removal; on Qwen2.5-7B-Instruct it reaches 65.63 at 14.3% removal versus dense 69.42, 57.22 at 28.6%, 50.30 at 42.9%, and 45.11 at 57.1%. Router overhead is reported as about 0.00285% for $G_t \in \{K_t, V_t, K_t+V_t\}$ 5 to 0.00498% for $G_t \in \{K_t, V_t, K_t+V_t\}$ 6 of backbone FLOPs in a Llama-7B-like configuration, while A100 latency breakdown attributes 79.50% of runtime to the backbone, 5.67% to the Decision Module, and 3.28% to gather/scatter. Ablations show that value states worked best at light-to-moderate sparsity, keys were slightly better at extreme sparsity, and the first-layer KV cache was best or most robust at low/moderate sparsity, with layer-3 slightly better at extreme sparsity.

3. BudCache as a bandwidth-effective DRAM cache for GPU–SCM hierarchies

In "Bandwidth-Effective DRAM Cache for GPUs with Storage-Class Memory" (Hong et al., 2024), BudCache is a bandwidth-effective DRAM cache architecture tailored for GPUs that use Storage-Class Memory to overcome the GPU memory capacity wall while preserving high effective bandwidth. It integrates an SCM-aware DRAM cache bypass policy, a Configurable Tag Cache in the GPU L2, and an Aggregated Metadata-In-Last-column DRAM-cache organization within a Heterogeneous Memory Stack that stacks SCM dies above DRAM dies and shares the same memory channels through TSVs. The design objective is explicitly bandwidth-effective service under GPU multithreading, where naive DRAM caches often hurt effective bandwidth because 100,000s of concurrent threads can thrash the DRAM cache and generate heavy tag-probe traffic.

The Heterogeneous Memory Stack places one DRAM-cache rank and one SCM rank on each channel. The DRAM cache is hardware-managed and not programmer-visible; SCM provides the expanded addressable memory. With 4× SCM density relative to DRAM, HMS delivers roughly 2× the addressable capacity of an equivalent HBM stack. On an L2 miss, the request first consults the CTC. A CTC hit resolves the DRAM-cache hit or miss without external traffic; a CTC miss triggers AMIL, which fetches all tags for a row in one DRAM column access and fills the CTC. On a DRAM-cache miss, the demand access is served from SCM immediately, after which the bypass policy decides whether the line should be installed in DRAM.

The bypass policy quantifies the utility of caching through a single performance-utility score that blends spatial locality, hotness, and write intensity. Its core latency discriminator is the SCM penalty score,

$G_t \in \{K_t, V_t, K_t+V_t\}$ 7

For reads only, the numerator is approximated by $G_t \in \{K_t, V_t, K_t+V_t\}$ 8; for reads plus writes it becomes $G_t \in \{K_t, V_t, K_t+V_t\}$ 9. A two-level decision algorithm then compares the current request’s discretized score to a moving-average SCM penalty level and, only if warranted, probes victim metadata to decide replacement. Most bypasses, more than 88%, are decided in Level 1, significantly reducing DRAM probe traffic.

CTC repurposes part of the L2 as a tag cache, with adjustable partitioning by workload. A single L2 way is subdivided into four 32 B Tag Cache ways, and each 32 B CTC line is split into eight 4 B sectors, each sector caching the tags for an entire DRAM-cache row. AMIL stores all tags and per-line metadata for a DRAM-cache row in the last column’s 32 B data field, so one column fetch retrieves all tags on a CTC miss. This organization preserves full ECC coverage for data and tags, unlike Tag-And-Data organizations that repurpose ECC bits.

The reported results are substantial. Compared to oversubscribed HBM, BudCache improves performance by up to 12.5× and 2.9× overall, while reducing energy by up to 89.3% and 48.1% overall. It reduces DRAM cache probe and SCM write traffic by 91–93% and 57–75%, respectively, compared to prior works. CTC achieves about 91% tag-cache hit rates overall, with a minimum of 59%, and reduces traffic overhead relative to an infinite-capacity HBM baseline from 2.93× to 2.45×. BudCache remains within 11.3% of InfHBM on average. Hardware overhead is reported as about 1.46 mm², or about 0.18% of die area for an A100-class GPU with 40 channels. The architecture also includes SCM throttling by stretching timing parameters such as $t$ 0 and $t$ 1 to keep stack power below HBM maximums, and SLC/MLC/TLC mode adaptation to trade performance against capacity as footprints vary.

4. BudCache as budget-constrained step-level diffusion caching

In "Budget-Constrained Step-Level Diffusion Caching" (Lei et al., 11 Jun 2026), BudCache is a method for accelerating diffusion and flow samplers by fixing a compute budget in advance and searching offline for the cache policy that best preserves final output quality under that budget. The central departure from threshold-based heuristics is explicit inversion of the control problem: rather than allowing per-step thresholds to determine runtime cost, BudCache fixes the number of fresh evaluations, and therefore the latency, before inference begins.

The method treats the model’s step-level prediction as the cacheable quantity: $t$ 2 in DDPM/DDIM-style samplers and $t$ 3 in PF-ODE or flow samplers. Let $t$ 4 be a binary mask over $t$ 5 logical steps, with $t$ 6 meaning fresh evaluation and $t$ 7 meaning reuse. If $t$ 8 is the index of the latest fresh evaluation, the Euler-style update becomes

$t$ 9

The budget-constrained objective is

$c_t = \mathrm{Pool}(G_{1:t})$ 0

where $c_t = \mathrm{Pool}(G_{1:t})$ 1 and $c_t = \mathrm{Pool}(G_{1:t})$ 2 is the full-compute teacher output.

Offline search combines Simulated Annealing with deterministic Hill Climbing. Simulated Annealing operates over the state space $c_t = \mathrm{Pool}(G_{1:t})$ 3 using swap and shift moves that preserve the budget exactly, with acceptance probability $c_t = \mathrm{Pool}(G_{1:t})$ 4 and exponential cooling. Deterministic Hill Climbing then exhaustively evaluates one-move neighbors and greedily moves to the best improvement until convergence. Under tight budgets, BudCache further introduces cache-aware schedule alignment, which fixes the searched mask $c_t = \mathrm{Pool}(G_{1:t})$ 5 and optimizes the continuous time discretization $c_t = \mathrm{Pool}(G_{1:t})$ 6 to reduce cache-induced trajectory mismatch.

Inference-time behavior is deliberately simple. The searched mask $c_t = \mathrm{Pool}(G_{1:t})$ 7 and, optionally, aligned schedule $c_t = \mathrm{Pool}(G_{1:t})$ 8 are fixed, so the number of fresh evaluations is exactly $c_t = \mathrm{Pool}(G_{1:t})$ 9 for all inputs. There is no runtime thresholding and no online search; the runtime logic is only “if $h_t^{\mathrm{new}}$ 0 then evaluate, else reuse cached prediction.” Storage overhead is negligible because the policy stores only $h_t^{\mathrm{new}}$ 1 bits and, at most, $h_t^{\mathrm{new}}$ 2 floats.

The empirical results emphasize matched-budget quality. On FLUX.1-dev at 1024×1024 with a 28-step teacher, BudCache-base at 10 NFE and 2.51× speedup reports LPIPS 0.1759, SSIM 0.8374, PSNR 24.90, ImageReward 0.9782, CLIP(L) 27.584, and HPSv2.1 0.2940; BudCache-fast at 9 NFE and 2.74× reports LPIPS 0.2020, SSIM 0.8201, and PSNR 23.59; BudCache-ultra at 8 NFE and 3.00× reports LPIPS 0.2479, SSIM 0.7917, and PSNR 22.16. Compared to MagCache at 2.74×, BudCache-fast yields lower LPIPS, 0.2020 versus 0.2498, and higher SSIM and PSNR. On Wan2.1-T2V-1.3B, BudCache reaches 82 s latency with PSNR 25.93, SSIM 0.8502, and LPIPS 0.1167, versus TeaCache at 100 s with PSNR 21.52, SSIM 0.7494, and LPIPS 0.1929. The paper also reports robust transfer across samplers, resolutions, CFG settings, and model sizes, and notes that policy search can finish within minutes on one H100, with a 9 NFE R1 SA20 search taking about 8m 55s.

5. BudCache as a memory-bounded software cache under prioritized garbage collection

In the managed-runtime interpretation grounded in "Prioritized Garbage Collection: Explicit GC Support for Software Caches" (Nunez et al., 2016), BudCache is essentially a Sache-inspired cache that uses prioritized garbage collection to enforce a memory budget with explicit policy and deterministic ordering. The underlying problem is that software caching in managed languages trades space for time in a way that directly affects garbage-collection cost, while conventional entry-count provisioning is brittle when values have heterogeneous sizes and soft references provide limited policy control.

The core runtime abstractions are PrioReference<T> and PrioSpace<T>. A priority reference holds a referent and an integer priority; the collector may clear these references to enforce a per-space memory bound, but only after all lower-priority references are cleared. Application code can set and query priorities, and can query the GC-computed size of the referent’s reachable subgraph when available. A PrioSpace provides an explicit target bound, either as bytes or as a fraction of used or free heap. During GC, the collector marks high-priority referents first until the budget is reached; remaining lower-priority references are cleared and their objects reclaimed by the sweeper.

For cached objects $h_t^{\mathrm{new}}$ 3 with sizes $h_t^{\mathrm{new}}$ 4 and priorities $h_t^{\mathrm{new}}$ 5, the enforced budget is

$h_t^{\mathrm{new}}$ 6

The collector sorts references in non-increasing priority order, traces referents in that order, and stops bounded marking once the running footprint reaches the space bound. LRU behavior is obtained by setting $h_t^{\mathrm{new}}$ 7 on access or insertion. The runtime also supports adaptive reserve computation with

$h_t^{\mathrm{new}}$ 8

where $h_t^{\mathrm{new}}$ 9 is heap size, $z_t = \mathrm{Concat}(c_t, h_t^{\mathrm{new}})$ 0 is live size not in the cache, and $z_t = \mathrm{Concat}(c_t, h_t^{\mathrm{new}})$ 1 is a free-memory reserve. This allows the cache to grow and shrink automatically with memory pressure.

This BudCache interpretation differs from Java soft references in two ways emphasized by the source material. First, clearing policy is explicit, per-space, and deterministic up to the specified priority order, rather than VM-specific and often globally ordered. Second, size accounting is integrated with GC traversal, so the cache can be bounded by true reachable footprint rather than entry count or programmer-supplied weigh functions. The source material further notes that eviction outside GC can create transient double residency, whereas prioritized GC reclaims only when memory is actually recovered.

The evaluation in the source paper reports that a single budget such as 40% of heap yields good performance across workloads, whereas a fixed entry capacity requires retuning. In a multi-cache experiment under HotSpot soft references, when two caches have a 10:1 access ratio, the slower cache achieves only about 25% of its potential hit rate, a 75% drop; with separate PrioSpaces, the hit rates of two Saches at 20% heap each remain within about 5% of each other. Under growing non-cache memory pressure, a Guava cache crashes when pressure exceeds about 77 MB in a 115 MB heap, whereas a Sache with adaptive budget targeting a 50% reserve does not crash and keeps GC time flat. The same source notes that large cached entries can make boundary-only eviction overshoot the budget temporarily, that fragmentation is partly orthogonal to the approach, and that full-heap enforcement points are especially relevant because cache-heavy applications are effectively non-generational.

6. BudCache as buffered commutative updates in caches

A related interpretation maps BudCache to the mechanisms of "Flexible Support for Fast Parallel Commutative Updates" (Balaji et al., 2017). Here the cache is not a budgeted store or a step-reuse policy, but a vehicle for on-demand privatization of data manipulated by commutative operations. Each core dynamically privatizes commutatively manipulated data, operates on a local copy, and later merges the local result into memory. The key property is that the updated copies remain in existing L1 cache lines rather than in duplicated heap structures, so the system avoids additional memory for private copies while also reducing coherence traffic and synchronization.

CCache introduces a small per-core source buffer, per-line metadata bits, merge registers, and a Merge Function Register File. The first commutative access to a cache line saves a source copy in the source buffer and uses the L1 line itself as the updated copy. Software accesses the data through c_read and c_write, while merge functions consume three 64 B values—source, updated, and memory—and apply an application-defined commutative merge such as memory := memory + (updated - source) or bitwise OR. soft_merge marks currently privatized lines as mergeable, enabling merge-on-evict, and merge serializes per-line updates by locking the corresponding LLC line during the merge.

The correctness model requires commutative updates, with associativity making the final value independent of merge order. CData lines do not initiate or respond to coherence messages while privatized; coherence is reestablished at merge points. The paper reports optimizations such as merge-on-evict and dirty-merge, which reduce source-buffer evictions by 2.2× in BFS and 409.9× in K-means, and reduce merges by 24× in PageRank’s read-mostly phases.

The quantitative results are application-level rather than cache-capacity benchmarks. Across an 8-core system, CCache achieves speedups up to 3.2× versus fine-grained locking and up to 10× versus static duplication, while also showing fewer L3 misses, fewer invalidations, or fewer directory messages across key–value store, K-means, BFS, and PageRank. This related interpretation suggests a BudCache meaning centered on cache-resident buffering of commutative updates rather than on replacement policy or memory-tiering. More broadly, the various BudCache usages indicate a recurring systems pattern: caching becomes most consequential when paired with an explicit control variable—depth, bandwidth, heap footprint, or merge boundary—rather than treated as an implicit optimization.