Memory Efficient PIC (MEPIC)
- Memory Efficient PIC (MEPIC) is a novel KV-cache system for LLM serving that creates canonical, shareable chunk representations via paged storage.
- It employs block-level recomputation and deferred positional encoding (RoPE at runtime) to reduce redundant computations and cut HBM usage by up to 2×.
- MEPIC integrates into vLLM and LMCache stacks with deterministic padding and a hybrid KV manager to efficiently manage shared HBM resources across requests.
Memory Efficient PIC (MEPIC) is a serving-system design for LLMs that targets a specific limitation of long-context inference: repeated document or code chunks may be reusable at arbitrary prompt positions, yet still remain physically duplicated in high-bandwidth memory (HBM). In the formulation introduced for LLM serving, MEPIC makes chunk-level position-independent caching into an actual memory-sharing mechanism by aligning chunk KV to paged storage, shifting recomputation from token- to block-level so only the first block is request-specific, storing positional-encoding-free KV, and applying Rotary Position Embedding (RoPE) inside the attention kernel at runtime. The resulting system is intended for workloads such as Retrieval-Augmented Generation, coding agents, and deep-research assistants, where the same passages recur across requests and batches (Wang et al., 18 Dec 2025).
1. Conceptual placement within KV-cache reuse methods
MEPIC is defined against two earlier reuse regimes. Prefix caching reuses KV only for identical prefixes, whereas prior position-independent caching (PIC) methods such as EPIC and CacheBlend permit chunk reuse at arbitrary positions but still allow the same logical chunk to diverge across requests because recomputation, positional encoding adjustments, and paged KV placement remain request-specific. The paper’s central distinction is therefore between algorithmic reuse of computation and physical reuse of HBM-resident KV pages (Wang et al., 18 Dec 2025).
| Method | Reuse condition | Consequence for HBM |
|---|---|---|
| Prefix caching | Identical leading tokens | Fails when shared content is reordered or appears in the middle of the prompt |
| Standard PIC | Arbitrary positions with request-specific recomputation and PE adjustment | The same chunk can diverge across requests and page layouts |
| MEPIC | Canonical, page-aligned chunk KV with NoPE storage and block-level recomputation | Chunk KV becomes shareable across positions, requests, and batches |
This distinction matters because long-prompt and multi-turn agent workloads exhibit a skewed reuse pattern. The reported production measurements indicate that a small hot set of passages, repositories, or authoritative sources is repeatedly reused, and that independent single-turn requests can reuse context as often as multi-turn conversations. In that setting, ordinary PIC may reduce prefill compute without preventing HBM blow-up, cache thrashing, and forced offloading or reloading. MEPIC is designed precisely to prevent such duplication by treating chunk KV as a first-class object in the paged KV cache rather than as a transient by-product of request-local execution (Wang et al., 18 Dec 2025).
2. Canonical chunk materialization and shared HBM management
MEPIC is integrated into the vLLM plus LMCache stack and preserves the broad split between a scheduling path and a computation path. Its scheduler partitions each request into two segment classes: immutable or reusable chunk segments and request-specific prompt segments. It then applies deterministic padding asymmetrically: chunk segments receive leading padding, prompt segments receive trailing padding, and the final prompt segment is left unpadded. The stated purpose is to force reusable chunks to begin on block boundaries so that the same logical chunk maps to the same paged KV blocks across requests (Wang et al., 18 Dec 2025).
The scheduling path further resolves HBM residency and remote availability, estimates the KV blocks required by non-resident segments, rejects a request if insufficient free blocks exist and evictable chunk KV cannot satisfy demand, and otherwise assigns paged KV blocks according to residency and segment type. This procedure is the mechanism by which MEPIC converts chunk reuse into deterministic physical layout rather than opportunistic logical matching.
HBM management is coordinated through a Hybrid KV Manager, which jointly manages prefix cache and chunk cache over a shared HBM block pool, together with a Chunk LRU Manager for eviction. The policy is asymmetric: prefix blocks are never evicted, whereas chunk blocks can be evicted when their reference count drops to zero. A plausible implication is that MEPIC treats prefix reuse as latency-critical state while treating chunk reuse as a canonical shared object store whose residency is managed explicitly. That design choice is central to its claim that chunk KV can be shared not only across requests but also across overlapping batches (Wang et al., 18 Dec 2025).
3. Block-level recomputation and position-free KV representation
MEPIC’s second major mechanism is its recomputation policy. Standard PIC methods often select tokens to recompute dynamically, but a single recomputed token can render an entire paged KV block non-shareable. MEPIC therefore moves from token-level to block-level recomputation. The policy is fixed: prompt segments are fully recomputed, newly encountered chunks are fully recomputed, cached chunks recompute only the first KV block, and all remaining blocks are reused as canonical shared chunk KV (Wang et al., 18 Dec 2025).
The paper describes this as a deterministic confinement of the request-specific “dirty” region to the first block. It further states that this is sufficient to capture boundary effects while preserving the rest of the chunk as reusable state. This suggests that MEPIC deliberately sacrifices finer-grained recomputation flexibility in order to stabilize physical page identity and maximize shareability.
The third mechanism removes positional information from stored KV. In standard vLLM-style execution, RoPE is applied before KV is written into cache, which makes cached KV position-specific. MEPIC instead stores NoPE KV: keys and values are cached without pre-applied RoPE, and a fused RoPE-attention kernel applies the appropriate rotary offsets on the fly when KV blocks are loaded into on-device buffers. This separation yields three forms of canonicalization: reuse across positions because KV is stored without RoPE, reuse across requests because chunk segments are padded to stable block boundaries and only the first block is request-specific, and reuse across batches because all later chunk blocks reside canonically in the shared paged HBM pool (Wang et al., 18 Dec 2025).
4. Evaluation, workloads, and measured system behavior
The reported evaluation uses Mistral-7B-Instruct-v0.3 on Ascend 910B NPUs with 64 GB HBM, with CacheBlend and EPIC as baselines, and SQuAD, NewsQA, NarrativeQA, and emrQA as datasets, where paragraphs are treated as chunks. The reuse characteristics span a wide range: mean reusable-token fractions are 61.4% for NewsQA, 84.9% for SQuAD, 93.9% for NarrativeQA, and 98.2% for emrQA. Mean request lengths are reported as roughly 1.4K–2.2K tokens, with only 29–521 tokens requiring recomputation depending on the dataset (Wang et al., 18 Dec 2025).
Accuracy is largely preserved. The reported scores are 0.74 for MEPIC on SQuAD versus 0.73 for CacheBlend and 0.72 for EPIC; 0.65 on NewsQA versus 0.66 and 0.66; 0.60 on NarrativeQA versus 0.61 and 0.61; and 0.39 on emrQA for all three systems. The paper’s interpretation is that recomputing only the first block of cached chunks does not hurt accuracy on these benchmarks (Wang et al., 18 Dec 2025).
Latency is generally competitive or better. End-to-end latency is reported as 116.03 s on SQuAD versus 119.41 s for CacheBlend and 114.73 s for EPIC; 112.36 s on NewsQA versus 117.00 s and 115.39 s; 97.71 s on NarrativeQA versus 104.72 s and 100.06 s; and 105.67 s on emrQA versus 110.97 s and 109.85 s. Under varying QPS, MEPIC is reported to achieve 9.1% lower latency than EPIC and 11.48% lower latency than CacheBlend (Wang et al., 18 Dec 2025).
The strongest gains are in HBM usage. Peak HBM usage is 27.67% on SQuAD versus 54.47% for CacheBlend and 54.13% for EPIC; 36.43% on NewsQA versus 45.97% and 45.43%; 29.67% on NarrativeQA versus 50.40% and 50.40%; and 23.20% on emrQA versus 37.37% and 37.40%. The paper summarizes these results as up to 2× lower HBM usage than state-of-the-art PIC at comparable latency and accuracy. Under varying QPS, it reports 5.74× lower HBM usage than CacheBlend and 5.25× lower HBM usage than EPIC. For long prompts with 2 to 16 chunks, MEPIC uses 2.97× to 5.21× less HBM than the baselines and remains below 40% HBM usage even as context length grows (Wang et al., 18 Dec 2025).
5. Implementation strategy, scope, and trade-offs
MEPIC is implemented by extending vLLM and LMCache rather than changing the model. The chunk cache coordinator resides inside vLLM’s paged KV allocator, LMCache provides remote persistence through CPU or disk storage, chunk identity is derived from the padded token sequence, and the attention kernel is modified so that RoPE is fused at runtime. The design is therefore characterized in the paper as software-only and intended to remain compatible with existing production serving stacks (Wang et al., 18 Dec 2025).
The paper is also explicit about trade-offs. Scheduler complexity increases because prefix and chunk KV must be jointly managed in a shared HBM block pool. Chunk-aware deterministic alignment is required; without the prescribed padding policy, reuse breaks. Selective recomputation is intentionally minimal: recomputing only the first block works well in the reported experiments, but it may ignore some cross-block boundary effects in exchange for much better memory efficiency. RoPE fusion adds some compute in the attention kernel, although the paper states that the overhead is negligible compared with the memory savings. Finally, when reuse is sparse, keeping large chunks resident in HBM provides less benefit, so the design is most effective when repeated shared content dominates the prompt distribution (Wang et al., 18 Dec 2025).
A recurrent misconception is to view MEPIC primarily as a better PIC recomputation heuristic. The paper argues instead that its novelty lies in changing storage and execution so that a chunk’s KV becomes canonical and shareable in a paged-attention engine. In that sense, MEPIC is better understood as a KV-cache memory-system redesign than as a narrowly algorithmic change in prefill reuse policy.
6. Related developments and acronym ambiguity
A closely related later system is MiniPIC, which adopts the same broad principle of a positional-encoding-free KV cache and applies RoPE to cached keys inside attention using per-request logical positions. MiniPIC exposes three token-level primitives—block-aligned padding, SSep, and PDep—and, excluding the modular attention backend, requires only 78 lines of vLLM core changes, of which 61 lines are new functionality. The paper positions it as a deliberately small and flexible realization capable of expressing Block-Attention, EPIC, and Prompt Cache within the same running vLLM instance, while integrating with KV cache CPU offload (Ordonez et al., 11 Jun 2026). This suggests that MEPIC’s canonicalization principles can also be implemented in a lighter-weight, more user-controlled form.
The acronym itself is not unique across arXiv literature. In an unrelated memristive processing-in-memory context, MEPIC denotes an error-correcting-code mechanism for memristive processing-in-memory, organized around diagonal parity, CMEM, processing crossbars, and in-memory syndrome checking rather than KV-cache sharing (Leitersdorf et al., 2021). That usage is conceptually separate from Memory Efficient Position Independent Caching for LLM serving.
Within LLM systems, however, MEPIC denotes a specific answer to the mismatch between logical chunk reuse and physical page reuse. Its defining claim is not merely that repeated chunks can be recomputed less often, but that they can exist as a single canonical HBM-resident representation that is reusable across positions, requests, and batches. In workloads dominated by repeatedly retrieved passages, repositories, or source documents, that distinction is the basis of its reported reductions in HBM usage at comparable latency and accuracy (Wang et al., 18 Dec 2025).