HYPIC: Accelerating Hybrid-Attention LLM Serving with Position-Independent Caching

Abstract: In retrieval augmented generation (RAG) and agentic LLM serving, prompts are assembled from independent segments into long contexts, making the prefill stage dominate the per-request computation cost. To this cost, two directions have emerged in parallel: position-independent caching (PIC) admits KV reuse for non-contiguous segments shared across different requests, while hybrid-attention models reduce computation complexity by replacing most full-attention layers with linear attention. However, they cannot coexist: applying PIC to hybrid-attention models breaks down because per-token KV-cache reuse primitives do not transfer to the per-request recurrent state. In this work, we present Hypic, the first serving system for hybrid-attention LLMs with position-independent caching. For linear-attention layers, we identify the segment-cumulative transition operator as the missing algebraic primitive, and cache it alongside each segment's zero-start end-state, enabling near-exact and constant-time state composition of independently cached segments. For the remaining full-attention layers, existing PIC methods also fail as linear layers do not expose the per-token hidden states for selective recomputation. We show that the most significant attention deviation concentrates at segment boundaries, so recomputing only a small seam window at each boundary suffices to restore cross-segment lookback. Finally, Hypic exploits segment-level self-containment to parallelize cache-miss prefill across instances, turning long cold requests -- a major tail-latency contributor under both prefix caching and prior PIC -- into an accelerable workload. Evaluated across four hybrid-attention models and five workloads, Hypic reduces time-to-first-token (TTFT) by 2.45x on average and improves peak throughput by up to 2.0x over existing systems, while staying within 3.3 points of full-recompute accuracy.

• The paper demonstrates a novel integration of position-independent caching with hybrid-attention LLMs that reduces time-to-first-token (TTFT) by up to 2.45× on average.
• It introduces constant-time state composition and localized recomputation at segment boundaries to efficiently accelerate long-context model inference.
• Experiments confirm significant throughput gains and scalability improvements, achieving up to 8.6× TTFT reduction with minimal loss in task accuracy.

Hypic: Accelerating Hybrid-Attention LLM Serving with Position-Independent Caching

Motivation and Problem Statement

LLM inference for retrieval-augmented generation (RAG) and agentic workflows involves assembling long contexts from independently retrieved or generated segments. This compositional prompt structure exacerbates prefill costs, leading to high time-to-first-token (TTFT) and limiting serving throughput. Two recent advances in LLM serving—position-independent caching (PIC) and hybrid-attention models—address scalability from different axes but have proved fundamentally incompatible.

PIC approaches, highly effective for full-attention Transformers, are incapable of splicing and correcting at the per-token level when linear-attention layers are present, since the recurrent state propagated therein is not individually tokenized or directly amenable to splicing operations. Meanwhile, hybrid-attention models, with the vast majority of layers replaced by various forms of linear attention, have become the de facto production baseline (e.g., Qwen3.5, MiniMax-M1, Kimi-Linear). Hence, serving efficiency for long-context, compositional workloads is bottlenecked by the inability to apply PIC to hybrid stacks.

System Overview

Hypic introduces the first serving system providing efficient, position-independent caching for hybrid-attention LLMs. The system comprises three main components:

• Hypic Router: Splits a request into segments and dispatches cache misses for parallel prefill.
• Hypic Store: Maintains both segment-level public caches (for reusable states) and per-request private caches.
• Hypic Assembler: Composes cached and freshly computed states into the running context for decoding.

This architecture allows Hypic to exploit segment self-containment for parallelism and cache state management efficiently.

Figure 1: Schematic of Hypic architecture, detailing the router, store, and assembler design.

Algorithmic Innovations

Constant-Time State Composition for Linear Attention

For all advanced forms of linear attention in use today, the core algebraic insight is that the segment-cumulative transition operator $T_C$ (a matrix product over per-token transitions) and the zero-start end-state $S_{C|0}$ fully determine the mapping from any input state to the segment's end-state. By caching $(T_C, S_{C|0})$ per segment, Hypic enables exact, constant-time state composition at each segment boundary.

Figure 2: For each segment, the tuple $(T_C, S_{C|0})$ is cached, enabling state composition during prompt assembly.

Given a sequence of $n$ segments, composition is simply:

$$S_{\text{final}} = T_{C_n}\cdots T_{C_1} S_{\text{prefix}} + \sum_{i=1}^n \left(\prod_{j=n}^{i+1} T_{C_j}\right) S_{C_i|0}$$

This operation is independent of segment length, resulting in substantial acceleration for long prompts with many segments.

Boundary-Anchored Alignment in Full Attention

Because hybrid stacks interleave sparse full-attention layers (carrying long-range dependency), naive PIC primitives cannot repair inter-segment context without per-token recurrent states, which are not available. Empirical analysis of attention-score deviations in hybrid-attention models reveals that errors are highly localized at segment boundaries—especially the $w$ tokens at the head and tail.

Figure 3: Attention-score deviation concentrates at segment boundaries, minimal in the interior; only the boundary windows require correction.

Thus, Hypic recomputes only a small seam window of $w$ tokens on each side of the boundary, repairing both the attention sink and cross-segment lookback by recomputing the contiguous window through the cached state.

Figure 4: Schematic of seam window handling—only the last and first $w$ tokens of each segment are excluded from the cache and recomputed.

Segment-Parallel Prefill for Cache Misses

Unlike prior systems constrained by sequential, monolithic prefill, Hypic leverages the algebraic self-containment guarantee of the PIC paradigm. Each cache-miss segment is dispatched to a separate worker for parallel prefill. The combine node collects prefilled segments, assembles the running state, and guarantees decoding can proceed once all segments are processed.

Figure 5: Segment parallelism: only cache-miss segments are distributed to a worker pool, prefilled in parallel, and reassembled at the combine node.

Load-balancing is handled via an LPT (Longest-Processing-Time-first) greedy policy. The design ensures the prefill bottleneck scales as $O(\lceil n/m \rceil \cdot |C| + c)$, rather than $S_{C|0}$0, essentially removing long cold requests as a source of tail latency.

Experimental Results and Empirical Analysis

Evaluation on four hybrid attention models (e.g., Qwen3.5-35B/122B, Ring-mini-linear-2.0) and diverse long-context workloads shows:

• TTFT Reduction: Hypic provides a mean $S_{C|0}$1 reduction in p50 TTFT over prefix cache (the incumbent baseline), and up to $S_{C|0}$2 in the extreme case (Qwen3.5-122B on MultiNews).
• Throughput Gains: Peak throughput improves up to $S_{C|0}$3 compared to existing deployments at the same SLO.
• Cache-Miss Prefill: On cold-only requests, segment parallelism yields a $S_{C|0}$4 TTFT speedup at 8 workers.
• Accuracy Envelope: The average drop in task metric (F1/ROUGE-L) is limited to $S_{C|0}$5 points, and state drift in deep linear attention layers is bounded below 10% relative $S_{C|0}$6.

  Figure 6: Pareto frontier of task accuracy versus TTFT for various models and datasets.

  

  Figure 7: Impact of system load (QPS) on p50 TTFT and per-GPU throughput.

• Scalability: Hypic's composition law achieves $S_{C|0}$7 compute for state assembly and $S_{C|0}$8 scaling with segment count, independent of segment length.

  Figure 8: State composition scaling: TTFT is constant in segment length and grows linearly in segment count.

• Seam Window Sensitivity: Task accuracy saturates for seam width $S_{C|0}$9, with negligible additional gains for wider windows and only a marginal increase in TTFT.

  Figure 9: Task accuracy and TTFT as seam window width $(T_C, S_{C|0})$0 is varied; $(T_C, S_{C|0})$1 is near-optimal.

• Parallelism Breakdown: As worker count increases, TTFT scales nearly inversely with $(T_C, S_{C|0})$2; communication and composition costs remain insignificant.

  Figure 10: TTFT breakdown by subphase as prefill worker count increases; dispatch forward dominates at small $(T_C, S_{C|0})$3, comm and combine costs are flat.

Theoretical and Practical Implications

Hypic's results establish that it is algebraically possible, with practical engineering tradeoffs, to unify PIC and hybrid attention in a production setting. For large-scale RAG, agentic orchestration, and any scenario where sub-contexts are reused or reordered, this removes the main architectural bottleneck. Theoretically, the algebraic insight—that all linear-attention variants' recurrent state transitions can be generalized and composed—may inform both future model designs and caching architectures.

For practitioners, Hypic enables operators to deploy advanced hybrid-attention models without the serving penalty incurred by long prompt compositions. Notably, the architecture is orthogonal to intra-instance parallelism; Hypic's loop-level parallelism in prefill coexists with hardware acceleration and batching inside each worker.

Potential for Future Directions

• Adaptive Seam Width: Dynamic seam sizing based on workload or model introspection could further minimize recompute without compromising alignment.
• Model Co-Design: Training models with Hypic's state-reuse semantics in mind may reduce state drift and improve compositional robustness.
• End-to-End System Integration: Extension of Hypic principles to multi-modal or cross-LLM agentic architectures can generalize to broader multi-context AI deployments.

Conclusion

Hypic provides the first PIC solution compatible with hybrid-attention LLMs, achieving both theoretical precision in linear attention composition and practical speedup in end-to-end serving. It leverages segment-local transition operators, localized window recomputation for full attention, and segment-parallel cold prefill to realize large, compositional prompt serving with controllable computational and accuracy costs. Hypic thus removes a critical obstacle to efficient, scalable long-context LLM inference in production.