Tangram: Unlocking Non-Uniform KV Cache for Efficient Multi-turn LLM Serving

Abstract: Multi-turn LLM serving is critical for consistent user experiences, yet the linear growth of the Key-Value (KV) cache imposes significant pressure on GPU memory and bandwidth. Non-uniform KV compression effectively preserves more information by considering the individual importance of each KV cache. However, such KV cache heterogeneity introduces various systemic challenges - including memory fragmentation, scheduling complexities, and diminished kernel utilization - which collectively lead to significant inefficiencies in existing LLM serving systems. To overcome these challenges, we present Tangram, a novel serving system designed to make Non-uniform KV caches practical. Tangram addresses systemic inefficiencies through three core techniques: (1) Deterministic Budget Allocation assigns a static memory footprint to each head based on its intrinsic pattern, entirely eliminating dynamic scheduling overhead and prefill stalls; (2) Head Group Page clusters attention heads with similar retention demands and manages them with independent, vectorized page tables, thereby maximizing physical memory reclamation; and (3) Ahead-of-Time (AOT) Load Balancing leverages static budget profiles to ensure uniform GPU utilization without runtime overhead. Experimental results show that Tangram improves throughput by up to 2.6x compared to existing baselines, while fully preserving model accuracy. Our implementation is publicly available at https://github.com/aiha-lab/TANGRAM.

• The paper introduces static headwise retention profiling to pre-allocate non-uniform KV cache budgets, preserving accuracy while reducing memory overhead.
• It details a novel grouping strategy that partitions attention heads for efficient memory reclamation, mitigating fragmentation and dynamic scheduling costs.
• The framework leverages ahead-of-time workload balancing to maximize GPU utilization and throughput under extreme multi-turn serving conditions.

Tangram: Systematic Non-Uniform KV Cache Compression for Scalable Multi-Turn LLM Serving

Problem Formulation and Background

Multi-turn LLM serving necessitates the persistent storage of growing attention-state histories—formally, the Key-Value (KV) cache—for each user session. Unlike single-turn inference, this leads to linear growth in cache size with each conversational turn, rapidly saturating GPU memory and constraining throughput and scaling efficiency. While KV compression is an established mitigation—removing less critical tokens to save memory—uniform allocation across all attention heads is fundamentally mismatched with model-intrinsic retrieval patterns, causing unnecessary information loss for certain heads and degrading accuracy in complex multi-turn settings.

Non-uniform compression, which assigns differing retention lengths to each head based on importance scoring, solves much of the preservation challenge but cannot be practically deployed in high-throughput serving frameworks such as vLLM or SGLang due to three primary architectural mismatches:

1. Page Fragmentation: Unified (monolithic) memory pages in current systems prevent effective reclamation of freed tokens from heads with low retention, causing significant waste.
2. Dynamic Reclamation Bottleneck: Runtime compression decisions require complex, high-overhead page tracking and reclamation, stalling scheduling and prefill efficiency.
3. GPU Workload Imbalance: Heterogeneous per-head retention disrupts thread block (KV-split) assignment, introduces stragglers, and causes severe underutilization in GPU backends optimized for regular workloads.

Empirical Motivation

The paper provides quantitative evidence that headwise retention rates in non-uniform compression are highly stable and model-intrinsic. Fine-grained analysis on models such as Qwen2.5-7B, Qwen3-4B, and LLaMA3.1-8B over diverse benchmarks shows that, while the absolute magnitude of required cache per head shifts across tasks, the relative distribution and ranking among heads is sharply preserved (Figure 1).

Figure 1: Per-head KV retention rates under non-uniform compression are highly stable across models and domains, motivating static group allocation.

This observation enables an offline profiling regime, wherein a model’s headwise retention signature is determined a priori and then fixed per deployment, obviating the need for dynamic runtime adaptation and the associated system costs.

The Tangram Framework: System Components

Deterministic Budget Allocation

The first key technique is replacing runtime, input-dependent cache allocation with static per-head budgets derived from offline profiling (via FastKVzip or similar methods). The main process:

• Calibrate average and variance of per-head retention using a modest pilot set ($n=50$).
• Assign static budgets with a configurable safety margin factor ($\alpha$), ensuring no head is starved for context, preventing performance loss even on out-of-distribution tasks.
• Use these budgets to allocate exactly the required pages per head before execution, eliminating all runtime memory over-provisioning and dynamic compress-and-reclaim cycles.

The static allocation unlocks accurate resource planning and maximum batching aggressiveness, as the upper bound for GPU memory is now known before scheduling.

Figure 2: KV cache grows rapidly with session length, but non-uniform strategies more effectively preserve important information than uniform compression.

Head Group Page Management

To resolve the architectural inefficiency at the memory management layer, Tangram clusters attention heads with similar static retention profiles into Head Groups, each governed by an independent (vectorized) page table rather than a monolithic global one.

• Heads are globally sorted by retention, and partitioned into groups of size $G$.
• Each group’s memory pool is managed independently, permitting precise reclamation whenever short-context heads are heavily compressed.
• The grouping granularity $G$ is tuned to optimize between physical fragmentation (small groups) and host-side page-table management overhead (large groups). Fast SIMD-accelerated table operations via OpenMP and AVX-512 parallelize these management tasks to minimize CPU bottlenecks.

  Figure 3: Comparison between unified and head group paging—smaller groups drastically improve reclamation, but incur higher management cost, mitigated by vectorized tables.

Ahead-of-Time (AOT) Workload Balancing

The final component leverages the fixed workload shape imposed by deterministic budgets to offline all GPU attention kernel scheduling:

• Per-group computational cost is statically calculated and partitioned into thread blocks at model startup, proportional to the cumulative retention of that group.
• No runtime cost for plan generation or partitioning, in contrast to FlashInfer, which incurs per-layer per-step planning overhead in the heterogeneous regime.
• This guarantees high GPU utilization even under severe cache compression rates and avoids both straggler-induced tail latency and control plane latencies.

  Figure 4: Attention kernel latency is minimized with AOT partitioning; planning overhead is eliminated compared to dynamic alternatives.

Experimental Results

Tangram’s evaluation considers both accuracy and system-level metrics under extreme, realistic multi-turn serving conditions (100K+ context, concurrent requests):

• Accuracy: Under aggressive non-uniform compression, Tangram’s deterministic headwise allocations preserve benchmark accuracy at parity with SOTA methods such as KVzip, even as retention rate drops to 20%, outperforming uniformly allocated approaches especially on long-context tasks (Figure 5).
• Throughput: End-to-end throughput increases by up to $2.6\times$ compared to strong vLLM baselines, owing primarily to (i) elimination of all page reclaim overhead (Figure 6), (ii) fine-grained physical memory reclamation (Figure 7), and (iii) AOT-kernel partitioning (Figure 8, 13).
• Latency: Time-to-first-token (TTFT) remains flat under heavy load, showing that deterministic planning prevents the prefill and scheduling backoff that arises with dynamic schemes (Figure 9).
• Trade-off: Optimal head group sizes ($G=4$–$8$) balance fragmentation and table management, universally maximizing effective throughput across tested hardware.

  Figure 5: Tangram’s accuracy remains at parity with SOTA non-uniform methods over broad context scales despite aggressive cache reduction.

  

  Figure 8: Throughput breakdown across benchmarks and methods reveals Tangram’s dominant gains under multi-turn, long-context conditions.

  

  Figure 6: Page reclamation latency is eliminated in Tangram, a major bottleneck in prior systems.

Implications and Future Directions

Tangram’s findings have significant implications for LLM serving design:

• Non-uniform compression is algorithmically essential for long-context LLMs, but only when co-designed with system-level innovations to eliminate the fragmentation, scheduling, and kernel inefficiencies that heterogeneous memory brings.
• Static headwise retention profiles are stable and model-intrinsic, simplifying the control plane and enabling deterministic resource orchestration at all system stack layers.
• Grouping techniques generalize beyond compression: Future system designs for mixtures-of-experts models, hybrid architectures, or compressive memory systems can exploit similar grouping abstractions for partitioned management and scheduling.

Potential future work includes adaptive online adjustment for models that undergo continual pretraining or task-specific finetuning, hierarchical or context-aware clusterings for hybrid models, and integration with hardware that natively supports sub-block-level memory management or SMC parallelism.

Conclusion

Tangram represents a comprehensive co-design for practical, efficient non-uniform KV cache compression in multi-turn LLM serving. By statically profiling headwise retention, clustering heads for grouped memory management, and fully offlining workload partitioning, it translates the theoretical gains of compression into realized system throughput and preserves task accuracy under modern multi-session workloads. These design patterns offer promising avenues for scalable, production-grade deployment of increasingly context-hungry LLM architectures.

Figure 10: System overview of Tangram, highlighting the integration of deterministic allocation, head group paging, and AOT balancing.