CodeComp: Structural KV Cache Compression for Agentic Coding

Abstract: Agentic code tasks such as fault localization and patch generation require processing long codebases under tight memory constraints, where the Key-Value (KV) cache becomes the primary inference bottleneck. Existing compression methods rely exclusively on attention signals to estimate token importance, systematically discarding structurally critical tokens such as call sites, branch conditions, and assignments that are essential for code understanding. We present CodeComp, a training-free KV cache compression framework that incorporates static program analysis into LLM inference via Code Property Graph priors extracted by Joern. Across bug localization and code generation benchmarks, CodeComp consistently outperforms attention-only compression baselines under equal memory budgets, recovering the majority of full-context accuracy under aggressive KV cache compression, while matching the patch generation quality of uncompressed full-context inference and integrating seamlessly into SGLang-based agentic coding pipelines without model modification.

• The paper presents CodeComp, a framework that leverages Code Property Graphs to explicitly preserve critical code tokens during compression.
• It employs a dual-stage approach with structure-aware chunk allocation and span-level protection to maintain essential program elements like function calls and control flows.
• Experimental results show significant improvements in bug localization accuracy and code generation fidelity compared to attention-only methods, even at aggressive compression levels.

CodeComp: Structure-Aware KV Cache Compression for Agentic Coding

Motivation and Problem Formulation

Agentic code tasks such as bug localization and patch generation impose stringent requirements on LLMs due to the need for long-context reasoning over entire codebases. The primary computational bottleneck in these scenarios is the Key-Value (KV) cache, which grows linearly with context length and can consume the majority of inference-time GPU memory. Existing state-of-the-art KV cache compression approaches—including token-level eviction (e.g., StreamingLLM, H2O, SnapKV) and quantization (e.g., KIVI, KVQuant), as well as chunk-level strategies (e.g., ChunkKV, ParallelComp)—predominantly rely on attention statistics to estimate token importance for retention.

However, applying attention-centric heuristics to source code is fundamentally flawed. Attention signals serve as proxies for semantic relevance in natural language, but in source code, true semantic significance is dictated by program structure (e.g., control/data dependencies, call relationships), which attention scores often fail to capture.

Figure 1: Attention-only compression discards structurally critical tokens such as function calls, branch predicates, and return statements. CodeComp uses CPG priors to explicitly preserve these tokens.

Empirical analyses demonstrate a marked misalignment between attention-based importance and structural importance, resulting in systematic mis-pruning of semantically critical tokens during compression. Specifically, critical program elements such as function callsites, control predicates, and return values are frequently evicted under attention-based strategies (as visualized in Figure 2).

Figure 2: Disparities between attention and structure, with attention-based approaches almost orthogonal to CPG-derived structural importance.

This structural blind spot significantly degrades LLM performance on code-centric tasks under memory constraints, indicating the necessity of an explicitly structure-aware compression methodology.

The CodeComp Framework

CodeComp introduces a fundamentally distinct, training-free approach for KV cache compression by directly incorporating static program analysis via Code Property Graphs (CPG), as extracted with Joern. The compression pipeline consists of the following steps:

Figure 3: Overview of CodeComp: context retrieval; static analysis for extracting structural anchors (e.g., function calls, control-flow predicates); structure-aware chunk budget allocation and span protection; attention-based compression within structure-enforced constraints.

1. Query-Conditioned Context Selection: Initial retrieval of relevant code chunks based on perplexity-scored query conditioning, ensuring context is semantically aligned to the user request.
2. Structural Feature Extraction: Each chunk's CPG is used to extract semantically meaningful program elements (function calls, control structures, return statements, assignments) with normalized structural statistics.
3. Structure-Aware Budget Allocation: Compression budgets are modulated for each chunk in proportion to its structural score, guaranteeing that structurally significant code receives higher retention capacity.
4. Span-Level Structural Protection: Within each chunk, contiguous spans corresponding to critical CPG-identified events are assigned hard retention—these are preserved irrespective of attention scores.
5. Residual Attention-Based Compression: Remaining budget is filled using standard attention-driven selection for non-structural tokens, enabling fine-grained adaptive compression.

This dual-tiered approach (chunk-level allocation, span-level protection) induces a two-stage prioritization: structural anchors are never evicted, while non-critical tokens undergo lossy selection based on attention dynamics.

Experimental Results

Comprehensive experiments and ablations are performed across representative bug localization (InfiniteBench-CodeDebug, DebugBench, LongCodeQA) and code generation (SWE-bench Lite, LCA CodeGen) benchmarks, comparing CodeComp to established attention-centric methods (ParallelComp, SnapKV).

Bug Localization Accuracy

Consistently, CodeComp outperforms baselines under identical retention ratios, despite aggressive compression (as low as 40% capacity):

• On DebugBench (Llama3-8B, cap=0.4): CodeComp achieves accuracy of 0.43 vs. ParallelComp's 0.03 (14× improvement).
• Structure Score (fraction of retained critical tokens): CodeComp maintains 0.54 vs. 0.35 (ParallelComp) at cap=0.4, evidencing explicit retention of structural evidence.
• Pareto frontier: CodeComp achieves near full-context accuracy at sharply reduced memory budgets.

Code Generation Robustness

• On SWE-bench Lite, CodeComp recovers majority of ground-truth file-level F1 lost by ParallelComp under strong compression, and achieves near-baseline patch generation fidelity (edit distance: 0.743 at cap=0.4 vs. uncompressed 0.740).
• On LCA CodeGen, API-level precision, recall, and F1 consistently exceed ParallelComp and SnapKV at equal budgets, with lower generation edit distance.

Ablation Study and Mechanistic Insights

Ablation experiments confirm that:

Figure 4: Feature ablation of span scoring; removing call or control-flow as features consistently degrades performance.

• Span-level preservation, anchoring on CPG-derived events, is the principal source of performance gain. Removing structural features (calls, control, returns) sharply reduces accuracy, highlighting their indispensability.
• Chunk-level allocation provides secondary complementary improvements, but is not sufficient alone to prevent semantic degradation.
• Throughput analysis demonstrates that integration of static program analysis incurs negligible inference overhead, supporting deployment within high-throughput SGLang-based agent pipelines.

Practical and Theoretical Implications

CodeComp demonstrates that structure-guided KV cache compression fundamentally resolves the semantic mismatch of attention-based token retention in code-centric LLM workflows. By jointly leveraging static program analysis at inference, it bridges the gap between compiler-inspired reasoning and deep model deployment without model modification or retraining.

Pragmatically, this enables scalable, accurate agentic coding pipelines by alleviating GPU memory bottlenecks, preserving the critical evidence needed for fault localization, patch synthesis, and cross-file reasoning within repository-scale contexts. The structure score metric acts as a reliable proxy for downstream utility and is amenable to future model-agnostic compression research.

Theoretically, the results motivate broader research on hybrid symbolic–neural inference optimization, particularly structure-aware context management, semantic compression, and long-context reasoning in domains rich in formal structure.

Conclusion

CodeComp establishes structure-aware compression as a necessary paradigm for code LLM inference under tight memory constraints. By integrating CPG-guided chunk allocation and span-level protection, it maintains task-critical evidence, yielding strong empirical gains over attention-centric methods in both bug localization and code generation tasks. The framework requires no retraining and integrates efficiently with high-throughput pipelines, laying the groundwork for future structure-aware inference and compression systems in software engineering and program synthesis tasks.