SWEzze: Lightweight Code Compression

• SWEzze is a context compression model that retains essential code segments, reducing computational expense and distracting noise in LLM-based bug fixing.
• It combines Oracle-guided Code Distillation with a fine-tuned Transformer reranker to identify minimal sufficient context for successful patch synthesis.
• Performance evaluations show a 6x compression ratio, 51–71% token reduction, and up to 9% improvement in patch resolution rates across LLM pipelines.

SWEzze is a lightweight context compression model for LLM-based issue resolution pipelines, addressing the dual challenges of computational overhead and degraded effectiveness arising from over-approximated code context windows. By combining Oracle-guided Code Distillation (OCD) for training data generation with a fine-tuned Transformer-based reranker, SWEzze enables the retention of only those code segments (“fix ingredients”) minimally sufficient for patch synthesis, thus improving both efficiency and LLM success rates in code repair and bug-fixing tasks (Jia et al., 30 Mar 2026).

1. Motivation: Code Context Compression in LLM Issue Resolution

Recent advances in LLMs have enabled practical resolution of real-world software issues (e.g., via Agentless or SWE-Agent workflows), where a typical pipeline retrieves an over-approximate set of files, functions, and code blocks related to a GitHub issue. Retrieval is deliberately broad—often selecting the top-$k$ elements via retrieval augmented generation (RAG) or heuristics—to avoid missing dependencies. This “maximal context” paradigm leads to hundreds of lines of code as model input, which incurs two principal costs: linear growth in computational expense with respect to context length, and empirical reductions in patch effectiveness as noise in the context window distracts the model from relevant bug-fix signals (see Shi et al., 2023; Liu et al., 2024).

Conventional compressors either degrade code’s semantic integrity by treating it as plain text, or apply superficial code-similarity heuristics that risk omitting essential but non-obvious fix ingredients. SWEzze aims to compress code context by learning, from oracle-generated data, to identify only those segments actually required for a successful fix.

2. Oracle-guided Code Distillation (OCD): Dataset Construction

OCD is a search-based algorithm that constructs for each issue a minimal sufficient code context. Starting from an initial candidate set $\mathcal{C}_0$, OCD reduces it to a “1-minimal sufficient context” $\hat{C}$: a smallest subset such that an LLM, when provided the issue, fault location, and $\hat{C}$, can synthesize a patch that passes all validation tests—removing any further segment causes patch validation to fail.

OCD represents the search space hierarchically at three granularity levels: file, function/method, and statement-level block. Leaf units (segments) may be omitted, replaced by placeholders (“# … N lines omitted”) to maintain syntactic validity.

Each segment $u$ receives a priority score:

$$\Phi(u) = w_p \cdot I(u \in \mathcal{F}_\text{patch}) + w_c \cdot \log(1 + |\mathcal{L}_\text{cov}(u)|) + w_s \cdot \text{SymScore}(u, P_\text{gold})$$

where $I(\cdot)$ indicates whether $u$ belongs to files changed by the patch, $\mathcal{L}_\text{cov}(u)$ is the set of test-covered lines in $u$, and $\mathcal{C}_0$0 measures lexical overlap with gold patch identifiers.

OCD proceeds in two phases:

1. Genetic Search (GA): Candidate contexts are represented as binary genomes, with search operators (tournament selection, crossover at file level, bit-flip mutation) and a fitness function driven by pass/fail or summed $\mathcal{C}_0$1. The process stops upon finding any sufficient context.
2. Hierarchical Delta Debugging (HDD): The minimality criterion is enforced by iteratively attempting to remove low-priority units at each granularity (file, function, block), provided removal does not break patch validation.

This produces (issue, fault, uncompressed context, 1-minimal sufficient context) tuples for downstream model training.

3. SWEzze Model Architecture and Training

SWEzze is trained using the OCD-generated datasets to score and filter context at inference time. For each (issue $\mathcal{C}_0$2, fault $\mathcal{C}_0$3, candidate segment $\mathcal{C}_0$4), pointwise training examples are created with $\mathcal{C}_0$5 if $\mathcal{C}_0$6, else $\mathcal{C}_0$7. The input query $\mathcal{C}_0$8 encodes $\mathcal{C}_0$9, and the goal is to predict, for each $\hat{C}$0, its retention label.

Architecture and training details:

• Base model: Qwen3-Reranker-0.6B, a Transformer cross-encoder.
• Adaptation: LoRA with rank $\hat{C}$1, $\hat{C}$2 on attention projections.
• Prediction: $\hat{C}$3, interpreted as the retention probability.
• Objective: Weighted binary cross-entropy loss,

$\hat{C}$4

where weights $\hat{C}$5 correct for a severe positive-to-negative class imbalance ($\hat{C}$6), and encode semantic-role importance.

• Hyperparameters: 3 epochs, effective batch 16, AdamW learning rate $\hat{C}$7, 10% warmup, weight decay $\hat{C}$8, gradient clip $\hat{C}$9, mixed precision ($\hat{C}$0), checkpointing.

At inference, the uncompressed context is decomposed into segments, scored, and the top segments are greedily selected up to a preset token limit, with omitted segments replaced by placeholders.

4. Performance Evaluation and Baseline Comparisons

Evaluation is conducted on SWE-bench Verified (500 issues), measuring absolute resolution rates (fraction of instances with test-passing patches), compression rates (input token contraction), total token counts (prompt plus completion), and compression latency per instance.

Key results across three downstream LLMs (DeepSeek-V3.2, Qwen3-Coder-Next, GPT-5.2):

• Stable compression ratio: $\hat{C}$1 (6.03, 5.95, 6.55).
• Token budget reduction: $\hat{C}$2 to $\hat{C}$3 relative to uncompressed baselines.
• Resolution rate improvement: $\hat{C}$4–$\hat{C}$5 absolute, $\hat{C}$6 average (e.g., DeepSeek-V3.2 from 47.8% to 52.2%).
• Compression latency: 6.1 s/instance ($\hat{C}$7 faster than LongCodeZip’s 24.4 s; slower than token-level compressors at $\hat{C}$8 s).

Comparison with state-of-the-art compressors:

Compressor	Compression Ratio	Resolution Gains	Latency
LLMLingua-2	$\hat{C}$9	Small/Negative	Low
LongCodeZip	$u$0–$u$1	Modest/Variable	High
SWE-Pruner	Up to $u$2	Often Negative	Moderate
SWEzze	$u$3	Highest	Low-Medium

SWEzze consistently achieves a balance of moderate compression, high resolution rate, and low-to-medium latency.

5. Design Principles and Deployment Recommendations

SWEzze distinguishes itself by prioritizing segment sufficiency (“distilling for sufficiency”), in contrast to traditional methods (“pruning for similarity”). Key practical principles arising from these findings:

• Effective LLM-based issue resolution requires retention of causal fix ingredients rather than maximally broad or heuristically similar code snippets.
• The two-stage SWEzze pipeline—expensive OCD distillation followed by efficient segment-wise reranking—offers a practical balance between cost, compression, and patch success.
• Recommended workflow:
  1. Apply OCD to a held-out issue set for dataset creation.
  2. Fine-tune a reranker (e.g., with LoRA) for segment retention prediction.
  3. At runtime, decompose, score, and compress input contexts, targeting $u$4–$u$5 reduction.

• Outcomes include significant token savings (50–70%), reduced inference cost, and improved or at least non-degraded patch success ($u$6–$u$7) across LLMs.

6. Implications and Application Scope

The results demonstrate that model-driven context compression grounded in semantic sufficiency is a viable route for scaling LLM-based code reasoning to large repositories and complex patches. Application of SWEzze may yield improvements in both computational efficiency and practical bug-fixing success in automated software engineering pipelines, with lessons potentially generalizable to related domains where input window saturation and noisy context remain bottlenecks (Jia et al., 30 Mar 2026).

7. Summary

SWEzze establishes that LLMs tasked with source code repair benefit from context compression strategies focused on Oracle-guided sufficiency, balancing minimal input size against retention of critical patch-enabling information. This achieves reductions in computational cost, improved patch resolution rates, and a robust performance profile compared to competitive baselines in the domain of LLM-driven software maintenance.