- The paper demonstrates a novel two-stage extraction method using a fine-tuned Qwen 3.5 2B model to prune tool outputs with 0.86 recall and 0.80 F1.
- It compresses 92% of input tokens, reducing redundant computations and improving processing efficiency in coding agent workflows.
- Empirical comparisons show Squeez outperforms zero-shot LLMs and heuristic methods, establishing state-of-the-art performance in tool-aware evidence extraction.
Managing long, mixed-format tool outputs is a persistent efficiency bottleneck for coding agents that manipulate code repositories, build systems, logs, and various ecosystem tools within contemporary software engineering automation frameworks. The vast majority of tokens in tool observations do not significantly impact the next decision step, yet standard paradigms require the agent to repeatedly process entire outputs. The central research question addressed is: given a focused query and a single tool observation, how can one extract the minimal verbatim evidence block necessary for the agent’s immediate reasoning step?
This focus delineates the problem from generic prompt-compression approaches, which treat documents or prompts as undifferentiated token streams, and from abstractive summarization, which tolerates loss of local evidence traceability. The intent is to enable coding agents to process information more efficiently while ensuring faithfulness by preserving source-grounded evidence.
Benchmark and Dataset Construction
The authors introduce a comprehensive benchmark consisting of 11,477 annotated examples, built from two primary data sources:
- SWE-bench-Derived Tool Outputs: 10,713 raw tool outputs (9,205 released) reflecting real repository interactions with 14 tool types, including file reads, grep, test runners, linters, VCS logs, and build outputs.
- Synthetic Multi-Ecosystem Samples: 2,039 realistic synthetic outputs (1,697 positives released) spanning TypeScript, Go, Rust, Java, Docker, Terraform, and Kubernetes. Additionally, 575 explicit negative examples are constructed by mismatching queries and tool outputs.
Each example pairs a focused, tool-aware extraction query (not a full task statement) with a single tool observation. A two-stage pipeline generates line-level span annotations: an LLM teacher first formulates a focused query and then selects the minimal contiguous span or set of spans that supplies the evidential basis for the agent’s next step. These labels are mapped to verbatim source line indices, supporting precise extractive supervision.
The final split ensures diversity: 10,508 train, 240 dev, and a manually curated set of 618 test examples, balancing repository and ecosystem diversity and eliminating trivial or duplicate cases.
Model Architecture and Training
A compact, task-tuned Qwen 3.5 2B model serves as the main architecture, fine-tuned with LoRA adapters. The model is trained to take as input a focused query and a raw tool observation, and to output the verbatim evidence block wrapped in XML tags. Key hyperparameters include a maximum sequence length of 20,000 tokens, batch size 8, accumulated gradients 4, learning rate 2×10−4, and training for three epochs. The model, once tuned, is merged and can be deployed via vLLM servers or as a CLI filter, facilitating integration into existing coding agent stacks with negligible system modification.
Baseline comparisons include:
- Zero-shot large models (Qwen 3.5 35B A3B, Kimi K2).
- The untuned Qwen 3.5 2B base.
- Heuristic extractive methods: BM25, First-N, Last-N, Random, all tuned to preserve approximately 10% of input lines.
Evaluation Protocol and Results
The primary metrics are recall and F1 overlap at the line-level, with a tolerant matching criterion accepting fuzzy substring similarity above 0.5. Compression is quantified as the fraction of input tokens removed.
- The LoRA-tuned Squeez-2B model achieves 0.86 recall and 0.80 F1 while removing 92% of input tokens, strongly surpassing both stronger and weaker baselines.
- Qwen 3.5 35B A3B lags by 11 recall points, with F1 of 0.73, while Kimi K2 and untuned Qwen 2B further degrade due to over-compression or noisy selection.
- Heuristic methods (e.g., BM25 at 0.22 recall) are largely ineffective, confirming the inadequacy of lexical matching heuristics on heterogeneous, position-variable tool outputs.
Qualitative analysis underscores the model’s precision on tasks requiring pinpoint selection within structured outputs (e.g., git logs), succinct isolation of failure blocks in logs, and correct empty outputs on negatives, with error cases often involving adjacent or slightly over-broad span selection rather than outright irrelevance.
Prior work in prompt compression (e.g., LLMLingua [jiang2023llmlingua], LongLLMLingua [jiang2024longllmlingua]) primarily addresses context window efficiency for LLMs but does not support task-focused, tool-specific, query-conditioned extraction from agent tool outputs. Document-level context pruners in retrieval-augmented generation (e.g., EXIT [hwang2024exit], Provence [chirkova2025provence]) and semantic highlighter models also fail to support the mixed-format and tool-aware scenario here, where observations blend code, logs, traces, and structured data.
Agent-specific pruning systems (e.g., FocusAgent [focusagent2025], SWE-Pruner [ayanami2026swepruner]) either target web agents or repository-level code, not the narrow task of single-observation, query-focused extraction. Extractive QA datasets (e.g., SQuAD, FEVER) motivate the faithfulness objective but operate on prose rather than tool output. This work uniquely formalizes and addresses the faithfulness and efficiency trade-off in coding agent observability.
Implications and Future Directions
The findings indicate that standard zero-shot LLMs and retrieval heuristics are insufficient for robust, tool-aware evidence extraction from agent tool output. State-of-the-art results are achievable with small, supervised models leveraging domain-specific annotation, enhancing efficiency (through strong compression) without sacrificing evidence retention.
Practically, Squeez enables seamless deployment as an intermediate tool-output filter within agent stacks, reducing redundant agent computation and context window saturation. Theoretically, it demonstrates the value of narrow, labeled extractive tasks for mixed-format artifacts, portending greater modularity in agent architectures where extraction, reasoning, and generation are increasingly disjoint and independently tunable.
Future avenues may include:
- Extending beyond single observation pruning to multi-step agent trajectory optimization and reinforcement learning over agent-in-the-loop extraction.
- Exploring adaptive, cross-tooltype extraction strategies, and hierarchical pruning as tool ecosystem complexity increases.
- Integrating broader modal coverage (e.g., images, telemetry data) in multi-modal agent pipelines.
- Evaluating the downstream impact on end-to-end software engineering task success, agent reliability, and cost-efficiency.
Conclusion
Squeez defines and solves the task of task-conditioned, tool-output pruning for coding agents through a benchmark of grounded span annotations across 27 tool types, a focused dataset, and deployment-ready fine-tuned LLMs. The approach demonstrates that narrow, supervised extraction models—rather than large, generic LLMs or document-level retrieval heuristics—yield state-of-the-art evidence preservation under high compression, directly supporting agent efficiency and faithfulness in practical software engineering automation. The code, model, and data are publicly available, providing a new foundation for research and deployment in code-oriented agent systems.
Reference: "Squeez: Task-Conditioned Tool-Output Pruning for Coding Agents" (2604.04979)
