Correctness-Aware Repository Filtering Under Maximum Effective Context Window Constraints

Abstract: Context window efficiency is a practical constraint in LLM-based developer tools. Paulsen [12] shows that all tested models degrade in accuracy well before their advertised context limits the Maximum Effective Context Window (MECW) which makes context construction a quality problem, not just a cost one. Modern software repositories routinely contain large non-code artifacts compiled datasets, binary model weights, minified JavaScript bundles, and gigabyte-scale log files that overflow the context window and push out task-relevant source code. We present a correctness-aware context hygiene framework: a pre-execution, size-based heuristic filter that intercepts repository scans before tokenization, using only OS-level stat() metadata with sub-millisecond overhead. Semantic retrieval approaches such as RepoCoder, GraphRAG, and AST-based chunking require index construction and query-time inference before any filtering decision is reached. Our framework, by contrast, requires no indexing and operates at <0.01 ms per file decision. Across 10 real open-source repositories (22,046 files, 5 languages), the proposed SizeFilter at θ=1 MB achieves 79.6% (\pm13.2%) mean token reduction at 0.30 ms overhead: the HybridFilter achieves 89.3% (\pm9.0%) the lowest variance of any filter evaluated. A token-density study across 2,688 files confirms a strong linear correlation (Pearson r=0.997, k=0.250 tokens/byte). A limited-scope evaluation (18 tasks, CodeLlama-7B-Instruct) yields 72% file-level accuracy under filtering versus 25% at baseline; hallucination frequency declines from 61% to 17%. All code and data are released for reproducibility.

• The paper presents a pre-execution filtering framework that reduces token counts by up to 94.1% while preserving correctness in LLM inputs.
• The approach uses a size-based heuristic and hybrid filter with near-zero false positives and sub-millisecond overhead per file.
• Empirical results show CodeLlama-7B-Instruct accuracy rising from 25% to 72% and hallucination rates dropping from 61% to 17% after filtering.

Correctness-Aware Repository Filtering for LLM Context Optimization

Motivation and Problem Statement

The practical cost of LLM-driven developer tools is fundamentally limited by context window quality, not capacity. Empirical findings by Paulsen (Paulsen, 21 Sep 2025) demonstrate that all tested LLMs exhibit accuracy degradation well before their nominal context window is reached, concretizing the Maximum Effective Context Window (MECW) constraint. In contemporary repositories, large, semantically irrelevant artifacts (e.g., log files, datasets, binary model weights) dominate token budgets, displacing task-relevant code and reducing prompt efficacy. Shi et al. (ICML 2023) showed that even 10% irrelevant context reduces LLM accuracy by up to 23%.

Framework Overview and Contributions

The paper introduces a pre-execution, size-based heuristic filtering framework that operates at the metadata level, utilizing only OS-level stat() calls and incurring sub-millisecond overhead per file. Key contributions include:

• Framework Design: Stack-agnostic, deployable without manual configuration, zero-index, non-blocking, override-capable filtering architecture that intercepts repository scanning before tokenization.
• Formal Cost Model: Empirically validated linear relationship between file size and token count ($k=0.250$ tokens/byte; Pearson $r=0.997$, $n=2688$).
• Filter Taxonomy and Empirical Evaluation: Eight filters evaluated across 10 repositories, with SizeFilter reducing token count by 79.6% (±13.2%) and HybridFilter by 89.3% (±9.0%) at near-zero FPR for standard workflows.
• Zero Disk I/O CI Methodology: Fully deterministic test suite with in-memory virtual filesystems for 45 cases in under 50 ms.
• Correctness Preservation: File-level CodeLlama-7B-Instruct accuracy improved from 25% (baseline) to 72% (filtered), hallucination rate reduced from 61% to 17% across 18 tasks.

Empirical Analysis and Filter Efficacy

Token Reduction and Latency

SizeFilter (1 MB threshold) offers a stable and statistically significant reduction in token count—79.6% mean, 13.2% standard deviation, 0.30 ms latency—outperforming extension-based and binary-based techniques, which are either unreliable or bounded by incomplete artifact coverage. The HybridFilter, which sequentially applies binary check, size check, minified detection, and semantic scoring, achieves 89.3% reduction (±9.0%), with most computational cost incurred by the semantic layer. Importantly, the false positive rate is near zero for relevant source files at the recommended 1 MB threshold, validating practical deployability.

Tail-at-Scale Structure

Data-heavy repositories exhibit pronounced tail-at-scale distribution: fewer than 2% of files account for 80–94% of total bytes and thus dominant token bloat. This structural property renders simple size-based heuristics highly effective—filtering a small subset yields near-optimal reductions. Per-repository analysis confirms that large files (>1 MB) contribute 84.3% of tokens removed and HybridFilter compresses overall token budgets from 154.0M to 4.6M (94.1% reduction).

Formal Validation

Token-density analysis ($k=0.250$ tokens/byte) shows near-perfect linearity (Pearson $r=0.997$), enabling accurate pre-tokenization heuristics at negligible cost. Wilson 95% CIs and Wilcoxon signed-rank tests substantiate statistical superiority of size-based filters over extension-based approaches.

Comparison with Semantic Retrieval and Compression

Semantic retrieval systems (RepoCoder, GraphRAG, AST chunking, embedding-based methods) address context optimization post-ingestion, necessitating index construction and query-time inference, with per-file latency orders of magnitude higher (10–300 ms/file) and inability to reliably handle binary artifacts. By contrast, the presented framework acts as a first-pass gate: it reduces the candidate file set by 80–97%, lowering downstream indexing and chunking costs, and removing non-parseable files ab initio. Context compression approaches (LLMLingua, RECOMP, Active Context Compression, GemFilter) similarly operate post-ingestion and cannot prevent context window overflow by large artifacts.

Correctness-Driven Outcomes

Limited-scope evaluation (18 tasks, 2 repositories, CodeLlama-7B-Instruct) reveals strong correctness preservation: file-level accuracy increases from 25% to 72%, function-level from 12.5% to 56.3%, and hallucinations decrease significantly. The improvement is directly attributable to the increased signal-to-noise ratio achieved by removing irrelevant artifacts and maximizing the inclusion of source code within the MECW.

Limitations and Directions for Future Work

• Large Legitimate Files: Heuristic filtering may exclude large, auto-generated source files (e.g., protocol buffer outputs), motivating adaptive thresholding and developer override facilities.
• Binary Detection Generalization: Magic-byte detection coverage is limited; expanding format signatures would improve reliability in ML-heavy and multimodal repositories.
• Semantic Filtering Overhaul: English keyword-based scoring lacks language agnosticism. Lightweight embedding-based relevance assessment is proposed.
• Benchmark Scale: Evaluation is preliminary; future work will port the protocol to SWE-bench with current frontier models.
• Dynamic Context Management: Dynamic adjustment of threshold ($\theta$) based on residual MECW capacity (multi-turn sessions) is an open avenue, as is feedback-based architecture integration.

Practical and Theoretical Implications

Pre-execution, correctness-driven context hygiene reframes prompt optimization from a cost constraint to a quality imperative. The ability to reliably admit only task-relevant files at minimal computational cost under real-world repository conditions (22,046 files, five languages) addresses a fundamental bottleneck in LLM-based developer tools. The framework is broadly deployable, robust to repository heterogeneity, and compatible as a frontend to semantic retrieval or compression paradigms. Future developments in adaptive, language-agnostic filtering and dynamic thresholding will further reinforce prompt consistency and correctness as context windows scale.

Conclusion

This work establishes correctness-aware, pre-execution repository filtering as a foundational mechanism for LLM context optimization under MECW constraints. Size-based heuristics, validated empirically and formally, deliver high token reduction at negligible overhead, preserving task accuracy and reducing hallucinations. The hybrid filter architecture Pareto-dominates competing strategies and is compatible as a first-pass gate for semantic approaches. Adaptive and language-agnostic extensions are natural future directions, and systematic evaluation on broad-scale benchmarks remains to be explored.