LongCodeBench: Evaluating Million-Token Code
- LongCodeBench is a long-context benchmark that evaluates coding LLMs on repository-scale comprehension and bug-fixing tasks using authentic GitHub repositories.
- It stratifies context lengths from 32K to 1M tokens and divides tasks between LongCodeQA for multiple-choice understanding and LongSWE-Bench for executable bug-fixing.
- Empirical results show that nominal long-context support often fails in repository-scale reasoning, exposing critical weaknesses in current coding models.
LongCodeBench (LCB) is a long-context benchmark for coding LLMs that evaluates whether models can actually understand and work within very large codebases, with context windows ranging from 32K tokens up to 1M tokens. It is organized around two complementary software engineering abilities—code comprehension, measured through LongCodeQA, and code repair / bug fixing, measured through LongSWE-Bench—and is built from real-world GitHub repositories and issues rather than synthetic long-context data. Its central empirical message is that nominal support for very large context windows does not imply effective long-context software engineering ability, especially when the task requires repository-scale reasoning or executable repair (Rando et al., 12 May 2025).
1. Scope, motivation, and problem setting
LongCodeBench was introduced to address a gap between rapidly expanding advertised context windows and the quality of long-context evaluation for coding models. The benchmark is motivated by the claim that existing long-context benchmarks tend to emphasize either synthetic retrieval or memory tests, such as Long Range Arena, Needle-in-a-Haystack, and RULER, or document-style reading and QA benchmarks, which the benchmark authors describe as insufficient for evaluating repository-scale coding workflows. In that framing, software repositories are treated as a natural testbed for million-token contexts because real projects often span tens of thousands to millions of tokens and require inter-file dependency tracking, bug localization, and repository-consistent edits (Rando et al., 12 May 2025).
The benchmark targets two distinct long-context bottlenecks. The first is repository-level comprehension: a model must inspect a large repository context and answer a repository-specific question that cannot be solved by generic programming knowledge alone. The second is precise generation under long context: a model must generate a valid patch for a real GitHub issue, with success determined by unit tests. The benchmark therefore separates information retrieval and understanding from executable repair, rather than collapsing both into a single score.
This design also reflects an explicit benchmark philosophy. LongCodeBench is not only “long” in aggregate; it is stratified across fixed context-length brackets so that performance can be examined as the amount of provided repository context increases. A plausible implication is that the benchmark is intended less as a single leaderboard and more as an instrument for studying degradation curves across scales.
2. Benchmark structure and task formulation
LongCodeBench consists of two sub-benchmarks evaluated across six context-length brackets:
The two tasks are deliberately complementary. LongCodeQA is a multiple-choice repository-comprehension task, whereas LongSWE-Bench is an execution-based bug-fixing task inspired by SWE-Bench but extended to much larger contexts (Rando et al., 12 May 2025).
| Aspect | LongCodeQA | LongSWE-Bench |
|---|---|---|
| Task type | Multiple-choice QA | Bug-fixing generation |
| Output | Final Answer: <LETTER> |
Single git-style patch |
| Metric | Accuracy (%) | Percentage of solved issues |
| Scale | 443 instances from 98 repositories | 600 instances from 61 repositories |
In LongCodeQA, each prompt contains instructions, the full repository with files sorted alphabetically by name, and a multiple-choice question. The task is zero-shot, and answer choices are shuffled uniformly. The metric is accuracy, defined as the percentage of correctly answered questions over all questions in the set. The benchmark contains 443 question-answer instances drawn from 98 public GitHub repositories. The per-bracket counts are 113, 76, 92, 65, 47, and 50 instances for 32K through 1M respectively; the associated repository counts are 27, 16, 20, 15, 11, and 9.
In LongSWE-Bench, each prompt contains instructions, repository files, issue title and body, and the expected patch format. The model must generate a single patch file in diff format, directly applicable with git apply. The reported score is the percentage of solved issues, where an instance is counted as solved only if all unit tests pass; any failed test or compilation error is a failure. LongSWE-Bench contains 600 bug-fixing instances, with 100 instances per context bracket, from 61 public Python repositories. The corresponding repository counts per bracket are 12, 11, 11, 10, 10, and 7.
The benchmark also reports bracket-level repository statistics. For LongCodeQA, the average files per repository are 26, 56, 88, 93, 209, and 202, and the average tokens per file are 690, 862, 1047, 1840, 1782, and 3419. For LongSWE-Bench, the average files per repository are 8, 17, 27, 62, 135, and 203, and the average tokens per file are 5200, 3668, 4646, 4219, 3211, and 4372. The benchmark authors explicitly interpret context length as being driven by both number of files, which increases inter-file dependency complexity, and tokens per file, which increases intra-file complexity.
3. Dataset construction, filtering, and validation
LongCodeQA and LongSWE-Bench are constructed differently, but both are derived from public Python GitHub repositories and both emphasize realism and manual validation (Rando et al., 12 May 2025).
For LongCodeQA, repositories are selected through GitHub REST APIs using criteria that include sufficient total token length and enough public issues to yield candidate questions. Closed GitHub issues are then filtered with GPT-4 to remove bug fixing, feature addition, installation problems, project roadmap discussion, and development practices. For the remaining issues, GPT-4 reformulates issue discussions into one repository-specific question, one correct answer, and three incorrect answers. A major construction problem was that many generated questions did not actually require repository context. The benchmark reports that 64% of generated questions could be answered using general coding knowledge alone. To remove such cases, GPT-4 is asked to answer each question without repository context, and any question answerable from internal knowledge alone is removed. After all filtering, only 3.99% of the initial issues are converted into suitable LongCodeQA items.
The benchmark also includes a validation step for LongCodeQA. A stratified random subset of 9 questions per context bracket, for a total of 54 questions, was manually verified. Of these, 52/54, or 96.3%, were judged to truly require in-depth repository research to answer. The benchmark authors further describe the question-generation procedure as “fair” because questions are generated using only the issue discussion, not the full codebase.
LongSWE-Bench is built through repository selection, issue selection, and manual filtering of ill-posed samples. Repositories are chosen for quality and size; issues must be resolved through a pull request, associated with unit tests, and constitute real bug fixes rather than refactors or ambiguous changes. Manual filtering removes unclear problem descriptions, ambiguous solutions, feature requests, and open-ended design decisions. For reproducibility, each issue is paired with a dedicated Docker image, exact dependencies are installed, and the original bug is verified before patching.
The long-context construction strategy differs across the two tasks. LongCodeQA uses the full repository text, with files concatenated in alphabetical order. LongSWE-Bench always includes the ground-truth files from the pull request that fixed the issue, then adds randomly sampled files from the repository until the target context bracket is reached. This choice is explicitly contrasted with BM25-style retrieval. The rationale is to isolate model long-context ability from retrieval quality by ensuring that oracle-relevant files are present. The appendix reports a retrieval robustness analysis on a smaller version of the benchmark: random retrieval yields a logistic regression trend with , whereas BM25 retrieval yields , above the paper’s significance threshold of $0.01$. This is used to justify the final design.
4. Evaluation protocol and main empirical findings
LongCodeBench evaluates both open- and closed-source long-context models under zero-shot prompting. The open-source models are Qwen2.5-14B-Instruct, Jamba 1.5 400B Large, Llama 3.1 405B Instruct, and Llama 4 Scout. The closed-source models are GPT-4o, GPT-4.1, Claude 3.5 Sonnet, Gemini 2 Flash, Gemini 1.5 Pro, and Gemini 2.5 Pro. The LongCodeQA prompt asks the model to analyze a GitHub repository and answer one question about it; the LongSWE-Bench prompt provides the issue body, code context, an example patch format, and an instruction to generate a single patch file (Rando et al., 12 May 2025).
The headline result is that long context remains a major weakness for current coding LLMs. Two examples are emphasized by the benchmark authors. On LongSWE-Bench, Claude 3.5 Sonnet drops from 29% at 32K to 3% at 256K. On LongCodeQA, Qwen2.5 drops from 70.2% at 512K to 40.0% at 1M.
LongCodeQA reveals substantial heterogeneity across models and scales. GPT-4.1 is identified as the strongest and most consistent QA model, reaching 78.3% at 128K and 80.0% at 1M. Llama 4 Scout also remains strong at large scales, with 78.7% at 512K and 76.0% at 1M. The Gemini family is described as relatively stable across brackets, generally staying in the mid-60s to low-70s. By contrast, some models degrade sharply as context increases: Jamba 1.5 400B reaches 72.8% at 128K but falls to 54.2% at 256K, and Qwen2.5 falls sharply at 1M. The benchmark authors explicitly note that performance is not monotonic with context length and that many models peak around 64K–128K rather than at their maximum supported window.
LongSWE-Bench is markedly harder. Most models solve very few issues, and several open-source models solve effectively none. Qwen2.5-14B scores 0 across all brackets; Llama 4 Scout also scores 0 across all brackets; Llama 3.1 405B is effectively 0; and Jamba 1.5 400B reaches a maximum of 3%. Among proprietary models, Claude 3.5 Sonnet is strongest at shorter contexts, with 29%, 19%, 15%, and 3% at 32K, 64K, 128K, and 256K respectively. Gemini 2.5 Pro is the strongest model over the full range, scoring 23%, 25%, 22%, 24%, 12%, and 7% from 32K to 1M. GPT-4.1 is a notable contrast case: it is strong on LongCodeQA but scores only 1, 1, 1, 1, 1, and 2 across the six LongSWE-Bench brackets.
These results support several benchmark claims. First, short-context performance does not predict long-context performance. Second, comprehension and repair diverge: a model can be strong at repository-scale QA and still be weak at repository-scale repair. Third, improvements across model generations appear specialized rather than uniform. The benchmark authors specifically note that GPT-4.1 improves over GPT-4o on LongCodeQA but not on LongSWE-Bench, whereas Gemini 2.5 Pro improves dramatically over Gemini 1.5 Pro on LongSWE-Bench but not comparably on LongCodeQA.
5. Analyses, design implications, and limitations
LongCodeBench includes several analyses intended to show that its difficulty is not an artifact of weak filtering or prompt design. A filtering ablation on a small dataset of approximately 60 LongCodeQA samples shows that, after applying the filter for repository-specific questions, Jamba 1.5 drops from 92.9% to 62.3% and Llama 3.1 drops from 76.2% to 64.6%. On the complete dataset before the final no-context filtering step, scores are also substantially higher than in the final benchmark: for example, Qwen2.5 reaches up to 82.5% at 128K and 82.0% at 512K, Gemini 1.5 Pro reaches up to 86.0% at 512K and 83.4% at 1M, and Claude 3.5 Sonnet reaches 84.3% at 128K. The final filtering step therefore materially increases repository dependence (Rando et al., 12 May 2025).
A file-length analysis on LongCodeQA yields an unexpected result. Using Gemini 2 Flash, the benchmark authors split examples within each bracket by median average file length and report that longer files correlate with higher accuracy in 5 of 6 brackets, with the largest gap near 40% in the 512K bracket. The paper does not resolve the cause of this effect. A plausible implication is that file length alone is not a reliable proxy for repository difficulty.
The benchmark also includes a topic analysis. Using GPT-4o to infer question topics, the authors analyze Gemini 1.5 Pro’s accuracy by topic and report only moderate variation by domain, generally around the overall average of approximately 70%. This is presented as evidence of relatively stable comprehension across repository domains.
LongCodeBench has several explicit limitations. Evaluation cost is high: Claude 3.5 Sonnet costs 100 USD on LongCodeQA and 140 USD on LongSWE-Bench; GPT-4o costs 50 USD per task; Gemini 1.5 Pro costs 300 USD on LongCodeQA and 700 USD on LongSWE-Bench. Self-hosted evaluation is also expensive, with Qwen2.5 taking 35 hours for LongCodeQA and 12 hours for LongSWE-Bench, and Jamba and Llama 3.1 taking 40 hours on LongCodeQA. The benchmark is Python-only, uses public Python repositories, and therefore does not address multilingual or multi-language repository evaluation. LongSWE-Bench intentionally includes oracle-relevant files, which isolates long-context reasoning from retrieval but differs from an end-to-end engineering setting in which retrieval itself is part of the task. The paper also leaves open the mechanistic question of whether failures arise primarily from attention dilution, retrieval failure, reasoning breakdown, or generation instability.
6. Position in the benchmark landscape and later extensions
LongCodeBench occupies a specific position within long-code evaluation. Relative to "LONGCODEU" (Li et al., 6 Mar 2025), it combines coding comprehension and repair with real-world data and context lengths up to 1M, whereas LONGCODEU defines long code understanding itself as a standalone evaluation target across four aspects and eight tasks—code unit perception, intra-code unit understanding, inter-code unit relation understanding, and long documentation understanding—with a maximum code length of 128K tokens and an average length of 54.8K. LONGCODEU’s main finding that performance drops dramatically beyond 32K tokens is consistent with LongCodeBench’s broader observation that declared context support does not translate into robust repository-scale reasoning.
Relative to "BigCodeBench" (Zhuo et al., 2024), LongCodeBench addresses a different realism axis. BigCodeBench is a function-level, execution-based Python benchmark for diverse function calls and complex instructions, covering 1,140 programming tasks across 139 libraries and 7 domains. It emphasizes multi-library tool use and compositional reasoning rather than repository-scale context or million-token inputs. The two benchmarks are therefore complementary: BigCodeBench stresses realistic function-level synthesis under complex specifications, whereas LongCodeBench stresses repository-scale comprehension and repair.
LongCodeBench is also relevant to work on long-context inference efficiency. "KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approaches" (Yuan et al., 2024) includes code completion tasks such as LCC and RepoBench-P, which emphasize predicting subsequent lines of code with cross-file dependencies, and concludes that keeping the prefill process uncompressed is crucial for performance maintenance. That paper further reports that quantization methods are broadly reliable across tasks, while token-dropping approaches can perform exceptionally well on some code-related tasks, though with substantial backbone dependence. "LongAttnComp: Cross-Family Context Compression for Long-Context Reasoning" (Ji et al., 31 May 2026) shows that a learned query-conditioned compressor can match or exceed full-context accuracy on InfiniteBench Code-Debug, where inputs average roughly 115k tokens and can exceed 200k, indicating that retrieval-side compression can be effective on some long-code workloads even though it is not a direct repository-scale repair benchmark.
A later robustness-focused extension, "Robustness and Reasoning Fidelity of LLMs in Long-Context Code Question Answering" (Maharaj et al., 19 Feb 2026), broadens the LongCodeBench QA setting across languages and evaluation modes. It reevaluates a Python subset of LongCodeBench under shuffled multiple-choice and open-ended settings, adds COBOL and Java benchmarks, and introduces needle-in-a-haystack tests with relevant and irrelevant snippets. Its central claim is that the standard LongCodeBench multiple-choice setup can overestimate true reasoning ability. The reported results show a substantial recognition–generation gap, especially in COBOL, along with strong positional effects and distractor sensitivity; the authors characterize part of this behavior as recency bias and describe a “Lost at the Start” phenomenon. This extension suggests that LongCodeBench is not only a benchmark for long-context coding performance but also a starting point for studying reasoning fidelity under long-context code QA.
Taken together, these comparisons indicate that LongCodeBench’s distinctive contribution is the combination of real GitHub-derived repository tasks, million-token context brackets, explicit separation between comprehension and repair, and graded evaluation across scales. Within the broader benchmark landscape, it serves as a repository-scale counterpart to function-level execution benchmarks, task-decomposed long-code understanding benchmarks, and efficiency-oriented long-context evaluations.