RepoExploreBench: Iterative Test Exploration

• RepoExploreBench is a specialized benchmark for sequential test generation that evaluates LLMs on exploring deep, corridor-structured Python modules through multi-round interactions.
• It employs iterative planning with repeated executions and coverage feedback to guide test generation, emphasizing future reachability over immediate branch gains.
• Constructed from 93 modules across 9 popular Python packages, the benchmark provides robust metrics such as branch and line coverage to assess exploration efficiency.

Searching arXiv for RepoExploreBench and directly related repository-exploration benchmark papers. {"query":"RepoExploreBench arXiv", "max_results": 10} {"query":"\"Planning to Explore: Curiosity-Driven Planning for LLM Test Generation\" arXiv", "max_results": 5} RepoExploreBench is a benchmark for iterative, exploration-based test generation with LLMs, introduced in “Planning to Explore: Curiosity-Driven Planning for LLM Test Generation” (Amayuelas et al., 6 Apr 2026). It was designed to evaluate a regime that the paper argues is largely missed by existing LLM test-generation benchmarks: code in which reaching deep behavior requires multi-round interaction, sequential exploration, and exploitation of execution feedback rather than one-shot test writing. In this formulation, the code under test is treated as an unknown environment, generated tests are actions, coverage feedback is the observation channel, and the benchmark is intended to stress modules with corridor structure—cases where setup actions may yield little immediate coverage but are necessary to unlock later branch discovery (Amayuelas et al., 6 Apr 2026).

1. Definition and research role

RepoExploreBench is not presented as a generic unit-test corpus. Its role is narrower and more specific: it evaluates whether an LLM-based tester can behave like an exploration agent in a repository setting, using repeated execution and coverage feedback to discover additional program behavior. The motivating claim is that many prior evaluations effectively assess one-shot generation, whereas RepoExploreBench targets code where multi-round feedback is essential and where greedy uncovered-branch targeting often fails (Amayuelas et al., 6 Apr 2026).

The benchmark is framed around the paper’s analogy between exploratory test generation and Bayesian exploration. In that mapping, the program’s hidden branch reachability structure is the unknown environment, the interaction history is a sequence of tests and coverage results, and the evolving coverage map serves as a proxy posterior over what has already been discovered. The paper defines the exploration components as follows:

• Unknown environment: $\Theta$ = the program’s branch reachability structure
• History: $h = (a_1,o_1)\cdots(a_t,o_t)$ = sequence of test/coverage-result pairs
• Exploration state: $p(\Theta \mid h)$ = approximated by the coverage map $\mathcal{C}_t$
• Action: $a$ = a test plan
• Observation: $o$ = branches hit by executing the plan

This positioning makes RepoExploreBench distinct from benchmarks centered on repository-level autocomplete, issue localization, or proactive bug discovery. A plausible implication is that the benchmark is best understood as a specialized evaluation of sequential repository exploration for test generation, rather than a general benchmark of repository understanding.

2. Corpus construction and benchmark contents

RepoExploreBench is built from 93 modules drawn from 9 popular open-source Python packages, totaling 77,242 lines of code (Amayuelas et al., 6 Apr 2026). The repositories were selected from the top 150 most-downloaded PyPI packages, using a 30-day rolling download window as of March 2026. Selection followed three package-level criteria: Pure Python source with at least 5K lines, No C extensions, build tools, or type stubs, and evidence of corridor structure, such as deep import chains, class hierarchies, and configuration requirements (Amayuelas et al., 6 Apr 2026).

The selected packages are click, requests, flask, rich, jinja2, httpx, pydantic, werkzeug, and starlette. Within those packages, candidate modules were selected by enumerating all submodules and checking line counts. The module-level criteria were: minimum 200 source lines, preference for larger modules, functional diversity within each repository, exposure of callable classes or functions, exclusion of test-only modules except testing utilities such as click.testing, exclusion of private modules except for httpx, where _-prefixed modules are conventional, and exclusion of vendored code and legacy compatibility layers (Amayuelas et al., 6 Apr 2026).

Package	Modules	Lines
click	10	8,293
flask	10	5,369
httpx	10	6,925
jinja2	12	12,751
pydantic	10	11,727
requests	6	4,375
rich	12	12,792
starlette	11	4,625
werkzeug	12	10,385

The package choice was deliberate rather than representative. The appendix descriptions emphasize corridor-like access patterns such as command setup $\rightarrow$ option parsing $\rightarrow$ type coercion for click, environment setup $\rightarrow$ lexer $\rightarrow$ parser $h = (a_1,o_1)\cdots(a_t,o_t)$0 AST $h = (a_1,o_1)\cdots(a_t,o_t)$1 compiler for jinja2, and app $h = (a_1,o_1)\cdots(a_t,o_t)$2 routing $h = (a_1,o_1)\cdots(a_t,o_t)$3 middleware $h = (a_1,o_1)\cdots(a_t,o_t)$4 request/response for starlette (Amayuelas et al., 6 Apr 2026). This suggests that RepoExploreBench is curated to foreground modules where exploratory sequencing matters, rather than to approximate the overall distribution of Python package structure.

3. Task formulation and interaction loop

On RepoExploreBench, the model is asked to iteratively generate executable Python test scripts for a target module so as to maximize cumulative code coverage under a fixed execution budget (Amayuelas et al., 6 Apr 2026). The target is a module rather than a single branch, and the benchmark evaluates the entire iterative interaction: generation, execution, coverage update, and replanning.

The benchmark loop exposes three inputs to the model: the target module source code, a coverage map, and recent test history. The system then generates candidate tests or plans, executes the selected scripts, collects branch coverage, updates the coverage map, and feeds the updated state back into the next round. The coverage map is the benchmark’s central state variable and is explicitly treated as a deterministic proxy for the posterior over discovered reachability structure. The update rule is:

$h = (a_1,o_1)\cdots(a_t,o_t)$5

where $h = (a_1,o_1)\cdots(a_t,o_t)$6 denotes the branches reached by the executed action (Amayuelas et al., 6 Apr 2026).

The textual feedback supplied through the coverage map is structured rather than raw instrumentation output. The paper states that it can include: branches discovered so far, the number of new branches from the last step, average discovery rate, stagnation warnings, and the most informative prior tests (Amayuelas et al., 6 Apr 2026). This feedback channel is what makes the benchmark genuinely iterative: the model is expected to alter its strategy based on prior execution outcomes.

For the CovQValue method introduced in the same paper, the action is a plan of $h = (a_1,o_1)\cdots(a_t,o_t)$7 scripts rather than a single script, and the benchmark therefore supports multi-step test trajectories. The paper’s implemented scoring reduces the recursive exploration objective to an LLM-estimated proxy:

$h = (a_1,o_1)\cdots(a_t,o_t)$8

$h = (a_1,o_1)\cdots(a_t,o_t)$9

with the practical score

$p(\Theta \mid h)$0

where both $p(\Theta \mid h)$1 and $p(\Theta \mid h)$2 are predicted on a 0–50 scale (Amayuelas et al., 6 Apr 2026). A plausible implication is that RepoExploreBench is expressly designed to reward actions whose value lies in future reachability, not only in immediate branch gain.

4. Evaluation protocol and metrics

The principal metric on RepoExploreBench is branch coverage, defined in the paper as “the cumulative number of unique branches executed across all test scripts” (Amayuelas et al., 6 Apr 2026). The benchmark treats this as the natural exploration metric because branch discovery is the direct operational proxy for behavioral reachability. Two secondary metrics are also reported: Line coverage, to connect the results to more conventional test-generation evaluations, and Pass rate, the fraction of generated scripts that execute without error (Amayuelas et al., 6 Apr 2026).

The benchmark uses a fixed execution budget of $p(\Theta \mid h)$3 test script executions for every strategy. In the reported experiments, Random, Greedy, and CovGreedy each execute 24 individual scripts, whereas CovQValue executes 8 rounds of 3-script plans, so again totals 24 executions (Amayuelas et al., 6 Apr 2026). Candidate sampling is fixed at $p(\Theta \mid h)$4, and the CovQValue configuration uses plan length $p(\Theta \mid h)$5 and discount factor $p(\Theta \mid h)$6.

RepoExploreBench is operationalized in a relatively controlled environment. The paper states that it runs inside a single Docker image with all 9 packages pre-installed, and that branch coverage is measured using Python’s standard branch coverage tooling (Amayuelas et al., 6 Apr 2026). Unlike repository-specific testbeds derived from issue-fixing tasks, the environment is therefore unified across the benchmark. The paper also notes that no train/dev/test split is described; RepoExploreBench is presented as an evaluation benchmark over the full set of 93 targets (Amayuelas et al., 6 Apr 2026).

An important methodological feature is that equal execution budget does not imply equal feedback frequency. CovQValue commits to 3-step plans and therefore receives only 8 map updates rather than 24. The paper explicitly highlights this as a disadvantage in update frequency, which strengthens the interpretation of any gains as coming from planning rather than from more opportunities to react (Amayuelas et al., 6 Apr 2026).

5. Empirical results and difficulty profile

RepoExploreBench is the benchmark on which the paper reports the largest gains from planning-aware exploration. Across Gemini 3 Flash, GPT-5.4 Mini, and Mistral Large 3, CovQValue substantially outperforms Random, Greedy, and CovGreedy on mean per-module branch coverage (Amayuelas et al., 6 Apr 2026).

The reported branch-coverage numbers are:

• Gemini 3 Flash: Random 43.2 ± 4.5, Greedy 41.8 ± 3.9, CovGreedy 51.7 ± 5.7, CovQValue 72.4 ± 6.5
• GPT-5.4 Mini: Random 31.2 ± 3.0, Greedy 28.6 ± 2.8, CovGreedy 36.5 ± 3.5, CovQValue 49.8 ± 5.0
• Mistral Large 3: Random 41.2 ± 4.2, Greedy 42.3 ± 4.4, CovGreedy 40.0 ± 4.5, CovQValue 59.1 ± 6.0 (Amayuelas et al., 6 Apr 2026)

The paper’s abstract summarizes these as roughly 40–74% relative branch-coverage improvement over Greedy, depending on model (Amayuelas et al., 6 Apr 2026). It also reports Cohen’s $p(\Theta \mid h)$7 of 0.77, 0.54, and 0.49, win rates of 83%, 84%, and 78%, and a paired $p(\Theta \mid h)$8-test with $p(\Theta \mid h)$9 for CovQValue versus Greedy (Amayuelas et al., 6 Apr 2026).

The secondary metrics show a consistent trade-off. CovQValue tends to achieve higher branch and line coverage with somewhat lower pass rate, suggesting that stronger exploration involves more ambitious and more failure-prone tests. For example, on RepoExploreBench with Gemini 3 Flash, CovQValue reaches 197 line coverage with 71% pass rate, whereas Greedy reaches 142 line coverage with 83% pass rate (Amayuelas et al., 6 Apr 2026).

The paper attributes the benchmark’s difficulty to its intended corridor structure. Case studies report that baselines can remain nearly flat for many steps because they fail to discover the required initialization or parser-state setup. In flask.app, all three baselines remain at 2 branches for all 24 steps, whereas CovQValue reaches 182 branches after breaking through the application-factory initialization sequence (Amayuelas et al., 6 Apr 2026). In werkzeug.http, Greedy plateaus after shallow parsing functions, while CovQValue progresses to deeper functions such as parse_options_header, dump_options_header, parse_cookie, and dump_cookie, which the paper interprets as traversal of a parser/tokenizer corridor (Amayuelas et al., 6 Apr 2026).

The ablations reinforce that RepoExploreBench is not simply rewarding more attempts. On RepoExploreBench with Gemini 3 Flash, increasing the execution budget from $\mathcal{C}_t$0 to $\mathcal{C}_t$1 raises CovQValue from 58.0 to 71.2 branches, while Random stays around 41; adding diversity hints raises coverage from 49.3 ± 5.4 to 67.9 ± 6.1; and adding Q-value scoring raises it further to 72.4 ± 6.5 (Amayuelas et al., 6 Apr 2026). This suggests that benchmark performance depends jointly on state feedback, plan diversity, and future-aware plan scoring.

6. Relation to adjacent repository benchmarks and limitations

RepoExploreBench occupies a distinct position among repository-scale benchmarks. Compared with RepoBench, which evaluates repository-level auto-completion through retrieval, code completion, and an end-to-end retrieve-then-complete pipeline, RepoExploreBench is not a completion benchmark and does not define repository understanding as next-line prediction under cross-file context (Liu et al., 2023). Compared with SWE-Explore, which isolates repository exploration as ranked line-region selection under a fixed line budget for issue-conditioned code localization, RepoExploreBench does not ask for ranked evidence spans or issue-based exploration, but for iterative executable test generation under coverage feedback (Zhang et al., 5 Jun 2026). Compared with TestExplora, which evaluates documentation-grounded proactive bug discovery through repository-level test generation, RepoExploreBench is narrower in oracle design and is centered on coverage-driven exploration rather than fail-to-pass defect exposure against buggy and fixed repository versions (Liu et al., 11 Feb 2026).

It is also narrower than repository-exploration infrastructures such as GraphRepo, which aim to support fast, generic mining and querying of software repositories through a persistent graph database (Serban et al., 2020), or exploratory interfaces such as ExIF, which support fragment-centered navigation through repository history and revision context (Kim et al., 2021). RepoExploreBench does not benchmark general-purpose repository mining or revision exploration. Its scope is specifically iterative test generation on corridor-structured Python modules.

The benchmark’s limitations follow directly from its construction. It is curated rather than representative, explicitly favoring packages and modules with corridor structure (Amayuelas et al., 6 Apr 2026). It is Python-only, limited to pure-Python packages without native extensions, and currently covers only 93 modules from 9 packages (Amayuelas et al., 6 Apr 2026). It uses coverage metrics, especially branch coverage, as a proxy for exploration success, and the coverage map is explicitly described as a deterministic proxy rather than a true posterior over branch reachability (Amayuelas et al., 6 Apr 2026). The paper also does not report a contamination analysis, despite using popular open-source packages. These constraints do not negate the benchmark’s purpose, but they make clear that RepoExploreBench should be read as a benchmark for sequential exploration difficulty in test generation, not as a complete measure of repository understanding.

In that sense, RepoExploreBench’s main significance is methodological. It reframes a portion of repository-level software evaluation from one-shot generation to interactive exploration under execution feedback, and it operationalizes that shift with a benchmark in which the value of a test often lies not in what it covers immediately, but in what it makes reachable next (Amayuelas et al., 6 Apr 2026).