---
title: 'RepoExploreBench: Iterative Test Exploration'
url: https://www.emergentmind.com/topics/repoexplorebench
type: topic
---

# RepoExploreBench: Iterative Test Exploration

Searching arXiv for RepoExploreBench and directly related repository-exploration benchmark papers.
{"query":"RepoExploreBench arXiv", "max_results": 10}
{"query":"\"Planning to Explore: Curiosity-Driven Planning for LLM Test Generation\" arXiv", "max_results": 5}
RepoExploreBench is a benchmark for **iterative, exploration-based test generation** with large language models, introduced in “Planning to Explore: Curiosity-Driven Planning for LLM Test Generation” [2604.05159]. It was designed to evaluate a regime that the paper argues is largely missed by existing LLM test-generation benchmarks: code in which reaching deep behavior requires **multi-round interaction**, **sequential exploration**, and exploitation of execution feedback rather than one-shot test writing. In this formulation, the code under test is treated as an unknown environment, generated tests are actions, coverage feedback is the observation channel, and the benchmark is intended to stress modules with **corridor structure**—cases where setup actions may yield little immediate coverage but are necessary to unlock later branch discovery [2604.05159].

## 1. Definition and research role

RepoExploreBench is not presented as a generic unit-test corpus. Its role is narrower and more specific: it evaluates whether an LLM-based tester can behave like an **exploration agent** in a repository setting, using repeated execution and coverage feedback to discover additional program behavior. The motivating claim is that many prior evaluations effectively assess **one-shot** generation, whereas RepoExploreBench targets code where **multi-round feedback is essential** and where greedy uncovered-branch targeting often fails [2604.05159].

The benchmark is framed around the paper’s analogy between exploratory test generation and Bayesian exploration. In that mapping, the program’s hidden branch reachability structure is the unknown environment, the interaction history is a sequence of tests and coverage results, and the evolving coverage map serves as a proxy posterior over what has already been discovered. The paper defines the exploration components as follows:

- **Unknown environment**: $\Theta$ = the program’s branch reachability structure  
- **History**: $h = (a_1,o_1)\cdots(a_t,o_t)$ = sequence of test/coverage-result pairs  
- **Exploration state**: $p(\Theta \mid h)$ = approximated by the coverage map $\mathcal{C}_t$  
- **Action**: $a$ = a test plan  
- **Observation**: $o$ = branches hit by executing the plan

This positioning makes RepoExploreBench distinct from benchmarks centered on repository-level autocomplete, issue localization, or proactive bug discovery. A plausible implication is that the benchmark is best understood as a specialized evaluation of **sequential repository exploration for test generation**, rather than a general benchmark of repository understanding.

## 2. Corpus construction and benchmark contents

RepoExploreBench is built from **93 modules** drawn from **9 popular open-source Python packages**, totaling **77,242 lines of code** [2604.05159]. The repositories were selected from the **top 150 most-downloaded PyPI packages**, using a **30-day rolling download window as of March 2026**. Selection followed three package-level criteria: **Pure Python source with at least 5K lines**, **No C extensions, build tools, or type stubs**, and evidence of **corridor structure**, such as deep import chains, class hierarchies, and configuration requirements [2604.05159].

The selected packages are **click**, **requests**, **flask**, **rich**, **jinja2**, **httpx**, **pydantic**, **werkzeug**, and **starlette**. Within those packages, candidate modules were selected by enumerating all submodules and checking line counts. The module-level criteria were: minimum **200 source lines**, preference for **larger modules**, **functional diversity** within each repository, exposure of **callable classes or functions**, exclusion of **test-only modules** except testing utilities such as `click.testing`, exclusion of **private modules** except for **httpx**, where `_`-prefixed modules are conventional, and exclusion of **vendored code** and **legacy compatibility layers** [2604.05159].

| Package | Modules | Lines |
|---|---:|---:|
| click | 10 | 8,293 |
| flask | 10 | 5,369 |
| httpx | 10 | 6,925 |
| jinja2 | 12 | 12,751 |
| pydantic | 10 | 11,727 |
| requests | 6 | 4,375 |
| rich | 12 | 12,792 |
| starlette | 11 | 4,625 |
| werkzeug | 12 | 10,385 |

The package choice was deliberate rather than representative. The appendix descriptions emphasize corridor-like access patterns such as **command setup $\rightarrow$ option parsing $\rightarrow$ type coercion** for click, **environment setup $\rightarrow$ lexer $\rightarrow$ parser $\rightarrow$ AST $\rightarrow$ compiler** for jinja2, and **app $\rightarrow$ routing $\rightarrow$ middleware $\rightarrow$ request/response** for starlette [2604.05159]. This suggests that RepoExploreBench is curated to foreground modules where exploratory sequencing matters, rather than to approximate the overall distribution of Python package structure.

## 3. Task formulation and interaction loop

On RepoExploreBench, the model is asked to **iteratively generate executable Python test scripts for a target module** so as to maximize cumulative code coverage under a fixed execution budget [2604.05159]. The target is a module rather than a single branch, and the benchmark evaluates the entire iterative interaction: generation, execution, coverage update, and replanning.

The benchmark loop exposes three inputs to the model: the **target module source code**, a **coverage map**, and **recent test history**. The system then generates candidate tests or plans, executes the selected scripts, collects branch coverage, updates the coverage map, and feeds the updated state back into the next round. The coverage map is the benchmark’s central state variable and is explicitly treated as a deterministic proxy for the posterior over discovered reachability structure. The update rule is:

$$
\mathcal{C}_t = \mathcal{C}_{t-1} \cup B(a_t)
$$

where $B(a_t)$ denotes the branches reached by the executed action [2604.05159].

The textual feedback supplied through the coverage map is structured rather than raw instrumentation output. The paper states that it can include: branches discovered so far, the number of new branches from the last step, average discovery rate, stagnation warnings, and the most informative prior tests [2604.05159]. This feedback channel is what makes the benchmark genuinely iterative: the model is expected to alter its strategy based on prior execution outcomes.

For the CovQValue method introduced in the same paper, the action is a **plan** of $S=3$ scripts rather than a single script, and the benchmark therefore supports **multi-step test trajectories**. The paper’s implemented scoring reduces the recursive exploration objective to an LLM-estimated proxy:

$$
\Tilde{g}(a \mid h) = I(O; \Theta \mid h, a)
$$

$$
q(h, a) = \Tilde{g}(a \mid h) + \gamma \cdot \mathbb{E}_{o \mid h,a}[v(hao)]
$$

with the practical score

$$
Q = \hat{g} + \gamma \cdot \hat{v}
$$

where both $\hat{g}$ and $\hat{v}$ are predicted on a 0–50 scale [2604.05159]. A plausible implication is that RepoExploreBench is expressly designed to reward actions whose value lies in **future reachability**, not only in immediate branch gain.

## 4. Evaluation protocol and metrics

The principal metric on RepoExploreBench is **branch coverage**, defined in the paper as “the cumulative number of unique branches executed across all test scripts” [2604.05159]. The benchmark treats this as the natural exploration metric because branch discovery is the direct operational proxy for behavioral reachability. Two secondary metrics are also reported: **Line coverage**, to connect the results to more conventional test-generation evaluations, and **Pass rate**, the fraction of generated scripts that execute without error [2604.05159].

The benchmark uses a **fixed execution budget** of $N=24$ test script executions for every strategy. In the reported experiments, Random, Greedy, and CovGreedy each execute **24 individual scripts**, whereas CovQValue executes **8 rounds** of **3-script plans**, so again totals 24 executions [2604.05159]. Candidate sampling is fixed at $K=3$, and the CovQValue configuration uses plan length $S=3$ and discount factor $\gamma=0.5$.

RepoExploreBench is operationalized in a relatively controlled environment. The paper states that it runs inside a **single Docker image** with all 9 packages pre-installed, and that branch coverage is measured using **Python’s standard branch coverage tooling** [2604.05159]. Unlike repository-specific testbeds derived from issue-fixing tasks, the environment is therefore unified across the benchmark. The paper also notes that **no train/dev/test split is described**; RepoExploreBench is presented as an **evaluation benchmark** over the full set of 93 targets [2604.05159].

An important methodological feature is that equal **execution budget** does not imply equal **feedback frequency**. CovQValue commits to 3-step plans and therefore receives only 8 map updates rather than 24. The paper explicitly highlights this as a disadvantage in update frequency, which strengthens the interpretation of any gains as coming from planning rather than from more opportunities to react [2604.05159].

## 5. Empirical results and difficulty profile

RepoExploreBench is the benchmark on which the paper reports the largest gains from planning-aware exploration. Across **Gemini 3 Flash**, **GPT-5.4 Mini**, and **Mistral Large 3**, CovQValue substantially outperforms Random, Greedy, and CovGreedy on mean per-module branch coverage [2604.05159].

The reported branch-coverage numbers are:

- **Gemini 3 Flash**: Random **43.2 ± 4.5**, Greedy **41.8 ± 3.9**, CovGreedy **51.7 ± 5.7**, CovQValue **72.4 ± 6.5**
- **GPT-5.4 Mini**: Random **31.2 ± 3.0**, Greedy **28.6 ± 2.8**, CovGreedy **36.5 ± 3.5**, CovQValue **49.8 ± 5.0**
- **Mistral Large 3**: Random **41.2 ± 4.2**, Greedy **42.3 ± 4.4**, CovGreedy **40.0 ± 4.5**, CovQValue **59.1 ± 6.0** [2604.05159]

The paper’s abstract summarizes these as roughly **40–74%** relative branch-coverage improvement over Greedy, depending on model [2604.05159]. It also reports **Cohen’s $d$** of **0.77**, **0.54**, and **0.49**, win rates of **83%**, **84%**, and **78%**, and a paired $t$-test with **$p < 0.0001$** for CovQValue versus Greedy [2604.05159].

The secondary metrics show a consistent trade-off. CovQValue tends to achieve higher branch and line coverage with somewhat lower pass rate, suggesting that stronger exploration involves more ambitious and more failure-prone tests. For example, on RepoExploreBench with Gemini 3 Flash, CovQValue reaches **197** line coverage with **71%** pass rate, whereas Greedy reaches **142** line coverage with **83%** pass rate [2604.05159].

The paper attributes the benchmark’s difficulty to its intended **corridor structure**. Case studies report that baselines can remain nearly flat for many steps because they fail to discover the required initialization or parser-state setup. In `flask.app`, all three baselines remain at **2 branches for all 24 steps**, whereas CovQValue reaches **182 branches** after breaking through the application-factory initialization sequence [2604.05159]. In `werkzeug.http`, Greedy plateaus after shallow parsing functions, while CovQValue progresses to deeper functions such as `parse_options_header`, `dump_options_header`, `parse_cookie`, and `dump_cookie`, which the paper interprets as traversal of a parser/tokenizer corridor [2604.05159].

The ablations reinforce that RepoExploreBench is not simply rewarding more attempts. On RepoExploreBench with Gemini 3 Flash, increasing the execution budget from **$N=8$** to **$N=24$** raises CovQValue from **58.0** to **71.2** branches, while Random stays around **41**; adding diversity hints raises coverage from **49.3 ± 5.4** to **67.9 ± 6.1**; and adding Q-value scoring raises it further to **72.4 ± 6.5** [2604.05159]. This suggests that benchmark performance depends jointly on state feedback, plan diversity, and future-aware plan scoring.

## 6. Relation to adjacent repository benchmarks and limitations

RepoExploreBench occupies a distinct position among repository-scale benchmarks. Compared with **RepoBench**, which evaluates repository-level auto-completion through retrieval, code completion, and an end-to-end retrieve-then-complete pipeline, RepoExploreBench is not a completion benchmark and does not define repository understanding as next-line prediction under cross-file context [2306.03091]. Compared with **SWE-Explore**, which isolates repository exploration as ranked line-region selection under a fixed line budget for issue-conditioned code localization, RepoExploreBench does not ask for ranked evidence spans or issue-based exploration, but for iterative executable test generation under coverage feedback [2606.07297]. Compared with **TestExplora**, which evaluates documentation-grounded proactive bug discovery through repository-level test generation, RepoExploreBench is narrower in oracle design and is centered on **coverage-driven exploration** rather than fail-to-pass defect exposure against buggy and fixed repository versions [2602.10471].

It is also narrower than repository-exploration infrastructures such as **GraphRepo**, which aim to support fast, generic mining and querying of software repositories through a persistent graph database [2008.04884], or exploratory interfaces such as **ExIF**, which support fragment-centered navigation through repository history and revision context [2104.13568]. RepoExploreBench does not benchmark general-purpose repository mining or revision exploration. Its scope is specifically **iterative test generation on corridor-structured Python modules**.

The benchmark’s limitations follow directly from its construction. It is **curated rather than representative**, explicitly favoring packages and modules with corridor structure [2604.05159]. It is **Python-only**, limited to **pure-Python** packages without native extensions, and currently covers only **93 modules from 9 packages** [2604.05159]. It uses **coverage metrics**, especially branch coverage, as a proxy for exploration success, and the coverage map is explicitly described as a **deterministic proxy** rather than a true posterior over branch reachability [2604.05159]. The paper also does not report a contamination analysis, despite using popular open-source packages. These constraints do not negate the benchmark’s purpose, but they make clear that RepoExploreBench should be read as a benchmark for **sequential exploration difficulty in test generation**, not as a complete measure of repository understanding.

In that sense, RepoExploreBench’s main significance is methodological. It reframes a portion of repository-level software evaluation from one-shot generation to **interactive exploration under execution feedback**, and it operationalizes that shift with a benchmark in which the value of a test often lies not in what it covers immediately, but in what it makes reachable next [2604.05159].

Source: https://www.emergentmind.com/topics/repoexplorebench