e-Otter++: Fail-to-Pass Test Generation System
- e-Otter++ is a system that automatically generates reproduction tests which fail on the old code and pass with the eventual patch.
- It integrates LLM-driven execution-augmented repair, heterogeneous issue morphs, and surrogate-patch selection to refine candidate tests.
- Empirical results on benchmarks like TDD-Bench Verified demonstrate significant improvements in fail-to-pass rates through structured prompt diversity and execution feedback.
Searching arXiv for papers on e-Otter++ and closely related work.
e-Otter++ is a system for automatically generating reproduction tests from software engineering issues before the fixing patch is available. Its target artifacts are fail-to-pass tests: tests that fail on the old code c_old and pass on the new code c_new, where c_new is unavailable during generation and used only for evaluation through a golden patch. The system combines execution-augmented test repair, heterogeneous prompting through issue morphs and context masks, and test selection via surrogate execution on generated code patches. On TDD-Bench Verified, it reaches a 63.0% fail-to-pass rate with Claude-3.7-Sonnet, substantially above the previously published results reported in the same study (Ahmed et al., 8 Aug 2025).
1. Problem formulation and target behavior
The formal objective is to construct a generator whose input consists of an issue description d_issue and the current buggy or incomplete codebase c_old, and whose output is a reproduction test. The desired output is not merely a failing test, but specifically a fail-to-pass test: it should fail on c_old and pass on the developer-written fix c_new, even though c_new is not available during generation (Ahmed et al., 8 Aug 2025).
This setting differs materially from conventional automated test generation. Traditional systems such as EvoSuite and Randoop assume a single, fixed, correct program and use execution feedback such as coverage, crashes, and assertion failures as guidance. In the issue-driven setting, by contrast, the code is missing or wrong precisely because the issue exists. Execution errors such as ModuleNotFoundError, TypeError, or environment misconfiguration are therefore common and often orthogonal to the behavior described in the issue. A further complication is that issue descriptions are informal, noisy, and sometimes misleading, with partial code, incorrect assumptions, or project-specific jargon. The central challenge is therefore not simply to synthesize a failing test, but to synthesize a test that fails for the reason described in the issue and will cease failing on the eventual fix.
A persistent misconception in this area is that any failing test is a useful reproduction. e-Otter++ explicitly rejects that view. The fail-to-pass requirement is stricter than simple failure because many candidate tests fail on c_old for the wrong reasons and may still fail after the patch. Such tests are not valid reproductions in the benchmark sense and are also of limited value for downstream patch validation.
2. System organization and execution-feedback loop
The architecture has three stages: a Test Generator, a Code Patch Generator, and a Test Selector. The Test Generator, called e-Otter, takes d_issue and c_old and produces 10 refined candidate tests. The Code Patch Generator reuses Agentless localization and patch generation to obtain up to 40 generated patches, normalizes them, and retains the 5 most frequent variants. The Test Selector then executes the candidate tests on c_old and on those 5 candidate patches, filters and ranks them, and returns a single final test (Ahmed et al., 8 Aug 2025).
This organization uses execution feedback in two distinct places. In the repair loop, failures on c_old are analyzed to improve tests. In the selector, the behavior of candidate tests on generated code patches is used as a surrogate signal for future pass behavior on the unknown c_new. The system is fully LLM plus rule-based: there is no symbolic execution, fuzzing, or constraint solving, and no deep learning training is performed. The implementation relies on inference with existing LLMs, including Claude 3.7 Sonnet, GPT-4o, and Mistral-large.
The design rationale is that execution remains informative even when the code is buggy, provided the system can distinguish target-aligned failures from incidental failures. e-Otter++ therefore treats execution not as a direct oracle of correctness, but as a black-box signal that must be filtered, interpreted, and paired with issue semantics.
3. Test synthesis: Otter lineage, morphs, masks, and repair
e-Otter++ extends the earlier Otter and Otter++ line. Otter comprises an LLM-based localizer, a self-reflective planner, and a test generator that writes imports, fixtures, and assertions while using static analysis such as Flake8 to fix syntax and semantic issues. Otter++ introduced prompt masks and simple execution-based selection. e-Otter++ adds execution-augmented repair, issue description morphs, and patch-based test selection (Ahmed et al., 8 Aug 2025).
The issue-morphing component produces five rewritten versions of the issue description: standard, simple, dropCode, initTest, and initPatch. These rewrites respectively canonicalize the report, simplify language, remove code snippets, embed an initial failing test, or embed an initial patch suggestion. In parallel, the system uses five context masks inherited from Otter++: planner, full, testLoc, patchLoc, and none. The masks selectively hide or expose localization information so that incorrect localization or noisy context does not bias every generated test in the same direction. The paper describes this as producing 10 diverse test candidates per issue.
The repair stage is the main execution-augmented component. For each candidate test, the system executes the test on c_old and captures the log. A critic LLM at temperature 0 then receives the issue description, the current test, and the execution log, and decides whether the test is failing for the right reason described in the issue. It also identifies a suspected buggy line in the test, the most relevant issue text or code snippet, and any functions or code fragments from c_old that would help repair the test. If the critic judges the failure to match the issue, the loop stops. Otherwise, the requested snippets are retrieved from c_old, a repair prompt is assembled, and another LLM at temperature 0.8 generates a repaired test. The loop repeats for up to 10 iterations.
Several empirical details clarify the role of this loop. A GPT-4o-based critic achieves approximately $0.78$–$0.83$ F1 on the task of deciding whether a test fails for the right reason, measured against golden tests. Many tests either require no repair iterations or hit the 10-iteration limit, and improvements plateau around 8 iterations. Ablation results indicate that every part of the repair prompt matters: removing the critic causes a 21% relative drop in fail-to-pass rate; removing buggy-line selection, relevant-snippet extraction, or function lookup each reduces fail-to-pass by about 8–9%; and using temperature 0 rather than 0.8 for repair reduces fail-to-pass by about 7%.
4. Surrogate-patch selection and patch-relative coverage
After repair, e-Otter++ has up to 10 candidate tests and 5 candidate patches. Test selection begins by executing each candidate on c_old and on each of the candidate patches. If no test is fail-to-pass on any patch, all 10 candidates are retained; otherwise, the selector restricts attention to tests that are fail-to-pass on at least one patch (Ahmed et al., 8 Aug 2025).
The retained candidates are then grouped by failure type on c_old. Assertion failures are preferred; other non-assertion failures are secondary; syntax errors, import errors, and runtime errors are least preferred. The selector keeps the highest-priority non-empty group. Within that group, it computes patch-relative coverage. If a patch contributes added lines added and deleted lines deleted, and a test executes covered_added and covered_deleted, then patch-relative coverage is defined as
The final score for a test is the average of this quantity over the 5 candidate patches, and the system returns the test with the highest average coverage.
This selector is notable because it uses imperfect generated patches as proxies for the unknown true fix. The paper argues that passing on those patches correlates strongly with passing on the golden patch in the benchmark. Empirically, the code-patch pre-filter is the major source of improvement in final selection: on TDD-Bench Verified with Claude, the fail-to-pass rate rises from 44.7% with no patches and random selection to 61.8% when patches are used with random tie-breaking, and then to 63.0% when coverage is added as the tie-breaker. The coverage heuristic is therefore secondary but consistently helpful, whereas the principal gain comes from patch-conditioned filtering.
A plausible implication is that surrogate patches act as a weak but operational approximation to the latent c_new. e-Otter++ does not assume those patches are correct; it only assumes that a test that is fail-to-pass on at least one plausible patch is more promising than one that fails uniformly.
5. Benchmarks, quantitative results, and ablations
The principal evaluation is on TDD-Bench Verified, derived from SWE-bench Verified and containing 449 issues, each with an old version c_old and a golden patch c_new. The study also evaluates on SWT-bench Lite, derived from SWE-bench Lite and containing 276 Python issues. The harness counts a generated test as successful if it fails on c_old and passes on c_new, considering only the contributing tests added or modified by the system (Ahmed et al., 8 Aug 2025).
| Benchmark | System / model | F→P |
|---|---|---|
| TDD-Bench Verified | Otter++ (Claude-3.7-Sonnet) | 38.8% |
| TDD-Bench Verified | e-Otter++ (Claude) | 63.0% |
| TDD-Bench Verified | e-Otter++ (GPT-4o) | 51.4% |
| SWT-bench Lite | Otter++ | 30–30.4% |
| SWT-bench Lite | Amazon Q Developer Agent | 37.7% |
| SWT-bench Lite | e-Otter++ (Claude) | 52.5% |
| SWT-bench Lite | e-Otter++ (GPT-4o) | 40.2% |
All reported improvements are statistically significant under the McNemar test with . The results establish that the combination of repair, structured diversity, and patch-based selection materially outperforms the previously published systems discussed in the paper.
The ablation studies further disaggregate the gains. Execution-augmented repair improves fail-to-pass across all masks and morphs by 11–74% relative, depending on prompt type and benchmark. On TDD-Bench Verified with Claude and the planner mask, fail-to-pass rises from 32.3% before repair to 39.9% after repair. Structured prompt diversity also contributes. For TDD-Bench Verified with Claude, the paper reports fail-to-pass-at- of 67.9% at for “planner + masks (5 prompts)” and 71.7% for “planner + masks + morphs (10 prompts),” a gain of 3.8 absolute points and 5.6% relative. The same analysis reports that heterogeneous prompts outperform pure high-temperature multi-sampling at the same .
The generated tests also serve as external validators for patch-generation systems. When e-Otter++ tests are used to filter patched solutions in SWE-bench Verified, the higher fail-to-pass-at- improves recall while preserving or improving precision for systems such as tools_claude-4-opus and sweagent_claude-4-sonnet. In that respect, the system is not only a standalone test generator but also an evaluation component for broader software engineering agents.
6. Position in the literature, limitations, and later developments
Within the literature on issue-driven test generation, e-Otter++ occupies a distinct position. Relative to EvoSuite and Randoop, it does not assume a correct target program and does not optimize primarily for coverage or generic bug finding. Relative to Otter and Otter++, it adds execution-augmented repair, issue morphs, and selection through candidate patches. Relative to AEGIS, it shares the use of execution feedback for repair but differs in augmenting repair prompts with localized code snippets, using morphs and masks for systematic diversity, and using candidate patches for selection. The paper also contrasts it with Libro, SWE-Agent+, EvoCoder, USEAgent, BRT Agent, and Issue2Test, and situates it against patch-centric systems such as CodeT, Agentless, CodeMonkeys, PatchPilot, and R2E-Gym. Those systems use tests to select or refine patches; e-Otter++ explicitly explores the inverse direction, using patches to help select tests (Ahmed et al., 8 Aug 2025).
The system also has clear limitations. The experiments are restricted to Python projects derived from SWE-bench benchmarks. The cost is nontrivial: the paper estimates approximately $2.75 per issue for Claude and $1.80 for GPT-4o, assuming shared Agentless patches. e-Otter++ generates only a single contributing test block in one file, whereas real projects may require multiple files or multiple tests. It assumes a buildable project, runnable tests, and a Python environment comparable to SWE-bench and TDD-Bench. The study also discusses threats to validity. A contamination concern arises because initTest and initPatch morphs may inject memorized structure; manual analysis of 50 cases found only about 2% suspicious for contamination, though the authors note residual ambiguity. The critic is helpful but imperfect, so misjudgments can terminate repair prematurely or waste iterations.
Despite these constraints, the system is designed for practical integration. The paper describes deployment scenarios in CI pipelines, issue trackers such as GitHub, and developer IDEs. It is particularly natural as a module in a larger agentic workflow in which localization and issue parsing are upstream and patch selection, regression testing, and quality assurance are downstream.
A subsequent paper, “EvoOtter: Evolutionary Reproduction Test Generator,” positions e-Otter++ as a prior inference-scaling approach and reports its 63.0% fail-to-pass result on TDD-Bench-Verified for Claude-3.7-Sonnet while arguing that evolutionary programming can sharpen feedback and reduce cost (Ahmed et al., 3 Jul 2026). This later comparison underscores the role of e-Otter++ as a pivotal baseline: it established that execution feedback on buggy code, when mediated by critic-guided repair and surrogate-patch selection, is sufficiently reliable to produce state-of-the-art reproduction tests in the issue-driven setting.