EvoOtter: Evolutionary Reproduction Test Generator

Abstract: Before fixing an issue, it is useful to first reproduce it by generating a bug reproduction test (BRT). However, generating a BRT is itself a challenging task, because issue descriptions tend to be informal, making it difficult to determine whether a candidate BRT indeed fails for the reason in the issue. Prior work has attempted to tackle this problem via inference scaling, using LLMs to generate many BRTs and patches, then using execution feedback to select and improve them. Unfortunately, this is expensive and the feedback is unreliable. This paper explores evolutionary programming for BRT generation to sharpen the feedback, while enhancing evolutionary programming to keep costs in check. Our new approach, EvoOtter, controls test execution costs via successive halving. Furthermore, it controls LLM costs via batched crossover for an entire generation in a single LLM call, as well as via rule-based code mutations, with a new fitness score tailored for BRTs. As a result, EvoOtter generates state-of-the-art quality BRTs at the fraction of the cost of prior inference-scaling approaches to this problem. More broadly, this paper points at how to efficiently and effectively combine evolutionary programming with LLMs for software engineering.

• The paper introduces EvoOtter, an evolutionary framework that generates fail-to-pass bug reproduction tests using mutation-based fitness evaluation.
• It employs heterogeneous and single prompting methods to balance diversity and inference cost, achieving state-of-the-art performance on TDD-Bench and SWE-Rebench.
• The methodology integrates rule-based code mutants, successive halving, and batched LLM crossover to enhance both test quality and operational efficiency.

EvoOtter: Evolutionary Reproduction Test Generation for Software Engineering

Problem Formulation and Motivation

EvoOtter addresses the challenge of automated bug reproduction test (BRT) generation given only an informal issue description and outdated code ($c_\textrm{old}$), aiming to produce fail-to-pass tests ($t_\textrm{gen}$) that fail on buggy code but pass once the issue is resolved. The lack of reliable feedback due to the absence of the fixed code ($c_\textrm{new}$) complicates selection and improvement of BRTs. Existing inference-scaling approaches that rely on LLMs for generating and refining BRTs are expensive and suffer from unreliable feedback. EvoOtter proposes evolutionary programming with tailored enhancements to overcome these obstacles by sharpening feedback while controlling execution and inference costs.

Figure 1: EvoOtter workflow overview, demonstrating the integration of heterogeneous prompting, mutation-based selection, and batched crossover within an iterative evolutionary loop.

EvoOtter Methodology

Initial Candidate Generation

EvoOtter initializes with $n$ diverse tests using BaseBRT, which combines multi-step LLM-based localization (file, function, body selection), contextualization (import and location optimization), and test generation. To maximize diversity, EvoOtter employs heterogeneous prompting (description morphing, context masking), overcoming the constraints of recent frontier LLMs with disabled temperature controls.

Figure 2: The base BRT generation workflow, highlighting sequential localization, contextualization, and test synthesis.

Evolutionary Loop and Cost Control

The main evolutionary loop is inspired by successive halving: each generation halves test candidates (predators) and doubles code mutants (prey). Code mutants are produced via rule-based transformations on localized functions—without LLM involvement—applying nine mutation types (arithmetic, logical, constant replacement, assignment, force-failure, etc.), ensuring robust fitness assessment and minimizing inference cost.

Figure 3: The timeline of EvoOtter’s evolutionary process, visualizing exponential halving of tests and doubling of mutants per generation.

Fitness, Selection, and Crossover

Fitness is defined as the “mutation score”—the number of mutants killed, determined by execution log differentiation rather than simple test outcomes. When mutation scores are indistinguishable, EvoOtter falls back to LLM-based selection using issue descriptions, test cases, and logs. Crossover is accomplished through a single batched LLM call, merging statements, augmenting diversity, and leveraging execution evidence. This approach avoids the exponential LLM call costs of prior repair-based methods.

Single Prompting vs. Heterogeneous Prompting

EvoOtter also investigates the use of single-prompt batch generation of initial populations. While this reduces diversity marginally (with a minor performance drop), it drastically cuts inference costs, enabling deployment of stronger reasoning models such as Claude-Opus-4.7.

Figure 4: Distribution of \failtopass tests in candidate populations, comparing heterogeneous and single prompting across TDD-Bench-Verified.

Empirical Evaluation

EvoOtter is evaluated on three prominent repository-level BRT benchmarks: TDD-Bench-Verified, SWT-Bench-Verified, and SWE-Rebench. On TDD-Bench-Verified and SWE-Rebench, EvoOtter achieves state-of-the-art fail-to-pass rates at substantially lower cost than inference-scaling baselines. With heterogeneous prompting and Claude-Sonnet-4.5, EvoOtter produces 65.9% fail-to-pass on TDD-Bench-Verified and 51.9% on SWE-Rebench. With single prompting and Claude-Opus-4.7, the rates rise to 75.3% and 66.3%, respectively, at reduced cost.

EvoOtter’s mutation-based selection outperforms LLM-based selection and random selection after sequential evolutionary phases, attributed to favoring tests with superior mutant-killing ability.

Figure 5: EvoOtter performance on TDD-Bench-Verified, showing high fail-to-pass rates and coverage across test generations.

Crossover is critical: it enables evolution of fail-to-pass tests even when initial candidates lack them, especially in populations selected via mutation score or random sampling. This synergistic effect is less pronounced when LLM-based selectors are used, likely due to selection bias and reduced diversity.

Figure 6: Coverage evolution in generated tests, showing improvement in both fail-to-pass and non-fail-to-pass tests during EvoOtter’s iterative process.

EvoOtter demonstrates robust performance across contamination-free post-cutoff instances, indicating resilience to LLM data memorization.

Analysis and Implications

The results establish EvoOtter’s evolutionary programming framework as an efficient, reliable, and practical method for BRT generation, overcoming unreliable feedback and cost barriers. The integration of mutation testing, successive halving, and batched LLM crossover efficiently leverages LLM capabilities. EvoOtter’s mutation-driven selection and repair are particularly effective—mutation scores correlate strongly with fail-to-pass behavior and test coverage, and the rule-based mutants provide broad applicability and competitive performance even when used in isolation.

Theoretical implications include the viability of evolutionary programming for AI-assisted code validation and its extensibility to other software engineering domains. Practically, EvoOtter can be adapted to cost-sensitive, contamination-resistant environments, supporting large-scale automated software testing and repair pipelines.

The impact on future AI developments is substantial. EvoOtter’s framework provides a template for integrating evolutionary search heuristics with LLM-driven workflows, overcoming the brittle nature of prompt engineering and execution feedback. Further research may explore larger population sizes, extended mutant categories, and direct generalization to other programming languages and contexts.

Conclusion

EvoOtter demonstrates that evolutionary programming, carefully adapted with mutation-based feedback and efficient scaling, produces high-quality, cost-effective bug reproduction tests from informal issue descriptions and legacy code. Its robust empirical performance and methodological innovations open new avenues for LLM-powered software engineering automation and research.