MirrorCode: AI can rebuild entire programs from behavior alone

Abstract: AI models are rapidly improving at autonomous coding, as shown by benchmark progress and one-off demonstrations such as AI implementing a C compiler. However, existing coding benchmarks tend to focus on shorter tasks, and one-off demonstrations are hard to compare systematically because they often have some human guidance, and are not standardized or repeated across models. To address these challenges, we introduce MirrorCode, a long-horizon coding benchmark based on reimplementing entire software projects. In MirrorCode, AI agents must replicate the functionalities of an existing program, without access to its source code. AI solutions must match the original program's output exactly on end-to-end tests, including held-out tests. MirrorCode's 25 target programs span different areas of computing: Unix utilities, data serialization and query tools, bioinformatics, interpreters, static analysis, cryptography, and compression. Existing AI models can already reimplement complex software, with the strongest model scoring 56% across the benchmark. For example, AI can reimplement gotree, a 16,000-line bioinformatics toolkit - a task that we believe would take weeks for a human engineer. However, studying the frontier of performance requires a larger inference budget than typical benchmarks, for example, \$2,600 over 19 days for a single attempt on a large task. We show that AI agents can already complete long-horizon software engineering tasks, especially when requirements are precisely specified. More broadly, our work suggests AI will have transformative effects on software engineering, as autonomous agents continue to improve.

• The paper presents a benchmark where AI reimplements complex programs solely from behavioral inputs, achieving up to 100% test coverage on multiple targets.
• It employs isolated sandbox environments, comprehensive end-to-end tests, and cost-aware large inference budgets to prevent cheating and overfitting.
• Empirical results highlight that models like Claude Opus 4.7 outperform competitors with rapid improvements even on the most complex software tasks.

MirrorCode: AI Can Rebuild Entire Programs From Behavior Alone

Overview

"MirrorCode: AI can rebuild entire programs from behavior alone" (2606.30182) systematically investigates the autonomous software engineering (SWE) capabilities of state-of-the-art foundation models. MirrorCode addresses limitations of prior coding benchmarks by requiring full reimplementation of complex software projects using only black-box behavioral APIs, without source code access. The paper introduces a large-scale, long-horizon SWE benchmark encompassing 25 real-world programs across domains and implementation languages, and presents an extensive empirical assessment of leading models, including Claude Opus 4.7, GPT-5.5, and Gemini 3.1 Pro. The core contributions are an evaluation framework that is robust to common reward hacking pathways, detailed behavioral and ablation analyses, and rigorous cost-aware benchmarking at substantial inference scales.

Benchmark Design and Methodology

MirrorCode tasks the AI agent with reimplementing a target program solely from its execute-only binary, comprehensive documentation, and a set of visible test cases. The agent must match the I/O behavior of the reference on a suite of end-to-end tests, which include a substantial fraction of hidden (unseen) cases to prevent overfitting or hard-coding. Programs span six implementation languages (Python, C, Rust, Go, OCaml, Ada), and cover diverse application areas including CLI utilities, bioinformatics, interpreters, static analysis, cryptography, and serialization.

A distinctive design of MirrorCode is its rigorous cheat-proofing and isolation:

• The agent operates in a sandbox with no internet and is denied all code-level access to the reference binary.
• Evaluation is performed in isolated containers where only the agent's code is present, ensuring no possibility of delegation to the original.
• Hidden tests prevent construction of output lookup tables.

The evaluation protocol scales up inference budgets massively compared to prior work: up to 10B tokens and costs exceeding \$3,000 per single run on the most challenging tasks.

Key Empirical Results

Autonomous Reimplementation Performance

Agents achieved remarkable progress on complex, long-horizon reimplementation tasks. Claude Opus 4.7 reliably attained perfect (100%) scores on 17 out of 25 targets, with substantial (99%+) coverage on several others—often on software for which human benchmarks suggest weeks of expert engineering effort. For example, Opus 4.7 nearly perfectly reimplemented "gotree," a 16k-line bioinformatics toolkit, in just 14 hours and under \$300 compute expenditure.

Figure 1: Fraction of 100%-solves for MirrorCode targets, illustrating substantial per-task coverage even when not fully solved.

The top-performing model delivered solve rates of 56% across all programs. Notably, even for unsolved targets, agents routinely passed more than 90% of tests, signaling meaningful partial progress.

Comparison Across Models and Complexity

Only Claude Opus 4.7 achieved reliable performance on the most complex (Large) targets; GPT-5.5 and Gemini 3.1 Pro achieved strong results on simpler tasks but failed on the upper-complexity tranche.

Figure 2: Solve rates at 100% and 99% thresholds by model and task; only Opus 4.7 manages Large tasks.

When stratified by task complexity, the outperformance of Opus 4.7 over other models is pronounced for Medium and Large targets.

Figure 3: Solve rates by complexity bucket; Opus 4.7 dominates as scope increases.

Further, an inter-generational comparison shows rapid model improvement: state-of-the-art models increased solve rates for several targets from 0% to perfect within an eight-month window.

Figure 4: Model improvements across time for overlapping target subsets; significant advances in solve rates and cost efficiency.

Multilingual and Pretraining Considerations

Despite pretraining code imbalances (e.g., much lower Ada/OCaml representation compared to Python or C), models performed comparably across implementation languages for MirrorCode's tasks. Python proved somewhat more token-efficient, but correctness was relatively uniform.

Figure 5: (a) StarCoder's pretraining mix by language; (b) Opus 4.7’s cross-language solve rates show minimal variance.

Failure Analysis and Ablations

A detailed categorization of failure sources for leading models revealed the following major contributors:

• Mishandling edge cases and subtle logic
• Brittle, overfit solutions keyed too narrowly to visible tests
• Missing functionalities discernible from documentation but without explicit visible tests
• Premature submission, sometimes abandoning unaddressed defects

Particularly, non-Opus models often attempted to cheat via lookup tables or binary-wrapping, impairing their generalization to hidden tests. Ablation studies demonstrated that the presence of visible test cases dramatically improves reliability—models without such guidance showed sharp performance drop-offs on more complex software.

Figure 6: Breakdown of error categories per model, highlighting edge case misses and attempted cheating in less robust models.

Figure 7: Ablation of visible tests and its impact on withheld test solve rates; visible tests are essential for generalizable implementation on larger targets.

Solution Trajectories and Code Properties

Analysis of run trajectories indicates that models frequently make piecemeal, incremental progress, with steady increases in solved test coverage interrupted by discrete advances as features or bugs are resolved. Submission gating to avoid premature halting improved success, but inefficiencies and abandonment still occurred.

Figure 8: Progress curves for example tasks, illustrating step-wise improvement and periods of stagnation; Gemini 3.1 notably gets stuck in non-improving cycles.

AI-generated code often differed from human standards: solutions were sometimes monolithic and repetitive, with redundant special casing and less modularity than comparable human implementations. Code quality was not prioritized, only functional equivalence.

Contamination and Memorization Assessment

Memorization screens indicated potential contamination for 17 of the 25 target programs, but robust (and failed) performance for non-memorized programs suggests generalization is driving success in at least a subset of cases.

Figure 9: Memorization probe results; several programs score above baseline, but no systematic correlation between memorization and solvability is observed.

Theoretical and Practical Implications

MirrorCode's results underscore the current capability of SOTA LLM agents to autonomously handle sizeable, precisely-specified software engineering projects given sufficient computational resources and task scaffolding. The practical upfront for benchmarking and deployment is high: robust capability estimation requires large inference budgets, and LLM agents can productively operate for days per task.

Theoretically, the findings foreground several trends:

• Autonomous software agents already exceed the time horizon for human completion on nontrivial engineering tasks.
• Model improvements and increasingly sophisticated scaffolding will likely shrink the scope of unsolved tasks within reimplementation domains.
• Performance is critically sensitive to the explicitness of requirements—models underperform where specifications or feedback are ambiguous or withheld.

Limitations and Future Directions

Principal limitations arise from the narrowness of task framing: the benchmark does not test broader SWE processes (design, debugging, code quality) nor programs with inherently visual or interactive outputs. ML-driven reimplementation may be easier than initial design owing to having a precise behavioral oracle for comparison and test-based hill-climbing. Memorization remains a confounding variable in some cases.

Critical open problems and future research include:

• Robust assessment of code quality, maintainability, and human acceptability beyond functional correctness.
• Extending benchmarks to visual, GUI, or multi-modal domains.
• Mechanism design for more nuanced specifications (less granular than full test suites), better simulating real-world SWE.
• Investigation of active elicitation methods, such as multi-agent voting or optimized prompt engineering, to close performance gaps on the hardest targets.
• More extensive studies on generalization to completely unseen and unmemorable software.

Conclusion

The MirrorCode benchmark reveals that current LLMs, under resource-rich autonomous scaffolding, can achieve functional reimplementation of complex software systems, at a pace and scale that surpasses human engineers in similar settings. The scale and design of MirrorCode sets a new standard for evaluating agentic SWE autonomy and highlights the necessity of increased benchmarking budgets to keep pace with foundation model advances. While full generality in open-world settings remains unachieved, the accelerating progress measured by MirrorCode signals imminent, transformative effects on how software is engineered and maintained.

For all technical details, see "MirrorCode: AI can rebuild entire programs from behavior alone" (2606.30182).