Whole-program reproduction from scratch remains unsolved

Establish reliable whole-program reproduction from scratch that achieves complete behavioral fidelity rather than only partial test-suite agreement.

Background

ProgramGym evaluates agents on program-reproduction tasks in which a system must reconstruct a program’s externally observable behavior from documentation and an executable reference, without access to the reference source code. The reported results show that test-pass rates can exceed 50%, while the percentages of almost-resolved and fully resolved tasks remain close to zero. In particular, the paper notes that flagship models achieve no fully resolved tasks under the evaluated setup, and that fine-tuning does not consistently improve high-fidelity completion. The unresolved challenge is therefore to develop methods that can reproduce entire programs from scratch with complete behavioral fidelity, rather than merely recovering a substantial subset of tested behaviors.