Test-Oriented Programming: rethinking coding for the GenAI era

Abstract: LLMs have shown astonishing capability of generating software code, leading to its use to support developers in programming. Proposed tools have relied either on assistants for improved auto-complete or multi-agents, in which different model instances are orchestrated to perform parts of a problem to reach a complete solution. We argue that LLMs can enable a higher-level of abstraction, a new paradigm we called Test-Oriented Programming (TOP). Within this paradigm, developers only have to check test code generated based on natural language specifications, rather than focusing on production code, which could be delegated to the LLMs. To evaluate the feasibility of this proposal, we developed a proof-of-concept tool and used it to generate a small command-line program employing two different LLMs. We obtained promising results and identified challenges for the use of this paradigm for real projects.

• The paper demonstrates that delegating production code generation to LLMs, via exclusive test artifact validation, is viable in constrained settings.
• The methodology leverages a CLI tool (Onion) with declarative YAML configurations to orchestrate iterative LLM prompt engineering for test and code synthesis.
• Empirical results reveal that errors in test code generation are the primary cause of failures, highlighting the need for scalable automated meta-verification.

Test-Oriented Programming: Rethinking Coding for the GenAI Era

Introduction

The proliferation of LLMs in software engineering is driving a reassessment of conventional development workflows. While current LLM-powered assistants and multi-agent systems have enhanced the productivity of developers, they essentially automate code synthesis without altering the fundamental abstraction level of programming: developers are still required to inspect, modify, and reason about production code. The paper "Test-Oriented Programming: rethinking coding for the GenAI era" (2604.08102) introduces the paradigm of Test-Oriented Programming (TOP), advocating a shift wherein developers interact exclusively with test artifact generation and validation. The synthesis of production code is entirely delegated to automated LLM systems, guided by natural language specifications and developer-verified test suites.

The Test-Oriented Programming (TOP) Paradigm

TOP extends the abstraction boundary beyond what is offered by TDD and low-code approaches. In TDD, developers alternate between specifying tests and writing production code, maintaining code and test at parity in abstraction. Low-code and domain-specific tools abstract away some syntactic and infrastructural complexities but target niche requirements and are not designed for general-purpose programming by professional developers. TOP, in contrast, positions test artifacts as the sole developer-facing code; all production code emanates from LLM-driven synthesis, triggered and validated solely via test conformance. Critical ambiguity resolution—an inherent risk in natural language—shifts to explicit, verifiable artifacts (i.e., test code), thereby combining the expressivity of NL-driven specs with the rigor of formal verification.

Importantly, TOP formally separates concerns. Developers specialize in specifying intent and verifying test correctness, not manual algorithmic implementation. The paradigm is model-agnostic and does not require domain-specific limitations inherent to low-code solutions.

Proof-of-Concept: Onion Tool and Empirical Results

The authors materialize TOP in Onion, an iterative CLI tool leveraging declarative YAML configuration files capturing high-level system goals, dependencies, and acceptance criteria described in NL. Onion orchestrates LLM-based code and test generation through prompt engineering, with human oversight limited to the modification and verification of the test code or system structure as needed.

Empirical assessment centered on synthesizing a CLI application for managing and querying entries. Experiments were conducted with OpenAI GPT-4o-mini and Gemini 2.5-Flash, contrasting reasoning-focused and non-reasoning LLMs. In multiple independent trials for each model:

• Developers did not intervene in production code, only in generated test code and occasionally in prompts/configuration.
• Failures in test code generation, not production code, accounted for most breakdowns in synthesis. Augmenting or clarifying the tests remedied the failures.
• For both LLMs, minor divergences in test code required fixing to facilitate successful production code synthesis.
• Notable divergence was observed in code verbosity and comment density: GPT-4o-mini code was more concise, whereas Gemini 2.5-Flash produced longer, heavily commented code.
• Output determinism remained a challenge, as repeated synthesis led to varied outcomes—even with the same LLM.

These results confirm that end-to-end code synthesis from rigorous test artifacts with minimal production code inspection is feasible in constrained settings. However, the volume and verification of generated test code present scalability challenges for human validators, raising the need for automated meta-verification techniques.

Theoretical Implications

TOP reframes the abstraction boundary between specification and implementation. It operationalizes the expressive power of LLMs to automate routine code generation reliably, provided test specifications are unambiguous and exhaustive. As with model-driven development or formal methods, the correctness guarantee in TOP becomes coterminous with the quality and completeness of the test suite, increasing the pressure on test design as the critical point of failure. The paper's empirical evidence that developer effort can be centered almost entirely on configuration and testing—while still completing non-trivial tasks—is a strong claim about LLM maturity for synthesis under well-posed constraints.

TOP challenges canonical boundaries between developer and machine agency in software development. It is especially relevant in high-change, requirements-driven or regulatory contexts, where aligning specification and executable contracts is paramount.

Practical Implications and Future Prospects

From a practical lens, the paper identifies several open research directions:

• Scalability of Test Verification: As systems scale, the human effort required to validate machine-generated test code becomes significant. This introduces opportunities for secondary automated verification or even machine-generated meta-tests.
• Prompt Engineering Sensitivity: The models' performance and test code generation are prompt-sensitive and model-specific, necessitating careful alignment.
• Variance in Synthesis Output: LLM determinism remains an open problem, possibly impacting reproducibility and consistency in larger projects.

Integrating TOP into real-world workflows will likely require modular decomposition (e.g., per-microservice) and robust methods for test code validation. Effective deployment may also necessitate automated test reduction, coverage analysis, oracles for test soundness, and possibly contract-driven test generation from formal NL specs.

Conclusion

The introduction of Test-Oriented Programming redefines the role of human developers in the GenAI era. Delegating production code authoring to LLMs while centering software design and validation around test artifacts represents an increased level of abstraction over mainstream LLM-based coding assistants. Initial experiments validate the paradigm's feasibility but also highlight verification and model variance challenges, especially relevant for scaling to enterprise-grade software. Further research into automated test code validation, deterministic synthesis, and meta-verification tooling will be vital for operationalizing TOP in practice and for solidifying its position as a general-purpose approach to software engineering in the era of generative AI.