N-Version Programming with Coding Agents

Abstract: This paper revisits the classical concept on N-version programming in the setting of contemporary AI coding agents. Revisiting the seminal Knight-Leveson experiment, we study whether diversity across agent systems, models, and implementation languages creates diverse failure modes. Using the Knight-Leveson's, Launch Interceptor Program Specification, we evaluate 48 agent-generated implementations on a shared oracle and a campaign of 1,000,000 randomized test inputs. The results show substantial common-mode failure, along the findings of Knight-Leveson. Further analysis that many of those co-occuring failures can be traced to where is specification is particularly hard or ambiguous. We also demonstrate that diversity from coding agents provides practical benefit: across majority voting three-version units, the mean failure count drops from 387.44 for single versions to 130.99 for triples, and 11,844 N-version units exhibit zero observed failures. Our original results is the strongest evidence to date that N-Version Programming with coding agents is a useful engineering strategy.

• The paper revives classical N-version programming by evaluating fault diversity in AI-generated LIP implementations using systematic agent-model testing.
• A rigorous experiment using 69 AML tuples and a million-case differential testing campaign reveals significant correlated faults from specification ambiguities.
• Despite correlated failures, majority-vote N-version systems improve reliability, highlighting the practical benefits of ensemble approaches in AI-driven software engineering.

N-Version Programming with Coding Agents: Revisiting Fault Diversity in Agent-Generated Implementations

Introduction

This work revisits the classical foundations of N-Version Programming (NVP) in the context of modern AI-generated code. By systematically replicating the Knight experiment with contemporary coding agents, the study addresses whether diversity across agent systems, foundation models, and target languages translates into genuinely independent failure behaviors, or whether correlated faults still dominate. Strict adherence to Knight's experimental protocol—using the Launch Interceptor Program (LIP) specification and extensive randomized testing—enables robust, quantitative comparisons to the classical era.

Experimental Framework and Methodology

The study leverages five distinct agentic coding harnesses (Cursor, Claude Code, OpenAI Codex, Gemini, OpenCode), each paired with multiple model variants and three target languages (Python, Pascal, Rust), yielding a total of 69 agent–model–language (AML) tuples. Each agent receives only the original LIP specification text, 15 I/O examples, and a reference comparator; no code skeletons or algorithmic hints are provided. Each resultant implementation is subjected to a Knight-style two-tier validation: a 200-case oracle-based acceptance filter and a million-case randomized differential testing campaign.

The experimental workflow comprises:

1. AML Tuple-Based Generation: Each AML configuration produces a candidate full implementation of the LIP DECIDE function.
2. Screening: Only versions passing strict acceptance criteria are admitted (48/69; ~70%).
3. Randomized Campaign: All admitted versions are evaluated across $10^6$ test cases using a validated Python oracle, with bitwise output comparison.
4. Fault Localization and Correlation Analysis: Detailed pairwise correlation ($\phi$), cross-language/cross-agent stratification, specification-level fault attribution, and synthetic N-version system simulations.

   Figure 1: Distribution of admitted versus excluded agent–model–language implementations; agent- and language-specific admission rates are annotated.

Population-Level Failure Analysis

Analysis of campaign outcomes underscores the marked non-independence in agentic faults. The distribution of total failures across admitted versions is strongly right-tailed, with a majority (27/48) registering zero failures, while select outliers incur thousands of faults.

Figure 2: Distribution of failure counts among the 48 admitted implementations, binned to highlight the long-tailed nature of their error distribution in million-case testing.

Rigorous application of the Knight $z$-statistic decisively rejects the hypothesis of independent Bernoulli failures. While the theoretical independence model predicts approximately 115 coincident-failure cases across all versions, 429 are observed ($z=29.20$, $p \ll 10^{-100}$). Stratified per-language slices compound the effect, with e.g., the Pascal slice showing $K/\mu = 155.1$.

Figure 3: Pairwise Pearson $\phi$ co-failure heatmap among 21 non-perfect implementations; dark off-diagonal blocks indicate strong fault clusters cutting across agents and languages.

Pairwise analysis unveils multiple blocks of implementations sharing identical or highly correlated failure profiles, with clusters not constrained by any single agent or language boundary.

Impact of Agent, Model, and Language Diversity

Subsetting pairwise comparisons by language and agent reveals that neither linguistic nor agentic diversity suffices to decorrelate failures:

• Cross-language co-failure: Among defined pairs, a substantial fraction ($\phi=1$) indicates perfect alignment, not diversity.
• Cross-agent co-failure: High $\phi$ values persist even between agents drawing upon distinct API providers and model families.

  Figure 4: Distributions of pairwise $\phi$ correlations, juxtaposed for cross-language and cross-agent pairs; perfect agreement is dominant.

The structural commonality in errors demonstrates that, as in human programming, diversity of interface is insufficient when root causes are specification-centric or universally misinterpreted by contemporary LLMs.

Root Cause Localization: Specification-Driven Faults

Analysis at the Launch Interceptor Condition (LIC) level identifies two predicates (LIC 9 and 14) as dominant sources for failure concentration. These relate to geometric inclusion (minimum enclosing circle predicates), where multiple agents adopt a circumcircle calculation—incorrect in the context—demonstrating convergent specification misinterpretation.

Figure 5: Per-LIC histogram of distinct AML tuples exhibiting errors, stratified by language and agent; failures concentrate almost exclusively in LIC 9 and 14 across the board.

Implementation inspection confirms that model-generated solutions, regardless of agent or target language, repeatedly commit logically equivalent algorithmic errors, vindicating earlier observations in human teams.

N-Version Systems: Practical Benefits Under Correlated Failures

Despite pronounced correlation, majority-vote N-version units provide nontrivial reliability improvements. Comprehensive simulation of all possible 3-version combinations reveals:

• The mean triple-unit failure count drops from 387.44 (single) to 130.99.
• 68.5% of triple units are failure-free compared to 56.3% of individual implementations.
• The high-fault tail is substantially compressed, emphasizing mitigation even when strict independence is unattainable.

  Figure 6: Comparative failure count distributions for single implementations and all possible majority-vote triples, with clear concentration toward lower-error bins in the triples.

These gains underscore the pragmatic utility of NVP with coding agents as robustifiers, not just when independence holds, but even under strong common-mode failure dynamics.

Implications, Limitations, and Future Directions

This empirical revisiting substantiates theoretical cautions from Littlewood, Bishop, and others: specification-induced common-mode errors remain centrally challenging, even for LLM-based generators. Agent and language diversity must be complemented with orthogonal diversity (e.g., methodology, training data curation, explicit fault injection) to realize the full reliability dividends posited by NVP.

The results also reinforce the diagnostic utility of multi-version testing: clusters of recurring faults and correlated failures pinpoint specification weaknesses and ambiguous requirements, inviting automated or semi-automated refinement.

Future research trajectories are evident:

• Specification Disambiguation: Automated identification of persistent fault clusters as signals for ambiguous or ill-posed requirements.
• Diversity Augmentation: Exploring synthetic diversity—purposeful perturbation, algorithmic diversity, or training-time interventions—to break specification-driven common modes.
• Dynamic Generation Regimes: Incorporating dynamic generation and repair in ensembles, as real system requirements drift and agent capabilities evolve.

Conclusion

N-version programming, instantiated with modern coding agents, is both operationally feasible and demonstrably beneficial for reliability enhancement. However, the classical independence assumption is systematically violated due to deep specification-centric convergence in fault modes, paralleling foundational results from the human-programmer era. Nevertheless, majority-vote redundancy among coding agent outputs confers substantial practical error reduction. These findings motivate a broader integration of ensemble and diversity-based methodologies in AI agentic software engineering, anchored by rigorous differential testing and informed by specification quality.