N-version Machine Learning (NVML)

• NVML is the adaptation of classical N-version programming to machine learning, employing multiple independently trained models and adjudication to reduce system-level failures.
• It uses diversity construction methods such as heterogeneous training, explicit specialization, and joint training to mitigate correlated errors.
• Sophisticated adjudication mechanisms—like majority voting, weighted soft voting, and confidence-based selection—are implemented to ensure reliable and safe decision-making.

Searching arXiv for recent and relevant papers on N-version machine learning and adjacent N-version methodologies. N-version Machine Learning (NVML) is the adaptation of classical N-version programming to machine learning: instead of relying on a single model or pipeline, a system executes multiple independently realized versions on the same task and uses an adjudicator—such as majority voting, weighted soft voting, confidence-based selection, or abstention—to produce a final decision. In the current literature, the term spans direct predictive-model settings, including API sequence recommendation and traffic sign recognition, as well as closely related methodological work on diversity training and automated N-version construction. Across these strands, the central technical question is the same: whether nominal diversity among versions is sufficient to reduce system-level failure when errors are often correlated rather than independent (Matsuda et al., 30 Apr 2026, Gao et al., 9 Jul 2025, Lee et al., 2017, Ron et al., 18 Jun 2026).

1. Conceptual lineage and scope

NVML inherits its core framing from classical N-version programming. In that formulation, multiple independent implementations of the same specification are executed in parallel, and an adjudicator—typically majority voting—selects the output. The reliability argument assumes that different versions fail independently or at least nearly independently; if that assumption holds and each version is individually fairly reliable, majority failure probability drops rapidly with the number of versions (Ron et al., 18 Jun 2026).

That classical framing maps directly onto machine learning. Independent versions correspond to independently trained or independently generated ML systems. Diversity sources correspond to different agents, base models, languages, prompts, training data slices, architectures, or inference procedures. Correlated failures or common-mode failure correspond to multiple systems making the same mistake on the same input because they share the same inductive biases, training priors, or interpretation of an ambiguous task. Adjudication corresponds to majority voting, consensus, confidence-weighted selection, or other ensemble aggregation. Reliability improvement is the hoped-for drop in system-level failure rate when combining versions (Ron et al., 18 Jun 2026).

The literature also distinguishes NVML from generic ensemble learning. In the API recommendation study, the system is described as much closer to fault-tolerant N-version design than to a standard ensemble, because it combines diverse versions, explicit modeling of reliability, and conservative acceptance or rejection to increase the trustworthiness of delivered outputs (Matsuda et al., 30 Apr 2026). A related boundary condition is that not all relevant work is about predictive ML outputs. "N-Version Programming with Coding Agents" is explicitly not about “N-version machine learning” in the usual sense of combining multiple predictive models for classification or regression, and "Galapagos" is about software N-version systems rather than ML ensembles. Even so, both are methodologically relevant because they examine automated diversity generation, validation, adjudication, and correlated failure in systems whose versions are produced by machine learning (Ron et al., 18 Jun 2026, Ron et al., 2024).

2. Diversity construction and structured specialization

A central NVML design problem is how versions are made different. The literature studies at least three distinct mechanisms: heterogeneous independently trained models, explicitly profiled complementary models, and jointly trained specialized models.

In traffic sign recognition, diversity is introduced primarily through architectural heterogeneity. The three-version system uses AlexNet, VGG16, and EfficientNetB0; the versions are trained independently on the same training set from GTSRB, and the paper does not describe different training datasets, input modalities, preprocessing pipelines, or random seeds as explicit diversity mechanisms (Gao et al., 9 Jul 2025). In API recommendation, the five versions are CodeBERT, CodeT5, MulaRec, UniXcoder, and CodeT5+, and the empirical premise is that these models have different pockets of competence, especially around head versus tail APIs (Matsuda et al., 30 Apr 2026).

A more explicit version-construction strategy appears in "Confident Multiple Choice Learning" (CMCL). The paper contrasts three objectives for training $M$ predictors $(f_1,\dots,f_M)$. Independent ensemble training minimizes

$$L_E(\mathcal D) = \sum_{i=1}^N \sum_{m\in [M]} \ell\!\left(y_i, f_m(\mathbf x_i)\right),$$

while Multiple Choice Learning minimizes the oracle loss

$$L_O(\mathcal D) = \sum_{i=1}^N \min_{m\in[M]} \ell\!\left(y_i, f_m(\mathbf x_i)\right).$$

CMCL modifies this with the confident oracle loss

$$\begin{aligned} L_C(\mathcal D)= \min_{v_i^m}\; \sum_{i=1}^N \sum_{m=1}^M \Big( & v_i^m \,\ell\!\left(y_i,P_{\theta_m}(y\mid \mathbf x_i)\right) \ &+ \beta(1-v_i^m)\, D_{KL}\!\left(\mathcal U(y)\parallel P_{\theta_m}(y\mid \mathbf x_i)\right) \Big) \end{aligned}$$

subject to

$\sum_{m=1}^M v_i^m = 1,\qquad v_i^m\in\{0,1\}.$

The key design move is that non-selected models are pushed toward a uniform predictive distribution rather than being left free to make arbitrarily confident errors outside their specialization regions (Lee et al., 2017).

CMCL adds two further mechanisms. Feature sharing injects lower-level representations across models through Bernoulli masks $$\boldsymbol\sigma_{nm}^{\ell}\sim \text{Bernoulli}(\lambda)$$, with $\lambda=0.7$ in the reported experiments, and stochastic labeling provides an efficient Monte Carlo surrogate for the KL-to-uniform term by sampling labels from $\mathcal U(y)$ (Lee et al., 2017). This produces a form of diversity that is not mere randomization: the versions are trained to be complementary experts while remaining usable together at inference time.

3. Adjudication, filtering, and abstention

NVML systems differ as much in their adjudication logic as in their version construction. The literature includes hard voting, weighted voting, soft voting, weighted soft voting, confidence-based candidate selection, and explicit rejection.

In the three-version traffic sign system, the paper studies majority voting, weighted voting, simple soft voting, weighted soft voting, LLM weighted voting, and LLM weighted soft voting. For soft voting, each model outputs a probability vector $g_i(x)$ over classes, with $(f_1,\dots,f_M)$0 and $(f_1,\dots,f_M)$1. Weighted soft voting then selects the class maximizing the weighted sum of model probabilities under weights $(f_1,\dots,f_M)$2 with $(f_1,\dots,f_M)$3 (Gao et al., 9 Jul 2025). What makes this mechanism specifically NVML rather than an ordinary ensemble rule is the source of the weights: the paper derives them from Failure Mode and Effects Analysis (FMEA), using severity and probability rather than accuracy alone. Severity levels are mapped as $(f_1,\dots,f_M)$4, $(f_1,\dots,f_M)$5, and $(f_1,\dots,f_M)$6, so severity-level-2 errors are heavily penalized (Gao et al., 9 Jul 2025).

NvRec introduces a different adjudication stack for API sequence recommendation. It first trains a Tail Analyzer, a binary classifier on concatenated query and code context, to predict whether an input is a likely tail case; if predicted tail, the request is rejected immediately (Matsuda et al., 30 Apr 2026). If predicted head, multiple recommender models generate candidate API sequences. These candidates are then filtered by one of three rules: Recorded filtering (all APIs must be present in the model profile), Head filtering (all APIs must be marked head), or Recorded + Head filtering (Matsuda et al., 30 Apr 2026).

When disagreement remains, NvRec first attempts majority voting over full API sequences. If no majority exists, it can either reject, choose the highest-scoring candidate, or defer to the best single model. The sequence-level score is defined as

$(f_1,\dots,f_M)$7

with acceptance threshold $(f_1,\dots,f_M)$8 (Matsuda et al., 30 Apr 2026). The geometric mean makes the procedure deliberately conservative: if any API in a sequence has zero successful history under the relevant model, the overall score becomes zero.

A recurring theme is that abstention is not peripheral. In NvRec, rejection rate, true accept rate, and false rejection rate are primary evaluation metrics rather than afterthoughts. In the traffic sign study, safety-aware weighting is motivated by the fact that not all mistakes are equally dangerous. This suggests that NVML is often best understood as a reliability-control architecture in which output suppression, escalation, or conservative fusion is part of the design objective rather than a deployment failure.

4. Representative empirical results

The direct empirical evidence for NVML comes from task-specific case studies, while adjacent work contributes methodological evidence about automated diversification and correlated failure.

Study	Setting	Selected reported result
"Tail-aware N-version Machine Learning Models for Reliable API Recommendation" (Matsuda et al., 30 Apr 2026)	Java API sequence recommendation	CodeT5 + MulaRec + UniXcoder with H-Filter + Simple Rejection: TAR $(f_1,\dots,f_M)$9, RR $$L_E(\mathcal D) = \sum_{i=1}^N \sum_{m\in [M]} \ell\!\left(y_i, f_m(\mathbf x_i)\right),$$0, FRR $$L_E(\mathcal D) = \sum_{i=1}^N \sum_{m\in [M]} \ell\!\left(y_i, f_m(\mathbf x_i)\right),$$1; five-version simple majority without filtering: TAR $$L_E(\mathcal D) = \sum_{i=1}^N \sum_{m\in [M]} \ell\!\left(y_i, f_m(\mathbf x_i)\right),$$2, RR $$L_E(\mathcal D) = \sum_{i=1}^N \sum_{m\in [M]} \ell\!\left(y_i, f_m(\mathbf x_i)\right),$$3
"Robust and Safe Traffic Sign Recognition using N-version with Weighted Voting" (Gao et al., 9 Jul 2025)	Three-version traffic sign recognition	Weighted Soft on GTSRB: Acc $$L_E(\mathcal D) = \sum_{i=1}^N \sum_{m\in [M]} \ell\!\left(y_i, f_m(\mathbf x_i)\right),$$4, Safety $$L_E(\mathcal D) = \sum_{i=1}^N \sum_{m\in [M]} \ell\!\left(y_i, f_m(\mathbf x_i)\right),$$5, SL2 $$L_E(\mathcal D) = \sum_{i=1}^N \sum_{m\in [M]} \ell\!\left(y_i, f_m(\mathbf x_i)\right),$$6, SL1 $$L_E(\mathcal D) = \sum_{i=1}^N \sum_{m\in [M]} \ell\!\left(y_i, f_m(\mathbf x_i)\right),$$7; under PGD with $$L_E(\mathcal D) = \sum_{i=1}^N \sum_{m\in [M]} \ell\!\left(y_i, f_m(\mathbf x_i)\right),$$8: RAcc $$L_E(\mathcal D) = \sum_{i=1}^N \sum_{m\in [M]} \ell\!\left(y_i, f_m(\mathbf x_i)\right),$$9, RSafety $$L_O(\mathcal D) = \sum_{i=1}^N \min_{m\in[M]} \ell\!\left(y_i, f_m(\mathbf x_i)\right).$$0, RSL2 $$L_O(\mathcal D) = \sum_{i=1}^N \min_{m\in[M]} \ell\!\left(y_i, f_m(\mathbf x_i)\right).$$1, RSL1 $$L_O(\mathcal D) = \sum_{i=1}^N \min_{m\in[M]} \ell\!\left(y_i, f_m(\mathbf x_i)\right).$$2
"Confident Multiple Choice Learning" (Lee et al., 2017)	Deep classification and segmentation	CMCL using 5 residual networks provides $$L_O(\mathcal D) = \sum_{i=1}^N \min_{m\in[M]} \ell\!\left(y_i, f_m(\mathbf x_i)\right).$$3 and $$L_O(\mathcal D) = \sum_{i=1}^N \min_{m\in[M]} \ell\!\left(y_i, f_m(\mathbf x_i)\right).$$4 relative reductions in top-1 error over IE on CIFAR and SVHN
"N-Version Programming with Coding Agents" (Ron et al., 18 Jun 2026)	48 admitted agent-generated implementations	Mean failure count drops from $$L_O(\mathcal D) = \sum_{i=1}^N \min_{m\in[M]} \ell\!\left(y_i, f_m(\mathbf x_i)\right).$$5 for single versions to $$L_O(\mathcal D) = \sum_{i=1}^N \min_{m\in[M]} \ell\!\left(y_i, f_m(\mathbf x_i)\right).$$6 for triples; $$L_O(\mathcal D) = \sum_{i=1}^N \min_{m\in[M]} \ell\!\left(y_i, f_m(\mathbf x_i)\right).$$7 triples exhibit zero observed failures
"Galapagos: Automated N-Version Programming with LLMs" (Ron et al., 2024)	Automated N-version software construction	$$L_O(\mathcal D) = \sum_{i=1}^N \min_{m\in[M]} \ell\!\left(y_i, f_m(\mathbf x_i)\right).$$8 of $$L_O(\mathcal D) = \sum_{i=1}^N \min_{m\in[M]} \ell\!\left(y_i, f_m(\mathbf x_i)\right).$$9 generated variants were formally verified equivalent; the system mitigated all three selected real Clang miscompilation bugs

In direct predictive-model NVML, the most important result is that reliability gains are highly configuration-dependent. NvRec’s best three-version configuration—CodeT5, MulaRec, and UniXcoder under H-Filter + Simple Rejection—achieves the highest reported TAR of $$\begin{aligned} L_C(\mathcal D)= \min_{v_i^m}\; \sum_{i=1}^N \sum_{m=1}^M \Big( & v_i^m \,\ell\!\left(y_i,P_{\theta_m}(y\mid \mathbf x_i)\right) \ &+ \beta(1-v_i^m)\, D_{KL}\!\left(\mathcal U(y)\parallel P_{\theta_m}(y\mid \mathbf x_i)\right) \Big) \end{aligned}$$0, but at a rejection rate of $$\begin{aligned} L_C(\mathcal D)= \min_{v_i^m}\; \sum_{i=1}^N \sum_{m=1}^M \Big( & v_i^m \,\ell\!\left(y_i,P_{\theta_m}(y\mid \mathbf x_i)\right) \ &+ \beta(1-v_i^m)\, D_{KL}\!\left(\mathcal U(y)\parallel P_{\theta_m}(y\mid \mathbf x_i)\right) \Big) \end{aligned}$$1. The five-version configuration reaches its highest TAR of $$\begin{aligned} L_C(\mathcal D)= \min_{v_i^m}\; \sum_{i=1}^N \sum_{m=1}^M \Big( & v_i^m \,\ell\!\left(y_i,P_{\theta_m}(y\mid \mathbf x_i)\right) \ &+ \beta(1-v_i^m)\, D_{KL}\!\left(\mathcal U(y)\parallel P_{\theta_m}(y\mid \mathbf x_i)\right) \Big) \end{aligned}$$2 with simple majority voting without filtering while reducing rejection to $$\begin{aligned} L_C(\mathcal D)= \min_{v_i^m}\; \sum_{i=1}^N \sum_{m=1}^M \Big( & v_i^m \,\ell\!\left(y_i,P_{\theta_m}(y\mid \mathbf x_i)\right) \ &+ \beta(1-v_i^m)\, D_{KL}\!\left(\mathcal U(y)\parallel P_{\theta_m}(y\mid \mathbf x_i)\right) \Big) \end{aligned}$$3, which the authors characterize as a better balance (Matsuda et al., 30 Apr 2026). This is a precise example of a core NVML trade-off: stronger reliability of accepted outputs often comes at the cost of much more aggressive abstention.

In safety-critical recognition, the traffic sign study reports that weighted soft voting is the strongest overall safety-aware mechanism. On GTSRB it achieves $$\begin{aligned} L_C(\mathcal D)= \min_{v_i^m}\; \sum_{i=1}^N \sum_{m=1}^M \Big( & v_i^m \,\ell\!\left(y_i,P_{\theta_m}(y\mid \mathbf x_i)\right) \ &+ \beta(1-v_i^m)\, D_{KL}\!\left(\mathcal U(y)\parallel P_{\theta_m}(y\mid \mathbf x_i)\right) \Big) \end{aligned}$$4 accuracy, $$\begin{aligned} L_C(\mathcal D)= \min_{v_i^m}\; \sum_{i=1}^N \sum_{m=1}^M \Big( & v_i^m \,\ell\!\left(y_i,P_{\theta_m}(y\mid \mathbf x_i)\right) \ &+ \beta(1-v_i^m)\, D_{KL}\!\left(\mathcal U(y)\parallel P_{\theta_m}(y\mid \mathbf x_i)\right) \Big) \end{aligned}$$5 safety, and the lowest severe error counts among the compared voting rules; under PGD attacks it continues to dominate majority voting in RSafety and in severity-level-2 misclassifications (Gao et al., 9 Jul 2025). In CMCL, the relevant evidence is training-side rather than task-specific adjudication: CMCL preserves the low oracle error associated with specialization while restoring usable top-1 behavior, unlike standard MCL, which can have excellent oracle error but catastrophic top-1 error (Lee et al., 2017).

The adjacent software studies matter because they test whether automatically generated diversity survives validation and whether it translates into practical reliability gains. This is methodologically close to NVML even when the artifacts are programs rather than class predictions (Ron et al., 18 Jun 2026, Ron et al., 2024).

5. Correlated failure and the independence problem

A persistent problem in NVML is that diversity does not imply independence. The strongest direct evidence for this comes from the coding-agent replication of the Knight–Leveson benchmark. For admitted versions $$\begin{aligned} L_C(\mathcal D)= \min_{v_i^m}\; \sum_{i=1}^N \sum_{m=1}^M \Big( & v_i^m \,\ell\!\left(y_i,P_{\theta_m}(y\mid \mathbf x_i)\right) \ &+ \beta(1-v_i^m)\, D_{KL}\!\left(\mathcal U(y)\parallel P_{\theta_m}(y\mid \mathbf x_i)\right) \Big) \end{aligned}$$6, with failure counts $$\begin{aligned} L_C(\mathcal D)= \min_{v_i^m}\; \sum_{i=1}^N \sum_{m=1}^M \Big( & v_i^m \,\ell\!\left(y_i,P_{\theta_m}(y\mid \mathbf x_i)\right) \ &+ \beta(1-v_i^m)\, D_{KL}\!\left(\mathcal U(y)\parallel P_{\theta_m}(y\mid \mathbf x_i)\right) \Big) \end{aligned}$$7, empirical failure rates $$\begin{aligned} L_C(\mathcal D)= \min_{v_i^m}\; \sum_{i=1}^N \sum_{m=1}^M \Big( & v_i^m \,\ell\!\left(y_i,P_{\theta_m}(y\mid \mathbf x_i)\right) \ &+ \beta(1-v_i^m)\, D_{KL}\!\left(\mathcal U(y)\parallel P_{\theta_m}(y\mid \mathbf x_i)\right) \Big) \end{aligned}$$8, and $$\begin{aligned} L_C(\mathcal D)= \min_{v_i^m}\; \sum_{i=1}^N \sum_{m=1}^M \Big( & v_i^m \,\ell\!\left(y_i,P_{\theta_m}(y\mid \mathbf x_i)\right) \ &+ \beta(1-v_i^m)\, D_{KL}\!\left(\mathcal U(y)\parallel P_{\theta_m}(y\mid \mathbf x_i)\right) \Big) \end{aligned}$$9 test cases on which two or more versions fail simultaneously, the paper defines

$\sum_{m=1}^M v_i^m = 1,\qquad v_i^m\in\{0,1\}.$0

with

$\sum_{m=1}^M v_i^m = 1,\qquad v_i^m\in\{0,1\}.$1

On $\sum_{m=1}^M v_i^m = 1,\qquad v_i^m\in\{0,1\}.$2 randomized inputs over 48 admitted versions, the study reports expected coincident failures under independence $\sum_{m=1}^M v_i^m = 1,\qquad v_i^m\in\{0,1\}.$3, observed coincident failures $\sum_{m=1}^M v_i^m = 1,\qquad v_i^m\in\{0,1\}.$4, $\sum_{m=1}^M v_i^m = 1,\qquad v_i^m\in\{0,1\}.$5, and $\sum_{m=1}^M v_i^m = 1,\qquad v_i^m\in\{0,1\}.$6, decisively rejecting independence (Ron et al., 18 Jun 2026).

The same study shows that correlation can survive nominally strong diversity manipulations. Among 146 defined cross-language $\sum_{m=1}^M v_i^m = 1,\qquad v_i^m\in\{0,1\}.$7 pairs, 81 fall exactly in the $\sum_{m=1}^M v_i^m = 1,\qquad v_i^m\in\{0,1\}.$8 bucket, meaning that many versions written in different languages fail on exactly the same campaign inputs. Exact co-failure clusters also appear across different agent systems (Ron et al., 18 Jun 2026). The diagnosis is that co-failures are concentrated in a small number of hard or ambiguous parts of the task, notably LICs #9 and #14, where many implementations computed the circumcircle of a triple rather than the minimum enclosing circle, and LICs #3 and #10, where tolerance-based angle semantics admit multiple reasonable interpretations (Ron et al., 18 Jun 2026).

This has a direct NVML interpretation. Apparent diversity—different architectures, vendors, languages, or model families—may still leave versions anchored to the same hard boundary in the task. NvRec reaches an analogous conclusion from a different angle: API recommendation is distorted by a long-tail distribution, and reliability controls become most valuable around tail APIs rather than uniformly across the output space (Matsuda et al., 30 Apr 2026). A plausible implication is that NVML failure dependence is often structured by semantically difficult regions, ambiguous labels, or rare events rather than by average marginal error alone.

6. Misconceptions, limitations, and open questions

One common misconception is that NVML is simply another name for ordinary ensembling. The literature repeatedly treats that as incomplete. NvRec combines multi-model redundancy with Tail Analyzer, API-level model profiling, candidate filtering, and rejection (Matsuda et al., 30 Apr 2026). The traffic sign work derives fusion weights from FMEA-based risk rather than generic validation accuracy (Gao et al., 9 Jul 2025). CMCL does not merely average multiple networks; it trains them to be complementary and confidence-aware so that non-specialists are discouraged from sharp predictions outside their competence regions (Lee et al., 2017).

A second misconception is that more versions are automatically better. NvRec explicitly shows otherwise: filtered five-version systems perform worse than the best three-version configuration, and the five-version system becomes strong only when filtering is removed and simple majority voting is used instead (Matsuda et al., 30 Apr 2026). This suggests that version count, calibration, and adjudication interact nontrivially.

The current evidence base also has substantial limitations. Direct predictive-model NVML evidence is domain-specific: Java API sequence recommendation on one public benchmark and traffic sign recognition on GTSRB and ArTS (Matsuda et al., 30 Apr 2026, Gao et al., 9 Jul 2025). High computational cost is explicit in NvRec because inference runs multiple recommendation models plus Tail Analyzer (Matsuda et al., 30 Apr 2026). The best three-version API-recommendation result comes with an $\sum_{m=1}^M v_i^m = 1,\qquad v_i^m\in\{0,1\}.$9 rejection rate, so reliability is achieved partly by refusing service on most inputs (Matsuda et al., 30 Apr 2026). In adjacent software N-version work, “zero observed failures” does not imply zero true failure probability, because the evaluation is finite randomized testing rather than formal reliability proof (Ron et al., 18 Jun 2026). Galapagos, by contrast, obtains formal equivalence only because its target is pure functions and LLVM IR rather than approximate learned systems; the paper itself emphasizes that there is no realistic direct analog for modern deep networks at task scale (Ron et al., 2024).

A final boundary issue concerns scope. The software papers are highly relevant to NVML, but they are not direct evidence about learned predictive systems. Their significance lies in showing that automated diversity is feasible, that validation pipelines matter, and that common-mode failure can remain strong even when surface diversity looks substantial (Ron et al., 2024, Ron et al., 18 Jun 2026).

The open questions are therefore structural rather than incremental: what kinds of diversity create genuinely different failure distributions; how hidden common-mode failures can be detected before deployment; and how disagreement among versions can be used not just for voting, but to identify ambiguity in the underlying task or labels (Ron et al., 18 Jun 2026). Taken together, the literature supports a mature definition of NVML: a fault-tolerant, reliability-oriented multi-model design paradigm in which redundancy is necessary, adjudication is consequential, abstention is often rational, and measured co-failure structure matters more than nominal diversity alone.