TEVV: Testing, Evaluation, Verification & Validation

• TEVV is a family of technical and managerial activities designed to ensure systems meet performance, quality, and safety requirements through rigorous testing, evaluation, verification, and validation.
• It incorporates diverse methodologies including simulation-based testing, digital twins, continuous integration, and scenario analysis to address challenges in software, autonomy, and AI.
• TEVV emphasizes traceability, stakeholder alignment, and evidence-based assurance to systematically manage risks and improve system reliability throughout the lifecycle.

Testing, Evaluation, Verification and Validation (TEVV) denotes a family of technical and managerial activities used to improve system quality, system reliability, and assurance that a product satisfies users’ operational needs (Al-Neaimi et al., 2012). Across systems engineering, software assurance, autonomy, and digital twins, TEVV spans execution-oriented testing, broader evaluation of effectiveness and utility, verification of conformance to requirements and design intent, and validation of adequacy with respect to stakeholder needs and real-world behavior (Waters, 6 Jul 2025). Recent work also treats verification and validation as knowledge-building activities, emphasizing evidence, confidence, traceability, and consistency rather than isolated end-of-line checks (Kannan et al., 11 Apr 2025).

1. Conceptual foundations and terminology

Terminology is not fully uniform across the literature. In multi-agent-system guidance, verification is defined as “the assurance that the products of a particular development phase are consistent with the requirements of that phase and preceding phase(s),” while validation is “the assurance that the final product meets system requirements”; testing is a supporting mechanism inside this broader V&V process, and assessor independence is emphasized through Independent V&V (IV&V) or internal-but-separate V&V (Al-Neaimi et al., 2012). Maritime-autonomy work uses the terms more reconstructively: verification concerns whether software or AI satisfies intended logical requirements, validation concerns whether the algorithm behaves safely and acceptably in representative settings, testing is the practical mechanism for exercising scenarios, and evaluation is performance assessment in those tests (Porres et al., 2021).

Digital-twin work makes the distinctions explicit in a standard systems-engineering form. Testing examines whether the twin functions as intended under controlled conditions; evaluation assesses effectiveness, usefulness, usability, performance, and value in practice; verification asks whether the twin has been built correctly according to specifications and formal properties; validation asks whether the right twin has been built, namely whether it adequately represents the real-world system and intended use (Waters, 6 Jul 2025). This sharper partition is useful because TEVV evidence is often heterogeneous: an executable test may establish local correctness, while a field comparison or pilot study speaks to representational adequacy.

At a more formal level, V&V can be modeled as epistemic activities in which evidence generated by verification or validation changes an agent’s epistemic state and justifies beliefs about criteria, requirements, needs, and goals (Kannan et al., 11 Apr 2025). In that account, inconsistency in needs, requirements, criteria, or activities makes V&V claims invalid, and the overlap between verification and validation depends on the logical relation between requirements and needs. This suggests that TEVV is not only a collection of procedures but also an evidence-structuring discipline.

2. Lifecycle placement, governance, and alignment

A recurring theme is that TEVV is lifecycle-wide rather than a terminal gate. In MAS guidance, a V&V plan is established first, requirements, design, implementation, and application each receive dedicated V&V activities, reverse transitions are expected when defects are found, and results are integrated and archived in a repository (Al-Neaimi et al., 2012). Digital-twin work expresses the same idea through both a lifecycle loop—concepts and requirements, creation, model, update, data integration, verification and validation, real-time monitoring and analytics, optimization and decision support, decommissioning or evolution—and a project workflow of planning, design, execution, analysis, reporting, and continuous improvement (Waters, 6 Jul 2025).

The organizational dimension is equally central. A six-company case study of RE–VV alignment found that weak alignment degrades test completeness, coverage of changed requirements, regression selection, and confidence in delivered quality; the identified challenge areas include aligning goals and perspectives, cooperating successfully, requirements specification quality, VV quality, maintaining alignment under change, abstraction-level mismatches, traceability, time and resources, large document spaces, and outsourcing or offshoring (Bjarnason et al., 2023). The same study found that human aspects are central, especially cooperation and communication, and that requirements engineering practices are a critical basis for alignment. In practical TEVV terms, this means that traceability, change control, and shared design intent are part of the assurance mechanism itself, not ancillary process hygiene.

For learning-enabled autonomous systems, recent mapping work organizes assurance techniques against a traditional systems-engineering V-model under three top-level categories—development, acquisition, and sustainment—so that TEVV planning can inform analysis of alternatives and communicate risk to leaders without requiring radical changes to existing systems-engineering processes (Ellis et al., 2022). A parallel digital-engineering approach for off-road autonomous vehicles pushes this further by coupling digital twins with MBSE and MBD, enabling traceable requirements engineering, efficient variant management, systematic test-case definition, automated execution, and report generation “with traceability and tractability across the digital thread” (Samak et al., 18 Mar 2025).

3. Methods, evidence modalities, and automation

TEVV practice spans review-based, formal, simulation-based, oracle-based, and continuous-integration modes. MAS guidance groups techniques into formal, semiformal, hybrid, and conventional classes, and repeatedly highlights traceability analysis, compliance checking, interface analysis, requirements evaluation, design evaluation, criticality assessment, and peer reviews as core V&V activities (Al-Neaimi et al., 2012). These activities are document- and model-centric, but they are routinely complemented by executable testing.

In autonomous maritime navigation, the dominant evidence modality is simulation. A systematic mapping study retrieved 427 papers, retained 132 after screening, and found that 86 out of 132 used simulation-based validation, generally with only 1 to 12 manually designed scenarios; three studies used either a real boat or a model boat, and the remainder used no verification and validation approach (Porres et al., 2021). Because such small handcrafted sets provide little confidence in rare but safety-critical corner cases, the paper advocates systematic scenario-based testing and reports a maritime-specific experiment in which 6000 simulation scenarios were created to test a reinforcement-learning-based collision-avoidance algorithm, with evaluation criteria including risk of collision and compliance to COLREGs (Porres et al., 2021). The same literature also cites Perera’s three-level validation scheme: Level 1 software simulation for all vessels, Level 2 restricted-water testing with own ship physical and others simulated, and Level 3 open-sea testing with all involved vessels physical (Porres et al., 2021).

Executable oracles provide a different kind of evidence. In a specification-based software-testing framework, VDM++ specifications are translated into C++ oracle classes that generate expected results and compare them with implementation outputs; the oracle is explicitly decomposed into an expected result generator and a comparator, and the method supports inheritance and aggregation at class level, though not concurrency (Ahmad et al., 2014). This is TEVV in a narrow but important sense: verification by conformance testing against a formalized behavioral reference.

Continuous, automated TEVV has also become a first-class concern. For OpenMP implementations, a CI/CD workflow organizes setup, build, testing, and cleanup, uses OpenMP VV primarily for verifying compiler correctness and SPEChpc benchmarking for evaluating implementation quality, and runs across heterogeneous HPC platforms and compilers (Jarmusch et al., 2024). In smart-contract ecosystems, the surveyed VV&T solution space is organized into public test networks, security analysis tools, blockchain emulators, and blockchain simulators, each offering a different balance of realism, configurability, confidentiality, and predefined versus user-defined checks (Benabbou et al., 2021).

4. AI-specific TEVV: brittleness, scenario spaces, and human involvement

AI systems, especially deep neural networks, expose TEVV stressors that differ from conventional software. In safety-critical framing, current DNN performance is far from the reliability culture reflected in DAL, ASIL, and SIL guideposts: aviation DAL A is placed at about $10^{-9}/h$, DAL D at about $10^{-3}/h$, automotive ASIL D around $10^{-8}/h$, and IEC 61508 SIL 4 around $10^{-8}/h$ (Lohn, 2020). Against those guideposts, ImageNet-level top-1 accuracies in the high 80% range and top-5 accuracies in the high 90% range imply per-use failure rates greater than $10^{-2}/use$, and at 10 images per second one would need accuracy $0.99999997$ merely to reach aviation’s lowest level $10^{-3}/h$ (Lohn, 2020). The same work argues that TEVV for AI must center on brittleness estimation across distributional boundaries: current practice overemphasizes in-distribution holdout accuracy, while real deployment requires estimating how performance degrades as inputs become out-of-distribution (OOD), and that degradation often appears gradual rather than cliff-like (Lohn, 2020).

Scenario realism and environment-coupled degradation become especially important in autonomy. A maritime simulation framework for MASS adds weather severity controls for rain, fog, and sea state, weather-induced radar propagation loss and SNR degradation, bathymetric terrain ingestion from NOAA/USGS data, depth-aware occupancy grids, and depth-informed wave-load computations (Patil et al., 3 Mar 2026). The reported integrated performance indicators are Minimum Passing Distance, $RMSE_V$, and $RMSE_\psi$, and the framework exposes failure modes such as uncertainty-induced premature collision avoidance rather than only outright collision (Patil et al., 3 Mar 2026). In off-road autonomy, a digital-engineering workflow evaluates an autonomous light tactical vehicle against a battery of 128 procedurally generated test cases formed by $2\times2\times2$ subsystem variants crossed with 4 times of day and 4 weather conditions; requirement checks include number of detections greater than 1, peak jerk less than $10^{-3}/h$0, average estimated velocity error within $10^{-3}/h$1, and zero collisions (Samak et al., 18 Mar 2025).

Human involvement also becomes part of the system under test. Human-centred military-AI work argues that for most military applications the relevant system is the human-machine team rather than the algorithm alone, so TEVV must include the effect of the human to reflect operationally realized system performance, continue across the lifecycle, and communicate residual risk to operators and decision-makers (Helmer et al., 2024). A related automotive vision paper proposes explainable testing methodologies in which requirements are refined through literature review and stakeholder input, safety-critical scenarios are generated by LLMs in structured formats, simulation-based validation runs in real time, and outputs include a test oracle, explanation generation, and a test chatbot (Eris et al., 20 Jun 2025). This suggests a shift from pure defect detection toward diagnosable, interpretable evidence.

5. Domain-specific evidence and metric regimes

TEVV metrics are highly domain-specific, and the literature repeatedly warns against mistaking surrogate measures for complete assurance. In maritime navigation, 82% of surveyed papers defined safety using TCPA or DCPA, and only 48% were judged to comply with COLREGs in their study design, illustrating how easily evaluation can collapse onto proximity-based surrogates (Porres et al., 2021). By contrast, digital-twin work enumerates technical, usability, reliability, and value metrics in the same framework, while combustion-CFD work combines code-to-code verification, comparison with experiment, and numerical-sensitivity analysis (Waters, 6 Jul 2025, Zhukov, 2011).

Domain	TEVV focus	Example metrics or evidence
Maritime autonomous navigation in adverse weather	Scenario-based V&V under rain, fog, sea state, and bathymetric constraints	MPD, $10^{-3}/h$2, $10^{-3}/h$3, collision-trigger activations (Patil et al., 3 Mar 2026)
Off-road autonomous ground vehicles	Requirement-linked digital-thread V&V over 128 procedurally generated tests	detections $10^{-3}/h$4, peak jerk $10^{-3}/h$5, average estimated velocity error within $10^{-3}/h$6, collisions $10^{-3}/h$7 (Samak et al., 18 Mar 2025)
Digital twins	Continuous lifecycle TEVV of dynamic cyber-physical models	MAE, RMSE, $10^{-3}/h$8, NRMSE, F1 Score, MTBF, Availability Percentage, SUS Score, ROI (Waters, 6 Jul 2025)
Hydrogen combustion CFD	Verification against CHEMKIN/PREMIX and validation against experiment	ignition delay, laminar burning velocity, grid spacing sensitivity, CPU hours per simulated millisecond (Zhukov, 2011)
OpenMP implementations on HPC platforms	Continuous validation, verification, and benchmarking across compilers and accelerators	OpenMP VV pass counts, SPEChpc runtime, build error, execution error (Jarmusch et al., 2024)

The examples show that TEVV evidence may target different objects: software correctness, model fidelity, closed-loop controller behavior, operational utility, or compile-time/runtime quality. They also show that metric choice is inseparable from system decomposition. For LEAS, this is especially acute because assurance may be needed at the whole-system level, at the learning-enabled component level, and at the traditional component level, each with different observables and failure modes (Ellis et al., 2022).

6. Open questions and research directions

Several research questions recur across domains. For AI assurance, open questions include how OOD should be classified in a way useful to certifiers, which distance measures correlate with failure, how frequently different severities of shift occur in deployment, what performance thresholds should be required at each OOD level, and when OOD detection is sufficient versus when residual competence under shift must itself be certified (Lohn, 2020). Digital-twin research raises parallel problems around continuous and online validation, uncertainty quantification, interoperability standards, self-learning twins, and scalable TEVV for interconnected twins (Waters, 6 Jul 2025).

Organizationally, RE–VV alignment remains a persistent bottleneck. The industrial case-study literature shows that better verification and validation do not begin with more tests alone; they also depend on clearer and more verifiable requirements, cross-role reviews, change-management practices that involve testing roles, fit-for-purpose traceability structures, and tool support that does not make maintenance prohibitively expensive (Bjarnason et al., 2023). For military AI, an additional unresolved issue is human scalability: AI systems may require very large scenario spaces, but human-centred TEVV often has only tens of participants, leaving residual unknown risk difficult to characterize (Helmer et al., 2024).

Foundational work suggests a more formal research agenda. If V&V are treated as knowledge-building activities, then consistency of needs, requirements, criteria, and activities becomes a precondition for meaningful assurance, minimal valid subsets become relevant for reducing TEVV burden, and model-checkers or reasoning engines integrated into MBSE environments become a plausible next step (Kannan et al., 11 Apr 2025). At the lifecycle level, mapping techniques into development, acquisition, and sustainment provides a pragmatic way to plan comprehensive test and evaluation and to communicate risk objectively to leadership (Ellis et al., 2022). The resulting picture is not of a single TEVV method, but of a layered discipline in which requirements quality, executable evidence, formal reasoning, operational realism, and governance all interact.