New Assessment/Test Method (NATM)

• NATM is a family of assessment methods that replaces narrow, one-shot evaluations with multi-component evidence and explicit criteria.
• In automated driving, the method uses a scenario-based, multi-pillar framework that integrates simulation, track trials, real-world testing, audits, and post-deployment monitoring.
• Across domains, NATM standardizes evidence production by hybridizing data sources and controlled test environments to enhance validity and traceability.

Searching arXiv for the cited NATM-related papers to ground the article. “New Assessment/Test Method” (NATM) is not a single universally standardized technique across the research literature. In the sources considered here, the term denotes either a specific regulatory framework—most prominently the UNECE WP.29/GRVA framework for safety assessment of Automated Driving Systems (ADS)—or a domain-specific designation for newly structured assessment procedures in education, clinical imaging, intelligent tutoring systems, psychometrics, software engineering, and human-factors experimentation (Camp et al., 30 Jul 2025, Slepkov, 2013, Perkins et al., 2024, Hou et al., 2024, Sheng et al., 2023, Harmankaya et al., 2024, Harmankaya et al., 2024). Across these usages, NATM consistently refers to an assessment architecture that replaces narrow, one-shot, or purely manual evaluation with multi-component evidence, explicit criteria, and stronger operational traceability.

1. Scope and principal usages

In current usage, NATM is best understood as a family resemblance term. In one strand, it is the proper name of a regulatory framework for ADS safety assurance. In other strands, it denotes a newly proposed assessment or testing protocol tailored to a specific problem domain.

Domain	NATM form	Citation
Automated driving	UNECE WP.29/GRVA multi-pillar, scenario-based ADS safety assessment	(Camp et al., 30 Jul 2025)
Educational assessment	IF-AT with integrated testlets; AI Assessment Scale; rule+LLM grading	(Slepkov, 2013, Perkins et al., 2024, Wang et al., 18 Aug 2025)
Intelligent tutoring systems	Adaptive and implicit IRT-based language proficiency assessment	(Hou et al., 2024)
Clinical imaging	“Noise-Aware Template Matching” for retinal video stabilization in SVP assessment	(Sheng et al., 2023)
Psychometrics	Single-administration CTT reliability estimation via fast dichotomisation	(Chakrabartty et al., 2015)
Software testing and QA	Diagnostic test generation, SPL mutation-based test assessment, test automation maturity self-assessment	(Kim et al., 2021, Lackner et al., 2015, Wang et al., 2019)
Human-factors experimentation	Compact-track recreation of on-road motion-sickness exposure	(Harmankaya et al., 2024)

This diversity matters because the acronym does not by itself identify a unique method. A plausible implication is that “NATM” functions less as a canonical technical standard than as a marker for assessment redesign: a method becomes “new” when it changes what is measured, how evidence is collected, and how validity is argued.

2. NATM as the UNECE scenario-based framework for automated driving

The most formalized use of NATM in the present sources is the ADS safety framework under development within the UNECE WP.29 Working Party on Automated/Autonomous and Connected Vehicles (GRVA). It is already recognized in major markets including Japan, South Korea, the EU, and the USA (Camp et al., 30 Jul 2025). The framework arises from the inadequacy of traditional vehicle type-approval, which has relied on a relatively small number of prescribed physical tests, deterministic test conditions, and human drivers as the default reference. For ADS, that model is insufficient because the relevant traffic-situation space is virtually infinite, software-intensive AI-based systems require evidence over distributions of scenarios, safety must be assessed across the whole lifecycle, and many rare but critical situations are unsafe or impractical to test exclusively on physical tracks (Camp et al., 30 Jul 2025).

NATM therefore adopts a multi-pillar structure. The pillars are virtual testing, track testing, real-world testing before deployment, an audit pillar, and in-service monitoring and reporting (ISMR) (Camp et al., 30 Jul 2025). Virtual testing covers large numbers of scenarios quickly and safely; track testing validates and complements simulation; real-world testing covers controlled public-road trials; the audit pillar addresses the manufacturer’s Safety Management System, internal procedures, and the tools and methods used for testing and validation; ISMR provides the post-deployment feedback loop (Camp et al., 30 Jul 2025).

Its conceptual core is scenario-based assessment. Scenarios serve two distinct functions: they describe the ADS Operational Design Domain (ODD), and they drive testing through concrete test scenarios instantiated from more generic descriptions (Camp et al., 30 Jul 2025). The paper adopts the functional–logical–concrete classification aligned with PEGASUS and many EU projects: functional scenarios are high-level descriptions, logical scenarios add parameter ranges, and concrete scenarios fully instantiate parameter values for execution as test cases (Camp et al., 30 Jul 2025). NATM itself specifies what kinds of assessment activities are needed, but does not prescribe concrete methods, tools, or data structures, which is precisely the implementation gap that subsequent work addresses (Camp et al., 30 Jul 2025).

3. Operationalizing the ADS NATM: scenario databases, risk, and safety cases

The operationalization described for ADS centers on a scenario-based Safety Assessment Framework (SAF) and a federated scenario-database approach (Camp et al., 30 Jul 2025). Inputs include the ODD description, ADS system requirements, test objectives, and available scenario databases. A federation layer sits between the assessor and distributed databases; the user issues a query based on ODD and test objectives, the layer translates it into database-specific queries, aggregates the returned scenarios, adds metadata including exposure and quality metrics, and returns a unified scenario set (Camp et al., 30 Jul 2025).

The framework then proceeds through test-scenario generation, metrics and test-objective definition, test-method allocation across simulation/track/road, execution, and finally test analysis, coverage analysis, system analysis, and assessment results feeding the safety case and deployment decisions (Camp et al., 30 Jul 2025). The paper distinguishes knowledge-based scenario databases from data-driven databases derived from accident data or everyday driving data, and cites SAFETYPOOL, STREETWISE, ADSCENE, and SCENARIO.CENTER as examples (Camp et al., 30 Jul 2025). Because individual databases are incomplete, localized, and heterogeneous, the federated approach is positioned as necessary for coverage and representativeness.

The Horizon Europe projects SUNRISE and SYNERGIES provide the main technical backbone. SUNRISE developed a federation layer with a common interface for heterogeneous scenario databases, while SYNERGIES extends this toward an EU-wide scenario dataspace emphasizing coverage, representativeness metrics, and traceability of scenario selection (Camp et al., 30 Jul 2025). HEADSTART is aligned with the three testing pillars, V4SAFETY provides guidance on validation of data sources, models, and simulation tools, and CERTAIN is intended to operationalize the ISMR pillar through detection of unexpected ADS responses and new scenarios in operation (Camp et al., 30 Jul 2025).

Risk is proposed as a central quantity because it is fair, explainable, and understandable for non-experts (Camp et al., 30 Jul 2025). The paper explicitly invokes Positive Risk Balance (PRB) and GAMAB (“Globalement Au Moins Aussi Bon”) as comparative safety concepts (Camp et al., 30 Jul 2025). Its formulation is given in prose: “By integrating the product of crash probability, the consequences of the crash (e.g., as an injury severity) and the exposure of the scenario over all scenarios within the ODD, the safety risk induced by the ADS can be estimated” (Camp et al., 30 Jul 2025). That estimate, together with coverage analysis, supports the safety case submitted to type-approval authorities.

The same source also identifies the main unresolved issues: harmonized scenario formats and interfaces, quantitative coverage and representativeness metrics, translation of soft requirements such as “competent driver” and “unreasonable risk” into quantitative criteria, safety case structure including Acceptable Means of Compliance (AMC), validation of tools and models, and full operationalization of ISMR (Camp et al., 30 Jul 2025). A recurrent misconception is that NATM already defines the detailed implementation. It does not; it defines what must be assessed, while the practical “how” remains the subject of framework, tooling, and standardization work (Camp et al., 30 Jul 2025).

4. Educational and skills-assessment NATMs

In educational assessment, several distinct methods are explicitly framed as new assessment architectures rather than merely new item types. One influential example is the combination of the Immediate Feedback Assessment Technique (IF-AT) with integrated testlets. IF-AT is an answer-until-correct multiple-choice response system using scratch-off answer sheets that provide immediate item-level feedback and support partial credit based on attempt number (Slepkov, 2013). Integrated testlets are purposefully interdependent and sequenced item sets in which earlier answers and feedback are meant to support later steps, thereby making multiple-choice examinations operate as a hybrid between standard multiple-choice and constructed response (Slepkov, 2013). The study reports excellent discrimination, with a mean polychotomous item-total correlation of $\overline{r}'_{i-t} = 0.41 \pm 0.13$ across 45 items, and a final-examination reliability of $\alpha = 0.82$ for $n=25$ items; criterion-related validity against independent constructed-response quizzes is reported as $r = 0.67$ (Slepkov, 2013). The paper explicitly presents this pattern as a robust candidate NATM for domains requiring structured multi-step reasoning.

A second educational framework is the revised AI Assessment Scale (AIAS), which is not a test in itself but a five-level framework governing how generative AI may be used in an assessment task (Perkins et al., 2024). Its levels are “No AI,” “AI Planning,” “AI Collaboration,” “Full AI,” and “AI Exploration” (Perkins et al., 2024). The framework is grounded in social constructivist principles and assessment validity, and it deliberately shifts the assessment problem from policing AI use toward designing valid tasks whose AI permissions are aligned with learning outcomes (Perkins et al., 2024). Level 1 now explicitly requires a controlled environment rather than a mere prohibition statement, and the earlier traffic-light representation has been replaced by a neutral, circular representation emphasizing that no level is inherently better than the others (Perkins et al., 2024). This is a NATM in the sense of redesigning assessment conditions, task structure, and communication protocols rather than inventing a new psychometric score.

A third strand is the IRT-based language-assessment framework in intelligent tutoring systems. Here the innovation is twofold: explicit tests are made adaptive using a 3PL IRT model, and ability is also estimated implicitly from ordinary practice exercises without a separate test session (Hou et al., 2024). The adaptive tests are reported as 4–5 times shorter than the original exhaustive tests while maintaining comparable accuracy, and exercise-based implicit assessment reaches correlations up to $\rho = 0.75$ with teacher-assigned CEFR levels under the strongest data condition reported (Hou et al., 2024). This is a NATM because it relocates proficiency estimation from isolated test events into the full tutoring workflow.

RUM extends this trajectory into software-testing education by combining rules and LLMs in a single comprehensive assessment pipeline for test case documents, test scripts, screenshots, and test reports (Wang et al., 18 Aug 2025). Its rubric has 6 dimensions and 17 indicators, with rules handling objective checks and LLMs performing subjective analysis (Wang et al., 18 Aug 2025). On the 2024 National College Student Contest on Software Testing, RUM achieved a total-score Pearson correlation of 0.992 and QWK of 0.889 against human assessment, improved assessment efficiency by 80.77%, and reduced costs by 97.38% relative to manual grading (Wang et al., 18 Aug 2025). A common theme across these educational NATMs is that validity is strengthened by making process evidence, interaction structure, or AI use itself part of the assessed construct.

5. Clinical, psychometric, software-engineering, and human-factors NATMs

Outside education, NATM design often appears as a reconfiguration of the test environment itself. In ophthalmic imaging, “NATM” denotes Noise-Aware Template Matching within an automated pipeline for stabilizing smartphone-based retinal videos so that spontaneous venous pulsations can be assessed more reliably (Sheng et al., 2023). The method combines ODR Spatio-Temporal Localization with noise-aware template matching, removes ODR-invisible and huge-jitter frames, suppresses specular artifacts, and produces stabilized ODR-centered clips (Sheng et al., 2023). Objective evaluation uses variance of optical flow, and subjective evaluation with 25 individuals from four clinics shows that the majority preferred the proposed stabilized videos for SVP observation (Sheng et al., 2023). Here NATM is a new standardized pre-test condition rather than a new clinical endpoint.

In classical psychometrics, a different NATM is the single-administration CTT reliability method based on fast splitting of a test into near-parallel halves by item difficulty (Chakrabartty et al., 2015). The goal is to estimate reliability exactly as Classical Test Theory defines it, namely as the ratio of true-score variance to observed-score variance, using a new fast dichotomisation method whose complexity is $\mathcal{O}((n/2)^2)$ in the number of items (Chakrabartty et al., 2015). The same work extends the method to interval estimates of true scores and weighted batteries of tests (Chakrabartty et al., 2015). The methodological novelty lies not in changing the test content but in changing how uncertainty is inferred from a single administration.

Software-engineering NATMs are especially explicit about assessment of assessment. In SPL engineering, mutation testing is lifted to the level of feature models, mapping models, delta models, and UML state machines in order to assess the error-detection capability of SPL test suites (Lackner et al., 2015). The mutation score is the classical $MS = \frac{M_k}{M_t - M_e}$, but the mutants are SPL-specific, such as Delete Mapping (DMP), Delete Mapped Element (DME), Insert Mapped Element (IME), Swap Feature (SWP), Change Feature Value (CFV), and several state-machine operators (Lackner et al., 2015). The reported results show that superfluous-behavior faults are badly detected—for example, DMP and DME are killed 0% of the time in the reported case studies—demonstrating that conventional coverage criteria do not guarantee SPL fault detection (Lackner et al., 2015).

A related organizational NATM is the self-assessment instrument for test automation maturity. It is a survey-based instrument with 15 knowledge areas and a revised total of 80 items, intended as a component of the TESTOMAT project’s Test Automation Improvement Model (Wang et al., 2019). Its content validity was evaluated through expert review using the Content Validity Index, yielding a revised scale-level value of $\text{S-CVI/Ave} = .91$ (Wang et al., 2019). The method does not define discrete maturity levels yet; instead it provides a structured self-assessment across strategy, resources, organization, knowledge transfer, tools, environments, requirements, design, execution, verdicts, process, SUT, measurements, and quality attributes (Wang et al., 2019).

Another assessment redesign appears in diagnostic test generation from learner histories. The method formulates question assembly as a combinatorial search over a learner–question probability matrix estimated by a knowledge-tracing model, and optimizes for low RMSE against whole-pool performance and high standard deviation across learners (Kim et al., 2021). A genetic algorithm outperforms greedy and random baselines across one private and four public datasets (Kim et al., 2021). This is a NATM because the test itself becomes a data-driven optimization product rather than a manually assembled questionnaire.

In motion-sickness research, the NATM is a compact-track procedure that recreates on-road longitudinal and lateral accelerations within a 70 m by 175 m test area using model predictive control (Harmankaya et al., 2024). In a within-subject experiment with 47 participants, the average maximum MISC was 2.69 on-road and 2.29 on the test track, with no significant effect of condition on the average motion-sickness trajectory; there was also an overall correspondence of individual sickness levels between conditions (Harmankaya et al., 2024). The assessment innovation here is the replacement of uncontrolled road exposure by a controlled, repeatable, dynamically equivalent test-track protocol.

6. Common design principles, recurrent tensions, and open questions

Across these otherwise unrelated literatures, several structural regularities recur. First, NATM usually emerges when legacy assessment is judged too narrow, too manual, or too poorly aligned with the construct of interest. In ADS safety, prescribed physical tests are insufficient for virtually infinite scenario spaces (Camp et al., 30 Jul 2025). In educational testing, ordinary multiple-choice items, detector-based AI policies, or purely manual grading fail to capture multi-step reasoning, AI-mediated performance, or scalable subjective evaluation (Slepkov, 2013, Perkins et al., 2024, Wang et al., 18 Aug 2025). In clinical and human-factors settings, raw observational conditions are too noisy or too irreproducible to support reliable assessment (Sheng et al., 2023, Harmankaya et al., 2024).

Second, NATMs typically hybridize evidence sources rather than replacing one source with another. ADS NATM combines simulation, track, road, audit, and ISMR (Camp et al., 30 Jul 2025). RUM combines rules and LLMs (Wang et al., 18 Aug 2025). IF-AT combines objective multiple-choice scoring with feedback-mediated partial credit (Slepkov, 2013). The ITS framework combines adaptive tests with implicit assessment during practice (Hou et al., 2024). This suggests that NATM is less about automation alone than about evidence orchestration.

Third, validity and representativeness are persistent fault lines. ADS NATM still requires harmonized scenario formats, coverage metrics, and quantitative acceptance criteria for soft requirements (Camp et al., 30 Jul 2025). AIAS explicitly prioritizes validity over cheating control and warns that the framework is a starting point rather than a prescriptive tool (Perkins et al., 2024). Self-assessment of test automation maturity is threatened by response bias and satisficing (Wang et al., 2019). LLM-based scoring in RUM exhibits some run-to-run variability, although the reported ranges remain limited relative to the total score scale (Wang et al., 18 Aug 2025). A common misconception is that a more technologically sophisticated NATM eliminates the need for validation; the cited works consistently say the opposite.

Finally, NATM often redefines not only scoring but also the admissible test environment. That is explicit in ADS scenario operationalization, retinal-video stabilization, and compact-track motion-sickness recreation (Camp et al., 30 Jul 2025, Sheng et al., 2023, Harmankaya et al., 2024). A plausible implication is that many “new assessment methods” are best understood as controlled evidence-production systems: they standardize inputs, intermediate representations, and traceability so that evaluation becomes arguable at scale.

In that sense, NATM is both a technical and epistemic concept. Whether in type approval, classroom assessment, clinical imaging, tutoring systems, or software-testing education, it names an attempt to convert messy real-world performance into structured evidence without collapsing the construct into a single fixed test.