---
title: New Assessment/Test Method (NATM)
url: https://www.emergentmind.com/topics/new-assessment-test-method-natm
type: topic
---

# New Assessment/Test Method (NATM)

Searching arXiv for the cited NATM-related papers to ground the article.
“New Assessment/Test Method” (NATM) is not a single universally standardized technique across the research literature. In the sources considered here, the term denotes either a specific regulatory framework—most prominently the UNECE WP.29/GRVA framework for safety assessment of Automated Driving Systems (ADS)—or a domain-specific designation for newly structured assessment procedures in education, clinical imaging, intelligent tutoring systems, psychometrics, software engineering, and human-factors experimentation [2507.22433][1308.4365][2412.09029][2409.16133][2305.06043][2412.14982][2412.14982]. Across these usages, NATM consistently refers to an assessment architecture that replaces narrow, one-shot, or purely manual evaluation with multi-component evidence, explicit criteria, and stronger operational traceability.

## 1. Scope and principal usages

In current usage, NATM is best understood as a family resemblance term. In one strand, it is the proper name of a regulatory framework for ADS safety assurance. In other strands, it denotes a newly proposed assessment or testing protocol tailored to a specific problem domain.

| Domain | NATM form | Citation |
|---|---|---|
| Automated driving | UNECE WP.29/GRVA multi-pillar, scenario-based ADS safety assessment | [2507.22433] |
| Educational assessment | IF-AT with integrated testlets; AI Assessment Scale; rule+LLM grading | [1308.4365], [2412.09029], [2508.12922] |
| Intelligent tutoring systems | Adaptive and implicit IRT-based language proficiency assessment | [2409.16133] |
| Clinical imaging | “Noise-Aware Template Matching” for retinal video stabilization in SVP assessment | [2305.06043] |
| Psychometrics | Single-administration CTT reliability estimation via fast dichotomisation | [1503.03297] |
| Software testing and QA | Diagnostic test generation, SPL mutation-based test assessment, test automation maturity self-assessment | [2112.11188], [1504.02443], [1904.05816] |
| Human-factors experimentation | Compact-track recreation of on-road motion-sickness exposure | [2412.14982] |

This diversity matters because the acronym does not by itself identify a unique method. A plausible implication is that “NATM” functions less as a canonical technical standard than as a marker for assessment redesign: a method becomes “new” when it changes what is measured, how evidence is collected, and how validity is argued.

## 2. NATM as the UNECE scenario-based framework for automated driving

The most formalized use of NATM in the present sources is the ADS safety framework under development within the UNECE WP.29 Working Party on Automated/Autonomous and Connected Vehicles (GRVA). It is already recognized in major markets including Japan, South Korea, the EU, and the USA [2507.22433]. The framework arises from the inadequacy of traditional vehicle type-approval, which has relied on a relatively small number of prescribed physical tests, deterministic test conditions, and human drivers as the default reference. For ADS, that model is insufficient because the relevant traffic-situation space is virtually infinite, software-intensive AI-based systems require evidence over distributions of scenarios, safety must be assessed across the whole lifecycle, and many rare but critical situations are unsafe or impractical to test exclusively on physical tracks [2507.22433].

NATM therefore adopts a multi-pillar structure. The pillars are virtual testing, track testing, real-world testing before deployment, an audit pillar, and in-service monitoring and reporting (ISMR) [2507.22433]. Virtual testing covers large numbers of scenarios quickly and safely; track testing validates and complements simulation; real-world testing covers controlled public-road trials; the audit pillar addresses the manufacturer’s Safety Management System, internal procedures, and the tools and methods used for testing and validation; ISMR provides the post-deployment feedback loop [2507.22433].

Its conceptual core is scenario-based assessment. Scenarios serve two distinct functions: they describe the ADS Operational Design Domain (ODD), and they drive testing through concrete test scenarios instantiated from more generic descriptions [2507.22433]. The paper adopts the functional–logical–concrete classification aligned with PEGASUS and many EU projects: functional scenarios are high-level descriptions, logical scenarios add parameter ranges, and concrete scenarios fully instantiate parameter values for execution as test cases [2507.22433]. NATM itself specifies what kinds of assessment activities are needed, but does not prescribe concrete methods, tools, or data structures, which is precisely the implementation gap that subsequent work addresses [2507.22433].

## 3. Operationalizing the ADS NATM: scenario databases, risk, and safety cases

The operationalization described for ADS centers on a scenario-based Safety Assessment Framework (SAF) and a federated scenario-database approach [2507.22433]. Inputs include the ODD description, ADS system requirements, test objectives, and available scenario databases. A federation layer sits between the assessor and distributed databases; the user issues a query based on ODD and test objectives, the layer translates it into database-specific queries, aggregates the returned scenarios, adds metadata including exposure and quality metrics, and returns a unified scenario set [2507.22433].

The framework then proceeds through test-scenario generation, metrics and test-objective definition, test-method allocation across simulation/track/road, execution, and finally test analysis, coverage analysis, system analysis, and assessment results feeding the safety case and deployment decisions [2507.22433]. The paper distinguishes knowledge-based scenario databases from data-driven databases derived from accident data or everyday driving data, and cites SAFETYPOOL, STREETWISE, ADSCENE, and SCENARIO.CENTER as examples [2507.22433]. Because individual databases are incomplete, localized, and heterogeneous, the federated approach is positioned as necessary for coverage and representativeness.

The Horizon Europe projects SUNRISE and SYNERGIES provide the main technical backbone. SUNRISE developed a federation layer with a common interface for heterogeneous scenario databases, while SYNERGIES extends this toward an EU-wide scenario dataspace emphasizing coverage, representativeness metrics, and traceability of scenario selection [2507.22433]. HEADSTART is aligned with the three testing pillars, V4SAFETY provides guidance on validation of data sources, models, and simulation tools, and CERTAIN is intended to operationalize the ISMR pillar through detection of unexpected ADS responses and new scenarios in operation [2507.22433].

Risk is proposed as a central quantity because it is fair, explainable, and understandable for non-experts [2507.22433]. The paper explicitly invokes Positive Risk Balance (PRB) and GAMAB (“Globalement Au Moins Aussi Bon”) as comparative safety concepts [2507.22433]. Its formulation is given in prose: “By integrating the product of crash probability, the consequences of the crash (e.g., as an injury severity) and the exposure of the scenario over all scenarios within the ODD, the safety risk induced by the ADS can be estimated” [2507.22433]. That estimate, together with coverage analysis, supports the safety case submitted to type-approval authorities.

The same source also identifies the main unresolved issues: harmonized scenario formats and interfaces, quantitative coverage and representativeness metrics, translation of soft requirements such as “competent driver” and “unreasonable risk” into quantitative criteria, safety case structure including Acceptable Means of Compliance (AMC), validation of tools and models, and full operationalization of ISMR [2507.22433]. A recurrent misconception is that NATM already defines the detailed implementation. It does not; it defines what must be assessed, while the practical “how” remains the subject of framework, tooling, and standardization work [2507.22433].

## 4. Educational and skills-assessment NATMs

In educational assessment, several distinct methods are explicitly framed as new assessment architectures rather than merely new item types. One influential example is the combination of the Immediate Feedback Assessment Technique (IF-AT) with integrated testlets. IF-AT is an answer-until-correct multiple-choice response system using scratch-off answer sheets that provide immediate item-level feedback and support partial credit based on attempt number [1308.4365]. Integrated testlets are purposefully interdependent and sequenced item sets in which earlier answers and feedback are meant to support later steps, thereby making multiple-choice examinations operate as a hybrid between standard multiple-choice and constructed response [1308.4365]. The study reports excellent discrimination, with a mean polychotomous item-total correlation of $\overline{r}'_{i-t} = 0.41 \pm 0.13$ across 45 items, and a final-examination reliability of $\alpha = 0.82$ for $n=25$ items; criterion-related validity against independent constructed-response quizzes is reported as $r = 0.67$ [1308.4365]. The paper explicitly presents this pattern as a robust candidate NATM for domains requiring structured multi-step reasoning.

A second educational framework is the revised AI Assessment Scale (AIAS), which is not a test in itself but a five-level framework governing how generative AI may be used in an assessment task [2412.09029]. Its levels are “No AI,” “AI Planning,” “AI Collaboration,” “Full AI,” and “AI Exploration” [2412.09029]. The framework is grounded in social constructivist principles and assessment validity, and it deliberately shifts the assessment problem from policing AI use toward designing valid tasks whose AI permissions are aligned with learning outcomes [2412.09029]. Level 1 now explicitly requires a controlled environment rather than a mere prohibition statement, and the earlier traffic-light representation has been replaced by a neutral, circular representation emphasizing that no level is inherently better than the others [2412.09029]. This is a NATM in the sense of redesigning assessment conditions, task structure, and communication protocols rather than inventing a new psychometric score.

A third strand is the IRT-based language-assessment framework in intelligent tutoring systems. Here the innovation is twofold: explicit tests are made adaptive using a 3PL IRT model, and ability is also estimated implicitly from ordinary practice exercises without a separate test session [2409.16133]. The adaptive tests are reported as 4–5 times shorter than the original exhaustive tests while maintaining comparable accuracy, and exercise-based implicit assessment reaches correlations up to $\rho = 0.75$ with teacher-assigned CEFR levels under the strongest data condition reported [2409.16133]. This is a NATM because it relocates proficiency estimation from isolated test events into the full tutoring workflow.

RUM extends this trajectory into software-testing education by combining rules and large language models in a single comprehensive assessment pipeline for test case documents, test scripts, screenshots, and test reports [2508.12922]. Its rubric has 6 dimensions and 17 indicators, with rules handling objective checks and LLMs performing subjective analysis [2508.12922]. On the 2024 National College Student Contest on Software Testing, RUM achieved a total-score Pearson correlation of 0.992 and QWK of 0.889 against human assessment, improved assessment efficiency by 80.77%, and reduced costs by 97.38% relative to manual grading [2508.12922]. A common theme across these educational NATMs is that validity is strengthened by making process evidence, interaction structure, or AI use itself part of the assessed construct.

## 5. Clinical, psychometric, software-engineering, and human-factors NATMs

Outside education, NATM design often appears as a reconfiguration of the test environment itself. In ophthalmic imaging, “NATM” denotes Noise-Aware Template Matching within an automated pipeline for stabilizing smartphone-based retinal videos so that spontaneous venous pulsations can be assessed more reliably [2305.06043]. The method combines ODR Spatio-Temporal Localization with noise-aware template matching, removes ODR-invisible and huge-jitter frames, suppresses specular artifacts, and produces stabilized ODR-centered clips [2305.06043]. Objective evaluation uses variance of optical flow, and subjective evaluation with 25 individuals from four clinics shows that the majority preferred the proposed stabilized videos for SVP observation [2305.06043]. Here NATM is a new standardized pre-test condition rather than a new clinical endpoint.

In classical psychometrics, a different NATM is the single-administration CTT reliability method based on fast splitting of a test into near-parallel halves by item difficulty [1503.03297]. The goal is to estimate reliability exactly as Classical Test Theory defines it, namely as the ratio of true-score variance to observed-score variance, using a new fast dichotomisation method whose complexity is $\mathcal{O}((n/2)^2)$ in the number of items [1503.03297]. The same work extends the method to interval estimates of true scores and weighted batteries of tests [1503.03297]. The methodological novelty lies not in changing the test content but in changing how uncertainty is inferred from a single administration.

Software-engineering NATMs are especially explicit about assessment of assessment. In SPL engineering, mutation testing is lifted to the level of feature models, mapping models, delta models, and UML state machines in order to assess the error-detection capability of SPL test suites [1504.02443]. The mutation score is the classical $MS = \frac{M_k}{M_t - M_e}$, but the mutants are SPL-specific, such as Delete Mapping (DMP), Delete Mapped Element (DME), Insert Mapped Element (IME), Swap Feature (SWP), Change Feature Value (CFV), and several state-machine operators [1504.02443]. The reported results show that superfluous-behavior faults are badly detected—for example, DMP and DME are killed 0% of the time in the reported case studies—demonstrating that conventional coverage criteria do not guarantee SPL fault detection [1504.02443].

A related organizational NATM is the self-assessment instrument for test automation maturity. It is a survey-based instrument with 15 knowledge areas and a revised total of 80 items, intended as a component of the TESTOMAT project’s Test Automation Improvement Model [1904.05816]. Its content validity was evaluated through expert review using the Content Validity Index, yielding a revised scale-level value of $\text{S-CVI/Ave} = .91$ [1904.05816]. The method does not define discrete maturity levels yet; instead it provides a structured self-assessment across strategy, resources, organization, knowledge transfer, tools, environments, requirements, design, execution, verdicts, process, SUT, measurements, and quality attributes [1904.05816].

Another assessment redesign appears in diagnostic test generation from learner histories. The method formulates question assembly as a combinatorial search over a learner–question probability matrix estimated by a knowledge-tracing model, and optimizes for low RMSE against whole-pool performance and high standard deviation across learners [2112.11188]. A genetic algorithm outperforms greedy and random baselines across one private and four public datasets [2112.11188]. This is a NATM because the test itself becomes a data-driven optimization product rather than a manually assembled questionnaire.

In motion-sickness research, the NATM is a compact-track procedure that recreates on-road longitudinal and lateral accelerations within a 70 m by 175 m test area using model predictive control [2412.14982]. In a within-subject experiment with 47 participants, the average maximum MISC was 2.69 on-road and 2.29 on the test track, with no significant effect of condition on the average motion-sickness trajectory; there was also an overall correspondence of individual sickness levels between conditions [2412.14982]. The assessment innovation here is the replacement of uncontrolled road exposure by a controlled, repeatable, dynamically equivalent test-track protocol.

## 6. Common design principles, recurrent tensions, and open questions

Across these otherwise unrelated literatures, several structural regularities recur. First, NATM usually emerges when legacy assessment is judged too narrow, too manual, or too poorly aligned with the construct of interest. In ADS safety, prescribed physical tests are insufficient for virtually infinite scenario spaces [2507.22433]. In educational testing, ordinary multiple-choice items, detector-based AI policies, or purely manual grading fail to capture multi-step reasoning, AI-mediated performance, or scalable subjective evaluation [1308.4365][2412.09029][2508.12922]. In clinical and human-factors settings, raw observational conditions are too noisy or too irreproducible to support reliable assessment [2305.06043][2412.14982].

Second, NATMs typically hybridize evidence sources rather than replacing one source with another. ADS NATM combines simulation, track, road, audit, and ISMR [2507.22433]. RUM combines rules and LLMs [2508.12922]. IF-AT combines objective multiple-choice scoring with feedback-mediated partial credit [1308.4365]. The ITS framework combines adaptive tests with implicit assessment during practice [2409.16133]. This suggests that NATM is less about automation alone than about evidence orchestration.

Third, validity and representativeness are persistent fault lines. ADS NATM still requires harmonized scenario formats, coverage metrics, and quantitative acceptance criteria for soft requirements [2507.22433]. AIAS explicitly prioritizes validity over cheating control and warns that the framework is a starting point rather than a prescriptive tool [2412.09029]. Self-assessment of test automation maturity is threatened by response bias and satisficing [1904.05816]. LLM-based scoring in RUM exhibits some run-to-run variability, although the reported ranges remain limited relative to the total score scale [2508.12922]. A common misconception is that a more technologically sophisticated NATM eliminates the need for validation; the cited works consistently say the opposite.

Finally, NATM often redefines not only scoring but also the admissible test environment. That is explicit in ADS scenario operationalization, retinal-video stabilization, and compact-track motion-sickness recreation [2507.22433][2305.06043][2412.14982]. A plausible implication is that many “new assessment methods” are best understood as controlled evidence-production systems: they standardize inputs, intermediate representations, and traceability so that evaluation becomes arguable at scale.

In that sense, NATM is both a technical and epistemic concept. Whether in type approval, classroom assessment, clinical imaging, tutoring systems, or software-testing education, it names an attempt to convert messy real-world performance into structured evidence without collapsing the construct into a single fixed test.

Source: https://www.emergentmind.com/topics/new-assessment-test-method-natm