Model-Based Relational Testing (MRT)

• Model-Based Relational Testing (MRT) is a testing methodology that assesses software correctness by comparing multiple executions, models, or implementations using relational oracles.
• It leverages various model types—formal, inferred, or requirements-based—to define and check relationships such as metamorphic, value, status equivalences, and leakage contracts across testing scenarios.
• Applications of MRT range from deep-learning API validation to CPU security testing, with reported results including increased API coverage, 100% branch conformance, and the detection of novel vulnerabilities.

Model-Based Relational Testing (MRT) is a testing methodology in which correctness is assessed through relations over multiple executions, multiple models, or multiple implementations, rather than through a single input–output oracle. Across the literature, MRT appears in several concrete forms: metamorphic relations for model transformations, inferred value and status equivalence between deep-learning APIs, TLA\textsuperscript{+}-driven conformance testing, and contract-based noninterference checks for CPU leakage. In each case, a model—formal, inferred, metamodel-based, or requirements-based—defines which behaviors should coincide, refine each other, or remain observationally indistinguishable (Troya et al., 2018, Deng et al., 2022, Davis et al., 2020, Oleksenko et al., 8 Jul 2025).

1. Relational semantics

The central departure from conventional testing is that MRT does not rely on a predicate over a single execution such as $\text{Oracle}(x,f(x))$. Instead, it uses a relational oracle over several executions. A convenient general schema, recurring across metamorphic-testing work, is

$$\forall x.\; C(x) \Rightarrow R\bigl(x,T(x),f(x),f(T(x))\bigr),$$

where $T$ constructs a follow-up input, $R$ constrains the corresponding outputs, and $C$ expresses applicability conditions. In model transformations this becomes a relation such as $R(M,M',T(M),T(M'))$; in API testing it becomes value or status equivalence between two operations on the same input domain; in security testing it becomes a noninterference-style condition over executions with different secrets but the same public input (Troya et al., 2018, Deng et al., 2022, Oleksenko et al., 8 Jul 2025).

This family resemblance suggests a broad conception of MRT. Metamorphic testing supplies the canonical relational form: “relationships between input-output pairs in consecutive executions of the System Under Test,” captured by metamorphic relations (MRs) (Duque-Torres et al., 2023). DeepREL makes the same idea explicit at the API level by defining value-equivalent APIs and status-equivalent APIs, while Revizor and its multi-domain extension replace output equality by equality of hardware observations conditioned on equal contract traces (Deng et al., 2022, Oleksenko et al., 2021). MongoDB’s model-based trace-checking and model-based test-case generation instantiate the same pattern with formal conformance: a recorded trace must be a permitted behavior of a TLA\textsuperscript{+} specification, or a model-generated scenario must produce the outputs predicted by that specification (Davis et al., 2020).

The strength of the relation varies by domain. Some MRT oracles require observational equality, some require only equality of statuses, some require trace inclusion, and some require isolation under a leakage contract. This variation is structural rather than incidental: MRT is defined less by one fixed relation than by the use of an explicit model to formulate and check a relation among executions.

2. What counts as the model

MRT does not require a single model formalism. In model-driven engineering, the model is a metamodel-typed transformation setting: source models, target models, and transformation rules provide the basis for generic metamorphic relations that can be instantiated for a specific transformation (Troya et al., 2018). In requirements-driven metamorphic testing, the model is textual: requirements specifications and API descriptions are treated as behavioral sources from which MRs and executable metamorphic relations (EMRs) can be derived (Shin et al., 2024).

In formal conformance testing, the model is an executable specification. MongoDB’s work uses TLA\textsuperscript{+} specifications of a replication protocol and an operational-transformation algorithm. In the first case, implementation traces are related to the set of traces admitted by a specification; in the second, the state graph of the model is projected into concrete tests for two implementations (Davis et al., 2020). In microarchitectural testing, the model is neither a full microarchitectural simulator nor a functional oracle, but a leakage contract. Revizor uses speculation contracts to specify the permitted side effects of program execution on the CPU’s microarchitectural state, and the later multi-domain work extends this to actors representing security domains such as virtual machines, kernel, and processes (Oleksenko et al., 2021, Oleksenko et al., 8 Jul 2025).

Component-based MBT adds another variant: the model is a family of labelled transition systems, and the key relation may hold between component models themselves. “Mutual acceptance” is introduced as a relation over component models guaranteeing that they communicate correctly; this is what restores compositionality to MBT under $\mathbf{uioco}$ (Cuyck et al., 2023). Taken together, these works show that “model-based” in MRT can mean metamodels, specifications, inferred API graphs, leakage contracts, or component transition systems.

3. Relational oracles, inference, and constrainting

A large part of MRT research concerns the construction of usable relational oracles. In model transformations, the basic idea is to define generic MRs as templates,

$\forall M,M'.\; P(M,M') \Rightarrow Q(T(M),T(M')),$

and then automatically instantiate them for a specific transformation. The contribution is not a hand-written oracle for one transformation, but a reusable family of relational properties derived from source and target metamodel elements and their mappings (Troya et al., 2018).

DeepREL pushes this toward automated inference. It hypothesizes that DL libraries contain “relational APIs” sharing similar inputs and outputs, infers candidate relations from signatures and documentation, synthesizes paired invocations, validates them dynamically, and then uses the validated relations as fuzzing oracles. Two formally distinguished relations are central: value equivalence,

$$S \equiv T \pmod D \iff \forall x \in D.\; S(x)=T(x),$$

and status equivalence,

$$S \sim T \pmod D \iff \forall x \in D.\; \{S(x)\}=\{T(x)\}.$$

The resulting relational graph functions as a partial specification of the library (Deng et al., 2022).

Later metamorphic-testing work emphasizes that MR generation alone is insufficient. “Towards a Complete Metamorphic Testing Pipeline” separates MR generation, MR constraint definition, and explainability of MR outcomes, and frames applicability constraints as first-class objects rather than afterthoughts (Duque-Torres et al., 2023). MetaTrimmer makes the same point operationally: it generates random test data, performs test-data transformations and logs MR violations, then uses manual inspections to derive constraints. The novelty claimed there is the avoidance of complex prediction models requiring labeled datasets about MR applicability, while enabling integration with advanced fuzzing (Duque-Torres et al., 2023).

LLM-assisted work extends oracle construction one step further by deriving EMRs from requirements and API specifications. Rather than merely asking the model to produce EMRs, the approach uses few-shot prompting to teach the MT process and the SMRL DSL, so that natural-language requirements become executable relational oracles (Shin et al., 2024). This suggests an emerging MRT pipeline in which models generate relations, relations become executable tests, and violations feed back into constraint refinement.

4. Execution workflows and conformance mechanisms

Across the cited work, MRT workflows repeatedly follow the same high-level structure: construct related executions from a model, run the SUT or implementations, and check a relation that spans those executions. For metamorphic testing, this means generating an initial input, deriving a follow-up input satisfying an MR precondition, executing both, and checking the instantiated MR. For inferred API relations, it means pairing source and target API calls on borrowed inputs and comparing outputs or statuses. For formal conformance, it means either checking implementation traces against a specification or extracting model behaviors and replaying them as tests (Troya et al., 2018, Deng et al., 2022, Davis et al., 2020).

MongoDB’s two case studies make the distinction particularly clear. Model-Based Trace-Checking (MBTC) observes implementation traces and verifies that the trace is a permitted behavior of the specification. Model-Based Test-Case Generation (MBTCG) uses the specification to generate test inputs and expected outputs, then checks whether the implementation responds as specified (Davis et al., 2020). The first is a trace–spec relation; the second is a model-driven stimulus–response relation.

In CPU security testing, the workflow is contract-centered rather than output-centered. Revizor generates random programs and inputs, computes contract traces using an executable model, collects hardware traces on the real CPU, partitions inputs into classes with equal contract traces, and then checks whether the corresponding hardware traces remain equal within each class (Oleksenko et al., 2021). The multi-domain extension adds actor-based templates, a kernel-module sandbox, multi-domain transitions, and statistical comparison of trace distributions using Pearson’s $$\forall x.\; C(x) \Rightarrow R\bigl(x,T(x),f(x),f(T(x))\bigr),$$0 test and adaptive sample sizes, because cross-domain execution introduces substantial nondeterminism (Oleksenko et al., 8 Jul 2025).

These workflows also show why MRT is not reducible to ordinary fuzzing. The distinctive step is not random generation by itself, but the use of a model to define which pairs or sets of executions are comparable and which divergences count as faults.

5. Representative applications and reported results

Empirical evidence for MRT is heterogeneous. Some papers are conceptual and high-level, while others report large-scale industrial or systems evaluations. The following examples illustrate the range of reported outcomes.

Setting	Relational model	Reported result
DL library fuzzing	Value/status-equivalent relational APIs	Covered 157% more APIs than FreeFuzz; detected 162 bugs in total, with 106 confirmed and 14 documentation bugs (Deng et al., 2022)
MongoDB Realm Sync OT	TLA\textsuperscript{+} model driving two implementations	Generated 4,913 C++ test cases and achieved 86/86 branches (100%) in the array merge rules (Davis et al., 2020)
Multi-domain CPU isolation	Leakage contracts over victim/attacker actors	Exposed four new leaks, with only two false positives across 304 campaigns and more than 20 billion measurements (Oleksenko et al., 8 Jul 2025)
Requirements-to-EMR generation	Requirements and API specifications translated into SMRL	107/138 generated EMR statements were correct; 77% of 64 MRs were rated positively for understandability (Shin et al., 2024)

Other results sharpen the picture. Revizor automatically surfaced Spectre, MDS, and LVI, as well as several previously unknown variants, showing that contract-based MRT can recover both known and novel speculative vulnerabilities on black-box CPUs (Oleksenko et al., 2021). By contrast, the MongoDB replication case study is explicitly negative: MBTC was found impractical for testing conformance to a highly abstract specification, while MBTCG was highly successful for the more self-contained OT algorithm (Davis et al., 2020). In model transformations, the 2018 paper is much shorter and more conceptual, but its stated contribution is already recognizably MRT: generic metamorphic relations for model transformations that can be automatically instantiated for a specific transformation (Troya et al., 2018).

6. Limits, misconceptions, and current directions

A recurring misconception is that local correctness automatically composes. The compositionality paper shows that $$\forall x.\; C(x) \Rightarrow R\bigl(x,T(x),f(x),f(T(x))\bigr),$$1 is not compositional: correctly tested components, when composed into a system, can still lead to a faulty system. The introduced remedy is “mutual acceptance,” a relation over component models under which mutually accepting components are guaranteed to communicate correctly (Cuyck et al., 2023). In MRT terms, this is a shift from relating implementations to specifications only, toward also relating specifications to each other.

A second misconception is that inferred or generic relations are automatically sound. DeepREL explicitly validates inferred relations on sampled inputs and still reports false positives: the overall false-positive rate is 30.72%, with 15.13% for value relations and 37.25% for status relations (Deng et al., 2022). This is not a peripheral issue. The method distinguishes intended equivalences from merely similar behavior only empirically, and later fuzzing may reveal that a “verified” relation was accidental rather than semantic.

A third difficulty is the abstraction gap between model and implementation. MongoDB’s replication study reports approximately 10 weeks of work by two engineers, including 570 lines of C++ tracing code, 252 lines added or modified in the TLA specification, and 484 lines of Python post-processing, yet only one handwritten test fully passed MBTC. The explicit conclusion is that MBTC is only practical if the specification is written with MBTC in mind (Davis et al., 2020). This is a fundamental MRT design constraint: abstraction helps reasoning, but mappability governs testability.

Metamorphic-testing work adds another correction: MRs are rarely global laws over the entire valid input space. Both the “complete pipeline” agenda and MetaTrimmer emphasize constraint definition, applicability, and explainability. The mature form of an MR is therefore not simply $$\forall x.\; C(x) \Rightarrow R\bigl(x,T(x),f(x),f(T(x))\bigr),$$2, but $$\forall x.\; C(x) \Rightarrow R\bigl(x,T(x),f(x),f(T(x))\bigr),$$3, where $$\forall x.\; C(x) \Rightarrow R\bigl(x,T(x),f(x),f(T(x))\bigr),$$4 captures where the relation is supposed to hold (Duque-Torres et al., 2023, Duque-Torres et al., 2023). Requirements-to-EMR generation reinforces the same point from a different angle: LLM-generated MRs and EMRs can be comprehensible and pertinent, yet some are too generic to guide expected outputs precisely, and some generated functions are infeasible in the real testing environment (Shin et al., 2024).

Current directions therefore converge on a more elaborate MRT stack: automated relational specification inference, explicit applicability constraints, explainable counterexample analysis, compositional relations between models, and statistical treatment of nondeterministic environments. The recent microarchitectural work states this succinctly in its own setting: MRT is not just random fuzzing plus a model, but a methodology in which generation, execution, and analysis are all shaped by the relational contract being tested (Oleksenko et al., 8 Jul 2025).