Assertion-Oracles for Cyber-Physical Systems

• Assertion-based test oracles for CPS are formal or executable artifacts that validate system behavior through runtime or post-mortem assertions.
• They integrate formal specification languages, FSM models, and ML-driven techniques to capture continuous dynamics and manage environmental uncertainties.
• Empirical studies demonstrate these oracles can significantly reduce test execution time while enhancing fault detection in domains such as aerospace and autonomous vehicles.

Assertion-based test oracles are formal or executable artifacts that specify criteria for validating the correctness of system executions by evaluating assertions—expressions over states, signals, or observable variables—at runtime or post-mortem. In cyber-physical systems (CPS), where software and physical elements interact continuously, the oracle problem is compounded by uncertainties, time- and magnitude-continuous behaviors, and the cost or impracticality of manually engineering explicit output expectations. Recent research has established that assertion-based oracles for CPS must be expressive, efficient, robust to system flakiness, and suitable for early or online evaluation.

1. Theoretical Foundations and Oracle Problem in CPS

The primary roles of assertion-based test oracles for CPS are to (i) algorithmically determine if execution outputs conform to the system’s requirements, (ii) do so continuously and efficiently—preferably online to enable early test termination—and (iii) handle the complexity introduced by non-determinism, continuous dynamics, and environmental uncertainties (Menghi et al., 2019). The inherent oracle problem—deciding for arbitrary test executions whether the system has deviated from acceptable behavior—is undecidable in general, especially when the exact ground-truth for outputs is unknown or unformulatable (Ibrahimzada et al., 2023).

Assertion-based CPS oracles address these challenges by formulating requirements in a way that naturally bridges physical behavior and discrete control, often either through logical requirement languages, formal models (finite state machines, hybrid automata), or learned representations that generalize from observed data (Menghi et al., 2019, Timo, 2022, Sadri-Moshkenani et al., 2023, Mandrioli et al., 4 Dec 2024, Jodat et al., 28 Aug 2025).

2. Formalization Frameworks and Oracle Specification Languages

Formal specification is central to assertion-based CPS oracles. Multiple frameworks have emerged:

• Restricted Signal First-Order Logic (RFOL): An expressive yet monitorable fragment of Signal First-Order Logic (SFOL), tailored for online assertion monitoring in continuous-time systems. RFOL formulas quantify over time intervals and signal variables, expressing requirements such as $$\forall t \in [0, T): \|q_\text{real}(t) - q_\text{target}(t)\| \leq \epsilon$$ (Menghi et al., 2019). The semantics are lifted from Boolean to quantitative using domain-specific diff functions.
• Finite State Machines (FSMs)/Hybrid Automata: Assertion oracles can be modeled as deterministic FSMs or hybrid automata to capture both discrete control and continuous mode invariants (Timo, 2022, Sadri-Moshkenani et al., 2023). An FSM oracle is a tuple $(S, s_0, X, Y, T)$ where $S$ is states, $X$ inputs, $Y$ outputs, and $T$ transitions. Hybrid automata generalize this with real-valued flows, guards, and invariants.
• Quantitative Satisfaction Metrics: Beyond Boolean verdicts, oracles often provide quantitative degrees of satisfaction/failure. In RFOL-based approaches, the diff function maps the predicate residual to $[-1,1]$ to denote (robust) satisfaction or violation (Menghi et al., 2019). Aggregation over time/intervals (min, max) aligns fitness values with logical structure.
• Metamorphic Relations (MRs): Assertions based on control-theoretical design assumptions, expressed as relations across tests (e.g., linearity, superposition, scaling), serve as oracles for input-output behavior when explicit output specifications are unavailable (Mandrioli et al., 4 Dec 2024).

3. Automated Oracle Generation: Algorithms and Tool Support

Research has yielded several automated workflows for generating assertion-based test oracles in CPS:

• Translation from Requirements to Simulation Monitors: The SOCRaTEs approach automatically translates RFOL requirements into Simulink blocks executable as online monitors. Time- and interval-shifting transformations ensure that all required future values are available at simulation time $t$ (e.g., $f(t+n)$ mapped to $f(t-n)$, with quantifier bounds adjusted accordingly) (Menghi et al., 2019).
• Oracle Generation via Genetic Programming and Fault-localization Metrics: Genetic Programming (GP) is applied to synthesize logical/arithmetic predicates over CPS input variables, with fitness functions guided by spectrum-based fault localization (SBFL) statistics such as Ochiai, Tarantula, and Naish. The Ochiai formula, for example,

$$\text{Ochiai}(c) = \frac{c_f(c)}{\sqrt{|\text{TS}_f| \cdot (c_p(c) + c_f(c))}}$$

where $c_f(c)$ is the count of failing tests satisfying condition $c$, is used to prioritize assertion candidates that maximize diagnostic correlation with observed failures (Jodat et al., 28 Aug 2025).

• Interpretable Machine Learning Approaches: Decision Trees (DT) and Decision Rules (DR) may also be used for oracle inference, trained on labeled test input/output data to yield rules that predict pass/fail verdicts over input features (Jodat et al., 28 Aug 2025).
• Test-driven FSM Oracle Mining: When requirements are underspecified or ambiguous, a space of candidate FSM oracles (represented as a nondeterministic FSM plus Boolean encodings) is iteratively partitioned by distinguishing tests, with expert selection of expected outputs narrowing to a precise, correct oracle (Timo, 2022).
• Hybrid Model-based Oracles: For systems specified via hybrid automata, the assertion-based oracle operationalizes model invariants and transitions as a "condition graph." At runtime, the oracle checks whether state sequences and transitions conform to the allowed hybrid model structure, rather than matching exact traces (Sadri-Moshkenani et al., 2023).

4. Scalability, Efficiency, and Online Monitoring

Online assertion-based oracles are designed for efficiency in large, high-fidelity CPS simulations where run time is a major constraint. Key mechanisms include:

• Embedded Simulink Oracles: By generating Simulink monitors that evaluate assertions during simulation, output signals continuously reflect quantitative requirement satisfaction. Monotonicity guarantees (e.g., the fitness output is non-increasing after threshold violation) support safe early-stopping: when violation is certain, the simulation can be terminated without risk of missing a restored satisfaction later (Menghi et al., 2019).
• Empirical Case Study Results: In industrial case studies, all 98 tested requirements for 11 CPS models were expressible in RFOL, with average Simulink oracle synthesis time $\sim$1.6 ms and an overhead of about 6% even for large models. Notably, online oracles reduced test execution time by 96% for the SatEx satellite control benchmark, showcasing the time savings of online assertion checking.
• Robustness to Flakiness: In domains where CPS simulators exhibit non-deterministic, "flaky" behavior, assertion-based oracles generated by the GP-Ochiai approach showed only 4% average accuracy variation across multiple flaky systems, and were more robust than DT/DR-based oracles (Jodat et al., 28 Aug 2025).

5. Expressiveness, Uncertainty Handling, and Quantitative Feedback

Expressiveness and uncertainty tolerance are critical for CPS oracles:

• Expressiveness: The restricted logic and translation approach presented in (Menghi et al., 2019) was sufficient to express all functional requirements from several industrial CPS systems, with requirements averaging 19.2 operators/quantifiers.
• Uncertainty Modeling: The presence of physical/environmental uncertainties and variable system parameters is explicitly modeled by running Simulink simulations with uncertain parameter ranges (ureal types) and input noise (specified by signal-to-noise ratio). The oracle evaluates the quantitative satisfaction measure for each output trace and ultimately reports the minimum, thereby soundly accounting for worst-case behavior (Menghi et al., 2019).
• Quantitative Feedback: Instead of binary verdicts, assertion-based oracles may compute a fitness value in $[-1,1]$ for each requirement, with aggregate calculus over logical connectives and quantifiers yielding a measure of partial or robust satisfaction/failure. This nuanced feedback informs engineers not only of the presence but the degree and severity of requirement breaches.

6. Applicability, Limitations, and Future Research Directions

Assertion-based oracles for CPS are broadly effective in practice, but several limitations and open directions remain:

• Applicability: The SOCRaTEs, GP-based, and FSM/Hybrid model oracle techniques are not tightly coupled to any specific CPS application, software platform, or simulation language. They are applicable to diverse domains, including aerospace, autonomous vehicles, robotics, and critical infrastructure.
• Limitations: Some assertion-based approaches rely on the availability of high-quality, formalized requirements or models. GP-based oracle synthesis relies on good coverage of training data (including both passing and failing cases), while FSM/mining techniques require expert intervention for final oracle selection when requirements are ambiguous.
• Integration with Other Techniques: Future work includes integrating assertion-based oracles into continuous integration pipelines, extending the expressiveness of assertion logics (for example, supporting temporal or probabilistic properties), and combining online assertion monitoring with post-mortem analysis tools.
• Early Lifecycle Use: Assertion oracles generated from formal requirements or hybrid models can be leveraged at model-in-the-loop (MiL), software-in-the-loop (SiL), and hardware-in-the-loop (HiL) testing stages, supporting early fault localization and model validation before deployment (Sadri-Moshkenani et al., 2023).
• Research Trajectory: There is ongoing development in making assertion-based oracles more readable for engineers, supporting richer language features (e.g., quantifiers, continuous properties), and incorporating adaptive or learned logic for evolving CPS requirements.

7. Empirical Validation and Industrial Impact

Assertion-based oracle frameworks have demonstrated strong empirical results:

• Case Study Summary Table

Study/Domain	Requirements Expressed	Online Overhead	Test Time Savings	Robustness (acc. var.)
SatEx (Simulink)	98/98 (RFOL)	$\sim$6%	$\sim$96% reduction	Not specified
Aerospace, Network, AV	Varies (GP/Ochiai)	N/A (Exec-free)	Simulator calls avoided	4%

All reported assertion-based oracle solutions for CPS focus on reducing manual effort, test execution cost, and increasing reliability by leveraging a mix of formal logic, interpretable ML, search, and model-based approaches (Menghi et al., 2019, Jodat et al., 28 Aug 2025, Sadri-Moshkenani et al., 2023).

---

In summary, assertion-based test oracles for CPS are characterized by their formal, automatic, online, and robust nature. By expressing requirements in monitorable logics, employing quantitative satisfaction measures, handling uncertainty, and supporting efficient execution, these oracles enable scalable and cost-effective testing of complex cyber-physical systems. Recent empirical studies validate their expressiveness, efficiency, and practical benefits in industrial contexts.