Observation Coverage Metric

• Observation coverage metric is a formal measure that quantifies the extent to which system behaviors are exercised through verified execution prefixes.
• It adapts classic test coverage by incorporating partial executions and under-approximation to assess progress in formal verification.
• Custom heuristics based on Abstract Reachability Trees optimize trace prioritization, enhancing the metric's accuracy within resource limitations.

Observation coverage metric is a formalized approach to quantify how comprehensively a process—such as formal verification or system testing—has exercised the behaviors or states of interest in a system. Evolving from classic test coverage, which measures how many code statements are executed during testing, observation coverage adapts these notions to contexts where executions can be partial, the analysis incomplete due to resource limitations, or the goal is to assess progress in systems where exhaustive testing or verification is intractable. The metric is especially central to verification techniques based on Abstract Reachability Trees (ARTs), providing an actionable basis for quantifying intermediate progress during formal analysis and enabling transfer to broader observational domains.

1. Formal Definition and Conceptual Foundation

Verification coverage is defined as the ratio of program statements for which at least one terminating execution (i.e., a feasible path from the initial to the final state satisfying the safety property) contains a prefix that has been concretely explored by the verification process. If $\mathcal{T}$ denotes the set of terminating executions and $s$ a statement, then coverage is established if $\exists t \in \mathcal{T}$ such that the safety property $\varphi(t)$ holds and there exists a prefix $\pi$ of $t$ with $s \in \pi$ and $\psi(\pi)$ (the predicate expressing that $\pi$ is within the known analyzed region of the ART):

$$\text{Coverage}(s) := \exists t \in \mathcal{T}, \; \varphi(t) \land (\exists \pi \; \text{isPrefix}(\pi, t) \land s \in \pi \land \psi(\pi))$$

Here, $\psi$ is derived from the Assumption Automaton, and only prefixes not leading to the special “FALSE” unexplored state are considered. This definition generalizes statement coverage from testing into verification, with the essential difference being the acceptance of partial analysis: statements may be considered “observed” even if the entire execution is not fully explored, provided the prefix reaches the statement under verified circumstances.

2. Methodological Framework in ART-Based Verification

ARTs are graph-based representations where nodes embody abstract states and edges encode transitions. Each node is mapped to a corresponding Control Flow Automaton (CFA) state. The Assumption Automaton tracks which transitions have been concretely analyzed. A statement is counted as covered only if it is reached via an analyzed path not touching the FALSE state.

To systematically compute coverage—even under incomplete verification—a property encoding “statement s not covered” is formulated as a safety property. Model checking then searches for counterexamples that refute this property, each corresponding to a newly analyzed terminating execution. Algorithmically, this proceeds by iteratively generating such executions and recording analyzed prefixes (using the exercisedWithinAnalysis function).

Under-Approximation Due to Resource Bounds

Because exhaustive analysis is infeasible (and most attempts are bounded by strict resource limits), the algorithm often yields an under-approximation: only those statements for which evidence exists of concrete exploration are counted, while the rest remain unassessed or labeled as “unknown.”

3. Heuristics and Performance Enhancement

A custom trace prioritization heuristic is introduced to optimize the search for analyzed executions. It assigns a tentative coverage score to each automaton state based on the reachability of CFA states composed with the Assumption Automaton. Formally, for each state $(s_a, s_{CFA})$, the “Reach” function recursively accumulates reachable CFA states unless $s_a$ is FALSE:

$$\text{Reach}((s_a, s_{CFA})) = \begin{cases} s_{CFA} \cup \bigcup_{(s_a', s_{CFA}') \in \text{succ}((s_a, s_{CFA}))} \text{Reach}((s_a', s_{CFA}')) & \text{if } s_a \neq \text{FALSE} \ \emptyset & \text{if } s_a = \text{FALSE} \end{cases}$$

The cardinality $|\text{Reach}(s_a, s_{CFA})|$ is the score guiding ART exploration. When traversing the ART, nodes with higher scores are prioritized, increasing the probability of expanding prefixes that cover more statements per exploration step.

4. Empirical Evaluation and Results

The proposed metric and heuristics are evaluated under a fixed time budget (900 s) using lazy predicate abstraction and value analysis techniques. Verification attempts typically yield “unknown” for many instances, but the subsequent under-approximation algorithm computes practical statement coverage ratios.

Empirically, the under-approximation aligns within 20–50% of the CPAchecker over-approximation (which may count unreachable code). The custom heuristic improves coverage estimation quality: in 22 of 25 benchmarks, it matches or improves the baseline; in 11 cases, it delivers strictly higher coverage. Thus, within resource limitations, the approach provides a rigorous measure of progress.

5. Relationship to Broader Observation Coverage Metrics

While the formalism targets statement coverage in software model checking, its principles generalize to observation coverage metrics wherever the “observed” behaviors can be mapped onto automata or event sequences:

• The definition of coverage in terms of analyzed prefixes generalizes to observable events, outputs, or stateful phenomena by replacing “statement” with “observation” and the ART with a behavioral automaton.
• The predicate $\psi_{obs}$ would evaluate whether an observation trace lies within the analyzed region, allowing the metric to address completeness and representativeness in observational contexts (e.g., runtime monitoring, sensor network coverage).
• Iterative under-approximation via trace generation and scoring heuristics applies equally: observations are considered covered only if they occur in concretely verified execution traces.

This mapping provides a blueprint for constructing observation coverage metrics in fields such as runtime verification, protocol monitoring, or event-driven testing where exhaustive analysis is infeasible.

6. Implementation Considerations and Limitations

Computational requirements are dictated by model checker efficiency, ART size, and the dimensionality of the analyzed system. To deploy the coverage metric effectively:

• ART and Assumption Automaton representation must scale to large state spaces.
• Heuristic trace prioritization is crucial in resource-bounded environments; naive traversal yields lower quality under-approximations.
• Special care must be taken to exclude unreachable code and spurious artifacts, as naive coverage computation risks inflation by dead states.

The metric is primarily suited to settings where terminating executions are well-defined and where the semantics of “exploration” map robustly to observed behavior. In highly non-deterministic or continuously reactive systems, adaptation may require domain-specific analogues to the CFA and ART concepts.

7. Summary and Impact

The verification coverage metric adapts classic statement coverage to the reality of incomplete formal verification, using ARTs and Assumption Automata to rigorously define “coverage” in terms of concretely analyzed prefixes of terminating executions. It enables intermediate progress measurement and under-approximation when full analysis is intractable, and performs well in practice with the support of custom heuristics. These concepts generalize naturally to observation coverage in diverse domains, offering an actionable pathway for quantifying the completeness and representativeness of observed behaviors under practical constraints (Castaño et al., 2017).