ODD Coverage Score for AI Systems Verification

• OCS is a quantitative metric certifying verification completeness by measuring the ratio of exercised, relevant discretized parameter bins to all safety-critical bins.
• The methodology involves discretizing continuous ODD parameters into criticality-weighted bins, applying constraint-based filtering, and performing dimension reduction to maintain focus on relevant subspaces.
• OCS supports Safety-by-Design by delivering auditable evidence of testing coverage, guiding targeted scenario generation for aviation certification.

The Operational Design Domain Coverage Score (OCS) is a quantitative metric designed to certify the completeness of verification efforts for AI-based systems operating within high-dimensional and safety-critical Operational Design Domains (ODDs). It is expressly formulated to address the requirements of aviation certification bodies such as EASA, which mandate demonstrable completeness of coverage over the system’s ODD. OCS serves as a rigorous and traceable indicator that all relevant parameter combinations in the ODD, after discretization, constraint-based filtering, and dimension reduction, have been adequately exercised and tested, providing foundational support for Safety-by-Design arguments in AI/ML system certification (Stefani et al., 2 Apr 2026).

1. Mathematical Definition

Let $p_1, \ldots, p_n$ represent the system’s ODD parameters. Each $p_i$ undergoes a discretization, partitioning its domain into $|B_i|$ bins based on the parameter’s criticality $c_i$, with criticality-weighted bin widths. The full bin space is

$$\mathcal{B} = B_1 \times B_2 \times \cdots \times B_n,$$

encompassing all possible parameter bin combinations. Physically plausible or safety-critical combinations are retained using a Boolean constraint predicate

$$\mathcal{C} : \mathcal{B} \rightarrow \{\mathrm{true}, \mathrm{false}\}.$$

The set of relevant bins is

$$\mathcal{B}_{\mathrm{rel}} = \{ b \in \mathcal{B} \mid \mathcal{C}(b) = \mathrm{true} \}.$$

For each executed scenario $d \in \mathcal{D}$, associate a unique bin $b(d)$. The set of covered bins, constrained to relevance, is

$$\mathcal{B}_{\mathrm{cov}} = \{ b(d) \mid d \in \mathcal{D} \} \cap \mathcal{B}_{\mathrm{rel}}.$$

The ODD Coverage Score is then

$p_i$0

An OCS of 1.0 asserts full coverage in the EASA sense of completeness.

2. Parameter Discretization, Constraint-Based Filtering, and Dimension Reduction

Parameter Discretization

Continuous ODD parameters $p_i$1 are discretized into $p_i$2 linearly or non-linearly spaced bins, with bin widths $p_i$3 set inversely proportional to criticality $p_i$4. Parameters of higher safety relevance receive finer partitioning.

Constraint-Based Filtering

The combinatorial parameter space $p_i$5 is filtered via $p_i$6, which encodes expert knowledge of physical, logical, and safety constraints (e.g., bounds of flight envelopes or exclusion of implausible aircraft configurations). Only the set $p_i$7 is retained for coverage analysis.

Criticality-Based Dimension Reduction

Parameters with uniformly low safety impact may be collapsed (merged into coarser bins) or eliminated, reducing the dimensionality $p_i$8 or the cardinality of $p_i$9. This mitigates the curse of dimensionality and focuses coverage efforts on pertinent ODD subspaces.

3. Methodological Procedure

The OCS calculation follows a structured, auditable process:

1. Input Specification: Define parameters $|B_i|$0; provide dataset $|B_i|$1; encode constraints $|B_i|$2
2. Discretization: For each $|B_i|$3: establish bin count $|B_i|$4; determine bin edges $|B_i|$5.
3. Cartesian Product Generation: Construct $|B_i|$6 implicitly using all bin ranges.
4. Constraint Filtering: Apply $|B_i|$7 to select $|B_i|$8.
5. Coverage Recording: For each scenario $|B_i|$9 in $c_i$0, map to bin $c_i$1 and update $c_i$2 if $c_i$3 holds.
6. Score Computation: Compute $c_i$4.
7. Reporting Gaps: List $c_i$5 for targeted scenario generation.

4. Metric Properties, Thresholds, and Normalization

The OCS naturally lies in the closed interval [0,1], with unity denoting exhaustive coverage of all physically and safety-relevant ODD bins. In practical applications, minimal statistical gaps may be tolerated by setting an OCS threshold (e.g., $c_i$6), subject to explicit justification within the safety case. There is no further normalization: the score is inherently a fraction of relevant bins covered by executed scenarios (Stefani et al., 2 Apr 2026).

5. Illustrative Example

Consider a 2-dimensional ODD:

• $c_i$7 with criticality $c_i$8 (5 bins of width 2);
• $c_i$9 with criticality $$\mathcal{B} = B_1 \times B_2 \times \cdots \times B_n,$$0 (2 bins of width 5).

There are $$\mathcal{B} = B_1 \times B_2 \times \cdots \times B_n,$$1 possible bin combinations. A constraint restricts analysis to bins with $$\mathcal{B} = B_1 \times B_2 \times \cdots \times B_n,$$2, yielding $$\mathcal{B} = B_1 \times B_2 \times \cdots \times B_n,$$3. If executed scenario data cover 5 of these relevant bins, then

$$\mathcal{B} = B_1 \times B_2 \times \cdots \times B_n,$$4

The set of uncovered bins is explicitly identified, enabling targeted scenario generation to incrementally close remaining coverage gaps.

6. Role in Certification and Safety-by-Design

The OCS is constructed to align with EASA’s DM-08 and LM-16 objectives, providing direct, quantifiable evidence of verification completeness across the joint ODD (Stefani et al., 2 Apr 2026). Iterative scenario generation—guided by the reported set of uncovered relevant bins—enables systematic progression toward an OCS of 1.0, thereby closing all safety-critical verification gaps. The process produces auditable artifacts documenting discretization strategies, constraint rationale, and dimension reduction, satisfying regulatory requirements for transparency and traceability and fully supporting a Safety-by-Design approach in AI/ML aviation systems.

7. Applications and Limitations

OCS provides a standardized, scalable, and formally grounded approach for demonstrating ODD coverage in high-dimensional, safety-critical domains, notably in AI-based mid-air collision avoidance research. A plausible implication is that the method can generalize to other safety-critical application domains that face similar certification demands and high-dimensional operational spaces. OCS is not claimed to assess scenario validity or effectiveness within covered bins; it measures solely the exercised breadth over the concretized ODD, subject to the fidelity of discretization, relevance constraints, and the criticality-weighted decomposition as implemented (Stefani et al., 2 Apr 2026).