A Scalability Analysis of Quantitative Confidence Assessment Methods for Assurance Cases

Abstract: This paper proposes a model to estimate the decision complexity and effort required to apply quantitative confidence assessment methods to assurance cases. The model considers both the worst and average case for these measures and characterizes how these quantities scale with argument size. Prior work has indicated that the additional effort required to apply these methods is a barrier to their adoption by assurance case practitioners. Researchers developing new methods, or improving existing methods, can use this model to estimate the effort required to apply their method. The proposed model is parameterized using data from published case studies and is applied to three existing quantitative confidence assessment methods: the Bayesian Belief Network method, the Dempster-Shafer Theory method, and the Certus method. The results show that, while Certus has the highest worst-case decision complexity, its average-case effort is lower than the BBN and DST methods.

• The paper introduces a formal model that quantifies decision complexity and user effort in applying quantitative confidence assessment methods to assurance cases.
• Comparative analysis reveals Certus's high worst-case complexity, while average-case tool support reduces annotation effort compared to BBN and DST.
• The study offers actionable guidance for CAM designers, tool developers, and practitioners on balancing expressivity with usability in assurance cases.

A Scalability Analysis of Quantitative Confidence Assessment Methods for Assurance Cases

Introduction

Assurance cases (ACs) are the primary structured artifacts used to systematize arguments and evidence supporting claims about the safety and security of critical systems. Given their increased adoption across regulated domains, systematic methods for evaluating confidence in the top-level claims of assurance cases have become crucial. Quantitative confidence assessment methods (CAMs), leveraging probabilistic and evidential reasoning frameworks, are increasingly applied, yet their scalability and usability in practical, large-scale cases remain unaddressed obstacles to their widespread adoption.

Simon Diemert and Jens H. Weber, in "A Scalability Analysis of Quantitative Confidence Assessment Methods for Assurance Cases" (2606.15480), introduce a formal model to estimate the decision complexity and user effort required when applying such quantitative CAMs to ACs of varying size. The model offers both worst-case and average-case analyses and is parameterized using data from recent, large assurance case studies. It is then used for a comparative evaluation of three significant methods: Bayesian Belief Networks (BBN), Dempster-Shafer Theory (DST), and the mixed quantitative-qualitative Certus method.

Overview of Quantitative Confidence Assessment Methods

Bayesian Belief Network (BBN) Method

The BBN method models the argument structure as a Bayesian Network, where each claim is annotated with subjective belief probabilities. For each parent node, users provide:

• Choice of combinator (noisy-AND or noisy-OR),
• A leakage parameter capturing residual uncertainty, and
• Per-child link weights modulating strength of support.

These parameters define the conditional probability tables used to propagate beliefs up the argument tree.

Dempster-Shafer Theory (DST) Method

The DST-based approach generalizes the probabilistic representation, distinguishing between decision and confidence axes on five-point ordinal scales. Each leaf is annotated with both a decision and confidence rating. At each argument step, users configure combinators and multiple forward/reverse propagation parameters and rules, allowing for nuanced modeling of evidential impact, but with significantly higher annotation complexity per node.

Certus Method

Certus is a mixed (quantitative-qualitative) method using a domain-specific language and possibility theory. Claims are annotated with linguistic belief levels (nine canonical fuzzy categories) or precise fuzzy set values. Internal nodes have expressive belief propagation expressions, optionally using macros and conditional constructs, which enable concise or highly customized specification of how children support their parent in the argument.

The workflow of Certus is illustrated on an adaptive cruise control argument fragment, visualizing how belief levels and propagation macros are embedded at node and step granularity.

Figure 1: Applying Certus to an argument fragment; belief annotations and propagation expressions are shown near each node.

Formal Model of Scalability

The scalability analysis models ACs as $n$-ary trees of height $h$, with each non-leaf node representing an argument step and each leaf node representing evidence. For each CAM, two primary components are modeled:

• Decision Complexity: Number of user decisions required, split into propagation-step decisions ($p$ per parent node) and leaf valuation decisions ($v$ per leaf).
• Effort Estimate: Time expenditure in minutes or hours, parameterized by average time per decision type.

Both worst-case (all parameters set explicitly) and average-case (defaults and tool support reduce decision burden for a proportion of nodes, reflecting real practice) analyses are provided.

A generic AC structure used for evaluation is depicted below, showing the recursive tree decomposition adopted for the complexity counting.

Figure 2: Model of assurance case argument structure as an $n$-ary tree for scalability analysis.

The parameterization of tree branching and leafening is anchored using several published large-case studies (e.g., from the CERN LHC Machine Protection System and other industrial-scale arguments), aligning the model to empirically observed sizes and structures.

Comparative Scalability Results

The three analyzed CAMs were instantiated with parameters capturing both their maximum possible per-node annotation complexity and expected average-case tool-assisted practice. The model computes the total decisions required and the time (in hours) to fully annotate a large case (typified by $h=10$, roughly $350$ nodes).

Key findings:

• Worst-case complexity: Certus can entail an exponential explosion of decisions per node due to the full expressivity of custom propagation expressions over nine-valued fuzzy variables. At this scale, the division in total decisions is stark: Certus ($>$10,000), DST ($\approx$1,700), BBN ($\approx$1,000).
• Average-case complexity: With typical tool support and macros, Certus' required decisions drop substantially, ending up lower than both BBN and DST (Certus: 390, BBN: 480, DST: 735 for $h$0).
• Effort (hours): Average-case effort is estimated at 13 hours for Certus, 14 for BBN, and 26 for DST for industrial-scale assurance cases.

The scaling trends as a function of case size ($h$1) are visualized below.

Figure 3: Scalability analysis: decision complexity and effort estimates as a function of argument height for BBN, DST, and Certus methods.

The results quantitatively demonstrate that, while the expressivity of Certus renders its worst-case complexity prohibitive, for most practical scenarios (where macros and tool defaults are leveraged), Certus can actually result in lower annotation effort than DST or BBN.

Implications and Future Directions

This work is the first to formally quantify the scalability of quantitative CAMs for assurance cases, operationalizing "decision complexity" and practitioner effort. The analysis delivers actionable guidance:

• CAM Designers: Quantitative CAMs should be evaluated with respect to both worst- and average-case user annotation burdens before being recommended for industrial adoption or scaled deployment.
• Tool Developers: The pronounced reduction in effort from the worst- to average-case hinges on the quality of tool support (effective propagation macros, intelligent defaults). There is value in further automating selection or inference of repetitive parameters to mitigate user effort.
• Practitioners: The choice of CAM should be aligned with the size of the AC, the perceived need for propagation expressivity, and the available tool support.

Theoretical implications include the need for new metrics of CAM usability beyond expressivity and inferential rigor, incorporating practical burden. Empirical validation of time and cognitive effort parameters in varied domains and user populations remains a future research agenda.

Several limitations are noted: the use of idealized $h$2-ary trees does not capture irregular structures of real-world ACs, the decision difficulty is not discriminated between types (e.g., ordinal versus continuous annotations), and average task durations are assumed rather than measured.

Conclusion

The presented scalability analysis model fills a critical knowledge gap regarding the practicality of quantitative confidence assessment methods for assurance cases. It rigorously quantifies both the annotation burden and time expenditure for three principal methods, exposing the tradeoff between expressivity/flexibility and user effort. The analysis demonstrates that mixed qualitative-quantitative methods (e.g., Certus), when coupled with strong macro and tool support, balance flexibility and usability more favorably than conventional probabilistic or evidential approaches, at least for average-case practice.

Future work should focus on empirical calibration of effort parameters, broader structural modeling of assurance cases, and the integration of cognitive usability studies to further refine decision complexity models and support further methodological innovation in assurance argument confidence quantification.