Probability of Acceptable Performance (PrAP)

• Probability of Acceptable Performance (PrAP) is defined as the probability that a system, model, or design satisfies a predefined acceptability criterion amid uncertainty.
• It is computed using methodologies like reliability analysis, surrogate modeling, and concentration inequalities, which balance between failure probabilities and performance thresholds.
• PrAP bridges average performance evaluation and strict failure analysis, influencing design optimization, robustness certification, and clinical sample-size planning.

Searching arXiv for recent and relevant papers on Probability of Acceptable Performance and related formulations. Probability of Acceptable Performance (PrAP) is a probabilistic quantity that denotes the chance that a system, model, or design satisfies a predefined acceptability criterion under uncertainty. Across the cited literature, PrAP appears in several technically distinct but structurally related settings: as the complement of failure probability in reliability analysis, as the probability that a prediction model’s calibration slope falls within a prespecified interval, as the probability that a performance index satisfies a threshold, and as a probabilistic robustness quantity under stochastic perturbations (Zhang et al., 2022, Nikbakht et al., 2019, Aleti et al., 2018, Pavlou et al., 17 Sep 2025, Iwazaki et al., 2020). The unifying idea is that performance is treated as a random variable induced by uncertain inputs, sampling variability, or environmental variation, and PrAP is the probability mass assigned to an “acceptable” region of outcomes.

1. Conceptual definitions and core formulations

In the most general form appearing in the literature, PrAP is defined by specifying an acceptable event and then taking its probability. In uncertainty propagation for software performance estimation, if $Y$ is a scalar performance index and $T$ is a deterministic acceptance threshold, the acceptable event is $\mathcal{A}=\{Y\le T\}$ or $\{Y\ge T\}$ if larger values are preferred, and

$$\mathrm{PrAP}=P(Y\in\mathcal A)=E[\mathbf{1}_{\mathcal A}(Y)]=\int_{\mathcal A} f_Y(y)\,dy$$

with the specialized form $\mathrm{PrAP}=\int_{-\infty}^{T} f_Y(y)\,dy$ for $Y\le T$ (Aleti et al., 2018).

In structural and system reliability, the formulation is typically expressed through a limit-state or performance function $g(x)$. For random input $x\in\mathbb{R}^d$ with density $f_X(x)$, acceptable performance corresponds to $T$0, giving

$T$1

while the failure probability is

$T$2

(Nikbakht et al., 2019). A closely related formulation in subset simulation writes PrAP as $T$3, reflecting an alternative sign convention for the performance function; in that source, the quantity denoted PrAP is the probability of the target event $T$4, factorized through nested subsets $T$5 (Šehić et al., 2020). This variation underscores that the meaning of “acceptable” depends on the chosen event definition and sign convention.

In probabilistic robustness assessment against functional perturbations, the relevant quantity is the probability that a classifier’s output remains sufficiently stable under random perturbations. Let $T$6 be a classifier, $T$7 an input, $T$8 a perturbation family with $T$9, $\mathcal{A}=\{Y\le T\}$0, and $\mathcal{A}=\{Y\le T\}$1. With

$\mathcal{A}=\{Y\le T\}$2

and

$\mathcal{A}=\{Y\le T\}$3

the PRoA value is

$\mathcal{A}=\{Y\le T\}$4

The source explicitly states that one may define $\mathcal{A}=\{Y\le T\}$5, with failure probability $\mathcal{A}=\{Y\le T\}$6 (Zhang et al., 2022).

In clinical prediction model development, PrAP is defined over the repeated-sampling distribution of a performance measure. If $\mathcal{A}=\{Y\le T\}$7 is the calibration slope estimated in validation for a model developed on a sample of size $\mathcal{A}=\{Y\le T\}$8, and $\mathcal{A}=\{Y\le T\}$9 is an a priori acceptability interval, then

$\{Y\ge T\}$0

(Pavlou et al., 17 Sep 2025). This formulation shifts attention from average performance to the probability that an individual fitted model attains acceptable performance.

2. Relation to failure probability, robustness, and acceptability regions

A recurring structural feature is that PrAP is defined by complementing or thresholding a failure event. In PRoA, a “failure” is the complement event

$\{Y\ge T\}$1

so $\{Y\ge T\}$2, and if at most $\{Y\ge T\}$3 failure-probability is allowed, then $\{Y\ge T\}$4 is required (Zhang et al., 2022). In reliability estimation via Hamiltonian MCMC, the same complementarity appears as $\{Y\ge T\}$5 (Nikbakht et al., 2019). For discrete-state performance functions, success probability is given as

$\{Y\ge T\}$6

which is the PrAP under that formulation (Vořechovský, 2022).

For safety-critical MooN systems subject to proof tests, the probability of acceptable performance at time $\{Y\ge T\}$7 is identified directly with system availability: $\{Y\ge T\}$8 and the average PrAP over a full-test interval is

$\{Y\ge T\}$9

where $$\mathrm{PrAP}=P(Y\in\mathcal A)=E[\mathbf{1}_{\mathcal A}(Y)]=\int_{\mathcal A} f_Y(y)\,dy$$0 denotes probability of failure on demand (Brissaud et al., 2010). This embeds PrAP within classical availability analysis rather than within a generic performance-threshold formulation.

In Bayesian quadrature optimization, the same conceptual object is called “probabilistic threshold robustness (PTR).” For controllable design parameter $$\mathrm{PrAP}=P(Y\in\mathcal A)=E[\mathbf{1}_{\mathcal A}(Y)]=\int_{\mathcal A} f_Y(y)\,dy$$1, environmental parameter $$\mathrm{PrAP}=P(Y\in\mathcal A)=E[\mathbf{1}_{\mathcal A}(Y)]=\int_{\mathcal A} f_Y(y)\,dy$$2, performance function $$\mathrm{PrAP}=P(Y\in\mathcal A)=E[\mathbf{1}_{\mathcal A}(Y)]=\int_{\mathcal A} f_Y(y)\,dy$$3, and threshold $$\mathrm{PrAP}=P(Y\in\mathcal A)=E[\mathbf{1}_{\mathcal A}(Y)]=\int_{\mathcal A} f_Y(y)\,dy$$4,

$$\mathrm{PrAP}=P(Y\in\mathcal A)=E[\mathbf{1}_{\mathcal A}(Y)]=\int_{\mathcal A} f_Y(y)\,dy$$5

(Iwazaki et al., 2020). Here PrAP is a function over the design space, not a single scalar, and the problem becomes one of maximizing this function or estimating its level set.

These formulations share the same logical pattern: define acceptability through a subset of the outcome space, then compute the probability assigned to that subset. A plausible implication is that PrAP is less a single metric than a family of probability-based performance summaries whose operational meaning is determined by the acceptability event.

3. Statistical estimation and concentration-based certification

Several lines of work focus on estimating PrAP with statistical guarantees when direct exhaustive evaluation is infeasible. In PRoA, for i.i.d. Bernoulli samples $$\mathrm{PrAP}=P(Y\in\mathcal A)=E[\mathbf{1}_{\mathcal A}(Y)]=\int_{\mathcal A} f_Y(y)\,dy$$6 with

$$\mathrm{PrAP}=P(Y\in\mathcal A)=E[\mathbf{1}_{\mathcal A}(Y)]=\int_{\mathcal A} f_Y(y)\,dy$$7

Hoeffding’s inequality yields

$$\mathrm{PrAP}=P(Y\in\mathcal A)=E[\mathbf{1}_{\mathcal A}(Y)]=\int_{\mathcal A} f_Y(y)\,dy$$8

and to guarantee $$\mathrm{PrAP}=P(Y\in\mathcal A)=E[\mathbf{1}_{\mathcal A}(Y)]=\int_{\mathcal A} f_Y(y)\,dy$$9 with confidence $\mathrm{PrAP}=\int_{-\infty}^{T} f_Y(y)\,dy$0, one needs

$\mathrm{PrAP}=\int_{-\infty}^{T} f_Y(y)\,dy$1

(Zhang et al., 2022).

PRoA further employs the adaptive Hoeffding inequality of Zhao et al. ’16 for stopping-time-valid inference. With $\mathrm{PrAP}=\int_{-\infty}^{T} f_Y(y)\,dy$2 being $\mathrm{PrAP}=\int_{-\infty}^{T} f_Y(y)\,dy$3-subgaussian and $\mathrm{PrAP}=\int_{-\infty}^{T} f_Y(y)\,dy$4 an almost surely finite stopping time,

$\mathrm{PrAP}=\int_{-\infty}^{T} f_Y(y)\,dy$5

where PRoA uses the working bound

$\mathrm{PrAP}=\int_{-\infty}^{T} f_Y(y)\,dy$6

after setting $\mathrm{PrAP}=\int_{-\infty}^{T} f_Y(y)\,dy$7, $\mathrm{PrAP}=\int_{-\infty}^{T} f_Y(y)\,dy$8, and choosing $\mathrm{PrAP}=\int_{-\infty}^{T} f_Y(y)\,dy$9 so that $Y\le T$0 (Zhang et al., 2022). This enables sequential sampling until the estimated PrAP is either certifiably above or below the target threshold.

In the clinical prediction setting, PrAP is estimated from the sampling distribution of a performance measure. Under large-$Y\le T$1 asymptotics and a very large validation set, $Y\le T$2, and the calibration slope satisfies

$Y\le T$3

Assuming this Gaussian approximation,

$Y\le T$4

(Pavlou et al., 17 Sep 2025). The source also provides an approximate closed-form variance formula for $Y\le T$5 in terms of prevalence $Y\le T$6, C-statistic $Y\le T$7, and sample size $Y\le T$8 (Pavlou et al., 17 Sep 2025).

In PCE-based uncertainty propagation, PrAP estimation proceeds by first building a surrogate for $Y\le T$9, then evaluating either the induced CDF

$g(x)$0

through quadrature when $g(x)$1, or by surrogate sampling: $g(x)$2 (Aleti et al., 2018). This approach does not produce concentration bounds of the PRoA type, but it yields a computationally efficient estimator once the surrogate is constructed.

4. Computational methodologies across domains

The methodological diversity surrounding PrAP is considerable. The following table summarizes the main formulations appearing in the cited sources.

Setting	PrAP definition	Main computational approach
Functional perturbation robustness	$g(x)$3	Sequential sampling with adaptive concentration inequality
Reliability via HMCMC	$g(x)$4	ASTPA with HMCMC and inverse importance-sampling post-processing
Subset simulation with local surrogates	$g(x)$5 under that sign convention	Nested subsets, MCMC conditional sampling, local GP, PLS
Software performance estimation	$g(x)$6	Polynomial Chaos Expansion and surrogate sampling
Clinical prediction model development	$g(x)$7	Simulation-based framework and analytical approximation
Design under environmental uncertainty	$g(x)$8	GP surrogate, Bayesian quadrature, active learning
Safety-critical MooN systems	$g(x)$9 or $x\in\mathbb{R}^d$0	Closed-form availability and proof-test scheduling
Discrete-state reliability	$x\in\mathbb{R}^d$1	Adaptive sequential sampling with nearest-neighbor classification

In ASTPA, the target density is constructed as

$x\in\mathbb{R}^d$2

where $x\in\mathbb{R}^d$3 is a one-dimensional Gaussian likelihood in the limit-state value. Samples are then drawn from $x\in\mathbb{R}^d$4 using standard HMCMC or QNp-HMCMC, followed by inverse importance-sampling correction to estimate $x\in\mathbb{R}^d$5 and hence PrAP (Nikbakht et al., 2019). The method uses a smooth proxy-failure likelihood to focus sampling near the boundary $x\in\mathbb{R}^d$6.

Local subset approximation Subset Simulation factorizes the target event probability as

$x\in\mathbb{R}^d$7

where intermediate thresholds $x\in\mathbb{R}^d$8 are chosen adaptively, for example as the $x\in\mathbb{R}^d$9-quantile of current samples. Standard Subset Simulation is accelerated by replacing expensive evaluations of $f_X(x)$0 during MCMC with local GP surrogates, optionally combined with PLS for dimension reduction (Šehić et al., 2020).

PCE-based estimation represents the uncertain performance index as

$f_X(x)$1

with mean $f_X(x)$2 and variance

$f_X(x)$3

Once the surrogate is available, PrAP is obtained from quadrature or cheap surrogate sampling (Aleti et al., 2018).

In Bayesian quadrature optimization, a GP prior is placed on $f_X(x)$4, yielding a posterior mean for the PrAP function

$f_X(x)$5

and an upper bound on its posterior variance

$f_X(x)$6

(Iwazaki et al., 2020). This supports active learning for both optimization and level-set estimation.

For discrete-state performance functions, the surrogate is a nearest-neighbor classifier over an adaptive experimental design, and the next sample is chosen by maximizing the $f_X(x)$7-criterion, which combines local density and geometric cell volume (Vořechovský, 2022). This is explicitly designed for settings where the model output is categorical and gradient-based schemes are inapplicable.

5. Applications and empirical findings

In robustness certification of deep neural networks, PRoA was evaluated on CIFAR-10 and ImageNet-scale models under rotation, translation, scaling, hue, saturation, brightness+contrast, and Gaussian blur. With $f_X(x)$8 and confidence levels $f_X(x)$9, PRoA certified accuracies on 1000 random test images were reported to be within a few percent of empirical grid-search robustness and consistently higher than competing statistical baselines, specifically Agresti–Coull CI and SRC (Zhang et al., 2022). Run-time was described as typically a few milliseconds per sample on CIFAR-10 models and tens of milliseconds on ImageNet-scale nets such as ResNet-50 and ViT (Zhang et al., 2022).

In reliability estimation by local subset approximations, benchmark examples showed that local GP surrogates reduced the number of expensive limit-state evaluations by over $T$00, with reported reductions of $T$01 and $T$02 on 2-D linear and nonlinear tests, $T$03 on the 4-branch function, and $T$04 on the hypersphere example (Šehić et al., 2020). In low to moderate dimension, the paper states that local GP reduces 80–90% of expensive runs while keeping $T$05 estimates within a few percent of Subset Simulation; in high dimension, PLS and PCA recover most of the gain, with approximately 50% reduction at acceptable error levels (Šehić et al., 2020). Because that source uses $T$06 as the target event, the same computational claims apply to the associated event probability under its sign convention.

In ASTPA, comparative results against Component-wise MH Subset Simulation were reported across four examples, with lower coefficient of variation for HMCMC-based ASTPA at the same model-call budget, and QNp-HMCMC being the most efficient in high-dimensional and highly nonlinear settings (Nikbakht et al., 2019). The examples include dimensions $T$07 and $T$08, with exact failure probabilities ranging from $T$09 to $T$10 (Nikbakht et al., 2019).

For software performance estimation, three case studies from different phases of software development and heterogeneous application domains showed that PCE could accurately estimate the robustness of various performance indices with better than $T$11 accuracy and save up to 225 hours of performance evaluation time compared with Monte Carlo Simulation (Aleti et al., 2018). The technical summary further states that PCE-based PrAP estimates agreed with brute-force Monte Carlo to better than 97% accuracy, while total CPU time was typically 5–20% of pure Monte Carlo cost, implying an 80–95% reduction in wall-clock time (Aleti et al., 2018).

In clinical sample-size calculation, one reported example with $T$12, prevalence $T$13, $T$14, $T$15, $T$16, and $T$17 yielded approximately $T$18 (Pavlou et al., 17 Sep 2025). In a heart valve surgery case study with prevalence $T$19, $T$20, and $T$21, the standard calculation targeting $T$22 gave $T$23, whereas the PrAP-based calculation targeting $T$24 on $T$25 gave $T$26, with simulation-based resampling confirming $T$27 and $T$28 (Pavlou et al., 17 Sep 2025). The same summary reports that when adhering to existing recommendations, performance variability increased substantially as the number of predictors decreased, and with 5 predictors, $T$29 was around 50% (Pavlou et al., 17 Sep 2025).

In product development under environmental uncertainty, Bayesian quadrature optimization for PTR demonstrated that BPT-UCB and BPT-TS found designs maximizing PrAP faster than GP-UCB, StableOpt, BQO variants for the mean objective, and random sampling on synthetic benchmarks; BPT-LSE achieved F$T$30-scores near 1 in fewer samples than standard LSE, StableLSE, BQLSE, or random; and in a newsvendor simulation, BPT-UCB/TS located near-optimal inventory levels in approximately 10 times fewer simulations than comparison methods (Iwazaki et al., 2020).

For MooN safety systems, a 2oo6 example with full tests every 360 days and partial tests every 90 days produced $T$31, $T$32, and $T$33, hence $T$34. Optimizing the three partial-test times within the 360-day window yielded $T$35, corresponding to about a 10% improvement over the periodic schedule (Brissaud et al., 2010).

6. Assumptions, limitations, and recurring points of caution

A central limitation is that PrAP is only as meaningful as the chosen acceptability region. In the clinical prediction framework, acceptability is defined through a pre-specified interval $T$36 for the calibration slope, and the source emphasizes that this interval is chosen a priori (Pavlou et al., 17 Sep 2025). In threshold-based design optimization, acceptability is specified by a user-selected threshold $T$37 and, for level-set estimation, a target $T$38 (Iwazaki et al., 2020). This suggests that PrAP is inherently decision-dependent rather than an invariant property.

PRoA explicitly provides probabilistic rather than worst-case guarantees; small-probability adversarial failures remain possible (Zhang et al., 2022). The adaptive bound $T$39 is conservative, and many samples may be needed when $T$40 is very close to $T$41 (Zhang et al., 2022). The guarantee also assumes faithful sampling from the true perturbation distribution $T$42; biased or unable-to-sample modes may yield over-optimistic certificates (Zhang et al., 2022).

In Bayesian quadrature optimization, all methods assume the environmental-parameter distribution $T$43 is known exactly, and the integrals required for $T$44 and $T$45 may be expensive in high-dimensional $T$46. The source also notes that performance depends on GP kernel and noise-level specification (Iwazaki et al., 2020).

For local subset approximation, local GP regression becomes computationally impractical with increasing dimension, which motivates the use of PLS as a gradient-free reduction method (Šehić et al., 2020). Even so, the reported high-dimensional oscillator example shows that error relative to Subset Simulation can become substantial without additional reduction steps (Šehić et al., 2020).

For discrete-state performance functions, the method assumes continuous inputs with known joint PDF and becomes costly in very high dimension because exploration layers must be sufficiently dense to discover isolated failure regions. The source also warns that tiny fragmented failure sets may be missed unless the exploration set is made much denser or a smooth surrogate is used (Vořechovský, 2022).

In sample-size planning for clinical models, the analytical development assumes binary outcomes, a true logistic model, multivariate normal predictors for the analytical results, and large-sample approximations (Pavlou et al., 17 Sep 2025). The framework is presented as flexible and simulation-based extensions to other measures and outcomes are discussed, but the explicit analytical results concern the calibration slope (Pavlou et al., 17 Sep 2025).

7. Position within related research areas

PrAP sits at the intersection of uncertainty quantification, reliability analysis, statistical validation, robustness assessment, and design optimization. In reliability engineering, it is closely linked to probability of failure, probability of failure on demand, system availability, subset simulation, importance sampling, and rare-event estimation (Nikbakht et al., 2019, Šehić et al., 2020, Brissaud et al., 2010). In machine learning robustness, it provides a probabilistic alternative to worst-case certification by focusing on the probability of acceptable behavior under a specified perturbation distribution (Zhang et al., 2022). In surrogate-based design, it appears as an objective over design space under environmental uncertainty, motivating GP-based Bayesian quadrature and active learning (Iwazaki et al., 2020). In software performance and uncertainty propagation, it functions as a threshold probability computed from the propagated distribution of a performance index, with PCE offering an efficient surrogate route (Aleti et al., 2018). In prediction-model methodology, it reframes sample-size determination in terms of the probability that a developed model achieves acceptable calibration rather than only the expected value of a performance measure (Pavlou et al., 17 Sep 2025).

A recurring misconception is that PrAP is a single universally standardized metric. The cited sources show instead that the same term is used for related but non-identical constructs. Some define it as $T$47, others as $T$48 under a different sign convention, still others as $T$49, $T$50, $T$51, or $T$52 for classifier stability (Nikbakht et al., 2019, Šehić et al., 2020, Aleti et al., 2018, Pavlou et al., 17 Sep 2025, Brissaud et al., 2010, Zhang et al., 2022). The common denominator is not a fixed formula but the probabilistic evaluation of an acceptability criterion under uncertainty.

This breadth also clarifies the methodological significance of PrAP. It serves as a bridge between average-case performance summaries and binary pass/fail compliance statements. Rather than asking only whether expected performance is high, or whether worst-case failure is impossible, PrAP asks how likely acceptable performance is under the uncertainties that matter in a given application. The literature indicates that this question can be addressed through concentration inequalities, surrogate modeling, rare-event simulation, Hamiltonian Monte Carlo, Bayesian quadrature, or analytical approximations, depending on the structure of the underlying uncertainty and the computational regime (Zhang et al., 2022, Nikbakht et al., 2019, Šehić et al., 2020, Iwazaki et al., 2020, Pavlou et al., 17 Sep 2025).