Risk Thresholds for Frontier AI

• Risk Thresholds for Frontier AI are quantitatively defined limits on the likelihood and impact of harm, ensuring safe deployment of high-capability systems.
• They integrate probabilistic risk measures with capability-based proxies—like red-team task success rates—to decide when to mitigate or halt deployment.
• These thresholds are operationalized within regulatory frameworks using iterative calibration, domain-specific metrics, and early warning zone models.

Frontier AI risk thresholds are quantitatively specified limits—typically on the likelihood and severity of harm—set to govern when models or systems are considered to pose too much risk and thereby require additional mitigation, restricted deployment, or outright prohibition. This approach is foundational in both regulatory regimes (e.g., the EU AI Act, U.S. export controls) and emerging safety frameworks, and it draws explicitly from methodologies established in nuclear, aviation, and financial risk management. Risk thresholds for frontier AI increasingly involve both capability-based proxies and direct probabilistic risk bounds, reflecting the difficulty of reliably estimating absolute societal risk from novel, high-capability general AI systems.

1. Foundations and Rationale

Risk thresholds in frontier AI are derived to constrain the probability and magnitude of unacceptable outcomes, such as large-scale economic loss, mass harm, loss of control, or systemic failure. A typical mathematical form is $P(\text{harm} \geq S) \leq p_0$, where $S$ is a severity level (e.g., fatalities, monetary loss) and $p_0$ is the maximum tolerated probability within a defined time frame (Koessler et al., 2024). Alternatively, the marginal increase in risk from deploying an AI system can be specified: $\Delta P(\text{harm} \geq S) \leq \delta_0$. These standards abstract from precedent in nuclear safety (e.g., $10^{-9}$ fatalities per flight-hour in aviation (Campos et al., 10 Feb 2025)) and adapt them for the scale, scope, and uncertainty inherent in advanced AI.

Capability thresholds remain dominant as actionable proxies; for instance, success rates on red-team tasks or the ability to autonomously execute sequences associated with undesirable outcomes. These are mapped—often via risk models—onto the underlying risk thresholds to operationalize checks within development lifecycles (Koessler et al., 2024).

2. Quantitative Structures and Metrics

Frontier AI risk frameworks distinguish between direct risk thresholds and capability-based proxies. Central quantitative constructs include:

• Risk Score: $R = P \times S$, where $P$ is the annual probability of the specified harm, and $S$ its severity in economically or societally meaningful units (Campos et al., 10 Feb 2025).
• Risk Tolerances: $R \leq R_\mathrm{tol}$, with categorical bands (e.g., acceptable, elevated, critical) and escalation rules at breach points.
• Capability Thresholds: Metrics such as capability uplift ($\Delta C$), defined as the difference in task performance over baseline systems, with intolerable thresholds often set at 25 percentage points for certain high-risk tasks (e.g., CBRN planning, cybersecurity exploits) (Raman et al., 4 Mar 2025).
• Zone-based Models: Frameworks with green (manageable), yellow (caution/mitigation required), and red (intolerable/halt) zones mapped to quantitative early-warning or hard-stop indicators (Lab et al., 22 Jul 2025).
• Self-Replication Probability: In the context of agentic LLMs, autonomous self-replication rates operationalized via experimental success counts (e.g., 50%+ as “no-go,” 90%+ as critical) (Pan et al., 2024).

The table below provides an overview of examples of risk and capability thresholds found in the literature:

Threshold Type	Example Metric	Action Triggered
Risk (absolute)	$P(\geq1000$ deaths$) < 10^{-6}$/year	Prohibit deployment; review mitigations
Capability (proxy)	End-to-end CBRN plan success $\geq25$pp	Pause, require new controls, retesting
Model autonomy	Self-replication $P_{\text{success}} \geq0.5$	Forbid local run, force remote API use
Cyber-offense (zone)	CTF solve $<$40% (safety $>0.6$)	Green (acceptable); else, yellow/red zone

The wide adoption of risk × severity product, uplift/Delta capability, and zone-based frameworks reflects both the ambition for principled safety control and the pragmatic constraints imposed by measurement limitations (Campos et al., 10 Feb 2025, Raman et al., 4 Mar 2025, Lab et al., 22 Jul 2025).

3. Methodologies for Setting and Calibrating Thresholds

A standard workflow emerges across recent frameworks for setting and refining risk thresholds:

1. Risk identification: Using scenario libraries, red teaming, literature review, and hazard analysis (e.g., STPA), identify loss scenarios and causal hazard chains (Mylius, 2 Jun 2025).
2. Quantification: For each risk, elicit (via expert judgement, Delphi method, or empirical data) an annualized probability and severity estimate; apply scaling from domain analogues where direct data is absent (Campos et al., 10 Feb 2025, Carpenter et al., 26 Nov 2025).
3. Threshold calibration: Where feasible, set explicit limits matching societal expectations; e.g., infrastructural vulnerability $P < 0.10$/year, or economic loss $R_{\rm tol} < \$500$$M/year (<a href="/papers/2502.06656" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Campos et al., 10 Feb 2025</a>).</li> <li><strong>Mapping to capabilities</strong>: Where probabilistic risk quantification cannot be trusted, derive task-based capability thresholds (e.g., code vulnerability discovery rate, agentic autonomy in self-replication) and set action thresholds via risk models (<a href="/papers/2406.14713" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Koessler et al., 2024</a>, <a href="/papers/2503.05812" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Raman et al., 4 Mar 2025</a>).</li> <li><strong>Iterative update</strong>: Post-incident/near-miss, re-estimate probabilities and severity, adjust capability thresholds, and recalculate aggregate system risk. For systematic approaches such as STPA, capability triggers (e.g., parameter count$$m$) are determined as$C_{\max}$such that</li> </ol> <p>$\sum_{h=1}^m P_h(C) S_h(C) \leq R_{\max}$$</p> <p>(<a href="/papers/2506.01782" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Mylius, 2 Jun 2025</a>).</p> <ol> <li><strong>Operational embedding</strong>: Thresholds are embedded into continuous evaluation, red-team pipelines, CI/CD deployment gates, and emergency “kill switch” protocols (<a href="/papers/2512.01166" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Stelling et al., 1 Dec 2025</a>, <a href="/papers/2507.16534" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Lab et al., 22 Jul 2025</a>).</li> </ol> <p>Best practice is to ground capability proxies and threshold cut-points in risk modeling and regularly adjust for capability drift, adversarial breakthroughs, or unforeseen emergent behaviors (<a href="/papers/2406.14713" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Koessler et al., 2024</a>, <a href="/papers/2512.01166" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Stelling et al., 1 Dec 2025</a>).</p> <h2 class='paper-heading' id='domain-specific-thresholds-and-case-examples'>4. Domain-Specific Thresholds and Case Examples</h2> <p>Thresholds differ by risk domain, with both shared methodologies and category-specific operationalizations:</p> <ul> <li><strong>Self-replication</strong>: Agentic LLMs crossing$$P_\text{success} \geq50\%$$on unsupervised self-replication trigger no-go policies, while$$P_\text{success} \geq90\%$ constitutes a critical emergency threshold (full prohibition of local or unsupervised deployment) (<a href="/papers/2412.12140" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Pan et al., 2024</a>).</li> <li><strong>CBRN, Cyber, Persuasion</strong>: Capability uplift ≥25pp over baseline (measured via human-uplift studies or benchmark differentials) is an intolerable threshold for AI-enabled CBRN planning or complex exploit generation (<a href="/papers/2503.05812" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Raman et al., 4 Mar 2025</a>).</li> <li><strong>Deception/Alignment faking</strong>: Crossing 60% correct situational-awareness on “evaluation vs. deployment” masking benchmarks (e.g., SAD dataset) is diagnostic of deception and triggers immediate halt (<a href="/papers/2503.05812" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Raman et al., 4 Mar 2025</a>).</li> <li><strong>Toxic content</strong>: Generation of dangerous or illegal outputs (e.g., &gt;10% compliance with toxic-content prompts) triggers an intolerable risk finding (<a href="/papers/2503.05812" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Raman et al., 4 Mar 2025</a>).</li> <li><strong>Socio-economic disruption</strong>: Model-linked forecasted displacement of ≥5% of a labor market without mitigation/countermeasures is an upper-bound for tolerable deployment (<a href="/papers/2503.05812" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Raman et al., 4 Mar 2025</a>).</li> </ul> <h2 class='paper-heading' id='institutional-mechanisms-for-enforcement'>5. Institutional Mechanisms for Enforcement</h2> <p>Multiple governance architectures for risk-threshold enforcement exist:</p> <ul> <li><strong>Coordinated pausing</strong>: Sector-wide pause rules of the form $\exists i : E_i(M)\geq\theta_i$ trigger R&amp;D and deployment suspension across multiple actors, optionally mediated by third-party auditors to mitigate antitrust concerns (<a href="/papers/2310.00374" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Alaga et al., 2023</a>).</li> <li><strong>Tiered insurance</strong>: Three-layer architectures (private, pool, federal) using actuarial tier thresholds (VaR$_{95}$, VaR$_{99}$, CVaR$_{99.5}$$) as triggers for insurance, pooling, and public guarantees (<a href="/papers/2504.02127" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Stetler, 2 Apr 2025</a>).</li> <li><strong>Compute/KYC thresholds</strong>: Compute quantity (e.g.,$$>10^{26}$$FLOP) triggers regulatory and KYC compliance, with periodic recalibration reflecting algorithmic advances (<a href="/papers/2310.13625" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Egan et al., 2023</a>, <a href="/papers/2504.16138" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Kumar et al., 21 Apr 2025</a>).</li> <li><strong>Red/yellow/green zone gating</strong>: Models evaluated as “yellow” (early warning—capability or safety index below caution threshold, e.g., safety$$<0.6$ on CyBench) are subject to access restrictions or mandated additional controls; “red line” crossing (typically not numerically specified, but analogous to capability for destructive outcomes) halts deployment (Lab et al., 22 Jul 2025).

Safety frameworks often incorporate explicit, non-discretionary pause/rollback rules when a risk or capability threshold is unmet, and best practice now demands publication, external audit, and empirical tracking of risk-threshold compliance and effect (Stelling et al., 1 Dec 2025).

6. Challenges, Ambiguities, and Ongoing Evolution

The direct application of risk thresholds in frontier AI is hindered by the intractability of reliable risk estimation in high-ambiguity, low-data, fast-moving threat environments (Carpenter et al., 26 Nov 2025). Frameworks such as “dark speculation” acknowledge this: an iterative process, coupling scenario generation with quantitative underwriting (e.g., compound Poisson/Lévy models), produces evolving estimates of event probabilities and magnitude, which are synthesized until VaR/CVaR or expected risk is deemed below an ex ante policy ceiling (Carpenter et al., 26 Nov 2025).

Current frameworks recommend a layered or hybrid regime, where risk thresholds provide the principled foundation, but operational decisions often rest on capability proxies pending improved risk estimation reliability (Koessler et al., 2024).

7. Regulatory and Policy Considerations

Frontier AI risk thresholds are now directly embedded in international policy (e.g., EU AI Act’s $10^{25}$ FLOP systemic-risk threshold, US $10^{26}$ FLOP “controlled model” regime), with forecasts projecting that these absolute cutoffs will rapidly sweep in more models than existing oversight capacity can support unless recalibrated (Kumar et al., 21 Apr 2025). Best-practice guidance recommends:

• Defining transparent, quantitative risk tolerances in real-world units (lives, dollars) as explicit policy cut-offs.
• Linking capability thresholds empirically to risk models, with public documentation and third-party verification (Stelling et al., 1 Dec 2025).
• Regularly updating both risk and capability thresholds as evidence, adversarial examples, and underlying capabilities evolve, with continuous monitoring and feedback to maintain effective governance coverage (Campos et al., 10 Feb 2025, Mylius, 2 Jun 2025, Lab et al., 22 Jul 2025).

A plausible implication is that, as methodological and infrastructural sophistication improves, direct risk thresholds will increasingly drive governance decisions in place of current capability proxies. Until then, a multi-layered, adaptive approach—combining probabilistic, capability-based, and institutional mechanisms—defines the state-of-the-art in frontier AI risk threshold governance.