Safety Warning Evaluation Framework

• Safety-warning-based evaluation frameworks are structured systems that issue discrete or graded warnings to assess AI risks using formal risk taxonomies and regulatory standards.
• They integrate self-evolving, multi-agent pipelines that dynamically recalibrate warning thresholds and harden tests in response to emerging vulnerabilities.
• The frameworks enhance AI safety by providing consistent, actionable risk assessments through evidence-driven feedback loops and proper scoring rules.

Safety-warning-based evaluation frameworks are principled, structured systems for assessing the potential or realized risk of AI system outputs by issuing discrete or graded warnings, typically grounded in formal risk taxonomies, regulatory standards, or operational policies. These frameworks aim to provide reliable, interpretable, and, most crucially, consistent warnings or risk assessments that can guide model deployment, system improvement, and stakeholder trust in safety-critical contexts. Recent advances in this field integrate self-evolving evaluation cycles, adversarial test generation, agentic pipeline design, and decision-theoretic scoring—augmenting static audits with dynamic, evidence-driven warning taxonomies (Wang et al., 30 Sep 2025, Taggart et al., 13 Feb 2025).

1. Conceptual Foundations and Formal Structure

A safety-warning-based evaluation framework starts from the need to move beyond static pass/fail audits, instead delivering actionable risk signals: discrete warnings (e.g., “low”, “medium”, “high”) or continuous risk scores in response to model outputs or behaviors. Central to this paradigm are:

• Multilevel Warning Scales: Warnings are often mapped to a monotonic, ordered set (e.g., ℓ₀ = Nil, ℓ₁ = Watch, ℓ₂ = Warning, ℓ₃ = Emergency), governed by mappings such as

$T: \{S_i\} \times \{C_j\} \to \mathcal{L}$

where $S_i$ are severity classes, $C_j$ are certainty bins, and $\mathcal{L}$ is the warning scale (Taggart et al., 13 Feb 2025).

• Taxonomic and Policy Foundations: Risk categories are grounded in regulatory texts, community standards, or formal semantic rules. For LLMs, these may follow multi-level taxonomies spanning 20–100+ risk subcategories (Yuan et al., 2024, Ying et al., 2024).
• Agentic and Self-Evolving Pipelines: Rather than fixed benchmarks, advanced frameworks implement multi-agent architectures that parse new policies, generate and iteratively harden tests, and dynamically recalibrate warning granularity (Wang et al., 30 Sep 2025).
• Consistency and Properness: Issued warnings are scored by strictly consistent rules (e.g., risk-matrix scores, strict calibration functions) ensuring decision-theoretical validity and comparability across systems (Taggart et al., 13 Feb 2025).

2. Multi-Agent and Self-Evolving Warning Generation

Contemporary frameworks such as SafeEvalAgent operationalize agentic warning generation via specialized LLM-driven agents:

• Specialist Agent ($\mathcal{A}_S$): Parses unstructured policy/regulatory text $R$ into atomic rules with explicit compliant/non-compliant guidance.
• Generator Agent ($\mathcal{A}_G$): Generates diverse question groups per rule, encompassing open-ended, adversarial, and multimodal variants.
• Evaluator Agent ($\mathcal{A}_E$): Applies policy-aware rubrics to issue warnings, potentially graded by magnitude (e.g., "low", "medium", "high" risk).
• Analyst Agent ($\mathcal{A}_A$): Analyzes failures and warning distributions, identifying patterns and informing further test hardening (Wang et al., 30 Sep 2025).

Crucially, these frameworks operate as self-evolving evaluation loops:

• Initialize test suites from regulatory knowledge extraction.
• Evaluate models, aggregate results (including warning levels).
• Analyze observed weaknesses/warning clusters.
• Automatically generate more challenging, targeted probes.
• Iterate, producing a progressively hardened, ever-more informative warning distribution.

Each iteration tightens the test suite and may recalibrate the warning thresholds and criteria based on empirical or policy feedback.

3. Warning Assignment, Severity Calibration, and Scoring

Warnings are typically mapped through explicit rules or learned rubrics, guided by measurable divergence from desired behavior or compliance templates. Common approaches include:

• Severity-tagged labeling: Each test item is assigned a warning value (such as $$z_q \in \{\mathrm{low}, \mathrm{medium}, \mathrm{high}\}$$) based on the distance from guidance (e.g., divergence from $\mathcal{G}_{\mathrm{should}}$) (Wang et al., 30 Sep 2025).
• Risk Matrices: Warnings are generated based on crossing probabilistic and severity thresholds, ensuring monotonicity: higher likelihood and more severe potential outcomes map to escalated warnings. The formal structure

$\text{Warning level} = \max_{i} T(S_i, f_i)$

guarantees that warnings respond both to the probability and the impact of hazards (Taggart et al., 13 Feb 2025).

• Proper Scoring Rules: Evaluations use scoring functions such as the risk-matrix score (RMaS) or warning score (WS), which reward proper calibration and penalize both over- and under-warning in proportion to their operational significance:

$$RMaS(F, y) = \sum_{i=1}^m \sum_{j=1}^n w_{i,j} \; ES_{i,j}(F,y)$$

where $ES_{i,j}$ is an elementary score for forecasts versus outcomes, and $w_{i,j}$ are nonnegative weights aligned with decision importance.

The operational implication is a feedback loop: frameworks can systematically track models’ warning patterns, reward improvements, and incentivize highly calibrated probabilistic risk assessments.

4. Metrics and Evaluation Protocols

Safety-warning based evaluation frameworks employ tailored metrics to quantify system performance and vulnerability exposure under dynamically evolving threats:

• Safety Rate/Pass Rate: Proportion of tests passed at each warning level, with progressive hardening showing safety declines (e.g., GPT-5’s EU AI Act compliance from 72.50% to 36.36% over three evaluation rounds) (Wang et al., 30 Sep 2025).
• Attack Success Rate (ASR): Fraction of tests that successfully elicit unsafe/undesirable behaviors, often under adversarial or stress conditions (Ying et al., 2024).
• Safety Risk Index (SRI) / Severity-weighted mean: Aggregates threat scores across test instances to reflect not just frequency, but average severity of warnings.
• Consistent/Proper Score Functions: Enables longitudinal tracking and direct comparison of improvements or regressions after model or policy updates (Taggart et al., 13 Feb 2025).

Empirical results demonstrate that static, single-snapshot safety tests frequently miss vulnerabilities only uncovered through iterative, warning-driven evaluation.

5. Examples of System Integration and Application Domains

Safety-warning frameworks are operational across diverse AI safety-critical application contexts:

• LLM and Policy Evaluation: Agentic, dynamically-evolving safety pipelines provide evolving warning profiles as LLMs and policies co-evolve. The frameworks easily adapt to new regulations by parsing and encoding them into new evaluable units (Wang et al., 30 Sep 2025).
• Multimodal and Multilingual Systems: Graded warnings in MLLM and VideoLLM settings provide nuanced risk assessments across text, image, and video modalities, critical for complex, culturally diverse safety regimes (Ying et al., 2024, Sun et al., 22 May 2025).
• Physical Systems and Robotics: Formal frameworks (e.g., SENTINEL) specify warnings grounded in temporal logic, enabling runtime shield synthesis and trajectory monitoring in robotics and embodied AI (Zhan et al., 14 Oct 2025).
• Risk Communication and Forecasting: In weather/climate/financial risk, risk-matrix-based warning assignment calibrated by consistent scoring enables CAP-compatible alerting and transparent comparison across forecasting systems (Taggart et al., 13 Feb 2025).

6. Strengths, Limitations, and Directions

Safety-warning-based frameworks provide:

• Progressive risk surfacing: Harder tests and refined warnings expose deeper model vulnerabilities.
• Alignment with operational priorities: Proper scoring rules ensure the system’s improvement targets real operational warning accuracy and calibration.
• Continuous, rather than one-shot, risk management: The agentic paradigm enables ongoing adaptation to new risks, regulations, and system behaviors.

However, these systems are subject to several challenges:

• Defining and calibrating warning thresholds and severity levels requires careful policy and stakeholder consultation.
• Scaling and hardening dynamic test suites can raise compute and interpretability demands.
• Complex edge cases—such as borderline “medium” warnings or subtle regulatory changes—may expose brittleness in warning calibration and require nuanced Analyst Agent interventions (Wang et al., 30 Sep 2025).

A plausible implication is that research focus is shifting toward fully automated, self-improving warning frameworks that tightly integrate dynamic policy ingestion, continual adversarial probe evolution, and decision-theoretically principled scoring into a closed, feedback-driven evaluation loop (Wang et al., 30 Sep 2025, Taggart et al., 13 Feb 2025).

7. Comparative Table: Representative Frameworks and Their Warning Strategies

Framework	Warning Mechanism	Evaluation Metric(s)
SafeEvalAgent	Agentic, graded warnings	Safety rate, Severity-tag, decline over rounds (Wang et al., 30 Sep 2025)
Risk Matrix Framework	Discrete risk-level mapping	RMaS, Warning Score (proper) (Taggart et al., 13 Feb 2025)
S-Eval	Category and risk score	Safety Score, RiskSeverity per prompt (Yuan et al., 2024)
SafeBench	Jury-based threat rating	Attack Success Rate, Safety Risk Index (Ying et al., 2024)
SENTINEL	Temporal logic (pass/fail with counterexamples)	Rule violation rate by scenario (Zhan et al., 14 Oct 2025)

These frameworks demonstrate the evolution of safety-warning-based evaluation from static, item-level checklists to dynamic, evidence-driven, and theoretically principled systems capable of issuing and evaluating multi-level warnings under real regulatory, operational, and adversarial constraints.