SafeRBench: ML Safety Benchmark Suite

• SafeRBench is a benchmark suite defining safety evaluation standards for machine learning systems in both large reasoning models and offline reinforcement learning.
• It employs novel methodologies such as prompt stratification, micro-thought segmentation, and composite scoring (RES, SAS) to assess and mitigate risk during the reasoning process.
• The benchmark’s effectiveness is validated through detailed human-LLM alignment, supporting reproducibility and enhanced risk management in safety-critical applications.

SafeRBench is a benchmark suite focused on safety evaluation in machine learning systems, with two distinct instantiations: (1) in the context of large language and reasoning models (LRMs), targeting end-to-end safety assessment throughout the entire reasoning process, and (2) in the domain of offline safe reinforcement learning (RL), providing reproducible tools for the development and evaluation of safety-critical RL algorithms. Both versions implement technically rigorous methodologies to dissect the nuanced risks and mitigation mechanisms in their respective areas, supporting multidimensional analysis, reproducibility, and domain-specific stress-testing (Gao et al., 19 Nov 2025, Liu et al., 2023).

1. Safety Challenges in Large Reasoning Models

SafeRBench for LRMs was motivated by the emergence of process-level risks associated with explicit chain-of-thought reasoning. Contrasted with traditional output-level safety assessments, the use of reasoning traces in LRMs introduces a new attack surface: harm can be injected covertly within intermediate reasoning steps (“micro-thoughts”), justified via misleading rationales, or only surface at the conclusion of a multi-step chain. Notable risk modes include rationale laundering, late-stage harmful revelations, and capability scaffolding. Existing safety benchmarks (e.g., SafetyBench, HarmBench) are limited by focusing solely on final outputs without tracing the evolution and stratification of risk throughout the reasoning process (Gao et al., 19 Nov 2025). SafeRBench directly addresses these deficiencies via input stratification, intermediate trace analysis, and composite, fine-grained safety metrics.

2. Input Characterization and Risk Taxonomy

SafeRBench’s input design systematically incorporates explicit risk categories and ordinal risk levels, pioneering prompt stratification along two axes: domain (six harm domains) and severity (low, medium, high), with annotation procedures calibrated for affected group sensitivity and escalation rules. The six domains are: Crimes & Illegal Activities, Cybersecurity & Attacks, Privacy & Data Abuse, Ethics & Legal Evasion, Social Safety & Well-being, and Environmental & Global Threats. Ordinal encoding is applied (0=safe; 1=low; 2=medium; 3=high), with escalation (e.g., group targeting → medium, advocacy of ideological harm → high). The benchmark includes 1,128 prompts evenly distributed by category and risk tier, with label validation conducted through GPT-3.5 and human experts (Gao et al., 19 Nov 2025). No further mathematical risk scoring beyond these discrete levels is used for input assessment.

3. Reasoning Trace Segmentation and Metric Construction

A micro-thought chunking paradigm is central to SafeRBench’s LRM methodology. Each reasoning trace is segmented by GPT-4o into the smallest coherent units and assigned one of six safety-relevant intent labels. This approach enables granular identification of risk emergence within the reasoning process and supports trace-level safety measurement. Earlier BERT-based segmentation approaches were insufficient for detecting intent shifts; GPT-4o demonstrated superior sensitivity. Risk assessment applies a position-weighted coherence metric (TrajectoryCoherence), with chunk-specific risk scores $s_t$, linear weights $w_t = t/T$, and aggregated risk trends: $R = \sum_{t=1}^T w_t s_t,$ normalized via sigmoid. Final coherence is: $$\mathrm{TrajectoryCoherence} = 1 - | r_\mathrm{trend} - r_\mathrm{ans} |,$$ where $r_\mathrm{ans}$ is the normalized risk level of the final answer. This formalism enables detection of risk “cliff-edge” accumulation near the end of reasoning chains, underscoring the requirement for tail-end safety controls (Gao et al., 19 Nov 2025).

4. Multidimensional Safety Metrics and Composite Scoring

SafeRBench for LRMs defines ten safety dimensions, segmented by analysis stage: reasoning trace (risk density, defense density, intention awareness, safe-strategy conversion), answer metrics (explicit refusal, risk level, execution level), and holistic response metrics (response complexity, trajectory coherence, risk reduction). Each dimension is scaled to $[0,1]$ prior to aggregation. Notable metrics include:

• Risk Density (RD): Fraction of tokens in direct harmful content
• Defense Density (DD): Fraction in defense-oriented tokens (norm violations, safe-strategy conversion)
• Intention Awareness (IA): Binary indicator for protective intent ordering
• Safe-Strategy Conversion (SSC): Cosine similarity in embedding space between queries and defense chunks
• Risk Reduction (RR): KL divergence between empirical and ideal risk score shifts

Composite scores include:

• Risk Exposure Score (RES): Mean over key risk metrics (lower=better)
• Safety Awareness Score (SAS): Mean over protective dimensions (higher=better)
• OverallSafety: $0.5 \times (1-\mathrm{RES}+\mathrm{SAS})$

These metrics are correlated with outcome safety properties, enabling empirical insights into mitigation efficacy, risk evolution, and the conditional relationships between reasoning density and final harms ($\rho \approx 0.92$ between reasoning risk and answer risk) (Gao et al., 19 Nov 2025).

5. Validation via Human–LLM Safety Alignment

SafeRBench incorporates systematic human validation, with 35 expert annotators assessing five subtasks spread over 100 items each: query categorization, risk level assignment, micro-thought segmentation, answer risk level, and practical execution level. Multi-class and ordinal subtasks are evaluated via multiple-choice and pairwise comparison questionnaires. Human-LLM agreement (percent match, no $\kappa$ reported) is robust across all axes: 84.57% for query category, 97.71% for query risk level, 89.43% for micro-thought labels, 98.86% for answer risk level, and 96.57% for execution level. This procedure grounds automated safety judgments and calibrates the reliability of metric extraction (Gao et al., 19 Nov 2025).

6. Experimental Findings and Model Analysis

SafeRBench’s LRM evaluation covers 19 models, spanning DeepSeek-R1 (1.5B–671B), Qwen3 (0.6B–235B), EXAONE, Kimi-thinking-p, and Hunyuan-T1. Single-sample reasoning per prompt is deployed for cost containment, interfaced via HuggingFace and official APIs. Safety Awareness (SAS) generally increases with scale up to a point (“always-help” failure mode in ultra-large mixture-of-experts architectures). RES and SAS are strongly anti-correlated ($\rho \approx -0.90$). Defensive design elements (intention awareness, response complexity) show protective correlations, with increased IA linked to reduced risk density ($\rho \approx -0.91$) and mitigated answer harms. High-risk inputs yield polarized refusal or dangerous but low-feasibility answers; domain-specific effects reflect varying robustness (cybersecurity and privacy show stronger defenses; social safety and ethics remain challenging). Notably, explicit reasoning (“Thinking mode”) improves safety in mid-scale models but introduces risks in extreme-scale or low-capacity networks (Gao et al., 19 Nov 2025).

7. Conclusions, Limitations, and Prospects

SafeRBench provides the first reproducible, holistic safety benchmark for LRMs, covering the full chain from input design, through process-trace segmentation, to answer-level evaluation. Its core contributions are the risk-level stratified query suite, granular micro-thought tagging, and composite metric design (RES/SAS/OverallSafety). Human-grounding enhances interpretability and reliability. Limitations include single-sample analysis, reliance on GPT-4o in quality control, and lack of detailed temporal modeling. Proposed future work includes multi-sample aggregation (variance estimation), extension to multimodal and continuous risk scoring, improved tail-end safety strategies, and automated training alignment optimizing for composite scores (Gao et al., 19 Nov 2025).

SafeRBench represents the current state of the art in safety benchmarking for both large reasoning models and offline safe RL frameworks, enabling nuanced analysis of risk emergence and mitigation throughout the decision and reasoning pipelines.