---
title: 'Omni-SafetyBench: Unified AI Safety Evaluation'
url: https://www.emergentmind.com/topics/omni-safetybench-framework
type: topic
---

# Omni-SafetyBench: Unified AI Safety Evaluation

Omni-SafetyBench is a unified, extensible framework for evaluating the safety of large-scale, multi-modal AI models—especially those exhibiting cross-modal reasoning, perception, and generation capabilities. The framework synthesizes rigorous risk taxonomies derived from regulations and policy, shared-target evaluation protocols, multi-dimensional metrics, adversarial testing, and modular pipelines. Omni-SafetyBench is designed to expose system-level safety vulnerabilities across any-to-any modality pairings and supports extension to new modalities, categories, and regulatory requirements. Its adoption provides a foundation for standardized, reproducible research at the intersection of AI safety, auditing, and societal alignment in the era of foundation and omni-modal models [2603.17476], [2508.07173], [2512.06589], [2407.17436].

## 1. Design Principles and Taxonomic Foundations

Omni-SafetyBench is guided by a set of principles that ensure regulatory relevance, extensibility, scenario realism, and scoring precision:

- **Regulation- and Policy-Grounded Taxonomy**: The risk taxonomy directly unifies explicit prohibitions and high-risk behaviors extracted from major government regulations (e.g., EU AI Act, U.S. Executive Order, China’s Generative AI Measures) and corporate acceptable-use policies (e.g., OpenAI, Google, Meta). This anchors evaluations in the emerging legal landscape and ensures comprehensive coverage of mandated or proscribed AI behaviors [2407.17436].
- **Hierarchical Granularity**: The base taxonomy is four-tiered: Level-1 (broad domains), Level-2 (high-level content/society/legal clusters), Level-3 (specific misuse types), Level-4 (314 granular categories, one-to-one mapped to regulatory statements). Level-4 enables precise, fine-grained auditing, while upper levels facilitate summary statistics and category-based tracking.
- **Cross-Modal and Contextual Diversity**: Dataset generation mechanisms instantiate each risk scenario across multiple I/O modalities (text, image, audio, video), inquiry types (consultative, imperative, declarative), dialectal/linguistic variants, and real-world context mutations [2512.06589], [2508.07173], [2407.17436].
- **Category-Specific, Human-Validated Scoring**: Each risk category is associated with a prompt-specific “judge” template, and output scoring follows calibrated multi-level rubrics, validated against human annotation to guarantee alignment (e.g., Cohen’s κ ≈ 0.86) [2407.17436].

## 2. Dataset Construction and Multimodal Scenario Generation

Omni-SafetyBench comprises benchmarks and toolboxes that instantiate thousands to tens of thousands of scenario instances per release, achieving extensive coverage by the following means:

- **Shared-Target Construction**: For each high-level risk scenario $s \in S$ (e.g., “knife attack,” “phishing email”), the framework projects $s$ into all supported task-modalities $t$ via a map:

  \[
  \varphi_t : S \longrightarrow (\mathcal{I}_t \times \mathcal{O}_t)
  \]

  where $\mathcal{I}_t$, $\mathcal{O}_t$ specify sets of inputs/outputs (text, image, audio, etc.) for task $t$ (e.g., text→image, image+text→image, audio+video→text) [2603.17476], [2508.07173].
- **Prompt Generation and Curation**: LLM-assisted prompt generation is coupled with manual curation and annotation, with further variation introduced via dialect mutation, authority-based reframing, and scenario-balancing heuristics [2407.17436]. Human auditing ensures that intent, category alignment, and realism are preserved, sometimes employing multi-stage or consensus protocols.
- **Scenario Instantiation and Modality Parallelism**: For consistency evaluation, parallel versions of each sample are generated—textual, visual (diffused, typographic, hybrid), audio (TTS, noisy-TTS), and video—all encoding the same target risk. This enables robust cross-modal consistency studies [2508.07173].
- **Inquiry Type Diversity**: Each instance is additionally labeled as consultative, imperative, or declarative, supporting analysis of how user intent structure impacts model safety [2512.06589].

| Tier             | Instance Count | Modalities         | Categories         |
|------------------|---------------|--------------------|--------------------|
| UniSAFE          | 6,802         | Text, Image        | 7 tasks, 7 domains |
| AIR-Bench 2024   | 5,694         | Text (+multi)      | 314                |
| Omni-SafetyBench | 23,328        | T, I, V, A (24 combos) | 972 base prompts   |
| OSB-MM           | 9,000+        | T+I                | 9 domains, 50 cats |

## 3. Evaluation Metrics and Protocols

Omni-SafetyBench employs multi-faceted metrics, with normalization and aggregation schemes designed for detailed safety analysis and cross-setting comparability:

- **Attack Success Rate (ASR) and Conditional ASR (C-ASR)**: Fraction of non-compliant outputs among prompts where the model understood the target risky intent:

  \[
  \text{C-ASR} = \frac{|\{i: \text{safe}_i = \mathrm{False} \,\wedge\, \text{understand}_i = \mathrm{True}\}|}{|\{i: \text{understand}_i = \mathrm{True}\}|}
  \]
  [2508.07173], [2603.17476]

- **Refusal Rate (RR) and Conditional RR (C-RR)**: Proportion of explicit refusals in the context of comprehension:

  \[
  \text{C-RR} = \frac{|\{i: \text{refuse}_i = \mathrm{True} \,\wedge\, \text{understand}_i = \mathrm{True}\}|}{|\{i: \text{understand}_i = \mathrm{True}\}|}
  \]
  [2508.07173], [2407.17436]

- **Safety-score**: Integrates both attack resilience and refusal proclivity:

  \[
  \text{Safety-score} = \frac{(1-\mathrm{C\text{-}ASR})\left[1 + \lambda\mathrm{C\text{-}RR}\right]}{1+\lambda}, \quad \lambda=0.5
  \]
  [2508.07173]

- **Cross-Modal Safety Consistency Score (CMSC-score)**: Measures behavioral invariance across $N$ modality settings:

  \[
  \mu = \frac{1}{N}\sum_{j=1}^N s_j,\quad
  \sigma = \sqrt{\frac{1}{N}\sum_{j=1}^N (s_j-\mu)^2}
  \]
  \[
  \text{CMSC-score} = \exp(-\alpha \sigma),\quad \alpha=5
  \]
  High CMSC ($\approx 1$) implies uniform safety response across modalities [2508.07173].

- **Three-Dimensional Evaluation (H–A–D protocol)** [2512.06589]:

  | Axis         | Scale         | Interpretation                          |
  |--------------|--------------|-----------------------------------------|
  | Harmfulness (H) | 1–10      | Severity: from insults to societal threats |
  | Alignment (A)   | 1–5       | Response fulfills harmful request       |
  | Detail (D)      | 1–5       | Explicitness: abstract → step-by-step   |

  Holistic adjudication (Eq. 5) maps H, A, D to discrete jailbreak event labeling.

- **Macro- and Subcategory Averaging**: ASR and ARR are first averaged within subcategories to avoid sampling biases, then macro-averaged for overall model assessment [2603.17476].

## 4. Attack and Defense Coverage

Omni-SafetyBench provides integrated infrastructure for adversarial red-teaming and mitigation benchmarking:

- **Attack Library**: Thirteen representative multi-modal jailbreak methods (e.g., gradient-based visual-adv, typographic prompt attack “FigStep,” black-box dispersion “CS-DJ,” cross-modal embedding “MML”) covering white-box, black-box, and out-of-distribution vectors [2512.06589].
- **Defense Suite**: Fifteen strategies encompassing:

  - Input Pre-Processing (e.g., visual purification, shield-prompting, cross-modal similarity detection)
  - Output Post-Processing (e.g., response classifiers or detoxifiers)
  - On-Model Interventions (hidden-layer gating, constitutional calibration, reinforcement fine-tuning) [2512.06589]

- **Evaluation Protocol**: Each (model, attack, defense, risk type) configuration is systematically benchmarked on (text, image) scenario pairs, with APIs for modular experimentation and performance visualization.

## 5. Empirical Results and Observed Vulnerabilities

Evaluation of current generation models (2025–2026) demonstrates multiple failure modes:

- **Elevated Multimodal Vulnerability**: Image-output tasks exhibit 4–6× higher ASR than text-only tasks for proprietary models (e.g., GPT-5, Gemini-2.5: image ASR up to 54%, text ASR as low as 6.2%) [2603.17476].
- **Modality and Task-Type Biases**: Highest ASRs occur in multi-image composition (IC) and multi-turn settings (MT), indicative of emergent compositional or escalation risks not captured in single-turn or text-only filter paradigms [2603.17476].
- **Auditory and Joint Modalities**: Audio-visual and image-audio-text attacks show the weakest safety-consistency, with Safety-scores dropping as low as 0.14 on specific typographic+diffusion image modalities [2508.07173].
- **Trade-offs**: Strong defense methods often decrease response explicitness, impacting utility, and may transfer vulnerability to new “holes” (e.g., “patch vs. new hole” effect reported with VLGuard vs. MML attacks) [2512.06589].
- **Policy Alignment Gaps**: Even top-performing models underperform in high-risk regulatory categories (e.g., refusal rates for advice in critical industries, privacy, and automated decision-making are often below 80% or even 20%) [2407.17436].

## 6. Extensibility and Generalization Strategies

Omni-SafetyBench is architected for systematic expansion:

- **Shared-Target Principle**: New modality-task combinations must be grounded in existing scenario sets for cross-task comparability [2603.17476].
- **Modular Pipelines**: Three-step data construction (trigger extraction, target expansion, scenario instantiation) is extensible to arbitrary modalities, including video, audio, structured data, and tabular risk [2407.17436].
- **Continuous Taxonomy Updates**: Monitoring and semi-automated integration of new regulatory or policy documents, with taxonomy augmentation as new risks emerge [2407.17436].
- **Versioned Releases and Community Input**: Formal documentation, code repositories, and open leaderboards support continuous benchmarking and crowd-sourced improvements.

## 7. Implications and Best Practices

The Omni-SafetyBench framework establishes rigorous standards and exposes significant research challenges:

- **Need for Modality-Invariant Alignment**: Specialized defenses trained on text and static image are inadequate for evolving cross-modal, audio-visual, or composition attacks [2508.07173], [2603.17476].
- **Self-Awareness and Refusal Calibration**: There is a quantifiable trade-off between automatic refusal accuracy (SAS), policy-aligned refusals (ARR), and overall utility, requiring explicit multi-objective calibration [2603.17476].
- **Dynamic, Adversarial Red-Teaming**: Automated scenario and attack generation, synthesized with manual curation, is essential for creating a moving target for both model and policy improvements [2512.06589].
- **Role in Audit and Regulation**: By aligning safety audit with legal and policy realities, and providing category-specific auto-graders, Omni-SafetyBench supports evidence-based oversight and technical support for compliance in highly regulated or safety-critical sectors [2407.17436].

Omni-SafetyBench serves as the de facto comprehensive template and open infrastructure for AI safety evaluation across modality, scenario, and regulatory boundaries, facilitating the next stage of both empirical research and policy-aligned deployment [2603.17476], [2512.06589], [2508.07173], [2407.17436].

Source: https://www.emergentmind.com/topics/omni-safetybench-framework