Omni-SafetyBench: Unified AI Safety Evaluation

• Omni-SafetyBench is a unified, extensible framework that evaluates multi-modal AI models using regulatory- and policy-grounded risk taxonomies.
• It employs modular pipelines, adversarial red-teaming, and multi-dimensional metrics to assess safety across text, image, audio, and video modalities.
• The framework standardizes scenario instantiation and scoring to support reproducible research and compliance in AI safety and auditing.

Omni-SafetyBench is a unified, extensible framework for evaluating the safety of large-scale, multi-modal AI models—especially those exhibiting cross-modal reasoning, perception, and generation capabilities. The framework synthesizes rigorous risk taxonomies derived from regulations and policy, shared-target evaluation protocols, multi-dimensional metrics, adversarial testing, and modular pipelines. Omni-SafetyBench is designed to expose system-level safety vulnerabilities across any-to-any modality pairings and supports extension to new modalities, categories, and regulatory requirements. Its adoption provides a foundation for standardized, reproducible research at the intersection of AI safety, auditing, and societal alignment in the era of foundation and omni-modal models (Lee et al., 18 Mar 2026, Pan et al., 10 Aug 2025, Jia et al., 6 Dec 2025, Zeng et al., 2024).

1. Design Principles and Taxonomic Foundations

Omni-SafetyBench is guided by a set of principles that ensure regulatory relevance, extensibility, scenario realism, and scoring precision:

• Regulation- and Policy-Grounded Taxonomy: The risk taxonomy directly unifies explicit prohibitions and high-risk behaviors extracted from major government regulations (e.g., EU AI Act, U.S. Executive Order, China’s Generative AI Measures) and corporate acceptable-use policies (e.g., OpenAI, Google, Meta). This anchors evaluations in the emerging legal landscape and ensures comprehensive coverage of mandated or proscribed AI behaviors (Zeng et al., 2024).
• Hierarchical Granularity: The base taxonomy is four-tiered: Level-1 (broad domains), Level-2 (high-level content/society/legal clusters), Level-3 (specific misuse types), Level-4 (314 granular categories, one-to-one mapped to regulatory statements). Level-4 enables precise, fine-grained auditing, while upper levels facilitate summary statistics and category-based tracking.
• Cross-Modal and Contextual Diversity: Dataset generation mechanisms instantiate each risk scenario across multiple I/O modalities (text, image, audio, video), inquiry types (consultative, imperative, declarative), dialectal/linguistic variants, and real-world context mutations (Jia et al., 6 Dec 2025, Pan et al., 10 Aug 2025, Zeng et al., 2024).
• Category-Specific, Human-Validated Scoring: Each risk category is associated with a prompt-specific “judge” template, and output scoring follows calibrated multi-level rubrics, validated against human annotation to guarantee alignment (e.g., Cohen’s κ ≈ 0.86) (Zeng et al., 2024).

2. Dataset Construction and Multimodal Scenario Generation

Omni-SafetyBench comprises benchmarks and toolboxes that instantiate thousands to tens of thousands of scenario instances per release, achieving extensive coverage by the following means:

• Shared-Target Construction: For each high-level risk scenario $s \in S$ (e.g., “knife attack,” “phishing email”), the framework projects $s$ into all supported task-modalities $t$ via a map:

$$\varphi_t : S \longrightarrow (\mathcal{I}_t \times \mathcal{O}_t)$$

where $\mathcal{I}_t$, $\mathcal{O}_t$ specify sets of inputs/outputs (text, image, audio, etc.) for task $t$ (e.g., text→image, image+text→image, audio+video→text) (Lee et al., 18 Mar 2026, Pan et al., 10 Aug 2025).

• Prompt Generation and Curation: LLM-assisted prompt generation is coupled with manual curation and annotation, with further variation introduced via dialect mutation, authority-based reframing, and scenario-balancing heuristics (Zeng et al., 2024). Human auditing ensures that intent, category alignment, and realism are preserved, sometimes employing multi-stage or consensus protocols.
• Scenario Instantiation and Modality Parallelism: For consistency evaluation, parallel versions of each sample are generated—textual, visual (diffused, typographic, hybrid), audio (TTS, noisy-TTS), and video—all encoding the same target risk. This enables robust cross-modal consistency studies (Pan et al., 10 Aug 2025).
• Inquiry Type Diversity: Each instance is additionally labeled as consultative, imperative, or declarative, supporting analysis of how user intent structure impacts model safety (Jia et al., 6 Dec 2025).

Tier	Instance Count	Modalities	Categories
UniSAFE	6,802	Text, Image	7 tasks, 7 domains
AIR-Bench 2024	5,694	Text (+multi)	314
Omni-SafetyBench	23,328	T, I, V, A (24 combos)	972 base prompts
OSB-MM	9,000+	T+I	9 domains, 50 cats

3. Evaluation Metrics and Protocols

Omni-SafetyBench employs multi-faceted metrics, with normalization and aggregation schemes designed for detailed safety analysis and cross-setting comparability:

• Attack Success Rate (ASR) and Conditional ASR (C-ASR): Fraction of non-compliant outputs among prompts where the model understood the target risky intent:

$$\text{C-ASR} = \frac{|\{i: \text{safe}_i = \mathrm{False} \,\wedge\, \text{understand}_i = \mathrm{True}\}|}{|\{i: \text{understand}_i = \mathrm{True}\}|}$$

(Pan et al., 10 Aug 2025, Lee et al., 18 Mar 2026)

• Refusal Rate (RR) and Conditional RR (C-RR): Proportion of explicit refusals in the context of comprehension:

$$\text{C-RR} = \frac{|\{i: \text{refuse}_i = \mathrm{True} \,\wedge\, \text{understand}_i = \mathrm{True}\}|}{|\{i: \text{understand}_i = \mathrm{True}\}|}$$

(Pan et al., 10 Aug 2025, Zeng et al., 2024)

• Safety-score: Integrates both attack resilience and refusal proclivity:

$$\text{Safety-score} = \frac{(1-\mathrm{C\text{-}ASR})\left[1 + \lambda\mathrm{C\text{-}RR}\right]}{1+\lambda}, \quad \lambda=0.5$$

(Pan et al., 10 Aug 2025)

• Cross-Modal Safety Consistency Score (CMSC-score): Measures behavioral invariance across $s$0 modality settings:

$s$1

$s$2

High CMSC ($s$3) implies uniform safety response across modalities (Pan et al., 10 Aug 2025).

• Three-Dimensional Evaluation (H–A–D protocol) (Jia et al., 6 Dec 2025):

| Axis | Scale | Interpretation | |--------------|--------------|-----------------------------------------| | Harmfulness (H) | 1–10 | Severity: from insults to societal threats | | Alignment (A) | 1–5 | Response fulfills harmful request | | Detail (D) | 1–5 | Explicitness: abstract → step-by-step |

Holistic adjudication (Eq. 5) maps H, A, D to discrete jailbreak event labeling.

• Macro- and Subcategory Averaging: ASR and ARR are first averaged within subcategories to avoid sampling biases, then macro-averaged for overall model assessment (Lee et al., 18 Mar 2026).

4. Attack and Defense Coverage

Omni-SafetyBench provides integrated infrastructure for adversarial red-teaming and mitigation benchmarking:

• Attack Library: Thirteen representative multi-modal jailbreak methods (e.g., gradient-based visual-adv, typographic prompt attack “FigStep,” black-box dispersion “CS-DJ,” cross-modal embedding “MML”) covering white-box, black-box, and out-of-distribution vectors (Jia et al., 6 Dec 2025).
• Defense Suite: Fifteen strategies encompassing:
  • Input Pre-Processing (e.g., visual purification, shield-prompting, cross-modal similarity detection)
  • Output Post-Processing (e.g., response classifiers or detoxifiers)
  • On-Model Interventions (hidden-layer gating, constitutional calibration, reinforcement fine-tuning) (Jia et al., 6 Dec 2025)
• Evaluation Protocol: Each (model, attack, defense, risk type) configuration is systematically benchmarked on (text, image) scenario pairs, with APIs for modular experimentation and performance visualization.

5. Empirical Results and Observed Vulnerabilities

Evaluation of current generation models (2025–2026) demonstrates multiple failure modes:

• Elevated Multimodal Vulnerability: Image-output tasks exhibit 4–6× higher ASR than text-only tasks for proprietary models (e.g., GPT-5, Gemini-2.5: image ASR up to 54%, text ASR as low as 6.2%) (Lee et al., 18 Mar 2026).
• Modality and Task-Type Biases: Highest ASRs occur in multi-image composition (IC) and multi-turn settings (MT), indicative of emergent compositional or escalation risks not captured in single-turn or text-only filter paradigms (Lee et al., 18 Mar 2026).
• Auditory and Joint Modalities: Audio-visual and image-audio-text attacks show the weakest safety-consistency, with Safety-scores dropping as low as 0.14 on specific typographic+diffusion image modalities (Pan et al., 10 Aug 2025).
• Trade-offs: Strong defense methods often decrease response explicitness, impacting utility, and may transfer vulnerability to new “holes” (e.g., “patch vs. new hole” effect reported with VLGuard vs. MML attacks) (Jia et al., 6 Dec 2025).
• Policy Alignment Gaps: Even top-performing models underperform in high-risk regulatory categories (e.g., refusal rates for advice in critical industries, privacy, and automated decision-making are often below 80% or even 20%) (Zeng et al., 2024).

6. Extensibility and Generalization Strategies

Omni-SafetyBench is architected for systematic expansion:

• Shared-Target Principle: New modality-task combinations must be grounded in existing scenario sets for cross-task comparability (Lee et al., 18 Mar 2026).
• Modular Pipelines: Three-step data construction (trigger extraction, target expansion, scenario instantiation) is extensible to arbitrary modalities, including video, audio, structured data, and tabular risk (Zeng et al., 2024).
• Continuous Taxonomy Updates: Monitoring and semi-automated integration of new regulatory or policy documents, with taxonomy augmentation as new risks emerge (Zeng et al., 2024).
• Versioned Releases and Community Input: Formal documentation, code repositories, and open leaderboards support continuous benchmarking and crowd-sourced improvements.

7. Implications and Best Practices

The Omni-SafetyBench framework establishes rigorous standards and exposes significant research challenges:

• Need for Modality-Invariant Alignment: Specialized defenses trained on text and static image are inadequate for evolving cross-modal, audio-visual, or composition attacks (Pan et al., 10 Aug 2025, Lee et al., 18 Mar 2026).
• Self-Awareness and Refusal Calibration: There is a quantifiable trade-off between automatic refusal accuracy (SAS), policy-aligned refusals (ARR), and overall utility, requiring explicit multi-objective calibration (Lee et al., 18 Mar 2026).
• Dynamic, Adversarial Red-Teaming: Automated scenario and attack generation, synthesized with manual curation, is essential for creating a moving target for both model and policy improvements (Jia et al., 6 Dec 2025).
• Role in Audit and Regulation: By aligning safety audit with legal and policy realities, and providing category-specific auto-graders, Omni-SafetyBench supports evidence-based oversight and technical support for compliance in highly regulated or safety-critical sectors (Zeng et al., 2024).

Omni-SafetyBench serves as the de facto comprehensive template and open infrastructure for AI safety evaluation across modality, scenario, and regulatory boundaries, facilitating the next stage of both empirical research and policy-aligned deployment (Lee et al., 18 Mar 2026, Jia et al., 6 Dec 2025, Pan et al., 10 Aug 2025, Zeng et al., 2024).