iSafetyBench and Omni-SafetyBench Overview

• iSafetyBench is a referential term denoting a safety evaluation style rather than a single benchmark, with Omni-SafetyBench as its closest concrete instantiation.
• It operationalizes safety for omni-modal language models by expanding 972 seed items into 23,328 samples across 24 modality configurations.
• Benchmark evaluations using metrics like C-ASR, C-RR, and CMSC-score reveal that no model excels in both overall safety and cross-modal consistency.

Searching arXiv for the benchmark name and closely related safety-benchmark papers. arXiv search query: iSafetyBench Within the available arXiv literature, iSafetyBench does not appear as a fully specified standalone benchmark with a single canonical paper. Instead, the name functions as an ambiguous reference point inside a broader safety-benchmark discourse. One directly relevant source states that, if the intended query is “iSafetyBench,” the relevant benchmark is Omni-SafetyBench, introduced as the first comprehensive safety benchmark for omni-modal / audio-visual LLMs (OLLMs) (Pan et al., 10 Aug 2025). Other papers use phrases such as “iSafetyBench-style safety evaluation” to denote a safety-centric, taxonomy-driven, and deployment-oriented benchmarking style rather than a single artifact (Meng et al., 18 Nov 2025). In that sense, iSafetyBench is most accurately situated as a referential node within the expanding ecosystem of AI safety benchmarks, with Omni-SafetyBench providing the clearest concrete instantiation in the supplied sources.

1. Referential status and benchmark identity

The most explicit resolution of the name occurs in the Omni-SafetyBench paper, which states that, if the query is “iSafetyBench,” the relevant benchmark in that paper is Omni-SafetyBench (Pan et al., 10 Aug 2025). This matters because the surrounding literature does not present a separate benchmark specification, dataset card, or evaluation protocol under the exact title iSafetyBench. Instead, the term surfaces indirectly, often as a shorthand for a class of safety evaluations concerned with fine-grained harm taxonomies, multimodal robustness, and operationally meaningful failure analysis.

This indirect usage is visible in later benchmarks that describe themselves as being “closer in spirit to iSafetyBench-style safety evaluation” or “highly relevant as an iSafetyBench-style benchmark”, particularly when they emphasize risk-aware reasoning rather than generic task success (Meng et al., 18 Nov 2025). A related ecosystem-level study further notes that iSafetyBench is not named there, while arguing that meaningful comparison across safety benchmarks requires benchmark-level metadata, metric-level semantics, aggregation rules, judge types, and maintenance metadata (Solanke, 14 Apr 2026). Taken together, these sources indicate that iSafetyBench is best understood not as a settled benchmark identity, but as part of a broader benchmark family whose members operationalize safety in different modalities, domains, and threat models.

2. Omni-SafetyBench as the closest concrete instantiation

Omni-SafetyBench is introduced as the first comprehensive safety benchmark for omni-modal / audio-visual LLMs (OLLMs), addressing the absence of dedicated safety evaluation for models that accept text, image, video, and audio together (Pan et al., 10 Aug 2025). Its core premise is that safety evaluation for OLLMs must assess not only whether a model blocks harmful content in a single format, but also whether that behavior remains stable when the same harmful intent is transformed across modalities.

The benchmark is built from 972 seed items selected from MM-SafetyBench, then expanded into a parallel benchmark with 24 modality variations across three paradigms: unimodal, dual-modal, and omni-modal. The unimodal setting includes Text-only, Image-only, Video-only, and Audio-only. The dual-modal setting includes Image-Text, Video-Text, and Audio-Text. The omni-modal setting includes Image-Audio-Text and Video-Audio-Text. Variants are constructed through diffusion-generated, typographic, and diffusion+typographic image or video forms, plus text-to-speech (TTS) and TTS+noise audio forms, yielding the explicitly enumerated 24 cases reported in the paper (Pan et al., 10 Aug 2025).

Each subcategory contains 972 samples, giving a total benchmark size of 23,328 samples. The paper additionally reports #Image: 3888, #Video: 3888, and #Audio: 2916. Harmful prompts span unsafe content inherited from MM-SafetyBench, including self-harm / weapon-making, illegal activity / gambling, fraud / cheating / manipulation, harassment / degradation / bias / objectification, and other harmful or unethical instruction-following scenarios. A distinctive feature is the inclusion of audio-visual joint harm cases, in which the harmful key phrase is redundantly embedded in both audio and visual channels (Pan et al., 10 Aug 2025).

This design makes Omni-SafetyBench parallel in a strict sense: the same underlying harmful intent is represented across multiple modality configurations. That structure is what enables explicit testing of cross-modal safety consistency, a property that earlier text-only, image-text, audio-only, or text-video benchmarks did not directly target.

3. Evaluation metrics and measurement semantics

A central methodological claim of Omni-SafetyBench is that raw attack success is misleading for OLLMs because a model may appear safe simply by failing to understand a complex multimodal input (Pan et al., 10 Aug 2025). To address this, the benchmark introduces conditional metrics that evaluate safety only on inputs the model actually understands.

The first two quantities are Conditional Attack Success Rate (C-ASR) and Conditional Refusal Rate (C-RR). C-ASR measures the chance that a model produces harmful content given that it understood the question. C-RR measures the chance that a model refuses to answer given that it understood the question. The source describes the procedure conceptually as labeling each response as understand, safe, and refuse, then computing safety only over understood examples. The reported formula snippets for C-ASR and C-RR are reproduced in the source with formatting corruption, so their conceptual role is clearer than the exact displayed expression (Pan et al., 10 Aug 2025).

The benchmark then defines a composite Safety-score:

\text{Safety-score} = \frac{(1-\text{C-ASR})(1+\lambda \cdot \text{C-RR})}{1+\lambda},

with $\lambda = 0.5$ in experiments. The intended interpretation is explicit: lower C-ASR improves the score, higher C-RR improves the score, and C-ASR is weighted more strongly because generating harmful content is the primary failure mode (Pan et al., 10 Aug 2025).

To quantify stability across the 24 parallel modality settings, the paper introduces the Cross-Modal Safety Consistency Score (CMSC-score). If the per-subcategory Safety-scores are $s_1, s_2, \ldots, s_N$, with mean $\mu$ and standard deviation $\sigma$, then

\text{CMSC-score} = e^{-\alpha \cdot \sigma}.

with $\alpha = 5$. Higher CMSC-score therefore indicates more consistent safety across modalities, while larger inter-modality variation sharply reduces the score (Pan et al., 10 Aug 2025).

These metrics are notable because they decouple three quantities that are often conflated in safety evaluation: comprehension, refusal, and harmful compliance. A plausible implication is that benchmarks in the iSafetyBench lineage are increasingly concerned not only with whether a model fails, but with how it fails and whether that failure is stable under representation change.

4. Empirical findings from Omni-SafetyBench

Omni-SafetyBench evaluates 6 open-source and 4 closed-source OLLMs, including Qwen2.5-Omni, MiniCPM-o, Baichuan-Omni, VITA, Unified-IO2, and Gemini 2.5 variants (Pan et al., 10 Aug 2025). The principal result is that no model excels simultaneously in both overall safety and cross-modal consistency. Only gemini-2.5-pro series and Qwen2.5-Omni-7b exceed 0.6 in both metrics, and even the best models top out around 0.8.

A second major finding is that safety degrades as inputs become more complex, following the pattern unimodal → dual-modal → omni-modal. Models that perform comparatively well on text, image, or audio alone become less safe when modalities are combined, and audio-visual joint inputs are especially effective at triggering unsafe behavior. This is directly tied to the benchmark’s motivating concern that OLLM safety does not automatically transfer across modality compositions (Pan et al., 10 Aug 2025).

Among open-source systems, Qwen2.5-Omni-7b performs best overall, while VITA-1.5 is described as strong on some unimodal cases, especially audio. However, most open-source models lose safety sharply in multimodal settings. The paper highlights particularly weak cases such as Minicpm-o-2.6, whose worst-case Safety-score is 0.14, and notes serious weaknesses for Baichuan-Omni-1.5 and others on specific modality combinations. Among closed-source systems, gemini-2.5-pro and gemini-2.5-pro-preview are the most robust overall, tending to hold Safety-scores around 0.8 and achieving CMSC-scores up to 0.83, but they still degrade under harder mixed-modal settings, especially with noise or typographic perturbations (Pan et al., 10 Aug 2025).

The benchmark also identifies each model’s weakest modality condition. For 6 out of 10 models, the weakest point is an omni-modal input, and only Gemini Pro variants remain above 0.7 in their worst cases. This finding reinforces the paper’s broader claim that single-modality safety evaluation can mask vulnerabilities that become visible only under cross-modal transformations.

5. Relation to adjacent safety benchmarks

The literature surrounding iSafetyBench is best understood through comparison with adjacent benchmarks that each specialize in a distinct safety surface. The following benchmarks are explicitly described in the supplied sources.

Benchmark	Scope	Distinguishing feature
SafetyBench	LLM safety understanding	11,435 multiple-choice questions, 7 categories, Chinese and English
Video-SafetyBench	Video LVLM safety	2,264 video-text pairs, 13 categories, 48 subcategories, harmful and benign query settings
Lingua-SafetyBench	Multilingual VLLM safety	100,440 harmful image-text pairs across 10 languages, partitioned into image-dominant and text-dominant subsets
LongSafetyBench	Long-context LLM safety	10 tasks, 1,203 instances, average length 41,889 words
MobileSafetyBench	Mobile device-control agents	80 tasks in Android emulators, including prompt injection robustness
DSBench	Autonomous driving VLM safety	external and in-cabin risks, 10 primary categories and 28 sub-categories

SafetyBench reformulates safety evaluation as multiple-choice question answering, emphasizing automatic, scalable, and multilingual assessment of safety understanding rather than open-ended generation (Zhang et al., 2023). Video-SafetyBench shifts the attack surface from image-text to video-text, introducing a controllable synthesis pipeline and a calibrated judge metric called RJScore, and showing that benign-query video composition can be more effective than explicit harmful prompts (Liu et al., 17 May 2025). Lingua-SafetyBench extends multimodal safety into the multilingual setting by separating image-dominant and text-dominant risk sources across 10 languages, exposing asymmetries between HRLs and Non-HRLs (Shi et al., 30 Jan 2026).

Other benchmarks broaden the operational setting rather than the modality inventory. LongSafetyBench evaluates whether long-context LLMs can detect, refuse, and safely handle harmful information embedded in very long inputs, using separate Harm Awareness (HA) and Safe Response (SR) metrics (Huang et al., 2024). MobileSafetyBench moves from QA to sequential action in Android emulators, where safety depends on environmental consequences, private data handling, and resistance to indirect prompt injection (Lee et al., 2024). DSBench adapts the safety-benchmarking logic to autonomous driving VLMs, unifying external environmental safety and in-cabin safety into a single taxonomy and explicitly presenting itself as closer in spirit to iSafetyBench-style safety evaluation (Meng et al., 18 Nov 2025).

This broader landscape shows that iSafetyBench, insofar as it names a recognizable benchmark style, is associated with three recurrent design commitments: explicit safety taxonomies, scenario realism beyond generic QA, and metrics that attempt to preserve semantic or operational nuance rather than relying only on surface accuracy.

6. Ecosystem significance, fragmentation, and frontier expansion

A large-scale ecosystem analysis catalogs 195 AI safety benchmarks released between 2018 and 2026 and argues that the field’s central structural problem is fragmentation rather than scarcity (Solanke, 14 Apr 2026). The catalogue reports 7 Popular, 68 High, 94 Medium, and 26 Low benchmarks; 165 English-only benchmarks; 170 evaluation-only resources; 137 stale GitHub repositories; and 96 stale Hugging Face datasets. At the metric level, it emphasizes that familiar labels such as accuracy, F1 score, safety score, and aggregate benchmark scores often conceal materially different judges, aggregation rules, and threat models. For a topic such as iSafetyBench, this is highly consequential: benchmark names can be stable while the actual semantics of evaluation differ sharply.

The frontier of benchmark design is also expanding beyond ordinary harmful-content refusal. ForesightSafety Bench is organized as a hierarchical framework with 20 core pillars and 94 fine-grained risk dimensions, spanning Fundamental Safety, Extended Safety, and Industrial Safety (Tong et al., 15 Feb 2026). Its coverage includes Risky Agentic Autonomy, AI4Science Safety, Embodied AI Safety, Social AI Safety, Environmental AI Safety, Catastrophic and Existential Risks, and 8 industrial safety domains. This suggests that the benchmark ecosystem associated with iSafetyBench-like evaluation is moving from surface semantic compliance toward broader assessments of autonomy, strategic behavior, dual-use scientific capability, and deployment-specific governance risk.

A plausible implication is that the term iSafetyBench is most useful not as a fixed benchmark identifier, but as a signpost for a methodological orientation: safety evaluation should be taxonomy-aware, threat-model-explicit, and sensitive to modality, context length, action consequences, language resource disparities, and cross-setting consistency. In the supplied literature, Omni-SafetyBench is the clearest concrete referent for that orientation, while the surrounding benchmark family shows how rapidly the underlying measurement problem is diversifying.