SceneJailEval: Scene-based Jailbreak Evaluation

• SceneJailEval is a benchmark framework that systematically evaluates jailbreak attacks exploiting scene context, narrative structure, and environmental cues.
• It employs scenario-adaptive, multi-dimensional metrics—including ASR, UR, and SU–HM—to quantify both adversarial success and potential harm.
• The framework supports extensive, annotated datasets and modular extensibility, paving the way for advanced research in multimodal and embodied AI security.

SceneJailEval is a benchmark family and evaluation methodology for systematically measuring, comparing, and understanding jailbreak attacks that exploit scene context, sequential narrative structure, or environmental cues—particularly in text-to-video (T2V) models, vision-LLMs (VLMs), and embodied AI systems. It unifies emerging research on indirect, scene-based, and scenario-adaptive jailbreak vulnerabilities spanning multimodal and embodied language systems. SceneJailEval frameworks distinguish themselves by supporting fine-grained, scenario-specific, and multi-dimensional quantification of both success and harm, incorporating comprehensive datasets, mathematically standardized metrics, and modular extensibility for new tasks, attacks, and defenses.

1. Conceptual Foundations and Scope

SceneJailEval formalizes the evaluation of jailbreak attacks that operate through scene-level or multi-turn manipulations, indirect prompt injection, and environmental strategy. Unlike classic prompting-based jailbreaks in LLMs, SceneJailEval frameworks are motivated by:

• T2V models, where distributed scene narratives can bypass per-prompt safety filters by composing an unsafe story from benign sub-scenes (Lee et al., 26 Sep 2025, Ying et al., 17 Nov 2025).
• Vision-language and embodied AI systems, where adversarial environmental cues (e.g., text embedded in the environment) or strategic visual-semantic mappings induce policy-violating actions without explicit textual queries (Li et al., 20 Nov 2025, Chen et al., 14 Apr 2026, Yeke et al., 19 May 2026).
• Scenario-adaptive evaluation, where a single binary label or uniform criteria fail to capture nuanced, context-sensitive, or regionally variable risks (Jiang et al., 8 Aug 2025).

SceneJailEval provides both attack- and evaluation-side advances, enabling red-teaming and defense research on broad attack surfaces exposed by scene-aware AI deployment.

2. Formal Evaluation Frameworks

SceneJailEval encompasses several instantiations that are united by shared principles:

Scenario-Adaptive Multi-Dimensional Evaluation

The general framework, formalized in (Jiang et al., 8 Aug 2025), introduces:

• Scenario classification agent $Agent_{SC}$ assigning scenario $s$ to query-response pairs $(q, r)$.
• Scenario-dimension adapter $Adapter(s)$ selecting appropriate detection and harm dimensions $(D_d^s, D_h^s)$, their scenario-specific criteria $(C_d^s, C_h^s)$, and expert-weighting $W^s$.
• Jailbreak detection agent $Agent_{JD}(q, r, d, c_{s,d}) \in \{0,1\}$ per relevant $d$.
• Harmfulness evaluator $Agent_{HE}(q, r, d, c_{s,d}) \in [0,5]$ and overall harm $s$0.
• Binary jailbreak is flagged when all relevant detection dimensions are violated.

This scenario-adaptive expansion overcomes one-size-fits-all limitations, enabling nuanced and precise evaluation across at least 14 scenarios, each with hand-crafted rulebases for dimension selection and harm weighting derived via an expert Delphi and Analytic Hierarchy Process.

Intent-Contrast and Security-Utility Metrics

SceneJailEval adopts matched intent-contrast designs, pairing every scene or task with both a benign and an adversarial goal (Yeke et al., 19 May 2026). This enables joint assessment of not just adversarial success (ASR), but also utility retention:

$s$1

$s$2

$s$3

$s$4 is the security rate, i.e., $s$5. $s$6 captures the harmonic mean of security and utility, penalizing defenses that overly restrict benign behavior.

3. Core Attack Models and Benchmarks

SceneJailEval evaluates and compares jailbreak attacks at the scene/narrative level across multiple model classes:

A. Distributed Narrative Attacks (SceneSplit, VEIL)

• SceneSplit (Lee et al., 26 Sep 2025) attacks T2V models by decomposing a blocked harmful prompt $s$7 into $s$8 “benign” scene subprompts $s$9, each individually accepted by model filters. The overall generative output space is constrained by $(q, r)$0, which, as scenes are composed, increasingly concentrates probability on unsafe outputs. Attack success is measured as the fraction of generated videos scoring above the unsafety threshold.
• VEIL (Ying et al., 17 Nov 2025) leverages cross-modal associative priors in T2V models by designing prompts of the form $(q, r)$1. Neutral scene anchors, latent auditory triggers, and cinematic style cues jointly activate implicit unsafe content, optimizing for both stealth (passing input filters) and semantic alignment with originally blocked intent.

B. Indirect Environmental Jailbreak and Embodied AI Attacks

• Indirect environmental jailbreak (IEJ) is formalized in SHAWSHANK (Li et al., 20 Nov 2025), modeling scenarios where malicious instructions are injected into the physical world (e.g., written in a scene) and read by the agent via its perception pipeline. Attack success occurs if the system executes a harmful plan $(q, r)$2 in the absence of a risky direct user query.
• SHAWSHANK-FORGE and SHAWSHANK-BENCH provide an end-to-end pipeline for constructing and evaluating such attacks in embodied agent simulators, with comprehensive scene diversity and task coverage.

C. Scene-Semantic Memory-Augmented Attacks

• MemJack (Chen et al., 14 Apr 2026) exploits deep scene semantics via multi-agent reinforcement: extracting vulnerabilities from scene objects, synthesizing adversarial multi-turn prompts, employing an iterative nullspace-projection filter to bypass latent refusal cones, and using experience-driven memory to transfer successful strategies across scene contexts. MemJack-Bench provides over 113,000 rounds of such attacks.

4. Datasets and Scenario Coverage

SceneJailEval comprises both purpose-built and widely-augmented datasets:

• A 14-scenario dataset spanning violent crime, professional advice, IP infringement, hate speech, political incitement, and regional regulatory differences, with ≈93 examples/scenario expertly annotated (Jiang et al., 8 Aug 2025).
• An 18-category taxonomy covering embodied system security consequences—ranging from collision, entrapment, and public disruption to discrimination and pornography—mirroring ISO and regulatory standards (Yeke et al., 19 May 2026).
• Large-scale, intent-contrast datasets generated by pairing adversarial and benign goals for curated (RJB-Instructions, 90 images) and augmented (RoboVQA, DROID, Autonomous Driving, etc.) scenes.
• Multimodal benchmarks (e.g., MemJack-Bench) exceeding 100,000 attack trajectories, enabling large-scale analysis of scene-driven vulnerabilities (Chen et al., 14 Apr 2026).

5. Metrics, Quantitative Results, and Scenario-Adaptive Gains

SceneJailEval evaluation combines:

• Attack Success Rate (ASR): proportion of attack attempts resulting in a safety-violating outcome.
• Harmful Risk Score (HRS): severity grading for embodied or physical risks (Li et al., 20 Nov 2025).
• Utility Rate (UR), Security Rate (SR), and SU–HM for mapping the defense-utility tradeoff (Yeke et al., 19 May 2026).
• Cross-scenario scenario-adaptive harm weighting, with weights $(q, r)$3 tuned per context.

Selected results:

Model/Task	Baseline ASR	Scene-Based Attack ASR	Maximum ASR (multi-turn)
Luma Ray2 (T2V)	39.5%	77.2% (SceneSplit)
Hailuo (T2V)	40.9%	84.1% (SceneSplit)	60% avg (VEIL)
Veo2 (T2V)	33.1%	78.2% (SceneSplit)
Gemini-ER (embodied)	n/a	75% (SHAWSHANK)
Qwen3-VL-Plus (multi)	n/a	71.48% (MemJack, R=20)	90% (R=100)

Scenario-adaptive evaluation delivers improvement in F1 and cross-benchmark generalization (+6% F1 over prior SOTA) (Jiang et al., 8 Aug 2025).

Ablations confirm that scenario-specific dimension selection and customized harm weighting substantially reduce both false negatives (e.g., hidden-help) and false positives (irrelevant dimension noise).

6. Defenses, Limitations, and Future Research

SceneJailEval enables systematic comparison of both attacks and defenses. Evaluated defenses include:

• Simple prompt-based reminders (Yeke et al., 19 May 2026), which provide modest gains (SR up by 4 points over no defense).
• Rule-based plan verification (RoboGuard): achieves higher security at near-perfect utility, but can be inflexible to novel strategies (Yeke et al., 19 May 2026).
• Emerging context-aware, multi-turn, and scene-synthesis safety checks, which are positioned as future research directions given current filter limitations (Lee et al., 26 Sep 2025, Ying et al., 17 Nov 2025).

Current safety mechanisms tend to lack narrative-level or cross-scene reasoning and thus are bypassed by distributed or environment-level strategies. Advances in context state maintenance, scene-level consistency checking, adversarial training with scene-attack patterns, and hierarchical multi-stage filters are advocated to mitigate these new classes of vulnerability.

7. Benchmark Extensions, Reproducibility, and Community Resource

SceneJailEval frameworks and datasets—including code, annotations, evaluation scripts, and leaderboards—are publicly released with reproducible scoring APIs and automated CI workflows (Yeke et al., 19 May 2026). Researchers contribute new attacks or defenses by fork-and-PR, running automatic evals on established subsets, with resulting metrics and per-method summaries displayed live. This enables extensibility and supports robust, standardized advancements in multimodal and embodied jailbreak research. The SceneJailEval methodology—through scenario-adaptive, scene-level, and security-utility dual metrics—now sets the reference for benchmark-driven discovery of both vulnerabilities and effective mitigations in deployed AI agents.