---
title: 'SceneJailEval: Scene-based Jailbreak Evaluation'
url: https://www.emergentmind.com/topics/scenejaileval
type: topic
---

# SceneJailEval: Scene-based Jailbreak Evaluation

SceneJailEval is a benchmark family and evaluation methodology for systematically measuring, comparing, and understanding jailbreak attacks that exploit scene context, sequential narrative structure, or environmental cues—particularly in text-to-video (T2V) models, vision-language models (VLMs), and embodied AI systems. It unifies emerging research on indirect, scene-based, and scenario-adaptive jailbreak vulnerabilities spanning multimodal and embodied language systems. SceneJailEval frameworks distinguish themselves by supporting fine-grained, scenario-specific, and multi-dimensional quantification of both success and harm, incorporating comprehensive datasets, mathematically standardized metrics, and modular extensibility for new tasks, attacks, and defenses.

## 1. Conceptual Foundations and Scope

SceneJailEval formalizes the evaluation of jailbreak attacks that operate through scene-level or multi-turn manipulations, indirect prompt injection, and environmental strategy. Unlike classic prompting-based jailbreaks in LLMs, SceneJailEval frameworks are motivated by:

- T2V models, where distributed scene narratives can bypass per-prompt safety filters by composing an unsafe story from benign sub-scenes [2509.22292, 2511.13127].
- Vision-language and embodied AI systems, where adversarial environmental cues (e.g., text embedded in the environment) or strategic visual-semantic mappings induce policy-violating actions without explicit textual queries [2511.16347, 2604.12616, 2605.19328].
- Scenario-adaptive evaluation, where a single binary label or uniform criteria fail to capture nuanced, context-sensitive, or regionally variable risks [2508.06194].

SceneJailEval provides both attack- and evaluation-side advances, enabling red-teaming and defense research on broad attack surfaces exposed by scene-aware AI deployment.

## 2. Formal Evaluation Frameworks

SceneJailEval encompasses several instantiations that are united by shared principles:

**Scenario-Adaptive Multi-Dimensional Evaluation**  
The general framework, formalized in [2508.06194], introduces:

- Scenario classification agent $Agent_{SC}$ assigning scenario $s$ to query-response pairs $(q, r)$.
- Scenario-dimension adapter $Adapter(s)$ selecting appropriate detection and harm dimensions $(D_d^s, D_h^s)$, their scenario-specific criteria $(C_d^s, C_h^s)$, and expert-weighting $W^s$.
- Jailbreak detection agent $Agent_{JD}(q, r, d, c_{s,d}) \in \{0,1\}$ per relevant $d$.
- Harmfulness evaluator $Agent_{HE}(q, r, d, c_{s,d}) \in [0,5]$ and overall harm $H = \sum_{d \in D_h^s} w_{s,d} h(d)$.
- Binary jailbreak is flagged when all relevant detection dimensions are violated.

This scenario-adaptive expansion overcomes one-size-fits-all limitations, enabling nuanced and precise evaluation across at least 14 scenarios, each with hand-crafted rulebases for dimension selection and harm weighting derived via an expert Delphi and Analytic Hierarchy Process.

**Intent-Contrast and Security-Utility Metrics**  
SceneJailEval adopts matched intent-contrast designs, pairing every scene or task with both a benign and an adversarial goal [2605.19328]. This enables joint assessment of not just adversarial success (ASR), but also utility retention:

\[
\mathrm{ASR}(\mathcal{D}, A) = \mathbb{E}_{(x, g^{\mathrm{adv}}, g^{\mathrm{ben}}) \sim \mathcal{D}} \left[ \mathbf{1} \big( \pi(x, g^{\mathrm{adv}}) \models g^{\mathrm{adv}} \big) \right]
\]
\[
\mathrm{UR}(\mathcal{D}) = \mathbb{E}_{(x, g^{\mathrm{adv}}, g^{\mathrm{ben}}) \sim \mathcal{D}} \left[ \mathbf{1} \big( \pi(x, g^{\mathrm{ben}}) \models g^{\mathrm{ben}} \big) \right]
\]
\[
\mathrm{SU\!-\!HM}(\mathcal{D}) = \frac{2\,\mathrm{SR}(\mathcal{D})\,\mathrm{UR}(\mathcal{D})}{\mathrm{SR}(\mathcal{D}) + \mathrm{UR}(\mathcal{D})}
\]

$\mathrm{SR}$ is the security rate, i.e., $1 - \mathrm{ASR}$. $SU\!-\!HM$ captures the harmonic mean of security and utility, penalizing defenses that overly restrict benign behavior.

## 3. Core Attack Models and Benchmarks

SceneJailEval evaluates and compares jailbreak attacks at the scene/narrative level across multiple model classes:

**A. Distributed Narrative Attacks (SceneSplit, VEIL)**  
- *SceneSplit* [2509.22292] attacks T2V models by decomposing a blocked harmful prompt $p$ into $k$ “benign” scene subprompts $(s_1, ..., s_k)$, each individually accepted by model filters. The overall generative output space is constrained by $\bigcap_{i=1}^k G(s_i)$, which, as scenes are composed, increasingly concentrates probability on unsafe outputs. Attack success is measured as the fraction of generated videos scoring above the unsafety threshold.
- *VEIL* [2511.13127] leverages cross-modal associative priors in T2V models by designing prompts of the form $P = P_{\text{anchor}} \oplus P_{\text{trigger}} \oplus P_{\text{modulator}}$. Neutral scene anchors, latent auditory triggers, and cinematic style cues jointly activate implicit unsafe content, optimizing for both stealth (passing input filters) and semantic alignment with originally blocked intent.

**B. Indirect Environmental Jailbreak and Embodied AI Attacks**  
- *Indirect environmental jailbreak* (IEJ) is formalized in SHAWSHANK [2511.16347], modeling scenarios where malicious instructions are injected into the physical world (e.g., written in a scene) and read by the agent via its perception pipeline. Attack success occurs if the system executes a harmful plan $u_{\mathrm{malicious}}$ in the absence of a risky direct user query.
- SHAWSHANK-FORGE and SHAWSHANK-BENCH provide an end-to-end pipeline for constructing and evaluating such attacks in embodied agent simulators, with comprehensive scene diversity and task coverage.

**C. Scene-Semantic Memory-Augmented Attacks**  
- *MemJack* [2604.12616] exploits deep scene semantics via multi-agent reinforcement: extracting vulnerabilities from scene objects, synthesizing adversarial multi-turn prompts, employing an iterative nullspace-projection filter to bypass latent refusal cones, and using experience-driven memory to transfer successful strategies across scene contexts. MemJack-Bench provides over 113,000 rounds of such attacks.

## 4. Datasets and Scenario Coverage

SceneJailEval comprises both purpose-built and widely-augmented datasets:

- A 14-scenario dataset spanning violent crime, professional advice, IP infringement, hate speech, political incitement, and regional regulatory differences, with ≈93 examples/scenario expertly annotated [2508.06194].
- An 18-category taxonomy covering embodied system security consequences—ranging from collision, entrapment, and public disruption to discrimination and pornography—mirroring ISO and regulatory standards [2605.19328].
- Large-scale, intent-contrast datasets generated by pairing adversarial and benign goals for curated (RJB-Instructions, 90 images) and augmented (RoboVQA, DROID, Autonomous Driving, etc.) scenes.
- Multimodal benchmarks (e.g., MemJack-Bench) exceeding 100,000 attack trajectories, enabling large-scale analysis of scene-driven vulnerabilities [2604.12616].

## 5. Metrics, Quantitative Results, and Scenario-Adaptive Gains

SceneJailEval evaluation combines:

- Attack Success Rate (ASR): proportion of attack attempts resulting in a safety-violating outcome.
- Harmful Risk Score (HRS): severity grading for embodied or physical risks [2511.16347].
- Utility Rate (UR), Security Rate (SR), and SU–HM for mapping the defense-utility tradeoff [2605.19328].
- Cross-scenario scenario-adaptive harm weighting, with weights $w_{s,d}$ tuned per context.

Selected results:

| Model/Task            | Baseline ASR | Scene-Based Attack ASR | Maximum ASR (multi-turn) |
|-----------------------|--------------|------------------------|--------------------------|
| Luma Ray2 (T2V)       | 39.5%        | 77.2% (SceneSplit)     |                          |
| Hailuo (T2V)          | 40.9%        | 84.1% (SceneSplit)     | 60% avg (VEIL)           |
| Veo2 (T2V)            | 33.1%        | 78.2% (SceneSplit)     |                          |
| Gemini-ER (embodied)  | n/a          | 75% (SHAWSHANK)        |                          |
| Qwen3-VL-Plus (multi) | n/a          | 71.48% (MemJack, R=20) | 90% (R=100)              |

Scenario-adaptive evaluation delivers improvement in F1 and cross-benchmark generalization (+6% F1 over prior SOTA) [2508.06194].

Ablations confirm that scenario-specific dimension selection and customized harm weighting substantially reduce both false negatives (e.g., hidden-help) and false positives (irrelevant dimension noise).

## 6. Defenses, Limitations, and Future Research

SceneJailEval enables systematic comparison of both attacks and defenses. Evaluated defenses include:

- Simple prompt-based reminders [2605.19328], which provide modest gains (SR up by 4 points over no defense).
- Rule-based plan verification (RoboGuard): achieves higher security at near-perfect utility, but can be inflexible to novel strategies [2605.19328].
- Emerging context-aware, multi-turn, and scene-synthesis safety checks, which are positioned as future research directions given current filter limitations [2509.22292, 2511.13127].

Current safety mechanisms tend to lack narrative-level or cross-scene reasoning and thus are bypassed by distributed or environment-level strategies. Advances in context state maintenance, scene-level consistency checking, adversarial training with scene-attack patterns, and hierarchical multi-stage filters are advocated to mitigate these new classes of vulnerability.

## 7. Benchmark Extensions, Reproducibility, and Community Resource

SceneJailEval frameworks and datasets—including code, annotations, evaluation scripts, and leaderboards—are publicly released with reproducible scoring APIs and automated CI workflows [2605.19328]. Researchers contribute new attacks or defenses by fork-and-PR, running automatic evals on established subsets, with resulting metrics and per-method summaries displayed live. This enables extensibility and supports robust, standardized advancements in multimodal and embodied jailbreak research. The SceneJailEval methodology—through scenario-adaptive, scene-level, and security-utility dual metrics—now sets the reference for benchmark-driven discovery of both vulnerabilities and effective mitigations in deployed AI agents.

Source: https://www.emergentmind.com/topics/scenejaileval