OrchJail: Orchestration-Guided Image Fuzzing

• OrchJail is an orchestration-guided fuzzing framework that targets multi-step tool chains in text-to-image systems to bypass traditional safety measures.
• It employs causal reasoning to learn high-risk tool-orchestration patterns, guiding prompt mutations and improving jailbreak success over prompt-only methods.
• Empirical evaluations demonstrate enhanced success rates, lower query counts, and better performance compared to alternative approaches in exploiting distributed safety vulnerabilities.

OrchJail is an orchestration-guided fuzzing framework for jailbreaking tool-calling text-to-image agents, introduced to exploit a safety failure mode that does not reduce to prompt text alone. In these systems, an LLM planner decomposes a request into a multi-step tool chain over image-generation and image-editing components. OrchJail treats the resulting tool orchestration as the critical attack surface: harmful outputs may arise when individually benign tool calls compose into an unsafe final image, so prompt-only jailbreak methods are insufficient. Its central mechanism is to learn high-risk tool-orchestration patterns from successful jailbreak traces, infer how prompt spans causally relate to those patterns, and use that knowledge to guide fuzzing toward prompts that are more likely to reproduce unsafe multi-step behavior (Chen et al., 8 May 2026).

1. Tool orchestration as the attack surface

OrchJail studies tool-calling text-to-image agents in which an LLM planner or controller reads a user prompt, decomposes it into sub-tasks, decides a multi-step plan, and chooses which tools to call and with what textual arguments. The tool set $\mathcal{T}$ may include a base image generator, object insertion, attribute or style editing, and composition or layout tools such as LMD and BoxDiff. Safety is typically distributed across a safety-aligned planner, tool-level filters, and possibly image-level safety checks. OrchJail identifies a structural weakness in this arrangement: a harmful final image can emerge from a sequence of tool calls where every single step looks safe in isolation (Chen et al., 8 May 2026).

This formulation differs from conventional text-to-image jailbreaking, which generally assumes a single prompt-to-image mapping and focuses on token perturbation, multilingual obfuscation, paraphrase search, or other surface-level text transformations. OrchJail argues that such prompt-only methods do not model how specific phrases induce particular tool chains, do not exploit the compositional loophole created by distributed safeguards, and can therefore miss attacks in which the planner and each tool appear locally compliant while the full orchestration is globally unsafe.

The paper formalizes the agent as returning

$$\big(\rho(p), I(p), \tau(p)\big) \leftarrow \mathcal{A}(p),$$

where $\rho(p)\in\{0,1\}$ is a refusal flag, $I(p)$ is the final image, and $\tau(p)=[(t_1,x_1),\dots,(t_L,x_L)]$ is the tool-calling trace. The attacker is black-box at the model level but trace-observable at the tool level: the attacker can send arbitrary prompts and observe the final image and textual tool log, but does not access internal planner states, model weights, gradients, or safety rules.

2. Threat model and formal representation

The attack begins from a harmful target prompt $p_t$ requesting prohibited content such as violence, bloody or gory scenes, illegal activities, discrimination or hate content, and related prohibited categories. A jailbreak is successful when the agent does not refuse and the resulting image is semantically aligned with the harmful intent. Image-text alignment is measured with CLIP ViT-B/32 as

$$\mathrm{ITA}(p, I(p)) \triangleq \cos\!\left(\phi_{\text{text}}(p), \phi_{\text{img}}(I(p))\right),$$

and the success indicator is

$$Y(p) \triangleq \mathbb{I}[\rho(p)=0]\cdot \mathbb{I}[\mathrm{ITA}(p,I(p))\ge \theta_{\text{i-t}}],$$

with $\theta_{\text{i-t}}$ set empirically (Chen et al., 8 May 2026).

A high-risk orchestration pattern is defined along three dimensions. The first is macro-planning, meaning the high-level sequence of sub-tasks. The second is micro-scheduling, meaning the fine-grained sequencing of operations. The third is tool selection, meaning which tools are preferred in which contexts. For a successful jailbreak prompt $p_s$, OrchJail abstracts the orchestration as

$$\big(\rho(p), I(p), \tau(p)\big) \leftarrow \mathcal{A}(p),$$0

Patterns are high-risk when they recur across successful jailbreak traces, tend to yield unsafe images despite local safeguards, and consist of steps that appear benign individually but harmful in composition.

The framework then performs prompt-orchestration causal reasoning. For each dimension $$\big(\rho(p), I(p), \tau(p)\big) \leftarrow \mathcal{A}(p),$$1, it stores

$$\big(\rho(p), I(p), \tau(p)\big) \leftarrow \mathcal{A}(p),$$2

where $$\big(\rho(p), I(p), \tau(p)\big) \leftarrow \mathcal{A}(p),$$3 is a specific orchestration element, $$\big(\rho(p), I(p), \tau(p)\big) \leftarrow \mathcal{A}(p),$$4 is a contiguous prompt span, and $$\big(\rho(p), I(p), \tau(p)\big) \leftarrow \mathcal{A}(p),$$5 is a natural-language explanation of why that span triggers that orchestration element. The full causal knowledge is

$$\big(\rho(p), I(p), \tau(p)\big) \leftarrow \mathcal{A}(p),$$6

This representation is central to OrchJail’s identity. The framework does not merely search for prompts that avoid refusal; it searches for prompts that are likely to recreate previously successful high-risk tool chains.

3. Orchestration-guided fuzzing pipeline

The pipeline has two intertwined subsystems: a standard fuzzing loop and an orchestration module. The standard loop generates seeds, queries the agent, evaluates jailbreak success, mutates prompts, scores candidates, and selects the next query. The orchestration module extracts pattern abstractions from successful traces and feeds causal knowledge back into mutation and scoring (Chen et al., 8 May 2026).

The first stage is tool-aware seed generation. An LLM is given the original harmful prompt and a description of the target agent’s tool configuration, including tool names and functionalities. It rewrites the harmful prompt into a more structured form, often using clauses or explicit sequencing such as “First, create a scene of … Then, add … Finally, adjust …”, with the goal of increasing the likelihood that the planner will decompose the request into multiple steps.

The second stage is trace collection and abstraction. For each queried prompt, OrchJail logs the entire trace $$\big(\rho(p), I(p), \tau(p)\big) \leftarrow \mathcal{A}(p),$$7 and applies regular-expression-based Trace2Orch rules to derive $$\big(\rho(p), I(p), \tau(p)\big) \leftarrow \mathcal{A}(p),$$8, $$\big(\rho(p), I(p), \tau(p)\big) \leftarrow \mathcal{A}(p),$$9, and $\rho(p)\in\{0,1\}$0. On successful prompts, a reasoning LLM infers prompt spans and explanations associated with those orchestration elements and stores $\rho(p)\in\{0,1\}$1 in a repository $\rho(p)\in\{0,1\}$2.

The third stage is orchestration-guided mutation. For a new prompt $\rho(p)\in\{0,1\}$3, OrchJail retrieves the top-$\rho(p)\in\{0,1\}$4 similar successful prompts from $\rho(p)\in\{0,1\}$5 using prompt embeddings and cosine similarity: $\rho(p)\in\{0,1\}$6 The framework sets $\rho(p)\in\{0,1\}$7. These neighbors provide in-context causal examples for the mutator.

Mutation then follows a two-branch strategy: $\rho(p)\in\{0,1\}$8 If the previous prompt was refused, the bypass-oriented mutator rewrites it into a more acceptable form while preserving harmful semantics and retaining orchestrational cues that previously induced successful tool chains. If the previous prompt was not refused but the image was not harmful enough, the semantics-oriented mutator reinforces missing semantic aspects while keeping the orchestration structure that enabled bypass. Each mutation step generates three candidate prompts.

The fourth stage is multi-objective scoring. Before making an expensive agent query, OrchJail uses an LLM-as-judge to assign three scores in $\rho(p)\in\{0,1\}$9: bypass probability $I(p)$0, prompt drift score $I(p)$1, and tool orchestration score $I(p)$2. The total score is

$I(p)$3

The top-1 candidate is selected as the next query. The loop repeats until the budget is exhausted or sufficient jailbreaks are found.

4. Empirical evaluation and main results

OrchJail was evaluated on three representative tool-calling text-to-image agents treated as black-box systems: GenArtist, CREA, and LayerCraft. It used the VBCDE-100 dataset from DACA, consisting of 100 harmful prompts across Violence, Bloody, Illegal activities, Discrimination, and a related sensitive content category from the original VBCDE taxonomy. Performance was measured with One-time Success Rate (O-SR), Re-use Success Rate (R-SR), Fréchet Inception Distance (FID), and number of queries (Chen et al., 8 May 2026).

Agent	OrchJail performance	Comparison summary
GenArtist	O-SR 72.63%, R-SR 96.26%, FID 156.26, #Queries 12.15	Best across all metrics reported
CREA	O-SR 66.27%, R-SR 79.11%, FID 160.73, #Queries 13.73	Best across all metrics reported
LayerCraft	O-SR 91.68%, R-SR 100%, FID 241.31, #Queries 7.23	Best O-SR, best FID, ties best R-SR

Across all three agents, OrchJail outperformed RING, DACA, SneakyPrompt, and JailFuzzer. On GenArtist, for example, OrchJail achieved O-SR 72.63%, exceeding JailFuzzer’s 63.48% and SneakyPrompt’s 57.12%; its R-SR was 96.26%, higher than JailFuzzer’s 93.20%; its FID was 156.26, lower than JailFuzzer’s 158.27; and its average query count was 12.15, below JailFuzzer’s 13.86. Comparable patterns held on CREA and LayerCraft.

The evaluation also showed that tool-aware seed generation materially improves the starting point of the search. Raw harmful prompts alone had 0% success across all three agents, whereas OrchJail’s rewritten seeds had non-zero jailbreak success rates of approximately 5–7%. This indicates that simply making a harmful intent multi-step and tool-compatible already changes the attack landscape in tool-calling systems.

Prompt naturalness was assessed with perplexity. On GenArtist, OrchJail achieved a perplexity of 35.91, lower than JailFuzzer’s 45.89, DACA’s 66.30, SneakyPrompt’s 337.97, and RING’s 1082.89, with similar patterns on CREA and LayerCraft. The qualitative implication drawn in the study is that orchestration-level exploitation does not require visibly garbled or typo-heavy prompts; natural language can remain grammatical and coherent while still steering the planner toward harmful tool chains.

5. Ablations, defenses, and limitations

Ablation studies isolate three modules: tool-aware seed generation, orchestration abstraction plus causal reasoning, and multi-objective scoring. Removing seed generation lowers O-SR and increases query cost. Removing orchestration abstraction and causal reasoning produces the largest drop in O-SR and increases query cost, indicating that causal orchestration reasoning is the most critical module. Removing scoring leaves O-SR less affected but produces the highest number of queries, showing that scoring is chiefly an efficiency mechanism. The study also reports that increasing the number of mutation candidates from 1 to 3 improves O-SR and reduces query count, but gains saturate beyond 3, so OrchJail fixes the candidate count at 3 (Chen et al., 8 May 2026).

Robustness was evaluated against a perplexity-based defense and three SmoothLLM variants on GenArtist. The perplexity defense was largely ineffective: O-SR remained 72.63%, FID shifted slightly from 156.26 to 159.41, and average queries rose from 12.15 to 13.05. SmoothLLM reduced performance but did not neutralize the attack. SmoothLLM-Insert yielded O-SR 70.32%, SmoothLLM-Swap 69.03%, and SmoothLLM-Patch 70.86%, while all three increased query cost and worsened FID. Even so, OrchJail under these defenses still exceeded JailFuzzer’s performance on an undefended GenArtist instance, where JailFuzzer reached 63.48% O-SR.

The limitations are specific and operational. OrchJail was evaluated only on three text-to-image agents. It assumes visibility of tool logs $I(p)$4, including tool names and textual arguments, which some closed systems may not expose. It depends on LLaVA-1.5-13B as mutation engine, scoring judge, and causal reasoning agent, so effectiveness depends on that model’s capability. The framework is demonstrated for text-to-image rather than video, audio, or other multi-modal agents. These constraints do not weaken the paper’s core claim, but they do bound its empirical scope.

6. Broader significance and related uses of the term

OrchJail’s broader significance lies in its reframing of jailbreaks as failures of orchestration rather than only failures of prompt filtering. The framework argues that tool-calling agents are structurally more vulnerable because orchestrated sequences of individually safe actions can be harmful as a whole, and therefore safety must be enforced at the agent level rather than only at the model or tool level (Chen et al., 8 May 2026).

The name also appears in a distinct, prospective sense in safety-critical clinical LLM deployments. A clinical education study describes an OrchJail-style orchestration layer sitting around LLMs and monitoring user prompts, LLM responses, and conversation context; detecting jailbreak attempts and deviations from domain constraints; and mitigating through blocking, rewriting, redirecting, or escalation to humans. That study is not an evaluation of the text-to-image fuzzing framework, but it uses the term to denote an orchestration-centric architecture for jailbreak handling and emphasizes spectrum-based risk scoring, explainability, and multi-stage monitoring (Nguyen et al., 21 Apr 2025).

A related line of work on commercial black-box LLMs proposed the MDH framework for malicious content detection based on LLMs with human assistance, along with RTA datasets built from explicitly harmful prompts. A plausible implication is that orchestration-aware red-teaming frameworks can benefit from such evaluation infrastructure when they need reliable harmful-response labeling or cleaned benchmark sets. MDH combines multiple LLM judgers, 0–10 scoring, fixed thresholds, multi-round voting, and minimal human review, achieving very low response-labeling error rates while keeping manual review small (Zhang et al., 14 Aug 2025).

Taken together, these results suggest a broader research program in which OrchJail denotes not only a specific orchestration-guided fuzzing framework for tool-calling text-to-image agents, but also a general shift toward orchestration-aware safety analysis. In that broader view, the key unit of analysis is no longer the prompt in isolation, but the full multi-step execution process: prompt wording, planner decomposition, tool selection, scheduling, intermediate control logic, and final output.