The Art of the Jailbreak: Formulating Jailbreak Attacks for LLM Security Beyond Binary Scoring

Abstract: Jailbreak attacks -- adversarial prompts that bypass LLM alignment through purely linguistic manipulation -- pose a growing operational security threat, yet the field lacks large-scale, reproducible infrastructure for generating, categorizing, and evaluating them systematically. This paper addresses that gap with three contributions. (1) Large-scale compositional jailbreak dataset. We construct 114,000 adversarial prompts by applying 912 composing strategies to 125 harmful seed prompts from JailBreakV-28K. Every prompt is assigned to one of 14 cybersecurity attack categories (e.g., malware, phishing, privilege escalation) via a six-model majority-vote pipeline, and each strategy is ranked by effectiveness per category, enabling principled strategy selection grounded in concrete adversarial objectives. (2) Automated jailbreak generation. We instruction-fine-tune category-aware LLMs on Moderate and Optimal subsets, producing models that synthesize fluent jailbreak prompts from a harmful seed at inference time -- no templates, no gradient search. Our generators achieve perplexity 24-39 versus 40-140 for AutoDAN and AmpleGCG, with safety-filter evasion rates of 0.29-0.51 Mal (LlamaPromptGuard-2-86M), enabling controllable, scalable red-teaming under realistic adversarial conditions. (3) OPTIMUS: a training-free jailbreak evaluator. OPTIMUS is a continuous metric J(S,H) that jointly captures semantic similarity between the harmful seed and the jailbreak (S) and harmfulness probability (H) via calibrated penalty functions. Unlike binary attack success rate (ASR), OPTIMUS requires no task-specific training, generalizes across evolving strategies, and exposes a stealth-optimal regime (S*=0.57, H*=0.43) that ASR misses. Experiments across 114,000 prompts confirm that OPTIMUS separates Weak, Moderate, and Optimal jailbreaks with category-level evidence binary evaluation cannot supply.

• The paper builds a 114,000-prompt, 14-category jailbreak dataset and instruction-tuned generators that create fluent attacks with perplexity of 24–39 and PromptGuard detection rates as low as 0.29, compared with 1.00 for key baselines.
• The paper introduces Optimus, a training-free score combining semantic similarity and harmfulness to identify stealthier attacks that binary attack success rate overlooks, with an empirical optimum near S=0.57 and H=0.43.
• The paper shows that category-aware jailbreak evaluation improves red-teaming insight but remains limited by weak agreement in broad taxonomic categories, single-turn open-source testing, and unverified transfer to proprietary, multimodal, or multi-turn systems.

Overview

This paper addresses a structural gap in LLM jailbreak research: the absence of large-scale, reproducible infrastructure for generating, categorizing, and evaluating adversarial prompts under an explicit threat model. The authors — Hossain et al. (2605.09225) — make three integrated contributions: (1) a compositional dataset of 114,000 adversarial prompts built from 912 in-the-wild strategies applied to 125 harmful seeds, each labeled into one of 14 cybersecurity attack categories via six-model majority voting; (2) category-aware generator LLMs instruction-tuned on Optimus-filtered subsets that synthesize jailbreaks from raw harmful seeds without templates or gradient search; and (3) Optimus, a training-free, two-dimensional continuous metric $\mathbf{J}(S,H)$ combining semantic similarity $S$ between seed and jailbreak with harmfulness probability $H$, designed to expose a "stealth-optimal" regime that binary attack success rate (ASR) collapses entirely.

The work is grounded in a black-box, single-turn threat model: the adversary manipulates only the natural-language input channel but may iterate offline with local LLMs, embedding models, and harmfulness classifiers; the defender operates static alignment plus lightweight classifiers such as LlamaGuard or PromptGuard. The paper explicitly concedes this excludes multi-turn manipulation and multimodal settings.

Dataset construction and categorization

The corpus is assembled by composing 125 deduplicated seed prompts from JailBreakV-28K with 912 strategy sets extracted from WildJailbreak using WildTeaming's tactic-extraction prompt, yielding 114,000 composed candidates (plus ~1,037 additional records from WildJailbreak itself). Each seed is assigned to one of 14 cyber-attack categories — spanning malware, phishing, privilege escalation, data exfiltration, keylogging, social engineering, and others aligned with MITRE ATT&CK tactics including T1059, T1068, T1110, T1566, and T1041 — via few-shot labeling by six open-source LLMs (Llama-3.1-8B-Instruct, Tulu-3-8B, Mistral-7B-v0.3, Ministral-8B, Gemma-3-270m-it, Qwen2.5-7B-Instruct) followed by majority voting.

Taxonomy reliability is assessed honestly rather than optimistically. Overall Fleiss' $\kappa$ under a conservative binary formulation is only 0.053, which the authors attribute to the 14-way label space; per-category agreement is substantially higher, with substantial consensus ($\kappa \geq 0.60$) for Backdoor Implantation ($\kappa = 0.799$), Keylogging (0.796), Password Cracking (0.799), DoS (0.752), Phishing (0.679), RCE (0.654), and USB-Based Attacks (0.648), versus Fair-to-Moderate agreement for semantically diffuse categories like Malware ($\kappa = 0.371$). A human audit on 300 stratified prompts shows that when both expert annotators agreed, the LLM majority-vote label matched at 71.1% overall, reaching 100% alignment for technically bounded categories (Data Exfiltration, Keylogging, Privilege Escalation) but near-zero for USB-Based Attack and below 40% for Social Engineering and RCE. The implication is that unanimous LLM consensus is a usable proxy for human judgment only within lexically narrow categories; boundary ambiguity in broad categories remains an unresolved weakness of the taxonomy.

Unsupervised validation via KMeans clustering over all-MiniLM-L6-V2 embeddings shows strong intra-cluster cohesion for narrow categories and multi-cluster dispersion for Malware, Social Engineering, and Other, consistent with the manual taxonomy. Embedding correlation analysis further reveals semantic coupling among categories (e.g., Phishing–Social Engineering at ~0.57), indicating that category boundaries are not linguistically orthogonal.

The Optimus metric

Optimus scores each composed prompt as

$$\mathbf{J}(S,H) = \frac{2S(1-H)}{S+(1-H)} \cdot P_S(S) \cdot P_H(H),$$

where $S$ is cosine similarity between seed and jailbreak embeddings, $H$ is the probability from a zero-shot MNLI entailment classifier with the hypothesis that the prompt enables dangerous content, and $S$0, $S$1 are sigmoid penalties discouraging verbatim copies (over-similarity) and sanitized rewrites (under-harmfulness). The harmonic-mean core rewards balance between intent preservation and covert phrasing.

Model-pair selection across nine combinations of three sentence encoders and three NLI classifiers identifies all-mpnet-base-v2 paired with deberta-large-mnli as best (mean score 0.193, deviation 0.108); a weighted nine-pair ensemble reaches 98% of that mean with lower variance (0.098), and the paper's selection rule adopts the ensemble when its mean is within 2% of the best pair. Analytically, solving $S$2 for the balanced configuration $S$3 yields an interior optimum $S$4 with $S$5, where both penalties are active but neither dominates. Empirical prompt pairs cluster near this region, supporting the metric's calibration. Hyperparameter sensitivity is characterized across strict and lenient regimes, shifting the optimum to $S$6 and $S$7 respectively — a dependence the paper documents but does not eliminate, meaning absolute score values are configuration-relative even if tier boundaries are normalized by $S$8.

Prompts are tiered into Safe/Fail (<0.212), Weak (0.212–0.283), Moderate (0.283–0.377), and Optimal (0.377–0.471). The central claim — that ASR collapses distinctions Optimus preserves — is supported by category-wise analysis: Exploit Kit Delivery attains mean Optimus 0.334 at ASR 1.000 while Backdoor Implantation attains ASR 1.000 at only 0.232, implying the latter succeeds through direct phrasing detectable by input-level classifiers, whereas the former achieves stealth-preserving composition. Malware dominates volume ($S$9, 32.5% Optimal), and Social Engineering combines ASR 0.974 with moderate stealth (0.257), a pattern lexical filters miss.

Generator fine-tuning and evaluation results

Instruction fine-tuning uses LoRA/DoRA (rank 8, $H$0, effective batch size 1024, learning rate $H$1) on the Moderate and Optimal subsets (24,220 instances) for three base models: Llama-3.1-8B-Instruct, Tulu-3-8B, and Vicuna-7B. Against AutoDAN and AmpleGCG suffix-based baselines, the fine-tuned generators achieve markedly lower perplexity (24–39 vs. 105–142 for AutoDAN and 40–43 for AmpleGCG), comparable semantic similarity (0.62–0.64 vs. AmpleGCG's 0.72–0.73), and substantially lower malicious-detection rates under Llama-Prompt-Guard-2-86M (0.29–0.51 vs. 1.00 for both baselines). On response-level evaluation, the generators reach ASR 0.84–0.98 with StrongReject 0.21–0.22, compared to AutoDAN(Vicuna)'s ASR 0.99 with detector flagging up to 95%. The authors frame this as producing adversarially potent yet linguistically covert prompts — high utility for red-teaming precisely because they evade surface-form detection.

Victim-model comparison across four models shows Tulu-3 most refusal-aligned (ASR 0.41, WildGuard harmful rate 14.8%) versus Mistral-8B (ASR 0.71, HarmBench 76.2%) and Vicuna-7B (ASR 0.61, HarmBench 73.4%). Extended comparisons against eight additional baselines (IJP, GCG, SAA, ZULU, PAIR, DrAttack, Puzzler, Base64) across five victim configurations show ZULU and PAIR evading PromptGuard almost entirely while producing low-quality or degenerate text (ZULU perplexity exceeds 950), underscoring the trade-off Optimus is designed to quantify: evasion without fluency or intent preservation scores poorly.

Safety-classifier sensitivity analysis quantifies composition's masking effect: under LlamaPromptGuard-86M, simple prompts are classified benign in 112,810 of 114,912 cases, but composed prompts flip 68,992 to malicious — a sixfold-plus increase in categories like Malware, Keylogging, and Social Engineering. Conversely, LlamaGuard-3 safe predictions rise substantially upon composition for Backdoor Implantation (8 → 445) and Data Exfiltration (8 → 1212), demonstrating that lightweight classifiers are inconsistently robust to reframing depending on category.

Limitations and open questions

The paper's stated limitations are concrete. Evaluation covers open-source models and single-turn attacks only; generalization to proprietary, multi-turn, or multimodal systems is untested. Multi-LLM voting reduces label variance but cannot remove correlated biases arising from shared pretraining distributions — a concern borne out by the USB-Based Attack audit failure (0% LLM-human alignment despite 97.2% inter-human agreement). Human evaluation of generated jailbreak outputs was not performed, though the authors cite StrongReject's reported alignment with human judgment as partial mitigation. Two questions remain open: whether the stealth-optimal operating point $H$2 transfers across embedding/entailment model families beyond the nine tested pairs, and whether category-aware generation improves targeted defense hardening relative to category-agnostic red-teaming.

Conclusion

This paper provides a unified pipeline — compositional synthesis at scale, security-grounded multi-model categorization with audited reliability, instruction-tuned generators, and a continuous training-free evaluator — that links jailbreak behavior to established intrusion taxonomies. Its strongest empirical claims are the fluency and detectability advantages of fine-tuned generators over suffix-based baselines (perplexity 24–39; PromptGuard-86M malicious rates as low as 0.29 vs. 1.00) and the demonstration that binary ASR conflates stealth-distinct attack modes. The framework's durability depends on the stability of its embedding and entailment components as attack strategies evolve, and its taxonomy on resolving boundary ambiguity in semantically diffuse categories — both explicitly left as open problems.