JailExpert: Experience-Driven LLM Jailbreak

• JailExpert is an automated black-box jailbreak framework that encodes and reuses structured past attack attempts as formalized experiences.
• It groups experiences by semantic drift using K-means clustering to enhance attack efficiency, achieving a 17% gain and 2.7× improvement over baselines.
• The framework formalizes each attack as a five-tuple and dynamically updates success and failure counts, enabling adaptive memory-centric attack generation.

JailExpert is an automated, experience-driven, black-box jailbreak framework for LLMs that is designed to reuse prior attack experience rather than re-optimize every new attack from scratch. It was introduced to address two shortcomings identified in iterative mutation and dynamic optimization methods: inefficiency stemming from outdated seed templates, and repetitive optimization that ignores the value of past attack cases. The framework formalizes each jailbreak attempt as structured experience, groups experiences by “jailbreak semantic drift,” and dynamically updates an experience pool as models and defenses evolve. In the reported experiments, it is described as the first framework to achieve a formal representation of experience structure, semantic-drift-based grouping, and dynamic updating of the experience pool, with average gains of 17% in attack success rate and 2.7 times in attack efficiency over current state-of-the-art black-box jailbreak baselines (Wang et al., 25 Aug 2025).

1. Conceptual basis and research setting

JailExpert is situated in the study of jailbreak attacks, namely prompt-based methods that circumvent safety alignment and induce LLMs to output malicious content. The framework starts from the observation that existing black-box jailbreak methods often treat each new model or malicious query as an entirely fresh optimization problem. JailExpert instead assumes that prior successful and failed attacks contain reusable information about how benign-looking instructions are transformed into effective jailbreak prompts, and that this information can be harvested, organized, and reused for future attacks (Wang et al., 25 Aug 2025).

The system’s stated goal is two-fold. First, it codifies every successful or failed jailbreak attempt into a structured “experience” so that prior attack insight is retained. Second, it organizes those experiences according to the semantic shift between the original harmful instruction and the final jailbreak prompt that elicited unsafe behavior. This design makes JailExpert an attack framework rather than a defense system, although the motivating premise is that jailbreak research can identify vulnerabilities and guide the development of robust security frameworks (Wang et al., 25 Aug 2025).

A common misconception is to treat automated jailbreak frameworks as purely mutation-based search procedures. JailExpert differs in that its central object is not merely a prompt mutation operator, but an evolving repository of attack cases, cluster representatives, and success/failure statistics. This suggests a memory-centric formulation of jailbreak automation rather than a purely online optimizer.

2. Formal representation of jailbreak experience

The core abstraction in JailExpert is the formal encoding of each past jailbreak attempt as a five-tuple

$$e = (\mathcal{I}, \mathcal{J}, A, s, f), \qquad A = \langle \mathcal{T}, \mathcal{M}\rangle,$$

where $\mathcal{I}$ is the original harmful instruction, $\mathcal{J}$ is the jailbreak prompt that induced the violation, $A$ is the jailbreak pattern, $s$ is the total number of successful executions of that exact experience, and $f$ is the total number of failures (Wang et al., 25 Aug 2025).

The “jailbreak pattern” $A = \langle \mathcal{T}, \mathcal{M}\rangle$ consists of a mutation strategy $\mathcal{T}(\cdot)$ and a template skeleton $\mathcal{M}$. Together, they transform the original instruction into the final jailbreak prompt. By explicitly tracking both $s$ and $\mathcal{I}$0, JailExpert treats each stored case as an object whose apparent reliability can change over time as the model environment changes (Wang et al., 25 Aug 2025).

Element	Meaning
$\mathcal{I}$1	Original harmful instruction
$\mathcal{I}$2	Final jailbreak prompt
$\mathcal{I}$3	Mutation strategy plus template skeleton
$\mathcal{I}$4	Total successful executions
$\mathcal{I}$5	Total failures

This representation is important because it moves beyond prompt collections or handcrafted templates. A stored experience includes both a reusable transformation pattern and an empirical history of success and failure. A plausible implication is that the framework can discount attack patterns that become obsolete after model updates without discarding them entirely.

3. Semantic drift, clustering, and pattern retrieval

To avoid an unstructured store of prior experiences, JailExpert groups them by “jailbreak semantic drift.” With embedding function $\mathcal{I}$6—implemented in practice with OpenAI’s text-embedding-3-small—the drift of an experience is defined as

$\mathcal{I}$7

JailExpert then applies K-means clustering to these drift vectors, with the number of clusters selected via silhouette score. Each cluster $\mathcal{I}$8 has centroid $\mathcal{I}$9, and is intended to collect jailbreaks that move an instruction in roughly the same direction in embedding space, which the paper interprets as exploiting a similar vulnerability (Wang et al., 25 Aug 2025).

Within each cluster, the framework selects one representative pattern $\mathcal{J}$0 by choosing the mutation-and-template pair with the highest product of frequency and success rate. When a new malicious instruction $\mathcal{J}$1 arrives, JailExpert generates a candidate prompt for each cluster,

$\mathcal{J}$2

and scores it by alignment between the candidate’s drift and the cluster centroid:

$\mathcal{J}$3

The clusters are then ranked by these scores, so the system tests first those families of transformations whose “signature drift” is closest to the projected needs of the new instruction (Wang et al., 25 Aug 2025).

This retrieval mechanism is the conceptual center of JailExpert. Rather than searching prompt space uniformly or relying on fixed seed templates, it searches a library of prior transformation types indexed by embedding-space displacement. This suggests that the framework models jailbreaks as recurring classes of semantic transformation rather than isolated prompt artifacts.

4. Dynamic updating and attack procedure

After ranking clusters, JailExpert enters an attack loop with maximum iteration budget $\mathcal{J}$4. For the top-scoring cluster, it first evaluates the representative-generated prompt. If that prompt fails, the system searches within the same cluster for the single experience whose initial instruction is most similar to the new instruction, weighted by empirical success ratio:

$\mathcal{J}$5

The pattern from the maximizing experience is then applied to generate a second candidate for that cluster (Wang et al., 25 Aug 2025).

After each trial, the framework updates success and failure counts. Failed attempts increase $\mathcal{J}$6 for experiences whose representative pattern was used; successful attempts increase $\mathcal{J}$7. The cluster centroid may also be updated if needed. The cluster is then removed from consideration for the current instruction, and the loop proceeds to the next-ranked cluster until success or exhaustion of the budget (Wang et al., 25 Aug 2025).

The practical significance of this design is adaptive memory. Because every failure reduces the relative prominence of the corresponding pattern and every success increases it, JailExpert can respond to model updates or new defenses without explicit retraining. The framework therefore combines offline structure—stored experiences, clusters, and representatives—with online adaptation through count-based updating.

5. Experimental setup and quantitative performance

The reported evaluation uses seven victim models: three open-source models—Llama2-7B-chat, Llama2-13B-chat, and Llama3-8B-Instruct—and four closed-source models—GPT-3.5-Turbo, GPT-4-Turbo, GPT-4, and Gemini-1.5-Pro. Initialization data come from the JBB jailbreak-benchmark, specifically attack results from ReNeLLM, CodeChameleon, Jailbroken, and GPTFuzzer. Evaluation is conducted on 110 instructions drawn from AdvBench, using 50 refined queries, and StrongReject. Baselines include GCG, CodeChameleon, PAIR, GPTFuzzer, ReNeLLM, Jailbroken, and AutoDAN-Turbo. The two reported metrics are Attack Success Rate (ASR), judged by a GPT-4-Turbo evaluator, and ASR Efficiency (ASR-E), defined as ASR divided by average query cost (Wang et al., 25 Aug 2025).

The central quantitative result is that JailExpert reaches an average ASR of 90% and an average ASR-E of 20.2, compared with 73% and 4.4, respectively, for AutoDAN-Turbo in the condensed comparison table. The paper also reports specific improvements on individual models, including Llama2-7B from 58% to 97% ASR, GPT-4-Turbo from 79% to 96%, and Gemini-1.5-Pro from 86% to 100% (Wang et al., 25 Aug 2025).

Model or average	AutoDAN-Turbo ASR	JailExpert ASR
Llama2-7B-chat	58%	97%
Llama2-13B-chat	56%	91%
GPT-4-Turbo	79%	96%
Gemini-1.5-Pro	86%	100%
Average	73%	90%

The efficiency result is equally emphasized. JailExpert’s ASR-E of 20.2 is reported as roughly 2.7 times higher than the next best baseline. In the high-level overview, the paper also states that the framework raises average ASR from under 70% for the best prior baseline to 90%, while improving attack efficiency, measured as queries per successful jailbreak, by roughly $\mathcal{J}$8 (Wang et al., 25 Aug 2025).

These results should be interpreted together with later work on jailbreak evaluation. StrongREJECT argues that many jailbreak studies overstate effectiveness because weak evaluators give credit to low-quality or capability-degraded outputs, and JADES reports that decompositional scoring can substantially reduce estimated attack success rates relative to earlier binary judgments (Souly et al., 2024, Chu et al., 28 Aug 2025). This contextualizes JailExpert’s reported gains as results under its stated evaluation protocol, not as evaluation-independent constants.

6. Relation to adjacent jailbreak, evaluation, and defense research

JailExpert belongs to a broader line of automated jailbreak systems that seek stronger search, better transfer, or richer memory. JPRO extends the automation problem to multimodal vision-LLMs through a four-agent framework—Planner, Attacker, Modifier, and Verifier—and reports over 60% attack success rate on multiple advanced VLMs, including GPT-4o. SRTJ further develops the memory-centric direction by coupling experience-driven attack generation with answer set programming-based rule selection and a hierarchical rule library that separates short-, middle-, and long-term rules (Zhou et al., 10 Nov 2025, Li et al., 1 May 2026).

The framework also interacts indirectly with benchmark construction and attack assessment. JBDistill treats jailbreak attacks as raw material for building renewable static safety benchmarks, selecting prompts that maximize effectiveness, separability, and diversity across held-out models. JADES addresses the evaluation side by decomposing a harmful question into weighted sub-questions and aggregating sub-scores into a final success judgment, achieving 98.5% agreement with human evaluators on JailbreakQR (Zhang et al., 28 May 2025, Chu et al., 28 Aug 2025).

On the defensive side, JailExpert’s authors note that standard filters—perplexity, RA-LLM random token removal, LlamaGuard, and OpenAI Moderation—are only marginally effective against the framework. Other defense papers in the same research area propose different countermeasures: a staged semantic linear-classification pipeline centered on TF-IDF and Linear SVM filtering; SelfGrader’s token-level-logit grading mechanism; and domain-specific two-layer detection via BERT-based linguistic feature extraction in clinical training LLMs (Wang et al., 25 Aug 2025, Rao et al., 22 Dec 2025, Zhang et al., 1 Apr 2026, Nguyen et al., 10 Feb 2026).

This broader landscape makes clear that JailExpert is not merely an isolated attack method. It occupies one point in an emerging cycle of attack memory, benchmark distillation, decompositional evaluation, and lightweight or staged defense.

7. Limitations, implications, and security significance

The paper identifies a restricted experience space as a current limitation. JailExpert presently integrates only two flavors of experience—mutation strategies and template skeletons—and these are drawn from black-box methods. The authors explicitly suggest that integrating white-box suffixes, real-time activation patterns, or reinforcement-learning feedback could enrich the experience pool (Wang et al., 25 Aug 2025).

A second implication concerns model-security dynamics. Because JailExpert groups attacks by recurring semantic drift classes and continuously updates empirical success statistics, it frames LLM vulnerabilities as recurring exploit families rather than one-off prompt tricks. This suggests that purely static defenses may be structurally mismatched to adaptive, experience-accumulating attackers.

From a security standpoint, the framework highlights a strategic asymmetry. An adversary that catalogues and reuses prior jailbreak experience can reduce both search cost and query burden over time. The paper therefore argues for more robust safety certifiers that can detect shifts in semantic drift or dynamically adversarial prompts, rather than relying on fixed prompt filters or isolated signature matching (Wang et al., 25 Aug 2025).

JailExpert’s implementation is reported as publicly available at XiZaiZai/JailExpert, which further positions it as an artifact for reproducible study and comparative analysis in jailbreak research (Wang et al., 25 Aug 2025).