MT-JailBench: A Modular Benchmark for Understanding Multi-Turn Jailbreak Attacks

Abstract: Multi-turn jailbreaks exploit the ability of LLMs to accumulate and act on conversational context. Instead of stating a harmful request directly, an attacker can gradually steer the conversation toward an unsafe answer. Recent methods demonstrate this risk, but they are usually evaluated as black-box pipelines with different budgets, judges, retry rules, and strategy generation procedures. As a result, it is often unclear whether reported gains reflect stronger attack mechanisms or different experimental conditions. We introduce MT-JailBench, a modular evaluation framework for benchmarking multi-turn jailbreaks under fixed conditions. MT-JailBench implements each attack as five interacting modules: evaluation function, attack strategy, prompt generation, prompt refinement, and flow control. This design enables fair comparison across attack methods and component-wise analysis of what drives attack success. Using MT-JailBench, we find that resource budgets and evaluation functions are major confounders: controlling turns, retries, interactions, sampled strategies, and judges substantially change the ranking of attacks. At the component level, prompt generation accounts for most performance variation, while refinement and flow control provide moderate gains. We also find that explicit dynamic strategy generation is not always necessary; stochastic sampling from a fixed strategy can rival more elaborate diversification mechanisms. Finally, recomposing the best components yields a strong attack configuration that outperforms its source attacks and generalizes across diverse target LLMs. MT-JailBench therefore provides a modular framework for comparing multi-turn jailbreaks, understanding the impact of components, and guiding stronger red-teaming evaluations.

• The paper introduces MT-JailBench, a modular framework that decomposes multi-turn jailbreak attacks into five interacting components for precise performance analysis.
• It highlights that prompt generation is the dominant factor in attack success, with experiments showing up to 67.3% ASR under fixed resource conditions.
• The framework's design enables systematic benchmarking of both attacks and defenses, fostering reproducibility and improved safety mechanisms in LLM evaluations.

MT-JailBench: Modular Benchmarking for Multi-Turn Jailbreak Attacks

Motivation and Problem Context

Converging evidence from recent literature demonstrates that multi-turn jailbreak attacks, which exploit conversational context accretion, pose significant vulnerabilities for LLMs. Unlike single-turn attacks localized in one prompt, multi-turn attacks progressively steer interactions towards unsafe outcomes, capitalizing on the model's capacity to retain and reason over extended conversational histories, adapt to user feedback, and revise objectives over multiple turns. This shift in attack modality amplifies the attack surface and circumvents conventional guardrails, which predominantly rely on explicit detection of harmful intent within isolated prompts. Evaluations of multi-turn attacks have lacked methodological consistency, with prior works differing in experimental budgets, judging protocols, retry policies, and attack strategy generation procedures, obscuring the causal factors behind measured attack success rates (ASR).

MT-JailBench Framework and Modular Design

MT-JailBench introduces a robust, modular benchmark for evaluating multi-turn jailbreaks under precisely controlled external conditions. The modularity is realized in a decomposition of an attack pipeline into five distinct interacting components:

• Evaluation Function ($\mathcal{E}$): Determines attack success via binary, graded, or semantic similarity metrics, and may provide auxiliary signals for adaptive interactions.
• Attack Strategy ($\mathcal{T}$): Provides high-level guidance (persona, framing, topic sequence) prior to prompt construction.
• Prompt Generation ($\mathcal{G}$): Synthesizes turn-level prompts conditioned on the strategy, history, and prior evaluation signals. Supports both offline and online modes, with adaptive prompt construction based on intermediate feedback.
• Prompt Refinement ($\mathcal{U}$): Repairs prompts upon failure, preserving local attack intent while adapting via the evaluation feedback.
• Flow Control ($\mathcal{C}$): Governs the trajectory, managing advancement, retries, restarts, and termination to efficiently allocate interaction budgets.

The design embodies three key interfaces (PromptGenerator, OutcomeEvaluator, FlowController), supporting interchangeable attack modules and extensible addition of new defenses. This facilitates controlled ablation, attack recomposition, and systematic benchmarking of both attacks and defenses.

Empirical Findings and Component Attribution

Extensive experiments reveal several salient findings:

• Resource Budget Sensitivity: ASR is highly contingent on resource allowances (turn count, retries, interactions, strategy samples). Attacks relying on extensive search lose much of their apparent efficacy under constrained budgets, indicating that certain reported gains derive from increased exploration rather than superior mechanisms.
• Judge Variability: Choice of evaluation function introduces substantial deviation in measured success. Permissive binary judges inflate ASR relative to more stable, fine-grained score-based judges. Cross-validation across multiple judges is necessary for reliable attack comparisons.
• Component-Wise Analysis: After controlling for budget and judge, prompt generation emerges as the dominant determinant of performance variation. Refinement and flow control provide moderate, but non-negligible, gains. Explicit strategy diversification is not strictly required—stochastic sampling within a fixed strategy achieves competitive diversity.
• Benchmark-Guided Recompositions: By recomposing Crescendo’s prompt generator with XTeaming’s refinement and flow control (CRESCENDOX), the resulting configuration outperforms both source attacks on 19 of 21 target LLMs, demonstrating transferable gains and generalizability across both open-weight and proprietary models.

Numerical Results and Strong Claims

• ASR Under Constrained Conditions: Under fixed resources and cross-validated judges, Crescendo achieves ASR as high as 67.3% (GPT-4o) and 63.5% (Llama-3-70B), with XTeaming second-best; ActorBreaker and FITD, which appeared strong under unconstrained budgets, lose much of their advantage.
• Component Ablation: Replacing Crescendo’s prompt generator reduces ASR by up to 30 percentage points, while refinement and flow control modifications yield improvements up to 7 percentage points.
• CRESCENDOX Performance: Absolute ASR gains of 20+ points over its source attacks on multiple targets (e.g., Llama-4-Scout, Gemma-4-E4B, Grok-4.1-Fast), and >80% ASR across diverse models, including robust proprietary targets, validating MT-JailBench’s recompositional capability.
• Robustness to Defenses: CRESCENDOX preserves >80% ASR against both processing-based (Backtranslation, Aligner) and guardrail-based (NBF) defenses, underscoring its persistence even when models are protected.

Theoretical and Practical Implications

MT-JailBench systematically decouples mechanism from experimental confounders, enabling component-wise attribution of attack performance. This diagnostic clarity is essential for constructing defenses resilient to adaptive, multi-turn attacks. The framework’s modularity supports scalable integration of new attack and defense strategies, promoting reproducibility and extensible comparative analysis. By identifying prompt generation as the primary performance driver, future research may prioritize the development of adaptive context management and semantic mapping defenses. The finding that stochastic search within strong generators suffices for strategy diversity raises important questions regarding the role of explicit planning and agentic diversification in multi-turn adversarial settings.

Practically, MT-JailBench facilitates rigorous model evaluation and red-teaming, guiding model providers and safety teams to understand vulnerabilities that arise from conversational context accumulation. The ability to benchmark multi-turn attacks and defenses under fixed conditions increases transparency and comparability across system deployments.

Impact and Future Directions

MT-JailBench is positioned not as a final arbiter of model safety, but as a methodological scaffold enabling controlled study of LLM vulnerabilities. The dual-use potential of benchmarking tools necessitates caution; however, systematic evaluation is indispensable for developing robust, context-aware safety mechanisms. Future directions may include automated defense generation, integration with multimodal models, and exploration of fine-grained semantic smoothing and goal prioritization as countermeasures. Component-level modularity will likely inform the construction of hybrid defenses and foster improved understanding of conversational dynamics in adversarial AI settings. Given the shifting landscape of instruction-tuned and agentic LLMs, MT-JailBench’s approach offers scalable, adaptable infrastructure for ongoing safety evaluation.

Conclusion

MT-JailBench establishes a modular, extensible benchmark for multi-turn jailbreaks, clarifying the underlying drivers of attack success by isolating and controlling evaluation confounders. Prompt generation is the principal factor in attack efficacy, with refinement and flow control offering moderate contributions. The recomposed CRESCENDOX attack demonstrates transferable strength, robustly generalizing across models and defenses. MT-JailBench's methodology advances rigorous red-teaming practice, illuminating the interplay between attack mechanism and evaluation protocol, and creating a foundation for scalable, component-wise analysis in LLM safety research (2605.11002).