- The paper presents a hierarchical two-level model that decouples turn encoding from global conversation dynamics to detect multi-turn jailbreak attacks.
- The model achieves state-of-the-art performance with 92.1% recall, 95.9% precision, and a notable F1 improvement over existing baselines.
- The work demonstrates that explicit cross-attention with token-level retrieval enables efficient moderation while mitigating quadratic self-attention complexity.
Jailbreak attacks on LLMs now increasingly exploit the multi-turn structure of conversations, distributing adversarial intent across several benign-appearing turns to evade turn-level moderation. Conventional classifiers, optimized for either overt toxicity or single-turn adversarial prompts, are structurally incapable of detecting this class of multi-turn attacks. Naive approaches that concatenate all turns and apply full-sequence Transformers suffer from prohibitive quadratic complexity in self-attention and indiscriminate allocation of modeling capacity over lengthy sequences. Hence, scalable architectures capable of cross-turn reasoning are essential for practical multi-turn jailbreak detection.
Hierarchical Attention Model Architecture
The core contribution is a hierarchical two-level model that efficiently decouples local turn encoding from global conversation dynamics, operationalized as a TurnEncoder and ConvTransformer stack.
TurnEncoder (Level 1): Each turn (prefixed with explicit role markers [USER], [ASST]) is independently processed by a frozen or fine-tuned multilingual E5-base encoder. This yields both (i) summary CLS embeddings for compact representation and (ii) token-level embeddings retained for downstream selective attention. Disjoint turn processing enables linear scaling in both computation and memory, with cross-turn dependencies addressed exclusively in the higher-level aggregation.
ConvTransformer (Level 2): Turn summaries are input to a 4-layer decoder stack (pre-LN, 8 heads, FFN size 2048), augmented with learned positional and role embeddings as inductive priors for turn order and speaker identity. Self-attention over summaries captures inter-turn relational patterns indicative of escalation, role manipulation, and context drifting. Critically, each layer also incorporates cross-attention to the full set of token-level embeddings, so that turn-level anomalies (subtle or isolated adversarial cues) can be dynamically retrieved during global reasoning. A residual attentive-pooling head enables selective emphasis of salient turns, mitigating the dilution problem seen in mean-pooling schemes.
Experimental Protocol
Datasets
Training leverages a curated 98k multi-turn corpus, balanced across sophisticated adversarial schemes (gradual escalation, framing, decomposition, sudden insertion, context narrowing, prompt injection) and hard negative safe conversations structurally similar to attacks. Evaluation employs a rigorously deduplicated 14k benchmark, combining real-world and synthetic conversations, covering 24 sources and all known multi-turn offense strategies. Jailbreak and safe labels are defined at the conversation level, not turn level.
Baselines
Comparisons include:
- Mean Pooling of turn embeddings (non-attentive aggregation)
- Llama 3.3 70B (concatenation over recent turns)
- Tree-of-Thought logical analysis (TOT OR, TOT Majority)
- Proprietary Claude Sonnet and Claude Opus 4.7 models (Anthropic's strongest safety-tuned offerings)
Main Results and Quantitative Analysis
The hierarchical model dominates across all primary metrics:
- Recall: 92.1%
- Precision: 95.9%
- F1: 0.9394
- False Positive Rate (FPR): 2.8%
These results reflect a 13.8% absolute recall improvement and 0.070 F1 lift compared to Claude Opus 4.7, while reducing FPR by more than half. Gains relative to Llama 3.3 70B and Tree-of-Thought variants are larger still, with F1 improvements in the 0.33–0.42 range. Mean-pooling ablation degrades recall by a striking 16.5 points, validating the necessity of explicit turn- and token-level attention.
Attention Mechanism Studies
Attention analysis elucidates the model's behavior: in safe conversations, self-attention from the [CONV_CLS] token concentrates on early turns, aligning with setting/context establishment; jailbreaks display a flatter distribution, peaking on later user turns, congruent with adversarial escalation patterns. Cross-attention exhibits heightened scrutiny on late user turns for jailbreaks, with Jailbreak/Safe ratios reaching 16x at deep turn positions. This dynamic allocation of modeling capacity confirms the model is not simply matching local content but operationalizing a learned theory of escalation and context manipulation.
Component ablations further show that:
- Self-attention alone detects escalation but over-indexes on benign instruction-following, inflating FPR.
- Cross-attention alone yields substantial recall loss, highlighting the necessity of combined mechanisms.
- Performance improvements saturate at 4 ConvTransformer layers, with no further substantive gains at deeper depths for the present dataset/task.
Theoretical and Practical Implications
This architecture confirms that compact per-turn embeddings, when equipped with both hierarchical self-attention and residual cross-attention to token-level information, are sufficient to robustly capture distributed adversarial intent. The model provides state-of-the-art robustness to sophisticated, multi-turn jailbreak strategies and outperforms even the best commercial safety detectors at lower computational cost.
Practically, this enables deployment of conversation-level moderation in real-world settings, especially as LLMs become more integrated with retrieval and tool-use pipelines where indirect, multi-modal context can be adversarially manipulated.
Theoretically, these findings articulate the limitations of pure long-context scaling and generic long-sequence Transformers (sparse, recurrent, or memory-augmented). Effective moderation requires explicit exploitation of conversation structure, role orderings, and non-uniform allocation of reasoning to high-risk regions—capabilities that emerge only under explicit hierarchical modeling.
Future Directions
Open areas include domain calibration, robustness to adaptive attacker policies, zero-shot transfer to new moderation policies or domains, and integration with tool/agent pipelines where the boundary of 'conversation' becomes diffuse. Addressing undetected compositional or multi-modal escalation strategies will likely require augmentation of the present hierarchy with richer intermediate supervision and submodule specialization for newly observed attack modes.
Conclusion
This work establishes a scalable, memory-efficient, and highly effective hierarchical attention architecture for the detection of multi-turn jailbreak attacks in LLM-mediated conversations. Through explicit separation of turn encoding and conversation-level reasoning—with the capacity to dynamically retrieve fine-grained token evidence—the proposed system sharply outperforms both research and industrial baselines across comprehensive multi-turn benchmarks, setting a new technical standard for robust, conversation-level LLM moderation.