Not All Turns Matter: Credit Assignment for Multi-Turn Jailbreaking

Abstract: Deploying LLMs in multi-turn dialogues facilitates jailbreak attacks that distribute harmful intent across seemingly benign turns. Recent training-based multi-turn jailbreak methods learn long-horizon attack strategies from interaction feedback, but often rely on coarse trajectory-level outcome signals that broadcast uniformly to every turn. However, we find that turn-level contributions in multi-turn jailbreaking are non-uniform, phase-dependent, and target-specific. Such coarse outcome supervision induces a credit assignment problem, leading to over-rewarding redundant turns in successful trajectories and under-crediting useful intermediate turns in failed ones. To address this, we propose TRACE, a turn-aware credit assignment framework for reinforcement learning (RL)-based multi-turn jailbreaking. For successful trajectories, TRACE estimates turn-level contributions via leave-one-turn-out semantic masking; for failed ones, TRACE assigns penalties based on prompt harmfulness and semantic relevance, with an additional local refusal-aware penalty. Furthermore, we reuse the attack-side credit signal for multi-turn defense alignment. Extensive experiments on open-source and closed-source targets show that TRACE achieves strong overall performance in effectiveness, transferability, and efficiency, yielding about a 25% relative improvement in attack success rate over the strongest RL baseline while also improving the safety-utility balance when reused for defense alignment.

• The paper introduces TRACE, an RL-based framework that uses leave-one-turn-out masking to identify attack-critical turns in multi-turn jailbreaking.
• It demonstrates a 25% improvement in attack success rate with more optimal phase structuring and reduced redundant exploration.
• The approach also repurposes turn-level credit for defense alignment by generating safer multi-turn responses, improving LLM robustness.

Turn-aware Credit Assignment for Multi-Turn Jailbreaking

Problem Formulation and Motivation

The paper "Not All Turns Matter: Credit Assignment for Multi-Turn Jailbreaking" (2605.08778) systematically analyzes a critical RL optimization problem in aligning attack strategies for multi-turn jailbreaking of LLMs. Such attacks distribute a harmful objective over several seemingly innocuous dialogue turns, ultimately circumventing safety mechanisms that would otherwise block direct, single-shot attacks. Existing RL-based attack optimizers largely assign single, trajectory-level scalar feedbacks to all turns, failing to account for the disparate, often non-uniform contributions of specific interactions. The result is imprecise credit assignment, with successful attacks rewarding extraneous or redundant context and failed attacks insufficiently penalizing crucial mistakes.

Experimental observations reveal that turn-level causal contribution is:

• Non-uniform: A minority of turns are "attack-critical," enabling harmful success, while many are functionally redundant.
• Phase-dependent: The contextual dialogue position modulates when sensitive or escalatory prompts are likely to advance versus derail the attack.
• Target-specific: Distinct LLMs exhibit idiosyncratic refusal boundaries, so the same prompt interacts with varying safety behaviors.

Uniform outcome broadcasting induces both inefficient exploration and suboptimal policy learning. Redundant turns are overfit, critical turns can remain underexplored, and safety signals from intermediate steps are diluted, leading to failure modes such as premature exposure or harmfulness drift.

The TRACE Framework

The authors propose TRACE (TuRn-level Assignment for CrEdit), an RL-based framework that implements turn-aware credit assignment for multi-turn jailbreaking. Instead of broadcasting trajectory reward signals, TRACE decomposes credit at the turn level, addressing both successes and failures with phase- and target-adaptive mechanisms.

Success-side Turn Credit

For successful trajectories, TRACE employs a leave-one-turn-out semantic masking. Each turn is omitted in turn, the dialogue "rolled out" under the new context, and the final harmful outcome is recomputed. If the attack fails when a turn is removed, that turn is assigned high causal credit (attack-critical); if success is unchanged, the omitted turn is deemed redundant. This approximates, in the absence of ground-truth causal labels, the true marginal contribution of each interaction.

Failure-side Penalty

For failed episodes, success-side masking is unreliable. Instead, TRACE assigns penalties based on:

• Prompt harmfulness: Penalty is concentration-adaptive, based on target- and phase-specific successful priors, penalizing harmfulness levels deviating from optimal phase patterns.
• Semantic relevance: Penalty activates if the prompt loses sufficient semantic alignment with the original harmful intent, based on trajectory-level statistics from successful attacks.

Both are normalized to prevent over-suppression and to provide stable gradients for RL optimization.

Refusal-aware Process Penalty

TRACE includes an explicit local penalty for target refusals, weighted more strongly for early-stage turns. This incentivizes the RL policy to avoid local refusals (where recoverability in later stages is target-dependent) and induces better phase structuring in prompt escalation.

Empirical Evaluation

TRACE is evaluated against a comprehensive suite of competitive baselines: single-turn attackers, multi-turn workflow methods, and RL-based multi-turn optimizers (e.g., TROJail, SEMA). Results are measured over several LLMs (Qwen2.5, gpt-oss-20b, Llama3/8B) using diverse harmful prompt sets (HarmBench, JailbreakBench, WildJailbreak), both open- and closed-source.

Key empirical findings:

• Effectiveness: TRACE achieves $\sim$25% relative improvement in attack success rate (ASR@1) over the strongest RL baseline, reaching 90%+ ASR in matched settings.
• Transferability: Attack policies trained under TRACE with mixed targets (i.e., TRACE-mix) generalize robustly across both open-source and commercial models, exceeding 80–90% ASR even against black-box models (GPT-4o, Gemini-2.5-Pro).
• Efficiency: Significant gains are achieved without added query or turn budget; policy shifts induced by turn-aware credit actually reduce redundant exploration.
• Credit Assignment Correction: Policy behaviors show more optimal phase structuring (fewer premature unsafe prompts, less late-stage harmfulness drift).
• Cross-judge Robustness: Improvements persist under external judge models (LlamaGuard, GPT-4o), ruling out reward-model overfitting.

Ablations

Disentangling each TRACE component, inclusion of success-side and failure-side turn-aware credit yields incremental, additive improvements in ASR, and the refusal-aware penalty further boosts in-target efficiency (with a mild trade-off against cross-target transferability).

Defense Alignment via Turn-level Credit

Beyond red-teaming, TRACE's attack-side credit signal is repurposed for multi-turn defense alignment. Attack-critical turns identified by leave-one-turn-out semantic masking are partitioned into latent-risk (intermediate, not directly harmful) and direct-harm buckets. These are then rewritten into safer response alternatives, yielding differentiated preference triples for DPO-style defense training. Empirical results indicate that this approach—particularly when modeling latent risk—enhances both multi-turn and single-turn robustness without substantial degradation in general capabilities (MMLU, GSM8K, GPQA).

Implications and Future Directions

The research demonstrates that fine-grained, turn-aware credit assignment is essential for both scalable attack and robust multi-turn defense alignment in LLMs, especially as adversarial behavior becomes more agentic and distributed. The leave-one-turn-out semantic masking used by TRACE provides a pragmatic avenue for causal ascription in domains lacking explicit ground-truth progress signals.

Broader implications include:

• RL-based jailbreaking research: Future attack optimizers should internalize trajectory-level attribution rather than rely on coarse, trajectory-level rewards. Credit design will become a key differentiator for LLM red-teaming competitiveness and interpretability.
• Defensive alignment: Utilizing turn-aware signals can enable earlier and more precise risk intervention, preventing both latent and explicit hazardous behaviors in agentic dialogues, and supporting more granular safety-utility trade-offs.
• Extension to other multi-turn settings: The framework naturally suggests application to other long-horizon agentic RL domains (e.g., sequential decision making, planning, multi-agent LLM interactions), where reward sparsity and delayed credit hinder efficient optimization.

Potential future work includes: (i) improving strategy diversity to handle more complex adversarial scenarios, (ii) refining defense alignment for enhanced safety-helpfulness calibration, and (iii) extending turn-aware credit mechanisms to broader classes of multi-turn, open-ended tasks beyond jailbreaking.

Conclusion

TRACE advances the state of the art in multi-turn LLM jailbreaking by introducing principled turn-aware credit assignment, empirically demonstrating substantial effectiveness and generalization improvements, and reusing the resulting signals to improve defense alignment. The approach identifies and exploits structural characteristics of long-horizon adversarial interactions and provides an architectural foundation for robust, data-efficient jailbreaking and safety alignment in emergent agentic AI systems.