RL-MTJail: Multi-Turn Jailbreaking Method

• RL-MTJail is a reinforcement learning framework that formulates multi-turn jailbreak generation as a Markov decision process.
• It employs heuristic process rewards to control harmfulness and maintain semantic relevance, thereby mitigating sparse supervision issues.
• Empirical evaluations demonstrate RL-MTJail achieving an average ASR of 86.23%, outperforming single-turn methods and other baselines.

RL-MTJail denotes a reinforcement-learning framework for automated black-box multi-turn jailbreaking of LLMs. It trains an attacker LLM to elicit harmful content from black-box models through a sequence of prompt-output interactions, rather than through single-turn prompt optimization. The method formulates jailbreak generation as a multi-turn reinforcement learning task, directly optimizing the harmfulness of the final-turn output as the outcome reward. To mitigate sparse supervision and promote long-term attack strategies, it introduces two heuristic process rewards: one controls the harmfulness of intermediate outputs to avoid triggering rejection mechanisms, and the other maintains semantic relevance to the original harmful target so that the dialogue does not drift into irrelevant content (Xiong et al., 8 Dec 2025).

1. Problem formulation and scope

RL-MTJail models multi-turn jailbreaking as a Markov decision process. At turn $t$, the state is the original harmful target together with the interaction history up to turn $t-1$: $s_t = (x_0, \tau_{t-1}),$ where

$\tau_{t-1} = [(x_1,y_1),\ldots,(x_{t-1},y_{t-1})].$

The action $a_t$ is the attacker’s next prompt $x_t$, sampled from policy $\pi_\theta(\cdot \mid s_t)$. After the attacker emits $x_t$, the black-box victim model $\pi_\phi$ returns a response $y_t \sim \pi_\phi(\cdot \mid s_t,x_t)$, and the next state appends the new prompt-response pair to the transcript (Xiong et al., 8 Dec 2025).

The episode terminates in either of two conditions: when the victim’s response is judged “harmful enough,” written as $t-1$0, or when the interaction reaches a fixed horizon $t-1$1. This framing distinguishes RL-MTJail from single-turn jailbreak optimization. Instead of treating each prompt as an isolated candidate, the method optimizes a trajectory whose earlier turns are valuable only insofar as they improve the probability of obtaining a harmful final response.

This design is explicitly targeted at black-box settings. The attacker does not require access to model internals, gradients, or training data of the victim model. The black-box assumption is central to the method’s practical orientation: the attacker observes only the victim’s outputs and learns a multi-turn policy that exploits those outputs as intermediate signals.

2. Reinforcement-learning objective and heuristic rewards

The reward structure in RL-MTJail has two layers. The outcome reward is defined on the completed dialogue: $t-1$2 so that optimization is tied to the harmfulness of the final response. Because this reward is sparse, the method supplements it with a process reward

$t-1$3

The first term, “over-harm mitigation,” penalizes prompts that cause the victim to refuse immediately. If the response $t-1$4 is a refusal, then $t-1$5; otherwise $t-1$6. Refusal is detected via “a small refusal-detection prompt to the victim LLM itself.” The second term, “target-guided progression,” encourages the intermediate responses to become more semantically similar to the original harmful target: $t-1$7 where $t-1$8 is cosine similarity in embedding space using MiniLMv2 (Xiong et al., 8 Dec 2025).

Optimization uses a clipped-PPO (GRPO) style objective with KL and entropy regularization. The combined advantage estimate is

$t-1$9

where the outcome term is normalized at the trajectory level and the process term discounts future process rewards over turns. This means that RL-MTJail does not merely reward success; it shapes the trajectory toward delayed success by rewarding intermediate behaviors that postpone refusal and preserve semantic alignment with the original harmful target.

A common misunderstanding is to interpret the heuristic rewards as substitutes for the final harmfulness objective. They are not. The paper presents them as auxiliary signals for sparse-reward mitigation. The final-turn outcome reward remains the principal optimization target, while the process rewards provide empirically motivated guidance for long-term attack strategy learning.

3. Training procedure and evaluation protocol

The training loop samples $s_t = (x_0, \tau_{t-1}),$0 trajectories per batch by rolling out the current attacker policy up to horizon $s_t = (x_0, \tau_{t-1}),$1 or until the harmfulness threshold is reached. For each trajectory and each turn, the method computes the outcome reward and the process reward, then computes normalized outcome and process advantages, combines them, and performs one PPO-style gradient step on the objective $s_t = (x_0, \tau_{t-1}),$2 (Xiong et al., 8 Dec 2025).

The reported attacker model is Qwen2.5-3B–Instruct. The victim models are Llama-3.1-8B-Instruct, Qwen2.5-7B-Instruct, Gemma-2-9B-IT, and Mistral-7B-Instruct-v0.3. Training uses AdvBench with 520 harmful targets. Evaluation is conducted on the HarmBench standard subset with 200 targets, StrongREJECT† with 288 targets, and JailbreakBench† with 55 targets. The primary metric is Attack Success Rate (ASR@3), defined as the percentage of targets for which at least one of three attempts yields a harmful response, with the HarmBench classifier as judge model at threshold $s_t = (x_0, \tau_{t-1}),$3 (Xiong et al., 8 Dec 2025).

The evaluation protocol emphasizes cross-model and cross-benchmark behavior. This is significant because the method is intended for automated red teaming under black-box access constraints. A plausible implication is that the authors view the learned attacker as a reusable policy rather than as a prompt search routine tied to a single victim or benchmark.

4. Empirical performance and ablations

Across the reported benchmarks and victims, RL-MTJail is described as consistently outperforming all single-turn and prior multi-turn methods. In the excerpted average ASR comparison, the method achieves the highest value among the listed baselines (Xiong et al., 8 Dec 2025).

Method	Avg. ASR
ActorAttack	65.75
MTSA	67.31
GRPO (outcome-only)	81.43
GRPO w/IPR	83.68
RL-MTJail	86.23

The ablation study on Gemma-2-9B isolates the contribution of the reward components. “Outcome only” yields an average of $s_t = (x_0, \tau_{t-1}),$4. Adding the over-harm term $s_t = (x_0, \tau_{t-1}),$5 raises the average to $s_t = (x_0, \tau_{t-1}),$6. Adding only the target progression term $s_t = (x_0, \tau_{t-1}),$7 yields $s_t = (x_0, \tau_{t-1}),$8. Using both yields $s_t = (x_0, \tau_{t-1}),$9 (Xiong et al., 8 Dec 2025). These numbers support the paper’s claim that the process rewards mitigate sparse supervision in complementary ways.

The method also exhibits several trajectory-level behaviors. ASR improves as the turn limit $\tau_{t-1} = [(x_1,y_1),\ldots,(x_{t-1},y_{t-1})].$0 increases from $\tau_{t-1} = [(x_1,y_1),\ldots,(x_{t-1},y_{t-1})].$1, then plateaus. Under “difficulty-aware adaptivity,” ASR degrades more gently as target difficulty rises, and the attacker automatically uses more turns on harder targets. In transfer experiments, an attacker trained on a robust victim such as Llama-3.1 or Gemma-2 transfers better to other victims, reaching approximately $\tau_{t-1} = [(x_1,y_1),\ldots,(x_{t-1},y_{t-1})].$2 average ASR, than attackers trained on “easier” targets. The paper further reports that RL-MTJail maintains high ASR while matching or exceeding the diversity of other RL and multi-agent baselines (Xiong et al., 8 Dec 2025).

These findings matter because they characterize RL-MTJail as a long-horizon policy learner rather than a narrowly tuned prompt optimizer. The plateau after five turns suggests diminishing returns from longer dialogues under the reported setup, while the transfer results suggest that training against more resistant victims exposes strategies that generalize better across model families.

5. Position within reinforcement-learning jailbreak research

RL-MTJail sits within a broader line of work that uses RL for automated jailbreak generation, but its defining feature is explicit multi-turn black-box optimization. Earlier black-box work such as RLbreaker casts jailbreaking as a search problem over prompt structures, with a DRL agent selecting among five mutator operations—rephrase, expand, shorten, crossover, and generate_similar—and using a cosine-similarity reward against a reference answer. RLbreaker uses a hard time-horizon $\tau_{t-1} = [(x_1,y_1),\ldots,(x_{t-1},y_{t-1})].$3 and a customized PPO variant without a learned value function (Chen et al., 2024). Relative to that formulation, RL-MTJail operates on dialogue trajectories rather than prompt-structure search trajectories.

Another adjacent approach is MTSA, a multi-turn safety-alignment framework built around an adversarial loop between a red-team model and a target model. MTSA uses a thought-guided attack learning stage followed by adversarial iterative optimization with a future-reward–based RLHF algorithm. In its attack evaluation on AdvBench, MTSA after three iterations reaches $\tau_{t-1} = [(x_1,y_1),\ldots,(x_{t-1},y_{t-1})].$4 ASR across models, with average $\tau_{t-1} = [(x_1,y_1),\ldots,(x_{t-1},y_{t-1})].$5, while the target model improves on safety benchmarks (Guo et al., 22 May 2025). RL-MTJail differs in objective: it is an attacker-training method rather than a joint red-team/defense co-optimization framework.

Jailbreak-R1 is another RL-based red-teaming system. It decomposes training into three stages—Cold Start, Warm-up Exploration, and Enhanced Jailbreak—and uses consistency, diversity, and progressive jailbreak rewards with GRPO. Its own discussion states a limitation of “Single-round jailbreaks only; no multi-turn dialogue refinement” (Guo et al., 1 Jun 2025). That limitation sharply contrasts with RL-MTJail’s explicit focus on multi-turn attack strategy.

More recent work on large reasoning models has introduced attention-guided rewards. The method in “Attention-Guided Reward for Reinforcement Learning-based Jailbreak against Large Reasoning Models” augments the action space to 17 prompt-refinement operators, including ten “cognitive persuasion” actions, and defines reward through a linear SVM over prompt and chain-of-thought attention proportions. On AdvBench, it reports ASR values such as $\tau_{t-1} = [(x_1,y_1),\ldots,(x_{t-1},y_{t-1})].$6 on Qwen3-1.7B and DeepSeek-R1-Distill-Llama-8B, and $\tau_{t-1} = [(x_1,y_1),\ldots,(x_{t-1},y_{t-1})].$7 on Qwen3-8B (Lin et al., 19 May 2026). This suggests a distinct branch of the literature in which RL is coupled to model-specific internal signals; RL-MTJail, by contrast, is framed for black-box victims.

6. Defensive context, threat surface, and terminological ambiguity

The RL-MTJail paper’s limitations section states that the process rewards “capture only a subset of factors influencing jailbreak success,” that victim models are static, and that diversity and controllability are not explicitly optimized beyond entropy regularization. Its discussion of potential defenses, via the ethical statement and broader literature, points toward “multi-turn safety alignment, early-dialogue interventions, or meta-learning defenses that detect progressive semantic drift” as countermeasures to RL-driven jailbreaks (Xiong et al., 8 Dec 2025).

A procedural defense aligned with that multi-turn threat model is RLM-JB, which treats jailbreak detection as a bounded recursive program rather than as one-shot classification. Its four stages are input canonicalization and de-obfuscation, coverage-guaranteeing chunking, parallel segment screening, and compositional evidence aggregation. The paper explicitly describes how to adapt this framework to an “RL-MTJail” scenario by chunking on tool boundaries and conversation turns, aggregating tool-intent signals across modalities, incorporating RL feedback as “confidence modifiers,” adding tool-aware overrides, and maintaining an append-only multi-turn state log (Shavit, 18 Feb 2026). This places RL-MTJail within a broader defense conversation centered on long-context hiding, semantic camouflage, split-payload attacks, and tool-augmented agents.

A related but distinct threat model arises in reinforcement learning with verifiable rewards. “Backdoors in RLVR: Jailbreak Backdoors in LLMs From Verifiable Reward” identifies a latent vulnerability to backdoor attacks within RLVR and proposes the Asymmetric Chain Backdoor. With less than $\tau_{t-1} = [(x_1,y_1),\ldots,(x_{t-1},y_{t-1})].$8 poisoned data in the training set, the backdoor can be implanted across model scales, safety performance degrades by an average of $\tau_{t-1} = [(x_1,y_1),\ldots,(x_{t-1},y_{t-1})].$9 when the trigger is activated, and clean-task performance remains within $a_t$0 of the clean model (Guo et al., 10 Apr 2026). Although this is not the same attack class as RL-MTJail, it broadens the security landscape: RL-based systems can be exploited both at inference time through learned attacker policies and at training time through poisoned reward-driven adaptation.

The term “RL-MTJail” is also not used uniformly across adjacent summaries. It appears most directly as the title of the black-box multi-turn jailbreaking method (Xiong et al., 8 Dec 2025), but it is also used as a scenario label in procedural-defense discussion (Shavit, 18 Feb 2026), and one technical summary uses “RL-MTJail” as an alias for “Jailbreak-R1” (Guo et al., 1 Jun 2025). In the most precise usage, RL-MTJail refers to the 2025 method for reinforcement learning–based automated black-box multi-turn jailbreaking.