DialTree-RPO: Multi-Turn Adversarial RL

• DialTree-RPO is an on-policy reinforcement learning framework that integrates breadth-first dialogue tree search with GRPO to discover multi-turn adversarial strategies against LLMs.
• It combines structured exploration, adaptive masking, and custom state-action encoding to enhance policy optimization and significantly improve attack success rates.
• Experimental results show a 25.9-point average ASR boost with only ~2.8 queries per dialogue across 10 target models, demonstrating its query efficiency and robustness.

DialTree-RPO is an on-policy reinforcement learning (RL) framework that augments Group Relative Policy Optimization (GRPO) with breadth-first dialogue tree search and pruning for the discovery of multi-turn adversarial attack strategies against LLMs. The framework treats multi-turn red-teaming as a sequential decision process, enabling systematic exploration of adversarial dialogue trajectories and outperforming prior methods on benchmarks of attack success. DialTree-RPO combines structured exploration, adaptive masking, policy optimization, and custom state-action encoding to induce sophisticated jailbreak tactics that exploit multi-turn vulnerabilities in both closed- and open-source LLMs (Guo et al., 2 Oct 2025).

1. Formal Problem Definition and MDP Formulation

DialTree-RPO frames the interaction between an attacker and a frozen target LLM as a fully observed, episodic Markov decision process (MDP) $$\mathcal{M} = (\mathcal{S}, \mathcal{A}, T, R, \gamma)$$, where:

• State space $(\mathcal{S})$: Each dialogue state $s_t$ consists of a fixed goal $g$ (e.g., jailbreak objective) and the dialogue history as a sequence of triplets $(c_i, q_i, r_i)$ for $i \leq t$, where $c_i$ is a chain-of-thought prompt, $q_i$ is the attack query, and $r_i$ is the target response.
• Action space $(\mathcal{A})$: At each turn $(\mathcal{S})$0, the action $(\mathcal{S})$1, generated by the attacker’s policy $(\mathcal{S})$2, specifies the next CoT and query components.
• Transition dynamics $(\mathcal{S})$3: The environment deterministically extends the state with the action and the sampled response $(\mathcal{S})$4.
• Reward function $(\mathcal{S})$5: Rewards are sparse and terminal, given by a safety-guard classifier $(\mathcal{S})$6 (HarmAug-Guard), providing $(\mathcal{S})$7 if any $(\mathcal{S})$8 within the trajectory $(\mathcal{S})$9, and $s_t$0 otherwise.
• Discount $s_t$1: Set to $s_t$2; only the final reward determines policy update.

This construction enables the adversarial attacker policy $s_t$3 to maximally exploit target vulnerabilities by planning over multi-turn conversational interactions (Guo et al., 2 Oct 2025).

2. Dialogue Tree Rollout and Exploration

Contrasting standard linear trajectory rollouts, DialTree-RPO utilizes a dialogue tree with branching factor $s_t$4 and depth up to $s_t$5:

• Tree Breadth and Depth: At each turn, every frontier node is expanded by sampling $s_t$6 candidate actions from $s_t$7.
• Pruning: Candidate branches are immediately pruned if the quality checker $s_t$8 detects malformed outputs or off-topic queries.
• Simulation: Surviving branches are evaluated by querying the (frozen) target model, generating a response $s_t$9.
• Frontier Advancement: New child nodes become active for the next turn.

This breadth-first expansion allows concurrent evaluation and comparison of multiple attack strategies at each turn, which enhances the discovery of diverse adversarial trajectories (Guo et al., 2 Oct 2025).

3. Policy Optimization: Group Relative Policy Optimization (GRPO)

The optimization objective avoids explicit value-function learning by leveraging per-group relative advantage:

$g$0

where:

• $g$1 is the likelihood ratio between current and previous policies,
• $g$2 is the stepwise group-relative advantage,
• $g$3 is the SFT-initialized reference policy,
• $g$4 tunes KL-regularization strength.

Rollout groups facilitate stable importance-weighted policy updates despite relying entirely on sparse, non-verifiable end-of-episode rewards (Guo et al., 2 Oct 2025).

4. State and Action Encoding, Adaptive Masking

Both states and actions are serialized as token sequences, enabling input to transformer-based generative policies:

• State Serialization: Combines the goal (in special tokens) and the entire prior dialogue history with explicit markers (e.g., <Goal>, <Thinking i>, <Attack question i>).
• Action Generation: The attacker decoder produces the next CoT prompt and query tokens in sequence.
• Pruning Module: Enforces format and topicality constraints, discarding non-conformant outputs during expansion.
• Adaptive Masking: For rollouts with negative group-relative advantage, token-level losses on format tokens are masked to ensure well-formed dialogue generation throughout optimization. This mechanism is critical for reducing collapse into malformed outputs and aids in achieving higher attack success rates (Guo et al., 2 Oct 2025).

5. Experimental Protocol and Quantitative Performance

The framework is evaluated across 10 target models and multiple benchmarks:

• Datasets: Pre-training (SFT) utilizes 397 human/LLM-curated dialogues. RL is trained on 500 unique goals (AdvBench, DangerousQA, CatQA); evaluation is on 200 held-out HarmBench goals.
• Backbone: Llama-3.1-8B-Instruct attacker, 10 diverse targets (five closed-source, five open-source).
• Baselines: GCG, PAIR, TAP, MTSA (SFT+DPO), Jailbreak-R1, AutoDAN-Turbo.
• Metric: Attack Success Rate (ASR), based on GPT-4o Likert-scale scoring.

Model	AutoDAN-Turbo	DialTree-RPO (Ours)
GPT-4o	60.0	86.0
GPT-4.1-mini	65.5	90.0
o3-mini	71.5	86.5
Gemini-2.0	56.5	87.5
Grok-4	38.0	75.0
Avg. (Closed)	58.3	85.0
Avg. (Open)	60.4	85.5

DialTree-RPO achieves an average ASR improvement of 25.9 percentage points over the strongest baseline. Query efficiency exceeds all prior methods, requiring only ~2.8 queries per dialogue to reach peak ASR (Guo et al., 2 Oct 2025).

Ablations demonstrate:

• Tree rollout with pruning increases ASR by ≈9.8 points over linear GRPO.
• Pruning for format and topicality is essential; removing both results in >25 point ASR reduction.
• Adaptive masking is required to prevent collapse to malformed outputs.

6. Strategic Insights, Contributions, and Limitations

DialTree-RPO enables the discovery of previously unseen multi-turn attack strategies within LLM dialogue red-teaming:

• Sequential MDP Formulation: Facilitates long-horizon, compositional attack planning.
• Breadth-First Tree Rollout: Supports exploration of diverse adversarial pathways in parallel.
• Novel Tactics Discovered: Pretexting, gradual escalation, policy-laundering via keywords, persistent gap-filling, and cross-lingual evasion—not found in SFT data.
• Explicit Advantage of GRPO and Adaptive Masking: Stability with sparse, non-verifiable terminal rewards; avoidance of malformed output collapse.

Limitations are recognized:

• Sparse rewards hinder learning for longer dialogues due to delayed feedback.
• Reliance on a proxy classifier $g$5 may permit reward hacking.
• Increased training compute cost due to tree expansion and pruning.
• Defensive adaptation remains an open requirement; improved guardrails and adversarial-robust training must co-evolve with attacker models.

This suggests that DialTree-RPO is both a potent methodology for testing LLM defenses and an analytic lens on emergent adversarial tactics, offering guidance for subsequent safety and robustness research (Guo et al., 2 Oct 2025).