---
title: 'DialTree-RPO: Multi-Turn Adversarial RL'
url: https://www.emergentmind.com/topics/dialtree-rpo
type: topic
---

# DialTree-RPO: Multi-Turn Adversarial RL

DialTree-RPO is an on-policy reinforcement learning (RL) framework that augments Group Relative Policy Optimization (GRPO) with breadth-first dialogue tree search and pruning for the discovery of multi-turn adversarial attack strategies against large language models (LLMs). The framework treats multi-turn red-teaming as a sequential decision process, enabling systematic exploration of adversarial dialogue trajectories and outperforming prior methods on benchmarks of attack success. DialTree-RPO combines structured exploration, adaptive masking, policy optimization, and custom state-action encoding to induce sophisticated jailbreak tactics that exploit multi-turn vulnerabilities in both closed- and open-source LLMs [2510.02286].

## 1. Formal Problem Definition and MDP Formulation

DialTree-RPO frames the interaction between an attacker and a frozen target LLM as a fully observed, episodic Markov decision process (MDP) $\mathcal{M} = (\mathcal{S}, \mathcal{A}, T, R, \gamma)$, where:

- **State space** $(\mathcal{S})$: Each dialogue state $s_t$ consists of a fixed goal $g$ (e.g., jailbreak objective) and the dialogue history as a sequence of triplets $(c_i, q_i, r_i)$ for $i \leq t$, where $c_i$ is a chain-of-thought prompt, $q_i$ is the attack query, and $r_i$ is the target response.
- **Action space** $(\mathcal{A})$: At each turn $t$, the action $a_t = (c_t, q_t)$, generated by the attacker’s policy $\pi_\theta(\cdot | s_{t-1})$, specifies the next CoT and query components.
- **Transition dynamics** $(T)$: The environment deterministically extends the state with the action and the sampled response $r_t \sim \pi_{\mathrm{tgt}}(\cdot | q_t, \text{history})$.
- **Reward function** $(R)$: Rewards are sparse and terminal, given by a safety-guard classifier $r_\phi$ (HarmAug-Guard), providing $R(\tau) = 1$ if any $r_\phi(g; q_t, r_t) > \eta$ within the trajectory $\tau$, and $0$ otherwise.
- **Discount** $(\gamma)$: Set to $1$; only the final reward determines policy update.

This construction enables the adversarial attacker policy $\pi_\theta$ to maximally exploit target vulnerabilities by planning over multi-turn conversational interactions [2510.02286].

## 2. Dialogue Tree Rollout and Exploration

Contrasting standard linear trajectory rollouts, DialTree-RPO utilizes a dialogue tree with branching factor $n$ and depth up to $T_{\max}$:

- **Tree Breadth and Depth**: At each turn, every frontier node is expanded by sampling $n$ candidate actions from $\pi_\theta$.
- **Pruning**: Candidate branches are immediately pruned if the quality checker $Q$ detects malformed outputs or off-topic queries.
- **Simulation**: Surviving branches are evaluated by querying the (frozen) target model, generating a response $r_{t,j}$.
- **Frontier Advancement**: New child nodes become active for the next turn.

This breadth-first expansion allows concurrent evaluation and comparison of multiple attack strategies at each turn, which enhances the discovery of diverse adversarial trajectories [2510.02286].

## 3. Policy Optimization: Group Relative Policy Optimization (GRPO)

The optimization objective avoids explicit value-function learning by leveraging per-group relative advantage:

\[
\mathcal{J}_{\mathrm{GRPO}}(\theta) = \mathbb{E}_{g,\{\tau^{(i)}\} \sim \pi_{\theta_{\mathrm{old}}}} \Biggl[ \frac{1}{G} \sum_{i=1}^G \frac{1}{|\tau^{(i)}|} \sum_{t=1}^{|\tau^{(i)}|} \min\bigl(r_t^{(i)} \hat{A}_t^{(i)}, \mathrm{clip}(r_t^{(i)}, 1-\varepsilon, 1+\varepsilon) \hat{A}_t^{(i)}\bigr) \Biggr] - \beta D_{\mathrm{KL}}[\pi_\theta \| \pi_{\mathrm{ref}}]
\]

where:
- $r_t^{(i)}$ is the likelihood ratio between current and previous policies,
- $\hat{A}_t^{(i)}$ is the stepwise group-relative advantage,
- $\pi_{\mathrm{ref}}$ is the SFT-initialized reference policy,
- $\beta$ tunes KL-regularization strength.

Rollout groups facilitate stable importance-weighted policy updates despite relying entirely on sparse, non-verifiable end-of-episode rewards [2510.02286].

## 4. State and Action Encoding, Adaptive Masking

Both states and actions are serialized as token sequences, enabling input to transformer-based generative policies:

- **State Serialization**: Combines the goal (in special tokens) and the entire prior dialogue history with explicit markers (e.g., `<Goal>`, `<Thinking i>`, `<Attack question i>`).
- **Action Generation**: The attacker decoder produces the next CoT prompt and query tokens in sequence.
- **Pruning Module**: Enforces format and topicality constraints, discarding non-conformant outputs during expansion.
- **Adaptive Masking**: For rollouts with negative group-relative advantage, token-level losses on format tokens are masked to ensure well-formed dialogue generation throughout optimization. This mechanism is critical for reducing collapse into malformed outputs and aids in achieving higher attack success rates [2510.02286].

## 5. Experimental Protocol and Quantitative Performance

The framework is evaluated across 10 target models and multiple benchmarks:

- **Datasets**: Pre-training (SFT) utilizes 397 human/LLM-curated dialogues. RL is trained on 500 unique goals (AdvBench, DangerousQA, CatQA); evaluation is on 200 held-out HarmBench goals.
- **Backbone**: Llama-3.1-8B-Instruct attacker, 10 diverse targets (five closed-source, five open-source).
- **Baselines**: GCG, PAIR, TAP, MTSA (SFT+DPO), Jailbreak-R1, AutoDAN-Turbo.
- **Metric**: Attack Success Rate (ASR), based on GPT-4o Likert-scale scoring.

|  Model         | AutoDAN-Turbo | DialTree-RPO (Ours) |
|----------------|---------------|---------------------|
| GPT-4o         | 60.0          | 86.0                |
| GPT-4.1-mini   | 65.5          | 90.0                |
| o3-mini        | 71.5          | 86.5                |
| Gemini-2.0     | 56.5          | 87.5                |
| Grok-4         | 38.0          | 75.0                |
| Avg. (Closed)  | 58.3          | 85.0                |
| Avg. (Open)    | 60.4          | 85.5                |

DialTree-RPO achieves an average ASR improvement of 25.9 percentage points over the strongest baseline. Query efficiency exceeds all prior methods, requiring only ~2.8 queries per dialogue to reach peak ASR [2510.02286].

Ablations demonstrate:
- Tree rollout with pruning increases ASR by ≈9.8 points over linear GRPO.
- Pruning for format and topicality is essential; removing both results in >25 point ASR reduction.
- Adaptive masking is required to prevent collapse to malformed outputs.

## 6. Strategic Insights, Contributions, and Limitations

DialTree-RPO enables the discovery of previously unseen multi-turn attack strategies within LLM dialogue red-teaming:

- **Sequential MDP Formulation**: Facilitates long-horizon, compositional attack planning.
- **Breadth-First Tree Rollout**: Supports exploration of diverse adversarial pathways in parallel.
- **Novel Tactics Discovered**: Pretexting, gradual escalation, policy-laundering via keywords, persistent gap-filling, and cross-lingual evasion—not found in SFT data.
- **Explicit Advantage of GRPO and Adaptive Masking**: Stability with sparse, non-verifiable terminal rewards; avoidance of malformed output collapse.

Limitations are recognized:
- Sparse rewards hinder learning for longer dialogues due to delayed feedback.
- Reliance on a proxy classifier $r_\phi$ may permit reward hacking.
- Increased training compute cost due to tree expansion and pruning.
- Defensive adaptation remains an open requirement; improved guardrails and adversarial-robust training must co-evolve with attacker models.

*This suggests* that DialTree-RPO is both a potent methodology for testing LLM defenses and an analytic lens on emergent adversarial tactics, offering guidance for subsequent safety and robustness research [2510.02286].

Source: https://www.emergentmind.com/topics/dialtree-rpo