---
title: Multi-Turn Preference Optimization
url: https://www.emergentmind.com/topics/multi-turn-preference-optimization-mtpo-df501921-91c6-4be5-877f-caef74c137b5
type: topic
---

# Multi-Turn Preference Optimization

Multi-Turn Preference Optimization (MTPO) is an advanced framework for aligning language agents and large language models (LLMs) with human or task-specific preferences over extended interactions. Unlike single-turn preference optimization—which focuses on pointwise or isolated decision steps—MTPO operates over multi-step trajectories within Markov Decision Processes (MDPs), directly optimizing for agent behaviors that maximize cumulative reward or preference signals at the trajectory or segment level. This paradigm has catalyzed substantial advances in agent alignment, dialogue control, tool usage, and sequential planning.

## 1. Formalization and Motivation

MTPO is formulated in the context of finite-horizon MDPs, where a policy $\pi_\theta$ generates action sequences $(a_0, \ldots, a_{T-1})$ in response to evolving textual states $(s_0, \ldots, s_{T-1})$. A trajectory $\tau = (s_0, a_0, \ldots, s_{T}, a_{T})$ accumulates a discounted reward $E[\sum_{t=0}^{T-1}\gamma^t r(s_t,a_t)]$ under transition dynamics. The essential challenge is to directly optimize policies using preference data—typically, labeled pairs of full multi-turn trajectories $(\tau^w, \tau^l)$ marked as “win” (preferred) and “lose” (dispreferred)—as a substitute for explicit reward models or hand-crafted feedback.

Classic Direct Preference Optimization (DPO) is provably effective in single-turn or myopic settings, leveraging a Bradley–Terry probabilistic model for pairwise preferences. However, applying DPO naively to multi-turn tasks fails due to two core obstacles:

1. The partition function $Z(s_t)$ in the MaxEnt-optimal policy is state- and step-dependent, thus cannot be cancelled between differing trajectories.
2. Preference trajectory pairs $(\tau^w, \tau^l)$ commonly differ in length ($T_w \ne T_l$), introducing unavoidable additive biases proportional to length disparities, which classical DPO cannot normalize away.

This constrains the expressivity and robustness of single-turn preference optimization in sequential environments, motivating new approaches that properly aggregate and normalize preference signals over whole-agent behaviors [2406.14868].

## 2. Core Methodological Advances and Loss Construction

#### 2.1 Occupancy Measure Regularization

MTPO generalizes DPO by lifting optimization from raw policy distributions to discounted state–action occupancy measures (SAOMs). Instead of per-step policy KL penalties, the regularized RL objective is expressed as:

\[
\max_{\pi_\theta} \mathbb{E}_{(s,a)\sim d^{\pi_\theta}}[r(s,a)] - \beta\, \mathbb{D}_{\mathrm{KL}}[d^{\pi_\theta} \| d^{\pi_{\mathrm{ref}}}]
\]

where $d^{\pi}(s,a)$ is the normalized, discounted occupancy measure, and $\pi_{\mathrm{ref}}$ is a fixed reference policy (e.g., a supervised fine-tuned baseline). Crucially, in the MaxEnt RL framework, the ratio $d^{\pi^*}/d^{\pi_{\mathrm{ref}}}$ eliminates any residual partition function dependence, removing state- and trajectory-length biases [2406.14868].

#### 2.2 Length-normalized Multi-turn Bradley–Terry Model

To convert preference supervision over whole trajectories into a practical loss, MTPO introduces a length-normalized Bradley–Terry likelihood:

\[
p(\tau^w \succ \tau^l) = \sigma\left[ 
  \frac{1-\gamma}{1-\gamma^{T_w}} \sum_{t=0}^{T_w-1} \gamma^t r(s_t^w, a_t^w)
  - \frac{1-\gamma}{1-\gamma^{T_l}} \sum_{t=0}^{T_l-1} \gamma^t r(s_t^l, a_t^l)
\right]
\]

This normalization re-weights each turn’s contribution and ensures equitable comparison across different-length trajectories. The practical implementation then substitutes $r(s,a)$ with its occupancy-measure-based closed form, neutralizing partition function artifacts [2406.14868].

#### 2.3 Closed-Form MTPO Loss and Implementation

The complete MTPO (DMPO) loss is:

\[
\mathcal{L}_{\mathrm{DMPO}} = -\mathbb{E}_{(\tau^w, \tau^l)} \log \sigma \left[
  \sum_{t=0}^{T_w-1} \phi(t,T_w) \frac{\pi_\theta(a_t^w|s_t^w)}{\pi_{\mathrm{ref}}(a_t^w|s_t^w)} -
  \sum_{t=0}^{T_l-1} \phi(t,T_l) \frac{\pi_\theta(a_t^l|s_t^l)}{\pi_{\mathrm{ref}}(a_t^l|s_t^l)}
\right]
\]

with $\phi(t, T) = (1-\gamma^{T-t})/(1-\gamma^T)$ prioritizing early decisions. The optimization can be implemented efficiently using gradient steps over minibatches of trajectory pairs, requiring only likelihood ratios from current and reference models; no supplementary reward model is needed during training [2406.14868]. Pseudocode for a gradient-update is:

```python
for (s0, τw, τl) in minibatch D:
    Vw = sum(φ(t,T_w) * (π_θ(a_t^w|s_t^w)/π_ref(a_t^w|s_t^w)) for t in range(T_w))
    Vl = sum(φ(t,T_l) * (π_θ(a_t^l|s_t^l)/π_ref(a_t^l|s_t^l)) for t in range(T_l))
    loss += -log σ(Vw - Vl)
update θ via ∇_θ loss
```

## 3. Theoretical Properties and Insights

The MTPO (DMPO) formulation yields several rigorous properties:

- **Compounding error mitigation:** The time-dependent reweighting $\phi(t,T)$ decreases with $t$, so earlier state-actions are emphasized. This directly addresses the compounding error problem that plagues naive per-turn objectives.
- **Single-turn reduction:** In the limit $\gamma \to 0$, $\phi(0,T)=1$ and all other weights vanish, so DMPO collapses exactly to the single-turn DPO objective.
- **Adaptive weighting:** The sigmoidal BT likelihood automatically amplifies loss on trajectory pairs where the model is uncertain.
- **Stability:** DMPO is a principled saddle-point approximation to the regularized maximum-entropy RL objective and retains empirical optimization stability. No formal global convergence proof is provided due to nonconvexity, but loss curves are well-behaved in extensive experiments [2406.14868].

## 4. Empirical Evidence and Benchmarks

DMPO’s effectiveness has been extensively validated on MDP-style agent tasks:

- **Datasets:** Evaluated on WebShop (shopping/task completion, [0,1] reward), ScienceWorld (science-experiment execution, [0,1] subgoal reward), and ALFWorld (embodied household tasks, binary completion reward). All environments utilize ReAct-inspired text observation/action schemas.
- **Baselines:** Compared to no tuning, supervised fine-tuning (SFT), best-of-$N$ sampling, rejection-sampling fine-tune (RFT), PPO, ETO (trial-and-error + DPO), and single-turn DPO.
- **Main findings:**
  - In noisy settings (base LlamA-2-7B-Chat, Mistral-7B), DMPO outperforms single-turn DPO by 2–5 points mean reward.
  - In clean settings (Llama-2-7B, WebShop/ScienceWorld) DMPO outperforms ETO, PPO, SFT, and best-of-$N$ (0.701/0.724 vs. 0.698/0.685 for ETO), and surpasses GPT-3.5.
  - Robustness under noisy or length-imbalanced preference data is demonstrated; single-turn DPO deteriorates when “lose” trajectories lengthen, but DMPO remains robust due to explicit normalization.
- **Ablations:** Key hyperparameters ($\gamma$, normalization functions) are probed. Smaller $\gamma$ is favored under noisy data, higher $\gamma$ for cleaner settings. Length normalization is critical for performance stability [2406.14868].

## 5. Specializations and Comparative Methods

Numerous extensions and domain-specific variants of MTPO have emerged:

- **Dialog and Tool-Augmented Settings:** DiaTool-DPO adapts multi-turn DPO with length and margin normalization in tool-augmented language models, demonstrating strong slot-filling and rejection capabilities over baselines [2504.02882].
- **Segment-Level Optimization:** SDPO restricts loss computation to minimal “error” segments in dialogue (as determined by LLMs), dramatically reducing label noise and demonstrating superior performance on social agent benchmarks [2501.01821].
- **Iterative Policy Optimization:** Iterative PPO recasts the multi-turn RLHF problem into an alternation between fitting a multi-turn Q-function and performing single-turn PPO updates, simplifying implementation while ensuring policy improvement [2511.21638].
- **Preference-Guided Molecular Optimization:** In molecular lead optimization, PGPO combines trajectory-level reinforcement learning with intra-trajectory pairwise preference learning, harvesting direct supervision at both granularities and achieving state-of-the-art sample efficiency [2509.21737].
- **Multimodal and Math Agents:** Other works integrate multi-turn preference optimization with multimodal (vision–language) settings [2505.23950] and chain-of-thought math agents [2409.02392], demonstrating consistent improvements over fine-tuning and single-turn preference methods.

A summary table of representative techniques:

| Method         | Domain/Setting    | Key Features                    | Benchmark Gains        |
|----------------|------------------|----------------------------------|-----------------------|
| DMPO [2406.14868] | Language agents   | SAOM normalization, trajectory-level BT | 2–5 points avg. reward |
| DiaTool-DPO [2504.02882] | Tool-augmented LLM | State-mapped DPO, margin, turn-norm | 44% slot accuracy↑    |
| SDPO [2501.01821]      | Social dialogue     | Segment-level loss, expert segmenting | SOTOPIA SOTA         |
| Iterative PPO [2511.21638] | Conversational RL | Q-function reduction, single-turn PPO | 10–15% conv. rate↑   |
| PGPO [2509.21737]     | Chemistry           | Dual-level (traj + intra-traj) pref   | 2.3× best baseline   |

*All values as reported in respective references.*

## 6. Practical Considerations, Limitations, and Future Work

Key practical insights for MTPO deployment include:

- Early decision weighting via $\phi(t,T)$ mitigates downstream error accumulation, promoting expert-like behavior traces.
- Robustness to noisy, imbalanced, and varying-length preference data is a direct benefit of length normalization.
- Minimal infrastructure is required: only instantaneous likelihood ratios are needed; there is no need to train auxiliary reward models.
- Hyperparameters to tune include $\beta$ (policy divergence), $\gamma$ (time discounting), and, for some variants, margin or segment-length settings.
- Principal limitations: Demonstrated predominantly on 7B-parameter models, with reward signals at turn or trajectory granularity. Extending to token-level objectives and scaling to larger models remain open topics.
- Future directions involve richer behavioral granularity (token-level, multi-tool actions), more sophisticated segment selection (via learned or human-in-the-loop processes), multi-agent or adversarial settings, and expansion to further domains such as tool-use, planning, and multi-modal reasoning [2406.14868].

## 7. Significance in Sequential Learning and Agent Alignment

MTPO stands as a theoretically principled and empirically validated framework for aligning agentic LLMs with desired long-horizon, sequential behaviors. By addressing the structural limitations of per-turn objectives, introducing length and trajectory normalization, and generalizing to occupancy measures and segment-level preferences, MTPO achieves both robust theoretical justifications and substantial empirical gains across reinforcement learning tasks, tool-use, and complex agentic domains [2406.14868, 2504.02882, 2501.01821, 2511.21638]. Its architectural and methodological flexibility enables its deployment in diverse agentic systems, providing a scalable foundation for dynamic, user-aligned autonomous agents.

Source: https://www.emergentmind.com/topics/multi-turn-preference-optimization-mtpo-df501921-91c6-4be5-877f-caef74c137b5