Turn-Level Preference Sampling

• Turn-Level Preference Sampling is a methodology that annotates each interaction turn to extract dense, temporally-localized feedback signals.
• It employs strategies like ECPO, IterChat, and PGPO to generate pairwise comparisons and optimize intermediate outputs efficiently.
• The approach enhances sample efficiency and reduces annotation overhead, with empirical gains in metrics such as EM and QED across various domains.

Turn-Level Preference Sampling is a set of methodologies for extracting, modeling, and utilizing fine-grained feedback signals within each turn of an iterative or multi-turn system—such as dialogue agents, reinforcement learning environments, or molecular optimization pipelines. Unlike trajectory-level or final-outcome supervision, turn-level preference sampling directly annotates, generates, or compares intermediate outputs or states, yielding dense, temporally-localized preference information. This has enabled substantial advances in sample efficiency, interpretability, and outcome alignment across diverse domains including conversational recommendation, dialogue system preference extraction, and multi-turn reinforcement learning for optimization tasks.

1. Formal Definitions and Paradigms

Turn-level preference sampling constructs datasets or learning signals where each unit corresponds to a single interaction (“turn”) within a sequential process. In dialogue systems, this can refer to the tuple (dialogue context, agent response, user feedback) at turn $t$; in reinforcement learning, to intermediate states or actions and associated comparative rewards.

For conversational recommendation, a formal episode is: $$H^T = \bigl\{u_0,\,(cr_1,p_1,u_1),\dots,(cr_T,p_T,u_T)\bigr\}$$ with $p_t$ as the agent’s response at turn $t$ and $r_t$ as a per-turn scalar reward, typically assigned via an external rubric or simulated user (Feng et al., 17 Jun 2025). In molecular optimization, the objects of comparison are intermediate solutions (e.g., molecules $m_t$), each associated with a reward $r_t$ and included in inter-turn pairwise preference sets (Wang et al., 26 Sep 2025). For slot-based task-oriented dialogue, turn-level preference sampling may operate over slot-value assignment tuples, representing state transitions elicited at each turn (Wang et al., 3 Aug 2025).

Key objective functions include direct preference-loss formulations: $$\max_\theta\,\mathbb{E}_{(s_t,p_t,\tilde p_t)\sim D_\text{pre}} \Bigl[\log\pi_\theta(\tilde p_t|s_t) - \log\pi_\theta(p_t|s_t)\Bigr]$$ and the Bradley–Terry probabilistic model for pairwise comparisons: $$P_\theta(m_t^i \succ m_t^j) = \frac{\exp(u_\theta(m_t^i))}{\exp(u_\theta(m_t^i))+\exp(u_\theta(m_t^j))}$$ where $u_\theta(.)$ is a model score (Wang et al., 26 Sep 2025).

2. Methodological Variants and Sampling Strategies

Three salient paradigms illustrate the diversity of turn-level preference sampling:

• Expectation Confirmation Preference Optimization (ECPO): Implements turn-level preference sampling by simulating user feedback on agent outputs per turn, scoring responses on a multidimensional rubric (flexibility, coherence, guidance), and collecting unsatisfactory cases for targeted rewriting and preference pair formation (Feng et al., 17 Jun 2025).
• IterChat Data Generation: Operates via stochastic sampling of preference slots to determine per-turn updates in dialogue, then pairs each turn’s historical state with the newly introduced preferences to generate (context, one-turn dialogue, state gain, updated preference) records (Wang et al., 3 Aug 2025).
• Preference-Guided Policy Optimization (PGPO) in POLO: Samples all feasible intra-trajectory pairs of intermediate solutions (e.g., molecules in lead optimization), exploits oracle rewards to form dense pairwise preference signals, and applies an importance-weighted Bradley–Terry loss over these pairs (Wang et al., 26 Sep 2025).

The sampling mechanics differ. ECPO leverages a threshold $\lambda$ on the per-turn reward to define positive (improved) and negative (unsatisfactory) responses. IterChat uses Bernoulli or uniform subset sampling over slots per turn, ensuring at least one slot modification per turn to drive preference extraction. PGPO enumerates or truncates all intra-trajectory pairs where reward differentials exist, typically keeping the top $K$ by magnitude for computational efficiency.

3. Construction and Utilization of Turn-Level Preference Datasets

The principal process for constructing turn-level preference datasets is outlined in the following steps:

1. Per-Turn Evaluation: Each system output at time $t$ (response, molecule, etc.) is assigned a reward, rubric score, or labeled preference.
2. Negative Example Mining and Rewriting: For responses below a threshold or with low reward, explicit rewrites or alternate outputs are generated, providing positive examples for preference optimization.
3. Pairwise Comparison Generation: In some domains (notably POLO), all feasible intra-trajectory pairs $((m_t^i, m_t^j), r_i, r_j)$ are constructed where $r_j > r_i$.
4. Contextual Conditioning: Historical context is explicitly included (prior slot-values in dialogue, preceding design steps in optimization) to maintain turn-level locality (Wang et al., 3 Aug 2025).

Preference-optimization objectives (e.g., Direct Preference Optimization loss, weighted Bradley–Terry likelihood) are then used to fine-tune the system so as to maximize the probability of preferred outcomes in a per-turn fashion.

4. Sample Efficiency, Annotation Overhead, and Theoretical Implications

A distinguishing characteristic of turn-level preference sampling is its high sample efficiency relative to both trajectory-level reward methods (e.g., RLHF) and full-tree rollout strategies. For instance, ECPO reduces LLM oracle calls asymptotically to $\mathcal{O}(N)$, where $N$ is number of turns, in contrast to $\mathcal{O}(MCT)$ for tree-based multi-turn preference optimization (MTPO), with $C$ candidates per node and $M$ simulated episodes (Feng et al., 17 Jun 2025). POLO’s PGPO, by reusing intra-episode pairs, amplifies learning signals from $O(N)$ (RL) to $O(NT^2)$, dramatically increasing sample-efficiency under tight evaluation constraints (Wang et al., 26 Sep 2025).

In annotation, turn-level schemes that tightly localize updates (as in IterChat) show superior annotation time and label accuracy. For 288 samples, IterChat yielded a 28.4% reduction in annotation time and an 11.0 point EM gain over multi-turn baselines (Wang et al., 3 Aug 2025).

5. Domain-Specific Implementations

Conversational Recommendation Agents

• ECPO Framework: Combines per-turn reward modeling using an LLM-based simulator (AILO) to generate both naturalistic user utterances and scalar feedback, with backward “bed” rewriting to improve unsatisfactory turns.
• Optimization Loop: Consists of initial supervised fine-tuning, simulated multi-turn sampling, per-turn reward evaluation and rewriting, and preference-based alignment via DPO (Feng et al., 17 Jun 2025).

Dialogue Preference Extraction

• IterChat Method: Encodes each dialogue as a sequence of (history state, one-turn change, user query, agent response), with preference transitions sampled explicitly and context conditioned for each turn. This modular construction mitigates annotation inconsistencies and enables higher-fidelity preference extraction (Wang et al., 3 Aug 2025).

Reinforcement Learning for Optimization

• POLO/PGPO Algorithm: Extracts turn-level preference signals by forming all intra-trajectory pairs for which reward rankings differ, and optimizes a weighted Bradley–Terry likelihood (with LambdaMART-inspired weights) in combination with a standard PPO objective (Wang et al., 26 Sep 2025).

6. Empirical Performance and Comparative Evaluation

Empirical results consistently show that turn-level preference sampling delivers superior effectiveness and efficiency. Notable findings include:

Domain	System	Key Metric	Turn-Level vs. Baseline
Dialogue Rec.	ECPO	WR↑ to 0.55–0.63, SR↑	Outperforms SFT/KTO (WR≲0.40) (Feng et al., 17 Jun 2025)
Pref. Extraction	IterChat	EM=78.4%, F1=93.6%	Beats multi-turn EM=42.7%, F1=89.8% (Wang et al., 3 Aug 2025)
Optimization	POLO/PGPO	QED↑ 91%, Multi↑ 97%	Outperforms PPO by 8–22% depending on λ_pref (Wang et al., 26 Sep 2025)

In all cited domains, turn-level preference sampling magnifies the effective learning signal per episode, resulting in higher task success rates, denser supervision, and lower computational and annotation costs than trajectory-level or tree-simulation baselines.

7. Design Choices, Hyperparameters, and Scaling Considerations

Successful turn-level preference sampling depends on domain- and task-specific design decisions:

• Preference scoring schemes: scalar vs. multidimensional rubrics, thresholds for negative sample selection.
• Sampling probability and granularity: per-slot Bernoulli (dialogue) or top-$K$ reward pairs (RL).
• Reweighting and objective balancing: choice of importance weights (e.g., LambdaMART in PGPO), preference-objective weight $\lambda_\mathrm{pref}$
• Context encoding: inclusion of full interaction, slot-state, and policy-trace in prompt or trajectory.

A key empirical result is that tuning $\lambda_\mathrm{pref}$ in PGPO from 0.1 to 0.5 consistently yields 8–22% improvement relative to PPO-only; removing turn-level loss leads to significant degradation (e.g., QED rate dropping from 91% to 83%) (Wang et al., 26 Sep 2025). Similar hyperparameter tuning (e.g., rewriting threshold $\lambda$, generation temperature $\tau$) significantly affects annotation efficiency and accuracy in dialogue preference extraction (Wang et al., 3 Aug 2025).

• "Expectation Confirmation Preference Optimization for Multi-Turn Conversational Recommendation Agent" (Feng et al., 17 Jun 2025)
• "Enhancing the Preference Extractor in Multi-turn Dialogues: From Annotating Disasters to Accurate Preference Extraction" (Wang et al., 3 Aug 2025)
• "POLO: Preference-Guided Multi-Turn Reinforcement Learning for Lead Optimization" (Wang et al., 26 Sep 2025)