SDPO: Segment-Level Direct Preference Optimization for Social Agents (2501.01821v2)

Abstract: Social agents powered by LLMs can simulate human social behaviors but fall short in handling complex social dialogues. Direct Preference Optimization (DPO) has proven effective in aligning LLM behavior with human preferences across various agent tasks. However, standard DPO focuses solely on individual turns, which limits its effectiveness in multi-turn social interactions. Several DPO-based multi-turn alignment methods with session-level data have shown potential in addressing this problem.While these methods consider multiple turns across entire sessions, they are often overly coarse-grained, introducing training noise, and lack robust theoretical support. To resolve these limitations, we propose Segment-Level Direct Preference Optimization (SDPO), which dynamically select key segments within interactions to optimize multi-turn agent behavior. SDPO minimizes training noise and is grounded in a rigorous theoretical framework. Evaluations on the SOTOPIA benchmark demonstrate that SDPO-tuned agents consistently outperform both existing DPO-based methods and proprietary LLMs like GPT-4o, underscoring SDPO's potential to advance the social intelligence of LLM-based agents. We release our code and data at https://github.com/AlibabaResearch/DAMO-ConvAI/tree/main/SDPO.

• The paper introduces SDPO as a method to refine social dialogues by targeting critical segments instead of entire sessions or individual turns.
• The methodology minimizes training noise by isolating and optimizing key interaction segments, balancing precision and context.
• Evaluation on the SOTOPIA benchmark demonstrates that SDPO achieves higher goal and relationship scores compared to traditional DPO strategies and GPT-4 models.

Segment-Level Direct Preference Optimization for Social Agents: An In-depth Analysis

The paper "SDPO: Segment-Level Direct Preference Optimization for Social Agents" aims to address the inherent challenges in aligning LLMs for social intelligence tasks, specifically in complex, goal-oriented social dialogues. By introducing a novel approach named Segment-Level Direct Preference Optimization (SDPO), the authors seek to bridge the gap between excessively fine-grained turn-level methods and the overly coarse-grained session-level approaches.

Overview of Current Approaches and Limitations

Currently, Direct Preference Optimization (DPO) based alignment methods are paradigmatic in refining LLM behavior to resonate with human preferences. These methods operate at two main granularity levels: turn-level and session-level. Turn-level DPO, although effective in optimizing individual conversational turns, fails to capture long-term interaction dynamics essential for achieving goals in social contexts. Conversely, session-level methods such as Extended Trajectory Optimization (ETO) and Direct Multi-Turn Preference Optimization (DMPO) extend the optimization to entire sessions. However, these session-level approaches often introduce noise due to their lack of precision in targeting the critical interaction segments influencing outcome quality.

Introduction of SDPO

The SDPO method focuses on a segment-level granularity, identifying and optimizing key segments of interaction that contribute to the desired behavior. This strategy reduces the training noise associated with session-level approaches by isolating erroneous parts of interactions and concentrating optimization efforts on these segments. Conceptually, SDPO strikes a balance between the granularity of turn- and session-level approaches, offering flexibility and accuracy in modeling complex dialogues.

Evaluation and Results

Evaluation on the SOTOPIA benchmark demonstrates SDPO-enhanced agents outperform both existing DPO-based methods and proprietary LLMs, such as those from OpenAI's suite including GPT-4o. The results underscore SDPO's effectiveness in enhancing the social intelligence of LLMs. Specifically, SDPO yields better goal and relationship scores across various scenarios, substantiating its alignment efficiency.

Theoretical Insights and Novelty

The deviation from traditional methods is underpinned by a comprehensive analytical framework that revises standard DPO objectives to accommodate segment-level preferences. By ensuring error and segment alignments, SDPO provides a more realistic opportunity for learning socially intelligent behaviors compared to out-of-context session-level alignments.

Implications for Social Intelligence in AI

The practical and theoretical implications of this approach are profound. Firstly, by enhancing the efficiency of LLMs in social dialogues, SDPO opens pathways for more sophisticated human-agent collaborations. Theoretically, it provides a robust framework for future research aiming to enhance multi-turn interaction agents, with potential extensions to other complex AI scenarios such as negotiation agents and cooperative task-solving systems.

Future Prospects

Encouragingly, the paper hints at the applicability of SDPO beyond the social intelligence domain, suggesting a promising avenue for future exploration and refinement. This positions SDPO as a versatile tool for improving agent interaction quality across various LLM applications.

The introduction of SDPO marks a significant refinement in the toolkit available for optimizing agent behavior in complex interaction scenarios, emphasizing the importance of segment-level accuracy in preference optimization. The paper is a step forward in aligning LLM capabilities with nuanced human social behaviors, setting a new standard for subsequent research in this pervasive area of AI development. The authors offer the community a robust and effective approach with potential implications for a wide range of AI and human-computer interaction applications.