Beyond Overlap Metrics: Rewarding Reasoning and Preferences for Faithful Multi-Role Dialogue Summarization

Abstract: Multi-role dialogue summarization requires modeling complex interactions among multiple speakers while preserving role-specific information and factual consistency. However, most existing methods optimize for automatic metrics such as ROUGE and BERTScore, which favor surface-level imitation of references rather than genuine gains in faithfulness or alignment with human preferences. We propose a novel framework that couples explicit cognitive-style reasoning with reward-based optimization for multi-role dialogue summarization. Our method first distills structured reasoning traces (e.g., step-by-step inferences and intermediate reflections) from a large teacher model and uses them as auxiliary supervision to initialize a reasoning-aware summarizer via staged supervised fine-tuning. It then applies GRPO with a dual-principle reward that blends metric-based signals with human-aligned criteria targeting key information coverage, implicit inference, factual faithfulness, and conciseness. Experiments on multilingual multi-role dialogue benchmarks show that our method matches strong baselines on ROUGE and BERTScore. Specifically, results on CSDS confirm the framework's stability in semantic consistency, while in-depth analysis on SAMSum demonstrates clear gains in factual faithfulness and model-based preference alignment. These findings underscore the value of reasoning-aware and preference-aware training for reliable dialogue summarization. Checkpoints and datasets are available at https://huggingface.co/collections/NebulaPixel/summorchestra-multirole-summary.

• The paper introduces a framework that integrates cognitive-style reasoning with dual-principle rewards to improve faithfulness in multi-role dialogue summarization.
• It employs a teacher–student approach for structured reasoning distillation and composite rewards combining ROUGE, BERTScore, and principle-based evaluations.
• Experimental results on CSDS and SAMSum show enhanced factual consistency and human preference alignment, indicating potential for broader applications.

Rewarding Reasoning and Preferences for Faithful Multi-Role Dialogue Summarization

Motivation and Critique of Existing Paradigms

Large-scale dialogue summarization in multi-role, multilingual and multi-party settings remains impeded by superficial optimization objectives and the tendency of neural summarizers to hallucinate or misattribute information, especially as conversational complexity increases. Traditional training regimens targeting n-gram overlap metrics—most notably ROUGE and BERTScore—systematically reward surface imitation of references. This paradigm is decoupled from actual semantic faithfulness, omits speaker intent, and fails to cope with the implicit and informal structure of real-world dialogues. Abstractive models, especially those based on standard SFT or vanilla RLHF, lack explicit mechanisms to enforce logical consistency or human preference alignment in these contexts.

Framework: Cognitive-Style Reasoning and Dual-Principle Reward

The core methodological advance is an explicit integration of cognitive-style reasoning signals into supervised and reinforcement learning objectives for multi-role dialogue summarization. The proposed pipeline Figure 1 features two primary modules: (i) structured reasoning distillation via a teacher–student approach, and (ii) a dual-principle reward for group-based policy optimization.

Figure 1: The pipeline consists of reasoning distillation to extract thinking traces and a two-stage training procedure combining reasoning awareness and preference supervision.

During data construction, a high-capacity teacher LLM (e.g., DeepSeek-v3) generates stepwise reasoning traces capturing speaker intent, core content extraction, motivational inference, and consistency checking for each dialogue instance. These traces complement existing reference summaries, forming a richer, reasoning-augmented supervision corpus.

In stage one, a student model is initialized through sequential SFT—first on original dialogue-summary pairs, then on the reasoning-augmented data—imparting structured inference procedures directly into the summarizer. In stage two, GRPO (Group Relative Policy Optimization) is employed, leveraging a composite reward composed of: (1) reference-based metrics (ROUGE, BERTScore), (2) length normalization, and (3) a principle-driven reward assessing key information coverage, implicit inference, faithfulness, and concise style. The principle reward, computed with GPT-4 via the RewardAnything framework and weighted to bias the model towards factual and human-aligned content, enforces multidimensional quality criteria. This dual-principle approach guides both the intermediate reasoning and the summary generation phase, operationalizing expert-inspired evaluation desiderata.

Evaluation Methodology and Datasets

Evaluation is conducted on two distinct benchmarks: CSDS (a Chinese customer service multi-party dataset with rich role-oriented and topic-structured dialogues) and SAMSum (informal, short English messenger-style conversations). Metrics encompass ROUGE (tokenized with THULAC for Chinese and py-rouge for English), BERTScore (language-matched BERT backbones), model-based factual consistency (faithfulness as the proportion of supported atomic claims, detected by HHEM-2.1-Open), and model-based human preference (WorldPM-72B-RLHFLOW). This evaluation suite is robust to the limitations of overlap-based metrics and provides both semantic and practical assessments of summary utility.

Figure 2: Sampled dialogue scenarios within the CSDS dataset highlight the diversity and complexity of multi-role interactions.

Experimental Findings: Ablative and Aggregate Results

The SFT+GRPO training procedure is validated via several controlled comparisons on both datasets. Key findings include:

• On CSDS, the Qwen2.5 models trained with sequential SFT and GRPO variants achieve up to 59.05/34.25/50.05 in ROUGE-1/2/L with accompanying BERTScore improvements when BERTScore, ROUGE, and length rewards are jointly optimized. The results match or outperform strong instruction-tuned LLMs, with surface metrics saturating at the SFT+GRPO plateau. Omission of the principle reward (due to evaluator limitations in Chinese) induces an upper bound on factual optimization.
• On SAMSum, the principle reward adds substantial value. Notably, incorporating it (BRLP reward group) reduces explicit ROUGE/BERTScore slightly but produces marked increases in faithfulness (to 0.8352) and model-based human preference alignment (to 0.6683), surpassing both baseline models and ground-truth references. This result emphasizes a trade-off: metric-based rewards alone can saturate, but human-aligned factuality responds directly to principle-based supervision.

Figure 3: Preference alignment and factual faithfulness gains on SAMSum, particularly pronounced in the presence of principle-driven reward optimization.

Consistent performance across resampling trials signals generalization and stability, with GRPO’s group-based approach mitigating both overfitting and length exploitation issues.

Theoretical and Practical Implications

This work demonstrates that explicit reasoning distillation and preference shaping—beyond traditional overlap and semantic similarity objectives—produce more reliable, faithful, and preferred summaries in multi-role dialogue scenarios. The dual-principle reward framework can be interpreted as a structured information bottleneck, constraining LLMs not only with n-gram or latent-space alignment, but with high-level process supervision that directly reflects human usage criteria. The results empirically validate the hypothesis that simple metric maximization does not imply faithfulness or utility in conversational summarization; preference- and principle-driven RL is indispensable for this alignment.

From an applied perspective, the findings advocate for future LLM summarization pipelines in sensitive or high-value settings (e.g., compliance, medical, or legal dialogue summarization) to incorporate explicit reasoning and multi-objective reward shaping. Furthermore, the methodology is extendable: the reasoning distillation and reward balancing protocol is not model- or domain-specific and generalizes to other structured NLG tasks where faithfulness and preference alignment are critical.

Limitations and Future Directions

The study’s limitations include restricted application of principle rewards on non-English corpora due to the absence of robust factual consistency and preference evaluators (e.g., for CSDS, only surface and semantic metrics can be reliably computed). Dataset scale (particularly for SAMSum) and topical diversity remain modest, and the effectiveness of reasoning distillation across broader domains and more complex dialogue structures merits further investigation. The introduced reflection signals also increase computational costs, highlighting the necessity for efficient instantiation, distillation, or pruning in deployment scenarios.

Future work should focus on extending evaluator capacity to additional languages, constructing larger and more representative multi-role benchmarks, and reducing computational overhead through optimized reasoning representation learning. There is significant potential to adapt the framework to other open-ended generation tasks where factual reliability and alignment with nuanced user demands are dominant.

Conclusion

This work establishes that reasoning-aware and preference-aligned reward optimization is crucial for reliable multi-role dialogue summarization. By explicitly supervising intermediate inference pathways and applying principle-centric rewards, the proposed framework achieves parity on conventional metrics and surpasses baselines in factual faithfulness and preference alignment. The results underline the inadequacy of surface-centric metrics in isolation and set a concrete precedent for multidimensional reward engineering in conversational NLG.