---
title: Task-Oriented Utility Dialogue Summarization
url: https://www.emergentmind.com/topics/task-oriented-utility-based-dialogue-summarization
type: topic
---

# Task-Oriented Utility Dialogue Summarization

Task-oriented utility-based dialogue summarization denotes the generation of concise, fluent, and faithful summaries for goal-driven dialogues in which the summary is expected to preserve actionable information such as user intents, constraints, entities, resolutions, and next steps. In this setting, summary quality is not adequately characterized by surface similarity alone: the output must support downstream use in settings such as customer service review, clinical decision making, workflow support, and call-center operations, while remaining factually grounded in dialogues whose critical information is often scattered across turns and revised through negotiation [2210.09894][2110.12680][2409.10070].

## 1. Scope, scenarios, and distinguishing requirements

Task-oriented dialogue summarization occupies the task-oriented branch of abstractive dialogue summarization, in contrast to open-domain summarization. The scenario taxonomy includes customer service, legal, medical care, and official issue domains such as meetings and email threads, all of which require summaries that cover user intents and corresponding solutions rather than only salient conversational snippets [2210.09894]. The requirements are explicitly contextual: in a shopping-chatbot scenario, the summary should highlight user preferences, whereas in a customer call center, the summary must capture problem attributes and the final resolution provided. This domain dependence underlies the observation that one single model does not perform well across various areas for distinct summarization tasks [2402.16986].

The defining constraints of the task are stricter than in generic dialogue summarization. TODSum identifies severe scattered information issues, repetition and negotiation of constraints, multi-domain complexity, and strict factual consistency requirements as central obstacles for summarizing task-oriented dialogues [2110.12680]. The survey literature likewise emphasizes faithfulness, factual completeness, slot or field coverage, and negation correctness as utility-bearing properties that are especially consequential in medical, legal, and customer-service contexts [2210.09894].

A related but distinct formulation is Dialogue Description, introduced in Dial2Desc, which abstracts away from conversational flow to generate a concise description of the object or action being discussed. That objective is explicitly different from task-oriented summarization, which is centered on preserving intents, solutions, and other operationally useful dialogue content [1811.00185].

## 2. Utility signals and representational strategies

A core line of work treats utility as structured semantic content that should be injected into summarization models. TODSum conditions generation on dialogue history $\mathbf{x}$ and dialogue state $\mathbf{s}$, optimizing
$$
\arg \max _{\theta} \sum_{\left\langle\mathbf{x}^{i}, \mathbf{y}^{i}, \mathbf{s}^{i}\right\rangle \in\langle\mathcal{X}, \mathcal{Y}, \mathcal{S}\rangle} \log p\left(\mathbf{y}^{i} \mid \mathbf{x}^{i}, \mathbf{s}^{i} ; \theta\right).
$$
Here, the dialogue state is a structured representation of domains, intents, and slot-value pairs, introduced to enhance faithfulness and factual consistency in summaries [2110.12680].

A more radical reformulation appears in DS2, which hypothesizes that dialogue summaries are essentially unstructured dialogue states. In that framework, a structured state $\mathbf{y}$ is mapped to a synthetic summary $\mathbf{z}=\phi(\mathbf{y})$, and the generated summary is parsed back into a structured state through $\eta$. If the generated output matches the template family, state recovery is exact. The approach therefore uses summarization as a controlled surface form for dialogue state tracking, and its analysis shows that the naturalness of the summary templates plays a key role for successful training [2203.01552].

Recent work has extended utility signals beyond dialogue state to Spoken Language Understanding semantics. In DECODA-based summarization, the summarizer is conditioned on call type and task-relevant named entities. The conditioning procedure, denoted “Pipeline_C,” first predicts the semantic label from the dialogue transcript and then concatenates it to the dialogue:
$$
C = \textrm{intent}(D), \quad S = \textrm{summary}(D, C).
$$
The stated objective is to reduce hallucinations concerning task-related facts such as user intent and named entities, thereby improving semantically faithful summaries for goal-oriented human-human dialogues [2409.10070].

Another representational strategy defines utility directly through downstream question answering. QUARTZ generates multiple summaries and task-oriented question-answer pairs from a dialogue using a pool of LLMs, and then evaluates summaries according to their capacity to answer relevant task-targeted questions. In this framework, utility is not reference similarity but answerability of downstream questions; the selected summaries are then used to fine-tune the best LLM with LoRA in an unsupervised manner [2509.26302].

For long dialogues, rubric-driven summarization has also been advocated. The long-dialog analysis argues for natural language rubrics that guide LLM fine-tuning toward coherent, contextually relevant, and goal-focused summaries, explicitly aligning summary generation with application-specific utility rather than generic informativeness [2402.16986].

## 3. Utility-aligned evaluation and candidate selection

A recurring claim across the literature is that ROUGE and related lexical metrics are insufficient for utility-focused dialogue summarization. DECODA states that ROUGE and BERTScore do not adequately capture faithfulness for utility-focused, task-based dialogue summarization, and introduces task-semantic metrics: CT-Acc for call type alignment and entity-level Precision, Recall, and F1 for named-entity overlap between generated and reference summaries [2409.10070]. TODSum similarly supplements ROUGE with state-aware factual consistency metrics based on exact match of extracted $(\text{domain}, \text{intent}, \text{slot}, \text{value})$ tuples, using Precision, Recall, and F1 [2110.12680].

Candidate selection can itself be utility-driven. In DECODA, the probability distribution over call types predicted from a generated summary is compared with the distribution predicted from the full dialogue using KL divergence,
$$
D_{\mathrm{KL}(G \parallel R)} = \sum_{x \in CT} G(x) \log \left(\frac{G(x)}{R(x)}\right),
$$
and the system also computes the Named Entity Hallucination Risk as the proportion of named entities not present in the source dialogue. The combined criterion first minimizes NEHR and then selects, among those candidates, the one with the smallest $D_{\mathrm{KL}}$, thereby jointly constraining entity hallucination and task-label drift [2409.10070].

QUARTZ extends the evaluation problem into a two-stage LLM ranking procedure. Candidate answers are ranked relative to gold answers, ranking consensus is computed with Kendall tau distance, and responder or summary quality is aggregated with Mean Reciprocal Rank. The selected summary
$$
S_i^* = \arg\max_{l_S} \text{Score}_{i, l_S}
$$
is therefore the one that most consistently supports correct answers to task-oriented questions across multiple models and evaluation permutations [2509.26302].

Several works propose broader utility metrics. The long-dialog analysis introduces Intent-Entity Coverage, defined as Jaccard overlap between the set of intents or entities extracted from the dialogue and those extracted from the summary:
$$
\mathrm{Jaccard}(A,B) = \frac{|A \cap B|}{|A \cup B|}.
$$
The analysis argues that IEC is especially relevant in customer service, chatbots, and other task-driven scenarios where summarization must retain actionable items, intents, or entities critical to utility [2402.16986]. In an industrial sales setting, SumSim aggregates coverage, relevance, informativeness, and factuality into a practical model-selection metric for business-critical summaries [2204.12951]. A later industrial lifecycle study evaluates summaries along three explicit dimensions—Accuracy, Completeness, and Readability—using both expert A/B preference tests and LLM-as-a-Judge scoring closely aligned to the current requirement definitions [2601.08682].

## 4. Datasets, annotation pipelines, and pretraining corpora

Task-oriented utility-based summarization has been shaped as much by dataset construction as by modeling. TODSum introduced a large-scale public Task-Oriented Dialogue Summarization dataset built on MultiWOZ, with 9,906 English dialogues covering 5 domains and averages of 1.9 domains, 3.7 intents, and 12 slots per dialogue [2110.12680]. DECODA, by contrast, targets French spoken call-center conversations and was revised into a standardized suite with call-type and entity annotations, augmented summaries, and automatic transcripts for settings without manual transcripts [2409.10070].

| Resource | Role | Key properties |
|---|---|---|
| TODSum [2110.12680] | Public benchmark for task-oriented summarization with dialogue state | 9,906 dialogues; 5 domains |
| DECODA suite [2409.10070] | Faithfulness benchmark for French call-center summarization | DECODA-v1-hum, DECODA-v2-aug, DECODA-v2-auto |
| OmniDialog pre-training corpus [2312.16864] | Multi-task dialogue pre-training for summarization and related tasks | 7 tasks; 15 datasets; over 3.2 million dialogue utterances |
| QUARTZ evaluation datasets [2509.26302] | Cross-domain validation of utility-driven summarization | SAMSum, DialogSum, MTS-Dialog, SimSAMU |

The data-creation procedures themselves are utility-sensitive. TODSum uses user goals and a state-of-the-art NLG model, SC-GPT, to generate candidate summaries, followed by a human-in-the-loop refinement cycle repeated until annotation accuracy exceeds 95% [2110.12680]. DECODA-v1-hum retains the original manually summarized dialogues, with DECODA-3 providing 200 training, 100 validation, and 200 test examples; DECODA-v2-aug uses ChatGPT-3.5 few-shot prompting to generate additional French synopses for dialogues without human references, increasing the training set to 1390 dialogues; and DECODA-v2-auto extends the corpus with automatic transcriptions generated by WhisperX [2409.10070].

Data scarcity has also been addressed through pseudo-labeling and pretraining. The sales-call system uses GPT-3 offline as a zero-shot labeler to produce pseudo-summaries under privacy constraints, and combines those pseudo-labels with human labels to fine-tune DialogBART [2204.12951]. OmniDialog pre-trains jointly on seven dialogue-focused tasks, drawing from fifteen datasets, and includes MediaSum, DialogSum, and TODSum as summarization-related resources, thereby positioning summarization as one of several mutually informative dialogue tasks [2312.16864]. QUARTZ, in turn, is explicitly reference-free during training selection and only later fine-tunes on self-selected summaries rather than human-written gold summaries [2509.26302].

## 5. Model architectures and system designs

Architecturally, the field spans structured encoder-decoder models, prompt-based multi-task pretraining, agentic pipelines, and industrial hybrid systems. TODSum’s state-aware structured summarization model linearizes the dialogue state, encodes it with a Transformer encoder, and uses a decoder with dual cross-attention that first attends to the dialogue-state representations and then to dialogue-context representations. The model can operate with oracle dialogue state or with dialogue state predicted by a DST model such as TRADE, and it establishes dialogue structure guidance as a direct mechanism for improving summary faithfulness [2110.12680].

OmniDialog generalizes this structured perspective through multi-task learning. Built on T5, it unifies dialogue management, generation, and comprehension tasks into a single prompt-based sequence-to-sequence framework, with every sample cast as `Prompt(z_t) [ Input (x) ] ⇒ Output (y)`. Summarization is treated as a first-class comprehension task, and the model is trained with a maximum input length of 1000 tokens to accommodate long multi-turn dialogues [2312.16864].

Long-input handling has also produced specialized pipeline designs. The long-dialog analysis evaluates direct feeding, two variants of chunk-and-summarize, and extract-then-summarize with SentenceBERT-based sentence extraction. It recommends incremental summarization to retain global context and extractive pre-filtering to maximize informative input under context limits [2402.16986]. In production sales-call summarization, long transcripts that may exceed 50,000 tokens for 45-minute calls are first split into semantically coherent segments using word embeddings; each segment is summarized independently, yielding chronologically ordered abstractive highlights rather than a single paragraph [2204.12951].

The industrial sales-call system further adapts BART into DialogBART by adding speaker embeddings and turn position embeddings, with each token represented by the sum of token, position, speaker, and turn-position embeddings. It supplements generation with a grammatical acceptability model based on perplexity and with a human-in-the-loop editing interface that allows users to add, remove, or edit machine-generated highlights [2204.12951]. A later industrial case study adopts an explicitly agentic workflow composed of a Drafting Agent, three Evaluator Agents for Accuracy, Completeness, and Readability, a Refinement Agent, and a Redundancy Checker, iterated for up to $N$ rounds or until all issues are resolved [2601.08682].

QUARTZ combines generation, evaluation, and adaptation within a single reference-free pipeline. A pool of LLMs generates summaries and gold QA pairs, other LLMs answer those questions from the candidate summaries, evaluators rank the answers and then the summaries, and the best-performing summarizer is fine-tuned with LoRA on the selected outputs. This design decomposes summarization utility into question generation, answerability, ranking consensus, and subsequent self-bootstrapping [2509.26302].

## 6. Empirical findings, applications, and open problems

Empirical results consistently show gains when models are aligned with task semantics rather than evaluated only by lexical overlap. On DECODA, data augmentation with LLM-synthesized summaries improves ROUGE-L from 23.6 to 29.1 and BERTScore from 33.1 to 38.9 relative to human-only training, while automatic transcription with WhisperX Large remains competitive despite a 40% word error rate, reaching ROUGE-L = 27.7 and BERTScore = 37.1. Under the main comparison with BARThez + WhisperX Large + Pipeline_C, CT-Acc rises from 0.76 for the baseline to 0.82 with $\min D_{\mathrm{KL}}$ and with the combined criterion, while NE-F1 rises from 0.43 to 0.46 with $\min$ NEHR [2409.10070].

Dialogue-state guidance produces analogous effects. In TODSum, BART with DS(oracle) improves over BART by +3.06 ROUGE-1, +5.01 ROUGE-2, +3.79 ROUGE-L, and +10.62 factual consistency F1, while gains remain sizable with DS(pred), indicating practical value when state is predicted rather than manually provided [2110.12680]. In DS2, framing dialogue state tracking as summary generation yields multi-domain Joint Goal Accuracy of 36.15/33.76 at 1% data, 45.14/44.20 at 5%, and 47.61/45.38 at 10% on MultiWoZ 2.0/2.1, and the ablation analysis shows that disabling template naturalness can reduce JGA to 13.1 versus 28.3+ [2203.01552].

Large-scale pretraining and utility-aware unsupervised selection also produce strong benchmark results. OmniDialog reports 41.53 ROUGE-1, 15.40 ROUGE-2, and 38.74 ROUGE-L on DialogSum, surpassing PEGASUS at 38.40, 13.84, and 33.41, and it reaches 83.85, 73.19, and 82.99 on TODSum, exceeding BART w. DS(oracle) at 81.12, 67.87, and 78.94 [2312.16864]. QUARTZ reports 61.4 R-1 and 77.9 BERT-Score on SAMSum, compared with 55.9 and 75.7 for MoE, and states that selected summaries using the pool outperform all individuals; fine-tuning on self-selected summaries nearly closes the gap with full supervision, and human or LLM-as-judge evaluation rates QUARTZ summaries on par with or better than references in coherence, consistency, fluency, and relevance [2509.26302].

The principal limitations are also consistent across studies. No single model is best overall, and performance is application- and context-sensitive [2402.16986]. Long dialogues introduce input truncation, context dilution, and recency bias, particularly in prompted conversational LLMs [2402.16986]. In industry, upstream ASR noise is reported as a leading cause of summarization error, with approximately 40% of issues traced to that bottleneck, and prompt portability across LLMs is poor enough to create vendor lock-in [2601.08682]. Surveyed future directions therefore emphasize more domain-specific public datasets, evaluation beyond ROUGE toward automatic task success indicators, robust tracking of user intents and solutions, graph-based and partitioned modeling, personalized or role-specific summaries, and better auto-evaluators for utility components such as clinical fact extractors and customer intent classifiers [2210.09894].

Across these lines of work, the field progressively shifts from summarization as generic compression toward summarization as utility preservation. This suggests that the most durable formulations are those in which intents, entities, states, QA behavior, or evaluator dimensions are made explicit in training, decoding, or evaluation, so that summaries remain actionable under the operational constraints of real task-oriented dialogue systems.

Source: https://www.emergentmind.com/topics/task-oriented-utility-based-dialogue-summarization