DEAM: AMR-based Dialogue Coherence

• DEAM is a trainable dialogue coherence metric that leverages AMR-based semantic manipulations to generate natural incoherent dialogue samples.
• It employs a RoBERTa encoder with a binary classifier to quantitatively assess dialogue coherence, outperforming heuristic text-level baselines.
• AMR manipulations—including contradiction, coreference inconsistency, irrelevancy, and decreased engagement—enable precise simulation of real chatbot failures.

DEAM, short for "Dialogue coherence Evaluation using AMR-based Semantic Manipulations" (Ghazarian et al., 2022), is a trainable, conversation-level coherence metric for open-domain dialogue systems. It casts coherence evaluation as a binary classification problem: given a full dialogue $x$, predict whether it is coherent ($y=1$) or incoherent ($y=0$). Its distinguishing feature is the use of Abstract Meaning Representation (AMR) to apply semantic-level manipulations for incoherent data generation, rather than relying on heuristic text-level manipulations such as utterances shuffling or inserting random utterances. By parsing each dialogue into AMR graphs, perturbing those graphs to inject coreference inconsistency, irrelevancy, contradictions, and decrease engagement, and then back-translating the manipulated AMRs into text, DEAM produces more natural incoherent samples and scores dialogues with $\mathrm{Coherence}(x)=s(x)=P(y=1\mid x)\in(0,1)$ (Ghazarian et al., 2022).

1. Problem setting and design rationale

Automatic evaluation metrics are essential for the rapid development of open-domain dialogue systems because they facilitate hyper-parameter tuning and comparison between models. Recently proposed trainable conversation-level metrics have shown encouraging results, but their quality is strongly dependent on the quality of training data. Prior works mainly resort to heuristic text-level manipulations to bootstrap incoherent conversations from coherent dialogues. DEAM is motivated by the claim that such approaches are insufficient to appropriately reflect the incoherence that occurs in interactions between advanced dialogue models and humans (Ghazarian et al., 2022).

The framework therefore relocates negative-sample generation from surface form to semantics. AMRs naturally facilitate the injection of various types of incoherence sources at the semantic level, and the paper identifies four such sources: coreference inconsistency, irrelevancy, contradictions, and decrease engagement. A plausible implication is that the method is not only creating corrupted dialogues, but also controlling the type of corruption in a way that is closer to observed chatbot failures.

2. Formalization as a coherence classifier

DEAM defines a training set $\mathcal{D}=\{(x_i,y_i)\}_{i=1}^N$, where $y_i\in\{0,1\}$. A RoBERTa encoder $f_\theta$ plus a single linear classification layer $W\in\mathbb{R}^{2\times d}$ with bias $b$ produces logits

$$\ell_i = W\,h_i + b,\quad \text{where } h_i = f_\theta(x_i)\in\mathbb{R}^{d}.$$

Applying softmax yields

$y=1$0

Training minimizes the binary cross-entropy loss

$y=1$1

At test time, a new dialogue is scored directly by $y=1$2, which functions as the coherence score (Ghazarian et al., 2022).

This formulation places DEAM in the class of supervised, trainable evaluation metrics, but the novelty lies less in the classifier than in the construction of the negative class. The paper’s central claim is that the representation of incoherence in the training distribution largely determines downstream correlation with human judgments.

3. AMR-based semantic manipulations

Given a coherent human-human dialogue, DEAM proceeds utterance by utterance: each utterance is parsed into a root-labelled AMR graph, one or more semantic manipulations are sampled, the manipulations are applied at the AMR graph level, and the manipulated AMR is back-translated into text via a pretrained AMR-to-text T5 model (Ghazarian et al., 2022).

Manipulation type	AMR-level operation	Resulting incoherence
Contradiction	Negate polarity or replace a concept with its antonym	Contradictions
Coreference inconsistency	Replace a pronoun or noun with a different entity from elsewhere in the conversation	The referent no longer makes sense
Irrelevancy	Replace AMR nodes with nodes drawn from other dialogues	Fluent yet off-topic utterances
Engagement decrease	Delete questions, drop the most detailed utterance, or prune lower-level ARG/op children	Decrease engagement

In the contradiction module, a subgraph $y=1$3 such as a predicate and its arguments is identified, then its polarity is negated by adding a “-” polarity node or a concept is replaced with its antonym using ConceptNet relations like Antonym and NotCapableOf. The paper gives the AMR fragment $y=1$4 becoming $y=1$5, corresponding to “I am happy” becoming “I am not happy.”

Coreference inconsistency is generated by collecting all coreference slot fillers, including pronouns and named entities, from the dialogue AMRs and randomly replacing a pronoun or noun in one utterance with a different entity from elsewhere in the conversation. The AMR-to-text model re-regenerates grammatical utterances, but the referent no longer makes sense.

Irrelevancy is generated by randomly selecting one or more AMR nodes—concepts, ARG-slots, or operators—and replacing them with nodes drawn from other dialogues. Because the replacements are semantically mismatched, the back-translated utterance appears fluent yet off-topic. The example given is a swap from “watch-01” to “listen-01” in a TV-viewing context.

Engagement decrease contains three sub-strategies: removing all nodes under an utterance AMR that encodes a question through the AMR-unknown marker; identifying the utterance with maximal AMR graph depth and dropping it; and randomly pruning lower-level ARG or op children from selected AMR nodes, thereby deleting fine-grained information. The ablation results later indicate that irrelevancy and engagement decrease are especially important to cover the kinds of errors made by modern chatbots.

4. Architecture, data construction, and training procedure

DEAM consists of four main components: a T5-based AMR parser to convert each utterance into an AMR graph; four AMR-level manipulation modules; a T5-based AMR-to-text generator to produce fluent negative dialogues; and a RoBERTa-large classifier that outputs the coherence score $y=1$6 (Ghazarian et al., 2022).

The encoder is RoBERTa-large, specified as 24 layers with 1024-dim hidden and pre-trained on English. The classifier head is a single linear layer mapping the “[CLS]” representation to 2 logits. Training uses Adam with learning rate $y=1$7, batch size 16, and 3–8 epochs depending on setup. Positive examples are human-human dialogues from TopicalChat and PersonaChat. Negative examples are an equal-sized set generated by AMR manipulations, producing a balanced training set. The validation split is 10–12% of each dataset.

In Setup 1, DEAM is compared to three baselines—an entity-graph model, an LSTM model, and DynaEval—all trained on identical positive/negative sets from Topical_DEAM plus Persona_DEAM. This isolates the effect of the negative-example generation strategy from the effect of using different training corpora. Setup 2 is pairwise on each baseline’s original dataset, and Setup 3 fixes RoBERTa while varying manipulations. Across these setups, the reported purpose is to test whether AMR-based manipulations, rather than only the classifier backbone, account for the observed gains.

5. Evaluation benchmarks and empirical results

DEAM is evaluated on FED and DSTC9 Interactive using Spearman’s rank correlation $y=1$8 between model scores and mean human ratings (Ghazarian et al., 2022). FED contains 125 dialogues annotated by 5 annotators each on coherence (0–2) and overall (0–4). DSTC9 Interactive contains 2,200 dialogues with coherence (1–3) and overall (1–5).

Benchmark	DEAM $y=1$9	Baselines
FED coherence	0.47	0.10, 0.13, -0.36
FED overall	0.55	all $y=0$0
DSTC9 coherence	0.19	all $y=0$1
DSTC9 overall	0.20	all $y=0$2

These results are reported as significant-margin improvements over the baseline methods. Setup 2 and Setup 3 further confirm that AMR-based manipulations yield the highest correlations in all cases. The paper gives a representative comparison from Table 4: with only text-level manipulations, Spearman is approximately 0.29 on FED, whereas with DEAM manipulations it is 0.47.

The paper also reports an asymmetry in detectability: DEAM can distinguish between coherent and incoherent dialogues generated by baseline manipulations, whereas baseline models cannot detect incoherent examples generated by DEAM. This suggests that the AMR-generated negatives span coherence failures not well approximated by heuristic perturbations.

6. Ablations, limitations, and research significance

The ablation study evaluates the contribution of each manipulation type on FED (Ghazarian et al., 2022).

Variant	FED $y=0$3 (coherence)	FED $y=0$4 (overall)
Full DEAM	0.47	0.55
– Contradiction	0.39	0.42
– Coref Inconsistency	0.41	0.46
– Irrelevancy	0.35	0.35
– Decr Engagement	0.34	0.35

All four manipulation types contribute to performance. The largest drops occur when removing Irrelevancy or Engagement Decrease, indicating that these are especially important to cover the kinds of errors made by modern chatbots. This suggests that failures of topical continuity and interactive detail are major components of perceived dialogue incoherence in the tested benchmarks.

The reported limitations are primarily tied to the AMR pipeline and the coverage of error types. DEAM depends on the quality of AMR parsing and AMR-to-text generation, with occasional tense, passive-voice, or detail omissions. It does not directly model sensibility errors requiring external knowledge. Its current manipulations focus on four error types, and future work is described as extending the framework to knowledge-based contradictions or idiomatic misuse.

Within the reported experiments, DEAM demonstrates that semantically controlled negative-sample generation via AMR produces training data that better reflects real coherence failures of state-of-the-art dialogue systems, and that a classifier trained on these examples correlates much more strongly with human judgments than baselines relying on heuristic text-level perturbations. The broader significance of the work is the methodological claim that evaluation quality can be improved by moving corruption from the text surface to the semantic graph.