---
title: SMM Coherence Evaluation Framework
url: https://www.emergentmind.com/topics/smm-coherence-evaluation-framework
type: topic
---

# SMM Coherence Evaluation Framework

Searching arXiv for the specified framework and closely related coherence-evaluation papers.
arxiv_search({"query":"id:2509.02292 OR \"LLMs and their Limited Theory of Mind: Evaluating Mental State Annotations in Situated Dialogue\"","max_results":5,"sort_by":"submittedDate","sort_order":"descending"})
The **SMM Coherence Evaluation Framework** is a two-step framework for evaluating how well large language models track and assess **shared mental models (SMMs)** in situated, task-oriented team dialogues. In the framework, LLMs are used both as annotators of team dialogues and as automated discrepancy detectors among individuals’ mental states, with the goal of determining whether a model can accurately attribute beliefs, goals, and commitments of each team member and identify divergences between inferred and gold-standard mental representations [2509.02292]. The framework was introduced on dialogues from the **Cooperative Remote Search Task (CReST) corpus**, and positions SMM coherence as a proxy for evaluating Theory of Mind capabilities in LLMs, especially under conditions involving grounded collaboration, ambiguity, and partial information [2509.02292].

## 1. Conceptual scope and motivation

The framework’s stated goal is to evaluate whether LLMs can **track, infer, and assess shared mental models in situated, task-oriented team dialogues**, specifically whether they can accurately attribute **beliefs, goals, and commitments** of each team member and detect discrepancies between agents’ mental representations [2509.02292]. In this setting, SMM coherence concerns alignment between inferred mental-state annotations and the gold-standard representation of what the dialogue participants know, intend, and have committed to.

The use of SMM coherence as an evaluation target distinguishes the framework from discourse-only coherence measures. In summarization, for example, coherence is often operationalized as **logical connectedness and flow of the summary** or as **inter-sentential coherence**, that is, document-level fluency between adjacent sentences [2605.11693] [2011.13662]. Other work formalizes coherence through **cohesion, consistency, and relevance** [2310.00598]. By contrast, the SMM framework is centered on **team mental-state reconstruction** and on the detection of **mental misalignments** in dialogue [2509.02292]. This suggests that the framework evaluates a coordination-relevant notion of coherence rather than only textual well-formedness.

The framework also responds to a specific limitation of surface-level annotation quality. The reported results show that LLMs can appear coherent on straightforward natural-language annotation tasks while still failing in cases requiring **spatial reasoning** or **disambiguation of prosodic cues** [2509.02292]. A plausible implication is that SMM coherence is intended to expose failures that would not be visible through ordinary text similarity or annotation consistency alone.

## 2. Two-step architecture

The framework is explicitly organized as a **two-step LLM-based SMM coherence framework** [2509.02292].

### Step 1: Annotation of SMM elements

The first step annotates each utterance of team dialogue with explicit representations of both agents’ mental states. Dialogues are selected from the **CReST corpus**, in which a **searcher** and **director** collaborate via spoken communication about a shared spatial task [2509.02292]. For each utterance, the framework produces structured annotations covering beliefs, goals, commitments, and some second-order beliefs.

The paper describes **three annotation sets** [2509.02292]:

- **LLM Annotators**: Three different LLMs—**o3-mini, Claude Sonnet 4, Gemma 8.5B**—annotate every utterance, inferring SMM elements.
- **Naive Human Annotators**: Two naive humans, given only the dialogue and minimal context but no ground truth, independently label and then reconcile their annotations.
- **Ground Truth Annotators**: Two humans with full environmental and video access annotate each utterance, producing the gold-standard SMM labels.

The annotation schema is a structured JSON representation with the following slots [2509.02292]:

- `Searcher believes`
- `Director believes`
- `2nd order: Searcher believes that the director believes`
- `2nd order: Director believes that the searcher believes`
- `Searcher has committed to`
- `Director has committed to`
- `Director's goal is`
- `Searcher's goal is`
- `Common Belief`

Fields without updates receive **"no change"** [2509.02292]. LLM annotators are guided by detailed prompts containing instructions, definitions, and multiple examples to standardize annotation behavior [2509.02292].

### Step 2: LLM-based discrepancy evaluation

The second step is a **discrepancy analysis framework** in which another LLM compares naive annotations—whether produced by humans or LLMs—against the corresponding ground-truth annotation for the same utterance [2509.02292]. This comparison is performed by an LLM acting as an automated discrepancy detector.

The framework defines four discrepancy types [2509.02292]:

1. **Belief Contradictions**: One annotation posits belief \(b\), and the other its negation.
2. **Omissions**: The annotator omits a belief that appears in the ground truth.
3. **Unsupported Beliefs**: The annotator posits a belief not verifiable or relevant based on ground truth or context.
4. **False Beliefs**: The annotator posits a belief that directly contradicts the known ground truth.

For each discrepancy, the discrepancy detector outputs the type, the beliefs involved, and an explanation; if no discrepancy is found, an empty array is returned [2509.02292]. Structured prompts are used to enforce consistency and objectivity in discrepancy identification [2509.02292].

## 3. Annotation schema and representational commitments

The framework’s annotation design is notable for making SMM structure explicit at the utterance level. Rather than assigning a single dialogue-wide score, it encodes evolving mental-state information for both agents and includes **second-order belief** fields and a **Common Belief** field [2509.02292]. This yields a representation that can track not only what each participant believes, but also what each participant believes about the other participant’s beliefs.

This representational scheme matters because the framework is concerned with **shared mental models**, not merely with isolated speaker intentions. The inclusion of commitments and goals means that the target representation spans epistemic and action-oriented dimensions. The paper states that the framework evaluates whether models can **accurately attribute beliefs, goals, and commitments of each team member and detect discrepancies between agents’ mental representations** [2509.02292].

A plausible implication is that the framework treats SMM coherence as a structured property of dialogue understanding in which belief attribution, joint understanding, and commitment tracking must remain mutually aligned over time. This is consistent with the paper’s characterization of the output as a way to **track the team’s shared mental models** and to expose **discrepancies in the team members’ joint understanding** [2509.02292].

## 4. Formal discrepancy metric

To quantify SMM coherence, the framework defines a severity-sensitive discrepancy metric [2509.02292]. Let:

- \(B_{m,d}\) be the number of **Belief Contradictions** for model \(m\) on dialogue \(d\),
- \(F_{m,d}\) the number of **False Beliefs**,
- \(U_{m,d}\) the number of **Unsupported Beliefs**,
- \(O_{m,d}\) the number of **Omissions**,
- \(w_x\) the assigned weight to discrepancy type \(x\),
- \(N_d\) the number of utterances in dialogue \(d\).

The framework then defines the **weighted raw discrepancy score** as

$$
r_{m,d} = w_b B_{m,d} + w_f F_{m,d} + w_u U_{m,d} + w_o O_{m,d}
$$

and the **per-utterance discrepancy score** as

$$
s_{m,d} = \frac{r_{m,d}}{N_d}.
$$

For cross-model and cross-dialogue comparison, it defines a **normalized discrepancy score**

$$
\mathcal{S}_{m,d} = 1 - \frac{s_{m,d} - s_{\min}}{s_{\max} - s_{\min}},
$$

where \(s_{\min}\) and \(s_{\max}\) are the minimum and maximum \(s_{m,d}\) over all \((m,d)\) pairs [2509.02292]. Higher \(\mathcal{S}_{m,d}\) signifies better coherence, that is, fewer or milder discrepancies per utterance [2509.02292].

In the default experiments, all discrepancy weights \(w_x\) are set to \(1\), though the paper notes that they may be adjusted in applications prioritizing certain discrepancy types [2509.02292]. This weighting scheme makes the framework **severity-sensitive** while preserving modularity.

The metric design can be contrasted with other coherence evaluation traditions. Summary evaluation work such as **MM-Eval** uses a learned aggregation model over factual consistency, coherence, fluency, relevance, image-text relevance, and image-set diversity [2605.11693], while **FFCI** decomposes summarization quality into faithfulness, focus, coverage, and inter-sentential coherence [2011.13662]. The SMM framework instead aggregates **discrepancy counts** over explicit mental-state categories [2509.02292]. This suggests a different notion of interpretability: error types are directly tied to failures of mental-state attribution.

## 5. Experimental setting and empirical findings

The framework was applied to **6 dialogues** from the CReST corpus, comprising **1,142 utterances** annotated by **3 LLM models**, **2 naive humans**, and ground-truth annotators [2509.02292]. The study ultimately produced **a dataset of human and LLM annotations**, **a reproducible evaluation framework for SMM coherence**, and **an empirical assessment of LLM-based discrepancy detection** [2509.02292].

The reported discrepancy patterns are differentiated by annotator type and model [2509.02292]. **Claude Sonnet 4** had the highest total discrepancy counts, especially in **unsupported beliefs** and **contradictions**. **o3-mini** and **Gemma** showed fewer discrepancies, but with different profiles: **o3-mini** omitted information frequently, whereas **Gemma** showed more belief contradictions, indicating errors of inference rather than omission [2509.02292]. **Naive humans** produced fewer unsupported beliefs than LLMs, but their rates of belief contradictions and omissions were comparable, reflecting the difficulty of reconstructing ground truth from dialogue alone [2509.02292].

The normalized results indicate that **o3-mini and Gemma scored highest in SMM coherence, close to humans**, whereas Claude scored substantially lower [2509.02292]. The paper also reports that **dialogue difficulty affected discrepancy rates for all annotators**, whether human or LLM, with greater ambiguity or spatial complexity increasing errors [2509.02292].

The discrepancy detector itself was also evaluated. Human validation showed **good correspondence between LLM-automated discrepancy detection and manual analysis**, particularly for simpler models, with **o3-mini achieving approximately 88% accuracy** [2509.02292]. This indicates that the framework is not only a way to score annotation outputs but also a method for automating discrepancy analysis.

## 6. Interpretive significance, limitations, and relation to adjacent evaluation frameworks

The empirical findings support a nuanced view of LLM performance. The framework reports that LLMs can **annotate nuanced belief structures and follow annotation instructions well in simple contexts**, but their SMM coherence breaks down under **spatial ambiguity**, **implicit references**, or **lack of explicit context**, leading to elevated rates of hallucinated or unsupported beliefs and difficulties with grounding [2509.02292]. Naive humans, by comparison, are less prone to unsupported insertions, though they are still vulnerable to omissions and contradictions when deprived of environmental access [2509.02292].

From an evaluation-design perspective, the framework emphasizes that **multiple dimensions—contradiction, omission, unsupported belief, and false belief—drive a more nuanced assessment than classical binary correctness** [2509.02292]. This places it alongside other fine-grained coherence frameworks that seek more diagnostic signals than scalar quality judgments. For example, **SNaC** introduces a taxonomy of coherence errors in generated narrative summaries, including **CharE**, **RefE**, **SceneE**, and **InconE** [2205.09641]. Similarly, **FFCI** treats inter-sentential coherence as a standalone, interpretable dimension [2011.13662]. The SMM framework’s contribution is to define an analogous diagnostic space for **mental-state misalignment** rather than textual organization [2509.02292].

The framework also has explicit limitations [2509.02292]. LLM outputs are sensitive to **model versioning**, **non-determinism**, **prompt wording**, and unknown internal context. The experiment covers only **6 dialogues**, which constrains generalization. The weighting system is adjustable, but the true severity of discrepancy types may be application-specific. Most fundamentally, LLMs **lack direct access to the physical and social context of dialogue**, which encourages reliance on surface regularities rather than grounded understanding [2509.02292].

A plausible implication is that the framework is best understood as a diagnostic instrument for situated dialogue understanding rather than as a complete account of Theory of Mind. Its principal value lies in isolating where **mental modeling** fails: whether through contradiction, omission, unsupported inference, or false belief attribution [2509.02292].

## 7. Broader methodological position

Within the landscape of coherence evaluation, the SMM Coherence Evaluation Framework occupies a distinctive methodological position. It is **reproducible**, **interpretable**, and designed around explicit comparison between naive annotations and gold-standard labels [2509.02292]. Unlike reference-based or text-only coherence scoring, it evaluates whether an annotator or model preserves the internal alignment structure of a team’s mental states.

This orientation connects it indirectly to broader work on coherence assessment that seeks robustness beyond end-task accuracy. In text classification, coherence has been framed as whether predictions over relevant sub-spans are **consistent and human-aligned at the level of supporting evidence within a text** [2109.04922]. In summary coherence meta-evaluation, researchers have proposed **intra-system correlation** and **bias matrices** to reveal confounders hidden by aggregate performance metrics [2209.06517]. The SMM framework advances a parallel idea in situated dialogue: a model should not be judged only by whether it produces plausible annotations, but by whether those annotations align with the underlying evidence and with the structure of participants’ beliefs, goals, and commitments [2509.02292].

For collaborative AI, the framework is presented as useful because it identifies exactly what kind of **mental misalignments** LLMs produce, which can inform **error analysis**, **hybrid LLM-plus-human pipelines**, and future model development [2509.02292]. Its broader significance therefore lies less in general discourse scoring than in providing a formal apparatus for evaluating shared understanding in interactive settings.

Source: https://www.emergentmind.com/topics/smm-coherence-evaluation-framework