SKG-Eval: Stateful Evaluation of Multi-Turn Dialogue via Incremental Semantic Knowledge Graphs

Abstract: Evaluating multi-turn dialogue systems remains challenging because response quality depends not only on the current prompt, but also on previously established entities, claims, and conversational commitments. Existing automatic evaluators, including LLM-as-a-judge frameworks and embedding-based metrics, largely rely on flat or turn-isolated representations, making them less effective at detecting long-range issues such as contradiction, topic drift, and entity inconsistency. To address this, we propose SKG-Eval, a quasi-deterministic and interpretable framework that models dialogue as an evolving Semantic Knowledge Graph (SKG) of entities, relations, and commitments across turns. The framework incrementally updates the graph through structured triple extraction and computes three complementary signals: (i) local relevance, measuring alignment with the current prompt and optional reference; (ii) historical consistency, evaluating how newly introduced information connects to prior conversational context using graph-based and embedding-driven signals; and (iii) logical coherence, assessed by a geometric contradiction engine that detects cross-turn conflicts without relying on NLI models or LLM judges. These signals are adaptively fused and aggregated into a length-invariant session score via recency-weighted trend analysis. Across multiple benchmarks, SKG-Eval achieves higher correlation with human judgments and substantially improves detection of long-range inconsistencies in extended conversations. In addition, the framework produces explicit contradiction certificates and deterministic scores for fixed inputs, enabling reproducible and auditable evaluation. Overall, our results suggest that structured externalized state tracking through semantic knowledge graphs provides a scalable alternative to implicit reasoning in LLM-based dialogue evaluators.

• The paper introduces SKG-Eval, a stateful evaluator that incrementally builds typed semantic knowledge graphs to detect contradictions, topic drift, entity inconsistencies, and forgotten information across dialogue turns.
• The geometric contradiction engine, revision-aware filtering, and deterministic score aggregation achieve session-level Spearman correlations of .73 on MT-Bench and .74 on MultiChallenge, outperforming history-aware GPT-4o judges.
• SKG-Eval delivers more differentiated and reproducible scores at substantially lower cost, but its effectiveness depends on accurate triple extraction and does not assess external factual correctness or implicit pragmatic conflicts.

SKG-Eval addresses a persistent weakness in automatic dialogue evaluation: most evaluators, whether LLM-as-a-judge protocols or embedding-based metrics, score each turn against a flat or turn-isolated representation of context and therefore fail to detect cross-turn failures such as contradiction, topic drift, and entity inconsistency (2605.16650). The paper proposes an alternative paradigm in which conversational state is externalized into an incrementally constructed Semantic Knowledge Graph (SKG), and each new response is scored against this structured commitment store rather than against a serialized prefix. The framework is positioned as a quasi-deterministic, interpretable evaluator whose only stochastic component is a one-shot triple-extraction LLM call; all downstream scoring is deterministic given fixed extractor outputs.

Motivation and problem formulation

The authors ground their motivation in documented failure modes of multi-turn dialogue systems: performance degradation as conversations lengthen, contradictions of earlier assumptions, and the inability of frontier judges to surface session-level errors reliably. They formalize evaluation as a sequential decision problem with five desiderata: causality ($Q_t$ depends only on $\mathcal{D}_{1:t}$), statefulness, determinism, length invariance of the session score, and interpretability. Six cross-turn failure modes are enumerated—direct contradiction, numeric/value substitution, antonymic flip, topic drift, local irrelevance, and silent forgetting—and mapped onto three per-turn signals: local relevance, historical consistency, and logical coherence.

Framework architecture

The core data structure is a typed, time-stamped directed multigraph $G_t = (V_t, E_t)$ whose nodes carry normalized labels, entity types from a fixed taxonomy, embeddings, importance scores, and quarantine flags, and whose edges carry relation strings, attribute classes (definition, effect, property, comparison, requirement, quantity, negation), intent modality, and property type (Exclusive vs. Additive). Triples are extracted by a single LLM call at temperature 0 using a deterministic prompt encoding normalization and disambiguation rules. Cross-turn deduplication merges subjects when embedding cosine similarity exceeds $0.80$, enforcing label consistency—a precondition for contradiction detection—and semantic edges are added above a $0.50$ similarity threshold.

Local relevance is computed via a "Semantic Triangle": max-pooled sentence-level cosine similarity between the response and both the prompt and an optional reference, gated by reference availability and prompt length. Max-pooling implements a coverage notion that rewards focused answers embedded in elaboration without dilution.

Historical consistency combines two mechanisms. A graph anchor score assigns each newly introduced node a value based on its attachment type—factual edge ($\eta_F = 1.0$), semantic edge ($\eta_S = 0.65$), or disconnected drift ($\eta_D = 0.20$)—weighted by node importance. A session-anchor mechanism, based on cosine similarity to the first-turn embedding scaled by $\delta = 0.85$, rescues focused Q&A sessions where graph disconnection is structurally expected. The final score is the maximum of the two.

The geometric contradiction engine

The logical coherence score is the paper's central contribution. Rather than applying NLI models or LLM judges, the engine compares current-turn fact edges against historical fact edges incident to candidate nodes through a prioritized cascade of detectors: NegFlip (confidence 0.95), Antonym (0.88), NumMismatch (0.92), Exclusive-Object Conflict, Same-Type EOC, and Residual Semantic Drift, interleaved with abstaining guards (IntentGate, ElabGuard, NoiseFloor) that suppress categorically incomparable pairs. The cascade exits at the first firing detector, ensuring high-precision symbolic conflicts dominate softer geometric ones. The score is $S^{\text{log}}_t = 1 - \max c^*(u)$ over candidate nodes.

A notable design element is revision-aware filtering: user-directed updates ("change that to...") mark historical edges as revision targets, which are temporarily excluded from contradiction comparison without being deleted from the graph. This prevents penalizing models for correctly following authorized state changes. A worked example involving a slogan revision illustrates how unfiltered comparison would incorrectly trigger a semantic-drift conflict.

The authors state Proposition 2 identifying three regimes where the geometric engine should outperform string-level NLI: numeric substitution within otherwise identical claims, long-prefix contradictions outside NLI effective context, and antonymic paraphrase. Importantly, they concede this is not universal dominance—the proposition identifies favorable regimes rather than claiming general superiority.

Scores are fused via regime-adaptive convex weights selected by a lookup over Short/QA/General regimes, refined by three monotone guards: a hard logic gate capping turns at 0.40 when $\mathcal{D}_{1:t}$0, a joint weakness penalty, and non-sequitur softening. Session aggregation uses recency-weighted exponential weights plus a weighted least-squares trend slope scaled by a length-adaptive coefficient $\mathcal{D}_{1:t}$1, yielding shift-invariant, slope-aware session scores. Quarantine at $\mathcal{D}_{1:t}$2 excludes low-quality content from propagating through graph state.

Empirical results

Across MT-Bench and MultiChallenge, SKG-Eval achieves the strongest correlation with human judgments at both turn and session levels. Session-level Spearman correlation reaches .73 on MT-Bench and .74 on MultiChallenge, versus .66 and .61 for the strongest history-aware GPT-4o judge baseline—gains of +.07 and +.13 respectively, reported as significant at $\mathcal{D}_{1:t}$3 under Holm–Bonferroni-corrected bootstrap testing. On model ranking over six generated LLMs, SKG-Eval achieves Kendall's $\mathcal{D}_{1:t}$4 against human rank, compared to 0.73 for LLM-as-a-judge.

On the mechanism-targeted SKG-Probe diagnostic benchmark (six engineered sessions isolating individual detector pathways), full SKG-Eval achieves a mean F1 of 79.8% across contradiction categories, versus 60.4% for the history-aware GPT-4o judge and 38.2% for ECoh. Per-detector ablations show each symbolic detector contributes measurably—for example, removing NegFlip drops NegFlip-category F1 from 89 to 41. Component ablations on MultiChallenge attribute the largest single drop (−.09 Spearman) to replacing the geometric engine with an NLI premise-pool baseline, followed by removing cross-turn deduplication (−.07) and the attribute taxonomy (−.05), supporting the claim that structured state tracking rather than any auxiliary refinement drives the gains.

A striking numerical finding concerns the divergence between paradigms: LLM-as-a-judge assigned near-saturated scores (>0.95) to nearly all evaluated models, while SKG-Eval produced more differentiated scores averaging approximately 0.24 lower. For instance, GPTOSS-20B received 0.766 under SKG-Eval versus 0.988 from the judge, and Llama-3-70B received 0.741 versus 0.994. The paper attributes this to judges rewarding local fluency while under-penalizing delayed contradictions and progressive semantic degradation, evidenced by negative session slopes ($\mathcal{D}_{1:t}$5) for models such as MiniMax-M2.7 and Mistral-7B despite high judge scores. This implies that holistic judge prompting may systematically overestimate long-horizon reliability.

On efficiency, SKG-Eval costs $\mathcal{D}_{1:t}$60.71 total on a 1,000-turn run, runs on CPU alone, and produces exactly reproducible scores, whereas history-aware judging costs $27.10 with run-to-run standard deviation of 0.04. Baseline correlation degrades with session length while SKG-Eval remains stable across length bins, consistent with graph-indexed retrieval of historical claims independent of prefix length.

Limitations and open questions

The paper is candid about several constraints. First, the framework depends on triple-extraction quality; error analysis identifies extraction fragmentation and ambiguous entity normalization as the dominant practical failure sources, meaning the "quasi-determinism" guarantee is conditional on extractor reliability. Second, the curated antonym lexicon limits contradiction recall in specialized technical domains. Third, SKG-Eval evaluates internal semantic consistency only—it does not verify grounding against external factual knowledge, so a consistently wrong but self-coherent conversation scores well. Fourth, implicit pragmatic contradictions requiring deep world knowledge may be under-detected. The SKG-Probe benchmark itself is small (six sessions), which raises questions about the generality of the per-detector F1 figures beyond these engineered regimes. Open questions include multilingual contradiction modeling, adaptive extraction, integration with retrieval-grounded verification, and extension beyond explicit semantic inconsistency to higher-order reasoning failures.

Conclusion

SKG-Eval demonstrates that externalizing conversational state into an incremental, typed knowledge graph enables deterministic, interpretable, and auditable multi-turn evaluation that outperforms both turn-isolated metrics and history-aware LLM judges precisely where those baselines are weakest: long-horizon contradiction recall and length-stable correlation with human judgment. Its principal trade-off is dependence on upstream extraction fidelity and coverage-limited symbolic resources, and its scope is internal consistency rather than factual grounding. The framework's contradiction certificates and near-linear computational scaling make it a practical complement to holistic judge-based evaluation for long-form dialogue assessment.