CIG: Measuring Conversational Information Gain in Deliberative Dialogues with Semantic Memory Dynamics

Abstract: Measuring the quality of public deliberation requires evaluating not only civility or argument structure, but also the informational progress of a conversation. We introduce a framework for Conversational Information Gain (CIG) that evaluates each utterance in terms of how it advances collective understanding of the target topic. To operationalize CIG, we model an evolving semantic memory of the discussion: the system extracts atomic claims from utterances and incrementally consolidates them into a structured memory state. Using this memory, we score each utterance along three interpretable dimensions: Novelty, Relevance, and Implication Scope. We annotate 80 segments from two moderated deliberative settings (TV debates and community discussions) with these dimensions and show that memory-derived dynamics (e.g., the number of claim updates) correlate more strongly with human-perceived CIG than traditional heuristics such as utterance length or TF--IDF. We develop effective LLM-based CIG predictors paving the way for information-focused conversation quality analysis in dialogues and deliberative success.

• The paper introduces the novel CIG framework, which quantifies dialogue progress using dynamic semantic memory dynamics.
• It employs LLM-based claim extraction and NLI to assess utterances along novelty, relevance, and implication scope dimensions.
• It demonstrates that memory-derived signals surpass surface-level metrics, offering valuable insights for automated dialogue moderation.

Conversational Information Gain: Structured Measurement of Informational Progress in Deliberative Dialogue

Introduction and Motivation

Accurately measuring informational progress in deliberative dialogue is pivotal for applications in computational social science, democratic engagement, and intelligent systems that interact with multi-agent conversations. Traditional dialogue quality metrics predominantly address surface characteristics—civility, argument form, or structural flow—yet lack granularity regarding how utterances concretely advance collective knowledge. The "CIG: Measuring Conversational Information Gain in Deliberative Dialogues with Semantic Memory Dynamics" (2604.15647) introduces a formal, operationalizable framework—Conversational Information Gain (CIG)—which jointly models the evolving collective memory and delivers interpretable scores for each utterance along Novelty, Relevance, and Implication Scope dimensions. This method aims to bridge the gap between computational proxies and the nuanced, content-driven judgments central to deliberative interaction.

Framework: CIG and Semantic Memory Dynamics

CIG is formulated by explicitly modeling an evolving Semantic Memory, which serves as the dynamic context for evaluating contributions. Each utterance undergoes structured processing:

• Extraction: LLMs decompose utterances into atomic, self-contained claims by resolving context dependencies (coreference, ellipsis) and filtering semantic fillers, enabling robust claim manipulation downstream.
• Consolidation: Via LLM-based NLI, each claim is assessed against the memory. Equivalence, entailment, and contradiction are classified deterministically, triggering ADD, UPDATE, or NONE actions, ensuring robust memory state tracking without collapsing inter-speaker disagreement.

Figure 1: Overview of the CIG pipeline for utterance evaluation, claim extraction, and memory consolidation.

Urged by the need for content-based, not merely formal or rhetorical, progress tracking, the CIG rubric scores utterances on four-level ordinal scales in three dimensions:

• Novelty: Assesses incremental to transformative contribution versus prior knowledge.
• Relevance: Measures substantive connection to the discussion goal.
• Implication Scope: Quantifies the generality or public-applicability beyond immediate participants.

Notably, CIG is not a global dialog-level construct but an utterance-level functional—anchoring informativeness to specific conversation states.

Human Annotation, Reliability, and Data

The authors curated and annotated 80 dialogue segments from two corpora: televised expert debates (INSQ) and facilitated community meetings (FORA). Annotations are strictly referenced to a semantic-memory-derived context to ensure judgments reflect dialogue-internal informational movement rather than superficial markers or general topical alignment. The multi-tiered annotation approach entailed separate ratings for CIG and each of its components, capturing nuanced inter-annotator variability. Krippendorff’s alpha ranged from ≈0.5–0.7, varying by aspect and setting, demonstrating moderate-to-high reliability for such high-level subjective phenomena.

Validation and Model-Based Prediction

The central claim—memory-derived dynamics predict human-perceived CIG more accurately than heuristic proxies (such as utterance length, TF–IDF, token surprisal)—is substantiated by multiple analyses:

• Superior Correlation: Memory-based counts of extracted claims and claim updates outperformed all heuristics (Pearson $|r|$ with CIG > 0.72, compared to < 0.70 for TF–IDF or utterance length).
• Aspect Analysis: Ordinal regression ablations reveal Novelty and Relevance drive perceived CIG, while Implication Scope, though capturing deliberative style variation, adds little predictive power — contradicting the theoretical intuition that broader public framing increases informativeness.

Figure 2: Heatmap of MAE for utterance CIG prediction using different aspect-combination and claim-aggregation operators.

• LLM Prediction Fidelity: GPT-5, conditioned only on the same memory-based summary as human annotators, achieves mean absolute errors competitive with human leave-one-out error. Full historical transcript provides marginal improvement, indicating that the memory summary compresses the salient context for CIG estimation.

Claim Aggregation and Conjunctive Bottleneck

A granular claim-level rating, followed by systematic aggregation experiments, reveals a critical insight: the conjunctive bottleneck phenomenon—utterance-level CIG is upper-bounded by its weakest dimension among Novelty, Relevance, and Scope. The "min" operator, not average, outperforms in mapping claim-level scores to utterance-level assessments, indicating listener sensitivity to any critical deficiency regardless of strengths elsewhere.

Figure 2: Aggregation operator analysis shows the “min” operator dominates, evidencing conjunctive constraints in perceived informativeness.

Moderator Effects and Temporal Dynamics

Addressing diachronic informativeness, the study applies CIG to investigate how moderator interventions structure informational gain. In adversarial debates (INSQ), confrontational moderator moves yield immediate surges in participant CIG. In collaborative community discussions (FORA), CIG growth is distributed, peaking several lags post-intervention and lacking confrontational peaks. This finding demonstrates the sensitivity of CIG to facilitation regimes and highlights its potential as a process-level analytic for dialogue management and automated facilitation.

Figure 3: Temporal alignment of CIG by lag since moderator intervention, delineating sharp versus distributed informational gain dynamics.

Annotation Interface and Distributional Analysis

The paper presents a practical annotation UI, supporting memory-informed, aspectwise ratings and facilitating contextual grounding through prior-knowledge highlighting.

Figure 4: Annotation interface showcasing side-by-side prior knowledge and utterance presentation, with cross-referenced keyword highlighting.

Rating distributions underscore a broad utilization of the full CIG spectrum, with minor corpus-driven differences.

Figure 5: Normalized distribution of utterance-level annotation ratings by aspect and corpus.

Implications and Future Directions

Theoretical Implications

• The conjunctive composition of CIG dimensions demonstrates that dialogue informativeness is not linearly additive: modelers must avoid simplistic sum- or average-based scoring for progress metrics in deliberative settings.
• The empirical sidelining of Implication Scope as an informativeness driver questions assumptions in political and philosophical theory about public-oriented speech automatically conferring deliberative value.

Practical and Computational Implications

• Memory-based signals surpass surface-level text metrics for automated informativeness estimation, recommending their adoption in computational moderation, conversational analysis, and training regimes for dialogue agents.
• Memory-derived summaries offer highly efficient context compression for both human and automated raters, with implications for scalability in LLM-based dialogue evaluation pipelines.

Future Directions

Opportunities emerge for integrating advanced memory architectures [e.g., agentic memory in LLM agents; (Xu et al., 17 Feb 2025)] to further enhance consolidation and context fidelity, exploring richer NLI schemes for finer-grained multi-party knowledge state tracking, and employing CIG as a reward signal in reinforcement learning or dialogue-guided agent training. Application domains include deliberation support platforms, scalable civic engagement tools, and automated moderation.

Conclusion

CIG operationalizes information gain in deliberative dialogues with explicit, interpretable, and context-sensitive metrics grounded in dynamic semantic memory. The evidence supports the superiority of memory-derived features and conjunctive aspect aggregation over conventional heuristics for both human and LLM-based informativeness assessment. The framework’s modularity and empirical grounding pave the way for principled dialogue evaluation and the development of more sophisticated, content-aware conversational AI systems.