Conditional PMI (CPMI) in Generative Evaluation

• CPMI is an information-theoretic measure that quantifies the dependency between two variables conditioned on a third, providing precise evaluation in multi-context scenarios.
• It is estimated using neural probability models and log-likelihood calculations, making it efficient for high-dimensional textual and multimodal data.
• Integrating CPMI in generation systems significantly boosts dialogue evaluation, reduces hallucinations, and improves content alignment with human judgments.

Conditional Pointwise Mutual Information (CPMI) is an information-theoretic measure that quantifies the degree of dependency between two variables conditioned on a third. In contemporary language technology research, CPMI has emerged as a critical diagnostic and scoring tool for analyzing and improving turn-level dialogue evaluation, controlling factuality in conditional generation, and mitigating hallucinations in both natural language and multimodal generative systems. The measure is inherently model-agnostic, adapts seamlessly to neural probability models, and has proven effective in advancing unsupervised evaluation and calibration tasks across several modalities.

1. Mathematical Definition and Formal Properties

Let $R$ denote a random variable corresponding to a context (e.g., dialogue history), $X$ a system output (e.g., response), and $H$ a hypothesis or evaluation criterion. The standard pointwise mutual information (PMI) between $R$ and $X$ is

$$\mathrm{PMI}(r, x) = \log \frac{p(r, x)}{p(r)\,p(x)}.$$

Conditional pointwise mutual information (C-PMI) augments this by quantifying how the interaction between $R$ and $X$ explains the probability of $H$ (Ren et al., 2023): $$\mathrm{C\text{-}PMI}(r, x \mid h) = \log \frac{p(r, x, h)p(h)}{p(r, h)p(x, h)}.$$ The measure can be symmetrized as

$X$0

A related formulation, Pointwise Conditional Mutual Information (PCMI), defined for three random variables $X$1, $X$2, and $X$3, is

$X$4

quantifying the incremental informativeness of $X$5 about $X$6, given $X$7 (Paranjape et al., 2021, Fang et al., 26 May 2025).

2. Intuitive Motivation and Theoretical Significance

C-PMI provides a localized, information-theoretic perspective on how much two sources (e.g., a dialogue history and a response) interact to drive a third event (e.g., a human-likeness hypothesis, a domain prompt, or an image feature). In dialogue evaluation, this directly addresses the shortcoming of metrics that score turns independently or fail to capture contingent user–system dynamics. If the interaction is strong and specifically relevant to $X$8, $X$9 will dominate the denominator, producing a high C-PMI. If, conversely, $H$0 is frequent regardless of the interaction, C-PMI vanishes or becomes negative (Ren et al., 2023).

In natural language generation and vision–language settings, CPMI disaggregates the contributions of different context sources—insulating against models that achieve high likelihood by over-relying on context-independent language priors or domain-generic patterns (Chae et al., 2024, Paranjape et al., 2021, Fang et al., 26 May 2025).

3. Estimation Procedures and Practical Calculation

In practice, exact computation of joint and marginal probabilities is intractable for high-dimensional textual (and multimodal) data. C-PMI is estimated through neural LLMs or large vision–LLMs (LVLMs), via autoregressive log-likelihoods. For a sequence $H$1 under a pretrained causal LLM $H$2,

$H$3

and C-PMI reduces to (Ren et al., 2023): $H$4 This requires four forward passes per hypothesis, all of which can be parallelized, require no additional training, and can be efficiently batched using modern APIs.

In source-conditioned generation, CPMI is computed at the token level by log-likelihood ratios under models with and without the relevant context, e.g.,

$H$5

for visual grounding (Fang et al., 26 May 2025), or similar mechanisms for conditional language generation (Paranjape et al., 2021).

4. Applications in Language and Vision–Language Evaluation

Dialogue Evaluation

C-PMI is incorporated as a replacement for negative log-likelihood (NLL) scorers in unsupervised turn-level dialogue evaluation metrics, such as FED. For each turn and evaluation dimension, scores are computed using C-PMI over a set of positive and negative hypotheses. Empirically, this modification boosts correlation with human judgments by 62.6% relative (Spearman’s ρ), with pronounced improvements on dimensions requiring modeling of user–system interaction (Interesting, Engaging, Specific) (Ren et al., 2023).

Controlling Hallucination in Conditional Generation

Domain-Conditional PMI (PMI₍DC₎) and related CPMI-based strategies penalize generation steps that draw excessively on domain-common or context-free priors rather than specific conditional information, significantly reducing hallucination and improving faithfulness in abstractive summarization benchmarks (e.g., XSUM) (Chae et al., 2024). In LVLMs, C-PMI-based decoding with token disentanglement for visual and textual streams enables robust mitigation of hallucinations by adaptively calibrating the model to prefer outputs that truly depend on the image, as opposed to overgeneral language priors (Fang et al., 26 May 2025).

Fine-Grained Specificity in Dialogue and Content Generation

PCMI has been shown to better isolate the unique informational contribution of one context over another (e.g., conversational history relative to new factual content), enabling models to generate responses that properly acknowledge dialogue flow and context, instead of simply reproducing content-specific information (Paranjape et al., 2021). Fused-PCMI strategies that trade joint PMI for higher history-specific PCMI offer further gains in human-likeness and acknowledgement phenomena.

5. Algorithmic Integration and Implementation

The integration of C-PMI into contemporary pipelines is training-free and model-agnostic. For dialogue evaluation, C-PMI is computed over all dimension–hypothesis pairs using efficient caching and batching. In generative decoding, CPMI or its conditional variants are injected into step-wise scoring during beam search, often gated by uncertainty (e.g., token entropy) to only apply corrections when the base model is least confident (Chae et al., 2024).

Multimodal CPMI-based decoding for LVLMs involves a bi-level optimization: an inner loop calibrates textual token sampling via C-PMI, while an outer visual purification loop dynamically retains only those image tokens that are most predictive of the text, using attention-based rewards and Gumbel-Softmax masking to maintain differentiability and efficient inference (Fang et al., 26 May 2025). Practical recipes use lightweight purifier networks, batch inference, and GPU acceleration to maintain scalability.

6. Empirical Validation and Comparative Results

C-PMI and its variants outperform standard metrics and scoring methods in multiple benchmark settings. Empirical results include:

Setting	CPMI Variant	Relative Gain / Key Finding
FED dialogue evaluation (Ren et al., 2023)	C-PMI, C-PMI-SYM	+62.6% Spearman’s ρ over baseline (avg.)
XSUM summarization (Chae et al., 2024)	PMI₍DC₎	+2.0 AlignScore, +2.2 FactCC, reduced hallucination
LVLM hallucination (Fang et al., 26 May 2025)	C-PMI Decoding	22.3-point drop in sentence-level hallucination (MSCOCO CHAIR); ~16% reduction in GPT-4o SHR
Dialogue acknowledgment (Paranjape et al., 2021)	PCMI, Fused-PCMI	74% preference for higher PCMI in acknowledgements, 60% Fused-PCMI over Max-PMI (human eval)

These gains reflect both improved alignment with human judgments and significant reductions in hallucination or context-irrelevant output.

7. Generalizations and Broader Significance

CPMI operates as a general diagnostic for disentangling and correctly attributing informativeness in multi-source generative scenarios, including but not limited to dialogue evaluation, content-grounded generation, style transfer, evidence conditioning in fact verification, and multi-source translation (Paranjape et al., 2021, Ren et al., 2023). Its core value lies in deconfounding spurious correlations, highlighting the unique contribution of particular context sources, and enabling precise intervention at inference time. While most applications to date focus on text and vision–language, a plausible implication is that further research may extend CPMI-based calibration strategies to other modalities (e.g., speech, structured data integration) or more granular conditional settings.