---
title: Conditional PMI (CPMI) in Generative Evaluation
url: https://www.emergentmind.com/topics/conditional-pointwise-mutual-information-cpmi
type: topic
---

# Conditional PMI (CPMI) in Generative Evaluation

Conditional Pointwise Mutual Information (CPMI) is an information-theoretic measure that quantifies the degree of dependency between two variables conditioned on a third. In contemporary language technology research, CPMI has emerged as a critical diagnostic and scoring tool for analyzing and improving turn-level dialogue evaluation, controlling factuality in conditional generation, and mitigating hallucinations in both natural language and multimodal generative systems. The measure is inherently model-agnostic, adapts seamlessly to neural probability models, and has proven effective in advancing unsupervised evaluation and calibration tasks across several modalities.

## 1. Mathematical Definition and Formal Properties

Let $R$ denote a random variable corresponding to a context (e.g., dialogue history), $X$ a system output (e.g., response), and $H$ a hypothesis or evaluation criterion. The standard pointwise mutual information (PMI) between $R$ and $X$ is
\[
\mathrm{PMI}(r, x) = \log \frac{p(r, x)}{p(r)\,p(x)}.
\]
Conditional pointwise mutual information (C-PMI) augments this by quantifying how the interaction between $R$ and $X$ explains the probability of $H$ [2306.15245]:
\[
\mathrm{C\text{-}PMI}(r, x \mid h) = \log \frac{p(r, x, h)p(h)}{p(r, h)p(x, h)}.
\]
The measure can be symmetrized as
\[
\mathrm{C\text{-}PMI\text{-}SYM}(r, x \mid h) = \frac{1}{2}\left(\mathrm{C\text{-}PMI}(r, x \mid h) + \mathrm{C\text{-}PMI}(x, r \mid h)\right).
\]
A related formulation, Pointwise Conditional Mutual Information (PCMI), defined for three random variables $X$, $Y$, and $Z$, is
\[
\pcmi(x; y \mid z) = \log \frac{p(x\mid y, z)}{p(x\mid z)},
\]
quantifying the incremental informativeness of $y$ about $x$, given $z$ [2104.07831, 2505.19678].

## 2. Intuitive Motivation and Theoretical Significance

C-PMI provides a localized, information-theoretic perspective on how much two sources (e.g., a dialogue history and a response) interact to drive a third event (e.g., a human-likeness hypothesis, a domain prompt, or an image feature). In dialogue evaluation, this directly addresses the shortcoming of metrics that score turns independently or fail to capture contingent user–system dynamics. If the interaction is strong and specifically relevant to $h$, $p(r, x, h)$ will dominate the denominator, producing a high C-PMI. If, conversely, $h$ is frequent regardless of the interaction, C-PMI vanishes or becomes negative [2306.15245].

In natural language generation and vision–language settings, CPMI disaggregates the contributions of different context sources—insulating against models that achieve high likelihood by over-relying on context-independent language priors or domain-generic patterns [2404.09480, 2104.07831, 2505.19678].

## 3. Estimation Procedures and Practical Calculation

In practice, exact computation of joint and marginal probabilities is intractable for high-dimensional textual (and multimodal) data. C-PMI is estimated through neural language models or large vision–language models (LVLMs), via autoregressive log-likelihoods. For a sequence $\mathbf{s} = s_1 \ldots s_n$ under a pretrained causal language model $P_\theta$,
\[
\mathrm{LL}(\mathbf{s}) = \frac{1}{n}\sum_{i=1}^n \log P_\theta(s_i \mid \mathbf{s}_{<i})
\]
and C-PMI reduces to [2306.15245]:
\[
\mathrm{C\text{-}PMI}(r, x \mid h) \approx \mathrm{LL}(r, x, h) + \mathrm{LL}(h) - \mathrm{LL}(r, h) - \mathrm{LL}(x, h).
\]
This requires four forward passes per hypothesis, all of which can be parallelized, require no additional training, and can be efficiently batched using modern APIs.

In source-conditioned generation, CPMI is computed at the token level by log-likelihood ratios under models with and without the relevant context, e.g.,
\[
\sum_{t} \left[\log p_\theta(y_t \mid v, x, y_{<t}) - \log p_\theta(y_t \mid x, y_{<t})\right]
\]
for visual grounding [2505.19678], or similar mechanisms for conditional language generation [2104.07831].

## 4. Applications in Language and Vision–Language Evaluation

### Dialogue Evaluation

C-PMI is incorporated as a replacement for negative log-likelihood (NLL) scorers in unsupervised turn-level dialogue evaluation metrics, such as FED. For each turn and evaluation dimension, scores are computed using C-PMI over a set of positive and negative hypotheses. Empirically, this modification boosts correlation with human judgments by 62.6% relative (Spearman’s ρ), with pronounced improvements on dimensions requiring modeling of user–system interaction (Interesting, Engaging, Specific) [2306.15245].

### Controlling Hallucination in Conditional Generation

Domain-Conditional PMI (PMI₍DC₎) and related CPMI-based strategies penalize generation steps that draw excessively on domain-common or context-free priors rather than specific conditional information, significantly reducing hallucination and improving faithfulness in abstractive summarization benchmarks (e.g., XSUM) [2404.09480]. In LVLMs, C-PMI-based decoding with token disentanglement for visual and textual streams enables robust mitigation of hallucinations by adaptively calibrating the model to prefer outputs that truly depend on the image, as opposed to overgeneral language priors [2505.19678].

### Fine-Grained Specificity in Dialogue and Content Generation

PCMI has been shown to better isolate the unique informational contribution of one context over another (e.g., conversational history relative to new factual content), enabling models to generate responses that properly acknowledge dialogue flow and context, instead of simply reproducing content-specific information [2104.07831]. Fused-PCMI strategies that trade joint PMI for higher history-specific PCMI offer further gains in human-likeness and acknowledgement phenomena.

## 5. Algorithmic Integration and Implementation

The integration of C-PMI into contemporary pipelines is training-free and model-agnostic. For dialogue evaluation, C-PMI is computed over all dimension–hypothesis pairs using efficient caching and batching. In generative decoding, CPMI or its conditional variants are injected into step-wise scoring during beam search, often gated by uncertainty (e.g., token entropy) to only apply corrections when the base model is least confident [2404.09480].

Multimodal CPMI-based decoding for LVLMs involves a bi-level optimization: an inner loop calibrates textual token sampling via C-PMI, while an outer visual purification loop dynamically retains only those image tokens that are most predictive of the text, using attention-based rewards and Gumbel-Softmax masking to maintain differentiability and efficient inference [2505.19678]. Practical recipes use lightweight purifier networks, batch inference, and GPU acceleration to maintain scalability.

## 6. Empirical Validation and Comparative Results

C-PMI and its variants outperform standard metrics and scoring methods in multiple benchmark settings. Empirical results include:

| Setting                                | CPMI Variant        | Relative Gain / Key Finding                  |
|-----------------------------------------|---------------------|-----------------------------------------------|
| FED dialogue evaluation [2306.15245]    | C-PMI, C-PMI-SYM    | +62.6% Spearman’s ρ over baseline (avg.)     |
| XSUM summarization [2404.09480]         | PMI₍DC₎             | +2.0 AlignScore, +2.2 FactCC, reduced hallucination |
| LVLM hallucination [2505.19678]         | C-PMI Decoding      | 22.3-point drop in sentence-level hallucination (MSCOCO CHAIR); ~16% reduction in GPT-4o SHR  |
| Dialogue acknowledgment [2104.07831]    | PCMI, Fused-PCMI    | 74% preference for higher PCMI in acknowledgements, 60% Fused-PCMI over Max-PMI (human eval) |

These gains reflect both improved alignment with human judgments and significant reductions in hallucination or context-irrelevant output.

## 7. Generalizations and Broader Significance

CPMI operates as a general diagnostic for disentangling and correctly attributing informativeness in multi-source generative scenarios, including but not limited to dialogue evaluation, content-grounded generation, style transfer, evidence conditioning in fact verification, and multi-source translation [2104.07831, 2306.15245]. Its core value lies in deconfounding spurious correlations, highlighting the unique contribution of particular context sources, and enabling precise intervention at inference time. While most applications to date focus on text and vision–language, a plausible implication is that further research may extend CPMI-based calibration strategies to other modalities (e.g., speech, structured data integration) or more granular conditional settings.

Source: https://www.emergentmind.com/topics/conditional-pointwise-mutual-information-cpmi