Cross Mutual Information Overview

• Cross mutual information is a measure that quantifies dependence across different distributions, modalities, or views by comparing test data against a reference model.
• It separates the roles of distribution in dependence measurement, making it asymmetric and capable of yielding negative values in certain non-stationary scenarios.
• The concept is applied in machine learning for tasks like cross-modal alignment, regularization, and robust estimation in dynamic or multi-domain settings.

Searching arXiv for recent and foundational papers on Cross Mutual Information and closely related usages. arXiv search query: "Cross Mutual Information (Gohil et al., 21 Jul 2025) mutual information cross-modal (Wang et al., 2023) cross-sample mutual information (Zhu et al., 2021) cross-view mutual information (Zhao et al., 2020)" Cross mutual information is a context-dependent term in contemporary information theory and machine learning. In its most explicit formalization, it denotes a reference-relative dependence measure that evaluates how strongly an $X$-$Y$ dependence encoded by one distribution is expressed in samples drawn from another distribution (Gohil et al., 21 Jul 2025). In broader applied usage, the phrase often refers more loosely to mutual-information objectives defined across domains, modalities, views, speakers, samples, variables, or frequencies, rather than to a single universally standardized quantity. Across these literatures, the unifying theme is the use of mutual information to quantify or control dependence across a boundary—between distributions, cross-modal representations, cross-view latents, cross-sample factors, or cross-frequency components—while preserving or suppressing specific forms of shared structure (Wang et al., 2023).

1. Formal information-theoretic definition

The standard definition of mutual information between two random variables $X$ and $Y$ is

$$I(X;Y)=\int p(x,y)\log\frac{p(x,y)}{p(x)p(y)}\,dx\,dy.$$

This quantity can also be written as an expectation of the local or pointwise mutual information

$$i_p(x;y)=\log\frac{p(x,y)}{p(x)p(y)}, \qquad I_p(X;Y)=\mathbb{E}_p\{i_p(x;y)\}.$$

In this conventional form, both the dependence model and the averaging distribution are the same distribution $p$ (Gohil et al., 21 Jul 2025).

The 2025 paper titled "Cross Mutual Information" introduces a different object, denoted $CI_{pq}$, that separates these two roles (Gohil et al., 21 Jul 2025). It is defined as

$$CI_{pq} = \mathbb{E}_{x,y\sim p(x,y)} \left\{ \log\left(\frac{q(x,y)}{q(x)q(y)}\right) \right\} = \mathbb{E}_p\{i_q(x;y)\}.$$

Here, $q$ is the reference distribution that defines what counts as dependence, while $Y$0 is the test distribution over which that dependence is averaged. This yields a quantity that answers a different question from ordinary mutual information: not “how dependent are $Y$1 and $Y$2 in $Y$3?”, but rather “how strongly is the $Y$4-$Y$5 dependence encoded by $Y$6 expressed in samples from $Y$7?” (Gohil et al., 21 Jul 2025).

Several immediate properties follow from this definition. If $Y$8, then $Y$9. If the reference distribution is independent, $X$0, then $X$1 for every $X$2. Unlike ordinary mutual information, cross mutual information is generally asymmetric, because $X$3 in general, and it can be negative because the expectation is taken under a distribution different from the one appearing inside the logarithm (Gohil et al., 21 Jul 2025).

2. Interpretation, non-stationarity, and relation to model fit

The main motivation for formal cross mutual information is non-stationarity. When data are split by condition, state, or time window, ordinary mutual information estimated in each subset is really a conditioned quantity such as $X$4, not a common pairwise measure in a shared probability space. Conditioning can introduce redundancy or synergy effects, so direct comparison of those values may conflate changes in pairwise dependence with changes induced by the conditioning variable (Gohil et al., 21 Jul 2025). Cross mutual information addresses this by evaluating condition-specific or test samples in a common reference probability space.

The same paper gives an especially useful interpretation through conditional densities: $X$5 Under this view, cross mutual information measures how much more the reference conditional model $X$6 favors the observed $X$7 than the reference marginal model $X$8, when the observed sample comes from $X$9 (Gohil et al., 21 Jul 2025). A positive value indicates that the test samples conform to the dependence pattern encoded in the reference; a negative value indicates that they are anti-aligned or surprising relative to that pattern.

The paper also relates $Y$0 to model assessment in linear regression. Under a Gaussian reference model $Y$1, the local cross mutual information becomes

$Y$2

This shows that smaller predictive residuals under the reference conditional model increase cross mutual information, making it a likelihood-ratio-style measure of how well the reference dependence explains the test data (Gohil et al., 21 Jul 2025).

A practical implication is that cross mutual information can be evaluated even when the test set is too small to estimate its own joint density robustly, including sliding-window or online settings. This suggests why the formal concept is particularly attractive for non-stationary systems and state-dependent analyses (Gohil et al., 21 Jul 2025).

3. Broader uses in machine learning and signal processing

Outside the 2025 formalization, the phrase “cross mutual information” is used more broadly for mutual-information objectives defined across domains, modalities, views, variables, or frequencies. The quantities differ, but they share the same structural role: they measure or shape dependence across paired but non-identical sources of information.

Setting	Quantity used	Role
Non-stationary dependence	$Y$3	Reference-relative dependence across distributions
Zero-shot cross-modality diffusion	local-wise MI $Y$4	Cross-modality conditioning signal
Cross-view pose learning	$Y$5	View-invariant pose learning
Cross-modal hashing	$Y$6	Shared information between modality-specific codes
Debiasing	$Y$7	Cross-sample dependence minimization
Frequency analysis	$Y$8	Dependence between spectral components

In zero-shot cross-modality data translation, MIDiffusion defines mutual information as a local cross-domain statistical dependence between neighborhoods in source and target images. Its local-wise mutual information layer is used as a conditioning signal in a score-based diffusion model, rather than as a separate regularizer, so that reverse diffusion is guided by local statistical similarity across modalities (Wang et al., 2023). In that setting, “cross mutual information” means local shared structure across source and target modalities, computed directly from neighborhoods rather than learned paired embeddings.

In representation learning for human pose, CV-MIM uses a cross-view mutual information maximization objective. Starting from

$Y$9

the method derives a relaxed objective involving

$$I(X;Y)=\int p(x,y)\log\frac{p(x,y)}{p(x)p(y)}\,dx\,dy.$$0

which is interpreted as the shared pose information that remains stable across camera views (Zhao et al., 2020). The cross-view MI term is therefore not a new information-theoretic primitive, but a standard MI applied to paired latent codes from different views.

In unsupervised cross-modal hashing, CMIMH uses

$$I(X;Y)=\int p(x,y)\log\frac{p(x,y)}{p(x)p(y)}\,dx\,dy.$$1

where $$I(X;Y)=\int p(x,y)\log\frac{p(x,y)}{p(x)p(y)}\,dx\,dy.$$2 is the central cross-modal term between image and text hash representations (Hoang et al., 2021). The paper explicitly distinguishes this from mere modality-gap minimization, arguing that maximizing shared information across modalities is not equivalent to forcing the two modalities to become identical.

In bias-invariant representation learning, CSAD introduces cross-sample mutual information between target features and bias features. Its core estimator,

$$I(X;Y)=\int p(x,y)\log\frac{p(x,y)}{p(x)p(y)}\,dx\,dy.$$3

treats $$I(X;Y)=\int p(x,y)\log\frac{p(x,y)}{p(x)p(y)}\,dx\,dy.$$4 as positive pairs when samples $$I(X;Y)=\int p(x,y)\log\frac{p(x,y)}{p(x)p(y)}\,dx\,dy.$$5 and $$I(X;Y)=\int p(x,y)\log\frac{p(x,y)}{p(x)p(y)}\,dx\,dy.$$6 share similar bias, thereby extending dependence measurement beyond within-sample coupling (Zhu et al., 2021). This is a different usage again: the “cross” refers to sampling structure across different examples rather than across modalities or distributions.

In frequency-domain analysis, MI-in-frequency defines

$$I(X;Y)=\int p(x,y)\log\frac{p(x,y)}{p(x)p(y)}\,dx\,dy.$$7

which measures dependence between spectral-process increments at specific frequencies, either within a signal or across two signals (Malladi et al., 2017). This extends the same basic information-theoretic logic to cross-frequency and cross-signal coupling.

4. Cross mutual information as an alignment or regularization principle

A recurring pattern in recent work is that cross mutual information is used not merely to measure dependence, but to shape learned representations. In these models, MI terms act as alignment objectives, bottlenecks, or regularizers that preserve desired shared information while suppressing nuisance structure.

InfoCTM provides a clear example in cross-lingual topic modeling. It defines a topic alignment with mutual information objective over linked cross-lingual word pairs $$I(X;Y)=\int p(x,y)\log\frac{p(x,y)}{p(x)p(y)}\,dx\,dy.$$8 and optimizes a contrastive lower bound

$$I(X;Y)=\int p(x,y)\log\frac{p(x,y)}{p(x)p(y)}\,dx\,dy.$$9

The purpose is not only to align linked words across languages, but also to avoid degenerate topic representations by contrasting them against unlinked words (Wu et al., 2023). This suggests that cross mutual information can simultaneously enforce alignment and non-collapse.

In multivariate time-series forecasting, InfoTime introduces an information bottleneck over cross-variable dependence: $$i_p(x;y)=\log\frac{p(x,y)}{p(x)p(y)}, \qquad I_p(X;Y)=\mathbb{E}_p\{i_p(x;y)\}.$$0 Its Lagrangian form,

$$i_p(x;y)=\log\frac{p(x,y)}{p(x)p(y)}, \qquad I_p(X;Y)=\mathbb{E}_p\{i_p(x;y)\}.$$1

treats cross-variable mutual information as something that must be filtered rather than maximized indiscriminately (Qi et al., 2024). The method’s interpretation is that useful shared information across variables should survive because it helps prediction, whereas redundant or unrelated cross-channel information should be penalized.

A similar separation of useful and nuisance dependence appears in cross-domain nuclei segmentation. MaNi maximizes a Jensen–Shannon-based lower bound on mutual information between source-domain nuclei representations and target-domain nuclei representations, while contrasting them against source-background/target-nuclei pairs (Sharma et al., 2022). Here the cross-domain MI is class-aware and representation-based rather than global. This suggests a broader principle: cross mutual information is often most useful when it is defined over semantically matched components, not over entire inputs.

These examples show that the modern machine-learning use of cross mutual information is often selective. It is rarely “maximize all shared information”; it is more often “maximize the specific shared information that preserves semantics, structure, or invariance, while constraining or minimizing the rest” (Hoang et al., 2021).

5. Relation to cross-entropy, classifiers, and common misconceptions

A common misconception is to equate cross mutual information with anything involving cross-entropy. Several papers clarify that the relevant information-theoretic object is usually still ordinary mutual information, even when cross-entropy is the training loss.

In classifier theory, one line of work shows that softmax cross-entropy can be interpreted as maximizing a variational lower bound on $$i_p(x;y)=\log\frac{p(x,y)}{p(x)p(y)}, \qquad I_p(X;Y)=\mathbb{E}_p\{i_p(x;y)\}.$$2, the mutual information between inputs and labels, under balanced class priors (Qin et al., 2019). Under imbalance, probability-corrected softmax restores the same mutual-information interpretation (Qin et al., 2021). A related metric-learning paper argues that cross-entropy and pairwise losses are both proxies for maximizing $$i_p(x;y)=\log\frac{p(x,y)}{p(x)p(y)}, \qquad I_p(X;Y)=\mathbb{E}_p\{i_p(x;y)\}.$$3, the mutual information between embeddings and labels, though through different views of the same quantity (Boudiaf et al., 2020). These works are highly relevant to “cross-entropy and mutual information,” but they do not define a separate formal object called cross mutual information.

Another misconception is to assume that any MI maximization across views or modalities is the same as the 2025 formal $$i_p(x;y)=\log\frac{p(x,y)}{p(x)p(y)}, \qquad I_p(X;Y)=\mathbb{E}_p\{i_p(x;y)\}.$$4. It is not. Cross-view MI in pose learning, cross-modal MI in hashing, and local cross-modality MI in diffusion all use standard mutual information or MI-inspired surrogates across paired variables or representations (Zhao et al., 2020, Hoang et al., 2021, Wang et al., 2023). By contrast, $$i_p(x;y)=\log\frac{p(x,y)}{p(x)p(y)}, \qquad I_p(X;Y)=\mathbb{E}_p\{i_p(x;y)\}.$$5 specifically separates the reference distribution from the averaging distribution and is designed for comparison under non-stationarity (Gohil et al., 21 Jul 2025).

A third misconception is that maximizing cross mutual information always implies semantic robustness. Several application papers caution otherwise. Histogram-based whole-image mutual information can fail under environmental variation and background changes, behaving more like a global intensity-distribution comparator than a robust semantic matcher (Liao et al., 2024). This suggests that the effectiveness of cross mutual information depends strongly on which variables are paired, how the distributions are estimated, and whether nuisance factors dominate the statistics.

6. Limitations, caveats, and current scope

The main limitations of cross mutual information differ by formulation. For the formal $$i_p(x;y)=\log\frac{p(x,y)}{p(x)p(y)}, \qquad I_p(X;Y)=\mathbb{E}_p\{i_p(x;y)\}.$$6, the central caveat is reference dependence: if the reference distribution encodes independence, then $$i_p(x;y)=\log\frac{p(x,y)}{p(x)p(y)}, \qquad I_p(X;Y)=\mathbb{E}_p\{i_p(x;y)\}.$$7 for all test distributions, even if the test distribution itself contains strong dependence (Gohil et al., 21 Jul 2025). This means cross mutual information is not a universal dependence detector; it is a reference-relative dependence expression measure.

Estimation is another persistent issue. The 2025 paper uses KSG estimators and notes that model-free estimation can become unstable when test data lie outside the support of the reference distribution (Gohil et al., 21 Jul 2025). Applied machine-learning methods typically avoid exact MI estimation altogether and instead optimize surrogates: InfoNCE bounds, Jensen–Shannon bounds, CLUB upper bounds, or reconstruction-style variational bounds (Wu et al., 2023, Hoang et al., 2021, Qi et al., 2024). This suggests that much of the modern literature uses “mutual information” as a principled design target, while relying on tractable approximations operationally.

A further caveat is semantic mismatch. Several methods work precisely because the paired variables are carefully chosen: source and target nuclei regions, cross-view pose codes, linked bilingual words, or local statistical neighborhoods (Sharma et al., 2022, Zhao et al., 2020, Wu et al., 2023, Wang et al., 2023). When the pairing is poorly aligned, mutual-information objectives can become noisy, collapse-prone, or dominated by nuisance structure. A plausible implication is that the practical success of cross mutual information depends at least as much on pair construction and representation choice as on the estimator itself.

Taken together, the literature suggests two distinct meanings of cross mutual information. In the narrow formal sense, it is the distribution-asymmetric quantity

$$i_p(x;y)=\log\frac{p(x,y)}{p(x)p(y)}, \qquad I_p(X;Y)=\mathbb{E}_p\{i_p(x;y)\}.$$8

introduced to compare dependence across non-stationary settings (Gohil et al., 21 Jul 2025). In the broader applied sense, it is an umbrella label for MI-based dependence measures and regularizers defined across distributions, modalities, views, samples, variables, or frequencies. The shared conceptual core is the same: dependence is not treated as an isolated property of a single variable pair in a single distribution, but as a structured relation that must be measured or optimized across some boundary of interest.