Cross-View Mutual Information Maximization

• Cross-view mutual information maximization is an information-theoretic paradigm that aligns representations across different views of the same signal.
• It employs diverse objectives like contrastive losses and reconstruction to preserve both shared and view-specific features across modalities.
• The approach has enhanced clustering, segmentation, and recognition by robustly capturing inter-view dependencies and mitigating degenerate collapse.

Searching arXiv for recent and foundational work on cross-view mutual information maximization to support a comprehensive article. Cross-view mutual information maximization is an information-theoretic paradigm for representation learning in which two or more views of the same underlying signal are trained to exhibit high statistical dependence in latent space. In the cited literature, a “view” may denote different camera viewpoints of a 2D human pose, frontal and profile face images, feature and topology graphs, local and global image features, paired image–text observations, or image and text states inside multimodal LLMs. The shared objective is to retain information that is consistent across views while avoiding degenerate collapse, repetitive topics, or the destruction of view-specific structure through auxiliary terms such as reconstruction, entropy regularization, disentanglement, or disagreement penalties (Zhao et al., 2020, Fan et al., 2021, Saadabadi et al., 2022).

1. Information-theoretic basis

The central quantity is mutual information, written in several of the cited works as

$$\mathcal{I}(X;Y)=H(X)-H(X|Y)=\sum_x \sum_y p(x,y)\log \frac{p(x,y)}{p(x)p(y)}.$$

In this form, maximizing mutual information reduces conditional uncertainty and increases statistical dependence between the chosen variables. In metric learning, this principle is explicitly connected to both discriminative and generative interpretations:

$$\mathcal{I}(\widehat{Z};Y)=H(Y)-H(Y|\widehat{Z})=H(\widehat{Z})-H(\widehat{Z}|Y).$$

The discriminative view emphasizes minimizing label uncertainty given the embedding, whereas the generative view emphasizes low intra-class spread and high inter-class spread. The same analysis is used to relate cross-entropy and pairwise metric-learning losses to mutual-information maximization, showing that minimizing cross-entropy can be interpreted as maximizing mutual information and that several pairwise losses share the same tightness/contrastive decomposition (Boudiaf et al., 2020).

Within cross-view settings, the variables whose dependence is maximized vary by task. They may be pose representations across different viewpoints, local and global features within one image, local graph embeddings and summary vectors across feature and topology views, or text-token and visual states in multimodal models. In local multimodal learning, the sum of local mutual information terms is argued to be a lower bound on global mutual information, which provides a formal rationale for patch–sentence or local–global objectives instead of only whole-instance alignment (Liao et al., 2021).

2. What constitutes a “view”

The literature uses “view” in a broader sense than simple data augmentation. It can denote acquisition geometry, modality, graph construction, semantic granularity, or latent factorization. This breadth is one reason cross-view mutual-information objectives appear across clustering, recognition, graph learning, topic modeling, segmentation, and multimodal generation.

Representative setting	Views	Core mechanism
Human pose (Zhao et al., 2020)	Same pose from different viewpoints	Contrastive CV-MIM with pose/view disentanglement
Face recognition (Saadabadi et al., 2022)	Frontal and profile faces	Coupled-encoder, PAC, memory buffer, PADA
Graph learning (Fan et al., 2021)	Feature and topology views	Multi-view module, common encoder, reconstruction
Segmentation (Harb et al., 2021)	Local and global image features	Two-step segmentation and MI maximization
Cross-lingual topic modeling (Wu et al., 2023)	Linked cross-lingual word representations	TAMI regularization and CVL
MLLMs (Li et al., 16 May 2025)	Visual hidden states and text tokens	Explicit vision-text alignment loss

This range matters methodologically. Some works maximize mutual information between aligned instances across views, such as frontal/profile identities or multiview poses. Others maximize it between local and global summaries, or between continuous representations and discrete cluster probabilities. In graph representation learning, the feature view is constructed by a $K$-nearest neighbor graph based on feature similarity, while the topology view is the original graph; in unsupervised semantic segmentation, the two “views” are local patch descriptors and class-specific global features inside a single image rather than two augmentations (Fan et al., 2021, Harb et al., 2021).

3. Objective functions and estimators

Direct computation of mutual information is typically intractable in the high-dimensional continuous settings studied in these papers, so practical systems optimize lower bounds or surrogate objectives. The most common mechanisms are InfoNCE-style contrastive losses, Jensen–Shannon-based estimators, MINE-style neural estimators, and task-specific critics.

In image clustering, CRLC uses two heads sharing one backbone: a representation-learning head and a clustering head. The final objective is

$$L_{\text{CRLC}} = L_{\text{PC}} - \lambda_1 H(\tilde{Q}_{\text{avg}}) + \lambda_2 L_{\text{FC}},$$

where the feature contrastive loss acts at instance level and the probability contrastive loss acts at cluster level. For cluster probabilities, CRLC introduces the “log-of-dot-product” critic,

$f(\tilde{q}, q_i) = \log(\tilde{q}^\top q_i),$

because the standard dot-product critic is described as suboptimal for probabilities (Do et al., 2021).

In cross-lingual topic modeling, InfoCTM replaces direct topic alignment with topic alignment with mutual information. The paper adapts the InfoNCE estimator to linked cross-lingual word pairs and integrates the resulting TAMI regularization into a VAE-based neural topic-modeling objective. It further extends the positive pair set through cross-lingual vocabulary linking, which links a word not only to its direct translations but also to translations of its nearest neighbors in embedding space (Wu et al., 2023).

In the presence of missing and unaligned views, HmiMVC uses a hierarchical objective

$$\mathcal{L} = \mathcal{L}_{cl} + \mathcal{L}_{pre} + \mathcal{L}_{rec},$$

where $\mathcal{L}_{cl}$ is a noise-robust contrastive loss for class-level alignment, $\mathcal{L}_{pre}$ is a dual-prediction loss for instance-level recovery and alignment, and $\mathcal{L}_{rec}$ is a view-reconstruction loss introduced to avoid trivial solutions and collapse. The paper treats partially view-unaligned and partially sample-missing cases as distinct problems and addresses them with different learning paradigms (Wang et al., 2023).

In vision–text alignment for multimodal LLMs, VISTA adds an explicit alignment term to the standard cross-entropy objective:

$$\mathcal{L}_{\text{Total}} = \mathcal{L}_{CE} + \frac{1}{m}\sum_{t=1}^m \frac{t}{m}\|x_t - S_n^I\|_2^2.$$

The weighting function $$\mathcal{I}(\widehat{Z};Y)=H(Y)-H(Y|\widehat{Z})=H(\widehat{Z})-H(\widehat{Z}|Y).$$0 up-weights later tokens because the theoretical analysis in the paper argues that the relative contribution of vision to generation vanishes as the text sequence grows (Li et al., 16 May 2025).

4. Alignment, disentanglement, and collapse avoidance

A recurring design problem is that maximizing dependence between views is not, by itself, sufficient. Several papers therefore pair cross-view mutual-information maximization with mechanisms that preserve complementary or private structure.

In multiview clustering, IMVC concatenates multiple features into a unified feature representation to retrieve a common representation across views, while each view is also passed through an encoder to produce a compact view-specific representation. The model constrains the mutual information between the common and view-specific representations to be minimal, reconstructs each view from the refined representation by maximizing their mutual information, and finally maximizes the mutual information between an instance and its $$\mathcal{I}(\widehat{Z};Y)=H(Y)-H(Y|\widehat{Z})=H(\widehat{Z})-H(\widehat{Z}|Y).$$1-nearest neighbors to enhance intra-cluster aggregation and induce separation of different clusters (Lele et al., 2023).

In human-pose learning, CV-MIM explicitly decomposes 2D pose inputs into pose-dependent and view-dependent representations. It maximizes the mutual information of the same pose performed from different viewpoints in a contrastive learning manner, then adds a disentanglement regularization that minimizes the dependence between pose and view factors and a smoothness regularization that matches the representation to a uniform prior. The result is a view-disentangled pose representation designed for cross-view action recognition (Zhao et al., 2020).

In unsupervised domain adaptation, DPN argues that solely extracting domain-invariant representations can fail to preserve the representation that is private to the label-missing domain. Its formulation combines KL-based alignment with mutual-information maximization between the unlabeled domain and its latent space:

$$\mathcal{I}(\widehat{Z};Y)=H(Y)-H(Y|\widehat{Z})=H(\widehat{Z})-H(\widehat{Z}|Y).$$2

The stated objective is to preserve target-specific information while still mitigating domain divergence (Chen et al., 2022).

These formulations directly counter a common misconception that successful cross-view learning always requires complete invariance. Much of the literature instead seeks selective invariance: high dependence for shared semantics, low dependence between common and view-specific factors, and explicit preservation of private structure when that structure is necessary for generalization.

5. Representative domains and reported results

The empirical record shows that cross-view mutual-information maximization is not tied to a single task family. It has been used for deep clustering, face recognition under extreme pose, semantic segmentation, multimodal medical representation learning, and multimodal language modeling.

Domain	Benchmark context	Reported outcome
Image clustering (Do et al., 2021)	CIFAR10/20, STL10, ImageNet-Dogs	about 5–7% ACC improvement over best baseline
Single-phase clustering (Ntelemis et al., 2021)	CIFAR-10, CIFAR-100/20	average accuracy 89.1% and 49.0%
Extreme-pose face recognition (Saadabadi et al., 2022)	Multi-PIE, CFP-FP, IJB-B, IJB-C	94.64%, 95.85%, 88.35%, 90.05%
Unsupervised segmentation (Harb et al., 2021)	COCO-Stuff, COCO-Persons	38.8% PA, 69.6% PA; 26% relative PA increase on COCO-Stuff
Medical image–text learning (Liao et al., 2021)	EdemaSeverity, Pathology9	average AUC 0.88 and 0.84 with tuned local MI

In multiview clustering with missing and unaligned data, HmiMVC reports best values on several datasets, including $$\mathcal{I}(\widehat{Z};Y)=H(Y)-H(Y|\widehat{Z})=H(\widehat{Z})-H(\widehat{Z}|Y).$$3 NMI, $$\mathcal{I}(\widehat{Z};Y)=H(Y)-H(Y|\widehat{Z})=H(\widehat{Z})-H(\widehat{Z}|Y).$$4 ACC, and $$\mathcal{I}(\widehat{Z};Y)=H(Y)-H(Y|\widehat{Z})=H(\widehat{Z})-H(\widehat{Z}|Y).$$5 ARI on Scene-15; $$\mathcal{I}(\widehat{Z};Y)=H(Y)-H(Y|\widehat{Z})=H(\widehat{Z})-H(\widehat{Z}|Y).$$6 NMI and $$\mathcal{I}(\widehat{Z};Y)=H(Y)-H(Y|\widehat{Z})=H(\widehat{Z})-H(\widehat{Z}|Y).$$7 ARI on Deep Animal; $$\mathcal{I}(\widehat{Z};Y)=H(Y)-H(Y|\widehat{Z})=H(\widehat{Z})-H(\widehat{Z}|Y).$$8 NMI and $$\mathcal{I}(\widehat{Z};Y)=H(Y)-H(Y|\widehat{Z})=H(\widehat{Z})-H(\widehat{Z}|Y).$$9 ARI on MNIST-USPS; and $K$0 NMI on Caltech101. The paper states that it significantly outperforms state-of-the-art methods even in the cases of view missing and unalignment (Wang et al., 2023).

In graph representation learning, MVMI-FT is reported to achieve comparable or even better performance than previous supervised methods under unsupervised representation and linear evaluation, and on Amazon Photo it reaches $K$1, compared to $K$2 for the best supervised method APPNP. In cross-lingual topic modeling, InfoCTM is described as producing more coherent, diverse, and well-aligned topics and showing better transferability for cross-lingual classification tasks. In multimodal LLMs, VISTA is reported to significantly outperform baseline models across more than a dozen benchmark datasets, including VQAv2, MMStar, and MME, without additional trainable modules or extra training data (Fan et al., 2021, Wu et al., 2023, Li et al., 16 May 2025).

6. Recurring limitations and active questions

A central concern is degeneracy. In cross-lingual topic modeling, direct alignment is described as producing repetitive topics and degenerate topic representations of words, whereas topic alignment with mutual information is proposed specifically to prevent this failure mode. In clustering, entropy terms are used to prevent collapse to a single cluster, and in graph learning disagreement regularization is introduced to enlarge the distance between representations from the same view and thereby encourage diversity (Wu et al., 2023, Do et al., 2021, Fan et al., 2021).

Another recurring question is whether explicit cross-view mutual-information objectives are necessary when standard supervised losses already have an information-theoretic interpretation. The metric-learning analysis shows that cross-entropy can be seen as a proxy for maximizing mutual information. By contrast, the VISTA analysis argues that the implicit alignment objective of cross-entropy in multimodal LLMs has inherent limitations and that cross-modal alignment degrades as text sequence length increases. This places the current literature in a nuanced position: mutual-information maximization may already be implicit in common objectives, but explicit cross-view formulations become important when the desired alignment is weakly represented, imbalanced across modalities, or liable to vanish during training (Boudiaf et al., 2020, Li et al., 16 May 2025).

View selection is also an active design issue. In constrained multiview contrastive learning for medical image segmentation, the paper states that the unselected average mutual information among multi-views would obstruct the learning strategy, and therefore ranks frequency-domain views by estimated mutual information, keeping only the top-$K$3 for contrastive learning. This suggests that the main challenge is often not merely maximizing mutual information, but choosing the correct variables, granularity, and negative context over which that maximization should operate (Dai et al., 2024).

Across these works, cross-view mutual information maximization functions less as a single algorithm than as a reusable principle for coupling representations across views while controlling what information is shared, what is discarded, and what must remain private.