Weak-Modality Penalized Contrastive Learning

• Weak-modality penalized contrastive learning is a framework that revises traditional InfoNCE loss to handle noisy, weak, or imbalanced modalities.
• It employs robust loss functions, weighted rebalancing, and structured positive sets (e.g., group-level or hierarchical) to mitigate false positives and modality gaps.
• Empirical results demonstrate enhanced cross-modal matching, improved ImageNet accuracy in weak supervision settings, and reduced collapse in multimodal representations.

Searching arXiv for recent and foundational papers on weak-modality and penalized contrastive learning. Weak-modality penalized contrastive learning denotes a class of contrastive representation-learning methods that alter the standard positive–negative objective when one view, modality, or supervisory signal is weak: noisy positive views, false negatives, asynchronous multimodal evidence, weakly paired samples, missing modalities, or modality imbalance. The unifying premise is that vanilla InfoNCE-style learning assumes reliable positive correspondence and uniformly valid negatives; when that assumption fails, the literature reports suboptimal or collapsed representations, class collision, weak-modality suppression, and persistent cross-modal geometric separation (Chuang et al., 2022, Zheng et al., 2021, Liang et al., 2022, Gong et al., 23 Oct 2025). Recent work addresses these failures by redesigning the loss, redefining positives and negatives at group or event level, injecting weak structure such as hierarchies or group labels, or replacing positive imitation by negative suppression when direct alignment would erase modality-specific information (Yu et al., 2022, Bhalla et al., 2024, Gorla et al., 3 Feb 2026, Lin et al., 2024).

1. Problem setting and characteristic failure modes

The basic failure mode arises when a positive pair does not actually share the underlying information that contrastive learning expects. In robust contrastive learning against noisy views, this appears as false positive pairs with no apparent shared information, caused by aggressive augmentations, misalignments in multimodal data, or other real-world noise patterns (Chuang et al., 2022). In weakly supervised audio-visual violence detection, the same issue takes the form of modality asynchrony and undifferentiated instances: synchronized snippets need not jointly contain violent evidence, and top-$K$ selection can ignore useful negative evidence such as background or normal snippets (Yu et al., 2022).

A second failure mode concerns the definition of negatives. Weakly supervised contrastive learning for images identifies class collision as a consequence of instance discrimination: semantically similar samples are inevitably treated as negatives, which degrades representation quality (Zheng et al., 2021). Heterogeneous contrastive learning makes the same point in multi-view and multi-label settings, arguing that vanilla contrastive loss is sub-optimal in the presence of false negative pairs and that the problem is amplified by complex heterogeneity in modality and label structure (Zheng et al., 2021).

A third failure mode is weak correspondence rather than wrong correspondence. In weakly paired multimodal data, samples can share a perturbation or condition label without any instance-level pairing; this is precisely the setting targeted by GROOVE and GroupCLIP (Gorla et al., 3 Feb 2026). Related theory for multimodal contrastive learning with unpaired data shows that useful cross-modal structure can still be extracted when only part of the data are correctly matched or when additional unpaired samples are available (Nakada et al., 2023).

A fourth failure mode is geometric rather than semantic. Multi-modal contrastive models such as CLIP often embed modalities at arm’s length in the shared representation space, producing a modality gap whose magnitude is influenced by initialization, hard negatives, and temperature (Liang et al., 2022). More recent analyses argue that low-temperature InfoNCE with independent encoders can actively generate such a gap, even in a unimodal control experiment with two identical encoders (Mager et al., 12 Jul 2026).

2. Robust loss design for noisy positives and false negatives

One major response is to replace InfoNCE by losses that are explicitly robust to noisy views. The most direct example is Robust InfoNCE, or RINCE, defined for similarity scores $s=\{s^+,s_1^-,\ldots,s_K^-\}$ by

$$L^{\lambda, q}_\mathrm{RINCE}(s) = \frac{-e^{q s^+}}{q} + \frac{\left(\lambda \left(e^{s^+} + \sum_{i=1}^K e^{s_i^-}\right)\right)^q}{q}.$$

Here $q \in (0,1]$ is a symmetry or robustness controller and $\lambda>0$ is a class balancing term. As $q \to 0$, the loss recovers standard InfoNCE up to a constant; at $q=1$, it becomes fully symmetric and inherits robustness guarantees from symmetric losses for noisy binary classification (Chuang et al., 2022). The same work connects RINCE to a Wasserstein-based dependency measure, showing that minimizing the $q=1$ form maximizes a lower bound on mutual information measured by Wasserstein distance rather than KL divergence, which is presented as less sensitive to outliers and noise (Chuang et al., 2022).

A related strategy is to reweight rather than replace the contrastive objective. HeroCon combines a weighted unsupervised contrastive loss and a weighted supervised contrastive loss with a standard classification objective. Its key mechanism is automatic pairwise reweighting based on feature similarity, so that negatives that appear semantically close are penalized less aggressively. The authors provide a theoretical analysis showing that vanilla contrastive loss leads to a sub-optimal solution in the presence of false negatives, whereas the weighted losses mitigate this effect and still admit mutual-information lower-bound interpretations (Zheng et al., 2021).

Theory on weak supervision imposes a boundary on how far simple mixing can go. Under a unified framework of jointly training supervised and unsupervised contrastive losses,

$$\mathcal{L}_{\text{JointTraining}} = (1-\theta)\mathcal{L}_{\text{unsup}} + \theta\mathcal{L}_{\text{sup}},$$

clean semi-supervised labels improve the downstream error bound, but noisy labels have limited effects under this paradigm because the induced similarity graph loses class contrast as label noise increases (Cui et al., 2023). This distinguishes useful weak supervision from unreliable weak supervision: simple convex blending is theoretically sufficient in the former case but not in the latter.

3. Weak supervision, group structure, and alternative positive sets

Another line of work modifies what counts as a positive. Weakly Supervised Contrastive Learning (WCL) introduces two projection heads: one performs regular instance discrimination, while the other uses a graph-based method to explore similar samples, generate a weak label, and apply supervised contrastive learning on those pseudo-groups. A K-Nearest Neighbor based multi-crop strategy further expands the number of positive samples. On ImageNet semi-supervised learning with ResNet50, WCL achieves $65\%$ and $s=\{s^+,s_1^-,\ldots,s_K^-\}$0 Top-1 Accuracy using only $s=\{s^+,s_1^-,\ldots,s_K^-\}$1 and $s=\{s^+,s_1^-,\ldots,s_K^-\}$2 labeled examples, respectively (Zheng et al., 2021).

Some methods use structure that is not conventionally treated as a modality. In WikiScenes, hierarchical organization of cathedral components is treated as a weak modality, with positives sampled from the same node pool and negatives from sibling nodes. The triplet margin is made level-dependent,

$s=\{s^+,s_1^-,\ldots,s_K^-\}$3

so higher levels impose larger separation while deeper levels allow finer distinctions. Replay regularization revisits earlier hierarchy levels to reduce catastrophic forgetting. The reported outcome is that the hierarchical contrastive model outperforms the comparable weakly-supervised and baseline methods in downstream classification, suggesting that dataset structure itself can serve as a useful weak supervision signal (Bhalla et al., 2024).

Weak pairing replaces instance-level matching by group-level correspondence. GroupCLIP generalizes CLIP to the case where cross-modal samples share only a group label. For anchor $s=\{s^+,s_1^-,\ldots,s_K^-\}$4, positives are all samples in the other modality with the same label, and the per-anchor loss is

$s=\{s^+,s_1^-,\ldots,s_K^-\}$5

Integrated into GROOVE with an on-the-fly backtranslating autoencoder, this approach performs on par with or outperforms existing methods for downstream cross-modal matching and imputation on simulations and two real single-cell genetic perturbation datasets, and ablations identify GroupCLIP as the key component driving gains (Gorla et al., 3 Feb 2026).

Related theory under linear representations shows that a broad class of multimodal contrastive losses, including CLIP loss, can be interpreted through singular value decomposition of a contrastive cross-covariance matrix. The same framework shows that multimodal contrastive learning can outperform unimodal contrastive learning even under wrongly matched pairs, and introduces a semi-supervised loss for incorporating unpaired data by estimating pseudo-pairs from cross-modal similarity (Nakada et al., 2023).

4. Modality-aware contrast, distillation, imbalance, and missingness

In weakly supervised audio-visual violence detection, MACIL-SD explicitly avoids penalizing weak-modality snippets by abandoning rigid snippet-level synchronization. Separate unimodal bags are built for audio and visual streams; violent semi-bag representations form positive pairs, while violent semi-bags are contrasted against background and normal instances in the opposite modality as negatives. A self-distillation module transfers unimodal visual knowledge to the audio-visual model through an exponential moving average update,

$s=\{s^+,s_1^-,\ldots,s_K^-\}$6

which is reported to alleviate noises and close the semantic gap between unimodal and multimodal features. On XD-Violence, the framework outperforms previous methods with lower complexity and can be used as plug-in modules for other networks (Yu et al., 2022).

Cross-modality contrastive distillation addresses a different weak-modality regime: the target modality has limited knowledge or weak discriminative features, and labels are unavailable. CMCD first contrastively pretrains a source encoder, then distills to a target encoder using either CMD or CMC losses over paired but unlabeled samples, and finally fine-tunes or evaluates on the target modality. The central theoretical claim is that target-modality test error is bounded by the total variation distance between source and target latent feature distributions, along with complexity and sample terms; empirically, the algorithm is reported to outperform existing methods consistently by a margin of $s=\{s^+,s_1^-,\ldots,s_K^-\}$7 across image, sketch, depth map, and audio tasks (Lin et al., 2024).

Modality imbalance motivates a more radical departure from positive alignment. Multimodal Negative Learning argues that forcing weak modalities to align with dominant ones risks suppressing modality-specific information. Instead, it trains the inferior modality only to suppress non-target classes under guidance from a robust dominant modality selected dynamically by target confidence and the Unimodal Confidence Margin. The negative learning loss is

$s=\{s^+,s_1^-,\ldots,s_K^-\}$8

The framework is claimed to tighten the robustness lower bound by increasing the Unimodal Confidence Margin and to reduce empirical error of weak modalities, particularly under noisy and imbalanced scenarios (Gong et al., 23 Oct 2025). A common misconception is that helping weak modalities must mean making them imitate strong ones; this line of work instead treats selective non-target suppression as the relevant penalty.

Missing modalities create yet another weak-modality regime. In clinical multimodal fusion, improved modality dropout replaces static zero placeholders with learnable modality tokens and trains over all present–missing combinations within each iteration. Conventional unimodal contrastive objectives are augmented with fused multimodal representations, and the reported result is improved performance particularly in scenarios where only a single modality is available at inference time (Gu et al., 22 Sep 2025).

5. Modality gap, temperature, and negative sampling geometry

The modality gap literature studies the geometry induced by contrastive learning itself. One influential account attributes the gap to a combination of model initialization and optimization. At initialization, deep networks map random inputs into narrow cones; with two independently initialized encoders, the modalities begin separated. During optimization, contrastive learning keeps them apart by a distance influenced by temperature, and varying that distance can affect zero-shot classification performance and fairness (Liang et al., 2022).

A later gradient-flow analysis sharpens this explanation by identifying two concrete mechanisms: mismatched data pairs enlarge the gap early in training, and a learnable inverse temperature causes the gap to close only at the slow rate $s=\{s^+,s_1^-,\ldots,s_K^-\}$9 under the standard CLIP parameterization. Mitigation strategies include temperature scheduling, alternative temperature parameterizations, smaller learning rates for temperature, and modality swapping. Closing the gap is shown to improve image–text retrieval, but zero-shot and linear-probe image classification correlate more with feature-space uniformity than with the gap alone (Yaras et al., 2024).

A more recent diagnosis argues that the gap is a mode-failure of InfoNCE with independent encoders, but only at low temperatures. The proposed xNCE modifies the denominator by mixing intermodal and intra-modality negatives. On MS-COCO, xNCE matches retrieval performance while consistently reducing the gap even at low temperatures, and it improves zero-shot classification over the InfoNCE baseline across all benchmarks; in contrast, high-temperature InfoNCE and regularized InfoNCE reduce the gap without improving zero-shot transfer (Mager et al., 12 Jul 2026). This establishes a precise design lesson: gap mitigation can be built into negative sampling rather than appended as a separate regularizer.

6. Empirical synthesis, recurrent debates, and outlook

Across the surveyed literature, weak-modality penalized contrastive learning is not a single algorithmic template but a family of responses to different correspondence failures.

Regime	Representative method	Reported effect
Noisy views and false positives	RINCE (Chuang et al., 2022)	Consistent improvements on image, video, and graph benchmarks
Weak pseudo-labels and class collision	WCL (Zheng et al., 2021)	Better self-supervised representations; $$L^{\lambda, q}_\mathrm{RINCE}(s) = \frac{-e^{q s^+}}{q} + \frac{\left(\lambda \left(e^{s^+} + \sum_{i=1}^K e^{s_i^-}\right)\right)^q}{q}.$$0 and $$L^{\lambda, q}_\mathrm{RINCE}(s) = \frac{-e^{q s^+}}{q} + \frac{\left(\lambda \left(e^{s^+} + \sum_{i=1}^K e^{s_i^-}\right)\right)^q}{q}.$$1 ImageNet Top-1 at $$L^{\lambda, q}_\mathrm{RINCE}(s) = \frac{-e^{q s^+}}{q} + \frac{\left(\lambda \left(e^{s^+} + \sum_{i=1}^K e^{s_i^-}\right)\right)^q}{q}.$$2 and $$L^{\lambda, q}_\mathrm{RINCE}(s) = \frac{-e^{q s^+}}{q} + \frac{\left(\lambda \left(e^{s^+} + \sum_{i=1}^K e^{s_i^-}\right)\right)^q}{q}.$$3 labels
Weakly paired multimodal groups	GROOVE / GroupCLIP (Gorla et al., 3 Feb 2026)	Strong matching and imputation under group-level correspondence
Weak target modality transfer	CMCD (Lin et al., 2024)	$$L^{\lambda, q}_\mathrm{RINCE}(s) = \frac{-e^{q s^+}}{q} + \frac{\left(\lambda \left(e^{s^+} + \sum_{i=1}^K e^{s_i^-}\right)\right)^q}{q}.$$4 gains across modalities and tasks
Modality imbalance	MNL (Gong et al., 23 Oct 2025)	Improved robustness and reduced weak-modality error under noise
Missing modalities	Improved modality dropout (Gu et al., 22 Sep 2025)	Better performance when only a single modality is available

Several debates recur. One concerns whether alignment is always the correct goal. MACIL-SD and GroupCLIP retain alignment but redefine it at event or group level (Yu et al., 2022, Gorla et al., 3 Feb 2026); MNL rejects all-class imitation and supervises only non-target suppression (Gong et al., 23 Oct 2025). A second concerns whether weak supervision is uniformly beneficial. Clean semi-supervised labels improve contrastive learning, but noisy labels do not necessarily help under naive joint training (Cui et al., 2023). A third concerns modality gap: one body of work shows that gap magnitude can influence fairness and zero-shot behavior in task-dependent ways (Liang et al., 2022), whereas other work shows that particular gap-reduction mechanisms, such as xNCE or theoretically guided temperature control, can improve retrieval and sometimes zero-shot transfer (Yaras et al., 2024, Mager et al., 12 Jul 2026).

A plausible implication is that future progress will depend less on choosing between “contrastive” and “non-contrastive” learning than on specifying which correspondence assumptions are credible in a given dataset: instance pairing, group membership, hierarchy, temporal synchronization, or cross-modal teacher–student transfer. The surveyed work suggests that penalization is most effective when it is tied to that assumption explicitly—through symmetry, weighting, structured positive sets, negative-class suppression, or geometry-aware sampling—rather than imposed as a uniform correction after the fact.