Generalized Contrastive Learning (GCL)

• Generalized Contrastive Learning is a framework that extends standard binary contrastive methods by introducing structured similarity targets, such as affinity tensors and soft label distributions.
• It unifies diverse objectives—from supervised metric learning to multimodal retrieval—by replacing hard pair labels with continuous or soft similarity measures.
• The approach is applied across various domains, improving tasks like speaker verification, image classification, and universal multimodal retrieval with significant performance gains.

Generalized Contrastive Learning (GCL) denotes a family of contrastive objectives that extend the binary positive/negative supervision of standard contrastive learning by introducing richer target structure. In published work, the term has been used for an affinity-tensor formulation that unifies supervised metric learning and unsupervised contrastive learning (Inoue et al., 2020), a generalized supervised contrastive loss that matches latent similarity to soft label similarity (Kim et al., 2022), a weighted contrastive framework for ranked multi-modal retrieval (Zhu et al., 2024), and a generalized InfoNCE objective for universal retrieval across image, text, and fused image-text modalities (Lee et al., 30 Sep 2025). A plausible implication is that GCL is best understood less as a single canonical loss than as a recurring design pattern: replace hard pair labels with structured targets that encode more of the relevant supervisory geometry.

1. Core concept and principal formulations

Standard contrastive learning typically assumes a binary supervision pattern: matched pairs are positives, unmatched pairs are negatives, and training optimizes a softmax-style discrimination objective. The formulations grouped under GCL alter that assumption in different ways. Some generalize the target relation itself, some generalize the supervision signal from one-hot labels to distributions, and some generalize the set of modalities or the relevance signal attached to each pair.

Formulation	Generalized object	Reported setting
Generalized contrastive loss (Inoue et al., 2020)	Affinity tensor $\alpha_{ij}^{kl}$ over pair relations	Speaker verification; supervised, semi-supervised, unsupervised
Generalized supervised contrastive loss (Kim et al., 2022)	Cross-entropy between label similarity $S(i)$ and latent similarity $P(i)$	Image classification with MixUp, CutMix, and knowledge distillation
GCL for retrieval and ranking (Zhu et al., 2024)	Score-derived pair weight $w_i = \mathrm{STW}(s_i)$	Multi-modal retrieval and ranking
GCL for universal multimodal retrieval (Lee et al., 30 Sep 2025)	Generalized InfoNCE over $M=\{i,t,it\}$ and six positive directions	Image, text, and fused image-text retrieval

These formulations share a common departure from conventional InfoNCE or CLIP-style training. Rather than treating supervision as a binary mask over a single modality pair type, they introduce a more expressive target object: an affinity tensor, a soft similarity distribution, a continuous relevance weight, or a modality-complete positive-pair graph. This commonality is structural rather than nominal.

2. Affinity-tensor GCL and the semi-supervised formulation

The 2020 generalized contrastive loss formalizes GCL through a fourth-order affinity tensor $\alpha=\{\alpha_{ij}^{kl}\}$, where positive values encourage similarity, negative values encourage dissimilarity, and zeros impose no constraint (Inoue et al., 2020). Given a mini-batch

$Z=\{z_i^k : i=1,\dots,N;\; k=1,2\},$

the loss is written as

$$L_\alpha = \frac{1}{2N} \sum_{i=1}^N \sum_{k=1}^2 \left\{ \frac{ \sum_{j=1}^N \sum_{l=1}^2 \langle \alpha_{ij}^{kl}\rangle \,\tilde s(z_i^k,z_j^l) }{ \sum_{j=1}^N \sum_{l=1}^2 |\alpha_{ij}^{kl}|\,\tilde s(z_i^k,z_j^l)+\epsilon } \right\},$$

where $\langle a\rangle=\max(0,a)$, $\epsilon\to 0^+$, and the raw similarity can be either Euclidean-based,

$S(i)$0

or cosine-based,

$S(i)$1

The central claim of this formulation is unification. By choosing different affinity-tensor patterns and appropriate similarity and normalization functions, the same framework recovers contrastive loss, triplet loss, N-pair or prototypical losses, and NT-Xent (Inoue et al., 2020). In that sense, the generalization is algebraic: the loss does not specialize to one pretext task, but parameterizes a family of metric-learning and contrastive objectives through $S(i)$2.

The same paper extends the construction to semi-supervised learning by combining labeled and unlabeled samples in one batch, $S(i)$3, and lifting the affinity object to a sixth-order tensor $S(i)$4, with $S(i)$5 indicating labeled versus unlabeled partitions (Inoue et al., 2020). A simple choice assigns supervised Type-4 structure within $S(i)$6, unsupervised Type-4 structure within $S(i)$7, and negative affinities across labeled-unlabeled pairs. The authors emphasize that the GCL formula itself is unchanged; only the affinity specification changes. This is the paper’s strongest sense of “generalized”: supervised, semi-supervised, and unsupervised regimes are handled without redefining the loss.

The reported application is text-independent speaker verification on VoxCeleb with 40-dimensional log-mel features and a ResNet-18 embedding network (Inoue et al., 2020). The key Equal Error Rates are $S(i)$8 in the supervised setting, $S(i)$9 in the unsupervised setting, and $P(i)$0 in the semi-supervised setting. These results support the paper’s claim that a single contrastive formalism can span three supervision regimes.

3. Soft-label geometry in generalized supervised contrastive learning

A different generalization appears in generalized supervised contrastive learning, where the conventional positive/negative partition of supervised contrastive learning is replaced by a soft distribution over pairwise label similarities (Kim et al., 2022). The setting assumes a minibatch of size $P(i)$1 that produces $P(i)$2 augmented or mixed views $P(i)$3, with $P(i)$4 a class-probability vector. An encoder $P(i)$5 yields embeddings $P(i)$6, and a two-layer projection head $P(i)$7 produces normalized vectors $P(i)$8, $P(i)$9.

For anchor $w_i = \mathrm{STW}(s_i)$0, latent similarity is defined by a softmax over dot products,

$w_i = \mathrm{STW}(s_i)$1

while label similarity is the cosine similarity of soft labels,

$w_i = \mathrm{STW}(s_i)$2

The generalized supervised contrastive loss is then

$w_i = \mathrm{STW}(s_i)$3

This construction generalizes standard supervised contrastive learning because, under one-hot labels, $w_i = \mathrm{STW}(s_i)$4 for same-class pairs and $w_i = \mathrm{STW}(s_i)$5 otherwise, so the generalized loss collapses exactly to the standard supervised contrastive loss (Kim et al., 2022). The paper explicitly states that this permits direct integration of methods that rely on probabilistic label structure, including MixUp, CutMix, and knowledge distillation.

The full GenSCL training procedure adds teacher similarity $w_i = \mathrm{STW}(s_i)$6 from an arbitrary teacher classifier $w_i = \mathrm{STW}(s_i)$7, leading to per-anchor loss

$w_i = \mathrm{STW}(s_i)$8

after which the projector $w_i = \mathrm{STW}(s_i)$9 is discarded and a linear classifier is trained on the frozen encoder features (Kim et al., 2022). On CIFAR-10 and CIFAR-100, the setup uses a ResNet-50 encoder for $M=\{i,t,it\}$0 epochs with batch size $M=\{i,t,it\}$1, $M=\{i,t,it\}$2, SGD, RandAugment, MixUp or CutMix, and a pre-trained PyramidNet-272 teacher with $M=\{i,t,it\}$3. On ImageNet-1K, the model is trained from scratch with ResNet-50 and the Momentum Contrast framework for $M=\{i,t,it\}$4 epochs, batch size $M=\{i,t,it\}$5 GPUs, $M=\{i,t,it\}$6, and a ConvNeXt-Large teacher (Kim et al., 2022).

The reported quantitative gains are substantial. For a ResNet-50 encoder, CIFAR-10 improves from SupCon $M=\{i,t,it\}$7 to GenSCL with CutMix $M=\{i,t,it\}$8 to $M=\{i,t,it\}$9 with knowledge distillation; CIFAR-100 improves from $\alpha=\{\alpha_{ij}^{kl}\}$0 to $\alpha=\{\alpha_{ij}^{kl}\}$1 to $\alpha=\{\alpha_{ij}^{kl}\}$2; and ImageNet improves from SupCon+MoCo $\alpha=\{\alpha_{ij}^{kl}\}$3 to $\alpha=\{\alpha_{ij}^{kl}\}$4 to $\alpha=\{\alpha_{ij}^{kl}\}$5 (Kim et al., 2022). The paper additionally reports that CutMix yields larger gains than MixUp, that the best $\alpha=\{\alpha_{ij}^{kl}\}$6 is around $\alpha=\{\alpha_{ij}^{kl}\}$7, that the gradient-mining term $\alpha=\{\alpha_{ij}^{kl}\}$8 remains nonzero longer under mixing, and that t-SNE plots show tighter same-class clusters and better separation.

4. Continuous relevance weighting for retrieval and ranking

In multi-modal retrieval and ranking, GCL generalizes contrastive learning by replacing binary relevance with continuous ranking scores (Zhu et al., 2024). The paper starts from a CLIP-style setup in which a batch contains $\alpha=\{\alpha_{ij}^{kl}\}$9 pairs $Z=\{z_i^k : i=1,\dots,N;\; k=1,2\},$0. Conventional training assumes each diagonal pair is fully positive and all off-diagonal pairs are negative. GCL instead uses triplets $Z=\{z_i^k : i=1,\dots,N;\; k=1,2\},$1, where $Z=\{z_i^k : i=1,\dots,N;\; k=1,2\},$2 is derived from a score $Z=\{z_i^k : i=1,\dots,N;\; k=1,2\},$3 through a Score-to-Weight function.

With normalized query and document embeddings $Z=\{z_i^k : i=1,\dots,N;\; k=1,2\},$4 and $Z=\{z_i^k : i=1,\dots,N;\; k=1,2\},$5, and similarity matrix $Z=\{z_i^k : i=1,\dots,N;\; k=1,2\},$6, the weighted contrastive loss is

$Z=\{z_i^k : i=1,\dots,N;\; k=1,2\},$7

The paper evaluates five score-to-weight mappings: Constant, Linear, Inverse, Inverse-sqrt, and Piecewise (Zhu et al., 2024). Inverse is reported as the default and Piecewise as especially effective when optimizing NDCG@10.

The same framework is extended to multi-field inputs. Each left-hand-side and right-hand-side field is encoded independently, normalized, and combined by weighted averaging,

$Z=\{z_i^k : i=1,\dots,N;\; k=1,2\},$8

with main similarity $Z=\{z_i^k : i=1,\dots,N;\; k=1,2\},$9 and auxiliary cross-field similarities $$L_\alpha = \frac{1}{2N} \sum_{i=1}^N \sum_{k=1}^2 \left\{ \frac{ \sum_{j=1}^N \sum_{l=1}^2 \langle \alpha_{ij}^{kl}\rangle \,\tilde s(z_i^k,z_j^l) }{ \sum_{j=1}^N \sum_{l=1}^2 |\alpha_{ij}^{kl}|\,\tilde s(z_i^k,z_j^l)+\epsilon } \right\},$$0. The final loss is

$$L_\alpha = \frac{1}{2N} \sum_{i=1}^N \sum_{k=1}^2 \left\{ \frac{ \sum_{j=1}^N \sum_{l=1}^2 \langle \alpha_{ij}^{kl}\rangle \,\tilde s(z_i^k,z_j^l) }{ \sum_{j=1}^N \sum_{l=1}^2 |\alpha_{ij}^{kl}|\,\tilde s(z_i^k,z_j^l)+\epsilon } \right\},$$1

This explicitly trains title-only, image-only, and cross-modal interactions rather than collapsing them into one opaque fusion stage (Zhu et al., 2024).

The empirical setting uses a large-scale corpus of $$L_\alpha = \frac{1}{2N} \sum_{i=1}^N \sum_{k=1}^2 \left\{ \frac{ \sum_{j=1}^N \sum_{l=1}^2 \langle \alpha_{ij}^{kl}\rangle \,\tilde s(z_i^k,z_j^l) }{ \sum_{j=1}^N \sum_{l=1}^2 |\alpha_{ij}^{kl}|\,\tilde s(z_i^k,z_j^l)+\epsilon } \right\},$$2 million query-document pairs with ranking scores, generated from Google Shopping results for GPT-4-produced queries over $$L_\alpha = \frac{1}{2N} \sum_{i=1}^N \sum_{k=1}^2 \left\{ \frac{ \sum_{j=1}^N \sum_{l=1}^2 \langle \alpha_{ij}^{kl}\rangle \,\tilde s(z_i^k,z_j^l) }{ \sum_{j=1}^N \sum_{l=1}^2 |\alpha_{ij}^{kl}|\,\tilde s(z_i^k,z_j^l)+\epsilon } \right\},$$3 fashion and homeware leaf categories (Zhu et al., 2024). The continuous score is $$L_\alpha = \frac{1}{2N} \sum_{i=1}^N \sum_{k=1}^2 \left\{ \frac{ \sum_{j=1}^N \sum_{l=1}^2 \langle \alpha_{ij}^{kl}\rangle \,\tilde s(z_i^k,z_j^l) }{ \sum_{j=1}^N \sum_{l=1}^2 |\alpha_{ij}^{kl}|\,\tilde s(z_i^k,z_j^l)+\epsilon } \right\},$$4, yielding $$L_\alpha = \frac{1}{2N} \sum_{i=1}^N \sum_{k=1}^2 \left\{ \frac{ \sum_{j=1}^N \sum_{l=1}^2 \langle \alpha_{ij}^{kl}\rangle \,\tilde s(z_i^k,z_j^l) }{ \sum_{j=1}^N \sum_{l=1}^2 |\alpha_{ij}^{kl}|\,\tilde s(z_i^k,z_j^l)+\epsilon } \right\},$$5. The train/evaluation protocol includes in-domain, novel-query, novel-corpus, and zero-shot splits. The paper reports that GCL achieves a $$L_\alpha = \frac{1}{2N} \sum_{i=1}^N \sum_{k=1}^2 \left\{ \frac{ \sum_{j=1}^N \sum_{l=1}^2 \langle \alpha_{ij}^{kl}\rangle \,\tilde s(z_i^k,z_j^l) }{ \sum_{j=1}^N \sum_{l=1}^2 |\alpha_{ij}^{kl}|\,\tilde s(z_i^k,z_j^l)+\epsilon } \right\},$$6 increase in NDCG@10 for in-domain evaluations and $$L_\alpha = \frac{1}{2N} \sum_{i=1}^N \sum_{k=1}^2 \left\{ \frac{ \sum_{j=1}^N \sum_{l=1}^2 \langle \alpha_{ij}^{kl}\rangle \,\tilde s(z_i^k,z_j^l) }{ \sum_{j=1}^N \sum_{l=1}^2 |\alpha_{ij}^{kl}|\,\tilde s(z_i^k,z_j^l)+\epsilon } \right\},$$7 to $$L_\alpha = \frac{1}{2N} \sum_{i=1}^N \sum_{k=1}^2 \left\{ \frac{ \sum_{j=1}^N \sum_{l=1}^2 \langle \alpha_{ij}^{kl}\rangle \,\tilde s(z_i^k,z_j^l) }{ \sum_{j=1}^N \sum_{l=1}^2 |\alpha_{ij}^{kl}|\,\tilde s(z_i^k,z_j^l)+\epsilon } \right\},$$8 increases for cold-start evaluations compared to the finetuned CLIP baseline with MarqoGS-10M (Zhu et al., 2024). In the detailed GSFull-10M results, CLIP (ViT-L/14) improves from $$L_\alpha = \frac{1}{2N} \sum_{i=1}^N \sum_{k=1}^2 \left\{ \frac{ \sum_{j=1}^N \sum_{l=1}^2 \langle \alpha_{ij}^{kl}\rangle \,\tilde s(z_i^k,z_j^l) }{ \sum_{j=1}^N \sum_{l=1}^2 |\alpha_{ij}^{kl}|\,\tilde s(z_i^k,z_j^l)+\epsilon } \right\},$$9 to $\langle a\rangle=\max(0,a)$0 NDCG@10, with ERR boosted by $\langle a\rangle=\max(0,a)$1 (Zhu et al., 2024).

The significance of this formulation is architectural as much as objective-level. The paper frames GCL as a route to a single-stage retrieval system that learns both relevance and ranking, reducing dependence on a separate re-ranker (Zhu et al., 2024).

5. Generalized InfoNCE for universal multimodal retrieval

A further generalization is directed at universal multimodal retrieval, where the retrieval database may contain text-only entries, image-only entries, and fused image+text entries such as Wikipedia pages (Lee et al., 30 Sep 2025). Rather than curating new triplet datasets for each query-to-candidate modality combination, the method uses only off-the-shelf image-caption pairs to learn a unified space over three modalities:

1. image embeddings $\langle a\rangle=\max(0,a)$2,
2. text embeddings $\langle a\rangle=\max(0,a)$3,
3. fused image+text embeddings $\langle a\rangle=\max(0,a)$4.

Let a mini-batch contain $\langle a\rangle=\max(0,a)$5 paired samples $\langle a\rangle=\max(0,a)$6, and define $\langle a\rangle=\max(0,a)$7 and the positive modality-pair set

$\langle a\rangle=\max(0,a)$8

The generalized loss is

$\langle a\rangle=\max(0,a)$9

Compared with standard CLIP-style loss, which trains only $\epsilon\to 0^+$0 and $\epsilon\to 0^+$1, this formulation explicitly includes fused image-text as both query and candidate modality and expands the negative pool to all samples from all three modalities (Lee et al., 30 Sep 2025).

The implementation computes all $\epsilon\to 0^+$2 similarity maps among image, text, and fused embeddings, masks self-pairs on identical-modality diagonals, concatenates positives with all negatives, and applies cross-entropy with targets $\epsilon\to 0^+$3 (Lee et al., 30 Sep 2025). The fused embedding is model-dependent: for example, VISTA uses a joint encoder, whereas UniIR can use $\epsilon\to 0^+$4. The training data is LLaVA Visual Instruct Pretrain LCS-558K, and evaluation uses M-BEIR, MMEB, and CoVR on pretrained backbones including VISTA, CLIP + score-fusion, and TinyCLIP + score-fusion (Lee et al., 30 Sep 2025).

The reported gains are strongest on tasks involving fused modalities. On global M-BEIR Recall@50 with VISTA, performance improves from pretrained $\epsilon\to 0^+$5 to standard pairwise CL $\epsilon\to 0^+$6 to GCL $\epsilon\to 0^+$7; with CLIP-SF, performance improves from $\epsilon\to 0^+$8 to $\epsilon\to 0^+$9 to $S(i)$00 (Lee et al., 30 Sep 2025). On MMEB Recall@1, VISTA improves from $S(i)$01 to $S(i)$02, and CLIP-SF from $S(i)$03 to $S(i)$04. On CoVR R@1, VISTA improves from $S(i)$05 to $S(i)$06, while CLIP-SF improves from $S(i)$07 to $S(i)$08. An ablation on global M-BEIR shows that removing any of the three blocks—cross-modal $S(i)$09, $S(i)$10 as candidate, or $S(i)$11 as query—reduces average Recall@50 from $S(i)$12 to the $S(i)$13–$S(i)$14 range (Lee et al., 30 Sep 2025).

The theoretical intuition given in the paper is that GCL “fills in” the embedding gaps among $S(i)$15, $S(i)$16, and $S(i)$17, turning any semantic pair from any modality combination into a positive (Lee et al., 30 Sep 2025). This suggests a broader notion of generalization than in the supervised-label and ranking-weighted variants: the generalized object is the modality graph itself.

6. Terminological ambiguity and adjacent generalization programs

One recurring source of confusion is that “GCL” is also the dominant abbreviation for Graph Contrastive Learning. In that literature, the acronym does not denote Generalized Contrastive Learning but a self-supervised graph representation paradigm. Several graph papers nevertheless pursue goals that are adjacent to generalized contrastive learning in the broader sense of unifying objectives, controlling generalization, or enlarging the structure available to the loss.

UGCL defines contextual representations as $S(i)$18, where the power $S(i)$19 acts as a tunable contextual scope, and the paper states that the architecture can be considered as a general framework to unify existing GCL methods (Zheng et al., 2022). InfoAdv introduces GCL-GE, derives a mutual-information upper bound on graph-contrastive generalization error, and argues that reducing $S(i)$20 lowers that bound, in direct contradiction to the pure InfoMax principle (Yuan et al., 2022). AS-GCL introduces spectral-based augmentation, parameter-sharing asymmetric encoders with distinct diffusion operators, and an upper-bound loss that maintains a balanced distribution of intra- and inter-class distance, reaching average node-classification accuracy $S(i)$21 versus $S(i)$22 for the best prior SOTA (Liu et al., 19 Feb 2025). Khan-GCL combines a Kolmogorov-Arnold Network encoder with hard negatives generated from coefficient-derived critical features, reporting average ROC-AUC $S(i)$23 in transfer learning and average accuracy $S(i)$24 in unsupervised evaluation (Wang et al., 21 May 2025). “Less is More” argues that original node features and graph structure naturally provide two complementary views, and proposes a GCN-MLP model requiring neither data augmentation nor negative sampling, with strong results on heterophilic graphs and low computational overhead (Zhao et al., 30 Sep 2025).

A second ambiguity concerns the meaning of “generalized.” In some papers it means a loss that algebraically subsumes earlier objectives, as with the affinity-tensor formulation (Inoue et al., 2020). In others it means replacing one-hot or binary targets with soft distributions, as in generalized supervised contrastive learning (Kim et al., 2022). In retrieval work it means weighting contrastive terms by continuous relevance or expanding the modality set from two to three elements (Zhu et al., 2024, Lee et al., 30 Sep 2025). In the broader self-supervised literature, the proposed Generalized Learning Framework decomposes self-supervised contrastive learning into an aligning part and a constraining part, unifying BYOL, Barlow Twins, and SwAV, and introduces Adaptive Distribution Calibration as a plug-and-play constraint (Si et al., 19 Aug 2025). This suggests that “generalized” has become a family resemblance term in contrastive learning: it can refer to unification, richer supervision, expanded modality structure, or improved transfer behavior, depending on the problem formulation.

A common misconception is that generalized contrastive learning is synonymous with “more negatives” or “harder negatives.” The surveyed work does not support that simplification. Some variants indeed redesign the negative pool, such as the all-modality denominator in universal multimodal retrieval (Lee et al., 30 Sep 2025), but others generalize the positive target distribution instead of the negatives (Kim et al., 2022), and some graph methods explicitly remove negative sampling altogether (Zhao et al., 30 Sep 2025). Another misconception is that generalized objectives are necessarily supervised. The affinity-tensor formulation was explicitly designed to cover supervised, semi-supervised, and unsupervised learning with no change in the definition of the loss function (Inoue et al., 2020).

Taken together, the literature portrays Generalized Contrastive Learning as a technically heterogeneous but conceptually coherent movement away from binary pair supervision toward structured similarity targets. Whether instantiated through affinity tensors, soft label geometry, continuous ranking weights, or modality-complete contrastive graphs, the central objective is the same: align representation-space geometry with richer forms of task-relevant structure than standard contrastive learning can express.