Positive-Margin Contrastive Learning

• Positive-Margin Contrastive Learning is a framework that enforces explicit margins on positive pairs, preserving semantic gaps induced by data augmentations.
• It employs techniques such as additive cosine/angular margins and adaptive margin schedules to reshape gradients and enhance model separability.
• The approach has demonstrated empirical gains across applications like speaker verification, sentence embeddings, and medical imaging by refining contrastive objectives.

Positive-Margin Contrastive Learning denotes a family of contrastive objectives that do not merely maximize anchor–positive similarity relative to negatives, but explicitly impose a margin, preserve an augmentation-induced positive distance, or calibrate a certified positive neighborhood around the anchor. In the simplest form, the positive logit is made harder than in standard InfoNCE by subtracting an additive cosine margin or adding an angular margin; in broader formulations, the method preserves an “innate” semantic margin between non-identical positives, introduces prototype-level angular margins, learns pair-specific subspaces that act as a margin-like buffer, or calibrates a geometric set whose threshold induces a sample-specific positive margin (Lepage et al., 2023, Xiao et al., 2023, Liu et al., 2021, Alkhatib et al., 27 Mar 2026, Rho et al., 2023).

1. Formal definition and objective families

Standard contrastive learning with cosine similarity and temperature $\tau$ uses an InfoNCE or NT-Xent objective of the form

$$\mathcal{L}=-\log \frac{\exp(s_{ap}/\tau)}{\exp(s_{ap}/\tau)+\sum_n \exp(s_{an}/\tau)},$$

where $s_{ap}$ is the anchor–positive similarity and $s_{an}$ are anchor–negative similarities. Positive-margin variants modify the positive term while often leaving negatives unchanged. In additive-margin form, the positive logit becomes $s^{AM}_{ap}=\cos\theta_{ap}-m$; in additive-angular-margin form, it becomes $s^{AAM}_{ap}=\cos(\theta_{ap}+m)$ (Lepage et al., 2023, Lepage et al., 2024).

A more general formulation writes the contrastive logit as

$$\delta_{ij}=\frac{\cos(\theta_{ij}+m_1 p_{ij})-m_2 p_{ij}}{\tau},$$

with $p_{ij}$ indicating whether $j$ is a positive for anchor $i$. Here $$\mathcal{L}=-\log \frac{\exp(s_{ap}/\tau)}{\exp(s_{ap}/\tau)+\sum_n \exp(s_{an}/\tau)},$$0 is an additive angular margin and $$\mathcal{L}=-\log \frac{\exp(s_{ap}/\tau)}{\exp(s_{ap}/\tau)+\sum_n \exp(s_{an}/\tau)},$$1 is a subtractive cosine or logit margin applied only to positives. The resulting loss retains the cross-entropy structure of contrastive learning while changing the positive gradient geometry (Rho et al., 2023).

Not all positive-margin methods use a fixed scalar $$\mathcal{L}=-\log \frac{\exp(s_{ap}/\tau)}{\exp(s_{ap}/\tau)+\sum_n \exp(s_{an}/\tau)},$$2. In the IFTCL framework, the central object is an augmentation-aware semantic margin

$$\mathcal{L}=-\log \frac{\exp(s_{ap}/\tau)}{\exp(s_{ap}/\tau)+\sum_n \exp(s_{an}/\tau)},$$3

together with a Twins Loss

$$\mathcal{L}=-\log \frac{\exp(s_{ap}/\tau)}{\exp(s_{ap}/\tau)+\sum_n \exp(s_{an}/\tau)},$$4

which preserves the innate difference between “identical twins” and “fraternal twins” rather than collapsing both positives to the same similarity (Xiao et al., 2023).

Other formulations move from scalar margins to geometric neighborhoods. Contrastive Conformal Sets define a learnable set

$$\mathcal{L}=-\log \frac{\exp(s_{ap}/\tau)}{\exp(s_{ap}/\tau)+\sum_n \exp(s_{an}/\tau)},$$5

and then a sample-specific positive margin

$$\mathcal{L}=-\log \frac{\exp(s_{ap}/\tau)}{\exp(s_{ap}/\tau)+\sum_n \exp(s_{an}/\tau)},$$6

Under split conformal calibration, this yields a distribution-free coverage guarantee $$\mathcal{L}=-\log \frac{\exp(s_{ap}/\tau)}{\exp(s_{ap}/\tau)+\sum_n \exp(s_{an}/\tau)},$$7 under exchangeability, so positive-margin contrastive learning is interpreted as certified inclusion rather than only relative logit separation (Alkhatib et al., 27 Mar 2026).

A distinct line introduces max-margin objectives directly. In multi-modal long-tail learning, the bi-directional max-margin loss enforces

$$\mathcal{L}=-\log \frac{\exp(s_{ap}/\tau)}{\exp(s_{ap}/\tau)+\sum_n \exp(s_{an}/\tau)},$$8

and replaces the constant margin by a sample-specific schedule $$\mathcal{L}=-\log \frac{\exp(s_{ap}/\tau)}{\exp(s_{ap}/\tau)+\sum_n \exp(s_{an}/\tau)},$$9, where $s_{ap}$0 depends on local density estimated from text clusters (Sheludzko et al., 9 Mar 2026).

2. Gradient mechanisms and optimization effects

The principal analytical claim in the gradient-centered treatment of margins is that margins matter less because they shift a putative decision boundary and more because they reshape gradients on positives. For generalized contrastive learning, the logit derivative is

$s_{ap}$1

and, with angles,

$s_{ap}$2

This decomposition isolates a geometric angle multiplier and a probability or logit multiplier. The reported conclusion is that emphasizing positive samples and scaling gradients depending on positive sample angles and logits are the keys to improving the generalization performance of contrastive learning in both seen and unseen datasets, while other factors only marginally improve performance (Rho et al., 2023).

In additive-margin speaker verification, the same phenomenon appears in a simpler form. If $s_{ap}$3, then

$s_{ap}$4

Because the positive logit is reduced by $s_{ap}$5, the positive target becomes harder and the gradient magnitude on positives increases. The paper states that this accelerates tightening of intra-speaker clusters. For additive angular margin, the derivative additionally contains the factor $s_{ap}$6, which can amplify gradients and destabilize training when $s_{ap}$7 is too large (Lepage et al., 2023).

The optimization implications are not uniformly beneficial across implementations. In Contrastive Forward-Forward learning, the positive-pair margin is sometimes injected by clamping the similarity as $s_{ap}$8. That formulation truncates gradients when $s_{ap}$9. The same study proves that subtracting the margin after the log-probability,

$s_{an}$0

is gradient-neutral under the mean-over-positives reduction. On CIFAR-10, clamping produced $s_{an}$1 higher pooled test-accuracy variance with no difference in mean accuracy, but the effect did not transfer cleanly to CIFAR-100, SVHN, or Fashion-MNIST, making margin implementation itself a dataset-dependent source of training variance (Steier, 1 Mar 2026).

3. Principal design patterns

The literature now contains several distinct mechanisms that all instantiate a “positive-margin” principle, although they differ in whether the margin is explicit, adaptive, semantic, or certified.

Family	Representative formulation	Example papers
Positive-logit margin	$s_{an}$2 or $s_{an}$3 on positives	(Lepage et al., 2023, Lepage et al., 2024, Sheng et al., 2022, Liu et al., 2021)
Semantic-margin preservation	Match learned positive-distance gap to augmentation-induced $s_{an}$4	(Xiao et al., 2023)
Adaptive or scheduled margin	Pair- or sample-specific margin from teacher similarity or density	(Nguyen et al., 2024, Sheludzko et al., 9 Mar 2026)
Certified geometric margin	$s_{an}$5 from conformal set calibration	(Alkhatib et al., 27 Mar 2026)
Implicit margin-like support	Weighted soft positives or learned subspaces enlarge a positive buffer	(Ge et al., 2023, Wu et al., 2024)

The most direct family is the additive cosine or additive angular margin inserted into InfoNCE-style training. This appears in self-supervised speaker verification, multi-view medical image prediction, and prototype-based domain adaptation, with the common rule that only the positive term is altered and negatives remain unchanged (Lepage et al., 2023, Sheng et al., 2022, Liu et al., 2021).

A second family preserves non-zero positive distances instead of forcing all positives toward maximal similarity. IFTCL treats dropout-generated pairs as “identical twins” and cross-lingual fusion pairs as “fraternal twins,” then enforces the difference between their similarities to match an augmentation-aware input-level margin. This directly targets over-collapse of semantically non-identical positives (Xiao et al., 2023).

A third family makes the margin adaptive. KDMCSE uses teacher similarities to compute $s_{an}$6 and applies a pair-dependent angular margin to negatives through $s_{an}$7 after filtering suspicious negatives. MM-TS instead modulates either the temperature or the max-margin parameter using a cosine schedule plus a density-dependent shift derived from cluster size (Nguyen et al., 2024, Sheludzko et al., 9 Mar 2026).

Finally, some methods are not explicit margin losses but admit a margin interpretation. In SNCLR, each soft neighbor weight $s_{an}$8 multiplies a positive numerator term, which is exactly equivalent to an additive logit shift $s_{an}$9. In Hydra, no explicit margin is introduced, but the learned pair-specific subspace reduces interference from irrelevant features and increases positive similarity within that subspace, acting as a margin-like buffer without requiring a global margin parameter (Ge et al., 2023, Wu et al., 2024).

4. Positive construction, negatives, and representation geometry

Positive-margin contrastive learning is inseparable from how positives are defined. In speaker verification, the standard self-supervised construction treats two augmented, non-overlapping segments from the same utterance as a positive pair and all segments from other utterances in the batch as negatives, implicitly assuming different utterances correspond to different speakers; the same work reports that occasional class-collision has negligible impact on convergence. Its symmetric SNT-Xent formulation concatenates both augmented views into a set of size $s^{AM}_{ap}=\cos\theta_{ap}-m$0, doubles the number of positive anchors, and compares each anchor to $s^{AM}_{ap}=\cos\theta_{ap}-m$1 batch negatives (Lepage et al., 2023).

Sentence representation learning expanded the positive concept further. IFTCL distinguishes minimally distorted positives from semantically related but linguistically more divergent positives, then preserves a learned interval between them. This redefines positive-margin learning as stratified positive geometry rather than only positive-versus-negative separation (Xiao et al., 2023).

Supervised and domain-adaptive variants often replace instance-level positives by structured anchors. MMCon uses a T1 image as anchor, the T2, CT, and Dose images of the same patient as positives, and all views from other patients as negatives. MPSCL uses domain-adaptive class prototypes, so the positive pair for a source or target pixel is the pixel feature and its class prototype, with an additive angular margin applied to the positive prototype logit. In both cases, the margin is imposed on a richer geometry than single-instance augmentation pairs (Sheng et al., 2022, Liu et al., 2021).

The handling of negatives is equally variable. KDMCSE argues that multimodal sentence embedding suffers from suspicious and noisy negative examples, so it uses frozen CLIP similarities to build threshold masks and exclude negatives before applying adaptive angular margins to the remaining ones. By contrast, self-supervised speaker verification explicitly uses only batch negatives and reports that no memory bank or momentum queue is needed in the symmetric formulation. Contrastive Conformal Sets are more radical still: they establish that minimum-volume positive-covering sets can serve as a proxy for negative exclusion even when negative pairs are unavailable (Nguyen et al., 2024, Lepage et al., 2023, Alkhatib et al., 27 Mar 2026).

These designs imply different notions of “margin.” In some systems it is a stricter positive logit; in others it is an augmentation-induced semantic interval, a prototype-centered angular cushion, a learned subspace, or a calibrated positive neighborhood. The common structural feature is that positive relationships are not all treated as identical.

5. Empirical behavior across application domains

The most extensive quantitative evidence comes from speaker verification. In "Experimenting with Additive Margins for Contrastive Self-Supervised Speaker Verification" (Lepage et al., 2023), the NT-Xent baseline on VoxCeleb1 test with Thin ResNet-34 obtained $s^{AM}_{ap}=\cos\theta_{ap}-m$2 EER and $s^{AM}_{ap}=\cos\theta_{ap}-m$3 minDCF, while symmetric SNT-Xent improved this to $s^{AM}_{ap}=\cos\theta_{ap}-m$4 EER and $s^{AM}_{ap}=\cos\theta_{ap}-m$5 minDCF. With additive margin, $s^{AM}_{ap}=\cos\theta_{ap}-m$6 achieved $s^{AM}_{ap}=\cos\theta_{ap}-m$7 EER and $s^{AM}_{ap}=\cos\theta_{ap}-m$8 minDCF; with additive angular margin, $s^{AM}_{ap}=\cos\theta_{ap}-m$9 yielded $s^{AAM}_{ap}=\cos(\theta_{ap}+m)$0 EER and $s^{AAM}_{ap}=\cos(\theta_{ap}+m)$1 minDCF. In the larger final model, SNT-Xent-AM reached $s^{AAM}_{ap}=\cos(\theta_{ap}+m)$2 EER and $s^{AAM}_{ap}=\cos(\theta_{ap}+m)$3 minDCF, and the difference between mean positive and mean negative scores increased from $s^{AAM}_{ap}=\cos(\theta_{ap}+m)$4 to $s^{AAM}_{ap}=\cos(\theta_{ap}+m)$5, indicating improved speaker separability. A closely related study reported that symmetric NT-Xent-AM in SimCLR improved EER on VoxCeleb1-O from $s^{AAM}_{ap}=\cos(\theta_{ap}+m)$6 for asymmetric NT-Xent and $s^{AAM}_{ap}=\cos(\theta_{ap}+m)$7 for symmetric NT-Xent to $s^{AAM}_{ap}=\cos(\theta_{ap}+m)$8 at $s^{AAM}_{ap}=\cos(\theta_{ap}+m)$9 (Lepage et al., 2024).

In sentence embeddings, IFTCL reported that IFCL-BERT_base achieved $$\delta_{ij}=\frac{\cos(\theta_{ij}+m_1 p_{ij})-m_2 p_{ij}}{\tau},$$0 average Spearman on English STS, surpassing DCLR at $$\delta_{ij}=\frac{\cos(\theta_{ij}+m_1 p_{ij})-m_2 p_{ij}}{\tau},$$1, MoCoSE at $$\delta_{ij}=\frac{\cos(\theta_{ij}+m_1 p_{ij})-m_2 p_{ij}}{\tau},$$2, and PT-BERT at $$\delta_{ij}=\frac{\cos(\theta_{ij}+m_1 p_{ij})-m_2 p_{ij}}{\tau},$$3, while IFCL-BERT_large reached $$\delta_{ij}=\frac{\cos(\theta_{ij}+m_1 p_{ij})-m_2 p_{ij}}{\tau},$$4. On Chinese tasks, IFCL-BERT scored $$\delta_{ij}=\frac{\cos(\theta_{ij}+m_1 p_{ij})-m_2 p_{ij}}{\tau},$$5 on C-STS-B and $$\delta_{ij}=\frac{\cos(\theta_{ij}+m_1 p_{ij})-m_2 p_{ij}}{\tau},$$6 on SimCLUE, exceeding unsupervised SimCSE at $$\delta_{ij}=\frac{\cos(\theta_{ij}+m_1 p_{ij})-m_2 p_{ij}}{\tau},$$7 and $$\delta_{ij}=\frac{\cos(\theta_{ij}+m_1 p_{ij})-m_2 p_{ij}}{\tau},$$8. The ablation on STS-B showed $$\delta_{ij}=\frac{\cos(\theta_{ij}+m_1 p_{ij})-m_2 p_{ij}}{\tau},$$9 for full IFCL, $p_{ij}$0 without the hippocampus queue, $p_{ij}$1 without fraternal instances and Twins Loss, $p_{ij}$2 without Twins Loss and queue, and $p_{ij}$3 for the SimCSE baseline. The predicted cosine similarity range also expanded from $p_{ij}$4 for BERT to $p_{ij}$5 for IFCL, which the paper presents as evidence of improved fine-grained distinctions (Xiao et al., 2023).

Medical and multimodal settings show a similar pattern. In multi-view adaptive nasopharyngeal carcinoma radiotherapy prediction, MMCon with ViT achieved $p_{ij}$6 for Accuracy, Precision, Recall, and F1, outperforming both Cross-Entropy and SupCon, while the four-view setting far exceeded single-view T1 or CT inputs (Sheng et al., 2022). In cross-modal medical image segmentation, MPSCL reached $p_{ij}$7 Dice and $p_{ij}$8 ASD for MRI$p_{ij}$9CT, and $j$0 Dice and $j$1 ASD for CT$j$2MRI, improving on the corresponding no-margin CSCL ablations (Liu et al., 2021). In multimodal sentence embedding, KDMCSE improved average STS from $j$3 to $j$4 for MCSE-BERT on wiki+flickr and from $j$5 to $j$6 for MCSE-RoBERTa, while ablations showed a drop from $j$7 to $j$8 without AdapACSE and to $j$9 without threshold filtering (Nguyen et al., 2024).

The broader empirical picture extends beyond explicit positive-only margins. On ImageNet, the gradient-centered margin analysis reported linear-probe improvements for MoCo v3 from $i$0 to $i$1 and for SimCLR from $i$2 to $i$3 when positive-emphasis and curvature-based scaling were added (Rho et al., 2023). In contrastive learning with long-tail multimodal data, MM-TS improved EPIC-KITCHENS-100 average mAP from $i$4 to $i$5 and average nDCG from $i$6 to $i$7, and improved YouCook2 text-to-video retrieval from $i$8 to $i$9 in $$\mathcal{L}=-\log \frac{\exp(s_{ap}/\tau)}{\exp(s_{ap}/\tau)+\sum_n \exp(s_{an}/\tau)},$$00 (Sheludzko et al., 9 Mar 2026). These results suggest that positive-margin effects are not restricted to one architecture, pairing protocol, or modality.

6. Limitations, implementation trade-offs, and unresolved questions

The dominant limitation is sensitivity to how the margin is parameterized. In self-supervised speaker verification, overly large AAM values caused instability: $$\mathcal{L}=-\log \frac{\exp(s_{ap}/\tau)}{\exp(s_{ap}/\tau)+\sum_n \exp(s_{an}/\tau)},$$01 produced exploding gradients, and learnable margins performed worse than fixed ones. The reported practical guidance therefore recommends AM in $$\mathcal{L}=-\log \frac{\exp(s_{ap}/\tau)}{\exp(s_{ap}/\tau)+\sum_n \exp(s_{an}/\tau)},$$02 as a safe starting point and AAM in $$\mathcal{L}=-\log \frac{\exp(s_{ap}/\tau)}{\exp(s_{ap}/\tau)+\sum_n \exp(s_{an}/\tau)},$$03 with cosine scheduling over the first half of training (Lepage et al., 2023).

A second limitation is that margins interact strongly with the positive-generation mechanism. IFTCL states that the method depends on augmentation quality and on the reliability of input similarity as a proxy for semantic margin; poor translations or bad fusion may distort semantics, while near-identity augmentations collapse the innate margin toward zero (Xiao et al., 2023). MPSCL similarly notes that wrong pseudo-labels can enforce a margin to the wrong prototype, causing negative transfer, and that a single prototype per class may be insufficient for multi-modal intra-class distributions (Liu et al., 2021).

Implementation details can themselves dominate behavior. The Forward-Forward study shows that margin clamping and post-log subtraction are not interchangeable: under mean-over-positives reduction, subtraction is gradient-neutral, whereas clamping can inflate seed-to-seed variance when early-layer clamp activation rates are high and task accuracy is in a moderate regime. The proposed diagnostic is the layer-0 clamp activation rate, with CIFAR-10 displaying $$\mathcal{L}=-\log \frac{\exp(s_{ap}/\tau)}{\exp(s_{ap}/\tau)+\sum_n \exp(s_{an}/\tau)},$$04 at epoch 600 (Steier, 1 Mar 2026).

Certified and adaptive methods introduce their own assumptions. Contrastive Conformal Sets require exchangeability between calibration and test positives and a calibration mechanism matched to the test pairing process; when calibration is mismatched, under-coverage can occur despite the conformal guarantee’s nominal form (Alkhatib et al., 27 Mar 2026). MM-TS assumes that a modality, preferably text, provides embeddings that are clusterable enough to estimate local density shifts; the paper notes that text is preferred because it is semantically cleaner and less noisy than raw visual features (Sheludzko et al., 9 Mar 2026). Hydra, meanwhile, requires label-conditioned gating, and the pair space scales quadratically with the number of classes (Wu et al., 2024).

Across these lines of work, a common open issue is that “margin” no longer denotes a single object. It may mean a constant cosine offset, an angular perturbation, a semantic-distance constraint, a per-pair adaptive shift, a density-aware schedule, a queue-weighted soft support term, or a calibrated set radius. This suggests that positive-margin contrastive learning is better understood as a design principle—positives should satisfy stricter, structured, or certified constraints than standard InfoNCE imposes—than as one specific loss.