---
title: Contrastive Learning with MLM
url: https://www.emergentmind.com/topics/contrastive-learning-with-mlm
type: topic
---

# Contrastive Learning with MLM

Contrastive learning with masked language modeling (MLM) refers to a family of self-supervised or semi-supervised frameworks that combine two core representation learning objectives: 1) a contrastive objective that pulls together representations of related (“positive”) inputs and pushes apart unrelated (“negative”) inputs, and 2) token-level MLM, in which the model reconstructs randomly masked portions of the input sequence from unmasked context. Recent developments reveal that this hybrid approach consistently improves the quality, informativeness, and robustness of learned representations across domains including language, speech, code, protein sequences, and multimodal tasks. Architectures and training procedures vary, reflecting domain-specific requirements and nuances, but certain core design patterns and empirical findings are broadly shared.

## 1. Motivation and Theoretical Basis

The primary motivation for combining contrastive learning and MLM is to exploit complementary inductive biases at the sequence and token level. Contrastive learning (CL) enforces global or “holistic” discrimination between examples by aligning representations of positive pairs and separating negatives. This structure injects desirable geometric properties—such as alignment and uniformity—into the embedding space. However, it may overlook or underutilize fine-grained input details, leading to superficial similarity collapse or insensitivity to information not included in the contrastive construction [2210.06432], [2204.10298].

MLM, by contrast, operates at the token level, encouraging contextual composition and information aggregation required for masked token reconstruction. Pure MLM, however, may yield representations that are not globally discriminative, and is prone to overfitting, codebook collapse (in discrete quantization), or domain-drift when used alone [2210.06432], [2108.06209], [2510.16797].

Joint optimization of both objectives is motivated by the need for representations that are at once information-dense, generative (enabling reconstruction), and highly discriminative for retrieval, classification, and transfer learning. A further rationale is that simultaneous sequence- and token-level supervision allows for stable end-to-end training and circumvents the optimization issues encountered with either objective in isolation [2210.06432], [2108.06209].

## 2. Core Methodological Paradigms

### 2.1 Loss Formulations and Joint Optimization

Most approaches employ an additive or weighted combination of contrastive and MLM losses, commonly expressed as:
\[
\mathcal{L} = \mathcal{L}_\mathrm{contrast} + \lambda\,\mathcal{L}_\mathrm{MLM}
\]
where $\mathcal{L}_\mathrm{contrast}$ is typically a batch or queue-based InfoNCE or margin-ranking loss, and $\mathcal{L}_\mathrm{MLM}$ is the negative log-likelihood of masked token reconstruction over selected positions. The balance factor $\lambda$ is tuned empirically or by ablation [2210.06432], [2108.06209], [2510.16797].

Architecturally, some frameworks use a single backbone with two heads (e.g., MLM decoder and contrastive embedding projector), while others employ auxiliary networks to mediate gradient flow and prevent MLM-induced perturbation of contrastive representations [2210.06432], [2306.09594]. Careful gradient routing—such as freezing shared layers or restricting MLM computation to shallow auxiliary modules—is crucial to prevent adverse interference [2210.06432], [2306.09594].

### 2.2 Construction of Positive/Negative Pairs and "Views"

Different domains admit distinct augmentation strategies for view generation:
- In language, views may be generated by applying different dropout masks [2210.06432], paraphrasing via prompts/demonstrations [2205.01308], or generating MLM-edited sentences [2204.10298].
- For code, explicit program transformations (renaming, reordering, dead code insertion) and natural language paraphrasing serve as augmentations [2301.09072].
- In cross-modal regimes (e.g., speech, audio-text), views may correspond to paired latent or quantized representations produced by different modules within the network [2108.06209], [2604.01247].

Negatives are sampled from other in-batch or memory-queue examples as in MoCo [2301.09072], or all non-paired examples in supervised settings [2205.01308]. The supervision or “label” varies: pseudo-labels (unsupervised), task-defined classes, or hard negatives (matched by semantic similarity or structured mining).

### 2.3 Domain-Specific Adaptations

Hybrid contrastive + MLM objectives are tailored for different data types:
- **Text/sentence embeddings**: InfoCSE [2210.06432], DiffCSE [2204.10298], CMLM-CSE [2306.09594], Auto-MLM [2203.16187], and MOSAIC [2510.16797] all demonstrate variants of this paradigm, integrating modifications to loss structure, view construction, or auxiliary networks.
- **Speech**: w2v-BERT [2108.06209] discretizes latent speech features via a quantizer for contrastive learning, and predicts masked discrete targets with a context-aware MLM head.
- **Code**: ContraBERT [2301.09072] combines code- and text-based augmentations, supporting both MLM on corrupted code snippets and contrastive learning between original and augmented pairs.
- **Protein/biomedical sequences**: SCEPTR [2406.06397] employs joint autocontrastive and masked-learning on structured TCR inputs.
- **Multimodal TTS**: Methods such as [2604.01247] first pretrain text and phoneme encoders with MLM, then apply cross-modal contrastive objectives with intrinsic retrieval and generative downstream assessment.

## 3. Representative Algorithms and Implementation Strategies

The following table summarizes architectural and training characteristics of selected frameworks:

| Method         | Domains      | View Gen.              | Loss Structure                    |
|----------------|-------------|------------------------|------------------------------------|
| InfoCSE [2210.06432]     | Text        | Dropout masks          | Contrastive + aux MLM (aux head)   |
| DiffCSE [2204.10298]     | Text        | Dropout & MLM-edits    | Contrastive + RTD loss             |
| ContraBERT [2301.09072]  | Code/Text   | Code+text augment      | MoCo-based contrast + MLM          |
| KECP [2205.03071]        | QA          | Prompted masking       | Span-level contrast + MLM          |
| w2v-BERT [2108.06209]    | Speech      | Audio masking/quant    | Audio contrastive + discrete MLM   |
| SCEPTR [2406.06397]      | Protein     | Mask/chain drop + dropout | Autocontrastive + MLM           |

Auxiliary modules are employed to restrict MLM gradients, preserve discriminative power, or inject sequence-level features into token-level predictions. For instance, InfoCSE freezes lower encoder layers for MLM computation, combining outputs with [CLS] via aggregation before passing through specialized reconstruction heads [2210.06432]. CMLM-CSE concatenates sentence embeddings with local token features prior to MLM prediction by a lightweight fusion Transformer [2306.09594]. Auto-MLM sums sentence vectors with token positions in masked sentences, reflecting tight integration of global and local signals [2203.16187].

Optimization and regularization choices—including temperature settings for InfoNCE, weighting of objectives ($\lambda$), masking rates, and auxiliary network depth—are consistently established via ablation, with best values often diverging between contrastive-only, MLM-only, and hybrid models [2210.06432], [2306.09594], [2510.16797].

## 4. Empirical Findings and Domain Impact

Simultaneous optimization of contrastive and MLM losses yields:
- Substantial improvements in downstream performance for retrieval, clustering, similarity, and classification tasks, notably in under-resourced and few-shot regimes [2210.06432], [2301.09072], [2205.03071], [2510.16797].
- Greater robustness against spurious or adversarial augmentations, e.g., variable renaming attacks in code [2301.09072].
- Measurable gains in transfer metrics (STS, nDCG@10, AUROC) versus baseline models trained with only one objective [2210.06432], [2510.16797], [2406.06397].

Specific observations:
- In sentence encoding, InfoCSE achieves +2.6 pts Spearman’s correlation over SimCSE, with ablations showing that removal of either objective significantly reduces performance [2210.06432].
- In speech, w2v-BERT reduces WER by 30–40% relative to prior models on LibriSpeech “test-other” and yields >30% relative gain on Voice Search traffic when compared to conformer wav2vec 2.0 [2108.06209].
- In code, ContraBERT achieves 90.46 MAP@R on clone detection, robust to adversarial edits, and outperforms CodeBERT in all core tasks [2301.09072].
- Domain-adapted models (MOSAIC) trained with joint objectives and domain-restricted masking outperform naive transfer and pure-contrastive re-training by up to +13.4 NDCG@10 in biomedical and low-resource text retrieval [2510.16797].
- In TCR representation, SCEPTR’s joint AC+MLM pretraining consistently beats general PLMs and alignment-based methods across all tested metrics [2406.06397].

Ablation studies systematically demonstrate that discarding contrastive or MLM components reduces overall performance or leads to collapsed/degenerate representations, validating the necessity of combined objectives for effective representation learning.

## 5. Analysis of Failure Modes, Limitations, and Trade-Offs

The efficacy of hybrid contrastive+MLM frameworks is contingent on:
- Careful balance of loss weights; over- or under-weighting MLM can respectively wash out the discriminative structure or collapse token-level information [2210.06432], [2306.09594], [2510.16797].
- Design of view and negative sampling strategies; poorly chosen augmentations may distort semantic alignment or introduce artifacts, especially in structured or multimodal domains [2301.09072], [2604.01247].
- Domain adaptation: methods like MOSAIC demonstrate that restricting MLM supervision to domain-specific vocabulary is critical for achieving domain-relevant adaptation without corrupting base semantic structures [2510.16797].

A trade-off frequently encountered is that gains in intrinsic embedding quality or retrieval metrics (alignment, recall) do not always translate to improved downstream generative performance, as shown in prosody-aware TTS systems [2604.01247]. Over-optimization towards prosodic or fine-grained discriminative targets can erode the encoder’s ability to reliably encode core linguistic or structural attributes, e.g., phoneme identity.

A plausible implication is that continued progress requires adaptive weighting or curriculum strategies that dynamically balance global discrimination and local generative constraints—in some cases, staged or interleaved training appears optimal [2510.16797], [2604.01247].

## 6. Extensions, Variations, and Future Research Directions

Several methodological extensions and open problems are evident:
- **Auxiliary Networks**: Use of specialized auxiliary or conditional MLM heads (as in InfoCSE and CMLM-CSE) to control gradient interference and infuse coverage of global sentence features is increasingly standard, but optimal architectural design, depth, and pretraining regimes remain to be characterized [2210.06432], [2306.09594].
- **Equivariant Objectives**: The introduction of equivariant contrastive learning, where particular augmentations are treated as “harmful” and the encoder is encouraged to be sensitive to such edits (e.g., through RTD losses in DiffCSE), generalizes the contrastive+MLM paradigm [2204.10298].
- **Domain-Selective Masking**: Selectively masking only domain tokens (MOSAIC) or information-critical regions suggests a general pattern for controlled adaptation without generalization collapse, but requires robust identification/extraction pipelines [2510.16797].
- **Span-Level and Task-Specific Contrastive Losses**: In extractive QA and tasks involving structured outputs, MLM-based span generation and ranking (KECP) introduces task-oriented contrastive supervision, directly aligning the pretraining and downstream objectives [2205.03071].
- **Multimodal Hybridization**: Joint text-audio pretraining in speech and TTS, as with w2v-BERT and recent dual-stream alignments, suggests a general route for contrastive+MLM approaches to bridge distinct modalities [2108.06209], [2604.01247].

Promising research directions include dynamic objective scheduling, adaptive augmentation selection, and fine-grained introspection/evaluation of embedding geometries under combined supervision.

---

**References:**
- "W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training" [2108.06209]
- "Contrastive learning of T cell receptor representations" [2406.06397]
- "Contrastive Learning for Prompt-Based Few-Shot Language Learners" [2205.01308]
- "InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddings" [2210.06432]
- "DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings" [2204.10298]
- "KECP: Knowledge Enhanced Contrastive Prompting for Few-shot Extractive Question Answering" [2205.03071]
- "ContraBERT: Enhancing Code Pre-trained Models via Contrastive Learning" [2301.09072]
- "MOSAIC: Masked Objective with Selective Adaptation for In-domain Contrastive Learning" [2510.16797]
- "CMLM-CSE: Based on Conditional MLM Contrastive Learning for Sentence Embeddings" [2306.09594]
- "Auto-MLM: Improved Contrastive Learning for Self-supervised Multi-lingual Knowledge Retrieval" [2203.16187]
- "Combining Masked Language Modeling and Cross-Modal Contrastive Learning for Prosody-Aware TTS" [2604.01247]

Source: https://www.emergentmind.com/topics/contrastive-learning-with-mlm