---
title: 'CLEAR: Contrastive Learning for Sentence Representation'
url: https://www.emergentmind.com/topics/clear-framework
type: topic
---

# CLEAR: Contrastive Learning for Sentence Representation

Searching arXiv for the core CLEAR paper and directly related prior work mentioned in the source material.
CLEAR, short for **Contrastive LEArning for sentence Representation**, is a pre-training framework for learning improved **sentence embeddings** by augmenting standard masked language modeling with an explicit **sentence-level contrastive objective**. Its central aim is to learn a **noise-invariant sentence representation**: if two augmented views of the same sentence preserve its semantic content despite deletion, reordering, or substitution noise, the encoder should map them to nearby vectors while keeping different underlying sentences apart. In this formulation, CLEAR addresses a gap in dominant pre-training paradigms such as masked language modeling, which are highly effective at token-level denoising but do not directly optimize sentence-level semantic similarity [2012.15466].

## 1. Problem formulation and conceptual basis

CLEAR begins from a criticism of word-centric pre-training. Objectives such as **masked language modeling (MLM)** are effective for contextual token representations, but they do not explicitly optimize a model to produce sentence vectors in which semantically similar sentences are close and unrelated sentences are distant. The paper situates this problem in relation to prior observations about BERT-style sentence representations: although the `[CLS]` token is often treated as a sentence embedding, earlier sentence-embedding work had already suggested that token-level pre-training does not necessarily yield the best sentence-level geometry [2012.15466].

The framework is motivated by downstream settings in which the atomic unit is a whole sentence rather than a token sequence to be reconstructed. The paper names semantic textual similarity, paraphrase detection, natural language inference, and retrieval as representative cases. In these tasks, an encoder is valuable insofar as it summarizes sentence semantics in a single vector and does so robustly under superficial perturbation. CLEAR therefore defines the target representation as one that is stable under approximately meaning-preserving noise while remaining discriminative across different sentences.

This motivation also clarifies what CLEAR is not. It is not merely a fine-tuning recipe for sentence similarity; it is a **pre-training framework from scratch** that combines MLM and contrastive learning from the beginning. That distinction matters in the paper’s comparison to other sentence-representation approaches, especially those that begin from an already pre-trained encoder.

## 2. Training pipeline and objective

CLEAR follows a **SimCLR-style** contrastive template adapted to text. For each original sentence \(s\), the framework generates two stochastic augmented views,
\[
\widetilde{s}_1 = \mathrm{AUG}(s, \mathit{seed}_1), \qquad
\widetilde{s}_2 = \mathrm{AUG}(s, \mathit{seed}_2),
\]
encodes them with a Transformer \(f(\cdot)\), and maps the resulting sentence representations through a nonlinear projection head \(g(\cdot)\):
\[
H_1 = f(\widetilde{s}_1), \qquad H_2 = f(\widetilde{s}_2),
\]
\[
z_1 = g(H_1), \qquad z_2 = g(H_2).
\]

During contrastive training, CLEAR uses the representation of a manually inserted `[CLS]` token as the sentence vector. In a minibatch of \(N\) original sentences, the model constructs \(2N\) augmented sentences. The two views derived from the same source sentence form a **positive pair**, and all other augmented instances in the minibatch serve as **negative examples**. The contrastive loss for a positive pair \((i,j)\) is
\[
l(i, j) {=} {-}\log \frac{\exp\left(\mathrm{sim}(z_i, z_j) / {\tau}\right)}{\sum_{k=1}^{2N} \mathbbm{1}_{[k\neq i]}\exp\left(\mathrm{sim}(z_i, z_k) / {\tau}\right) } ,
\]
with cosine similarity
\[
\mathrm{sim}(u, v) = u^{\top}v/(\Vert u\Vert_2 \Vert v\Vert_2).
\]

The minibatch contrastive objective is written as
\[
\mathcal{L}_{\mathrm{CL} = \sum_{i=1}^{2N} \sum_{j=1}^{2N} m(i, j) l(i,j),
\]
where \(m(i,j)\) is \(1\) for a positive pair and \(0\) otherwise. The total training objective combines token-level and sentence-level supervision:
\[
\mathcal{L}_{\mathrm{total} = \mathcal{L}_{\mathrm{MLM} + \mathcal{L}_{\mathrm{CL}.
\]

The intended division of labor is explicit in the paper. **MLM** supplies token-level contextual understanding; **CL** supplies sentence-level semantic invariance. This suggests that CLEAR should be read less as a replacement for MLM than as a structural complement to it.

## 3. Augmentation design and invariance structure

A major contribution of CLEAR is its treatment of augmentation for text, where perturbation is much less straightforward than in vision. The paper studies four augmentation families and treats them as sources of “proper noise” that usually preserve enough semantics to support contrastive learning.

| Augmentation | Mechanism | Main setting |
|---|---|---|
| Word deletion | Replace randomly selected tokens with `[DEL]`; collapse consecutive `[DEL]` tokens | 70% of tokens |
| Span deletion | Replace spans with `[DEL]`; collapse consecutive `[DEL]` tokens | 5 spans, each about 5% of input |
| Reordering | Swap randomly selected span pairs | 5 pairs, each span about 5% of input |
| Substitution | Replace selected tokens with similar-meaning words | 30% of tokens |

Word deletion is severe by design: in the main pre-training experiments, **70% of tokens** are replaced by `[DEL]`, not physically removed from the sequence. Consecutive deletion markers are collapsed so that the model cannot trivially identify augmentation boundaries from long `[DEL]` runs. Span deletion is a structured version of the same idea, removing contiguous chunks rather than isolated tokens. Reordering perturbs surface order by swapping **5 pairs of spans**, and substitution introduces lexical variation by replacing **30% of tokens** with similar-meaning words drawn from a synonym inventory in which about **40% of tokens** in the corpus have at least one available substitute [2012.15466].

The paper distinguishes **single-augmentation models**—called “MLM+1-CL-objective”—from **mixed augmentation models**, called “MLM+2-CL-objective.” When isolating a single augmentation, the same augmentation type is applied to both \(\widetilde s_1\) and \(\widetilde s_2\) so that its effect can be measured directly. In mixed models, augmentations are composed, as in **MLM+subs+del-span** or **MLM+del-span+reorder**.

CLEAR treats augmentation choice as semantically consequential rather than merely regularizing. Deletion-based augmentations encourage robustness to missing information; reordering pushes abstraction beyond exact word order; substitution encourages lexical invariance and paraphrase sensitivity. At the same time, the paper explicitly acknowledges semantic corruption as a limitation: deleting a word such as *not* can change meaning. The method therefore relies on approximate, not guaranteed, preservation of semantics.

## 4. Architecture, implementation, and evaluation protocol

The encoder is a **12-layer Transformer** with **12 attention heads** and **hidden size 768**, making it base-sized in the BERT/RoBERTa regime. CLEAR includes a **nonlinear projection head** \(g(\cdot)\), but the paper does not state its exact internal architecture beyond that description. Pre-training uses **BookCorpus** and **English Wikipedia**; an ablation study uses **WikiText-103**.

The main training configuration is large-scale: **500K updates**, minibatch size **8,192 sequences**, maximum length **512 tokens**, peak learning rate \(6 \times 10^{-4}\) with warmup over the first **24,000 steps**, linear decay thereafter, **Adam** with \(\beta_1 = 0.9\), \(\beta_2 = 0.98\), \(\epsilon = 1\mathrm{e}{-6}\), weight decay **0.01**, dropout **0.1**, and **256 NVIDIA Tesla V100 32GB GPUs**. For MLM, **15%** of tokens are selected for masking, with **10% random replacement** and **10% keep unchanged**, following the familiar BERT-style masking variant.

Evaluation is split across two regimes. On **GLUE**, the model is fine-tuned with a task-specific fully connected layer and RoBERTa-style hyperparameters on **MNLI, QNLI, QQP, RTE, SST-2, MRPC, CoLA,** and **STS-B**. On **SentEval**, the encoder is evaluated as a frozen sentence representation on **SICK-R, STS-B, STS12, STS13, STS14, STS15,** and **STS16**. CLEAR reports both **CLS pooling** and **mean pooling** at evaluation time, even though `[CLS]` is the representation directly optimized during contrastive pre-training.

One of the paper’s more practically important observations is that CLEAR often favors **CLS pooling**, unlike earlier reports that simple mean pooling can outperform `[CLS]` in BERT-style models. The paper attributes this to direct supervision on the `[CLS]` representation during contrastive learning. A plausible implication is that pooling behavior is not a generic property of the backbone alone; it is partly induced by the pre-training objective.

## 5. Empirical results, ablations, and learned invariances

CLEAR improves over strong masked-language-model baselines on both transfer benchmarks used in the paper. On GLUE, baseline averages are **81.2** for **BERT-base** and **83.5** for **RoBERTa-base**. CLEAR variants achieve:
- **MLM + del-word**: **85.7**
- **MLM + del-span**: **85.6**
- **MLM + subs + del-word**: **85.0**
- **MLM + subs + del-span**: **85.5**
- **MLM + del-word + reorder**: **85.0**
- **MLM + del-span + reorder**: **85.7**

The best variants therefore improve over RoBERTa-base by **+2.2 points average** on the eight GLUE tasks [2012.15466]. Task-wise gains are especially notable on **RTE** and **CoLA**, and the paper emphasizes that the best score for each GLUE task is obtained by some CLEAR variant rather than by the baseline models.

On SentEval, baseline averages are **53.8** for **RoBERTa-base-mean** and **56.1** for **RoBERTa-base-[CLS]**. The strongest CLEAR numbers are:
- **MLM + subs + del-span-[CLS]**: **61.8**
- **MLM + del-span + reorder-[CLS]**: **61.5**
- **MLM + del-span + reorder-mean**: **59.1**

This corresponds to about **+5.7 points** over the best baseline on the seven STS-style tasks. The paper highlights particularly large improvements on **STS13**, **STS14**, **STS15**, and **STS16**, which supports the claim that CLEAR especially benefits sentence-level semantic transfer [2012.15466].

The ablation study is designed to test whether the improvements come from contrastive learning itself or merely from increased effective batch size and multiple views. Two controls are introduced on WikiText-103: **Double-batch RoBERTa-base** and **Double MLM RoBERTa-base**. Their reported averages are **76.8** and **75.8**, compared with **76.4** for the base model and up to **77.2** for CLEAR variants. The paper’s interpretation is that larger batches help a little, but the gain cannot be reduced to batch size alone; the **contrastive loss itself contributes**.

Augmentation analysis yields one of the paper’s most distinctive findings: **different augmentations learn different features**. **MLM + subs + del-span** is particularly strong on similarity and paraphrase-oriented tasks such as **QQP**, **STS**, and nearly **MRPC**, whereas **MLM + del-span** performs well on inference-oriented tasks such as **MNLI**, **QNLI**, and **RTE**. This suggests that augmentation type functions not just as noise injection, but as a way of shaping the semantic invariances encoded by the sentence representation.

## 6. Position in the literature, limitations, and significance

CLEAR is positioned at the intersection of pre-trained language models, sentence representation learning, and contrastive learning. Relative to **BERT** and **RoBERTa**, its novelty lies in adding an explicit **sentence-level contrastive objective** to token-level MLM pre-training. Relative to **Sentence-BERT**, it is not a supervised siamese fine-tuning method on labeled sentence pairs, but a **self-supervised pre-training framework**. Relative to earlier unsupervised sentence models such as **Skip-Thoughts**, **Quick-Thoughts**, and order-prediction objectives like **NSP/SOP**, it replaces surrounding-sentence prediction with **instance discrimination over augmented views of the same sentence**. Relative to **DeCLUTR**, the paper emphasizes that CLEAR is trained **from scratch** and systematically studies text augmentation choices; relative to **CERT**, it uses an **encoder-only** approach and avoids decoder-induced noise [2012.15466].

The method also has clear limitations. Some augmentation configurations were unstable enough that results were not reported: **MLM + subs**, **MLM + reorder**, and **MLM + subs + reorder** either converged too quickly or encountered **gradient explosion**. The paper interprets this as evidence that some augmentations are too easy to distinguish, which weakens or destabilizes contrastive learning. Hyperparameter tuning is explicitly described as only **slight**, broader augmentation mixing is left for future work, and the paper does not present geometric analyses in terms such as alignment, uniformity, or anisotropy. Its evidence for better sentence representations comes from downstream transfer and STS performance rather than from a dedicated representation-geometry study.

Within the development of sentence representation methods, CLEAR’s significance lies in making sentence-level invariance an explicit pre-training target. Its main empirical message is not merely that contrastive learning can be imported into NLP, but that when combined with MLM and carefully chosen sentence augmentations, it materially improves both fine-tuned transfer and frozen sentence embeddings. The broader implication is that pre-training objectives do not simply encode language structure in the abstract; they determine which units of meaning—tokens, spans, or sentences—are made geometrically accessible in the representation space.

Source: https://www.emergentmind.com/topics/clear-framework