CLEAR: Contrastive Learning for Sentence Representation

• CLEAR is a pre-training framework that augments masked language modeling with a sentence-level contrastive objective to produce noise-invariant sentence embeddings.
• It employs targeted text augmentations such as word deletion, span deletion, reordering, and substitution to enforce semantic consistency under perturbations.
• CLEAR demonstrates significant improvements on benchmarks like GLUE and STS, outperforming standard models such as BERT and RoBERTa.

Searching arXiv for the core CLEAR paper and directly related prior work mentioned in the source material. CLEAR, short for Contrastive LEArning for sentence Representation, is a pre-training framework for learning improved sentence embeddings by augmenting standard masked language modeling with an explicit sentence-level contrastive objective. Its central aim is to learn a noise-invariant sentence representation: if two augmented views of the same sentence preserve its semantic content despite deletion, reordering, or substitution noise, the encoder should map them to nearby vectors while keeping different underlying sentences apart. In this formulation, CLEAR addresses a gap in dominant pre-training paradigms such as masked language modeling, which are highly effective at token-level denoising but do not directly optimize sentence-level semantic similarity (Wu et al., 2020).

1. Problem formulation and conceptual basis

CLEAR begins from a criticism of word-centric pre-training. Objectives such as masked language modeling (MLM) are effective for contextual token representations, but they do not explicitly optimize a model to produce sentence vectors in which semantically similar sentences are close and unrelated sentences are distant. The paper situates this problem in relation to prior observations about BERT-style sentence representations: although the [[CLS](https://www.emergentmind.com/topics/chunk-level-sparsity-cls)] token is often treated as a sentence embedding, earlier sentence-embedding work had already suggested that token-level pre-training does not necessarily yield the best sentence-level geometry (Wu et al., 2020).

The framework is motivated by downstream settings in which the atomic unit is a whole sentence rather than a token sequence to be reconstructed. The paper names semantic textual similarity, paraphrase detection, natural language inference, and retrieval as representative cases. In these tasks, an encoder is valuable insofar as it summarizes sentence semantics in a single vector and does so robustly under superficial perturbation. CLEAR therefore defines the target representation as one that is stable under approximately meaning-preserving noise while remaining discriminative across different sentences.

This motivation also clarifies what CLEAR is not. It is not merely a fine-tuning recipe for sentence similarity; it is a pre-training framework from scratch that combines MLM and contrastive learning from the beginning. That distinction matters in the paper’s comparison to other sentence-representation approaches, especially those that begin from an already pre-trained encoder.

2. Training pipeline and objective

CLEAR follows a SimCLR-style contrastive template adapted to text. For each original sentence $s$, the framework generates two stochastic augmented views,

$$\widetilde{s}_1 = \mathrm{AUG}(s, \mathit{seed}_1), \qquad \widetilde{s}_2 = \mathrm{AUG}(s, \mathit{seed}_2),$$

encodes them with a Transformer $f(\cdot)$, and maps the resulting sentence representations through a nonlinear projection head $g(\cdot)$: $$H_1 = f(\widetilde{s}_1), \qquad H_2 = f(\widetilde{s}_2),$$

$z_1 = g(H_1), \qquad z_2 = g(H_2).$

During contrastive training, CLEAR uses the representation of a manually inserted [CLS] token as the sentence vector. In a minibatch of $N$ original sentences, the model constructs $2N$ augmented sentences. The two views derived from the same source sentence form a positive pair, and all other augmented instances in the minibatch serve as negative examples. The contrastive loss for a positive pair $(i,j)$ is

$l(i, j) {=} {-}\log \frac{\exp\left(\mathrm{sim}(z_i, z_j) / {\tau}\right)}{\sum_{k=1}^{2N} \mathbbm{1}_{[k\neq i]}\exp\left(\mathrm{sim}(z_i, z_k) / {\tau}\right) } ,$

with cosine similarity

$$\widetilde{s}_1 = \mathrm{AUG}(s, \mathit{seed}_1), \qquad \widetilde{s}_2 = \mathrm{AUG}(s, \mathit{seed}_2),$$0

The minibatch contrastive objective is written as

$$\widetilde{s}_1 = \mathrm{AUG}(s, \mathit{seed}_1), \qquad \widetilde{s}_2 = \mathrm{AUG}(s, \mathit{seed}_2),$$1

where $$\widetilde{s}_1 = \mathrm{AUG}(s, \mathit{seed}_1), \qquad \widetilde{s}_2 = \mathrm{AUG}(s, \mathit{seed}_2),$$2 is $$\widetilde{s}_1 = \mathrm{AUG}(s, \mathit{seed}_1), \qquad \widetilde{s}_2 = \mathrm{AUG}(s, \mathit{seed}_2),$$3 for a positive pair and $$\widetilde{s}_1 = \mathrm{AUG}(s, \mathit{seed}_1), \qquad \widetilde{s}_2 = \mathrm{AUG}(s, \mathit{seed}_2),$$4 otherwise. The total training objective combines token-level and sentence-level supervision: $$\widetilde{s}_1 = \mathrm{AUG}(s, \mathit{seed}_1), \qquad \widetilde{s}_2 = \mathrm{AUG}(s, \mathit{seed}_2),$$5

The intended division of labor is explicit in the paper. MLM supplies token-level contextual understanding; CL supplies sentence-level semantic invariance. This suggests that CLEAR should be read less as a replacement for MLM than as a structural complement to it.

3. Augmentation design and invariance structure

A major contribution of CLEAR is its treatment of augmentation for text, where perturbation is much less straightforward than in vision. The paper studies four augmentation families and treats them as sources of “proper noise” that usually preserve enough semantics to support contrastive learning.

Augmentation	Mechanism	Main setting
Word deletion	Replace randomly selected tokens with [DEL]; collapse consecutive [DEL] tokens	70% of tokens
Span deletion	Replace spans with [DEL]; collapse consecutive [DEL] tokens	5 spans, each about 5% of input
Reordering	Swap randomly selected span pairs	5 pairs, each span about 5% of input
Substitution	Replace selected tokens with similar-meaning words	30% of tokens

Word deletion is severe by design: in the main pre-training experiments, 70% of tokens are replaced by [DEL], not physically removed from the sequence. Consecutive deletion markers are collapsed so that the model cannot trivially identify augmentation boundaries from long [DEL] runs. Span deletion is a structured version of the same idea, removing contiguous chunks rather than isolated tokens. Reordering perturbs surface order by swapping 5 pairs of spans, and substitution introduces lexical variation by replacing 30% of tokens with similar-meaning words drawn from a synonym inventory in which about 40% of tokens in the corpus have at least one available substitute (Wu et al., 2020).

The paper distinguishes single-augmentation models—called “MLM+1-CL-objective”—from mixed augmentation models, called “MLM+2-CL-objective.” When isolating a single augmentation, the same augmentation type is applied to both $$\widetilde{s}_1 = \mathrm{AUG}(s, \mathit{seed}_1), \qquad \widetilde{s}_2 = \mathrm{AUG}(s, \mathit{seed}_2),$$6 and $$\widetilde{s}_1 = \mathrm{AUG}(s, \mathit{seed}_1), \qquad \widetilde{s}_2 = \mathrm{AUG}(s, \mathit{seed}_2),$$7 so that its effect can be measured directly. In mixed models, augmentations are composed, as in MLM+subs+del-span or MLM+del-span+reorder.

CLEAR treats augmentation choice as semantically consequential rather than merely regularizing. Deletion-based augmentations encourage robustness to missing information; reordering pushes abstraction beyond exact word order; substitution encourages lexical invariance and paraphrase sensitivity. At the same time, the paper explicitly acknowledges semantic corruption as a limitation: deleting a word such as not can change meaning. The method therefore relies on approximate, not guaranteed, preservation of semantics.

4. Architecture, implementation, and evaluation protocol

The encoder is a 12-layer Transformer with 12 attention heads and hidden size 768, making it base-sized in the BERT/RoBERTa regime. CLEAR includes a nonlinear projection head $$\widetilde{s}_1 = \mathrm{AUG}(s, \mathit{seed}_1), \qquad \widetilde{s}_2 = \mathrm{AUG}(s, \mathit{seed}_2),$$8, but the paper does not state its exact internal architecture beyond that description. Pre-training uses BookCorpus and English Wikipedia; an ablation study uses WikiText-103.

The main training configuration is large-scale: 500K updates, minibatch size 8,192 sequences, maximum length 512 tokens, peak learning rate $$\widetilde{s}_1 = \mathrm{AUG}(s, \mathit{seed}_1), \qquad \widetilde{s}_2 = \mathrm{AUG}(s, \mathit{seed}_2),$$9 with warmup over the first 24,000 steps, linear decay thereafter, Adam with $f(\cdot)$0, $f(\cdot)$1, $f(\cdot)$2, weight decay 0.01, dropout 0.1, and 256 NVIDIA Tesla V100 32GB GPUs. For MLM, 15% of tokens are selected for masking, with 10% random replacement and 10% keep unchanged, following the familiar BERT-style masking variant.

Evaluation is split across two regimes. On GLUE, the model is fine-tuned with a task-specific fully connected layer and RoBERTa-style hyperparameters on MNLI, QNLI, QQP, RTE, SST-2, MRPC, CoLA, and STS-B. On SentEval, the encoder is evaluated as a frozen sentence representation on SICK-R, STS-B, STS12, STS13, STS14, STS15, and STS16. CLEAR reports both CLS pooling and mean pooling at evaluation time, even though [CLS] is the representation directly optimized during contrastive pre-training.

One of the paper’s more practically important observations is that CLEAR often favors CLS pooling, unlike earlier reports that simple mean pooling can outperform [CLS] in BERT-style models. The paper attributes this to direct supervision on the [CLS] representation during contrastive learning. A plausible implication is that pooling behavior is not a generic property of the backbone alone; it is partly induced by the pre-training objective.

5. Empirical results, ablations, and learned invariances

CLEAR improves over strong masked-language-model baselines on both transfer benchmarks used in the paper. On GLUE, baseline averages are 81.2 for BERT-base and 83.5 for RoBERTa-base. CLEAR variants achieve:

• MLM + del-word: 85.7
• MLM + del-span: 85.6
• MLM + subs + del-word: 85.0
• MLM + subs + del-span: 85.5
• MLM + del-word + reorder: 85.0
• MLM + del-span + reorder: 85.7

The best variants therefore improve over RoBERTa-base by +2.2 points average on the eight GLUE tasks (Wu et al., 2020). Task-wise gains are especially notable on RTE and CoLA, and the paper emphasizes that the best score for each GLUE task is obtained by some CLEAR variant rather than by the baseline models.

On SentEval, baseline averages are 53.8 for RoBERTa-base-mean and 56.1 for RoBERTa-base-[CLS]. The strongest CLEAR numbers are:

• MLM + subs + del-span-[CLS]: 61.8
• MLM + del-span + reorder-[CLS]: 61.5
• MLM + del-span + reorder-mean: 59.1

This corresponds to about +5.7 points over the best baseline on the seven STS-style tasks. The paper highlights particularly large improvements on STS13, STS14, STS15, and STS16, which supports the claim that CLEAR especially benefits sentence-level semantic transfer (Wu et al., 2020).

The ablation study is designed to test whether the improvements come from contrastive learning itself or merely from increased effective batch size and multiple views. Two controls are introduced on WikiText-103: Double-batch RoBERTa-base and Double MLM RoBERTa-base. Their reported averages are 76.8 and 75.8, compared with 76.4 for the base model and up to 77.2 for CLEAR variants. The paper’s interpretation is that larger batches help a little, but the gain cannot be reduced to batch size alone; the contrastive loss itself contributes.

Augmentation analysis yields one of the paper’s most distinctive findings: different augmentations learn different features. MLM + subs + del-span is particularly strong on similarity and paraphrase-oriented tasks such as QQP, STS, and nearly MRPC, whereas MLM + del-span performs well on inference-oriented tasks such as MNLI, QNLI, and RTE. This suggests that augmentation type functions not just as noise injection, but as a way of shaping the semantic invariances encoded by the sentence representation.

6. Position in the literature, limitations, and significance

CLEAR is positioned at the intersection of pre-trained LLMs, sentence representation learning, and contrastive learning. Relative to BERT and RoBERTa, its novelty lies in adding an explicit sentence-level contrastive objective to token-level MLM pre-training. Relative to Sentence-BERT, it is not a supervised siamese fine-tuning method on labeled sentence pairs, but a self-supervised pre-training framework. Relative to earlier unsupervised sentence models such as Skip-Thoughts, Quick-Thoughts, and order-prediction objectives like NSP/SOP, it replaces surrounding-sentence prediction with instance discrimination over augmented views of the same sentence. Relative to DeCLUTR, the paper emphasizes that CLEAR is trained from scratch and systematically studies text augmentation choices; relative to CERT, it uses an encoder-only approach and avoids decoder-induced noise (Wu et al., 2020).

The method also has clear limitations. Some augmentation configurations were unstable enough that results were not reported: MLM + subs, MLM + reorder, and MLM + subs + reorder either converged too quickly or encountered gradient explosion. The paper interprets this as evidence that some augmentations are too easy to distinguish, which weakens or destabilizes contrastive learning. Hyperparameter tuning is explicitly described as only slight, broader augmentation mixing is left for future work, and the paper does not present geometric analyses in terms such as alignment, uniformity, or anisotropy. Its evidence for better sentence representations comes from downstream transfer and STS performance rather than from a dedicated representation-geometry study.

Within the development of sentence representation methods, CLEAR’s significance lies in making sentence-level invariance an explicit pre-training target. Its main empirical message is not merely that contrastive learning can be imported into NLP, but that when combined with MLM and carefully chosen sentence augmentations, it materially improves both fine-tuned transfer and frozen sentence embeddings. The broader implication is that pre-training objectives do not simply encode language structure in the abstract; they determine which units of meaning—tokens, spans, or sentences—are made geometrically accessible in the representation space.