SNCSE: Soft Negative Contrastive Sentence Embedding

• SNCSE is an unsupervised sentence representation framework that distinguishes semantic meaning by incorporating soft negative samples for finer contrast.
• It uses a bidirectional margin loss combined with InfoNCE to separate semantically opposite yet textually similar sentences, mitigating feature suppression.
• Experimental results on STS datasets show SNCSE improves Spearman’s correlations by up to 2.3% over traditional contrastive approaches.

The SNCSE (Soft Negative Contrastive Sentence Embedding) framework is an unsupervised sentence representation learning method designed to address the limitations of standard contrastive approaches in semantic textual similarity (STS) tasks. By incorporating "soft negative samples"—sentences structurally similar but semantically different from the anchor—together with a bidirectional margin loss, SNCSE seeks to disentangle textual and semantic similarity, overcoming feature suppression effects seen in prior unsupervised contrastive learning frameworks (Wang et al., 2022).

1. Motivation: Feature Suppression and Semantic Decoupling

Unsupervised contrastive learning methods such as SimCSE typically generate positive sentence pairs through data augmentation (e.g., dropout), while all other sentences in the minibatch serve as hard negatives. These frameworks primarily distinguish sentence representations by pulling together embeddings of augmented views and pushing apart all others under an InfoNCE loss. However, this leads to two key deficiencies:

• Feature Suppression: Since "positive" pairs are generated from virtually identical text, the encoder exploits surface cues, leading to trivial alignment based on bag-of-words overlap rather than true semantic consistency. This is analogous to feature suppression observed in vision contrastive learning.
• Conflation of Textual and Semantic Similarity: The model fails to distinguish semantic from lexical similarity. Sentences with high word overlap but divergent meanings (e.g., a statement and its negation) may be incorrectly embedded close together.

SNCSE introduces explicit soft negatives—textually similar but semantically opposing variants (e.g., negated sentences)—forcing the model to resolve fine-grained semantic distinctions even under high surface similarity.

2. Construction of Soft Negative Samples

Soft negative samples in SNCSE are generated by applying explicit negation to the anchor sentence via a rule-based pipeline (e.g., spaCy-based). This modifies the main verb or auxiliary form by inserting or prepending the negation "not," restricting to explicit negations and avoiding semantic drift from antonym substitution.

Examples:

Original	Soft Negative
Tom and Jerry became good friends.	Tom and Jerry did not become good friends.
She enjoys playing tennis.	She does not enjoy playing tennis.

The methodology ensures that the soft negative preserves nearly all tokens and grammatical structure from the source, contrasting sharply in semantic content. By forcibly distancing the anchor and its soft negative in embedding space, SNCSE compels the encoder to model semantic differences beyond surface similarity.

3. Objective Function: Bidirectional Margin Loss and InfoNCE

Let $X_i$ denote the original sentence, $X_i^+$ its positive view (dropout-augmented), and $X_i^\#$ its soft negative (negated version). Each sample has in-batch hard negatives $\{X_j^-\}$. The SNCSE loss involves:

• InfoNCE Loss: Encourages similarity between $h_i$ and $h_i^+$, and dissimilarity with in-batch hard negatives, using cosine similarity and temperature $\tau$:

$$\mathcal{L}_{\text{InfoNCE}} = -\frac{1}{N}\sum_{i=1}^N \log \frac{\exp(\text{sim}(h_i, h_i^+)/\tau)}{\exp(\text{sim}(h_i, h_i^+)/\tau) + \sum_{j=1}^N \exp(\text{sim}(h_i, h_j^-)/\tau)}$$

• Bidirectional Margin Loss (BML): Defines a margin $m$ between positive and soft negative similarities, penalizing positives that fall below $m$ or soft negatives above $m$:

$$\mathcal{L}_{\text{BML}(i)} = [m - \text{sim}(h_i, h_i^+)]_+ + [\text{sim}(h_i, h_i^\#) - m]_+$$

where $[x]_+ = \max(x, 0)$.

• Total Loss: The final SNCSE objective combines both terms:

$$\mathcal{L}_{\text{SNCSE}} = \mathcal{L}_{\text{InfoNCE}} + \lambda \frac{1}{N}\sum_{i=1}^N \mathcal{L}_{\text{BML}(i)}$$

with $\lambda$ controlling the trade-off.

This explicit margin-based penalization enforces separation of soft negatives and anchors, directly targeting the confound between surface similarity and semantic overlap.

4. Training Protocol and Hyperparameterization

SNCSE employs a minibatch-based training regime. Hyperparameters include batch size $B$ (default 256 for base models), temperature $\tau = 0.05$, margin $m \approx 0.2$ (set via $\alpha = 0.1$, $\beta = 0.3$), and $\lambda$ (1e-3 for BERT-base, 5e-4 for RoBERTa-base). Learning rates follow standard transformer tuning practices: $1\times 10^{-5}$ (base) and $5\times 10^{-6}$ (large).

Algorithmic Steps:

1. Sample $B$ sentences $\{X_i\}$.
2. For each $X_i$, create $X_i^+$ (dropout-based) and $X_i^\#$ (negated).
3. Obtain embeddings ($h_i$, $h_i^+$, $h_i^\#$) via a shared Transformer + MLP encoder.
4. Compute cosine similarities among anchor, positive, in-batch negatives, and soft negatives.
5. Evaluate $\mathcal{L}_{\text{InfoNCE}}$ and per-instance $\mathcal{L}_{\text{BML}(i)}$.
6. Minimize $\mathcal{L}_{\text{SNCSE}}$ via backpropagation.

This process ensures separation of semantic and lexical analogs, leveraging bidirectional margins for robust generalization.

5. Experimental Evaluation and Results

Experiments utilize the STS-12, STS-13, STS-14, STS-15, STS-16, STS-B, and SICK-R datasets, with evaluation via average Spearman’s rank correlation coefficient ($\rho$) across tasks. Encoders include BERT$_{\text{base}}$, RoBERTa$_{\text{base}}$, and their larger counterparts; positive samples are dropout-based as in SimCSE.

Summary of Results:

Model	Encoder	Average $\rho$ (STS)
SNCSE	BERT$_{\text{base}}$	78.97%
ESimCSE	BERT$_{\text{base}}$	78.27%
SNCSE	RoBERTa$_{\text{base}}$	79.23%
ESimCSE	RoBERTa$_{\text{base}}$	77.44%

SNCSE provides absolute improvements over baselines ranging from +0.7% to +2.3%, with ablation showing that:

• Removing BML ($\lambda = 0$) reduces SNCSE (BERT$_{\text{base}}$) from 78.97% to 78.21%.
• Naively treating negations as pure positives or negatives lowers performance further ($\approx$77% and $\approx$74%).
• Best results are obtained for margin intervals $\text{sim}_+ - \text{sim}_- \in [0.1, 0.3]$.

6. Error Analysis and Limitations

A rank-based error metric computes per-pair squared error between estimated and gold similarities, revealing dominant failure modes:

• Typos: Lexical deviations (misspellings) lead to underestimated similarity.
• Negation logic: Over-estimation for explicit or implicit negation pairs persists.
• Textual independence: Paraphrases with little lexical overlap are under-scored.
• Word-order flips: "A kicks B" vs. "B kicks A" are textually similar but semantically opposite, indicating insensitivity to argument structure.

This suggests that SNCSE, while mitigating feature suppression, remains limited by Transformer-level aggregation mechanisms that insufficiently model compositional and syntactic nuance.

7. Future Directions and Extensions

The primary bottleneck is the decoupling of textual from semantic content in sentence embeddings. Potential enhancements include:

• Generation of richer soft negatives (antonym substitution, subject-object swaps, role reversals).
• Adaptive, per-example margins in the BML objective.
• Augmentation with small-scale labeled NLI or STS supervision for targeted semantic distinctions.
• Architectural extensions that explicitly encode negation and argument structure, such as polarity flags or syntactic tree encoders.

SNCSE advances unsupervised contrastive sentence embedding by forcing representations to reflect genuine semantic relations even for sentences with nearly identical surface forms, primarily through explicit soft negative construction and the bidirectional margin loss (Wang et al., 2022).