AnglE: Angle-Optimized Text Embeddings

• AnglE is a supervised text-embedding framework applying angle optimization to overcome cosine similarity saturation in semantic similarity training.
• It splits embeddings into real and imaginary components to compute precise angular differences, enhancing training signals alongside cosine and in-batch negatives.
• AnglE integrates with backbone models like BERT and LLaMA, achieving superior performance on both short-text and long-text STS benchmarks.

AnglE, short for Angle-optimized Text Embeddings, is a supervised text-embedding framework designed for semantic textual similarity (STS) and related retrieval settings. Its central claim is that many supervised sentence-embedding objectives optimize cosine similarity, but cosine has saturation zones in which the gradient becomes very small; as a result, learning signals can become weak when sentence pairs are already strongly aligned or strongly misaligned. AnglE addresses this by introducing angle optimization in a complex space, while retaining cosine-based and in-batch-negative terms in a composite objective. The framework is presented as a training method that can be placed on top of backbone encoders such as BERT, RoBERTa, and LLaMA, and is evaluated on short-text STS, a newly collected long-text STS benchmark, domain-specific low-resource STS, and LLM-annotated data (Li et al., 2023).

1. Conceptual basis and motivation

AnglE is motivated by a specific failure mode in supervised sentence embedding: objectives based on cosine similarity can suffer from vanishing gradients because the cosine function contains saturation zones. In the formulation used for supervised STS, the standard cosine-based ranking loss is

$$\mathcal{L}_{cos} = \log \left[1 + \sum_{s(\mathbf{X}_i,\mathbf{X}_j) > s(\mathbf{X}_m,\mathbf{X}_n)} \exp\left(\frac{\cos(\mathbf{X}_m,\mathbf{X}_n)-\cos(\mathbf{X}_i,\mathbf{X}_j)}{\tau}\right)\right],$$

where $\tau$ is a temperature and $s(\cdot,\cdot)$ denotes the human similarity ordering. The intended effect is to enforce larger cosine similarity for more similar sentence pairs than for less similar ones. The reported problem is not that cosine similarity is unsuitable in itself, but that its slope becomes very small near the extremes of the cosine range, reducing the informativeness of the loss for optimization (Li et al., 2023).

This issue is described as particularly relevant for STS-style datasets such as MRPC or QQP, where labels are often binary. Under such labeling regimes, many pairs accumulate in high-similarity or low-similarity regions, precisely where cosine-based objectives may provide little gradient. AnglE therefore shifts the optimization target from cosine alone to the angle difference derived from a complex-space representation. The stated intuition is that even when cosine values lie in a saturation zone, the corresponding angle can still provide a meaningful optimization signal. This suggests that AnglE is best understood not as a replacement for sentence encoders, but as an alternative supervisory geometry for embedding training.

2. Complex-space angle optimization

The core mathematical construction splits each embedding into real and imaginary components. For a pair of sentence embeddings $(\mathbf{X}_i,\mathbf{X}_j)$, AnglE defines

$$\mathbf{z} = \mathbf{a} + \mathbf{b}i,\qquad \mathbf{w} = \mathbf{c} + \mathbf{d}i,$$

with $\mathbf{a}=\mathbf{X}_i^{re}$, $\mathbf{b}=\mathbf{X}_i^{im}$, $\mathbf{c}=\mathbf{X}_j^{re}$, and $\mathbf{d}=\mathbf{X}_j^{im}$. In polar form, complex division is written as

$$\frac{\mathbf{z}}{\mathbf{w}} = \gamma \Delta\theta_{zw}, \qquad \gamma = \frac{r_{\mathbf{z}}}{r_{\mathbf{w}}} = \frac{\sqrt{\mathbf{a}^2+\mathbf{b}^2}}{\sqrt{\mathbf{c}^2+\mathbf{d}^2}}, \qquad \Delta\theta_{zw} = \theta_{\mathbf{z}} - \theta_{\mathbf{w}}.$$

Using the complex division rule,

$\tau$0

AnglE then normalizes by the magnitude ratio $\tau$1 to isolate the angular component:

$\tau$2

This quantity becomes the basis of the angle-optimized loss,

$\tau$3

which encourages smaller angle differences for more similar pairs (Li et al., 2023).

The complete AnglE objective combines cosine supervision, in-batch negatives, and the complex-space angle term:

$\tau$4

The in-batch negative component is

$\tau$5

This term is described as acting both as a contrastive regularizer and as a form of data augmentation. A practical correction is also applied: if identical sentences appear in the same batch but are not explicitly labeled as positives, they would otherwise become false negatives. AnglE therefore detects identical sentence pairs and treats them as positives to reduce training noise. The reported interpretation of the ablation results is that the angle term is the central innovation, but that it works best in combination with the cosine and in-batch-negative terms rather than in isolation.

3. Framework, backbone models, and training procedure

AnglE is explicitly described as not a brand-new encoder. It is a training framework that can be used with backbones including BERT, RoBERTa, and LLaMA. In the main experiments, uncased BERT-base (110M parameters) is used as the default backbone for fairness against BERT-based baselines. In LLaMA-based experiments, fine-tuning is performed with LoRA, and the prompt is

“Summarize sentence {sentence} in one word:”

The training pipeline follows the usual sentence-embedding procedure: sentences are tokenized and padded to length $\tau$6, embedded, encoded by the backbone, and then pooled. Among the tested pooling variants, the paper reports that CLS pooling performs best (Li et al., 2023).

Hyperparameters are selected by grid search. The reported temperatures are $\tau$7 for the cosine and in-batch-negative objectives, and $\tau$8 for the angle objective. For transfer experiments, the maximum sequence length is 128, and batch sizes up to 50 are explored. Larger batches are reported to help, but experiments could not go higher because of GPU memory constraints. This practical detail is used to frame one limitation of the method: performance may still benefit from larger effective batch sizes, but the reported experiments were resource-bounded.

The framework also extends to long-text and LLM-based settings. The paper reports experiments with long-text backbones and with pseudo-labels produced by LLMs, including ChatGPT, LLaMA, ChatGLM, and an ensemble of them. In that sense, AnglE is presented as a supervision scheme that can be combined with different encoder scales and different label-generation regimes, rather than as a single fixed architecture.

4. Evaluation datasets and benchmark design

A major part of the AnglE study is the benchmark construction. The method is evaluated on standard short-text STS benchmarks: MRPC, QQP, QNLI, STS 2012–2016, SICK-R, and STS-B. To supplement these established tasks, the paper introduces a new GitHub Issues Similarity Dataset intended for long-text STS (Li et al., 2023).

The GitHub Issues dataset is collected from 55 popular open-source repositories using the GitHub API. Duplicate issues are treated as positive pairs and non-duplicate issues as negative pairs. The reported dataset size is about 21K samples, with the following splits.

Split	Total	Positive / Negative
Train	18,565	9,457 / 9,108
Validation	1,547	774 / 773
Test	1,548	807 / 741

A notable property of this dataset is that around 60%+ of samples in each split have token length greater than 512. The paper presents this as evidence that the benchmark is genuinely long-text, unlike most sentence-similarity datasets dominated by short inputs. On this basis, the long-text evaluation is meant to probe whether an embedding method trained for semantic similarity remains effective when documents substantially exceed standard sentence lengths.

The study also includes domain-specific low-resource STS and LLM-annotated data. The low-resource scenario is simulated by training on pseudo-labeled examples generated by LLMs from STS-style inputs. A plausible implication is that the authors regard label scarcity, rather than modeling capacity alone, as a central bottleneck for domain-specific embedding models. Within the paper’s scope, LLM supervision is proposed as a workaround for that bottleneck.

5. Empirical results

On the standard transfer STS setting, where the model is trained on MNLI and SNLI and then evaluated on seven STS benchmarks, AnglE is reported to improve over prior baselines. The paper gives the following average Spearman scores: AnglE-BERT = 82.37 and AnglE-LLaMA2-7B = 85.96, with the latter identified as the best overall result in the reported table. The paper also notes that AnglE-LLaMA2-7B exceeds the previous best SimCSE-LLaMA2-7B average of 85.24 (Li et al., 2023).

In the non-transfer supervised setting, AnglE-BERT reaches an average of 73.55 across MRPC, STS-B, QQP, QNLI, and the GitHub Issues dataset, while SBERT reaches 68.03. On the long-text GitHub Issues benchmark specifically, AnglE-RAN achieves 71.25, slightly exceeding AnglE-BERT at 70.55. The interpretation given in the paper is that a long-text backbone can be advantageous for long documents.

The transfer-task appendix reports that AnglE-LLaMA reaches 91.38 average accuracy over several downstream classification tasks, outperforming SimCSE-RoBERTa and DiffCSE-RoBERTa. The paper also reports a retrieval gain on Flickr30k caption matching, stating that AnglE achieves higher strict accuracy than SimCSE and SBERT. These results are used to position AnglE not only as an STS method narrowly defined, but as a general-purpose embedding framework whose benefits transfer to retrieval and classification settings.

The results are presented as evidence that angle optimization mitigates the deficiencies of pure cosine-based supervision, especially in settings where similarity judgments cluster near the extremes. The paper further states that the cosine-similarity distribution produced by AnglE more closely matches the “gold” similarity distribution than those of SBERT or supervised SimCSE, with particular improvement in the low- and high-similarity regions associated with cosine saturation zones.

6. Ablation findings, interpretation, and limitations

The ablation study is one of the most informative parts of the AnglE presentation. On STS-B, AnglE-BERT-all obtains 86.26 Spearman. Removing the in-batch negative term yields 86.00, while removing the angle objective yields 85.30. Using only cosine gives 85.28, using only angle gives 85.15, and using only the in-batch negative term yields 72.48. The paper interprets these outcomes as showing that the angle objective is the key innovation, but that the full performance gain depends on combining it with supervised cosine training and in-batch negatives rather than relying on any single term alone (Li et al., 2023).

The study also reports that CLS pooling is best among the tested pooling methods, and that identical-sentence-pair detection within the in-batch-negative mechanism yields a small but consistent gain of about 0.18%. These results reinforce the characterization of AnglE as a framework with both conceptual and implementation-level components: the complex-space angle objective is the principal novelty, but practical details such as pooling choice and false-negative handling are also consequential.

The paper identifies several limitations. AnglE remains a supervised framework, so it depends on labeled data or pseudo-labels. Data scarcity in domain-specific settings is explicitly noted as an ongoing challenge, motivating the LLM-supervised variant. Another limitation is computational: larger batch sizes appear beneficial, but the experiments were constrained by GPU memory. A plausible implication is that the reported results may not exhaust the attainable performance of the method under more favorable training budgets.

More broadly, AnglE is positioned as particularly useful when embedding quality depends on fine-grained semantic similarity, when labels are coarse, when data are imbalanced toward extreme similarity values, or when texts are long. Within that framing, its distinctive contribution is not a new encoder family, but a modified optimization geometry for supervised embedding learning that seeks to remain informative where cosine-based objectives become flat.