---
title: Conditional Semantic Textual Similarity (C-STS)
url: https://www.emergentmind.com/topics/conditional-semantic-textual-similarity-c-sts
type: topic
---

# Conditional Semantic Textual Similarity (C-STS)

Searching arXiv for recent and foundational papers on Conditional Semantic Textual Similarity and closely related STS methodology.
Conditional Semantic Textual Similarity (C-STS) is a semantic similarity task in which the similarity of two sentences is judged with respect to an explicit natural-language condition rather than overall sentence meaning. In its standard formulation, each instance is a quadruplet \(\{s_1, s_2, c, y\}\), where \(s_1\) and \(s_2\) are sentences, \(c\) is a free-form condition, and \(y\) is a similarity score on a 5-point Likert scale [2305.15093]. The task was introduced to address a central ambiguity in ordinary STS: the same sentence pair can be similar under one aspect and dissimilar under another. The original benchmark established C-STS as a difficult problem; even models performing well on STS, including SimCSE, Flan-T5, and GPT-4, achieved Spearman correlation scores below 50 on the task [2305.15093].

## 1. Task formulation and semantic motivation

The conceptual basis of C-STS is that semantic similarity is not a single scalar property of a sentence pair, but depends on the aspect under consideration. The original formulation makes this point by describing standard STS as a latent mixture over possible conditions,
\[
\sum_{i=1}^{|\mathcal{C}|} w_i~\textrm{sim}_{c_i}\left (s_1, s_2 \right)\quad s.t. \sum_i w_i = 1,
\]
where annotators implicitly weight different semantic aspects \(c_i\) when assigning an unconditional similarity score [2305.15093]. C-STS removes that latent ambiguity by making the relevant aspect explicit.

Under this formulation, the same sentence pair can receive sharply different judgments depending on the condition. The benchmark paper illustrates this with examples such as two sentences about a basketball shot and a tennis serve: under “The motion of the ball,” the pair is highly similar, whereas under “The size of the ball,” it is not [2305.15093]. This conditionalization is intended both to reduce subjectivity and to enable fine-grained evaluation of language understanding across visual properties, physical reasoning, event structure, object attributes, and related semantic phenomena [2305.15093].

The label space is ordinal. In the original benchmark, annotators use a 5-point Likert scale, with 1 denoting complete dissimilarity and 5 denoting complete equivalence under the condition [2305.15093]. Later work preserves this basic structure, and one reinforcement-learning formulation restates each sample as \(x=(t_1,t_2,c,y)\), where \(y \in [1,5]\) [2510.04080]. Because the target is ordinal rather than categorical, Spearman correlation became the principal evaluation metric in the benchmark literature [2305.15093].

## 2. Benchmark construction and annotation design

The original C-STS benchmark, C-STS-2023, contains 18,908 instances after filtering and quality assurance, derived from an initial pool of 20,000 examples [2305.15093]. The final split is 60% train, 15% validation, and 25% test [2305.15093]. Sentence pairs were sourced from image-caption corpora, specifically the MS COCO 2014 train split and Flickr30K, with the motivation that image retrieval can surface semantically interesting caption pairs that text-only retrieval may not [2305.15093].

Construction proceeded in two stages. First, similar images were retrieved using CLIP-ViT embeddings; for each image \(I_i\), the method selected top-\(k\) nearest image neighbors by cosine similarity, then converted image pairs into caption pairs through the Cartesian product of caption sets [2305.15093]. Second, the resulting sentence pairs were filtered to retain moderately ambiguous cases rather than trivially similar or trivially dissimilar ones. The filtering criteria included bag-of-words intersection-over-union after stopword removal, cosine distance between SimCSE embeddings, a sentence-length ratio, and a minimum character-length constraint [2305.15093]. For COCO, the thresholds were \(k=64\), \(iou \le 0.12\), \(d_{\text{text}} \ge 0.4\), ratio \(\ge 0.7\), and length \(\ge 50\); for Flickr30K, they were \(k=128\), \(iou \le 0.2\), \(d_{\text{text}} \ge 0.4\), ratio \(\ge 0.7\), and length \(\ge 48\) [2305.15093].

Annotation was organized into three stages on Mechanical Turk. Workers were first qualified on writing conditions and making conditional similarity judgments, yielding a pool of 271 workers [2305.15093]. For each sentence pair, qualified annotators wrote two conditions: one intended to produce high similarity, \(c_{\rm high}\), and one intended to produce low similarity, \(c_{\rm low}\) [2305.15093]. New annotators then verified that each condition was relevant to the sentence pair and assigned a similarity score on the 5-point scale [2305.15093]. The paper reports Fleiss’ kappa of 0.61 on a random sample of 100 examples with three annotations each, as well as an average sentence length of 12.6 words and an average condition length of 5.3 words [2305.15093].

The benchmark’s open-ended condition design is central to its scope. Conditions are free-form natural-language phrases rather than members of a fixed taxonomy, and examples include “The color of the object,” “The size of the object,” “The arrangement of objects,” and “The number of people” [2305.15093]. This design broadens the semantic coverage of the task, but later work argues that it also introduces major annotation and definitional difficulties.

## 3. Annotation problems, reannotation, and data cleaning

Subsequent work identified substantial quality problems in the original benchmark. “Linguistically Conditioned Semantic Textual Similarity” reports that the validation set suffers from annotation errors, ill-defined conditions, and ambiguity in the task definition, and that these issues can impede proper evaluation [2406.03673]. In a pilot study on 150 instances sampled to reflect different condition-feature frequencies, the authors observed only 40% exact label match with the original labels and a Spearman correlation of 50.4 [2406.03673]. On the full validation split, they report annotator discrepancy on 55% of the instances [2406.03673].

That reannotation study organizes the disagreements into four explicit cases: incommensurable mapping, ambiguous condition, inference degree, and invalid condition [2406.03673]. Binary conditions, such as some number-related comparisons, were restricted to labels 1, 5, or 3, corresponding to equal, unequal, or possibly equal [2406.03673]. Ambiguous singleton conditions were defaulted to the entity type, invalid cases were assigned label \(-1\), and 214 invalid instances were removed, producing a relabeled validation set of 2,620 samples [2406.03673]. The paper’s examples show that some conditions are effectively binary rather than 5-way ordinal, some are underspecified expressions such as “the table” or “the kite,” and some require implicit decisions about how much inference from context is permitted [2406.03673].

The same paper turns reannotation into an automatic error-detection problem by rewriting a condition as a question, generating answers for each sentence with GPT-3.5 or GPT-4, and then comparing answer embeddings [2406.03673]. The authors report that the cosine similarity of GPT-generated answer embeddings has Spearman correlation 55.44 with the reannotated labels, compared with 49.22 between the original and reannotated labels [2406.03673]. Their automatic pipeline clusters answer pairs with k-means, ranks answer-pair cosine similarities within each cluster, maps ranks to labels 1 through 5 according to a preset distribution, and flags mismatches between the new label and the original one as likely annotation errors [2406.03673]. The best reported result is 82.4 F1 with clustering and the reannotated label distribution, with Spearman 65.7; even without clustering, the pipeline reaches 79.4 F1 under the reannotated distribution [2406.03673].

A later paper extends the cleaning effort from validation to training data. “Annotating Training Data for Conditional Semantic Textual Similarity Measurement using Large Language Models” argues that the original dataset still contains major problems in both conditions and ratings, including imbalanced conditions, subjective conditions, inconsistent phrasing styles, varying granularity, ambiguous conditions, invalid conditions, unrelated conditions, and overlap between train and test [2509.14399]. Using GPT-4o to refine conditions and GPT-4o plus Claude-3.7-Sonnet to reannotate ratings, the paper cleans 14,176 training instances while leaving the test set unchanged [2509.14399]. Ratings are aggregated by taking the arithmetic mean of the original human label, the GPT-4o label, and the Claude label, then rounding to the nearest integer [2509.14399].

That cleaning study includes several quality-control measures. Repeated annotation of 100 instances with Claude-3.7-Sonnet produced Krippendorff’s Alpha \(=0.865\) [2509.14399]. Manual verification of 300 instances suggested that 23% of original human ratings were inaccurate, about one-third of those inaccuracies were serious, and only 9% of reannotated labels differed from humans by 1 point [2509.14399]. The appendix also reports substantial overlap in the original dataset, including 1,196 sentence-only overlaps, 804 condition-only overlaps, 185 single sentence-condition overlaps, 9 sentence-pair overlaps, and 2 full-instance overlaps [2509.14399]. These findings established dataset quality as a central issue in C-STS research rather than a peripheral concern.

## 4. Modeling paradigms and performance trajectories

The original C-STS paper evaluated cross-encoder, bi-encoder, and tri-encoder architectures. In the bi-encoder setting, conditional similarity is scored as
\[
\text{cos}(f_\theta(s_1;c), f_\theta(s_2;c)),
\]
whereas the cross-encoder directly models the triple \((s_1,s_2,c)\) and the tri-encoder separately combines sentence and condition representations [2305.15093]. Because each sentence pair is paired with both a high-similarity and a low-similarity condition, the paper also introduced a quadruplet contrastive loss,
\[
\text{Quad}(p_1, p_2, n_1, n_2) = \max(\lambda + \text{cos}(n_1, n_2) - \text{cos}(p_1, p_2), 0),
\]
and experimented with MSE, Quad, and Quad + MSE training objectives [2305.15093].

The early empirical picture was sobering. Fine-tuning helped, but the task remained difficult: the best fine-tuned test result reported in the original study was SimCSE-large bi-encoder at 47.5 Spearman and 47.6 Pearson, followed by DiffCSE-base bi-encoder at 43.4 Spearman and RoBERTa-large cross-encoder at 40.7 Spearman [2305.15093]. Tri-encoder variants were generally weaker [2305.15093]. Instruction-tuned LLMs improved with prompting, and GPT-4 reached about 43.6 Spearman in the short-instruction 4-shot setting, but the benchmark paper emphasized that strong STS models and strong LLMs alike struggled with C-STS [2305.15093].

A major shift came from decomposing C-STS into answer generation and answer comparison. The QA-based method in [2406.03673] rewrites the condition as a question, generates one answer per sentence, and then measures the similarity of the two answers rather than the original sentences. Using supervised SimCSE backbones, this transformation produced large gains across architectures. In the bi-encoder setting, SimCSE-base improved from 49.6 to 73.9 Spearman, and SimCSE-large improved from 71.7 to 75.9 [2406.03673]. In the cross-encoder setting, SimCSE-base increased from 37.2 to 71.4 and SimCSE-large from 43.0 to 72.9 [2406.03673]. In the tri-encoder setting, the QA-based version reached 73.9 Spearman [2406.03673]. The same reformulation also improved zero-shot prompting for GPT-3.5 from 9.1 to 66.1 Spearman, while GPT-4 remained roughly unchanged at 64.2 versus 64.4 [2406.03673]. The paper’s interpretation is that answer generation makes the condition more explicit and reduces the reasoning burden on the similarity model [2406.03673].

Another influential line embeds the condition rather than the sentence. CASE, or Condition-Aware Sentence Embeddings, represents a sentence under a condition by encoding the condition with a prompt that contains the sentence,
\[
f(c; I(s)),
\]
and then subtracting an unconditional condition embedding \(f(c; I(\emptyset))\) before applying a supervised projection [2503.17279]. The intended CASE representation is
\[
{\rm CASE}(s,c) = g\!\left(f(c; I(s)) - f(c; I(\emptyset)); \theta\right),
\]
with training by mean squared error between the human rating and the cosine similarity of paired CASE embeddings [2503.17279]. The paper reports that encoding the condition given the sentence is consistently better than encoding the sentence given the condition, that subtracting the unconditional condition embedding improves both Spearman and accuracy almost everywhere, and that LLM-based embeddings outperform MLM-based ones on the benchmark [2503.17279]. At 512 dimensions, supervised projections dramatically outperformed PCA and ICA; for NV-Embed-v2, the reported Spearman values were 69.08 for nonlinear MLP, 69.95 for linear MLP, 32.04 for PCA, and 19.13 for ICA [2503.17279].

The cleaned-data paper [2509.14399] then showed that annotation quality itself materially changes model performance. Using Supervised Non-Linear Projection (SNPro), a Siamese bi-encoder style model with a 2-layer feed-forward projection network, and condition-aware embeddings generated by prompts such as “Retrieve semantically similar texts to the [CONDITION], given the Sentence: [SENTENCE],” the authors reported the strongest result when training on cleaned and reannotated data: Train-Mod-Reanno \(\rightarrow\) ReTest-Mod yielded 73.93 Spearman [2509.14399]. They state that this corresponds to a 5.4% statistically significant improvement in Spearman correlation over the prior best bi-encoder performance with the same architecture and training settings [2509.14399].

Reinforcement learning added a further modeling direction. PoLi-RL formulates C-STS as a ranking-oriented RL problem and argues that RL is well suited because the benchmark metric is Spearman correlation, which is rank-based and non-differentiable [2510.04080]. Its two-stage curriculum first uses pointwise and binary rewards, then introduces a hybrid reward with pointwise, pairwise, and listwise components, together with the Parallel Slice Ranking Reward (PSRR) mechanism for granular credit assignment [2510.04080]. On the official benchmark, PoLi-RL achieved 48.18 Spearman and 48.27 Pearson, which the paper presents as a new state of the art for the cross-encoder architecture [2510.04080]. This did not surpass the strongest recent bi-encoder-style systems built on cleaned data, but it established RL as a viable optimization strategy for C-STS [2510.04080].

A related sentence-embedding paper, “2-Tier SimCSE,” is not a C-STS model because it does not encode an explicit condition, but it is relevant as an STS optimization reference. It reports that contrastive fine-tuning on STS improves sentence representations substantially, with supervised SimCSE stronger than unsupervised SimCSE and a sequential unsupervised-then-supervised “2-Tier” procedure giving the best STS dev Pearson at 0.811 [2501.13758]. The paper explicitly notes that this is unconditional STS rather than condition-specific similarity, which suggests that C-STS requires additional machinery beyond generic sentence-embedding improvement [2501.13758].

## 5. Similarity functions, thresholds, and optimization criteria

C-STS inherits many methodological questions from STS, especially the question of how similarity should be measured once representations have been constructed. A relevant STS analysis shows that, for common word vectors with means concentrated near zero, cosine similarity is essentially equivalent to the Pearson correlation coefficient, and reports that only 0.03% of mean values exceed magnitude 0.05 across popular embeddings [1905.07790]. The same work argues that Pearson correlation is appropriate only for some embedding families and is fragile under non-normality and outliers; when embeddings are non-normal, rank correlations such as Spearman’s \(\rho\) and Kendall’s \(\tau\) can significantly improve performance on word-level and sentence-level STS benchmarks [1905.07790]. This suggests that in C-STS, where conditioning may alter the distribution of sentence representations across conditions, the choice between cosine, Pearson, and rank-based alternatives is itself part of the model design rather than a fixed constant.

The benchmark literature’s emphasis on Spearman correlation reflects this broader concern with ranking rather than absolute calibration. The original C-STS benchmark used Spearman as its primary metric because the labels are ordinal Likert judgments [2305.15093]. Later work retained Spearman for model comparison and, in some cases, reported Pearson alongside it [2406.03673; 2509.14399]. PoLi-RL goes further by directly aligning training with rank-based evaluation through RL reward design [2510.04080].

A different but related perspective comes from a binary STS setting in which similarity is operationalized as a thresholded decision. “A Distribution-Based Threshold for Determining Sentence Similarity” addresses a task where two sentences are “similar” only if they match with respect to highly specific information such as names, addresses, or identification codes [2311.16675]. The architecture uses a Siamese network with a frozen Transformer-based multilingual Universal Sentence Encoder producing embeddings in \(\mathbb{R}^{512}\), followed by a dense layer with 50 ReLU units and a distance function—Manhattan, Euclidean, or Minkowski with \(p=3\)—trained with mean squared error, Adam, learning rate 0.005, batch size 512, gradient clipping with clip norm 2.0, and early stopping up to 100 epochs [2311.16675]. After training, the method computes the distributions of predicted distances for similar and dissimilar pairs and defines the threshold \(\tau\) as the empirical intersection point of the two class distributions [2311.16675]. The resulting rule is
\[
\hat{y}= \begin{cases} \text{similar} & \text{if } d(\mathbf{x},\mathbf{y}) < \tau,\\
\text{dissimilar} & \text{otherwise}.
\end{cases}
\]
Although this work is not framed as C-STS, it is closely aligned in spirit because similarity is determined by a task-specific matching criterion rather than global paraphrase equivalence [2311.16675]. A plausible implication is that some applied C-STS settings may be more naturally modeled as condition-specific decision problems than as pure ordinal regression.

The thresholding study also proposes a distribution-aware “accuracy score” obtained by combining closeness to an ideal class endpoint with closeness to the peak of the empirical class histogram, normalized to \([0,100]\) and applied by interpolation over a stored accuracy table [2311.16675]. This is distinct from the rank-based evaluation protocols in C-STS, but it underscores a general theme across the literature: the semantics of “similar” depend not only on the representation, but also on the decision rule or scoring function imposed on top of that representation.

## 6. Linguistic structuring, misconceptions, and open problems

One common misconception is that adding a condition automatically removes ambiguity. The empirical record does not support that claim. The benchmark paper introduced C-STS specifically to reduce subjectivity and ambiguity [2305.15093], but later reannotation work showed that conditions can themselves be ambiguous, invalid, binary rather than ordinal, or dependent on underspecified inference policies [2406.03673]. The task therefore replaces one kind of ambiguity with a more structured but still difficult annotation problem.

To address this, the reannotation literature proposes a more explicit linguistic basis for conditionality. The typed-feature structure (TFS) discussion in [2406.03673] argues that many current conditions are too coarse because they are surface phrases such as “the food” or “the activity.” In the TFS view, a condition should be grounded in an entity type plus typed lexical features such as type, size, color, participant, or instrument, and the final similarity score is calculated from the weighted sum of the individual similarity labels annotated for each non-empty feature, with the type feature carrying the highest weight [2406.03673]. The paper further suggests that entity types can be drawn from a hierarchy such as WordNet and feature inventories from resources such as Schema.org or ConceptNet [2406.03673]. This does not yet define a standard benchmark, but it offers a linguistic foundation for more principled condition construction.

Another recurrent issue is dataset leakage and domain specificity. The cleaned-data study documents overlap between train and test in the original dataset [2509.14399], while the threshold-based binary STS study notes that its learned threshold depends on the training distribution and may shift if the domain changes [2311.16675]. This suggests that C-STS evaluation depends heavily on both annotation quality and data partition hygiene. A related limitation is language coverage: several papers explicitly note that current work is English-only and that multilingual C-STS datasets do not appear to be publicly available [2503.17279; 2509.14399].

Bias and social sensitivity remain underexplored. The LLM-based cleaning paper warns that conditions involving age, gender, and related attributes may reproduce or amplify social biases already encoded in the annotating LLMs, and states that such bias was not evaluated [2509.14399]. CASE likewise notes that social bias in LLMs and MLMs was not analyzed [2503.17279]. These omissions are consequential because many conditions directly reference human attributes.

The current state of the field therefore combines conceptual clarity with methodological instability. C-STS has established that conditional similarity is a meaningful and distinct problem from ordinary STS [2305.15093]. Reannotation work has shown that benchmark quality materially affects both evaluation and training [2406.03673; 2509.14399]. Condition-aware representation methods such as QA-based decomposition and CASE have produced large gains [2406.03673; 2503.17279]. Reinforcement learning has shown that metric-aligned optimization is feasible [2510.04080]. At the same time, the literature consistently identifies unresolved problems: cleaner and more principled conditions, multilingual resources, bias analysis, overlap control, and model designs that remain robust when the condition changes the relevant semantic subspace.

Source: https://www.emergentmind.com/topics/conditional-semantic-textual-similarity-c-sts