SemScore: Evaluating Instruction-Tuned LLMs
- SemScore is a semantic evaluation metric that quantifies similarity between model outputs and gold responses using cosine similarity of sentence embeddings.
- It addresses the limitations of manual and overlap-based evaluations, offering a scalable and reproducible approach aligned with human judgment.
- Leveraging models like all-mpnet-base-v2, SemScore achieved top Pearson (0.970) and Kendall’s tau (0.879) correlations in benchmarking instruction-tuned LLMs.
Searching arXiv for the specified paper and closely related evaluation work. SemScore is a reference-based automatic evaluation metric for instruction-tuned LLMs that scores a model response by measuring semantic textual similarity between the generated output and a gold target response. It was introduced to address a central evaluation problem in instruction following: manual judgments are slow and expensive, while overlap-based metrics can underrate responses that are semantically correct but lexically different. In the reported experiments, SemScore achieved the strongest correlation with human evaluation among the compared automatic metrics, indicating that semantic similarity is a particularly effective basis for evaluating instruction-tuned outputs (Aynetdinov et al., 2024).
1. Motivation and problem setting
Instruction-tuned LLMs are commonly assessed by how well they respond to natural-language instructions, but the paper argues that prevailing evaluation practice has two structural weaknesses. First, manual evaluation is time-consuming and does not scale well to the comparison of many models or model variants. Second, traditional reference-based text-generation metrics such as BLEU and ROUGE-L are poorly matched to instruction-following tasks in which a response may be correct despite having low lexical overlap with the reference. This issue is especially acute when evaluation datasets contain only one human-written target response and when tasks involve code or other structured text (Aynetdinov et al., 2024).
The motivating principle behind SemScore is that, for instruction-following evaluation, meaning match matters more than exact word overlap. The paper gives a concrete example in which a response receives a human A-rating while BLEU and ROUGE-L assign very low scores because the wording differs from the gold answer despite semantic equivalence. SemScore is proposed as a direct remedy: instead of asking whether two responses share tokens, it asks whether they express similar semantics (Aynetdinov et al., 2024).
2. Formal definition and computation
SemScore is computed in three steps. The model output and the gold target response are embedded separately with a sentence transformer, cosine similarity is computed between the two embeddings, and that cosine similarity is used as the score. In mathematical form,
where is the model-generated output, is the gold reference response, and are sentence embeddings. The score lies in : values near $1$ indicate highly similar semantics, values near $0$ indicate unrelated content, and negative values indicate semantically opposite content (Aynetdinov et al., 2024).
The default embedding backbone is all-mpnet-base-v2, which is based on MPNet-Base and fine-tuned with a contrastive objective on 1 billion sentence pairs. The paper also studies encoder choice through an ablation with deberta-xlarge-mnli, comparing CLS token pooling and mean pooling. This comparison is important because SemScore is not defined independently of the embedding model: the sentence encoder determines the geometry in which semantic similarity is measured (Aynetdinov et al., 2024).
3. Benchmark design and evaluation protocol
The empirical study uses the evaluation dataset from Wang et al. (2023c), comprising 252 instructions spanning text completion, blog post suggestions, coding, formal logic, and general language tasks. Each instruction is paired with a model output, a human-written target response, and human ratings from the original study. The paper extends prior human evaluation by manually evaluating four additional models, yielding a total of 12 instruction-tuned LLMs: Vanilla GPT-3, GPT-3 + T0, GPT-3 + SuperNI, GPT-3 Self-Inst, GPT-3 Self-Inst + SuperNI, text-davinci-001, text-davinci-002, text-davinci-003, gpt-3.5-turbo, GPT-4, LLaMA, and Alpaca (Aynetdinov et al., 2024).
Human evaluation uses a 4-point category system: A for a correct and satisfying response, B for an acceptable response with minor imperfections, C for a response with significant errors, and D for an irrelevant or invalid response. For ranking, these labels are converted to numeric scores $1$–$4$ and averaged over the dataset. Automatic metrics are then compared against these averaged human scores using Kendall’s rank correlation and Pearson’s 0 correlation (Aynetdinov et al., 2024).
SemScore is evaluated against eight widely used metrics: BLEU, ROUGE-L, BERTScore, BLEURT, BARTScore, BARTScorepara, DiscoScore, and G-Eval. G-Eval is further instantiated as G-Eval-4, G-Eval-3.5, and G-Eval-3.5-instruct. For fairness, when G-Eval uses a model as evaluator, outputs generated by that same model are excluded in order to avoid self-evaluation (Aynetdinov et al., 2024).
4. Empirical performance
The central quantitative result is that SemScore attains the highest reported correlation with human judgment among the compared metrics. Its Pearson correlation of 1 is the strongest value in the benchmark, and its Kendall’s 2 of 3 is also the highest reported value. The comparison reported in the paper is summarized below (Aynetdinov et al., 2024).
| Metric | Kendall 4 | Pearson 5 |
|---|---|---|
| SEMSCORE | 0.879 | 0.970 |
| G-Eval-4* | 0.855 | 0.863 |
| G-Eval-3.5* | 0.855 | 0.831 |
| BERTScore | 0.848 | 0.944 |
| G-Eval-3.5-instruct | 0.840 | 0.911 |
| ROUGE-L | 0.788 | 0.933 |
| BARTScore | 0.788 | 0.621 |
| BARTScorepara | 0.697 | 0.884 |
| BLEU | 0.667 | 0.865 |
| BLEURT | 0.485 | 0.485 |
| DiscoScore | 0.364 | 0.583 |
The paper interprets these results as evidence that sentence-embedding similarity is particularly well aligned with human judgments in instruction-following settings. A noteworthy comparison is BERTScore: SemScore slightly outperforms it despite BERTScore using a larger and stronger transformer backbone for sentence-level similarity. This suggests that direct response-to-reference sentence embedding similarity is an effective evaluation signal for the benchmark considered (Aynetdinov et al., 2024).
5. Encoder ablations and interpretive analysis
The encoder ablation shows that the effectiveness of SemScore depends in part on how the sentence representation is formed. Using deberta-xlarge-mnli with mean pooling produces stronger results than using the CLS token, while the default SemScore configuration with all-mpnet-base-v2 remains best overall (Aynetdinov et al., 2024).
| Variant | Kendall 6 | Pearson 7 |
|---|---|---|
| SEMSCORE | 0.879 | 0.970 |
| SEMSCORE-DeBERTa-Mean | 0.870 | 0.929 |
| BERTScore | 0.848 | 0.944 |
| SEMSCORE-DeBERTa-CLS | 0.756 | 0.892 |
Two technical conclusions are explicit in the paper. First, sentence-embedding-based scoring is effective for this evaluation setting. Second, mean pooling works better than CLS token pooling in the reported ablation. The qualitative interpretation is that SemScore succeeds because it captures meaning-level similarity rather than surface-form similarity, which is particularly useful when a response is valid but paraphrased, when lexical overlap is low, or when the task is not naturally suited to n-gram matching (Aynetdinov et al., 2024).
6. Uses, strengths, and limitations
SemScore is presented as simple, cheap, reproducible, effective, and interpretable. Operationally, it is most appropriate when evaluation has three components: an instruction, a generated response, and a human-written gold target response. In that setting it can support model comparison, regression testing, rapid iteration during instruction-tuning, and large-scale evaluation where manual scoring is infeasible (Aynetdinov et al., 2024).
Its limitations are equally explicit. SemScore requires a gold-standard target response and therefore cannot evaluate outputs without a reference. Its behavior depends on the chosen sentence transformer, and the paper specifically recommends all-mpnet-base-v2. It is not uniformly successful on all task types: the appendix notes difficult cases, especially for some code-related or unusual tasks, and the metric can over- or under-penalize extra content. The benchmark itself contains only 252 instructions, which is broad but not exhaustive. As a reference-based metric, SemScore also inherits any incompleteness or suboptimality in the gold response (Aynetdinov et al., 2024).
A common misconception is that semantic similarity alone makes an evaluator universally applicable. The paper does not support that stronger claim. What it does support is narrower and more precise: in a benchmark of 12 instruction-tuned LLMs evaluated against human-written references, cosine similarity between sentence embeddings correlates with human judgment better than the compared alternatives (Aynetdinov et al., 2024).
7. Place within instruction-tuned LLM evaluation
SemScore occupies a specific position in the evaluation landscape. It is neither a manual protocol nor an LLM-as-a-judge method; it is a deterministic reference-based metric that replaces surface overlap with semantic textual similarity. In the reported study, it outperforms both classical overlap metrics and the tested G-Eval variants in correlation with human judgments, while avoiding dependence on a proprietary evaluator model (Aynetdinov et al., 2024).
Its broader significance lies in the methodological claim embedded in its design: for instruction-following outputs, evaluation should prioritize semantic equivalence over lexical identity. This does not eliminate the need for human judgment, nor does it remove the constraints of reference-based evaluation, but it suggests that a simple STS-based formulation can provide a strong automatic proxy when scalable and reproducible assessment is required (Aynetdinov et al., 2024).