Semantic F1 Scores Explained

• Semantic F1 scores extend traditional F1 by incorporating graded semantic similarity to give partial credit for near-miss predictions.
• They are applied in diverse tasks such as duplicate question detection, semantic role labeling, relation extraction, and medical segmentation.
• Their task-dependent design adjusts evaluation units and averaging schemes to ensure precise calibration, protocol integrity, and meaningful comparisons.

Searching arXiv for the cited papers to ground the article in current literature. Semantic F1 scores are evaluation measures that use the F-measure to assess semantic predictions rather than purely surface-form matches. In the classical setting, they are ordinary precision–recall harmonic means computed on semantic units such as duplicate questions, labeled predicate–argument spans, semantic relations, retrieved medical concepts, category assignments, joined tuple pairs, or oracle-aligned semantic SQL outputs. In a more specific recent usage, “Semantic F1” also denotes a soft multi-label metric in which exact label identity is replaced by graded semantic similarity, so that near-miss predictions receive partial credit rather than being treated as complete failures (Al-Theiabat et al., 2020, Ouchi et al., 2018, Chochlakis et al., 25 Sep 2025).

1. Formal foundations and evaluation units

Across the literature, the classical F1 definition remains unchanged. Precision and recall are

$P = \frac{TP}{TP+FP}, \qquad R = \frac{TP}{TP+FN},$

and the F1 score is

$$F_1 = 2 \cdot \frac{P \cdot R}{P + R} = \frac{2TP}{2TP + FP + FN}.$$

This representation makes explicit that F1 depends on true positives, false positives, and false negatives, while ignoring true negatives (Ouchi et al., 2018, Lipton et al., 2014, Hand et al., 2020).

What varies across “semantic F1” applications is not the algebra but the evaluation unit. In Arabic semantic question similarity, the positive class is the label for semantically similar question pairs, and F1 is computed over pairwise duplicate detection (Al-Theiabat et al., 2020). In PropBank-style semantic role labeling, the unit is a labeled argument span for a given predicate, represented as a triple $\langle i,j,r\rangle$, and the CoNLL scorer computes corpus-level micro-F1 over exact span-and-label matches (Ouchi et al., 2018). In picture-naming EEG, by contrast, the task is nine-class semantic-category decoding and the central aggregate is Macro-F1, defined as the unweighted average of per-class F1 scores across the nine categories (Hu et al., 12 Jun 2026). This suggests that “semantic F1” is best understood as a task-dependent family of F1 instantiations whose semantics are determined by the prediction object and the averaging scheme rather than by a single universal protocol.

A recurrent source of confusion is the assumption that semantic F1 is always either binary or macro-averaged. The surveyable cases do not support that assumption. Binary positive-class F1 is used for duplicate-question detection (Al-Theiabat et al., 2020); corpus-level micro-F1 is used for SRL and effectively for TACRED-style relation extraction (Ouchi et al., 2018, Olek, 2023); Macro-F1 is central in EEG semantic-category decoding (Hu et al., 12 Jun 2026); and mean per-centroid F1 is used when ontology-based similarity measures are thresholded to form concept clusters in pharmacovigilance (Painter et al., 26 Mar 2025).

2. Exact-match semantic classification

A clear binary case study is NSURL-2019 Task 8 on semantic question similarity in Arabic. The task asks whether a pair of Arabic questions are semantically similar or not, with label 1 for semantically similar and label 0 for non-similar. The Mawdoo3 AI dataset contains 11,997 labeled training pairs and 3,715 test pairs, with the training split slightly imbalanced at 55% label 0 and 45% label 1. Within that setting, the reported models range from roughly 88% to 96% F1: the Bi-GRU obtains 88.312 private F1, the CNN 88.773, multi-head attention 87.889, a single multilingual BERT 95.617, and an ensemble of multilingual BERT models trained with different random seeds reaches an official private F1 of 95.924, while a five-seed ensemble reaches the highest private F1 of 96.232 (Al-Theiabat et al., 2020). In this case, semantic F1 is simply binary F1 on the duplicate-question class, but its practical meaning is immediately semantic: it quantifies how reliably a system separates paraphrases from non-paraphrases.

Relation extraction provides a complementary example in which the formulas remain standard but evaluation details become decisive. In the TACRED setting analyzed in the discussion of the RECENT system, instances are sentence-level subject–object pairs labeled with one relation or no_relation, and the evaluation is effectively micro-F1 over non-no_relation predictions aggregated across the full test set. Michał Olek’s reevaluation shows how sensitive semantic F1 is to implementation correctness: the recreated original experiment yields precision 89.86, recall 65.62, and F1 75.85, whereas the corrected experiment preserves recall at 65.62 but increases false positives from 246 to 1190, reducing precision to 64.70 and F1 to 65.16 (Olek, 2023). The underlying lesson is not about a new metric variant but about protocol integrity: semantic F1 in information extraction is only as trustworthy as the mapping from system outputs to TP, FP, and FN.

These exact-match settings also show why F1, rather than accuracy, is repeatedly chosen. In duplicate detection, relation extraction, and related semantic classification tasks, the principal costs are spurious positive predictions and missed positive predictions. Because F1 omits true negatives, it focuses the evaluation on that trade-off rather than on majority-class prevalence (Al-Theiabat et al., 2020, Olek, 2023).

3. Structured outputs, boundaries, and explanations

In semantic role labeling, semantic F1 is attached to a richer structural object: the labeled argument span. Ouchi et al. formulate span-based SRL as direct span selection and evaluate with the official CoNLL-2005 scorer on CoNLL-2005 and CoNLL-2012. A true positive requires exact agreement on predicate, start index, end index, and semantic role label; there is no partial credit. Under that protocol, their ELMo-Span ensemble reaches 87.4 F1 on CoNLL-2005 Test ALL and 87.0 F1 on CoNLL-2012 Test (Ouchi et al., 2018). Their boundary-only and label-only analyses further show that overall semantic F1 conflates two distinct subproblems: exact span recovery and correct semantic labeling.

Medical semantic segmentation exposes a different structural mismatch between what F1 can measure and what practitioners may care about. The paper on blob loss argues that volumetric Dice or volumetric F1 can remain high even when many small lesions are missed, because a few large foreground instances dominate voxel-wise overlap. It therefore targets instance-level detection metrics such as F1 score and sensitivity by treating connected components as blobs rather than aggregating all foreground voxels. Relative to soft Dice loss, the reported improvements are 5% for MS lesions, 3% for liver tumor, and an average 2% for microscopy segmentation tasks considering F1 score (Kofler et al., 2022). Here, semantic F1 is still standard F1, but applied to object instances rather than to pixels or voxels.

Dialogue topic segmentation shows that exact boundary F1 can become semantically misleading when annotation granularity is unstable. The granularity-aware evaluation paper argues that strict boundary matching treats small boundary shifts as full errors, penalizes semantically coherent oversegmentation, and conflates boundary detection quality with boundary density. It therefore proposes evaluating dialogue segmentation with window-tolerant F1 together with Boundary Oversegmentation Ratio, purity, and coverage, and argues that topic segmentation is better viewed as selecting an appropriate granularity than as reproducing one uniquely correct boundary set (Coen, 18 Dec 2025). This suggests that in segmentation problems, plain F1 may underestimate semantic adequacy when boundaries are gradual, nested, or sparsely annotated.

Explainable question answering adds an explicitly user-centered critique. On HotpotQA, answer-F1, supporting-facts F1, and Joint-F1 are all defined through lexical overlap with gold annotations, but the paper “F1 is Not Enough!” argues that these scores do not measure answer–explanation coupling. A model can obtain a high answer score even when the explanation omits the actual answer evidence, and a wrong answer paired with an explanation that accurately reveals the model’s reasoning can still receive zero under standard F1-style metrics. The proposed FaRM and LocA scores are designed to quantify coupling more directly, and the accompanying user study reports that these scores are better aligned with user experience than standard F1 (Schuff et al., 2020). The broader implication is that high semantic F1 on output labels does not automatically imply usefulness, faithfulness, or appropriate granularity.

4. Multiclass semantic spaces and clustering

Semantic F1 is not confined to binary decisions. In picture-naming EEG, semantic-category information is operationalized by embedding French picture names with Qwen3-Embedding-0.6B, clustering them with hierarchical agglomerative clustering using Ward’s linkage, and manually merging the resulting clusters into nine interpretable semantic categories. The decoding task is then nine-class classification from channel-wise SingLEM EEG embeddings. The reported Macro-F1 increases from 0.566 in the EARLY window to 0.613 in the NAMING window and reaches 0.784 when EARLY and NAMING are combined; per-class F1 scores in the combined condition range from 0.774 to 0.798 (Hu et al., 12 Jun 2026). In this setting, semantic F1 measures how well neural activity supports recovery of a text-derived semantic category structure rather than a single lexical label.

Ontology-based clustering in pharmacovigilance uses F1 in yet another way. For each MedDRA Preferred Term centroid, ontology-based semantic similarity measures are thresholded to create a cluster of related PTs, and F1 is computed against a reference set defined by expert review, SMQ membership, or HLGT membership. The reported expert-based peak F1 values are approximately 0.404 for INTRINSIC_LCH, 0.403 for INTRINSIC_LIN and SOKAL, 0.36 for WUPALMER, and 0.28 for LCH (Painter et al., 26 Mar 2025). Because the scores are obtained by sweeping thresholds and then averaging F1 across centroid concepts, they evaluate how effectively a semantic similarity function induces clinically meaningful retrieval sets. This suggests that semantic F1 can serve as a bridge between continuous similarity spaces and discrete task judgments whenever a thresholded semantic neighborhood is the object of interest.

Both cases depart from the common intuition that semantic F1 is about exact label recovery alone. In EEG decoding, the labels themselves are data-driven semantic clusters (Hu et al., 12 Jun 2026). In pharmacovigilance, F1 evaluates induced concept sets rather than single classifications (Painter et al., 26 Mar 2025). The common element is not exactness of label names, but the use of precision–recall trade-offs to assess recovery of a target semantic structure.

5. Thresholds, calibration, and interpretability

Theoretical work on threshold selection shows that F1 is not merely descriptive; it also induces distinctive decision rules. For real-valued classifier outputs, the paper “Thresholding Classifiers to Maximize F1 Score” derives the F1-optimal decision rule and shows that, when scores are well-calibrated conditional probabilities, the optimal threshold is half the optimal F1 score. It also proves a striking special case: if a classifier is completely uninformative, expected F1 is maximized by predicting all examples as positive (Lipton et al., 2014). In multilabel settings, this interacts strongly with label prevalence and with the distinction between micro, macro, and per-instance averaging, which helps explain why F1-based threshold tuning can behave counterintuitively for rare labels.

Interpretability concerns motivate an alternative transformation rather than a new operating rule. The paper “F*: An Interpretable Transformation of the F-measure” defines

$$F^* = \frac{F}{2-F} = \frac{TP}{TP+FP+FN} = \frac{TP}{n-TN},$$

and shows that it is a strictly increasing transformation of F1, so rankings are preserved. Because $F^*$ equals the Jaccard coefficient between the predicted-positive set and the true-positive set, it provides a more immediate description of what an F1-like score means: the proportion of relevant-or-predicted-positive items that are correctly identified positives (Hand et al., 2020). The transformation does not change model selection, but it changes the semantics of reporting.

In database settings, thresholding and calibration are coupled to computational cost. SemJoin defines semantic join quality by comparing the predicted set of joined tuple pairs with the ground-truth set under a natural-language predicate, and reports that an LLM advisor routing workloads to either Cluster Join or Classifier strategy outperforms adaptive block join by 20–33 F1 points across IMDb reviews, email contradictions, and Stack Overflow tags, while also improving token efficiency on two of the three datasets (Gou et al., 28 Jun 2026). In semantic SQL, streaming model cascades use proxy models, oracle models, and two-threshold routing policies; across six datasets, both SUPG-IT and GAMCAL achieve F1 greater than 0.95 on every dataset, with GAMCAL delivering higher F1 per oracle call at cost-sensitive operating points and SUPG-IT reaching a higher quality ceiling with formal guarantees on precision and recall (Liskowski et al., 1 Apr 2026). These systems make explicit that semantic F1 can be an optimization target embedded in calibration, routing, and token-budget decisions rather than only a retrospective score.

6. Soft Semantic F1 under fuzzy class boundaries

The most explicit redefinition of the term appears in the recent proposal “Semantic F1 Scores: Fair Evaluation Under Fuzzy Class Boundaries,” which addresses subjective or fuzzy multi-label classification. The paper’s starting point is that hard F1 treats semantically related predictions such as “anger” versus “disgust,” or “sad” versus “depressed,” as total failures, even when annotators disagree systematically and downstream actions would be nearly identical. It therefore introduces a label-similarity matrix

$$S \in [0,1]^{C \times C}, \qquad S_{a,b} = \text{semantic similarity between labels } a \text{ and } b,$$

and the asymmetric coverage operator

$$\text{BestMatch}(A,B,S) = \frac{1}{|A|}\sum_{a \in A}\max_{b \in B} S_{a,b}.$$

For an instance with predicted label set $P_i$ and gold label set $T_i$, semantic precision and semantic recall are

$$\text{Precision}_i^s = \text{BestMatch}(P_i, T_i, S), \qquad \text{Recall}_i^s = \text{BestMatch}(T_i, P_i, S),$$

and pointwise Semantic F1 is

$$F_1 = 2 \cdot \frac{P \cdot R}{P + R} = \frac{2TP}{2TP + FP + FN}.$$0

This two-step construction preserves the usual precision–recall interpretation while granting partial credit to semantically related labels (Chochlakis et al., 25 Sep 2025).

The same paper defines micro and macro variants by replacing hard true positives, false positives, and false negatives with semantic versions. For instance-level micro aggregation,

$$F_1 = 2 \cdot \frac{P \cdot R}{P + R} = \frac{2TP}{2TP + FP + FN}.$$1

which leads to

$$F_1 = 2 \cdot \frac{P \cdot R}{P + R} = \frac{2TP}{2TP + FP + FN}.$$2

A key property is reduction to the conventional metric when $$F_1 = 2 \cdot \frac{P \cdot R}{P + R} = \frac{2TP}{2TP + FP + FN}.$$3 is the identity matrix: exact-match hard F1 is recovered as the special case in which nonidentical labels receive zero similarity. The proposal is therefore backward-compatible rather than discontinuous with prior practice (Chochlakis et al., 25 Sep 2025).

The empirical claim of the soft metric is not merely that it is more forgiving, but that it is more faithful to fuzzy label spaces. The paper reports synthetic evidence that Semantic F1 decreases smoothly as semantic error magnitude increases, remains informative under moderate similarity-matrix misspecification, and distinguishes near-miss predictors from far-miss predictors more effectively than hard F1. On real data, it yields smoother threshold behavior, more stable model rankings, stronger correlation with downstream ROC-AUC in a persuasion task, and better early-stopping behavior than hard sample F1 (Chochlakis et al., 25 Sep 2025). A plausible implication is that exact-match F1 and similarity-aware Semantic F1 should often be reported together: the former for strict label identity, the latter for semantic proximity under disputed or overlapping category boundaries.