---
title: Semantic Fidelity Score (SFS) Metrics
url: https://www.emergentmind.com/topics/semantic-fidelity-score-sfs
type: topic
---

# Semantic Fidelity Score (SFS) Metrics

Semantic Fidelity Score (SFS) denotes a family of evaluation constructs that quantify whether an output preserves intended meaning rather than merely reproducing wording, syntax, pixels, or bits. Across recent literature, the underlying question is stable—whether semantics survive a transformation—but the formalization varies substantially by task, supervision regime, and modality. One important terminological caveat is that in the literary-translation framework SAMAS, **SFS** does **not** mean Semantic Fidelity Score; it means **Stylistic Feature Spectrum**, an 81-dimensional wavelet-derived signature used for style control, while semantic fidelity is evaluated separately by machine-translation quality metrics [2602.19840].

## 1. Conceptual scope and semantic invariants

A Semantic Fidelity Score is typically motivated by the inadequacy of exact-form fidelity. In semantic communication, the central contrast is between Shannon-style replication of transmitted symbols and preservation of intended meaning; the target shifts from zero bit error to zero semantic symbol error [2101.12649]. In literary translation, the same distinction appears as a separation between **Fidelity**, **Fluency**, and **Felicity**, where fidelity is “the accurate transfer of the source text’s core meaning,” fluency concerns naturalness in the target language, and felicity concerns style, tone, and literary grace [2602.19840]. In instruction-following evaluation, the analogous claim is that lexical-overlap metrics can underrate semantically correct paraphrases, so semantic similarity rather than surface overlap becomes the relevant object [2401.17072].

Taken together, these works suggest that semantic fidelity is best understood as preservation of content-bearing invariants under admissible variation. The invariant may be a proposition, a set of semantic symbols, a supported subsequence, a topic-transport pattern, a set of labels, a scene graph, or a document’s content structure. What changes across formulations is the representation of meaning and the error model used to detect semantic drift.

A recurring misconception is that semantic fidelity is interchangeable with stylistic, perceptual, or structural fidelity. The available evidence argues otherwise. SAMAS explicitly separates semantic accuracy from style fidelity [2602.19840]; high-level-fidelity work in super-resolution separates semantic preservation from perceptual quality [2512.07037]; and low-level image-processing work distinguishes semantic similarity from conventional fidelity-oriented IQA [2604.25408].

## 2. Canonical mathematical patterns

Despite strong domain variation, recent SFS formulations cluster around a small number of mathematical templates. Some compare semantic representations directly, some measure supported content, and some decompose fidelity into directional error modes such as omission, contradiction, and hallucination.

| Family | Core construction | Representative setting |
|---|---|---|
| Embedding similarity | cosine similarity between semantic embeddings | instruction-tuned LLM evaluation |
| Supported-content projection | score claim against its supported subsequence | summarization, QA faithfulness |
| Bidirectional cross-examination | question-derived Coverage, Conformity, Consistency | text-to-text generation |
| Channel/divergence model | KL divergence between intended and realized semantic transformations | reference-free QCA evaluation |
| Soft set matching | best-match precision/recall over semantic units | fuzzy multi-label classification |
| Structured semantic matching | entity, relation, and structural consistency | vision and document parsing |

The simplest scalar form is direct semantic similarity. SemScore defines
\[
\mathrm{SemScore}(\hat{y}, y) = \cos\bigl(e(\hat{y}), e(y)\bigr)
\]
using sentence embeddings from `all-mpnet-base-v2`, and averages this score over a dataset [2401.17072]. A more structured but still scalar formulation appears in semantic communication, where the paper explicitly proposes a semantic error
\[
err = f(s,\hat{s}), \qquad err \in [0,1],
\]
which naturally induces
\[
\mathrm{SFS} = 1 - f(s,\hat{s}).
\]
In the paper’s closed-vocabulary audio case study, this reduces to semantic-symbol accuracy [2101.12649].

A second pattern measures how much of an output is semantically supported. The Longest Supported Subsequence framework defines
\[
\mathrm{LSS}(r,c)=\arg\max_{s \preceq c} |s| \quad \text{such that } s \text{ is supported by } r,
\]
then evaluates faithfulness by comparing the claim to its supported subsequence rather than directly to the source [2308.12157]. A third pattern treats semantic fidelity as directional set coverage. Semantic F1 defines
\[
\text{BestMatch}(A, B, S) = \frac{1}{|A|} \sum_{a \in A} \max_{b \in B} S_{a,b},
\]
and combines forward and reverse best-match scores with the harmonic mean, thereby generalizing exact-match F1 to fuzzy semantic boundaries [2509.21633].

## 3. Reference-based textual formulations

Reference-based SFS formulations assume an available gold answer, supported rewrite, or gold semantic label set. Their principal strength is strong anchoring; their principal weakness is dependence on the chosen reference.

SemScore is the most direct reference-based textual SFS. It embeds the model output and gold target separately, computes cosine similarity, and uses the average score to rank systems. On a benchmark of 12 instruction-tuned or base LLMs and 252 instructions, it achieved the strongest reported correlation to human model rankings among the compared metrics, with Kendall’s \(\tau = 0.879\) and Pearson’s \(r = 0.970\) [2401.17072]. The same paper emphasizes that this formulation is sentence-level rather than token-level and therefore differs from BERTScore’s alignment mechanism.

The LSS framework refines reference-based scoring by extracting the faithful core of a claim before scoring. Rather than asking whether the full generated text matches the reference context, it isolates the longest subsequence of the claim that is actually supported by the reference and then computes a standard metric between the claim and that subsequence. On its human-annotated dataset, the best traditional reference–claim correlation was \(0.30\), whereas generated-LSS scoring reached \(0.48\) or \(0.49\), and human-written LSS yielded correlations up to \(0.86\) [2308.12157]. This suggests that a good SFS need not compare whole outputs monolithically; it may benefit from first factoring outputs into supported and unsupported content.

Semantic F1 generalizes the same logic to arbitrary semantic label sets. For each instance \(i\), with predicted set \(P_i\) and gold set \(T_i\),
\[
\text{Precision}_i^s = \text{BestMatch}(P_i,T_i,S), \qquad
\text{Recall}_i^s = \text{BestMatch}(T_i,P_i,S),
\]
and
\[
\text{SeF1}_i = \frac{2 \cdot \text{Precision}_i^s \cdot \text{Recall}_i^s}{\text{Precision}_i^s + \text{Recall}_i^s}.
\]
If the similarity matrix is the identity, the metric reduces exactly to ordinary F1 [2509.21633]. This makes Semantic F1 a useful abstraction for SFS design whenever outputs can be converted into sets of semantic units.

These reference-based methods share a common philosophy: lexical or structural mismatch is not automatically semantic failure. At the same time, they remain reference-conditioned. SemScore explicitly requires a gold target [2401.17072], LSS depends on supportedness relative to a source reference [2308.12157], and Semantic F1 depends on a domain-appropriate similarity matrix over gold and predicted labels [2509.21633].

## 4. Reference-free and process-aware formulations

Reference-free SFS formulations attempt to measure semantic fidelity without a gold output. They are especially important in translation, summarization, question answering, and grounded generation where multiple outputs may be valid.

The Cross-Examination Framework formalizes semantic fidelity as answer-equivalence under grounded questioning. Let \(D\) be a source document and \(T\) a generated candidate. CEF generates factual yes/no questions from each text and cross-answers them using the other text. It then computes three directional scores:
\[
\text{Coverage}(D,T) = 100 - \% \big(A^{T}_{Q_D} = \text{IDK}\big),
\]
\[
\text{Conformity}(D,T) = 100 - \% \big(A^{T}_{Q_D} = \text{NO}\big),
\]
\[
\text{Consistency}(D,T) = 100 - \% \big(A^{D}_{Q_T} = \text{IDK}\big).
\]
Coverage measures retained source information, Conformity measures contradiction avoidance, and Consistency measures absence of unsupported additions [2601.19350]. In translation, the reference-free and with-reference variants showed strong correlations for Coverage and Consistency, \(r = 0.845\) and \(r = 0.846\), and human validation showed that CEF mismatching questions aligned with semantic errors far more than with non-semantic errors [2601.19350].

A more explicitly information-theoretic SFS is the paper’s Semantic Faithfulness metric for Question–Context–Answer triplets. It represents context, query, and answer as topic distributions \(p^{(c)}, p^{(q)}, p^{(a)}\), infers two row-stochastic transition matrices \(Q\) and \(A\) satisfying
\[
\mathbf{p}^{(q)} = \mathbf{p}^{(c)} \mathbf{Q}, \qquad
\mathbf{p}^{(a)} = \mathbf{p}^{(c)} \mathbf{A},
\]
and minimizes the conditional KL divergence
\[
D(A \parallel Q) = \sum_{i=1}^{N} p_i^{(c)} \sum_{j=1}^{N} A_{ij}\log\frac{A_{ij}}{Q_{ij}}.
\]
The final score is
\[
\mathcal{F}_S = \frac{1}{1 + D_{\min}}.
\]
This yields a reference-free scalar in \((0,1]\), where higher values indicate greater semantic faithfulness of the answer to the context-conditioned intent of the query [2512.05156].

Semantic communication offers a simpler process-aware reference-free perspective. The paper argues that performance should be measured by bitrate and semantic error rate rather than by bit accuracy, and explicitly states that CTSF targets zero semantic symbol error [2101.12649]. In the audio case study, semantic fidelity is instantiated as
\[
Accuracy = 1 - err = 1 - \frac{1}{N}\sum_{i=0}^{N}{s_i \ne \hat{s}_i},
\]
showing that, in closed semantic vocabularies, SFS can be realized as end-to-end semantic-symbol accuracy rather than text similarity [2101.12649].

A plausible synthesis is that reference-free SFS designs tend to outperform surface metrics when semantic failure is primarily about omissions, contradictions, hallucinations, or misdirected semantic transformations rather than wording differences. Their main cost is evaluator complexity: question generation and answering, latent topic estimation, or semantic-symbol inference becomes part of the metric.

## 5. Multimodal and domain-specific extensions

In vision and document understanding, SFS shifts from sentence similarity to content-preservation under perceptual or structural transformation. The same conceptual question remains—whether the output still depicts, encodes, or preserves the same meaning—but the semantic unit changes.

For super-resolution, high-level fidelity is defined as preservation of the image content that humans care about semantically. The paper constructs a 723-example dataset from KonIQ-10k, asks annotators whether there is a high-level semantic fidelity change between the GT and SR output, and aggregates binary responses into
\[
s_i = \frac{1}{n_i}\sum_{j=1}^{n_i} y_{ij}, \qquad s_i \in [0,1],
\]
where higher means worse fidelity [2512.07037]. Foundation-model-based predictors using cosine similarity between GT and SR embeddings correlate more strongly with these human scores than conventional IQA metrics; after fine-tuning, BLIP achieved SRCC \(0.8417\) and PLCC \(0.8955\), while DINOv2 achieved SRCC \(0.8315\) and PLCC \(0.9030\) [2512.07037]. A plausible SFS interpretation is the fidelity-oriented reparameterization \(1-s_i\).

For low-level image processing more broadly, semantic similarity is formalized using entities and relations. An image is represented as
\[
I=\{e_i\}_{i=1}^{N},
\]
and the paper’s abstract semantic similarity score combines matched entities and matched relations. Its concrete metric, Triplet-based Semantic Similarity Score (T3S), harmonically combines foreground, background, and relation branches:
\[
T3S(I_1, I_2) = \frac{2 \tilde{\epsilon}_{\mathrm{ent}} \tilde{\epsilon}_{\mathrm{R}}}{\tilde{\epsilon}_{\mathrm{ent}} + \tilde{\epsilon}_{\mathrm{R}}}.
\]
On COCO, T3S reached \(0.8671\) overall versus \(0.7034\) for ViTScore and lower values for SeSS, DeepSSIM, and SSIM, while also showing clearer monotonic decline under progressive semantic change [2604.25408].

For generative document parsing, SCORE treats semantic fidelity as interpretation-agnostic preservation of content, table meaning, and hierarchical roles. Its content backbone is adjusted normalized edit distance,
\[
\text{NED}(s, g) = 1 - \min\left(\max\left(\frac{L_{\text{edit}(s, g)}}{\max(|g|, |s|)}, 0\right), 1\right),
\]
augmented by semantically aligned weighted similarity over document elements [2509.19345]. It also distinguishes omissions from hallucinations using
\[
\text{TokensFound}(s, g) = \frac{\sum_{t} \min(\text{freq}_s(t), \text{freq}_g(t))}{\sum_{t} \text{freq}_g(t)}
\]
and
\[
\text{TokensAdded}(s, g) = \frac{\sum_{t} \max(0, \text{freq}_s(t) - \text{freq}_g(t))}{\sum_{t} \text{freq}_s(t)}.
\]
Across 1,114 pages, the paper reports that in 2–5% of pages with ambiguous table structures, traditional metrics penalized systems by 12–25% on average, whereas SCORE recovered equivalence between alternative but valid interpretations [2509.19345].

These multimodal extensions imply that SFS is not tied to text. What matters is the availability of a semantic representation in which benign perceptual or structural variation can be separated from meaning change.

## 6. Limitations, controversies, and design implications

No cited paper establishes a universal SFS formula across tasks. Instead, the literature supports a family resemblance. SemScore makes SFS a reference-based embedding similarity [2401.17072]; CEF makes it a vector of grounded QA diagnostics [2601.19350]; Semantic Faithfulness makes it a KL-derived channel-alignment score [2512.05156]; Semantic F1 makes it a soft best-match harmonic mean [2509.21633]; and recent vision and document-parsing work makes it a structured semantic-consistency measure over entities, relations, tables, or functional hierarchy [2604.25408; 2509.19345].

Several limitations recur. First, semantic fidelity is representation-dependent. SemScore depends on the chosen sentence encoder [2401.17072]; Semantic F1 depends on the quality of the similarity matrix and explicitly warns that associative similarity can be misleading [2509.21633]; the topic-transport formulation depends on topic extraction and clustering choices [2512.05156]; and T3S depends on the reliability of entity extraction, class labeling, and relation extraction [2604.25408]. Second, many automatic metrics remain proxies rather than definitions. High-level-fidelity work in super-resolution treats human semantic-change judgments as the real target and uses model-based scores only as regressors toward that target [2512.07037]. Third, task structure matters. In summarization, omissions may dominate; in translation, contradictions and entity distortions may be more salient; in super-resolution, local semantically dense details such as text or QR codes can dominate human judgment [2512.07037].

A further controversy concerns whether semantic fidelity should be scalar or multidimensional. The evidence increasingly favors decomposition. CEF separates Coverage, Conformity, and Consistency [2601.19350]; SCORE separates content fidelity, hallucinations, tables, and hierarchy [2509.19345]; SAMAS explicitly treats semantic fidelity and style fidelity as distinct dimensions [2602.19840]. This suggests that a single-number SFS is often useful operationally, but a diagnostically adequate evaluation regime usually requires companion axes.

The most defensible general design principle is therefore two-stage. First, choose semantic units appropriate to the task: sentence embeddings, supported subsequences, semantic symbols, labels, propositions, entities, relations, or structural elements. Second, score preservation of those units while remaining invariant to transformations that do not alter meaning. Under that view, SFS is less a single metric than a research program for replacing surface fidelity with meaning-preservation as the primary criterion of correctness.

Source: https://www.emergentmind.com/topics/semantic-fidelity-score-sfs