Semantic Fidelity Score (SFS) Metrics

• Semantic Fidelity Score (SFS) is a family of evaluation constructs that measure if an output preserves the intended meaning rather than exact wording or style.
• SFS employs various methodologies—including embedding similarity, supported-content projection, and bidirectional cross-examination—to quantify semantic consistency across text and multimodal data.
• Despite its versatile applications, SFS approaches face challenges such as dependency on reference quality, representation choices, and the complexity of decomposing semantic invariants.

Semantic Fidelity Score (SFS) denotes a family of evaluation constructs that quantify whether an output preserves intended meaning rather than merely reproducing wording, syntax, pixels, or bits. Across recent literature, the underlying question is stable—whether semantics survive a transformation—but the formalization varies substantially by task, supervision regime, and modality. One important terminological caveat is that in the literary-translation framework SAMAS, SFS does not mean Semantic Fidelity Score; it means Stylistic Feature Spectrum, an 81-dimensional wavelet-derived signature used for style control, while semantic fidelity is evaluated separately by machine-translation quality metrics (Wu et al., 23 Feb 2026).

1. Conceptual scope and semantic invariants

A Semantic Fidelity Score is typically motivated by the inadequacy of exact-form fidelity. In semantic communication, the central contrast is between Shannon-style replication of transmitted symbols and preservation of intended meaning; the target shifts from zero bit error to zero semantic symbol error (Shi et al., 2021). In literary translation, the same distinction appears as a separation between Fidelity, Fluency, and Felicity, where fidelity is “the accurate transfer of the source text’s core meaning,” fluency concerns naturalness in the target language, and felicity concerns style, tone, and literary grace (Wu et al., 23 Feb 2026). In instruction-following evaluation, the analogous claim is that lexical-overlap metrics can underrate semantically correct paraphrases, so semantic similarity rather than surface overlap becomes the relevant object (Aynetdinov et al., 2024).

Taken together, these works suggest that semantic fidelity is best understood as preservation of content-bearing invariants under admissible variation. The invariant may be a proposition, a set of semantic symbols, a supported subsequence, a topic-transport pattern, a set of labels, a scene graph, or a document’s content structure. What changes across formulations is the representation of meaning and the error model used to detect semantic drift.

A recurring misconception is that semantic fidelity is interchangeable with stylistic, perceptual, or structural fidelity. The available evidence argues otherwise. SAMAS explicitly separates semantic accuracy from style fidelity (Wu et al., 23 Feb 2026); high-level-fidelity work in super-resolution separates semantic preservation from perceptual quality (Rocafort et al., 7 Dec 2025); and low-level image-processing work distinguishes semantic similarity from conventional fidelity-oriented IQA (Wang et al., 28 Apr 2026).

2. Canonical mathematical patterns

Despite strong domain variation, recent SFS formulations cluster around a small number of mathematical templates. Some compare semantic representations directly, some measure supported content, and some decompose fidelity into directional error modes such as omission, contradiction, and hallucination.

Family	Core construction	Representative setting
Embedding similarity	cosine similarity between semantic embeddings	instruction-tuned LLM evaluation
Supported-content projection	score claim against its supported subsequence	summarization, QA faithfulness
Bidirectional cross-examination	question-derived Coverage, Conformity, Consistency	text-to-text generation
Channel/divergence model	KL divergence between intended and realized semantic transformations	reference-free QCA evaluation
Soft set matching	best-match precision/recall over semantic units	fuzzy multi-label classification
Structured semantic matching	entity, relation, and structural consistency	vision and document parsing

The simplest scalar form is direct semantic similarity. SemScore defines

$$\mathrm{SemScore}(\hat{y}, y) = \cos\bigl(e(\hat{y}), e(y)\bigr)$$

using sentence embeddings from all-mpnet-base-v2, and averages this score over a dataset (Aynetdinov et al., 2024). A more structured but still scalar formulation appears in semantic communication, where the paper explicitly proposes a semantic error

$err = f(s,\hat{s}), \qquad err \in [0,1],$

which naturally induces

$\mathrm{SFS} = 1 - f(s,\hat{s}).$

In the paper’s closed-vocabulary audio case study, this reduces to semantic-symbol accuracy (Shi et al., 2021).

A second pattern measures how much of an output is semantically supported. The Longest Supported Subsequence framework defines

$$\mathrm{LSS}(r,c)=\arg\max_{s \preceq c} |s| \quad \text{such that } s \text{ is supported by } r,$$

then evaluates faithfulness by comparing the claim to its supported subsequence rather than directly to the source (Mittal et al., 2023). A third pattern treats semantic fidelity as directional set coverage. Semantic F1 defines

$$\text{BestMatch}(A, B, S) = \frac{1}{|A|} \sum_{a \in A} \max_{b \in B} S_{a,b},$$

and combines forward and reverse best-match scores with the harmonic mean, thereby generalizing exact-match F1 to fuzzy semantic boundaries (Chochlakis et al., 25 Sep 2025).

3. Reference-based textual formulations

Reference-based SFS formulations assume an available gold answer, supported rewrite, or gold semantic label set. Their principal strength is strong anchoring; their principal weakness is dependence on the chosen reference.

SemScore is the most direct reference-based textual SFS. It embeds the model output and gold target separately, computes cosine similarity, and uses the average score to rank systems. On a benchmark of 12 instruction-tuned or base LLMs and 252 instructions, it achieved the strongest reported correlation to human model rankings among the compared metrics, with Kendall’s $\tau = 0.879$ and Pearson’s $r = 0.970$ (Aynetdinov et al., 2024). The same paper emphasizes that this formulation is sentence-level rather than token-level and therefore differs from BERTScore’s alignment mechanism.

The LSS framework refines reference-based scoring by extracting the faithful core of a claim before scoring. Rather than asking whether the full generated text matches the reference context, it isolates the longest subsequence of the claim that is actually supported by the reference and then computes a standard metric between the claim and that subsequence. On its human-annotated dataset, the best traditional reference–claim correlation was $0.30$, whereas generated-LSS scoring reached $0.48$ or $0.49$, and human-written LSS yielded correlations up to $err = f(s,\hat{s}), \qquad err \in [0,1],$0 (Mittal et al., 2023). This suggests that a good SFS need not compare whole outputs monolithically; it may benefit from first factoring outputs into supported and unsupported content.

Semantic F1 generalizes the same logic to arbitrary semantic label sets. For each instance $err = f(s,\hat{s}), \qquad err \in [0,1],$1, with predicted set $err = f(s,\hat{s}), \qquad err \in [0,1],$2 and gold set $err = f(s,\hat{s}), \qquad err \in [0,1],$3,

$err = f(s,\hat{s}), \qquad err \in [0,1],$4

and

$err = f(s,\hat{s}), \qquad err \in [0,1],$5

If the similarity matrix is the identity, the metric reduces exactly to ordinary F1 (Chochlakis et al., 25 Sep 2025). This makes Semantic F1 a useful abstraction for SFS design whenever outputs can be converted into sets of semantic units.

These reference-based methods share a common philosophy: lexical or structural mismatch is not automatically semantic failure. At the same time, they remain reference-conditioned. SemScore explicitly requires a gold target (Aynetdinov et al., 2024), LSS depends on supportedness relative to a source reference (Mittal et al., 2023), and Semantic F1 depends on a domain-appropriate similarity matrix over gold and predicted labels (Chochlakis et al., 25 Sep 2025).

4. Reference-free and process-aware formulations

Reference-free SFS formulations attempt to measure semantic fidelity without a gold output. They are especially important in translation, summarization, question answering, and grounded generation where multiple outputs may be valid.

The Cross-Examination Framework formalizes semantic fidelity as answer-equivalence under grounded questioning. Let $err = f(s,\hat{s}), \qquad err \in [0,1],$6 be a source document and $err = f(s,\hat{s}), \qquad err \in [0,1],$7 a generated candidate. CEF generates factual yes/no questions from each text and cross-answers them using the other text. It then computes three directional scores: $err = f(s,\hat{s}), \qquad err \in [0,1],$8

$err = f(s,\hat{s}), \qquad err \in [0,1],$9

$\mathrm{SFS} = 1 - f(s,\hat{s}).$0

Coverage measures retained source information, Conformity measures contradiction avoidance, and Consistency measures absence of unsupported additions (Raha et al., 27 Jan 2026). In translation, the reference-free and with-reference variants showed strong correlations for Coverage and Consistency, $\mathrm{SFS} = 1 - f(s,\hat{s}).$1 and $\mathrm{SFS} = 1 - f(s,\hat{s}).$2, and human validation showed that CEF mismatching questions aligned with semantic errors far more than with non-semantic errors (Raha et al., 27 Jan 2026).

A more explicitly information-theoretic SFS is the paper’s Semantic Faithfulness metric for Question–Context–Answer triplets. It represents context, query, and answer as topic distributions $\mathrm{SFS} = 1 - f(s,\hat{s}).$3, infers two row-stochastic transition matrices $\mathrm{SFS} = 1 - f(s,\hat{s}).$4 and $\mathrm{SFS} = 1 - f(s,\hat{s}).$5 satisfying

$\mathrm{SFS} = 1 - f(s,\hat{s}).$6

and minimizes the conditional KL divergence

$\mathrm{SFS} = 1 - f(s,\hat{s}).$7

The final score is

$\mathrm{SFS} = 1 - f(s,\hat{s}).$8

This yields a reference-free scalar in $\mathrm{SFS} = 1 - f(s,\hat{s}).$9, where higher values indicate greater semantic faithfulness of the answer to the context-conditioned intent of the query (Halperin, 4 Dec 2025).

Semantic communication offers a simpler process-aware reference-free perspective. The paper argues that performance should be measured by bitrate and semantic error rate rather than by bit accuracy, and explicitly states that CTSF targets zero semantic symbol error (Shi et al., 2021). In the audio case study, semantic fidelity is instantiated as

$$\mathrm{LSS}(r,c)=\arg\max_{s \preceq c} |s| \quad \text{such that } s \text{ is supported by } r,$$0

showing that, in closed semantic vocabularies, SFS can be realized as end-to-end semantic-symbol accuracy rather than text similarity (Shi et al., 2021).

A plausible synthesis is that reference-free SFS designs tend to outperform surface metrics when semantic failure is primarily about omissions, contradictions, hallucinations, or misdirected semantic transformations rather than wording differences. Their main cost is evaluator complexity: question generation and answering, latent topic estimation, or semantic-symbol inference becomes part of the metric.

5. Multimodal and domain-specific extensions

In vision and document understanding, SFS shifts from sentence similarity to content-preservation under perceptual or structural transformation. The same conceptual question remains—whether the output still depicts, encodes, or preserves the same meaning—but the semantic unit changes.

For super-resolution, high-level fidelity is defined as preservation of the image content that humans care about semantically. The paper constructs a 723-example dataset from KonIQ-10k, asks annotators whether there is a high-level semantic fidelity change between the GT and SR output, and aggregates binary responses into

$$\mathrm{LSS}(r,c)=\arg\max_{s \preceq c} |s| \quad \text{such that } s \text{ is supported by } r,$$1

where higher means worse fidelity (Rocafort et al., 7 Dec 2025). Foundation-model-based predictors using cosine similarity between GT and SR embeddings correlate more strongly with these human scores than conventional IQA metrics; after fine-tuning, BLIP achieved SRCC $$\mathrm{LSS}(r,c)=\arg\max_{s \preceq c} |s| \quad \text{such that } s \text{ is supported by } r,$$2 and PLCC $$\mathrm{LSS}(r,c)=\arg\max_{s \preceq c} |s| \quad \text{such that } s \text{ is supported by } r,$$3, while DINOv2 achieved SRCC $$\mathrm{LSS}(r,c)=\arg\max_{s \preceq c} |s| \quad \text{such that } s \text{ is supported by } r,$$4 and PLCC $$\mathrm{LSS}(r,c)=\arg\max_{s \preceq c} |s| \quad \text{such that } s \text{ is supported by } r,$$5 (Rocafort et al., 7 Dec 2025). A plausible SFS interpretation is the fidelity-oriented reparameterization $$\mathrm{LSS}(r,c)=\arg\max_{s \preceq c} |s| \quad \text{such that } s \text{ is supported by } r,$$6.

For low-level image processing more broadly, semantic similarity is formalized using entities and relations. An image is represented as

$$\mathrm{LSS}(r,c)=\arg\max_{s \preceq c} |s| \quad \text{such that } s \text{ is supported by } r,$$7

and the paper’s abstract semantic similarity score combines matched entities and matched relations. Its concrete metric, Triplet-based Semantic Similarity Score (T3S), harmonically combines foreground, background, and relation branches: $$\mathrm{LSS}(r,c)=\arg\max_{s \preceq c} |s| \quad \text{such that } s \text{ is supported by } r,$$8 On COCO, T3S reached $$\mathrm{LSS}(r,c)=\arg\max_{s \preceq c} |s| \quad \text{such that } s \text{ is supported by } r,$$9 overall versus $$\text{BestMatch}(A, B, S) = \frac{1}{|A|} \sum_{a \in A} \max_{b \in B} S_{a,b},$$0 for ViTScore and lower values for SeSS, DeepSSIM, and SSIM, while also showing clearer monotonic decline under progressive semantic change (Wang et al., 28 Apr 2026).

For generative document parsing, SCORE treats semantic fidelity as interpretation-agnostic preservation of content, table meaning, and hierarchical roles. Its content backbone is adjusted normalized edit distance,

$$\text{BestMatch}(A, B, S) = \frac{1}{|A|} \sum_{a \in A} \max_{b \in B} S_{a,b},$$1

augmented by semantically aligned weighted similarity over document elements (Li et al., 16 Sep 2025). It also distinguishes omissions from hallucinations using

$$\text{BestMatch}(A, B, S) = \frac{1}{|A|} \sum_{a \in A} \max_{b \in B} S_{a,b},$$2

and

$$\text{BestMatch}(A, B, S) = \frac{1}{|A|} \sum_{a \in A} \max_{b \in B} S_{a,b},$$3

Across 1,114 pages, the paper reports that in 2–5% of pages with ambiguous table structures, traditional metrics penalized systems by 12–25% on average, whereas SCORE recovered equivalence between alternative but valid interpretations (Li et al., 16 Sep 2025).

These multimodal extensions imply that SFS is not tied to text. What matters is the availability of a semantic representation in which benign perceptual or structural variation can be separated from meaning change.

6. Limitations, controversies, and design implications

No cited paper establishes a universal SFS formula across tasks. Instead, the literature supports a family resemblance. SemScore makes SFS a reference-based embedding similarity (Aynetdinov et al., 2024); CEF makes it a vector of grounded QA diagnostics (Raha et al., 27 Jan 2026); Semantic Faithfulness makes it a KL-derived channel-alignment score (Halperin, 4 Dec 2025); Semantic F1 makes it a soft best-match harmonic mean (Chochlakis et al., 25 Sep 2025); and recent vision and document-parsing work makes it a structured semantic-consistency measure over entities, relations, tables, or functional hierarchy (Wang et al., 28 Apr 2026, Li et al., 16 Sep 2025).

Several limitations recur. First, semantic fidelity is representation-dependent. SemScore depends on the chosen sentence encoder (Aynetdinov et al., 2024); Semantic F1 depends on the quality of the similarity matrix and explicitly warns that associative similarity can be misleading (Chochlakis et al., 25 Sep 2025); the topic-transport formulation depends on topic extraction and clustering choices (Halperin, 4 Dec 2025); and T3S depends on the reliability of entity extraction, class labeling, and relation extraction (Wang et al., 28 Apr 2026). Second, many automatic metrics remain proxies rather than definitions. High-level-fidelity work in super-resolution treats human semantic-change judgments as the real target and uses model-based scores only as regressors toward that target (Rocafort et al., 7 Dec 2025). Third, task structure matters. In summarization, omissions may dominate; in translation, contradictions and entity distortions may be more salient; in super-resolution, local semantically dense details such as text or QR codes can dominate human judgment (Rocafort et al., 7 Dec 2025).

A further controversy concerns whether semantic fidelity should be scalar or multidimensional. The evidence increasingly favors decomposition. CEF separates Coverage, Conformity, and Consistency (Raha et al., 27 Jan 2026); SCORE separates content fidelity, hallucinations, tables, and hierarchy (Li et al., 16 Sep 2025); SAMAS explicitly treats semantic fidelity and style fidelity as distinct dimensions (Wu et al., 23 Feb 2026). This suggests that a single-number SFS is often useful operationally, but a diagnostically adequate evaluation regime usually requires companion axes.

The most defensible general design principle is therefore two-stage. First, choose semantic units appropriate to the task: sentence embeddings, supported subsequences, semantic symbols, labels, propositions, entities, relations, or structural elements. Second, score preservation of those units while remaining invariant to transformations that do not alter meaning. Under that view, SFS is less a single metric than a research program for replacing surface fidelity with meaning-preservation as the primary criterion of correctness.