Semantic Preservation Score (SPS)
- Semantic Preservation Score (SPS) is a metric that measures the degree to which a rewritten text retains the original meaning via cosine similarity of full-text sentence embeddings.
- It distinguishes semantic fidelity from cultural identity by decoupling propositional content preservation from the removal of culturally marked linguistic features.
- SPS is computed per text pair in a simple pipeline and validated through robust correlation with human judgments, making it actionable for evaluating LLM outputs.
Searching arXiv for the cited papers to ground the response. Semantic Preservation Score (SPS) is a metric used to quantify whether a rewritten text preserves the meaning of the original while remaining agnostic to stylistic, cultural, or pragmatic changes. In the study of cultural ghosting in world English varieties, SPS is defined as cosine similarity between sentence embeddings of an original text and a model output , and is used to separate meaning retention from identity retention: a text can preserve propositional content while erasing culturally marked linguistic features (Navneet et al., 25 Feb 2026). The same acronym also appears in other research with different meanings, including a governance-derived semantics for effect-transparent AI workflows, a retrieval metric called Spectrum Projection Score, and a faithful extrapolation for model-version analysis on Hugging Face; accordingly, the term requires domain-specific disambiguation (McCann, 1 May 2026).
1. Definition, scope, and motivation
In "When AI Writes, Whose Voice Remains? Quantifying Cultural Marker Erasure Across World English Varieties in LLMs" (Navneet et al., 25 Feb 2026), SPS quantifies the extent to which an LLM’s rewritten text preserves the meaning of the original, regardless of stylistic, cultural, or pragmatic changes. Its role is to show that “meaning” can be retained even as culturally marked linguistic features are erased. The paper introduces SPS together with Identity Erasure Rate (IER) to disentangle semantic fidelity from cultural retention.
The metric is motivated by the paper’s central phenomenon, termed "Cultural Ghosting," defined as the systematic erasure of linguistic markers unique to non-native English varieties during text processing. SPS is needed because cultural ghosting can occur when LLMs remove culturally specific markers while still preserving the propositional content. This separation allows the study to demonstrate what it calls the Semantic Preservation Paradox: high semantic fidelity can coexist with cultural marker loss (Navneet et al., 25 Feb 2026).
The paper distinguishes SPS from several common evaluation metrics. Versus BLEU and ROUGE, SPS is embedding-based and tolerant to paraphrase, making it more suitable for assessing meaning preservation after style changes. Versus BERTScore, SPS uses sentence-level embeddings and a global cosine similarity to capture overall semantic equivalence of the whole text. Versus generic embedding similarity, the distinction lies not in a different mathematical definition but in its role and validation within the study’s framework.
The paper further states that SPS ranges from $0$ to $1$, where $0$ denotes entirely different meaning and $1$ denotes identical meaning. In the study’s framing, SPS greater than $0.7$ is described as “high semantic similarity,” and values around the overall mean of approximately $0.75$ are treated as typical semantic fidelity in the reported experiments (Navneet et al., 25 Feb 2026).
2. Mathematical formulation and computational pipeline
The paper defines SPS as a cosine similarity between sentence embeddings of the original and the model output. The exact LaTeX reproduced in the source is:
$\text{SPS} = \frac{\mathbf{e}_{\text{orig} \cdot \mathbf{e}_{\text{out} {\|\mathbf{e}_{\text{orig}\|\,\|\mathbf{e}_{\text{out}\|}$
The source notes that this expression is recognizable as the standard cosine similarity between vectors and 0. Conceptually, the score is the cosine between the embedding of the original text 1 and the embedding of the LLM output 2. The inputs are the original text and the rewritten output, and the paper applies a single embedding to each full text string; no sentence-level aggregation is specified (Navneet et al., 25 Feb 2026).
The computational pipeline is specified as follows. For each pair 3, the system obtains a single sentence embedding for 4 and a single sentence embedding for 5, computes cosine similarity between them, and records this value as SPS. The paper does not specify token-level normalization, lowercasing, or sentence segmentation for SPS. It treats “sentence embeddings” as embeddings of the entire text. SPS is computed per pair and then averaged within reporting strata such as model, prompt condition, and all outputs.
The paper cites Feng et al. (2022) "Language-agnostic BERT Sentence Embedding" and reports robustness against an alternate encoder, M-USE, with cross-correlation 6. Cosine similarity is the similarity function. While the exact encoder name is not declared in the body text, the cited setup and robustness check indicate that the metric is implemented with sentence embeddings and evaluated for stability across encoder choice (Navneet et al., 25 Feb 2026).
The study evaluates SPS on outputs generated from five instruction-tuned open-source LLMs—Mistral-7B-Instruct, Apertus-8B-Instruct, DeepHat-7B, MiMo-7B, and Qwen3-8B—under three prompt conditions: Baseline, Neutral, and Preservation. The prompts are, respectively, “Make this text more professional and grammatically correct,” “Improve the clarity and grammar of this text,” and “Improve clarity and grammar while preserving cultural voice and regional expressions.” Decoding parameters are fixed at temperature 7, top-p 8, and seed 9.
A minimal pseudocode form given in the source is:
$0.7$9
This implementation detail is central to the study’s use of SPS: the metric is simple, per-output, and then aggregated by model and prompt for dataset-level analysis (Navneet et al., 25 Feb 2026).
3. Relationship to Identity Erasure Rate and the Semantic Preservation Paradox
SPS is paired with Identity Erasure Rate (IER), which quantifies the proportion of annotated cultural markers erased from the original during rewriting. In prose, IER is the fraction of culturally annotated markers in $0$0 that do not appear in $0$1, computed only when the original has at least one marker. The exact LaTeX reproduced in the source is:
$0$2
The central analytical contribution of the paper is the Semantic Preservation Paradox: across 22,350 outputs, SPS is high on average while IER is non-zero. The reported overall means are IER $0$3 with SD $0$4, and SPS $0$5 with SD $0$6. This indicates that LLMs preserve semantic content while erasing cultural markers (Navneet et al., 25 Feb 2026).
The paper reports a weak statistical relationship between the two metrics. A linear regression gives $0$7 and $0$8, meaning only $0$9 of SPS variance is explained by erasure. The paper states that increasing erasure from $1$0 to $1$1 predicts a negligible SPS decrease from $1$2 to $1$3. This finding supports the claim that high semantic similarity does not guarantee preservation of culturally significant form.
The marker-level breakdown further sharpens the distinction between semantic and identity preservation. Pragmatic markers are reported as $1$4 more vulnerable than lexical markers, with $1$5 versus $1$6 erasure. This suggests that forms of meaning not fully captured by propositional equivalence—such as politeness conventions and other pragmatic features—are especially susceptible to flattening under LLM rewriting, even when SPS remains high (Navneet et al., 25 Feb 2026).
A worked example in the source illustrates the mechanism. The original Indian English sentence “Kindly do the needful and revert back at the earliest.” is rewritten under the Baseline prompt as “Please complete the task and respond promptly.” The paper notes that one should expect a relatively high SPS because the paraphrase preserves meaning, while IER would be greater than zero because markers such as “kindly,” “do the needful,” and “revert back” are removed. This is presented as a direct instance of the Semantic Preservation Paradox.
4. Dataset, experimental design, and empirical results
The study evaluates SPS over a corpus of 1,490 texts. The dataset includes Indian English $1$7, Singaporean English $1$8, Nigerian English $1$9, and an American English baseline $0$0. The cultural marker lexicon contains 108 markers in total: Indian 52, Singaporean 32, and Nigerian 24. The corpus contains 624 total marker instances, broken down into Lexical 260, Pragmatic 198, and Syntactic 166 (Navneet et al., 25 Feb 2026).
The outputs evaluated by SPS total 22,350, corresponding to 1,490 texts processed by 5 models under 3 prompt conditions. For every output, SPS is computed between the original text and the model-produced rewrite under the specific prompt. The resulting scores are then aggregated by model and prompt to produce reported statistics.
The following table summarizes the main SPS results reported in the study.
| Condition | SPS | SD |
|---|---|---|
| Overall | 0.7482 | 0.204 |
| Baseline prompt | 0.7421 | 0.204 |
| Neutral prompt | 0.7498 | 0.198 |
| Preservation prompt | 0.7548 | 0.196 |
For the Baseline prompt, the paper reports the following per-model SPS values, each with $0$1 per model: Mistral-7B at $0$2 (SD $0$3), Apertus-8B at $0$4 (SD $0$5), DeepHat-7B at $0$6 (SD $0$7), MiMo-7B at $0$8 (SD $0$9), and Qwen3-8B at $1$0 (SD $1$1). These figures show substantial model-level variation in semantic preservation.
The paper reports significant model differences overall via one-way ANOVA, $1$2, $1$3, $1$4. Prompt effects are also significant via repeated-measures ANOVA, $1$5, $1$6, $1$7. The Preservation prompt slightly improves SPS while reducing IER. Confidence intervals are not reported (Navneet et al., 25 Feb 2026).
Human validation is an important part of the metric’s empirical support. SPS correlates with human similarity judgments at Pearson $1$8 on $1$9 examples. Robustness against encoder choice is also reported: using M-USE as an alternate encoder yields cross-correlation $0.7$0 with the primary setup. These results support the interpretation of SPS as a practical semantic-fidelity metric for paraphrastic or style-transforming outputs.
5. Interpretation, practical use, and limitations
The paper recommends using SPS to verify that LLM editing did not alter intended meaning, but also emphasizes that SPS should always be reported alongside IER. High SPS does not ensure cultural authenticity; it ensures meaning preservation. The metric therefore functions as one axis in a two-axis evaluation regime: semantic fidelity is tracked by SPS, while cultural marker retention is tracked by IER (Navneet et al., 25 Feb 2026).
The study offers an interpretation guide grounded in its observed distributions. SPS greater than or equal to $0.7$1 is treated as generally high meaning preservation, SPS around the overall mean of approximately $0.7$2 is typical of the reported outputs, and cases with high SPS but IER greater than zero instantiate the Semantic Preservation Paradox. The paper specifically advises investigating pragmatic markers in such cases, because they are the most vulnerable category.
Several best practices are stated. The paper recommends reporting distributions, including mean and SD, of both SPS and IER; stratifying by prompt condition and model to expose trade-offs; and considering preservation-oriented prompts, since the reported Preservation prompt reduces IER by $0.7$3 while maintaining or slightly improving SPS. This makes SPS useful not only as an evaluation metric but also as an intervention-sensitive diagnostic.
The paper also notes known failure modes and boundary conditions. One failure mode is “false preservation,” where surface form may be retained but pragmatic function is flattened. Another is embedding bias: sentence encoders may reflect Western-centric semantics. The paper states that this is mitigated via cross-encoder checks but should be acknowledged. A plausible implication is that SPS is strongest as a measure of global semantic equivalence, but incomplete as a standalone account of sociolinguistic fidelity or interactional meaning.
Reproducibility is partly constrained. The paper does not provide code or versioned libraries, and the encoder is not explicitly named in the body text, though LaBSE is cited and M-USE is used for validation. Reproduction therefore requires a sentence-embedding model, a library to compute cosine similarity, access to the 1,490-text corpus and the same LLMs with the given prompts and decoding parameters, and the cultural marker lexicon for IER. SPS itself does not require the lexicon (Navneet et al., 25 Feb 2026).
6. Acronym ambiguity and related uses in adjacent research
The acronym SPS is not unique. In "Effect-Transparent Governance for AI Workflow Architectures: Semantic Preservation, Expressive Minimality, and Decidability Boundaries" (McCann, 1 May 2026), the paper does not define SPS explicitly; instead, it derives SPS from a formal semantic transparency relation over governed and ungoverned executions. In that setting, programs are Interaction Trees over directive events, governance is imposed through an operator $0.7$4, and semantic transparency means that permitted governed runs are observationally equivalent to ungoverned runs modulo governance-only events. From this, the paper derives an exact-equivalence SPS over permitted traces, with the ideal property that under the formal model $0.7$5. This use of SPS concerns effect-transparent governance, not textual rewriting.
A second distinct use appears in "Spectrum Projection Score: Aligning Retrieved Summaries with Reader Models in Retrieval-Augmented Generation" (Hu et al., 8 Aug 2025). There, SPS stands for Spectrum Projection Score rather than Semantic Preservation Score. The metric is defined over reader hidden states in retrieval-augmented generation, using a max-pooled “boundary” vector from penultimate-layer token states and a principal subspace derived from SVD or PCA of a reader representation matrix. The score is the residual norm $0.7$6, where smaller values indicate stronger alignment between a retrieved summary and the reader’s internal geometry. The paper explicitly states that “Semantic Preservation Score” does not appear as the canonical name in that work.
A third nearby use appears in "An Empirical Framework for Evaluating Semantic Preservation Using Hugging Face" (Jia et al., 8 Dec 2025). That paper does not define an explicit Semantic Preservation Score; instead, it presents a faithful extrapolation aligned with metric stability across commits for learning-enabled software systems. The extrapolated SPS is based on normalized drift of comparable evaluation metrics across adjacent commits, with $0.7$7 and values near $0.7$8 indicating high semantic preservation relative to acceptable change bands. This is a repository-evolution metric rather than a text-pair semantic similarity metric.
These alternate uses show that SPS is a polysemous acronym across current AI research. In the cultural ghosting literature, SPS refers specifically to cosine similarity between sentence embeddings of an original and a rewritten text; in governance, it is a transparency-derived trace metric; in retrieval-augmented generation, it denotes Spectrum Projection Score; and in model evolution analysis, it is a faithful extrapolation over metric drift. This suggests that SPS should always be defined locally when used in technical discourse, especially in cross-disciplinary settings where the acronym alone is insufficiently specific.