Semantic Alignment Score (SAS)

• SAS is a family of metrics that quantifies semantic alignment by mapping representations like tokens, regions, and sentences into a shared embedding space.
• It employs techniques such as cosine similarity, sigmoid-MLP, and CCA to evaluate alignment across domains including language, vision, and question answering.
• SAS has practical applications in detecting hallucinations, improving cross-modal matching, and providing robust, human-like evaluation in image captioning and QA.

The Semantic Alignment Score (SAS) is a family of evaluation and interpretability metrics that quantify the degree to which representations in machine learning models align with semantic content across modalities or within knowledge-grounded generation settings. SAS is used in diverse contexts: as a mechanistic probe of knowledge grounding in LLMs, a cross-modal similarity measure in vision-language tasks, an automatic answer similarity metric in question answering, and an image-text alignment diagnostic for image captioning. Despite the different domains, all SAS variants share the central principle of mapping elements—tokens, regions, sentences, or answers—into a shared (often learned) representation space and scoring their alignment under a well-defined mathematical criterion.

1. Formal Definitions across Domains

SAS is instantiated in several distinct but closely related ways, each tailored to the representational structure of the domain and the mechanics of the underlying models.

GraphRAG and LLM Knowledge Grounding (SAS-GR)

In "Detecting Hallucinations in Graph Retrieval-Augmented Generation via Attention Patterns and Semantic Alignment" (Li et al., 9 Dec 2025), SAS is defined to measure how the hidden states $h_t$ of generated answer tokens in a frozen decoder align with embeddings $g_i$ of retrieved subgraph triples from a knowledge graph. The score for answer token $y_t$ is: $$SAS(y_t) = \max_{T_i \in TES(q)} \left( \frac{h_t \cdot g_i}{\|h_t\| \|g_i\|} \right)$$ with sentence-level SAS given by $\frac{1}{T} \sum_{t=1}^T SAS(y_t)$, clipped to $[0,1]$.

Semantic Image Retrieval and Detection (SAS-VL)

"End-to-end Semantic Object Detection with Cross-Modal Alignment" (Ferreira et al., 2023) defines SAS as the output of a cross-modal alignment module. For region proposal $r_j$ and text query $t$: $SAS_j = \sigma \left( h_j W_2 + b_2 \right)$ where $h_j = \text{ReLU}([v_j;u]W_1 + b_1)$, $v_j$ and $u$ are L2-normalized embeddings from the visual and text encoders, and $\sigma$ denotes the sigmoid activation.

Semantic Answer Similarity in QA (SAS-QA)

"Semantic Answer Similarity for Evaluating Question Answering Models" (Risch et al., 2021) proposes SAS as a cross-encoder-based regression model: $$SAS(a, b) = \sigma \left( w^\top \text{Dropout}(h_{\mathrm{CLS}}) + b \right)$$ where $h_{\mathrm{CLS}}$ is the final-layer contextual embedding for the concatenated answer pair $([CLS], a, [SEP], b, [SEP])$.

Image Captioning Alignment Diagnostic (SAS-CAP)

In "Adversarial Semantic Alignment for Improved Image Captions" (Dognin et al., 2018), SAS is a canonical-correlation-analysis (CCA) based cosine similarity: $$SAS(x, y) = \frac{\langle \Sigma U^\top x, V^\top y \rangle} {\| \Sigma U^\top x \|_2 \| V^\top y \|_2}$$ where $x$ and $y$ are sentence and image embeddings projected into the CCA subspace by $U$, $\Sigma$, and $V$.

2. Intuitive and Theoretical Motivation

The unifying thread in all variants of SAS is its focus on semantic fidelity of model outputs—either to external knowledge, user-specified queries, or paired data across modalities.

• In GraphRAG, even when the decoder attends to relevant knowledge, the feedforward layers may fail to encode this information due to inductive biases toward internal parametric knowledge, leading to hallucinations. SAS quantifies this mechanistic grounding of answers in retrieved facts (Li et al., 9 Dec 2025).
• In vision-language settings, SAS directly measures the compatibility between localized visual features and linguistic queries, enforcing that object proposals or captions remain closely tied to the semantics of the text (Ferreira et al., 2023, Dognin et al., 2018).
• For QA evaluation, SAS addresses the failure of n-gram metrics to capture semantic equivalence, providing a similarity score closely correlated with human judgment even when lexical overlap is absent (Risch et al., 2021).
• The theoretical validity of SAS flows from its embedding-based design—by optimizing model parameters (or selecting projection directions) such that vector representations align for semantically similar objects, answers, or regions, it places semantic equivalence on a quantifiable geometric footing.

3. Detailed Methodologies

Each SAS instantiation uses an ordered pipeline to compute alignment:

Domain	Inputs	Representation Extraction	Alignment/Scoring
GraphRAG	Linearized triples, question, answer	Layer-$(L{-}1)$ token/sequence embeddings (LM)	Max. cosine h_t, g_i, averaged
Vision-Lang	Image regions $r_j$, text $t$	CNN, FC vision; Transformer encoder for text	Sigmoid-MLP over [v_j; u]
QA Eval	Answer pair $(a, b)$	Cross-encoder (RoBERTa-large) $h_{\mathrm{CLS}}$	Linear regression (or sigmoid)
Captioning	Caption $x$, image $y$	HKSE (caption); ResNet-101 pool (image)	CCA-cosine in joint subspace

Key Computation Steps

GraphRAG SAS (Li et al., 9 Dec 2025)

1. Prune subgraph; construct TES via path and connectivity scores ($K=20$ triples).
2. Feed prompt through LM; extract $h_t$ at layer $l^*$ for generated tokens.
3. Independently, tokenize and encode each triple in TES; extract average hidden $g_i$ at $l^*$.
4. L2 normalization; compute $SAS(y_t) = \max_i \cos(h_t, g_i)$.
5. Aggregate across answer tokens.

Vision-Lang SAS (Ferreira et al., 2023)

1. Extract object proposals with RPN and encode $r_j$; L2-normalize.
2. Tokenize text; encode with Transformer; L2-normalize.
3. Concatenate feature vectors; project through MLP; apply sigmoid to get $SAS_j$.
4. Train with binary cross-entropy loss on positives/negatives.

QA SAS (Risch et al., 2021)

1. Concatenate ground-truth and predicted answers as cross-encoder input.
2. Feed through transformer; pool $h_{\mathrm{CLS}}$.
3. Predict similarity with linear or sigmoid regression; supervise with MSE loss on $[0,1]$-normalized human similarity ratings.

Captioning SAS (Dognin et al., 2018)

1. Extract high-dimensional image features from ResNet; HKSE for captions.
2. Learn CCA projections $U$, $\Sigma$, $V$ on large (image, caption) corpora.
3. Project features, take cosine similarity in joint subspace.

4. Empirical Findings and Correlation with Human Judgment

The effectiveness of SAS is demonstrated across multiple empirical studies:

• In GraphRAG settings, SAS shows a moderate correlation with ground-truth hallucination labels ($r=0.15$), effect size $d=0.60$, and separates truthful from hallucinated model outputs significantly better than attention-based metrics (Li et al., 9 Dec 2025).
• For vision-language object detection, restricting proposals to $SAS\geq 0.5$ increases precision from 0.93 to 0.95 and recall from 0.86 to 0.97 (Ferreira et al., 2023).
• In QA, cross-encoder SAS achieves Pearson $r=0.56$ on SQuAD when F1=0, where token overlap metrics are uninformative, and up to $r=0.75$ when F1$\neq$0 (Risch et al., 2021). Kendall's $\tau$ values also consistently favor SAS.
• For captioning, SAS correlates strongly ($r\approx 0.8-0.9$ from regression fit) with mean opinion scores collected from human annotators on both standard and out-of-context datasets, providing a reliable proxy for human semantic assessment (Dognin et al., 2018).

5. Comparative Metrics and Diagnostic Role

SAS is complementary to several other interpretability and evaluation metrics:

• Path Reliance Degree (PRD): In GraphRAG, PRD measures attention on shortest-path subgraphs, but fails to capture whether this attended content is integrated into final token states. SAS, by quantifying representational grounding, reveals failures not visible to PRD; the two are weakly correlated and jointly more effective for hallucination detection (Li et al., 9 Dec 2025).
• N-gram Metrics (BLEU, CIDEr, METEOR): BLEU, ROUGE, and CIDEr penalize semantic paraphrase and miss cross-modal alignment; SAS better reflects human perceptions of correctness when paraphrase or multimodal grounding is required (Dognin et al., 2018, Risch et al., 2021).
• Contrastive Losses: While contrastive learning objectives may train such alignment functions, SAS often directly leverages the output of such alignment modules or regression heads.

Metric	Scope	Weakness Addressed	SAS Advantage
PRD	KG attention in LM	Only attention, not representation	Mechanistic grounding
BLEU	String overlap	No semantic or cross-modal	Embedding-based similarity
CIDEr	N-gram count, image text	Insensitive to novel compositions	Sensitive to semantic novelty

6. Implementation Considerations and Limitations

Implementing SAS requires high-fidelity, domain-adapted encoders and, where applicable, well-chosen representation subspaces:

• Layer choice in LMs: For GraphRAG, extraction from layer $L-1$ is empirically optimal for retaining semantic content before the output head (Li et al., 9 Dec 2025).
• Normalization: L2-normalization of all feature vectors is standard to ensure cosine similarity properties.
• Batch inference: For QA, cross-encoder SAS requires O(n) passes for n answer pairs, leading to slower throughput compared to retrieval-style bi-encoders (Risch et al., 2021).
• Subspace learning: For captioning, CCA subspace must be trained on large, diverse corpora to capture broad alignments (Dognin et al., 2018).
• Domain adaptation: SAS trained on general STS data may remain suboptimal for domain-specific contexts, requiring curated training datasets for best alignment to task-specific semantic distinctions (Risch et al., 2021).

7. Applications and Impact on Model Design

By revealing representational (not merely attention-based) alignment failures, SAS has shaped both diagnostic methodologies and proactive model adaptations:

• Prompt engineering for LLMs: By observing SAS trends, practitioners may adjust linearization, emphasize specific triples, or add natural-language summaries to improve end-of-sequence alignment (Li et al., 9 Dec 2025).
• Object proposal filtering in vision tasks: SAS enables precision improvements by judicious non-max suppression and confidence re-weighting (Ferreira et al., 2023).
• Auxiliary loss for fine-tuning: SAS serves as a candidate for auxiliary objectives in supervised fine-tuning, directly regularizing semantic proximity in the embedding space (Li et al., 9 Dec 2025).
• Automated answer evaluation: SAS allows label-efficient, human-like scoring of QA outputs, directly closing gaps left by surface-form overlap metrics (Risch et al., 2021).
• Diagnostic tool for out-of-context reasoning: As in the OOC benchmark, SAS reveals generalization failures or successes in model handling of compositional or rare scenes (Dognin et al., 2018).

A plausible implication is that, as broader adoption of semantic alignment metrics continues, we can expect fine-grained improvements both in mechanistic understanding and practical reliability of retrieval-augmented, cross-modal, and knowledge-grounded generation systems.