---
title: 'SADS: Semantic Alignment for Captioning'
url: https://www.emergentmind.com/topics/semantic-alignment-development-score-sads
type: topic
---

# SADS: Semantic Alignment for Captioning

The Semantic Alignment Development Score (SADS) quantifies the semantic correspondence between images and language, addressing the limitations of n-gram–based evaluation in image captioning. SADS projects both image and caption embeddings into a shared latent space using Canonical Correlation Analysis (CCA) trained on paired vision–language data, and computes their alignment via cosine similarity. Empirical results reveal a high correlation with human judgment and demonstrate sensitivity to both compositional semantics and lexical choices, distinguishing it as a robust diagnostic and comparative tool for vision–language models [1805.00063].

## 1. Mathematical Formulation and Pipeline

Let $I$ denote an image and $w$ a candidate caption. The computational pipeline begins by extracting a visual embedding $x \in \mathbb{R}^{d_i}$ via a deep CNN (typically ResNet-101), and a sentence embedding $y \in \mathbb{R}^{d_s}$ constructed using the Hierarchical Kernel Sentence Embedding (HKSE) atop word2vec.

A CCA model is trained offline over approximately 1 million image–caption pairs, yielding projection matrices $U \in \mathbb{R}^{d_i \times k}$, $V \in \mathbb{R}^{d_s \times k}$, and a diagonal singular-value matrix $\Sigma \in \mathbb{R}^{k \times k}$ optimizing cross-modal correlation. Projected embeddings in the CCA space are:

- $E_i(x) = \Sigma \cdot (U^{\top} x)$
- $E_s(y) = V^{\top} y$

The SADS for an image–caption pair is the cosine similarity between these projections:
$$
\mathrm{SADS}(I, w) = 
\frac{\langle E_i(x), E_s(y) \rangle}{\|E_i(x)\|_2 \cdot \|E_s(y)\|_2}
$$
No thresholding is applied; the unmodified cosine is used directly.

### Computational Steps

1. **Feature extraction:** $x = \mathrm{CNN}(I)$; $y = \mathrm{HKSE}(w)$
2. **Mean-centering:** Subtract training-set mean $\mu_x$ and $\mu_y$
3. **CCA projection:** $E_i(x)$, $E_s(y)$ as above
4. **Cosine similarity:** Compute inner product and norms for SADS

Underlying assumptions include mean-centered, uncorrelated base features and CCA directions satisfying whitened-covariance constraints [1805.00063].

## 2. Empirical Validation and Human Correlation

SADS exhibits substantial empirical alignment with human caption judgments. Human annotators (on Amazon Mechanical Turk) assigned both mean opinion scores (MOS) and best-caption choices. A scatter plot of mean SADS versus MOS produces a linear fit with $R^2 \approx 0.85$ on COCO data, indicating strong predictive value.

| Model                  | Semantic (SADS) | MOS   |
|------------------------|-----------------|-------|
| CE baseline            | 0.189           | 3.22  |
| CIDEr-RL               | 0.186           | 3.30  |
| GAN₁ (SCST, log D)     | 0.195           | 3.40  |
| GAN₂ (SCST, +5·CIDEr)  | 0.194           | 3.44  |

Higher SADS correlates with higher "Turing-test fooling rates," indicating improved alignment between generated captions and reference semantics [1805.00063].

## 3. Model Comparison and Diagnostic Capability

SADS provides a direct comparative metric for diverse captioning systems. On the COCO test set (5,000 images), the highest observed SADS is 0.192 (SCST-based GAN₂), while cross-entropy (CE) and RL-trained models yield 0.184–0.186. On an Out of Context (OOC) benchmark—targeting compositional generalization—CE and RL models yield SADS of ~0.118, whereas GAN-based methods attain up to 0.124.

During model training, SADS increases monotonically for successful adversarial models, outpacing n-gram–based metrics such as CIDEr. Vocabulary coverage also positively correlates with SADS, affirming its role in rewarding semantic and lexical diversity rather than mere frequency overlap.

## 4. Sensitivity and Ablation Analyses

SADS demonstrates acute sensitivity to semantic mismatches. Single-word perturbations in captions (e.g., “tennis player”→“giraffe player”) cause SADS to drop from ~0.25 to 0.05–0.15, indicating realism-dependent alignment. Captions mismatched at random to images yield SADS near or below zero, and the metric distinguishes gender cues and object identities robustly.

Varying the projection dimension $k$ stabilizes a trade-off between expressivity and computational cost. Retention of the singular-value weighting $\Sigma$ is crucial; omitting it alters absolute scale and discriminative strength.

## 5. Computational Complexity and Practical Considerations

The principal cost arises in offline CCA training: with $N \sim 1$M pairs and embedding sizes $d_i \sim 2048$, $d_s \sim 300$, covariance estimation and singular-value decomposition are the dominating factors (complexity $O(\min(d_i^2 d_s, d_s^2 d_i) N)$ for covariances plus $O(d_i^3 + d_s^3)$ for SVD). Inference per image–caption pair requires two matrix–vector multiplies ($O(d_i k + d_s k)$) and one dot product plus two norms ($O(k)$), which is negligible relative to deep model feature extraction.

Larger projection dimensions $k$ yield higher recall and alignment at the cost of increased inference time. The singular-value reweighting component ($\Sigma$) can be toggled to modulate sensitivity and absolute scale.

## 6. Extensions and Generalization Beyond Image Captioning

The SADS architecture generalizes to other multimodal evaluation settings:

- **Video Captioning:** Substitute ResNet with a time-pooled 3D CNN for video feature extraction, then compute SADS using temporally summarized vectors.
- **Visual Question Answering (VQA) and Grounding:** Compute alignment between question–answer embeddings and spatially attended image regions.
- **Cross-modal Retrieval:** Rank target items by descending SADS to evaluate retrieval relevance.
- **Alternative Embeddings:** Replace ResNet-HKSE with other paired encoders (e.g., CLIP–ViT), recomputing the CCA layer for joint alignment.
- **Nonlinear Extensions:** Substitute linear CCA with Deep CCA (DCCA); here, SADS operates on nonlinear projected representations, becoming a “deep cosine” metric [1805.00063].

## 7. Significance and Interpretation

SADS provides a complementary, image-grounded alternative to n-gram–based language evaluation metrics, directly quantifying vision–language alignment in a model-agnostic, content-sensitive manner. Its strong correlation with human ratings, responsiveness to semantic errors, and low computational overhead establish it as a robust metric for both standard and compositional caption evaluation. The metric’s flexible architecture invites further adaptation to a variety of cross-modal learning tasks, including retrieval, question answering, and complex scene understanding [1805.00063].

Source: https://www.emergentmind.com/topics/semantic-alignment-development-score-sads