SADS: Semantic Alignment for Captioning

• SADS is a metric that quantifies semantic alignment by projecting image and caption embeddings into a shared latent space using Canonical Correlation Analysis.
• It achieves a high correlation (R² ≈ 0.85) with human caption judgments and is sensitive to both compositional semantics and lexical choices.
• The metric is computationally efficient during inference and adaptable to various multimodal tasks, including video captioning and visual question answering.

The Semantic Alignment Development Score (SADS) quantifies the semantic correspondence between images and language, addressing the limitations of n-gram–based evaluation in image captioning. SADS projects both image and caption embeddings into a shared latent space using Canonical Correlation Analysis (CCA) trained on paired vision–language data, and computes their alignment via cosine similarity. Empirical results reveal a high correlation with human judgment and demonstrate sensitivity to both compositional semantics and lexical choices, distinguishing it as a robust diagnostic and comparative tool for vision–LLMs (Dognin et al., 2018).

1. Mathematical Formulation and Pipeline

Let $I$ denote an image and $w$ a candidate caption. The computational pipeline begins by extracting a visual embedding $x \in \mathbb{R}^{d_i}$ via a deep CNN (typically ResNet-101), and a sentence embedding $y \in \mathbb{R}^{d_s}$ constructed using the Hierarchical Kernel Sentence Embedding (HKSE) atop word2vec.

A CCA model is trained offline over approximately 1 million image–caption pairs, yielding projection matrices $U \in \mathbb{R}^{d_i \times k}$, $V \in \mathbb{R}^{d_s \times k}$, and a diagonal singular-value matrix $\Sigma \in \mathbb{R}^{k \times k}$ optimizing cross-modal correlation. Projected embeddings in the CCA space are:

• $E_i(x) = \Sigma \cdot (U^{\top} x)$
• $E_s(y) = V^{\top} y$

The SADS for an image–caption pair is the cosine similarity between these projections:

$$\mathrm{SADS}(I, w) = \frac{\langle E_i(x), E_s(y) \rangle}{\|E_i(x)\|_2 \cdot \|E_s(y)\|_2}$$

No thresholding is applied; the unmodified cosine is used directly.

Computational Steps

1. Feature extraction: $w$0; $w$1
2. Mean-centering: Subtract training-set mean $w$2 and $w$3
3. CCA projection: $w$4, $w$5 as above
4. Cosine similarity: Compute inner product and norms for SADS

Underlying assumptions include mean-centered, uncorrelated base features and CCA directions satisfying whitened-covariance constraints (Dognin et al., 2018).

2. Empirical Validation and Human Correlation

SADS exhibits substantial empirical alignment with human caption judgments. Human annotators (on Amazon Mechanical Turk) assigned both mean opinion scores (MOS) and best-caption choices. A scatter plot of mean SADS versus MOS produces a linear fit with $w$6 on COCO data, indicating strong predictive value.

Model	Semantic (SADS)	MOS
CE baseline	0.189	3.22
CIDEr-RL	0.186	3.30
GAN₁ (SCST, log D)	0.195	3.40
GAN₂ (SCST, +5·CIDEr)	0.194	3.44

Higher SADS correlates with higher "Turing-test fooling rates," indicating improved alignment between generated captions and reference semantics (Dognin et al., 2018).

3. Model Comparison and Diagnostic Capability

SADS provides a direct comparative metric for diverse captioning systems. On the COCO test set (5,000 images), the highest observed SADS is 0.192 (SCST-based GAN₂), while cross-entropy (CE) and RL-trained models yield 0.184–0.186. On an Out of Context (OOC) benchmark—targeting compositional generalization—CE and RL models yield SADS of ~0.118, whereas GAN-based methods attain up to 0.124.

During model training, SADS increases monotonically for successful adversarial models, outpacing n-gram–based metrics such as CIDEr. Vocabulary coverage also positively correlates with SADS, affirming its role in rewarding semantic and lexical diversity rather than mere frequency overlap.

4. Sensitivity and Ablation Analyses

SADS demonstrates acute sensitivity to semantic mismatches. Single-word perturbations in captions (e.g., “tennis player”→“giraffe player”) cause SADS to drop from ~0.25 to 0.05–0.15, indicating realism-dependent alignment. Captions mismatched at random to images yield SADS near or below zero, and the metric distinguishes gender cues and object identities robustly.

Varying the projection dimension $w$7 stabilizes a trade-off between expressivity and computational cost. Retention of the singular-value weighting $w$8 is crucial; omitting it alters absolute scale and discriminative strength.

5. Computational Complexity and Practical Considerations

The principal cost arises in offline CCA training: with $w$9M pairs and embedding sizes $x \in \mathbb{R}^{d_i}$0, $x \in \mathbb{R}^{d_i}$1, covariance estimation and singular-value decomposition are the dominating factors (complexity $x \in \mathbb{R}^{d_i}$2 for covariances plus $x \in \mathbb{R}^{d_i}$3 for SVD). Inference per image–caption pair requires two matrix–vector multiplies ($x \in \mathbb{R}^{d_i}$4) and one dot product plus two norms ($x \in \mathbb{R}^{d_i}$5), which is negligible relative to deep model feature extraction.

Larger projection dimensions $x \in \mathbb{R}^{d_i}$6 yield higher recall and alignment at the cost of increased inference time. The singular-value reweighting component ($x \in \mathbb{R}^{d_i}$7) can be toggled to modulate sensitivity and absolute scale.

6. Extensions and Generalization Beyond Image Captioning

The SADS architecture generalizes to other multimodal evaluation settings:

• Video Captioning: Substitute ResNet with a time-pooled 3D CNN for video feature extraction, then compute SADS using temporally summarized vectors.
• Visual Question Answering (VQA) and Grounding: Compute alignment between question–answer embeddings and spatially attended image regions.
• Cross-modal Retrieval: Rank target items by descending SADS to evaluate retrieval relevance.
• Alternative Embeddings: Replace ResNet-HKSE with other paired encoders (e.g., CLIP–ViT), recomputing the CCA layer for joint alignment.
• Nonlinear Extensions: Substitute linear CCA with Deep CCA (DCCA); here, SADS operates on nonlinear projected representations, becoming a “deep cosine” metric (Dognin et al., 2018).

7. Significance and Interpretation

SADS provides a complementary, image-grounded alternative to n-gram–based language evaluation metrics, directly quantifying vision–language alignment in a model-agnostic, content-sensitive manner. Its strong correlation with human ratings, responsiveness to semantic errors, and low computational overhead establish it as a robust metric for both standard and compositional caption evaluation. The metric’s flexible architecture invites further adaptation to a variety of cross-modal learning tasks, including retrieval, question answering, and complex scene understanding (Dognin et al., 2018).