---
title: 'Multimodal Alignment Score (MAS): An Overview'
url: https://www.emergentmind.com/topics/multimodal-alignment-score-mas
type: topic
---

# Multimodal Alignment Score (MAS): An Overview

Searching arXiv for the cited papers on multimodal alignment score variants and related formulations.
arxiv_search(query="2402.02055 Variance Alignment Score Multimodal Alignment Score", max_results=5)
search_arxiv(query="2402.02055 Variance Alignment Score", max_results=5)
Multimodal Alignment Score (MAS) denotes a class of metrics for quantifying alignment across modalities rather than a single standardized formula. In recent work, the term is used for per-sample covariance alignment in multimodal contrastive pretraining, externally computed response-quality assessment in visual instruction following, layerwise latent speech–text alignment, probabilistically calibrated image–text evaluation, geometric tri-modal similarity, and cosine-based internal alignment inside frozen large language models [2402.02055], [2508.12458], [2505.19937], [2504.12018], [2509.24734], [2405.16700]. This breadth of usage makes MAS a family resemblance concept: the common objective is to measure whether multimodal representations, outputs, or datasets conform to a desired cross-modal structure, but the scored object, supervision source, and score semantics differ substantially.

## 1. Scope, nomenclature, and score semantics

The literature assigns the label MAS to several technically distinct objects. In one formulation, MAS is any per-sample score measuring alignment to a desired multimodal statistical structure, with the Variance Alignment Score (VAS) as the concrete covariance-based instance [2402.02055]. In M3PO, MAS is an externally computed scalar for a candidate response conditioned on an image and instruction [2508.12458]. In ALAS, “MAS” refers to the Monotonic Alignment Search algorithm, while ALAS is the resulting score; conceptually, ALAS serves as a multimodal alignment score for speech–text models [2505.19937]. In iMatch, the paper does not explicitly name a “Multimodal Alignment Score,” but the global continuous image–text score induced by QAlign naturally plays that role [2504.12018]. In TRIANGLE, the paper likewise does not define MAS by name, but triangle-area similarity is proposed as an “unambiguous measure of their alignment” for three modalities [2509.24734]. In the frozen-LLM study, MAS maps to the paper’s “implicit alignment score,” computed from cosine similarity between perceptual and textual token representations [2405.16700].

A common misconception is that MAS always denotes a high-is-better scalar over image–text pairs. The surveyed formulations do not support that simplification. VAS, M3PO MAS, iMatch, and the implicit alignment score are used as larger-is-better alignment indicators, whereas ALAS is an average absolute deviation from a reference path and TRIANGLE’s area is minimized for better alignment [2402.02055], [2508.12458], [2505.19937], [2504.12018], [2509.24734], [2405.16700].

| Formulation | Scored object | Score interpretation |
|---|---|---|
| VAS | training sample or subset | higher is better |
| M3PO MAS | candidate response \(y\) for \((I,x)\) | higher is better |
| ALAS | layerwise speech–text alignment path error | lower is better |
| iMatch global score | image–text pair | higher is better |
| TRIANGLE area | tri-modal embedding triplet | lower is better |
| Implicit alignment score | internal perceptual/text token states | higher is better |

This suggests that MAS is best understood as a task-specific alignment functional rather than a universally calibrated benchmark.

## 2. Covariance-alignment MAS in multimodal contrastive pretraining

In "Variance Alignment Score: A Simple But Tough-to-Beat Data Selection Method for Multimodal Contrastive Learning" [2402.02055], MAS is instantiated by VAS. The problem setting is multimodal contrastive pretraining, where one wants a selected subset whose multimodal statistics match a target distribution expected at test time. Let fixed teacher encoders produce L2-normalized embeddings \(x_i = \bar f_v(v_i)\) and \(y_i = \bar f_l(l_i)\). The target (cross-)covariance estimated from a proxy test distribution is

\[
\bar{\Sigma}_{\text{test}} = \mathbb{E}_{(v,l)\sim \text{proxy}} \big[\bar f_{m_1}(v)\,\bar f_{m_2}(l)^\top\big].
\]

For the \(i\)-th sample, the outer-product tensor is \(\Sigma_i = x_i y_i^\top\) in the cross-modal case, or \(\Sigma_i = z_i z_i^\top\) in the single-modal case. The per-sample score is

\[
\mathrm{VAS}_i(m_1,m_2)
= \langle \bar{\Sigma}_{\text{test}}, \Sigma_i\rangle
= \operatorname{trace}\!\big(\bar{\Sigma}_{\text{test}}^\top \Sigma_i\big)
= \bar f_{m_1}(v_i)^\top \bar{\Sigma}_{\text{test}} \bar f_{m_2}(l_i).
\]

If features are nonzero mean, the formulation centers them before forming covariances:
\[
\tilde x_i = x_i - \mu_x,\qquad \tilde y_i = y_i - \mu_y,\qquad
\Sigma_i = \tilde x_i \tilde y_i^\top,\qquad
\bar{\Sigma}_{\text{test}} = \mathbb{E}\big[(x-\mu_x)(y-\mu_y)^\top\big].
\]
In practice, CLIP embeddings are L2-normalized and centering is optional, though it can improve robustness under domain shift [2402.02055].

The associated subset-selection problem is additive:
\[
\max_{S:\,|S|=k} \sum_{i\in S} \operatorname{trace}\!\big(\bar{\Sigma}_{\text{test}}^\top \Sigma_i\big).
\]
This admits a practical greedy implementation: compute all \(\mathrm{VAS}_i\), sort, and take top-\(k\). The paper also uses a combined quality-and-alignment score,
\[
\mathrm{score}_i = \alpha \cdot \mathrm{CLIP}_i + \beta \cdot \mathrm{VAS}_i,
\]
with conservative removal of very low-quality pairs before ranking. A dynamic-prior variant, VAS-D, updates \(\Sigma_{\text{test}}\) from the selected subset and greedily removes low-VAS samples [2402.02055].

The theoretical analysis is carried out in a simplified linearized CLIP-like regime. Training on subset \(S\) yields an approximate learned cross-map
\[
\hat G_v \hat G_l^\top \approx \frac{1}{\rho}\, G_v^* \Sigma_S (G_l^*)^\top + \text{noise},
\]
while a simplified test loss is proportional to \(-\operatorname{Tr}(\Sigma_{\text{test}}\Sigma_S)\). The method-of-moments intuition is that maximizing \(\langle \Sigma_{\text{test}}, \Sigma_i\rangle\) selects samples that most increase empirical covariance in target directions, thereby reducing contrastive test loss in the linearized regime [2402.02055].

Empirically, applying VAS and CLIP scores together improves strong baselines by \(1.3\%\) average across 38 evaluation sets on DataComp and by \(2.5\%\) on VTAB for CC12M. VAS alone is better on the high-quality CC12M data, whereas on noisy web data combining VAS with CLIP quality filtering is crucial. The ablations further show that vision-only VAS outperforms text-only and cross-modal VAS, and the paper attributes this to more stable visual embeddings and better signal-to-noise [2402.02055].

## 3. External-quality MAS in visual instruction following

In "M3PO: Multimodal-Model-Guided Preference Optimization for Visual Instruction Following" [2508.12458], MAS is an externally computed scalar that quantifies how well a candidate textual response \(y\) aligns with both an image \(I\) and an instruction/query \(x\). Formally,
\[
\mathrm{MAS}(I,x,y) \in \mathbb{R}.
\]
The paper states that MAS assesses visual relevance, semantic accuracy, and instruction adherence, but does not specify the exact numeric range, normalization, or a closed-form decomposition. It emphasizes instead that MAS is computed by a pre-trained visual-language assessment model distinct from the base LVLM being fine-tuned [2508.12458].

The operational role of MAS is preference-pair construction without human labels. For each \((I,x)\), the base LVLM generates \(N=32\) diverse candidates. Each candidate receives both an external MAS score and an internal confidence score \(c_i = \log P(y_i\mid I,x)\). The preferred response is chosen by pure external alignment:
\[
y_w = \arg\max_{y\in \mathcal{Y}} \mathrm{MAS}(I,x,y).
\]
Hard-negative selection then uses the M3P-Score
\[
S(y_j) =
\big[\mathrm{MAS}(I,x,y_w)-\mathrm{MAS}(I,x,y_j)\big]
-\alpha \cdot \max\!\bigl(0,\log P(y_w\mid I,x)-\log P(y_j\mid I,x)-\delta\bigr),
\]
and the dispreferred response is
\[
y_l = \arg\max_{y_j\in \mathcal{Y}\setminus\{y_w\}} S(y_j).
\]
The first term favors candidates that are clearly worse than \(y_w\) under external alignment, while the second term penalizes negatives only when the model is already much less confident in them. The defaults reported are \(\alpha=0.5\) and \(\delta=0.1\) [2508.12458].

MAS is computed with external assessors such as CLIP ViT-L/14 or BLIP-2, and optionally GPT-4V as a strong evaluator. The paper does not detail prompt templates, rubric scoring, or score aggregation mechanics, but describes MAS as “robust” and “comprehensive.” The computation is performed offline during preference-data generation. For LLaVA-1.5-7B, the reported end-to-end cost is approximately 10 GPU hours for data generation and 10 GPU hours for fine-tuning on A100-class GPUs [2508.12458].

The empirical contribution of MAS is clearest in combination with confidence-based hard-negative mining. On LLaVA-1.5-7B, full M3PO reaches MME 1402.3, POPE 87.35%, IFT 71.80, and Human Preference 3.38, whereas removing the confidence term (\(\alpha=0\)) yields MME 1395.0, POPE 87.00%, IFT 71.20, and Human Preference 3.25. The interpretation given in the paper is that MAS supplies the external alignment signal, while confidence mining turns misaligned but model-plausible responses into informative dispreferred samples [2508.12458].

## 4. Path-based MAS for latent speech–text alignment

In "ALAS: Measuring Latent Speech-Text Alignment For Spoken Language Understanding In Multimodal LLMs" [2505.19937], the score is defined over latent layerwise alignments between audio frames and text tokens. For transformer layer \(l\), the model exposes audio representations \(X_l \in \mathbb{R}^{A\times d}\) and text representations \(Y_l \in \mathbb{R}^{T\times d}\), with \(x_{l,i}\in\mathbb{R}^d\) and \(y_{l,j}\in\mathbb{R}^d\). The paper uses cosine similarity to build a cross-modal matrix
\[
S^l_{i,j} = \frac{\langle x_{l,i}, y_{l,j}\rangle}{\|x_{l,i}\|\,\|y_{l,j}\|},
\]
optionally rescaled to \([0,1]\) by \(\tilde S^l_{i,j}=(S^l_{i,j}+1)/2\) [2505.19937].

A monotonic alignment path is then computed with the Monotonic Alignment Search algorithm:
\[
P^l = \{p^l_1,p^l_2,\ldots,p^l_A\},\qquad p^l_i\in\{1,\ldots,T\},\qquad p^l_i \le p^l_{i+1}.
\]
The reference alignment
\[
G = \{g_1,g_2,\ldots,g_A\}
\]
is derived from Whisper-large-v3 word-level timestamps mapped to token indices. The layerwise score is the average absolute deviation between the latent path and the reference path:
\[
\mathrm{ALAS}^l = \frac{1}{A}\sum_{i=1}^A |p^l_i - g_i|.
\]
For sample \(n\), this becomes \(\mathrm{ALAS}_n^l\), and dataset-level aggregation is
\[
\mathrm{ALAS}_{\text{dataset}}(l) = \frac{1}{N}\sum_{n=1}^N \mathrm{ALAS}_n^l.
\]
The paper mainly reports \(\mathrm{ALAS}_{\text{dataset}}(l)\) as a function of depth rather than a single scalar [2505.19937].

The protocol runs the speech LLM twice for each example: once with audio plus instruction, and once with transcription-only plus instruction. Instruction tokens are trimmed so that only modality-specific segments remain. Qwen2-Audio-7B-Instruct and SALMONN-7B are examined; both use Whisper-v3 as the speech encoder and expose hidden states layer by layer. For Spoken Question Answering, the paper filters examples by Sentence-BERT similarity, discarding pairs with cosine similarity below 0.7 so that audio-only and text-only runs retain comparable task-relevant information [2505.19937].

The empirical behavior is task-dependent. On LibriSQA, late layers develop crisp diagonal similarity bands and \(\mathrm{ALAS}_{\text{dataset}}(l)\) generally decreases with depth for both QWEN and SALMONN, indicating stronger semantic speech–text alignment. On IEMOCAP emotion recognition, ALAS worsens in the last few layers, consistent with the task’s emphasis on prosody rather than lexical semantics. The paper reports SQA response similarity averages of 0.71 for QWEN and 0.76 for SALMONN, and emotion-recognition accuracies of 21.56% and 49.30%, respectively [2505.19937].

A notable terminological point is that in this paper “MAS” names the path-finding algorithm, not the score. Conceptually, however, ALAS is the multimodal alignment score, because it quantitatively measures how far a model’s latent alignment deviates from a trusted reference [2505.19937].

## 5. Human-aligned image–text MAS via iMatch

In "Instruction-augmented Multimodal Alignment for Image-Text and Element Matching" [2504.12018], MAS is naturally instantiated by the global image–text alignment score \(s\in[1,5]\). The model also predicts element-level scores \(S_{e_i}\in[0,1]\) for prompt elements, but the central global quantity is \(S_{\text{total}}\in[1,5]\), “representing the overall matching degree.” During instruction tuning, the global score is discretized to
\[
S'_{\text{total}} = \operatorname{round}\!\left(\frac{S_{\text{total}}-1}{4}\times 14 + 1\right), \qquad S'_{\text{total}}\in\{1,\ldots,15\},
\]
and mapped to alphabet labels \(\{a,\ldots,o\}\). Element scores are similarly discretized to seven ordinal levels [2504.12018].

The key mechanism is QAlign, which converts closed-set logits into a continuous score. Let \(x_{l_i}\) denote the logit for label \(l_i\in\{a,\ldots,o\}\), and define
\[
p_{l_i} = \frac{\exp(x_{l_i})}{\sum_{j=1}^{15}\exp(x_{l_j})},\qquad \sum_i p_{l_i}=1.
\]
The expected 15-level score is
\[
\hat S^{(15)} = \sum_{i=1}^{15} p_{l_i}\cdot i,
\]
which is then linearly rescaled to the dataset’s global range:
\[
s = 1 + 4\cdot \frac{\hat S^{(15)}-1}{14},\qquad s\in[1,5].
\]
This \(s\) is the paper’s continuous image–text alignment score and functions as MAS in the global evaluation setting [2504.12018].

The iMatch pipeline combines several augmentation strategies. Validation-set pseudo-labeling augments training data; element augmentation injects element category labels and confidences into the prompt; image augmentation applies random lighting, random grid distortion, and random crop; and, for the element task, prompt type augmentation and score perturbation further improve accuracy. At test time, predicted element scores can be embedded as pseudo-features into the global prompt, but the paper does not specify a deterministic aggregation rule from element scores to the final global score [2504.12018].

Evaluation is conducted on EvalMuse-40K using SRCC, PLCC, and ACC. On the validation set, iMatch reaches SRCC 0.8304 and PLCC 0.8294 for global alignment, compared with 0.7742 and 0.7722 for FGA-BLIP2. For the element task, ACC reaches 0.8317 with Ovis2-8B, 0.8284 with InternVL2.5-8B-MPO, and 0.7948 with Qwen2.5-VL-7B-Instruct, against 0.7680 for FGA-BLIP2. The method also won first place in the CVPR NTIRE 2025 Text to Image Generation Model Quality Assessment Track 1, with Main 0.8551, SRCC 0.8249, PLCC 0.8485, and ACC 0.8734 [2504.12018].

Relative to embedding-only scores such as CLIPScore, the paper positions this MAS as a human-aligned, ordinally calibrated measure. Because it is produced through closed-set MLLM judgments and QAlign expectation rather than direct cosine similarity, it is intended to be more sensitive to fine-grained entity, attribute, and relation mismatches [2504.12018].

## 6. Geometric and internal-representation views of MAS

Two further formulations push MAS beyond pairwise external scoring. In "A TRIANGLE Enables Multimodal Alignment Beyond Cosine Similarity" [2509.24734], the score is defined directly on three embeddings \(\mathbf{x},\mathbf{y},\mathbf{z}\in\mathbb{R}^n\). Let \(\mathbf{u}=\mathbf{x}-\mathbf{y}\) and \(\mathbf{v}=\mathbf{x}-\mathbf{z}\). The core similarity is the triangle area
\[
A = \frac{1}{2}\sqrt{\langle \mathbf{u},\mathbf{u}\rangle \langle \mathbf{v},\mathbf{v}\rangle - \langle \mathbf{u},\mathbf{v}\rangle^2}.
\]
Smaller \(A\) indicates tighter joint clustering of the three modalities, while larger \(A\) indicates poorer joint alignment. For downstream retrieval, the paper further uses
\[
\mathcal{A} =
\frac{1}{2}\sqrt{\langle \mathbf{u},\mathbf{u}\rangle \langle \mathbf{v},\mathbf{v}\rangle - \langle \mathbf{u},\mathbf{v}\rangle^2}
- \alpha \cos\theta_{\mathbf{xy}},
\]
where \(\theta_{\mathbf{xy}}\) is the angle for the task-critical pair. Inside the contrastive loss, the effective high-is-better score is \(\exp(-A/\tau)\). TRIANGLE improves cosine-based methods by up to 9 points of Recall@1 and achieves, for example, MSR-VTT zero-shot video–text retrieval results of T2V 55.2 and V2T 52.5, with comparable gains on DiDeMo, ActivityNet, VATEX, AudioCaps, and VGGSound 5K [2509.24734].

In "Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputs" [2405.16700], MAS is mapped to the paper’s implicit alignment score inside a frozen LLM. If \(P^{(l)}\) and \(T^{(l)}\) denote the perceptual and textual token subsequences at layer \(l\), with mean vectors \(\hat P^{(l)}\) and \(\hat T^{(l)}\), the default score is
\[
\mathrm{SimAvg}(P^{(l)},T^{(l)}) =
\frac{\hat P^{(l)}\cdot \hat T^{(l)}}{\|\hat P^{(l)}\|\;\|\hat T^{(l)}\|}.
\]
The paper also computes MaxSim, MinSim, AvgSim, and MedSim over token pairs, and complements token-level alignment with weight-level overlap,
\[
\mathrm{IoU}(l) = \frac{|M^{(l)}_{\text{perc}} \cap M^{(l)}_{\text{text}}|}{|M^{(l)}_{\text{perc}} \cup M^{(l)}_{\text{text}}|},
\]
where the masks are derived by Wanda pruning scores. Cross-modal token alignment rises across depth and is highest after self-attention; reported inside-block values are approximately 0.45 after self-attention versus approximately 0.10 in the residual stream for Vicuna-v1.5, and approximately 0.58 versus approximately 0.15 for LLaVA-1.5-4. Weight-level IoU is high and increases with depth, with values around 0.67–0.69 across tasks and modality pairs [2405.16700].

The paper further reports a positive correlation between the implicit alignment score and task performance, and a negative correlation with hallucinations on POPE and COCO object hallucination benchmarks. Because perceptual tokens change slowly across layers and activated subnetworks overlap strongly with textual ones, the study also proposes skipping FFN computations for some visual tokens and compressing the model into a single modality-agnostic subnetwork [2405.16700].

Taken together, these two formulations broaden MAS in opposite directions. TRIANGLE treats alignment as a geometric property of three jointly embedded modalities, while the frozen-LLM study treats alignment as an intrinsic property of internal token dynamics and activated weights. A plausible implication is that recent MAS formulations occupy at least three distinct levels of analysis: dataset/sample selection, output evaluation, and internal representational diagnostics [2509.24734], [2405.16700].

Source: https://www.emergentmind.com/topics/multimodal-alignment-score-mas