Multimodal Alignment Score (MAS): An Overview

• Multimodal Alignment Score (MAS) is a family of metrics that quantifies alignment between modalities, with varied formulations for different tasks.
• MAS formulations include methods for contrastive pretraining, visual instruction following, speech–text alignment, and geometric similarity analysis.
• Empirical findings indicate that combining MAS with quality scores enhances model selection, fine-tuning, and internal diagnostic evaluations in multimodal systems.

Searching arXiv for the cited papers on multimodal alignment score variants and related formulations. arxiv_search(query="(Wang et al., 2024) Variance Alignment Score Multimodal Alignment Score", max_results=5) search_arxiv(query="(Wang et al., 2024) Variance Alignment Score", max_results=5) Multimodal Alignment Score (MAS) denotes a class of metrics for quantifying alignment across modalities rather than a single standardized formula. In recent work, the term is used for per-sample covariance alignment in multimodal contrastive pretraining, externally computed response-quality assessment in visual instruction following, layerwise latent speech–text alignment, probabilistically calibrated image–text evaluation, geometric tri-modal similarity, and cosine-based internal alignment inside frozen LLMs (Wang et al., 2024, Gao et al., 17 Aug 2025, Mousavi et al., 26 May 2025, Yue et al., 16 Apr 2025, Cicchetti et al., 29 Sep 2025, Shukor et al., 2024). This breadth of usage makes MAS a family resemblance concept: the common objective is to measure whether multimodal representations, outputs, or datasets conform to a desired cross-modal structure, but the scored object, supervision source, and score semantics differ substantially.

1. Scope, nomenclature, and score semantics

The literature assigns the label MAS to several technically distinct objects. In one formulation, MAS is any per-sample score measuring alignment to a desired multimodal statistical structure, with the Variance Alignment Score (VAS) as the concrete covariance-based instance (Wang et al., 2024). In M3PO, MAS is an externally computed scalar for a candidate response conditioned on an image and instruction (Gao et al., 17 Aug 2025). In ALAS, “MAS” refers to the Monotonic Alignment Search algorithm, while ALAS is the resulting score; conceptually, ALAS serves as a multimodal alignment score for speech–text models (Mousavi et al., 26 May 2025). In iMatch, the paper does not explicitly name a “Multimodal Alignment Score,” but the global continuous image–text score induced by QAlign naturally plays that role (Yue et al., 16 Apr 2025). In TRIANGLE, the paper likewise does not define MAS by name, but triangle-area similarity is proposed as an “unambiguous measure of their alignment” for three modalities (Cicchetti et al., 29 Sep 2025). In the frozen-LLM study, MAS maps to the paper’s “implicit alignment score,” computed from cosine similarity between perceptual and textual token representations (Shukor et al., 2024).

A common misconception is that MAS always denotes a high-is-better scalar over image–text pairs. The surveyed formulations do not support that simplification. VAS, M3PO MAS, iMatch, and the implicit alignment score are used as larger-is-better alignment indicators, whereas ALAS is an average absolute deviation from a reference path and TRIANGLE’s area is minimized for better alignment (Wang et al., 2024, Gao et al., 17 Aug 2025, Mousavi et al., 26 May 2025, Yue et al., 16 Apr 2025, Cicchetti et al., 29 Sep 2025, Shukor et al., 2024).

Formulation	Scored object	Score interpretation
VAS	training sample or subset	higher is better
M3PO MAS	candidate response $y$ for $(I,x)$	higher is better
ALAS	layerwise speech–text alignment path error	lower is better
iMatch global score	image–text pair	higher is better
TRIANGLE area	tri-modal embedding triplet	lower is better
Implicit alignment score	internal perceptual/text token states	higher is better

This suggests that MAS is best understood as a task-specific alignment functional rather than a universally calibrated benchmark.

2. Covariance-alignment MAS in multimodal contrastive pretraining

In "Variance Alignment Score: A Simple But Tough-to-Beat Data Selection Method for Multimodal Contrastive Learning" (Wang et al., 2024), MAS is instantiated by VAS. The problem setting is multimodal contrastive pretraining, where one wants a selected subset whose multimodal statistics match a target distribution expected at test time. Let fixed teacher encoders produce L2-normalized embeddings $x_i = \bar f_v(v_i)$ and $y_i = \bar f_l(l_i)$. The target (cross-)covariance estimated from a proxy test distribution is

$$\bar{\Sigma}_{\text{test}} = \mathbb{E}_{(v,l)\sim \text{proxy}} \big[\bar f_{m_1}(v)\,\bar f_{m_2}(l)^\top\big].$$

For the $i$-th sample, the outer-product tensor is $\Sigma_i = x_i y_i^\top$ in the cross-modal case, or $\Sigma_i = z_i z_i^\top$ in the single-modal case. The per-sample score is

$$\mathrm{VAS}_i(m_1,m_2) = \langle \bar{\Sigma}_{\text{test}}, \Sigma_i\rangle = \operatorname{trace}\!\big(\bar{\Sigma}_{\text{test}}^\top \Sigma_i\big) = \bar f_{m_1}(v_i)^\top \bar{\Sigma}_{\text{test}} \bar f_{m_2}(l_i).$$

If features are nonzero mean, the formulation centers them before forming covariances: $$\tilde x_i = x_i - \mu_x,\qquad \tilde y_i = y_i - \mu_y,\qquad \Sigma_i = \tilde x_i \tilde y_i^\top,\qquad \bar{\Sigma}_{\text{test}} = \mathbb{E}\big[(x-\mu_x)(y-\mu_y)^\top\big].$$ In practice, CLIP embeddings are L2-normalized and centering is optional, though it can improve robustness under domain shift (Wang et al., 2024).

The associated subset-selection problem is additive: $(I,x)$0 This admits a practical greedy implementation: compute all $(I,x)$1, sort, and take top-$(I,x)$2. The paper also uses a combined quality-and-alignment score,

$(I,x)$3

with conservative removal of very low-quality pairs before ranking. A dynamic-prior variant, VAS-D, updates $(I,x)$4 from the selected subset and greedily removes low-VAS samples (Wang et al., 2024).

The theoretical analysis is carried out in a simplified linearized CLIP-like regime. Training on subset $(I,x)$5 yields an approximate learned cross-map

$(I,x)$6

while a simplified test loss is proportional to $(I,x)$7. The method-of-moments intuition is that maximizing $(I,x)$8 selects samples that most increase empirical covariance in target directions, thereby reducing contrastive test loss in the linearized regime (Wang et al., 2024).

Empirically, applying VAS and CLIP scores together improves strong baselines by $(I,x)$9 average across 38 evaluation sets on DataComp and by $x_i = \bar f_v(v_i)$0 on VTAB for CC12M. VAS alone is better on the high-quality CC12M data, whereas on noisy web data combining VAS with CLIP quality filtering is crucial. The ablations further show that vision-only VAS outperforms text-only and cross-modal VAS, and the paper attributes this to more stable visual embeddings and better signal-to-noise (Wang et al., 2024).

3. External-quality MAS in visual instruction following

In "M3PO: Multimodal-Model-Guided Preference Optimization for Visual Instruction Following" (Gao et al., 17 Aug 2025), MAS is an externally computed scalar that quantifies how well a candidate textual response $x_i = \bar f_v(v_i)$1 aligns with both an image $x_i = \bar f_v(v_i)$2 and an instruction/query $x_i = \bar f_v(v_i)$3. Formally,

$x_i = \bar f_v(v_i)$4

The paper states that MAS assesses visual relevance, semantic accuracy, and instruction adherence, but does not specify the exact numeric range, normalization, or a closed-form decomposition. It emphasizes instead that MAS is computed by a pre-trained visual-language assessment model distinct from the base LVLM being fine-tuned (Gao et al., 17 Aug 2025).

The operational role of MAS is preference-pair construction without human labels. For each $x_i = \bar f_v(v_i)$5, the base LVLM generates $x_i = \bar f_v(v_i)$6 diverse candidates. Each candidate receives both an external MAS score and an internal confidence score $x_i = \bar f_v(v_i)$7. The preferred response is chosen by pure external alignment: $x_i = \bar f_v(v_i)$8 Hard-negative selection then uses the M3P-Score

$x_i = \bar f_v(v_i)$9

and the dispreferred response is

$y_i = \bar f_l(l_i)$0

The first term favors candidates that are clearly worse than $y_i = \bar f_l(l_i)$1 under external alignment, while the second term penalizes negatives only when the model is already much less confident in them. The defaults reported are $y_i = \bar f_l(l_i)$2 and $y_i = \bar f_l(l_i)$3 (Gao et al., 17 Aug 2025).

MAS is computed with external assessors such as CLIP ViT-L/14 or BLIP-2, and optionally GPT-4V as a strong evaluator. The paper does not detail prompt templates, rubric scoring, or score aggregation mechanics, but describes MAS as “robust” and “comprehensive.” The computation is performed offline during preference-data generation. For LLaVA-1.5-7B, the reported end-to-end cost is approximately 10 GPU hours for data generation and 10 GPU hours for fine-tuning on A100-class GPUs (Gao et al., 17 Aug 2025).

The empirical contribution of MAS is clearest in combination with confidence-based hard-negative mining. On LLaVA-1.5-7B, full M3PO reaches MME 1402.3, POPE 87.35%, IFT 71.80, and Human Preference 3.38, whereas removing the confidence term ($y_i = \bar f_l(l_i)$4) yields MME 1395.0, POPE 87.00%, IFT 71.20, and Human Preference 3.25. The interpretation given in the paper is that MAS supplies the external alignment signal, while confidence mining turns misaligned but model-plausible responses into informative dispreferred samples (Gao et al., 17 Aug 2025).

4. Path-based MAS for latent speech–text alignment

In "ALAS: Measuring Latent Speech-Text Alignment For Spoken Language Understanding In Multimodal LLMs" (Mousavi et al., 26 May 2025), the score is defined over latent layerwise alignments between audio frames and text tokens. For transformer layer $y_i = \bar f_l(l_i)$5, the model exposes audio representations $y_i = \bar f_l(l_i)$6 and text representations $y_i = \bar f_l(l_i)$7, with $y_i = \bar f_l(l_i)$8 and $y_i = \bar f_l(l_i)$9. The paper uses cosine similarity to build a cross-modal matrix

$$\bar{\Sigma}_{\text{test}} = \mathbb{E}_{(v,l)\sim \text{proxy}} \big[\bar f_{m_1}(v)\,\bar f_{m_2}(l)^\top\big].$$0

optionally rescaled to $$\bar{\Sigma}_{\text{test}} = \mathbb{E}_{(v,l)\sim \text{proxy}} \big[\bar f_{m_1}(v)\,\bar f_{m_2}(l)^\top\big].$$1 by $$\bar{\Sigma}_{\text{test}} = \mathbb{E}_{(v,l)\sim \text{proxy}} \big[\bar f_{m_1}(v)\,\bar f_{m_2}(l)^\top\big].$$2 (Mousavi et al., 26 May 2025).

A monotonic alignment path is then computed with the Monotonic Alignment Search algorithm: $$\bar{\Sigma}_{\text{test}} = \mathbb{E}_{(v,l)\sim \text{proxy}} \big[\bar f_{m_1}(v)\,\bar f_{m_2}(l)^\top\big].$$3 The reference alignment

$$\bar{\Sigma}_{\text{test}} = \mathbb{E}_{(v,l)\sim \text{proxy}} \big[\bar f_{m_1}(v)\,\bar f_{m_2}(l)^\top\big].$$4

is derived from Whisper-large-v3 word-level timestamps mapped to token indices. The layerwise score is the average absolute deviation between the latent path and the reference path: $$\bar{\Sigma}_{\text{test}} = \mathbb{E}_{(v,l)\sim \text{proxy}} \big[\bar f_{m_1}(v)\,\bar f_{m_2}(l)^\top\big].$$5 For sample $$\bar{\Sigma}_{\text{test}} = \mathbb{E}_{(v,l)\sim \text{proxy}} \big[\bar f_{m_1}(v)\,\bar f_{m_2}(l)^\top\big].$$6, this becomes $$\bar{\Sigma}_{\text{test}} = \mathbb{E}_{(v,l)\sim \text{proxy}} \big[\bar f_{m_1}(v)\,\bar f_{m_2}(l)^\top\big].$$7, and dataset-level aggregation is

$$\bar{\Sigma}_{\text{test}} = \mathbb{E}_{(v,l)\sim \text{proxy}} \big[\bar f_{m_1}(v)\,\bar f_{m_2}(l)^\top\big].$$8

The paper mainly reports $$\bar{\Sigma}_{\text{test}} = \mathbb{E}_{(v,l)\sim \text{proxy}} \big[\bar f_{m_1}(v)\,\bar f_{m_2}(l)^\top\big].$$9 as a function of depth rather than a single scalar (Mousavi et al., 26 May 2025).

The protocol runs the speech LLM twice for each example: once with audio plus instruction, and once with transcription-only plus instruction. Instruction tokens are trimmed so that only modality-specific segments remain. Qwen2-Audio-7B-Instruct and SALMONN-7B are examined; both use Whisper-v3 as the speech encoder and expose hidden states layer by layer. For Spoken Question Answering, the paper filters examples by Sentence-BERT similarity, discarding pairs with cosine similarity below 0.7 so that audio-only and text-only runs retain comparable task-relevant information (Mousavi et al., 26 May 2025).

The empirical behavior is task-dependent. On LibriSQA, late layers develop crisp diagonal similarity bands and $i$0 generally decreases with depth for both QWEN and SALMONN, indicating stronger semantic speech–text alignment. On IEMOCAP emotion recognition, ALAS worsens in the last few layers, consistent with the task’s emphasis on prosody rather than lexical semantics. The paper reports SQA response similarity averages of 0.71 for QWEN and 0.76 for SALMONN, and emotion-recognition accuracies of 21.56% and 49.30%, respectively (Mousavi et al., 26 May 2025).

A notable terminological point is that in this paper “MAS” names the path-finding algorithm, not the score. Conceptually, however, ALAS is the multimodal alignment score, because it quantitatively measures how far a model’s latent alignment deviates from a trusted reference (Mousavi et al., 26 May 2025).

5. Human-aligned image–text MAS via iMatch

In "Instruction-augmented Multimodal Alignment for Image-Text and Element Matching" (Yue et al., 16 Apr 2025), MAS is naturally instantiated by the global image–text alignment score $i$1. The model also predicts element-level scores $i$2 for prompt elements, but the central global quantity is $i$3, “representing the overall matching degree.” During instruction tuning, the global score is discretized to

$i$4

and mapped to alphabet labels $i$5. Element scores are similarly discretized to seven ordinal levels (Yue et al., 16 Apr 2025).

The key mechanism is QAlign, which converts closed-set logits into a continuous score. Let $i$6 denote the logit for label $i$7, and define

$i$8

The expected 15-level score is

$i$9

which is then linearly rescaled to the dataset’s global range: $\Sigma_i = x_i y_i^\top$0 This $\Sigma_i = x_i y_i^\top$1 is the paper’s continuous image–text alignment score and functions as MAS in the global evaluation setting (Yue et al., 16 Apr 2025).

The iMatch pipeline combines several augmentation strategies. Validation-set pseudo-labeling augments training data; element augmentation injects element category labels and confidences into the prompt; image augmentation applies random lighting, random grid distortion, and random crop; and, for the element task, prompt type augmentation and score perturbation further improve accuracy. At test time, predicted element scores can be embedded as pseudo-features into the global prompt, but the paper does not specify a deterministic aggregation rule from element scores to the final global score (Yue et al., 16 Apr 2025).

Evaluation is conducted on EvalMuse-40K using SRCC, PLCC, and ACC. On the validation set, iMatch reaches SRCC 0.8304 and PLCC 0.8294 for global alignment, compared with 0.7742 and 0.7722 for FGA-BLIP2. For the element task, ACC reaches 0.8317 with Ovis2-8B, 0.8284 with InternVL2.5-8B-MPO, and 0.7948 with Qwen2.5-VL-7B-Instruct, against 0.7680 for FGA-BLIP2. The method also won first place in the CVPR NTIRE 2025 Text to Image Generation Model Quality Assessment Track 1, with Main 0.8551, SRCC 0.8249, PLCC 0.8485, and ACC 0.8734 (Yue et al., 16 Apr 2025).

Relative to embedding-only scores such as CLIPScore, the paper positions this MAS as a human-aligned, ordinally calibrated measure. Because it is produced through closed-set MLLM judgments and QAlign expectation rather than direct cosine similarity, it is intended to be more sensitive to fine-grained entity, attribute, and relation mismatches (Yue et al., 16 Apr 2025).

6. Geometric and internal-representation views of MAS

Two further formulations push MAS beyond pairwise external scoring. In "A TRIANGLE Enables Multimodal Alignment Beyond Cosine Similarity" (Cicchetti et al., 29 Sep 2025), the score is defined directly on three embeddings $\Sigma_i = x_i y_i^\top$2. Let $\Sigma_i = x_i y_i^\top$3 and $\Sigma_i = x_i y_i^\top$4. The core similarity is the triangle area

$\Sigma_i = x_i y_i^\top$5

Smaller $\Sigma_i = x_i y_i^\top$6 indicates tighter joint clustering of the three modalities, while larger $\Sigma_i = x_i y_i^\top$7 indicates poorer joint alignment. For downstream retrieval, the paper further uses

$\Sigma_i = x_i y_i^\top$8

where $\Sigma_i = x_i y_i^\top$9 is the angle for the task-critical pair. Inside the contrastive loss, the effective high-is-better score is $\Sigma_i = z_i z_i^\top$0. TRIANGLE improves cosine-based methods by up to 9 points of Recall@1 and achieves, for example, MSR-VTT zero-shot video–text retrieval results of T2V 55.2 and V2T 52.5, with comparable gains on DiDeMo, ActivityNet, VATEX, AudioCaps, and VGGSound 5K (Cicchetti et al., 29 Sep 2025).

In "Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputs" (Shukor et al., 2024), MAS is mapped to the paper’s implicit alignment score inside a frozen LLM. If $\Sigma_i = z_i z_i^\top$1 and $\Sigma_i = z_i z_i^\top$2 denote the perceptual and textual token subsequences at layer $\Sigma_i = z_i z_i^\top$3, with mean vectors $\Sigma_i = z_i z_i^\top$4 and $\Sigma_i = z_i z_i^\top$5, the default score is

$\Sigma_i = z_i z_i^\top$6

The paper also computes MaxSim, MinSim, AvgSim, and MedSim over token pairs, and complements token-level alignment with weight-level overlap,

$\Sigma_i = z_i z_i^\top$7

where the masks are derived by Wanda pruning scores. Cross-modal token alignment rises across depth and is highest after self-attention; reported inside-block values are approximately 0.45 after self-attention versus approximately 0.10 in the residual stream for Vicuna-v1.5, and approximately 0.58 versus approximately 0.15 for LLaVA-1.5-4. Weight-level IoU is high and increases with depth, with values around 0.67–0.69 across tasks and modality pairs (Shukor et al., 2024).

The paper further reports a positive correlation between the implicit alignment score and task performance, and a negative correlation with hallucinations on POPE and COCO object hallucination benchmarks. Because perceptual tokens change slowly across layers and activated subnetworks overlap strongly with textual ones, the study also proposes skipping FFN computations for some visual tokens and compressing the model into a single modality-agnostic subnetwork (Shukor et al., 2024).

Taken together, these two formulations broaden MAS in opposite directions. TRIANGLE treats alignment as a geometric property of three jointly embedded modalities, while the frozen-LLM study treats alignment as an intrinsic property of internal token dynamics and activated weights. A plausible implication is that recent MAS formulations occupy at least three distinct levels of analysis: dataset/sample selection, output evaluation, and internal representational diagnostics (Cicchetti et al., 29 Sep 2025, Shukor et al., 2024).