Good Scores, Bad Data: A Metric for Multimodal Coherence

Abstract: Multimodal AI systems are evaluated by downstream task accuracy, but high accuracy does not mean the underlying data is coherent. A model can score well on Visual Question Answering (VQA) while its inputs contradict each other. We introduce the Multimodal Coherence Score (MCS), a metric that evaluates fusion quality independent of any downstream model. MCS decomposes coherence into four dimensions, identity, spatial, semantic, and decision, with weights learned via Nelder-Mead optimization. We evaluate on 1,000 Visual Genome images using DETR, CLIP, and ViLT, and validate on 150 COCO images with no retraining. Across three fusion architectures, MCS discriminates quality with higher sensitivity than task accuracy alone (Spearman rho = 0.093 vs. 0.071). Perturbation experiments confirm each dimension responds independently to its failure mode with zero cross-talk. MCS is lightweight, requires no human annotation, and tells you not just that something broke, but what broke.

• The paper introduces Multimodal Coherence Score (MCS), a weighted metric combining identity, spatial, semantic, and decision coherence to diagnose inconsistencies across images, annotations, descriptions, relationships, and QA records.
• The paper finds that contract-enforced fusion substantially improves coherence over naive concatenation, raising MCS from 0.214 to 0.330 while downstream decision accuracy remains nearly unchanged at roughly 0.423–0.460.
• The paper shows that perturbation tests produce dimension-specific degradation with zero cross-talk, while semantic coherence dominates the learned weighting (0.722) and MCS achieves a modest but significant Spearman correlation of 0.093 with decision coherence.

Motivation and problem statement

The paper addresses a gap in multimodal evaluation: existing metrics (VQA accuracy, Recall@K, CIDEr, CLIP similarity) measure model performance or pairwise alignment, but none assesses whether the records fused into a single multimodal event are internally coherent. The motivating example is Visual Genome image #2320169, where DETR detects thirteen zebras while the annotations label most of them "animal," one "bunch," individual stripes as objects, and repeat region descriptions verbatim. A VQA system still answers correctly, so benchmark scores look fine while the data is not. The authors argue this class of incoherence propagates into production settings — radiology metadata contradicting reports, LiDAR/camera label disagreement in autonomous driving, extracted tables conflicting with document text — with no metric to flag it.

The Multimodal Coherence Score

MCS evaluates a multimodal event $E$ (image pixels, object annotations, relationship triplets, region descriptions, QA pairs) along four dimensions:

• Identity Coherence (IC): Jaccard similarity between DETR-ResNet-50 detections (confidence 0.7) and annotated entity sets, with a label normalization map aligning Visual Genome vocabulary to COCO categories.
• Spatial Coherence (SpC): mean IoU between annotated bounding boxes and detector boxes.
• Semantic Coherence (SC): minimum pairwise cosine similarity between the CLIP ViT-B/32 image embedding and all text-record embeddings; the min reflects the position that coherence is only as strong as the weakest pair.
• Decision Coherence (DC): binary correctness of a ViLT-B/32 VQAv2-fine-tuned reference model on up to three QA pairs per image.

The composite is a learned weighted sum optimized via Nelder-Mead to maximize Spearman correlation between MCS (computed from IC, SpC, SC only) and DC across the dataset. The design intent is diagnostic: DC alone reports failure, whereas the per-dimension decomposition attributes it.

Experimental setup

Primary evaluation uses 1,000 Visual Genome images; transfer is validated on 150 COCO 2017 images and 130 Open Images V7 images without retraining. Three fusion architectures are compared: naive concatenation, contract-enforced fusion (rejecting boxes with IoU < 0.1 against detections, descriptions with CLIP similarity < 0.3, and triplets referencing unvalidated objects), and foundation-model fusion (CLIP-based re-ranking at a 0.5 threshold). Perturbation experiments on 200 images apply object swaps, bounding-box shuffles (offsets of 100–200% of box dimension), caption swaps, and compound corruption at 10%, 20%, and 50% rates. Evaluation axes are discriminative power (Kruskal-Wallis), downstream correlation, decomposability (>20% target degradation, <5% cross-talk), and transfer.

Results

Architecture comparison: contract-enforced fusion roughly doubles IC (0.272 vs. 0.121) and SpC (0.500 vs. 0.239) over naive concatenation, and also leads on SC (0.265), yielding MCS = 0.330 vs. 0.214. The central finding is that DC is nearly flat across all three architectures (0.423–0.460): downstream accuracy cannot distinguish fusion quality, while MCS can. All differences are significant ($p < 0.001$, Kruskal-Wallis).

Decomposability: perturbation results confirm dimensional independence with zero cross-talk. At 50% corruption, object swap degrades IC by 35.6% with SpC and SC unchanged; bbox shuffle degrades SpC by 38.3%; caption swap degrades SC by 26.7%; compound corruption degrades all three (−37.2%, −39.4%, −25.6%). Degradation scales linearly with corruption rate.

Learned weights and correlation: SC dominates at $w_{\text{SC}} = 0.722$, SpC carries 0.276, and IC contributes only 0.002. The composite achieves Spearman $\rho = 0.093$ ($p = 0.003$) with DC, exceeding the best single dimension (SpC, $\rho = 0.071$). The near-zero IC weight indicates that on Visual Genome, detection is reliable enough that entity mismatches barely affect downstream success; the text–image semantic gap is the dominant failure mode. The authors correctly frame these weights as a dataset-specific diagnostic fingerprint rather than universal constants — in medical imaging or autonomous driving, other dimensions would plausibly dominate, though this is asserted rather than demonstrated.

Transfer: VG-learned weights produce meaningful scores on COCO (MCS = 0.571) and Open Images (0.272) with zero retraining, supporting H5.

An additional practical finding favors simple validation rules over general-purpose filtering: contract-enforced fusion outperforms CLIP-based re-ranking precisely on the dimensions its rules target, suggesting explicit filters suffice when failure modes are well-characterized.

Limitations and open questions

The paper concedes several dependencies. DC is computed from a single reference model (ViLT), biasing learned weights toward that model's failure modes. IC assumes a reliable detector, so the near-zero IC weight may partly reflect DETR's strength on Visual Genome rather than a domain-invariant property. Weights are learned against VQA-style decision coherence and may overfit to that task family; cross-task validation (training on VQA, evaluating on retrieval) is explicitly left as future work. Two further points deserve scrutiny: the absolute downstream correlation is modest ($\rho = 0.093$), so MCS's predictive value over accuracy is real but small in magnitude; and the transfer experiments report MCS values on COCO and Open Images without ground truth for what those values should be, so transfer validity rests on plausibility rather than calibrated agreement.

Conclusion

The paper introduces MCS, a lightweight, annotation-free metric that decomposes multimodal fusion quality into four independently measurable dimensions and produces actionable per-dimension diagnostics. Its strongest empirical results are the zero-cross-talk perturbation decomposition and the demonstration that downstream accuracy is nearly invariant across fusion architectures that differ substantially in coherence. The framework's weights function as a dataset-level audit signal — here identifying text–image misalignment as the primary weakness of Visual Genome — though the modest correlation magnitudes and VQA-specific weight learning leave open how well the approach calibrates across tasks and domains.