Multi-Modal Feature Alignment Loss

• Multi-Modal Feature Alignment (MMFA) loss is a functional category that unifies diverse modality representations into a coordinated embedding space.
• It employs various strategies such as contrastive, Gram-based, decoupled, and information bottleneck approaches to optimize alignment.
• MMFA techniques enhance downstream performance by balancing modality-specific features with global semantic compatibility.

Multi-Modal Feature Alignment (MMFA) loss is most accurately treated as an umbrella designation for objectives that constrain heterogeneous modality representations to become semantically compatible in a shared or coordinated embedding space. Recent work does not present a single canonical MMFA formula. Instead, alignment appears in several distinct forms: the "Gramian Representation Alignment Measure Loss" in drug-target interaction prediction, token-wise cosine alignment to frozen modality experts in multimodal video diffusion, decoupled alignment and uniformity objectives for multimodal representation learning, Information Bottleneck regularization that suppresses modality-specific nuisances, one-vs-all contrastive-plus-minimality training for arbitrary-modality alignment, and CLIP-guided contrastive feature alignment for multimodal semantic understanding (Li et al., 14 Oct 2025, Xiao et al., 3 May 2026, Yin et al., 10 Feb 2026, Almudévar et al., 5 Jun 2025, Li et al., 28 May 2026, Zhang et al., 2024).

1. Nomenclature and scope

The expression “MMFA loss” is not standardized across the recent literature. Several papers studied here explicitly solve an MMFA-type problem but use different names for the alignment mechanism. In "M3ST-DTI" the relevant objective is called Gram Loss, also referred to as GRAM Loss or Gramian Representation Alignment Measure Loss, and the paper does not introduce a loss named “MMFA” (Li et al., 14 Oct 2025). In "Zoom and Shift are All You Need," the alignment objective is embodied by Alternative Telescopic Displacement (ATD), which is an architectural alignment-and-fusion mechanism rather than a separately parameterized loss (Qin, 2024). In "Multimodal Representation Alignment for Cross-modal Information Retrieval," the contribution is an analysis of alignment metrics and learned similarity functions rather than a new canonical MMFA loss (Xu et al., 10 Jun 2025).

This non-uniform terminology reflects a broader methodological fact. Some works define MMFA through an explicit auxiliary loss term; others define it through a joint training objective in which alignment is only one component; still others implement alignment at the architectural level through calibration, projection, displacement, or fusion modules. A plausible implication is that “MMFA loss” is best understood as a functional category rather than a single formula.

The modalities covered by these formulations are correspondingly diverse. Recent instances include textual, structural, and functional features for drugs and targets (Li et al., 14 Oct 2025); RGB, depth, and segmentation masks in multimodal video generation (Xiao et al., 3 May 2026); arbitrary numbers of modalities in conflict-aware or one-vs-all representation learning (Yin et al., 10 Feb 2026, Li et al., 28 May 2026); image and text in semantic understanding and retrieval (Zhang et al., 2024, Xu et al., 10 Jun 2025); and time series, image, and text in ATD-based fusion (Qin, 2024). Across these settings, the common role of MMFA is to reduce the heterogeneity gap while preserving task-relevant information.

2. Principal objective families

A first family of MMFA objectives is contrastive or cosine-based alignment. In "CLFA," the auxiliary alignment loss is symmetric and bidirectional. If $L_i$ and $L_t$ denote image-side and text-side alignment, the final contrastive term is

$L_{con} = \frac{L_i + L_t}{2},$

and total training uses

$L = \alpha L_{con} + L_{ce}.$

The method aligns ViT features with CLIP image features and BERT features with CLIP text features, with cosine similarity and temperature scaling in an InfoNCE-like construction (Zhang et al., 2024). In "M$^2$-REPA," alignment is also cosine-based but operates token-wise against frozen expert foundation models: $\mathcal{L}_{\text{M%%%%0%%%%-REPA}} = -\frac{1}{K}\sum_{k=1}^{K} \mathbb{E}\left[\frac{1}{N}\sum_{n=1}^{N} \text{cos}(\hat{y}^{(k)}_{*,[n]}, \hat{h}^{(k)}_{\phi,[n]})\right].$ This is explicitly a feature-level, token-wise alignment loss rather than a contrastive loss (Xiao et al., 3 May 2026).

A second family is determinant- or Gram-based alignment. In "M3ST-DTI," the three modality features are normalized, arranged into a Gram matrix $G$, and converted into a volume-like quantity

$V = \sqrt{\det(G) + \varepsilon}.$

The alignment objective is then

$$\text{Gram Loss} = -\frac{1}{B} \sum_{i=1}^{B} \log \left( \frac{\exp(-V_i / \tau)}{\sum_{j=1}^{k} \exp(-V_j / \tau)} \right),$$

with total alignment

$L_g=(\text{Gram}_{d}+\text{Gram}_{t})/2.$

The paper states that this loss “constrains the semantic closeness of different modal features in the embedding space” (Li et al., 14 Oct 2025). The same paper notes a notational inconsistency between $L_t$0 and $L_t$1, but the intended quantity is the volume-like term $L_t$2.

A third family is decoupled alignment-plus-regularization. "UniAlign" separates intra-modality uniformity from cross-modality alignment: $L_t$3 where

$L_t$4

This design is presented as conflict-free because uniformity is enforced within each modality only, while cross-modal alignment uses a single anchor modality (Yin et al., 10 Feb 2026). "M$L_t$5-REPA" also uses two synergistic objectives: the token-wise alignment term above plus a CKA-based decoupling loss,

$L_t$6

and combines them with flow matching (Xiao et al., 3 May 2026).

A fourth family is Information Bottleneck-based MMFA. In "Aligning Multimodal Representations through an Information Bottleneck," the training objective is

$L_t$7

where $L_t$8 approximates a KL-based minimality term and, under isotropic Gaussian assumptions, becomes proportional to

$L_t$9

The argument is that InfoNCE promotes sufficiency, whereas the added term suppresses modality-specific nuisance information (Almudévar et al., 5 Jun 2025). "OVA-IB" generalizes this logic to arbitrary modality counts: $L_{con} = \frac{L_i + L_t}{2},$0 with a one-vs-all InfoNCE sufficiency term and a minimality regularizer defined against the remaining modalities rather than against a single partner (Li et al., 28 May 2026).

3. Placement within multimodal pipelines

MMFA objectives are often positioned before the main interaction or prediction stage. In "M3ST-DTI," alignment belongs to the early fusion stage, where multi-source cross-attention (MCA) is paired with Gram Loss as a structural constraint for the textual, structural, and functional modalities. This occurs before the late-stage BCA and DOF modules. The paper makes the stage ordering explicit: feature extraction, early fusion with MCA + Gram Loss, late fusion with BCA + DOF, and then prediction (Li et al., 14 Oct 2025). In "CLFA," the alignment loss is inserted before the main cross-attention fusion and classification block: BERT and ViT first encode text and image, small MLPs project them into the CLIP embedding dimension, the CLIP-guided contrastive alignment is applied, and only then are the aligned features passed to cross-attention and the classification head (Zhang et al., 2024).

A second architectural pattern applies MMFA to intermediate latent features rather than encoder outputs. "M$L_{con} = \frac{L_i + L_t}{2},$1-REPA" extracts intermediate diffusion features $L_{con} = \frac{L_i + L_t}{2},$2, decouples them into modality-specific subspaces using lightweight MLPs $L_{con} = \frac{L_i + L_t}{2},$3, and aligns each branch to the corresponding frozen expert feature map. The alignment therefore regularizes the denoising backbone internally rather than replacing the generative objective (Xiao et al., 3 May 2026).

A third pattern embodies alignment mainly as a fusion mechanism. "Zoom and Shift are All You Need" states that feature alignment serves as the primary mechanism for fusing multimodal data, but the paper does not define a separate MMFA loss. Instead, ATD uses modality-specific encoders, a modality-aware guide module that computes $L_{con} = \frac{L_i + L_t}{2},$4 and $L_{con} = \frac{L_i + L_t}{2},$5 for normalization, and an ATD fusion module that maps $L_{con} = \frac{L_i + L_t}{2},$6 and $L_{con} = \frac{L_i + L_t}{2},$7 into each other’s space via trainable displacement matrices before concatenation and projection (Qin, 2024). This suggests that in some systems the distinction between “alignment loss” and “alignment architecture” is intentionally blurred.

These placements matter because they determine whether alignment functions as an auxiliary regularizer, as the primary fusion operator, or as a global training principle. In the surveyed literature, all three roles occur.

4. Objects and granularity of alignment

The aligned entities vary from modality-specific sample embeddings to token grids, expert features, and one-vs-all subspaces.

Method	Alignment mechanism	Aligned objects
M3ST-DTI (Li et al., 14 Oct 2025)	Gram Loss with MCA	textual / structural / functional features within drugs and within targets
CLFA (Zhang et al., 2024)	CLIP-guided contrastive loss	BERT $L_{con} = \frac{L_i + L_t}{2},$8 CLIP text, ViT $L_{con} = \frac{L_i + L_t}{2},$9 CLIP image
M$L = \alpha L_{con} + L_{ce}.$0-REPA (Xiao et al., 3 May 2026)	token-wise cosine matching + CKA decoupling	projected diffusion tokens $L = \alpha L_{con} + L_{ce}.$1 frozen expert tokens
UniAlign (Yin et al., 10 Feb 2026)	anchor-based $L = \alpha L_{con} + L_{ce}.$2 alignment + intra-modality uniformity	all modalities aligned to one anchor modality
OVA-IB (Li et al., 28 May 2026)	one-vs-all InfoNCE + minimality regularizer	each modality against the joint context of all remaining modalities
ATD (Qin, 2024)	shift-and-expand displacement fusion	calibrated encoder outputs across modalities

"M3ST-DTI" aligns three modalities within drugs and within targets, then averages the resulting losses. The model therefore does not directly align drug features to target features at the early stage; it aligns modality-specific embeddings inside each entity type before downstream interaction modeling (Li et al., 14 Oct 2025). "CLFA" is also indirect in a different sense: it does not directly enforce similarity between arbitrary raw image-text pairs, because the paper emphasizes that in sarcasm detection or sentiment analysis the image and text may not always be semantically consistent. Instead, each task encoder is aligned to the relevant CLIP teacher space (Zhang et al., 2024).

"M$L = \alpha L_{con} + L_{ce}.$3-REPA" makes the granularity especially explicit. The alignment is feature-level, token-level, and applied to intermediate diffusion representations rather than to full outputs or labels (Xiao et al., 3 May 2026). "UniAlign" and "OVA-IB" instead emphasize the geometry of full sample embeddings. In "UniAlign," the key design is that all non-anchor modalities are pulled toward a single anchor, removing competing alignment directions (Yin et al., 10 Feb 2026). In "OVA-IB," the key object is not a single paired vector but the span of the other modalities, with a parameter-free geometry-aware projection

$L = \alpha L_{con} + L_{ce}.$4

used to score how well modality $L = \alpha L_{con} + L_{ce}.$5 lies in the subspace jointly supported by the others (Li et al., 28 May 2026).

A plausible synthesis is that MMFA has expanded from pairwise sample matching toward structured alignment across tokens, modality experts, and higher-order modality sets.

5. Theoretical interpretations

One theoretical line interprets MMFA as a response to InfoNCE conflicts. "Towards Uniformity and Alignment for Multimodal Representation Learning" argues that InfoNCE in multimodal settings suffers from two failures that worsen as the number of modalities $L = \alpha L_{con} + L_{ce}.$6 grows: an alignment–uniformity conflict, in which the repulsion needed for uniformity opposes pairwise alignment, and an intra-alignment conflict, in which alignment vectors from multiple modalities are not collinear and therefore partially cancel (Yin et al., 10 Feb 2026). The proposed remedy is to decouple the two roles: uniformity is computed within each modality via a Gaussian-kernel-based objective, while alignment is enforced across modalities through a single anchor. The same paper further relates this decomposition to a global Hölder divergence over modality distributions and presents the objective as a tractable proxy for reducing distribution gap.

A second line interprets MMFA through the Information Bottleneck Principle. "Aligning Multimodal Representations through an Information Bottleneck" distinguishes shared essence $L = \alpha L_{con} + L_{ce}.$7 from modality-specific nuisances $L = \alpha L_{con} + L_{ce}.$8. The paper states that aligned multimodal representations should be minimal sufficient representations of the essence: sufficient means preserving the shared content, while minimal means discarding nuisance. The proposed KL-derived regularizer is intended to minimize $L = \alpha L_{con} + L_{ce}.$9, thereby reducing nuisance retention that standard InfoNCE leaves unaddressed (Almudévar et al., 5 Jun 2025). "OVA-IB" generalizes this perspective to arbitrary modality counts: sufficiency should preserve what is predictable from the remaining modalities, and minimality should compress what is not supported by them. The resulting one-vs-all objective is linked to a Dual Total Correlation-style interpretation rather than to a collection of independent pairwise comparisons (Li et al., 28 May 2026).

A third line emphasizes distributional geometry. The image-text retrieval analysis in "Multimodal Representation Alignment for Cross-modal Information Retrieval" evaluates the modality gap through the Euclidean distance between centroids and the Wasserstein-2 distance $^2$0. The paper finds that Wasserstein distance is informative for measuring modality gap, but also states that a smaller gap does not necessarily translate into better retrieval (Xu et al., 10 Jun 2025). This is important because it distinguishes geometric alignment from downstream utility.

Taken together, these interpretations show that MMFA is not only a similarity-maximization device. It is also used to manage conflicts among modalities, enforce compression of modality-specific information, and control the geometry of multimodal distributions.

6. Empirical behavior, applications, and limitations

Empirical results consistently show that alignment mechanisms can materially affect downstream performance, but they also show that the effect depends on how alignment is formulated. In "M3ST-DTI," the ablation on BindingDB reports Only early at Accuracy 82.42, F1 58.69, AUROC 90.25, AUPRC 56.44, while the full model reports Accuracy 83.36, F1 60.38, AUROC 91.04, AUPRC 60.47. The same study states that the early fusion stage with MCA + Gram alignment is beneficial and that the full two-stage design performs best overall (Li et al., 14 Oct 2025). In "CLFA," the no-alignment baseline BERT+ViT reaches F1 = 79.80 on MMSD, whereas CLFA reaches F1 = 83.91; on MVSA-Single, BERT+ViT gives F1 = 68.84 and CLFA gives F1 = 72.45; on MVSA-Multiple, the values are 67.39 and 68.31 respectively. The paper also reports that Caption alignment reaches F1 = 81.76, below CLFA = 83.91 (Zhang et al., 2024).

Architectural alignment without an explicit MMFA loss can also yield large gains. In "Zoom and Shift are All You Need," on ETT time-series forecasting the Numerical only setting reports MAE 0.187, MSE 0.030, whereas multimodal ATD reports MAE 0.058, MSE 0.006. On MIT-BIH arrhythmia classification, Image only gives Accuracy 0.518, F1 0.479, Time-series only gives Accuracy 0.858, F1 0.893, and multimodal ATD gives Accuracy 0.989, F1 0.982. Replacing the ATD fusion module with LMF or Cross-attention degrades performance on both tasks (Qin, 2024).

The more recent theory-driven MMFA formulations report improvements alongside failure modes that are specific to alignment design. "UniAlign" reports improvements over VAST by about 8.7 R@1 on T2V and 6.4 R@1 on V2T, and over GRAM by 3.0 R@1 on T2V and 1.6 R@1 on V2T; for UnCLIP-style generation, the reported pattern is roughly 10–40 lower FID depending on decoder and setting (Yin et al., 10 Feb 2026). "M$^2$1-REPA" reports that naive multi-expert alignment can degrade performance or yield only marginal gains, and that CKA decoupling is clearly better than the ablated cosine-squared alternative (Xiao et al., 3 May 2026). "Aligning Multimodal Representations through an Information Bottleneck" reports that adding the CM regularizer to ITC + LM on COCO improves captioning from CIDEr 91.7, BLEU@4 28.6 to CIDEr 93.0, BLEU@4 29.4 at $^2$2, while also noting that retrieval performance drops somewhat as $^2$3 increases too much (Almudévar et al., 5 Jun 2025).

Several recurrent limitations follow directly from these findings. MMFA is not necessarily a single standalone loss; it may be auxiliary, architectural, or part of a composite objective (Li et al., 14 Oct 2025, Qin, 2024). Stronger geometric alignment does not automatically imply better retrieval, as indicated by the Wasserstein analysis (Xu et al., 10 Jun 2025). Naive aggregation of multiple experts can create feature conflicts unless decoupling is introduced (Xiao et al., 3 May 2026). Simple learned scorers, particularly shallow MLP-based similarity functions, can underperform cosine similarity for image-text retrieval and may be insufficient to close the modality gap created by independently trained encoders (Xu et al., 10 Jun 2025). In Information Bottleneck settings, larger $^2$4 can suppress nuisance information but can also over-compress and harm sufficiency or downstream accuracy (Almudévar et al., 5 Jun 2025).

These results suggest that MMFA is most effective when its alignment target, regularization strategy, and placement in the pipeline are matched to the modality structure of the task. The contemporary literature therefore treats MMFA less as a monolithic loss than as a design space for controlling semantic compatibility, redundancy, and modality-specific information in multimodal representation learning.