Cross-Modal Multi-Scale Alignment

• Cross-modal multi-scale alignment is a framework that establishes correspondences across modalities at various spatial, temporal, and semantic scales.
• It enables selective alignment and fusion strategies to mitigate granularity gaps, over-matching, and modality collapse in tasks like segmentation and retrieval.
• Empirical studies demonstrate its effectiveness in enhancing accuracy and efficiency, paving the way for advanced multimodal systems.

Cross-modal multi-scale alignment denotes the establishment of correspondences across modalities at multiple levels of abstraction and granularity rather than only within a single shared embedding space. In the recent literature, “multi-scale” spans spatial scales, temporal scales, semantic granularity, and hierarchical resolution, and it appears in tasks as varied as text-based person search, open-vocabulary segmentation, multimodal emotion recognition, remote sensing retrieval, multimodal sentiment analysis, and multimodal LLMs. Representative formulations include image–sentence, region–phrase, and image–phrase alignment; object-, region-, and pixel-level alignment; distribution-, instance-, and token-level alignment; and local token/segment-, prototype-, and global semantic-level alignment (Li et al., 2024, Ji et al., 2022, Liu et al., 2024, Wang et al., 2024, Qian et al., 14 Mar 2025).

1. Definition, Scope, and Conceptual Boundaries

In the surveyed literature, alignment and fusion are related but distinct operations. Alignment establishes “what corresponds to what,” either by learning a semantically consistent shared space or by imposing explicit matching relations such as region–word, patch–token, or token–token correspondences. Fusion instead determines “how to combine” the aligned signals into unified embeddings or predictions. This distinction is central because many modern systems adopt an “align before fuse” pattern, or they interleave progressive alignment and fusion rather than collapsing both into a single attention block (Li et al., 2024).

The term “multi-scale” is not restricted to image pyramids. It includes spatial scales such as multi-resolution visual features, temporal scales such as frame-level and segment-level alignment, and semantic granularity such as token/phrase/sentence in text or patch/object/scene in vision. In some systems, multi-scale is primarily hierarchical in representation depth; in others, it is explicitly semantic, moving from local details to global context. This is why image–text retrieval, multimodal sentiment analysis, open-vocabulary segmentation, and MLLM grounding can all be described with the same term while relying on different operational scales (Li et al., 2024).

A recurring misconception is that cross-modal multi-scale alignment is equivalent to global contrastive pretraining. The surveyed work does not support that equivalence. Global alignment remains important, but multiple papers identify a granularity gap between training and inference when only coarse correspondence is learned. In text-based person search, restricting alignment to individual scales overlooks image–phrase relations; in text-supervised open-vocabulary segmentation, training with coarse image- or region-level alignment while inferring at the group- or pixel-level leads to under- or over-segmentation (Ji et al., 2022, Liu et al., 2024).

2. Core Principles and Mathematical Patterns

A common pattern is to define separate alignment objectives for distinct scales and then aggregate them. The survey gives a generic multi-scale contrastive form,

$$L = \sum_{s \in S} w_s \, L_{\mathrm{InfoNCE}}^{(s)},$$

where the set of scales $S$ may correspond to global image–caption, region–phrase, or patch–token alignment. This formulation captures a broad family of models in which each scale contributes its own matching signal rather than being absorbed into a single fused representation (Li et al., 2024).

One large design axis concerns whether alignment is explicit or implicit. Attention-based models perform explicit matching through cross-attention, self-attention, or co-attention. MGCMA, for example, separates distribution/global, instance/utterance, and token/sequence-element alignment. Its token-level module stacks self-attention and cross-attention blocks, while utterance-level alignment is imposed with bidirectional InfoNCE and distribution-level alignment uses a Wasserstein-based contrastive loss between diagonal Gaussians (Wang et al., 2024). ACSA similarly treats cross-scale matching asymmetrically: it preserves image–sentence, region–phrase, and image–phrase relations while discarding region–sentence alignment because the latter can cause over-matching (Ji et al., 2022).

A second axis concerns transport- and distribution-based alignment. AlignMamba uses a relaxed OT procedure for local token-level correspondence to a language anchor and couples it with an RBF-kernel MMD loss for global distribution consistency. Its total objective is

$$\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{task}} + \lambda_g \, \mathcal{L}_{\text{align}},$$

where $$\mathcal{L}_{\text{align}} = \mathrm{MMD}^2(\tilde{X}_v, X_l) + \mathrm{MMD}^2(\tilde{X}_a, X_l)$$ after OT-based realignment of video and audio tokens to the language grid (Li et al., 2024). DecAlign pushes this further by explicitly decoupling modality-unique and modality-common streams. Heterogeneous features are aligned with prototype-guided multi-marginal OT plus local sample-to-prototype calibration, while homogeneous features are aligned with moment matching and MMD. Its end-to-end objective is

$$L_{\mathrm{total}} = L_{\mathrm{task}} + L_{\mathrm{dec}} + \alpha (L_{\mathrm{OT}} + L_{\mathrm{proto}}) + \beta (L_{\mathrm{sem}} + L_{\mathrm{MMD}}),$$

which formalizes a hierarchy from local token/segment alignment, through prototype-level distribution alignment, to global semantic consistency (Qian et al., 14 Mar 2025).

A third axis concerns whether modality-common and modality-specific information should be aligned together. DecAlign argues that aggressive alignment in a shared space can cause over-alignment and modality collapse, so it separates modality-unique features $F_{\mathrm{uni}}$ from modality-common features $F_{\mathrm{com}}$ and applies different alignment strategies to each. This design reframes multi-scale alignment as both a granularity problem and a heterogeneity problem (Qian et al., 14 Mar 2025).

3. Representative Architectural Realizations

The literature operationalizes cross-modal multi-scale alignment in task-specific ways, but several recurrent templates can be organized succinctly.

System	Scales	Main alignment mechanism
ACSA	image–sentence, region–phrase, image–phrase	global CMPM/CMPC plus asymmetric cross-attention
MGCA	object, region, pixel	pseudo correspondences, hard sampling, symmetric InfoNCE
MSA	four image scales with localized text	per-scale cross-attention, per-scale contrastive loss, cross-scale consistency
MGCMA	distribution/global, instance/utterance, token/local	Wasserstein distribution alignment, token cross-attention, bidirectional InfoNCE
AlignMamba	local token-level, global distribution	relaxed OT, MMD, time-priority Mamba fusion
DecAlign	local token/segment, prototype, global semantic	decoupling, GMM-guided multi-marginal OT, moment matching, transformer fusion

These systems show that “multi-scale” can mean semantic hierarchy, spatial hierarchy, temporal hierarchy, or a combination of them. ACSA extracts multi-scale representations with a Swin Transformer and BERT, then aligns them asymmetrically so that phrases may attend to either global image tokens or local body regions, but full sentences are not forced onto single regions (Ji et al., 2022). MGCA constructs pseudo multi-granular semantic correspondences from a pixel-to-text similarity matrix and learns object-, region-, and pixel-level contrastive alignment to bridge the train–test granularity gap in open-vocabulary semantic segmentation (Liu et al., 2024). MSA, designed for remote sensing image–text retrieval, aligns each visual scale separately to localized text features and regularizes smaller-scale matching matrices toward the largest scale with a KL-based consistency loss (Yang et al., 2024). MGCMA, AlignMamba, and DecAlign all define multi-scale hierarchies that move from local correspondences to global distributional agreement, but they differ in whether the transition is mediated by attention, OT, latent moments, or decoupled streams (Wang et al., 2024, Li et al., 2024, Qian et al., 14 Mar 2025).

Other recent systems extend the idea beyond standard dual encoders. EMMA defines structural alignment at the image level through a pixel-wise reconstruction loss inside a Mamba-based MLLM and hierarchical alignment at the feature level through multi-scale fusion of intermediate hidden states (Xing et al., 2024). S4ECA treats scale selection itself as language-conditioned: textual proxies distilled by learnable queries guide a visual adapter that performs gated scale selection, spatial reweighting, and language-guided visual adaptation while updating only 2.43% of the backbone parameters (Li et al., 29 Jun 2026).

4. Empirical Behavior Across Tasks

The empirical record in the provided literature is broad rather than uniform. In multimodal sentiment and emotion analysis, DecAlign reports results on CMU-MOSI, CMU-MOSEI, CH-SIMS, and IEMOCAP. On CMU-MOSI it achieves Acc-2 85.75, Acc-7 45.07, F1 85.82, MAE 0.735, and Corr 0.811; on CMU-MOSEI it reports Acc-2 86.48, Acc-7 55.02, F1 86.07, MAE 0.543, and Corr 0.768; on CH-SIMS it reports MAE 0.403 and F1 81.85; and on IEMOCAP it reports WAcc 73.35 and WAF1 73.43. The paper states that DecAlign consistently surpasses transformer-only fusion baselines and decoupling-only baselines, and its ablations show that removing heterogeneous alignment causes larger drops than removing homogeneous alignment (Qian et al., 14 Mar 2025).

On IEMOCAP, MGCMA reports WA 78.87% and UA 80.24%, compared with a best prior of WA 78.42% and UA 79.71%. Its ablation study reports WA/UA of 77.78/79.07 without the distribution alignment module, 78.00/79.14 without the token alignment module, 78.30/79.38 without the instance alignment module, and 76.20/77.62 without all alignments, indicating that the three scales are complementary and that distribution alignment contributes the most (Wang et al., 2024). AlignMamba reports 86.9% Accuracy / 86.9% F1 on CMU-MOSI and 86.6% / 86.5% on CMU-MOSEI for complete multimodal fusion, while also showing smaller performance degradation than baselines under missing-modality settings (Li et al., 2024).

In dense prediction and retrieval, MGCA reports VOC 53.1, Context (60) 31.3, COCO-Object (81) 31.9, and Avg 38.8 on datasets with a background class, and states that it outperforms TCL by +3.5 mIoU avg using only CC3M. On datasets without background class it reports VOC20 81.4, Context59 33.7, COCO-Stuff 22.0, Cityscapes 24.0, ADE20K 16.4, and Avg 35.5, improving over TCL by +2.4 mIoU avg (Liu et al., 2024). In remote sensing image–text retrieval, MSA reports mR 38.08 on RSITMD with ResNet-50, 23.45 on RSICD with ResNet-101, and 52.11 on UCM Caption with ResNet-50, and its ablations show that adding the cross-scale consistency loss yields further gains beyond multi-scale alignment alone (Yang et al., 2024).

In multimodal LLMs and referring segmentation, EMMA reports HallusionBench 51.0 versus Cobra 41.4 and improves TextVQA from 52.4 to 57.2 with EMMA-V1, alongside a substantial increase in MME from 1294.3 to 1572.8 (Xing et al., 2024). S4ECA-L reports on RRSIS-D [email protected] 81.03%, [email protected] 63.12%, [email protected] 31.73%, oIoU 80.07%, and mIoU 69.76%, and on RefSegRS [email protected] 88.51%, [email protected] 72.90%, [email protected] 25.12%, oIoU 82.35%, and mIoU 77.64% (Li et al., 29 Jun 2026).

5. Efficiency, Robustness, and Failure Modes

A major motivation for multi-scale design is not only accuracy but computational control. DecAlign explicitly pushes heavy distribution matching to prototype-level spaces. It states that naive multi-marginal OT over prototypes has complexity $O(KM)$ for constructing the full cost tensor and $O(KM)$ for storing $\Gamma$, while prototype grids reduce dimensionality and entropic regularization enables generalized Sinkhorn-style solvers. It also removes OT, MMD, and prototype computations at inference, retaining only encoder and transformer forward passes (Qian et al., 14 Mar 2025).

AlignMamba treats alignment as a preconditioning stage for a linear-time Mamba backbone. Under fixed $S$0, it reports 46.7G FLOPs versus 101.6G for a single-stream Transformer and 203.2G for a multi-stream Transformer, 8.53 GB memory versus 10.7 GB and 20.3 GB at 6.4k tokens, and 6.05 s inference time versus 36.13 s and 48.61 s. These numbers frame multi-scale alignment not as an alternative to efficiency, but as a mechanism that can make linear-time fusion more effective by injecting explicit local and global correspondence before scanning (Li et al., 2024).

EMMA and S4ECA use two different efficiency strategies. EMMA confines its multi-scale feature fusion and visual decoder to training, so they add no inference overhead, and the paper states that the model is nearly four times faster than transformer-based MLLMs of similar scale during inference (Xing et al., 2024). S4ECA freezes CLIP and DINOv2 backbones and trains 3.65M parameters, only 2.43% of the 149.97M parameters required by full fine-tuning, while keeping GFLOPs close to the frozen-backbone setting at 169.45G versus 168.73G (Li et al., 29 Jun 2026).

Failure modes remain highly task-dependent. MGCMA notes that mis-transcriptions or poor ASR text input degrade token-level alignment, that asynchrony between speech frames and text tokens can still cause misalignment, and that class imbalance and ambiguous labels can blur class boundaries (Wang et al., 2024). DecAlign states that prototype OT may underfit when modality heterogeneity is extreme or prototypes poorly capture distributions, and that domain shifts can invalidate learned prototype structures and latent moment statistics (Qian et al., 14 Mar 2025). S4ECA identifies ambiguous or underspecified expressions, extremely small targets at low SNR, and multiple disconnected referents as persistent difficulties, particularly for [email protected] (Li et al., 29 Jun 2026). AlignMamba notes kernel sensitivity in MMD and asymmetry induced by anchoring to language, which may motivate joint or cyclic alternatives (Li et al., 2024).

6. Methodological Debates and Future Directions

Several debates recur across the literature. One concerns symmetry. ACSA argues that cross-scale alignment should be asymmetric because a phrase may map to an entire image or to specific regions, whereas a full sentence is unlikely to map meaningfully to a single region; its ablations show that discarding region–sentence alignment improves performance (Ji et al., 2022). Another concerns whether all information should be aligned. DecAlign argues that entangling heterogeneous and homogeneous signals can cause over-alignment and modality collapse, with dominant modalities suppressing weaker ones, so selective alignment is preferable to indiscriminate alignment (Qian et al., 14 Mar 2025). These results suggest that the central problem is often not how to maximize alignment everywhere, but how to determine which scales and which subspaces should be aligned strongly, weakly, or not at all.

The most explicit forward-looking proposals in the papers on arXiv point toward richer hierarchies. DecAlign proposes explicit feature pyramids with scale-specific costs and transport plans, cross-scale attention, dynamic prototypes, barycentric alignment, exact Bures-Wasserstein geometry, sliced multi-marginal OT, and sample-based stochastic OT (Qian et al., 14 Mar 2025). MGCMA proposes alternative distribution divergences, token-level Sinkhorn alignment, uncertainty-aware instance matching, hierarchical multi-scale token alignment, and multi-task learning (Wang et al., 2024). The survey emphasizes efficient multi-scale transformers, weakly supervised fine-grained alignment, continual or adaptive alignment, modality gap reduction, and trustworthiness as open research directions (Li et al., 2024). S4ECA adds adaptive query counts and dynamic scale granularity as concrete extensions in dense aerial segmentation (Li et al., 29 Jun 2026).

A plausible implication is that future cross-modal multi-scale alignment systems will combine three ingredients that are currently dispersed across subfields: selective scale pairing, distribution-aware alignment, and lightweight task-specific adaptation. The present literature already contains each element separately—explicit scale selection, OT or MMD-style matching, asymmetric or decoupled alignment, and parameter-efficient adapters—but no single framework yet unifies them across the full range of multimodal tasks represented here.