Multimodal Dual-Alignment Fusion (MDAF)

• Multimodal Dual-Alignment Fusion (MDAF) is a framework that separates multiple alignment processes before fusing heterogeneous modality signals.
• It employs dual alignment—such as temporal and semantic—to establish robust correspondences, thereby reducing cross-modal mismatches and improving prediction accuracy.
• By decoupling alignment from fusion, MDAF enables efficient feature-level integration, as demonstrated in applications like multimodal sentiment analysis and object detection.

Multimodal Dual-Alignment Fusion (MDAF) denotes a class of multimodal architectures in which heterogeneous modalities are not fused immediately, but are first brought into correspondence by two distinct alignment processes and only then integrated by a dedicated fusion mechanism. In recent alignment-and-fusion literature, alignment is defined as establishing semantic correspondences across modalities so that their representations occupy a shared space, whereas fusion merges those aligned signals into unified predictions or embeddings (Li et al., 2024). The term itself is not a universally standardized framework name. Rather, it functions as an umbrella description for systems that explicitly decouple alignment from fusion and instantiate two complementary alignment axes—such as temporal and semantic, spatial and semantic, local geometric and global statistical, or visual-visual and visual-text—before adaptive integration (Wen et al., 5 Dec 2025).

1. Conceptual scope and taxonomy

Within the general taxonomy of multimodal learning, MDAF is most naturally situated at the feature level. The survey literature distinguishes data-level, feature-level, output-level, and hybrid fusion, and identifies feature-level fusion as especially effective because it models relationships between modalities before final prediction (Li et al., 2024). An MDAF-style system typically follows an align-before-fuse or progressive alignment-and-fusion pattern rather than raw early fusion or purely decision-level aggregation. The same survey highlights dual-stream and two-stage precedents, including dual-stream co-attentional transformers, align-before-fuse designs, and progressive alignment-fusion paradigms, which together provide the conceptual background for MDAF as a staged rather than monolithic operation (Li et al., 2024).

What distinguishes MDAF from generic multimodal fusion is the claim that “alignment” is not singular. Different forms of cross-modal mismatch require different mechanisms. In one setting, sequence lengths differ and temporal correspondence must be established; in another, features may already be temporally synchronized but remain semantically inconsistent; in yet another, modalities are weakly aligned in space and require geometric correction before any meaningful fusion can occur. MDAF therefore denotes architectures in which at least two such mismatch regimes are handled explicitly, after which a fusion stage operates on better-conditioned representations.

This interpretation is especially direct in DashFusion, whose description states that it is “very close in spirit to what one would call a Multimodal Dual-Alignment Fusion framework” because it separates temporal alignment and semantic alignment and then applies a dedicated fusion stage (Wen et al., 5 Dec 2025). Across the broader literature, similar structures recur even when different terminology is used: “dual alignment,” “dual-stage fusion,” “alignment-aware fusion,” “training-time fusion,” or “progressive alignment and fusion.”

2. Canonical formulation: DashFusion as a direct exemplar

DashFusion provides the clearest explicit formulation of MDAF in multimodal sentiment analysis (Wen et al., 5 Dec 2025). Its setting is a video utterance decomposed into text, audio, and visual streams with modality-specific features

$$X_m \in \mathbb{R}^{T_m \times d_m}, \qquad m \in \{t,v,a\},$$

where differing sequence lengths $T_m$ and feature dimensions $d_m$ encode the usual heterogeneity of text, speech, and video. The architecture has four stages: modality encoding, dual-stream alignment, supervised contrastive refinement, and hierarchical bottleneck fusion, followed by prediction.

The first alignment stream is temporal alignment. DashFusion uses text as the anchor modality and aligns audio-to-text and vision-to-text by cross-modal attention. Text tokens query non-linguistic sequences, producing a text-length-aligned multimodal representation

$$\begin{aligned} H &= X_t + X_{a\to t} + X_{v\to t} \ &= X_t + \operatorname{CA}(X_t,X_a) + \operatorname{CA}(X_t,X_v). \end{aligned}$$

This is a text-centric, pairwise alignment strategy rather than a single tri-modal operator. Its purpose is sequence synchronization: audio and vision are aligned to textual time steps without requiring equal sequence lengths.

The second stream is semantic alignment. Global unimodal features $\bar X_m$ are formed by average pooling over time, and a contrastive NT-Xent-style objective aligns text-audio and text-vision pairs from the same sample while separating mismatched pairs. In effect, the model imposes representation-space consistency after temporal correspondence has been established. DashFusion then adds supervised contrastive learning over both unimodal features and the temporally aligned multimodal feature $H$, using label-aware positives and hard negatives. The overall training objective is

$$\mathcal{L}_{\mathrm{all}}=\mathcal{L}_{\mathrm{pred}}+\lambda \mathcal{L}_{\mathrm{con}},$$

with $\lambda=0.2$ and $\tau=0.5$ in the reported experiments (Wen et al., 5 Dec 2025).

Fusion is then handled by hierarchical bottleneck fusion rather than by reusing the alignment mechanism itself. If the bottleneck has initial token count $p$ and the fusion stack has $T_m$0 layers, the bottleneck width shrinks as $T_m$1 at layer $T_m$2. The bottleneck is initialized from the current multimodal state,

$T_m$3

gathers information from each modality, and then writes information back to each modality through cross-modal attention. This yields a two-part MDAF sequence: align across time, align across semantics, then fuse through a compressed communication substrate (Wen et al., 5 Dec 2025).

DashFusion is also instructive because its dual alignment is explicitly asymmetric. Text acts as the anchor for both temporal and semantic alignment. This suggests that many practical MDAF systems are not symmetric co-encoders; they often rely on one modality as the organizing reference.

3. Major realizations of dual alignment

Across recent work, the “dual” in MDAF is instantiated in several non-equivalent ways. The table summarizes representative forms.

Method	Alignment axes	Fusion stage
DashFusion (Wen et al., 5 Dec 2025)	Temporal + semantic	Hierarchical bottleneck fusion
AlignMamba-2 (Li et al., 19 Mar 2026)	OT local geometric + MMD global statistical	Modality-Aware Mamba
CoDAF (Zongzhen et al., 20 Jun 2025)	Spatial offset-guided + shared semantic space	Dynamic gating + spatial-channel dual attention
DualAlign (Zhou et al., 8 Jul 2026)	Visual-visual + visual-text staged GRAM alignment	Two-stage enhancement and fusion
DADM (Yang et al., 1 Mar 2025)	Intra-domain modality + inter-domain domain alignment	MI-mask interaction and domain-aligned fusion
ConFu (Koutoupis et al., 26 Nov 2025)	Pairwise unimodal + fused-pair-to-third alignment	Lightweight MLP fusion

The best-known temporal-semantic instance is DashFusion. A different formulation appears in AlignMamba-2, which regularizes unimodal latent sequences using Optimal Transport for local geometric or token-level alignment and Maximum Mean Discrepancy for global statistical alignment, then concatenates the aligned streams and fuses them with a Modality-Aware Mamba layer (Li et al., 19 Mar 2026). The combined alignment loss is

$T_m$4

with reported best values $T_m$5 and $T_m$6 in the hyperparameter analysis (Li et al., 19 Mar 2026).

A spatial-semantic realization appears in CoDAF for weakly aligned UAV RGB-IR detection. Its Offset-guided Semantic Alignment module first predicts attention-guided offsets from an IR-weighted RGB-IR representation, then uses deformable convolution with a base offset prior $T_m$7 to align RGB features to the IR domain,

$T_m$8

and finally projects aligned RGB and IR features into a shared semantic space with an InfoNCE contrastive loss (Zongzhen et al., 20 Jun 2025). Here dual alignment means spatial correction plus shared-space semantic alignment.

DualAlign for Action Quality Assessment realizes dual alignment as an explicitly staged process. Stage 1 aligns RGB video, optical flow, and skeleton with a visual alignment loss $T_m$9; stage 2 aligns the stabilized visual representation with grounded text using $d_m$0 (Zhou et al., 8 Jul 2026). The model is explicit that text is incorporated only after visual stabilization, because premature semantic fusion can distort the visual manifold.

DADM uses a different duality: modality alignment within a domain and domain alignment across modalities. Its Mutual Information Mask module adaptively enhances favorable modalities and suppresses unfavorable ones, while PG-IRM and modality angle-margin losses align sub-domain hyperplanes and cross-domain modality geometry (Yang et al., 1 Mar 2025). ConFu generalizes the concept further by aligning both unimodal pairs and fused-pair-to-third-modality embeddings in a shared space, thereby extending pairwise alignment to higher-order multimodal structure (Koutoupis et al., 26 Nov 2025).

These examples show that MDAF is not one fixed algorithm. It is a family of architectures in which two complementary correspondence problems are isolated and solved before or alongside fusion. The alignment pair may be temporal and semantic, local and global, structural and semantic, modality and domain, or pairwise and higher-order. This suggests that the defining property of MDAF is architectural decomposition rather than a single canonical loss.

4. Fusion after alignment

The fusion stage in MDAF is neither incidental nor uniform. Once representations have been aligned, fusion may still need to handle residual modality conflict, computational constraints, or higher-order dependency structure.

DashFusion’s hierarchical bottleneck fusion illustrates fusion as compressed communication rather than full all-token self-attention. On CH-SIMS, the paper reports that Concat+SA requires 324M MAdds, standard bottleneck fusion 162M MAdds, and hierarchical bottleneck fusion 145M MAdds, while HBF attains the best Acc-5 and MAE among those fusion variants in that comparison (Wen et al., 5 Dec 2025). This is a central MDAF lesson: fusion can be made lighter once alignment has reduced heterogeneity.

In weakly aligned detection, fusion often remains reliability-aware. AR-CNN first aligns region features and then performs Confidence-Aware Fusion, in which modality-specific classification margins

$d_m$1

reweight region features, while a disagreement term suppresses contradictory sensed features for unpaired objects (Zhang et al., 2022). CoDAF adopts the same principle in a more modern form: its Modality-Adaptive Gating Network predicts per-pixel RGB and IR weights, and its Dual-Attention Cross-Modulation block refines the fused tensor with channel then spatial attention (Zongzhen et al., 20 Jun 2025). In both cases, alignment and fusion are coupled: fusion is not blind averaging, but a confidence-sensitive operation performed on already aligned features.

Other systems use fusion itself as an additional alignment signal. ConFu creates fused representations $d_m$2 and aligns each fused pair with the remaining modality under

$d_m$3

thereby preserving pairwise correspondence while capturing higher-order dependencies such as XOR-like structure (Koutoupis et al., 26 Nov 2025). ITO goes further by using a lightweight training-time multimodal fusion transformer that is discarded at inference; the paper argues that multiple alignment improves discriminative power, while training-time fusion acts as a structural regularizer that reduces modality gap and stabilizes optimization (Liu et al., 3 Mar 2026).

A complementary design principle is provided by the controlled comparison of cross-attention and concatenation. On Flickr8k with pre-aligned CLIP features, concatenation outperforms cross-attention by 4.1–5.1 percentage points across 2048–16384 samples, and the paper argues that concatenation requires $d_m$4 samples while cross-attention requires $d_m$5 samples to learn its interaction weights (Zhou et al., 31 May 2026). This suggests that in strongly aligned MDAF systems, the fusion stage need not be maximally expressive. If alignment has already removed most approximation burden, simple fusion may dominate on statistical efficiency grounds.

5. Empirical behavior across domains

The empirical literature consistently reports that alignment alone is insufficient and fusion alone is insufficient. Gains come from their combination.

In DashFusion, removing the entire dual-stream alignment on CH-SIMS lowers F1 from 79.39 to 76.37, lowers Acc-5 from 44.24 to 42.01, and worsens MAE from 0.412 to 0.436; removing HBF also causes a large degradation, to F1 77.76, Acc-5 42.67, and MAE 0.431 (Wen et al., 5 Dec 2025). The paper reports strong benchmark performance on three sentiment datasets, including $d_m$6 Acc-2 and $d_m$7 MAE on CMU-MOSI, $d_m$8 Acc-2 and $d_m$9 MAE on CMU-MOSEI, and $$\begin{aligned} H &= X_t + X_{a\to t} + X_{v\to t} \ &= X_t + \operatorname{CA}(X_t,X_a) + \operatorname{CA}(X_t,X_v). \end{aligned}$$0 Acc-2 with $$\begin{aligned} H &= X_t + X_{a\to t} + X_{v\to t} \ &= X_t + \operatorname{CA}(X_t,X_a) + \operatorname{CA}(X_t,X_v). \end{aligned}$$1 MAE on CH-SIMS (Wen et al., 5 Dec 2025).

AlignMamba-2 reports that removing the dual alignment loss causes the largest drop across all four evaluated benchmarks: MOSI F1 falls from 87.0 to 84.4, MOSEI F1 from 86.5 to 84.0, NYUDv2 F1 from 71.5 to 69.5, and MVSA F1 from 80.2 to 78.1 (Li et al., 19 Mar 2026). Removing the Modality-Aware Mamba also hurts consistently, indicating that dual alignment and heterogeneity-aware fusion are complementary rather than interchangeable.

SeaDATE offers a parallel result in multimodal object detection. On FLIR, the baseline obtains $$\begin{aligned} H &= X_t + X_{a\to t} + X_{v\to t} \ &= X_t + \operatorname{CA}(X_t,X_a) + \operatorname{CA}(X_t,X_v). \end{aligned}$$2 on $$\begin{aligned} H &= X_t + X_{a\to t} + X_{v\to t} \ &= X_t + \operatorname{CA}(X_t,X_a) + \operatorname{CA}(X_t,X_v). \end{aligned}$$3; adding only DTF reaches $$\begin{aligned} H &= X_t + X_{a\to t} + X_{v\to t} \ &= X_t + \operatorname{CA}(X_t,X_a) + \operatorname{CA}(X_t,X_v). \end{aligned}$$4, adding only contrastive semantic alignment reaches $$\begin{aligned} H &= X_t + X_{a\to t} + X_{v\to t} \ &= X_t + \operatorname{CA}(X_t,X_a) + \operatorname{CA}(X_t,X_v). \end{aligned}$$5, and using both reaches $$\begin{aligned} H &= X_t + X_{a\to t} + X_{v\to t} \ &= X_t + \operatorname{CA}(X_t,X_a) + \operatorname{CA}(X_t,X_v). \end{aligned}$$6 (Dong et al., 2024). The same complementarity appears on LLVIP and M$$\begin{aligned} H &= X_t + X_{a\to t} + X_{v\to t} \ &= X_t + \operatorname{CA}(X_t,X_a) + \operatorname{CA}(X_t,X_v). \end{aligned}$$7FD. CoDAF shows the same pattern under weak alignment: on DroneVehicle, the baseline gives 73.9 [email protected], baseline+OSA 77.8, baseline+full DAFM 77.7, and full CoDAF 78.6 (Zongzhen et al., 20 Jun 2025).

The pattern extends beyond sentiment and detection. DualAlign reports an average correlation improvement of 21.16% on MM-JDM over the state of the art and gains of 3.53% on RG and 5.95% on Fis-V; removing all alignment drops MM-JDM SRCC from 85.44 to 63.82 (Zhou et al., 8 Jul 2026). DADM improves average fixed-modal domain-generalization performance in face anti-spoofing to HTER 13.63 and AUC 92.96, outperforming MMDG’s 19.25 and 87.96, and its ablations show that modality alignment, domain alignment, and MI-guided optimization each contribute materially (Yang et al., 1 Mar 2025). These results suggest that MDAF is not task-specific. The same design logic appears in multimodal sentiment analysis, recommendation, anti-spoofing, action quality assessment, object detection, and graph-based multimodal reasoning.

6. Limitations, boundary conditions, and open questions

Despite its breadth, MDAF is not a settled or uniformly specified paradigm. Several limitations recur.

A first limitation is anchor dependence. Many systems are explicitly asymmetric: DashFusion uses text as the anchor for both temporal and semantic alignment; CoDAF keeps IR as the reference and aligns RGB to IR; DualAlign aligns visual modalities first and only later incorporates text (Wen et al., 5 Dec 2025, Zongzhen et al., 20 Jun 2025, Zhou et al., 8 Jul 2026). This works when one modality is reliably dominant, but it may be less appropriate when no modality is semantically privileged or when the anchor is itself noisy. This suggests that MDAF is often more practical than symmetric, but also more task-dependent.

A second limitation is incomplete coverage of higher-order interaction. Many methods align pairwise streams to an anchor or fuse only after pairwise conditioning. ConFu argues that this can miss higher-order dependencies and responds with fused-pair-to-third-modality alignment, but also acknowledges combinatorial growth as the number of modalities increases (Koutoupis et al., 26 Nov 2025). This indicates a tension between expressive alignment structure and tractable training.

A third limitation concerns computational placement. AlignMamba-2 removes OT and MMD at inference but admits additional training-time overhead (Li et al., 19 Mar 2026). ITO similarly pushes fusion into training only, preserving dual-encoder inference (Liu et al., 3 Mar 2026). By contrast, CoDAF and DashFusion keep alignment and fusion in the deployed model. This suggests that MDAF can be implemented either as an inference-time architecture or as a training-time regularization strategy, and the choice has major efficiency implications.

A fourth limitation is methodological opacity in some recent proposals. ATD frames multimodal fusion as alignment via “zoom” and “shift,” but its exact fusion equations and task losses are not fully specified in the available formulation (Qin, 2024). SeaDATE gives clear DTF and contrastive objectives but leaves some implementation-critical parameters, such as numbers of heads and channel groups, unspecified in the main description (Dong et al., 2024). Such gaps complicate reproducibility and blur the distinction between conceptual MDAF design and fully reproducible engineering practice.

Finally, the relation between alignment quality and fusion complexity remains unresolved in a universal sense. The controlled Flickr8k study argues that when features are already aligned, the approximation-error advantage of cross-attention largely vanishes and simpler fusion may dominate statistically (Zhou et al., 31 May 2026). This suggests a nontrivial design principle: MDAF should not assume that stronger alignment necessitates stronger fusion. In some regimes, the opposite may hold. A plausible implication is that the central research question in MDAF is not merely how to fuse more aggressively, but how to choose the minimal fusion operator consistent with the residual misalignment that remains after dual alignment.

In synthesis, MDAF is best understood as a multimodal design philosophy in which alignment is decomposed into two complementary problems and fusion is treated as a separate, task-conditioned operation on the aligned outputs. Its contemporary realizations differ sharply in the kinds of alignment they pair, the symmetry they assume, the stage at which fusion is applied, and whether fusion is retained at inference. What unifies them is the claim that multimodal heterogeneity is structurally plural, and that robust fusion improves when that plurality is modeled explicitly rather than collapsed into a single undifferentiated attention or concatenation step.