AMMNet: Asymmetric Multi-Modal Architectures

• AMMNet is a design paradigm for multimodal architectures where modalities are assigned unequal roles, computational capacities, and interaction directions.
• It leverages directional interaction, capacity asymmetry, and representation-space asymmetry to optimize cross-modal fusion in applications like remote sensing, medical imaging, and video retrieval.
• Empirical evaluations show that asymmetric designs can reduce model parameters and improve accuracy, though they often require additional mechanisms to ensure robust modal alignment.

Asymmetric Multi-Modal Network (AMMNet) denotes a family of multimodal architectures in which different modalities are assigned unequal representational roles, computational capacities, or interaction directions, rather than being processed as symmetric peers. In recent arXiv literature, the name is used explicitly for remote sensing semantic segmentation (Ye et al., 22 Jul 2025), while closely related asymmetric designs appear under other names in multi-label skin lesion classification (Tang et al., 2024), semantic segmentation and image translation (Wang et al., 2021), Alzheimer’s disease prognosis (Ming et al., 9 Jul 2025), video moment retrieval (Panta et al., 2023), and infrared–visible image fusion (Huang et al., 2024). The acronym is also overloaded: in semantic scene completion, AMMNet refers to “Adversarial Modality Modulation Network,” not “Asymmetric Multi-Modal Network,” although its fusion pathway is likewise asymmetric (Wang et al., 2024).

1. Terminology and naming

The designation is not standardized across the literature. Some works use “AMMNet” directly, others describe the same design logic through task-specific names, and at least one paper assigns the acronym a different expansion.

Paper	Name used in paper	Asymmetry emphasis
(Ye et al., 22 Jul 2025)	AMMNet	RGB–DSM asymmetric dual encoder, prior-guided fusion, distribution alignment
(Tang et al., 2024)	AMMFM	Light CI branch, heavy DI branch, one-way clinical-to-dermoscopy attention
(Wang et al., 2021)	AsymFusion	Shared encoder with private BN, bidirectional multi-layer fusion via asymmetric operators
(Wang et al., 2024)	AMMNet	TSDF-to-RGB modulation and adversarial gradient competition

In the explicit remote-sensing formulation, AMMNet is defined as an architecture for RGB–DSM semantic segmentation with three modules: Asymmetric Dual Encoder (ADE), Asymmetric Prior Fuser (APF), and Distribution Alignment (DA) (Ye et al., 22 Jul 2025). In the skin-lesion setting, the corresponding formulation is called “Asymmetrical Multi-Modal Fusion Method” (AMMFM), comprising an Asymmetric Fusion Framework (AFF) and an Asymmetric Attention Block (AAB) (Tang et al., 2024). In RGB-D segmentation and multimodal translation, the relevant term is “AsymFusion,” centered on a shared encoder with modality-specific BatchNorm and two parameter-free asymmetric fusion operators, channel shuffle and pixel shift (Wang et al., 2021). In Alzheimer’s disease prognosis, the paper names the model ACMCA, built around asymmetric cross-modal cross-attention (Ming et al., 9 Jul 2025).

A common misconception is that AMMNet designates a single canonical blueprint. The literature instead shows a broader architectural principle: asymmetry may be implemented through encoder depth, fusion direction, cross-scale pairing, latent distribution alignment, or even gradient coupling.

2. Architectural principles of asymmetry

A first recurring principle is capacity asymmetry. In the SPC skin-lesion setting, the clinical-image branch uses MobileNetV3, while the dermoscopy branch uses a heavier backbone such as Swin Transformer by default, with ResNet-50 and ConvNeXt-Tiny also evaluated; the stated rationale is that dermoscopy is the primary modality and clinical imagery is supplementary (Tang et al., 2024). In remote sensing, ADE assigns a deeper encoder to RGB imagery and a lighter encoder to DSM, with the best configuration reported as RGB using Swin-Base and DSM using Swin-Small (Ye et al., 22 Jul 2025). This suggests that asymmetric capacity allocation is intended not merely as compression, but as an explicit encoding of modality-specific informativeness.

A second principle is directional interaction asymmetry. In ACMCA for multi-omic Alzheimer’s prognosis, clinical and genetic embeddings serve as queries, while MRI and PET serve as keys and values; the two reported branches are clinical→MRI and genetic→PET, with no reverse imaging→structured attention (Ming et al., 9 Jul 2025). In AMMFM, the AAB computes attention only from clinical features to refine dermoscopy features, rather than using mutual attention in both directions (Tang et al., 2024). In video moment retrieval, the asymmetric co-attention block primarily generates visual-aware text features through repeated text-to-video attention over a long visual stream, motivated by the length imbalance $M \ll N$ between text and video sequences (Panta et al., 2023). In semantic scene completion, asymmetry appears as one-way TSDF→RGB modulation, so that TSDF-derived 3D features recalibrate RGB-derived 3D features and later decoder states (Wang et al., 2024).

A third principle is representation-space asymmetry. MMA-UNet for infrared–visible image fusion argues that visible and infrared features do not reach deep semantic space at the same rate; its cross-scale design therefore fuses the first four layers of VI-UNet with the last four layers of IR-UNet, rather than coupling same-depth features (Huang et al., 2024). AsymFusion adopts a different route: it keeps a single shared encoder but introduces asymmetric channel shuffle and pixel shift so that bidirectional fusion does not collapse into redundant representations (Wang et al., 2021). A plausible implication is that “asymmetry” in this literature is less a single module type than a constraint on how modality roles are specified.

3. Representative instantiations across research domains

In medical imaging, asymmetric multimodal design has been used in both lesion analysis and neurodegenerative disease modeling. AMMFM operates on paired clinical images and dermoscopy images from the Seven-Point Checklist dataset, with eight concurrent classification tasks: one diagnosis task with five classes and seven visual-criteria tasks (PN, STR, PIG, RS, DaG, BWV, VS) (Tang et al., 2024). ACMCA integrates PET, MRI, genetic SNP data, and seven clinical variables—gender, age, MoCA, MMSE, CDR, FAQ, and GDS—after projecting all four modalities to 100-dimensional embeddings (Ming et al., 9 Jul 2025). In both cases, the asymmetric assumption is clinically motivated: one modality is treated as primary and the others as structurally complementary.

In dense prediction, AMMNet for remote sensing semantic segmentation addresses RGB–DSM fusion on the ISPRS Vaihingen and Potsdam datasets using ADE, APF, and DA, followed by a decoder adapted from UNetFormer (Ye et al., 22 Jul 2025). MMA-UNet addresses infrared–visible image fusion with two separately trained modality-specific encoders, VI-guided SE attention for IR encoder training, and cross-scale fusion before a PixelShuffle-based decoder (Huang et al., 2024). In semantic scene completion from single-view RGB-D, the Adversarial Modality Modulation Network lifts 2D RGB features into 3D, combines them with TSDF features through one-way modulation, and adds adversarial supervision from geometry- and semantics-perturbed voxel volumes (Wang et al., 2024). These systems share the view that symmetric fusion can obscure differences in modality geometry, semantics, or noise characteristics.

In sequence modeling, asymmetry has been used to address severe modality imbalance. The video moment retrieval model of (Panta et al., 2023) pairs an asymmetric co-attention network with cross-modal momentum contrastive learning, using C3D video features, GloVe text embeddings, proposal ranking, and boundary regression. In RGB-D segmentation and multimodal image translation, AsymFusion instantiates asymmetry through modality-private BatchNorm, multi-layer bidirectional fusion, channel shuffle, and pixel shift, and is evaluated on NYUDv2, Cityscapes, and Taskonomy (Wang et al., 2021). Together, these works show that AMMNet-like design extends beyond paired image fusion into joint reasoning over spatial, temporal, structured, and volumetric modalities.

4. Mathematical operators and training objectives

A central mechanism in asymmetric multimodal models is directional cross-attention. In AMMFM, if $C \in \mathbb{R}^{H \times W \times C}$ denotes clinical features and $D \in \mathbb{R}^{H \times W \times C}$ dermoscopy features, the attention map is defined as

$M_c = \text{softmax}(C_q \cdot C_k^T),$

and the refined dermoscopy feature is

$D_{\text{refined}} = D_v \cdot M_c + D.$

Training uses

$$L_{\text{total}} = L_{\text{derm}} + L_{\text{clic}} + L_{\text{fusion}},$$

and test-time prediction is

$P_{FI} = W_D P_D + W_C P_C + W_{FU} P_{FU}.$

This combination couples feature-level interaction with late prediction fusion (Tang et al., 2024).

ACMCA formalizes asymmetry with structured queries and imaging keys/values. With 100-dimensional embeddings $C$, $G$, $M$, and $C \in \mathbb{R}^{H \times W \times C}$0 for clinical, genetic, MRI, and PET, the reported cross-modal attention branches are

$C \in \mathbb{R}^{H \times W \times C}$1

followed by

$C \in \mathbb{R}^{H \times W \times C}$2

The classifier is trained with standard cross-entropy over the three classes $C \in \mathbb{R}^{H \times W \times C}$3 (Ming et al., 9 Jul 2025).

Other AMMNet-like systems replace attention with modulation, reconstruction, or alignment losses. In semantic scene completion, the key modulation operator is

$C \in \mathbb{R}^{H \times W \times C}$4

where both $C \in \mathbb{R}^{H \times W \times C}$5 and $C \in \mathbb{R}^{H \times W \times C}$6 are generated from TSDF features, and the overall objective is

$C \in \mathbb{R}^{H \times W \times C}$7

The adversarial term is defined through a minimax game involving real voxel labels, generated outputs, geometry-perturbed labels, and semantics-perturbed labels (Wang et al., 2024). In MMA-UNet, the fusion loss is

$C \in \mathbb{R}^{H \times W \times C}$8

with $C \in \mathbb{R}^{H \times W \times C}$9 and $D \in \mathbb{R}^{H \times W \times C}$0 (Huang et al., 2024). In remote sensing AMMNet, the total objective is a supervised segmentation loss plus $D \in \mathbb{R}^{H \times W \times C}$1, where DA minimizes an asymmetric KL-like divergence between DSM and RGB latent distributions (Ye et al., 22 Jul 2025).

The broader pattern is consistent: asymmetric models rarely rely on a single fusion rule. They typically combine a directional interaction mechanism with a task loss and an auxiliary constraint that stabilizes cross-modal compatibility, such as late prediction weighting, latent distribution alignment, reconstruction-based regularization, or adversarial competition.

5. Reported empirical behavior and efficiency

The published results show that asymmetric designs are often motivated as accuracy–efficiency trade-offs, not only as fusion refinements.

System	Reported performance	Efficiency note
AMMFM on SPC (Tang et al., 2024)	AVG AUC 88.1%, AVG ACC 77.2%	33.06M params
AsymFusion on NYUDv2 (Wang et al., 2021)	IoU 50.8% with ResNet-101	118.2M params, about +0.1M over RefineNet
ACMCA on ADNI overlap (Ming et al., 9 Jul 2025)	Test accuracy 94.88%, F1-score 0.855, AUC 0.948	No FLOPs or latency reported
Video AMMNet on TACoS (Panta et al., 2023)	R@1, [email protected] = 49.77; R@1, [email protected] = 37.99	22M trainable params vs 36M for MSAT
MMA-UNet downstream segmentation (Huang et al., 2024)	mIoU = 74.12	Params and FLOPs not reported
Remote-sensing AMMNet on Vaihingen (Ye et al., 22 Jul 2025)	mOA 93.52, mF1 93.27, mIoU 87.56	28.82 G FLOPs, 151.26 M params, 1026 MB
Adversarial Modality Modulation Network on NYUCAD (Wang et al., 2024)	SC-IoU 83.3%, SSC-mIoU 67.2%	Model size and runtime not reported

In skin-lesion classification, asymmetry is explicitly tied to parameter savings. Replacing a symmetric Swin-T/Swin-T design with MobileNetV3 for clinical images and Swin-T for dermoscopy reduces parameters from 58.49M to 32.48M with negligible accuracy change, and the full AMMFM reaches 33.06M parameters with AVG AUC 88.1% and AVG ACC 77.2% (Tang et al., 2024). In RGB-D segmentation and translation, AsymFusion reports 50.8% IoU on NYUDv2 with ResNet-101 at 118.2M parameters, compared with RDFNet’s 49.1% IoU at 366.7M parameters, and 82.1% IoU with 43.52M parameters on Cityscapes (Wang et al., 2021).

In remote sensing, AMMNet on Vaihingen reports mIoU 87.56, mF1 93.27, and mOA 93.52, while using 28.82 G FLOPs, 151.26 M parameters, and 1026 MB; the strongest cited baseline FTransUNet reports 45.21 G FLOPs, 160.88 M parameters, 3463 MB, and mIoU 84.23 (Ye et al., 22 Jul 2025). In Alzheimer’s disease prognosis, ACMCA reports test accuracy 94.88% on the overlapping ADNI cohort and outperforms both symmetric co-attention and concatenation baselines in the reported comparisons (Ming et al., 9 Jul 2025). In video moment retrieval, the full model achieves the best TACoS scores reported in that paper while reducing trainable parameters from 36M to 22M relative to MSAT, though gains on ActivityNet are described as comparable rather than uniformly superior (Panta et al., 2023).

Taken together, these results suggest that asymmetry is most effective when it matches a genuine imbalance in modality informativeness, sequence length, or structural reliability. The literature does not support the simpler claim that asymmetry is universally superior irrespective of task or modality pair.

6. Limitations, misconceptions, and open problems

A common misconception is that asymmetry automatically solves multimodal misalignment. Several papers report the opposite: asymmetry improves results only when paired with an additional compatibility mechanism. In remote sensing AMMNet, DA alone can hurt if APF is absent, and its benefit depends on prior-guided fusion (Ye et al., 22 Jul 2025). In video moment retrieval, asymmetric co-attention or contrastive learning alone does not consistently beat the MSAT baseline; the paper attributes the strongest results to their combination (Panta et al., 2023). This suggests that asymmetric routing by itself is often insufficient.

Another recurring limitation is restricted robustness evaluation. AMMFM does not report cross-dataset or clinic tests, does not analyze image-quality sensitivity, and does not explicitly evaluate missing-modality scenarios (Tang et al., 2024). ACMCA is trained and evaluated on the overlapping ADNI subset with complete records for all four modalities and does not address missing or noisy modalities, cross-cohort generalization, or longitudinal modeling (Ming et al., 9 Jul 2025). AsymFusion assumes pixel-aligned multimodal inputs (Wang et al., 2021), MMA-UNet expects well-registered infrared–visible pairs (Huang et al., 2024), and remote-sensing AMMNet is evaluated on co-registered ISPRS RGB/IRRG–DSM pairs (Ye et al., 22 Jul 2025). A plausible implication is that many published AMMNet-style systems remain sensitive to sensor registration quality and modality availability.

Interpretability also remains limited. AMMFM does not provide qualitative attention visualizations (Tang et al., 2024), ACMCA does not provide attention maps, SHAP, or Grad-CAM analyses (Ming et al., 9 Jul 2025), and MMA-UNet’s asymmetric layer pairing is manually selected from CKA analysis rather than learned adaptively (Huang et al., 2024). In semantic scene completion, AMMNet’s adversarial mechanism improves SC-IoU and SSC-mIoU, but the paper does not report model size, FLOPs, or runtime (Wang et al., 2024). These gaps matter because asymmetric multimodal systems often justify their design through efficiency, yet efficiency is not always profiled comprehensively.

The main open problem is therefore not whether asymmetry is useful, but how it should be specified. Current work explores at least five non-equivalent answers: unequal encoder depth, one-way attention, asymmetric fusion operators, cross-scale layer matching, and gradient-level modulation. The literature indicates that AMMNet is best understood as a design paradigm for modality-aware allocation and interaction, rather than a single architecture with fixed modules or a settled acronym.