Modality-Aware Adaptive Encoder (M2AE)

• The paper introduces an M2AE design that conditions feature extraction on modality identity, availability, and reliability to handle missing or degraded inputs.
• It employs intra-modal processing, cross-modal interaction, and selective fusion techniques to dynamically adjust computation based on scene-dependent utility.
• Empirical results show that this adaptive encoder significantly outperforms fixed fusion models in tasks like arbitrary-modality segmentation and unified change detection.

Modality-Aware Adaptive Encoder (M2AE) denotes an encoder design paradigm in which feature extraction and cross-modal information exchange are conditioned on modality identity, modality availability, signal reliability, or domain metadata, rather than being governed by a single fixed fusion rule. In the recent literature considered here, the term is interpretive rather than official: neither CrossWeaver nor UniRoute explicitly introduces the name “M2AE,” yet both instantiate encoder-side adaptation that changes computation as a function of available modalities and scene-dependent utility. CrossWeaver does so for arbitrary-modality semantic segmentation through reliability-aware cross-modal interaction within a shared hierarchical Transformer encoder, while UniRoute does so for remote sensing change detection through conditional routing over receptive fields and differencing operators in a unified encoder–decoder framework (Zhang et al., 3 Apr 2026, Shu et al., 21 Jan 2026).

1. Definition and problem setting

An M2AE-style architecture addresses a recurrent limitation of multimodal models: static backbones and fixed fusion pathways assume that all modalities are present, equally informative, and comparably reliable. The papers considered here challenge that assumption in two distinct but related settings. CrossWeaver studies arbitrary-modality semantic segmentation, where the model is trained in a multimodal setting but must remain functional when only an arbitrary subset of modalities is available at inference. Its input is a synchronized set of $m$ sensing modalities,

$$\mathcal{I} = \{ I_i \in \mathbb{R}^{H \times W \times C_i} \mid i = 1,2,\dots,m \},$$

with heterogeneous sensors including RGB, depth, event, LiDAR, NIR, AoLP, and DoLP. UniRoute studies unified remote sensing change detection across both homogeneous and heterogeneous temporal pairs, especially Optical–Optical, Optical–UAV, and Optical–SAR, under a single deployment model rather than a specialist-per-pair regime (Zhang et al., 3 Apr 2026, Shu et al., 21 Jan 2026).

The motivating failure mode is similar in both works. CrossWeaver argues that mapping-based aggregation, prompt-based adaptation, exchange-based interaction, and stitch-based fusion tend toward either fixed fusion or indiscriminate information passing, which becomes brittle when a modality is degraded, missing, spatially misaligned, asynchronous, or semantically inconsistent with others. UniRoute identifies two additional conflicts in change detection: the receptive-field conflict between local boundary preservation and global semantic aggregation, and the operator conflict between subtraction-style differencing that works in aligned homogeneous imagery and more robust alternatives required in heterogeneous or geometrically distorted settings. In both cases, the central claim is that adaptation should occur inside the encoder, not only in a late fusion head (Zhang et al., 3 Apr 2026, Shu et al., 21 Jan 2026).

This yields a practical working definition of M2AE: an encoder is modality-aware when it estimates or conditions on modality-specific properties, and it is adaptive when its internal computation changes with modality subset, data quality, local content, or domain metadata. This definition is directly supported by CrossWeaver’s reliability-aware message passing and UniRoute’s domain-conditioned MoE routing.

2. Structural patterns of M2AE-like systems

Despite addressing different tasks, the two frameworks reveal a common design grammar. Both retain a largely shared backbone, then inject conditional computation into selected internal locations rather than constructing a bank of fully separate modality-specific encoders. Both also separate intra-stream strengthening from inter-stream adaptation: CrossWeaver performs per-modality self-attention before exchange, while UniRoute retains a shared ResNet-50 backbone and replaces deeper blocks with routed experts. Finally, both treat “adaptation” as selective computation rather than indiscriminate fusion.

Aspect	CrossWeaver	UniRoute
Task	Arbitrary-modality semantic segmentation	Unified remote sensing change detection
Shared core	SegFormer-style hierarchical Transformer with $S=4$ stages	ResNet-50 pretrained on ImageNet
Adaptive mechanism	Modality Interaction Block (MIB) with reliability-aware cross-modal interaction	AR$^2$-MoE receptive-field routing and MDR-MoE difference routing
Conditioning signal	Modality reliability, token confidence, semantic consistency, scale	Domain code $\mathbf{z}$, local content, temporal feature pair
Fusion path	Seam-Aligned Fusion (SAF) after stage-wise interaction	Decoder-side stage-wise differencing via MDR-MoE
Variable modality support	Arbitrary subsets at inference	Unified handling of multiple modality pairings

The comparison clarifies a common misconception: M2AE need not imply explicit modality tokens, dynamically generated network weights, or fully separate per-modality branches. CrossWeaver explicitly does not use modality tokens or modality embeddings; its modality awareness is expressed through reliability estimation, source-conditioned interaction, and stage-wise weighted fusion. UniRoute likewise keeps unified model weights and a shared architecture, but conditions expert routing and DSBN on a domain code $\mathbf{z}$. This suggests that M2AE is better understood as a functional property of the encoder than as one specific parameterization (Zhang et al., 3 Apr 2026, Shu et al., 21 Jan 2026).

A second misconception is that encoder adaptivity is equivalent to late fusion flexibility. The two papers separate these notions. CrossWeaver places the adaptive machinery inside each Transformer stage and then still applies SAF afterward to repair seams and preserve local coherence. UniRoute adapts both encoding and differencing, arguing that encoder adaptation alone is insufficient when the temporal comparison operator itself is modality-dependent.

3. CrossWeaver as an interaction-driven M2AE

CrossWeaver is a shared-encoder multimodal segmentation pipeline built on a SegFormer-style hierarchical Transformer backbone with $S=4$ stages. Each modality is processed through the same weight-shared encoder, producing stage features

$$Z_i^{(l)} \in \mathbb{R}^{B \times N_l \times C_l}, \quad N_l = H_l W_l.$$

This shared design is reported to improve parameter efficiency, encourage semantic alignment across modalities, and support arbitrary subsets without maintaining a combinatorial family of branches. Adaptivity is introduced by inserting a Modality Interaction Block (MIB) after both the self-attention and FFN components at every stage. After interaction, the enhanced features $\{\tilde{Z}_i^{(l)}\}_{i=1}^m$ are merged by Seam-Aligned Fusion (SAF),

$$F^{(l)} = \mathrm{SAF}\big(\{\tilde{Z}_i^{(l)}\}_{i=1}^{m}\big) \in \mathbb{R}^{B \times N_l \times C_l},$$

and the set $$\mathcal{I} = \{ I_i \in \mathbb{R}^{H \times W \times C_i} \mid i = 1,2,\dots,m \},$$0 is fed to a lightweight decoder $$\mathcal{I} = \{ I_i \in \mathbb{R}^{H \times W \times C_i} \mid i = 1,2,\dots,m \},$$1 to predict segmentation logits,

$$\mathcal{I} = \{ I_i \in \mathbb{R}^{H \times W \times C_i} \mid i = 1,2,\dots,m \},$$2

The reported operating mode is subset-invariant: the encoder processes whichever streams are present, MIB exchanges messages only among available modalities, and SAF fuses only the available enhanced features (Zhang et al., 3 Apr 2026).

MIB is the principal reason CrossWeaver can be interpreted as an M2AE-like encoder. At stage $$\mathcal{I} = \{ I_i \in \mathbb{R}^{H \times W \times C_i} \mid i = 1,2,\dots,m \},$$3, it begins with intra-modal self-attention,

$$\mathcal{I} = \{ I_i \in \mathbb{R}^{H \times W \times C_i} \mid i = 1,2,\dots,m \},$$4

preserving modality-specific structure before any exchange. It then extracts a global descriptor,

$$\mathcal{I} = \{ I_i \in \mathbb{R}^{H \times W \times C_i} \mid i = 1,2,\dots,m \},$$5

and predicts two modality-aware signals,

$$\mathcal{I} = \{ I_i \in \mathbb{R}^{H \times W \times C_i} \mid i = 1,2,\dots,m \},$$6

Here $$\mathcal{I} = \{ I_i \in \mathbb{R}^{H \times W \times C_i} \mid i = 1,2,\dots,m \},$$7 is a modality-level reliability weight normalized across modalities, and $$\mathcal{I} = \{ I_i \in \mathbb{R}^{H \times W \times C_i} \mid i = 1,2,\dots,m \},$$8 is a token coverage ratio controlling how many source tokens from modality $$\mathcal{I} = \{ I_i \in \mathbb{R}^{H \times W \times C_i} \mid i = 1,2,\dots,m \},$$9 should be retained. In parallel, token-wise confidence is estimated as

$S=4$0

followed by a soft Top-$S=4$1 mask,

$S=4$2

The calibrated source features are then

$S=4$3

This sequence makes modality quality explicit at both global and token scales.

Cross-modal interaction is pairwise and multi-scale. For an ordered pair $S=4$4, target modality $S=4$5 supplies queries from $S=4$6, while source modality $S=4$7 supplies keys and values from $S=4$8. With pooled source tokens over multiple spatial grids, each scale computes

$S=4$9

where $^2$0 is a Gaussian relative positional bias. The scale responses are aggregated as

$^2$1

A consistency filter then suppresses semantically incompatible messages,

$^2$2

with cosine similarity between L2-normalized features. Cross-modal updates are integrated residually,

$^2$3

followed by FFN refinement and a milder source-conditioned affine adaptation,

$^2$4

$^2$5

The residual coefficient of $^2$6 is initialized to $^2$7, preserving near-identity behavior at the start of training.

SAF is not merely a projection head after MIB. Given $^2$8, it first concatenates tokens, linearly projects them back to channel dimension, and applies spatial mixing: $^2$9 The $\mathbf{z}$0 operator is described as multi-branch depthwise convolutional mixing, such as $\mathbf{z}$1, $\mathbf{z}$2, and $\mathbf{z}$3, followed by channel attention. SAF then adds a modality-weighted residual,

$\mathbf{z}$4

This produces a fused map while preserving modality-specific cues.

CrossWeaver is trained with pixel-wise cross-entropy only. The paper does not introduce auxiliary reliability supervision, consistency losses, distillation, reconstruction, or regularization terms beyond standard optimization. This is notable because the reliability weights $\mathbf{z}$5, coverage ratios $\mathbf{z}$6, token confidence maps $\mathbf{z}$7, and consistency filtering are learned end-to-end from segmentation supervision alone. A plausible implication is that the method’s arbitrary-modality robustness is intended to emerge primarily from architecture rather than from explicit subset-sampling curricula (Zhang et al., 3 Apr 2026).

4. UniRoute as a routing-based M2AE

UniRoute addresses remote sensing change detection under a unified deployment setting in which a single model must handle Optical–Optical, Optical–UAV, and Optical–SAR pairs. Its main architecture is an encoder–decoder with Mixture-of-Experts routing in both the encoder/backbone and the difference/fusion pathway. The backbone is ResNet-50 pretrained on ImageNet, producing stage features

$\mathbf{z}$8

The adaptive encoder component is AR$\mathbf{z}$9-MoE, inserted in deeper stages, with the reported best accuracy-efficiency trade-off obtained by placing it only at Stages 2–4. The adaptive differencing component is MDR-MoE, and training is stabilized by Consistency-Aware Self-Distillation (CASD) and DSBN (Shu et al., 21 Jan 2026).

AR$\mathbf{z}$0-MoE reframes receptive-field selection as conditional routing. It contains two experts. The Local-Detail Expert is

$\mathbf{z}$1

where $\mathbf{z}$2 is a depth-wise separable convolution block with a small kernel size. The Global-Context Expert is

$\mathbf{z}$3

with depth-wise, depth-wise dilated, and point-wise convolutions, and dilation rate $\mathbf{z}$4. Routing is conditioned on both feature content and a domain code: $\mathbf{z}$5 The gate is hardened through the Straight-Through Estimator,

$\mathbf{z}$6

and the output is

$\mathbf{z}$7

This is a direct encoder-side realization of M2AE: per-location expert selection changes with local content and explicit modality/domain metadata.

MDR-MoE extends adaptation beyond the encoder into temporal differencing. UniRoute defines a compact library of differentiable primitives $\mathbf{z}$8, clarified in the paper as subtraction, concatenation, and multiplication. The gate predicts per-location primitive probabilities from the temporal feature pair,

$\mathbf{z}$9

and applies Top-1 hard routing via STE. The printed equation for the routing mask contains typographical errors in the manuscript, but the intended mechanism is clearly one-hot argmax routing in the forward pass with STE in the backward pass. The resulting difference feature is

$S=4$0

This means each spatial location chooses one primitive rather than averaging several noisy candidates.

CASD stabilizes unified routed training. The teacher prediction is formed by averaging predictions from an original and transformed input,

$S=4$1

and the student is matched by

$S=4$2

Routing decisiveness is encouraged by

$S=4$3

and unchanged-region feature consistency by

$S=4$4

The total objective is

$S=4$5

DSBN is also used so that each modality/domain $S=4$6 has separate normalization statistics and affine parameters; the printed DSBN equation is malformed in the paper, but its intended meaning is standard domain-specific normalization.

For M2AE interpretation, AR$S=4$7-MoE is the closest match to an adaptive encoder in the narrow sense, whereas MDR-MoE extends that philosophy into modality-aware differencing. UniRoute therefore functions less as a pure encoder-only proposal than as a unified modality-adaptive encoder-plus-fusion system (Shu et al., 21 Jan 2026).

5. Empirical evidence for encoder adaptivity

CrossWeaver’s strongest evidence for M2AE-like behavior comes from arbitrary-modality evaluation after full-modality training. On MCubeS, it reaches a mean arbitrary-subset mIoU of $S=4$8, compared with $S=4$9 for StitchFusion and $$Z_i^{(l)} \in \mathbb{R}^{B \times N_l \times C_l}, \quad N_l = H_l W_l.$$0 for CMNeXt. Under single-modality inputs it achieves $$Z_i^{(l)} \in \mathbb{R}^{B \times N_l \times C_l}, \quad N_l = H_l W_l.$$1 on RGB, $$Z_i^{(l)} \in \mathbb{R}^{B \times N_l \times C_l}, \quad N_l = H_l W_l.$$2 on AoLP, $$Z_i^{(l)} \in \mathbb{R}^{B \times N_l \times C_l}, \quad N_l = H_l W_l.$$3 on DoLP, and $$Z_i^{(l)} \in \mathbb{R}^{B \times N_l \times C_l}, \quad N_l = H_l W_l.$$4 on NIR, and on partial combinations $$Z_i^{(l)} \in \mathbb{R}^{B \times N_l \times C_l}, \quad N_l = H_l W_l.$$5 for RN and $$Z_i^{(l)} \in \mathbb{R}^{B \times N_l \times C_l}, \quad N_l = H_l W_l.$$6 for RDN. On DeLiVER, the mean arbitrary-subset mIoU is $$Z_i^{(l)} \in \mathbb{R}^{B \times N_l \times C_l}, \quad N_l = H_l W_l.$$7, versus $$Z_i^{(l)} \in \mathbb{R}^{B \times N_l \times C_l}, \quad N_l = H_l W_l.$$8 for StitchFusion and $$Z_i^{(l)} \in \mathbb{R}^{B \times N_l \times C_l}, \quad N_l = H_l W_l.$$9 for CMNeXt; the reported subset scores include $\{\tilde{Z}_i^{(l)}\}_{i=1}^m$0 on D, $\{\tilde{Z}_i^{(l)}\}_{i=1}^m$1 on RD, $\{\tilde{Z}_i^{(l)}\}_{i=1}^m$2 on RDE, $\{\tilde{Z}_i^{(l)}\}_{i=1}^m$3 on RDL, and $\{\tilde{Z}_i^{(l)}\}_{i=1}^m$4 on DEL. Under standard multimodal settings with MiT-B0, CrossWeaver reports $\{\tilde{Z}_i^{(l)}\}_{i=1}^m$5 mIoU for R-A, $\{\tilde{Z}_i^{(l)}\}_{i=1}^m$6 for R-A-D, and $\{\tilde{Z}_i^{(l)}\}_{i=1}^m$7 for R-A-D-N on MCubeS, and $\{\tilde{Z}_i^{(l)}\}_{i=1}^m$8 for R-D, $\{\tilde{Z}_i^{(l)}\}_{i=1}^m$9 for R-D-E, and $$F^{(l)} = \mathrm{SAF}\big(\{\tilde{Z}_i^{(l)}\}_{i=1}^{m}\big) \in \mathbb{R}^{B \times N_l \times C_l},$$0 for R-D-E-L on DeLiVER. Stage ablation shows $$F^{(l)} = \mathrm{SAF}\big(\{\tilde{Z}_i^{(l)}\}_{i=1}^{m}\big) \in \mathbb{R}^{B \times N_l \times C_l},$$1 for Stage1 only, $$F^{(l)} = \mathrm{SAF}\big(\{\tilde{Z}_i^{(l)}\}_{i=1}^{m}\big) \in \mathbb{R}^{B \times N_l \times C_l},$$2 for Stage2, $$F^{(l)} = \mathrm{SAF}\big(\{\tilde{Z}_i^{(l)}\}_{i=1}^{m}\big) \in \mathbb{R}^{B \times N_l \times C_l},$$3 for Stage3, $$F^{(l)} = \mathrm{SAF}\big(\{\tilde{Z}_i^{(l)}\}_{i=1}^{m}\big) \in \mathbb{R}^{B \times N_l \times C_l},$$4 for Stage4, and $$F^{(l)} = \mathrm{SAF}\big(\{\tilde{Z}_i^{(l)}\}_{i=1}^{m}\big) \in \mathbb{R}^{B \times N_l \times C_l},$$5 for Stage1–4, indicating that adaptation is most effective when distributed across the full encoder hierarchy. Architectural ablation further reports $$F^{(l)} = \mathrm{SAF}\big(\{\tilde{Z}_i^{(l)}\}_{i=1}^{m}\big) \in \mathbb{R}^{B \times N_l \times C_l},$$6M parameters and $$F^{(l)} = \mathrm{SAF}\big(\{\tilde{Z}_i^{(l)}\}_{i=1}^{m}\big) \in \mathbb{R}^{B \times N_l \times C_l},$$7 GFLOPs for full CrossWeaver; removing MIB yields $$F^{(l)} = \mathrm{SAF}\big(\{\tilde{Z}_i^{(l)}\}_{i=1}^{m}\big) \in \mathbb{R}^{B \times N_l \times C_l},$$8M, $$F^{(l)} = \mathrm{SAF}\big(\{\tilde{Z}_i^{(l)}\}_{i=1}^{m}\big) \in \mathbb{R}^{B \times N_l \times C_l},$$9 GFLOPs, and $$\mathcal{I} = \{ I_i \in \mathbb{R}^{H \times W \times C_i} \mid i = 1,2,\dots,m \},$$00 mIoU, removing SAF yields $$\mathcal{I} = \{ I_i \in \mathbb{R}^{H \times W \times C_i} \mid i = 1,2,\dots,m \},$$01M, $$\mathcal{I} = \{ I_i \in \mathbb{R}^{H \times W \times C_i} \mid i = 1,2,\dots,m \},$$02 GFLOPs, and $$\mathcal{I} = \{ I_i \in \mathbb{R}^{H \times W \times C_i} \mid i = 1,2,\dots,m \},$$03 mIoU, and removing both yields $$\mathcal{I} = \{ I_i \in \mathbb{R}^{H \times W \times C_i} \mid i = 1,2,\dots,m \},$$04M, $$\mathcal{I} = \{ I_i \in \mathbb{R}^{H \times W \times C_i} \mid i = 1,2,\dots,m \},$$05 GFLOPs, and $$\mathcal{I} = \{ I_i \in \mathbb{R}^{H \times W \times C_i} \mid i = 1,2,\dots,m \},$$06 mIoU. These results identify MIB as the principal adaptive encoder component (Zhang et al., 3 Apr 2026).

UniRoute’s empirical evidence centers on unified deployment across five datasets. The reported results are F1 $$\mathcal{I} = \{ I_i \in \mathbb{R}^{H \times W \times C_i} \mid i = 1,2,\dots,m \},$$07, IoU $$\mathcal{I} = \{ I_i \in \mathbb{R}^{H \times W \times C_i} \mid i = 1,2,\dots,m \},$$08 on LEVIR-CD; F1 $$\mathcal{I} = \{ I_i \in \mathbb{R}^{H \times W \times C_i} \mid i = 1,2,\dots,m \},$$09, IoU $$\mathcal{I} = \{ I_i \in \mathbb{R}^{H \times W \times C_i} \mid i = 1,2,\dots,m \},$$10 on WHU-CD; F1 $$\mathcal{I} = \{ I_i \in \mathbb{R}^{H \times W \times C_i} \mid i = 1,2,\dots,m \},$$11, IoU $$\mathcal{I} = \{ I_i \in \mathbb{R}^{H \times W \times C_i} \mid i = 1,2,\dots,m \},$$12 on HTCD; F1 $$\mathcal{I} = \{ I_i \in \mathbb{R}^{H \times W \times C_i} \mid i = 1,2,\dots,m \},$$13, IoU $$\mathcal{I} = \{ I_i \in \mathbb{R}^{H \times W \times C_i} \mid i = 1,2,\dots,m \},$$14 on MT-Wuhan; and F1 $$\mathcal{I} = \{ I_i \in \mathbb{R}^{H \times W \times C_i} \mid i = 1,2,\dots,m \},$$15, IoU $$\mathcal{I} = \{ I_i \in \mathbb{R}^{H \times W \times C_i} \mid i = 1,2,\dots,m \},$$16 on XiongAn, with average F1 $$\mathcal{I} = \{ I_i \in \mathbb{R}^{H \times W \times C_i} \mid i = 1,2,\dots,m \},$$17. Compared with the best unified baseline, AFENet re-trained under the unified setting, UniRoute improves average F1 by $$\mathcal{I} = \{ I_i \in \mathbb{R}^{H \times W \times C_i} \mid i = 1,2,\dots,m \},$$18. The full model has $$\mathcal{I} = \{ I_i \in \mathbb{R}^{H \times W \times C_i} \mid i = 1,2,\dots,m \},$$19M parameters and $$\mathcal{I} = \{ I_i \in \mathbb{R}^{H \times W \times C_i} \mid i = 1,2,\dots,m \},$$20G FLOPs, compared with an ensemble of specialists at $$\mathcal{I} = \{ I_i \in \mathbb{R}^{H \times W \times C_i} \mid i = 1,2,\dots,m \},$$21M parameters, $$\mathcal{I} = \{ I_i \in \mathbb{R}^{H \times W \times C_i} \mid i = 1,2,\dots,m \},$$22G FLOPs, and average F1 $$\mathcal{I} = \{ I_i \in \mathbb{R}^{H \times W \times C_i} \mid i = 1,2,\dots,m \},$$23. Ablation supports the routing thesis: removing AR$$\mathcal{I} = \{ I_i \in \mathbb{R}^{H \times W \times C_i} \mid i = 1,2,\dots,m \},$$24-MoE gives $$\mathcal{I} = \{ I_i \in \mathbb{R}^{H \times W \times C_i} \mid i = 1,2,\dots,m \},$$25 on LEVIR-CD, $$\mathcal{I} = \{ I_i \in \mathbb{R}^{H \times W \times C_i} \mid i = 1,2,\dots,m \},$$26 on HTCD, and $$\mathcal{I} = \{ I_i \in \mathbb{R}^{H \times W \times C_i} \mid i = 1,2,\dots,m \},$$27 on MT-Wuhan; removing MDR-MoE gives $$\mathcal{I} = \{ I_i \in \mathbb{R}^{H \times W \times C_i} \mid i = 1,2,\dots,m \},$$28, $$\mathcal{I} = \{ I_i \in \mathbb{R}^{H \times W \times C_i} \mid i = 1,2,\dots,m \},$$29, and $$\mathcal{I} = \{ I_i \in \mathbb{R}^{H \times W \times C_i} \mid i = 1,2,\dots,m \},$$30, respectively; without CASD the scores are $$\mathcal{I} = \{ I_i \in \mathbb{R}^{H \times W \times C_i} \mid i = 1,2,\dots,m \},$$31, $$\mathcal{I} = \{ I_i \in \mathbb{R}^{H \times W \times C_i} \mid i = 1,2,\dots,m \},$$32, and $$\mathcal{I} = \{ I_i \in \mathbb{R}^{H \times W \times C_i} \mid i = 1,2,\dots,m \},$$33. For static differencing, “Sub. Only” reaches $$\mathcal{I} = \{ I_i \in \mathbb{R}^{H \times W \times C_i} \mid i = 1,2,\dots,m \},$$34 on LEVIR-CD, $$\mathcal{I} = \{ I_i \in \mathbb{R}^{H \times W \times C_i} \mid i = 1,2,\dots,m \},$$35 on HTCD, and $$\mathcal{I} = \{ I_i \in \mathbb{R}^{H \times W \times C_i} \mid i = 1,2,\dots,m \},$$36 on MT-Wuhan, demonstrating the brittleness of fixed subtraction in Optical–SAR. The routing strategy comparison further reports that soft gating is acceptable but weaker on heterogeneous data, hard routing without STE fails because of gradient blocking, Gumbel-Softmax is competitive, and STE hard routing is best (Shu et al., 21 Jan 2026).

Taken together, these results support two related conclusions. First, encoder adaptation can improve robustness when modality utility varies strongly across scenes or modality pairings. Second, the specific adaptive variable differs by task: CrossWeaver adapts cross-modal exchange and reliability weighting for arbitrary subsets, whereas UniRoute adapts receptive fields and differencing primitives for cross-domain temporal comparison.

6. Conceptual boundaries, misconceptions, and limitations

A precise reading of the two systems discourages several overly narrow definitions of M2AE. It does not require explicit modality-conditioned parameter generation: CrossWeaver uses a shared-weight backbone and adapts message passing through reliability weights, sparse source masks, consistency filters, and token-wise source mixing. It does not require modality tokens or modality embeddings: CrossWeaver is explicitly modality-aware without them. Nor does it require adaptation to be confined strictly to encoding: UniRoute argues that modality conflict persists in the difference/fusion path, so adaptive routing must continue beyond feature extraction (Zhang et al., 3 Apr 2026, Shu et al., 21 Jan 2026).

At the same time, the papers identify concrete limits. CrossWeaver assumes synchronized modalities and sufficient co-registration for token-level interaction, even though it explicitly tries to mitigate residual misalignment and heterogeneity. Its pairwise interaction formulation may scale less gracefully as the number of modalities grows beyond four, though the shared encoder alleviates part of the burden. Reliability is learned only indirectly from segmentation loss, without explicit uncertainty calibration or auxiliary supervision, and some equations in the paper are typeset imperfectly, leaving a few implementation details under-specified. UniRoute depends on a domain code $$\mathcal{I} = \{ I_i \in \mathbb{R}^{H \times W \times C_i} \mid i = 1,2,\dots,m \},$$37 available as domain metadata and on DSBN with modality/domain-specific statistics, so it is not a metadata-free adaptive system. Its expert library is intentionally compact—two receptive-field experts in AR$$\mathcal{I} = \{ I_i \in \mathbb{R}^{H \times W \times C_i} \mid i = 1,2,\dots,m \},$$38-MoE and three primitives in MDR-MoE—which preserves efficiency but may limit expressiveness. The main text also leaves some details underspecified, including decoder architecture, exact primitive formulas, the finalized segmentation loss expression, exact stage resolutions, and the numerical values of $$\mathcal{I} = \{ I_i \in \mathbb{R}^{H \times W \times C_i} \mid i = 1,2,\dots,m \},$$39, $$\mathcal{I} = \{ I_i \in \mathbb{R}^{H \times W \times C_i} \mid i = 1,2,\dots,m \},$$40, and $$\mathcal{I} = \{ I_i \in \mathbb{R}^{H \times W \times C_i} \mid i = 1,2,\dots,m \},$$41 (Zhang et al., 3 Apr 2026, Shu et al., 21 Jan 2026).

These limitations clarify the present status of M2AE as a research concept. The cited literature supports an encoder-centered view in which modality awareness is implemented through conditional computation, reliability estimation, and routed interaction rather than through a single canonical architecture. CrossWeaver represents a lightweight, interaction-driven version of that idea for arbitrary-modality segmentation, while UniRoute represents a routing-based version that extends adaptivity from encoding into differencing and training stabilization. A plausible implication is that “M2AE” is best treated not as a fixed named model, but as a family of encoder designs for subset-robust and domain-sensitive multimodal perception.