MDR: Multimodal Decomposition Routers

• MDR are specialized routing mechanisms that disentangle shared and modality-specific features for effective multimodal domain adaptation.
• They employ multiple low-rank decomposers with unique and shared gating networks to tailor feature extraction per modality.
• Empirical studies on video benchmarks confirm that MDR improve few-shot adaptation by optimizing decomposer orthogonality and activation consistency.

Multimodal Decomposition Routers (MDR) are a specialized routing mechanism in deep learning systems, principally designed to facilitate fine-grained partitioning and dynamic utilization of shared and modality-unique features in multimodal domain adaptation scenarios. MDR have been introduced in the context of Modality-Collaborative Low-Rank Decomposers (MC-LRD), where their core function is to selectively activate banks of low-rank decomposers for each modality, supporting robust adaptation under significant domain shift with few target domain samples (Wanyan et al., 24 Nov 2025).

1. Motivation and Conceptual Basis

MDR address the need for adaptive cross-modal representation disentanglement in settings where modalities manifest different shifts across domains. In video domain adaptation (e.g., RGB and optical flow), conventional feature fusion or naive domain alignment methods are inadequate because each modality encapsulates both shared information (e.g., object identity) and modality-specific signals (e.g., RGB-texture vs. flow-motion) that are affected differently by domain shifts. MDR enable explicit modeling of these differences by allocating a set of decomposers—each targeting distinct feature components—and dynamically gating their outputs for every sample. This mechanism ensures that domain-invariant shared structure can be aligned aggressively, while modality-unique factors are preserved or aligned with greater flexibility (Wanyan et al., 24 Nov 2025).

2. MDR Architecture and Interaction with MC-LRD

In the MC-LRD framework, MDR are inserted at multiple levels (clip-level and video-level) for each modality. Each modality $m$ (e.g., RGB "r" or flow "o") is associated with a bank of $N$ low-rank decomposers $D_{m,1}, \ldots, D_{m,N}$, parameterized via LoRA-style adaptations of the underlying MLP projector. Immediately following this decomposition, MDR provides three distinct gating (“sub-router”) networks:

• $R^r_u$ (unique router for RGB)
• $R^o_u$ (unique router for flow)
• $R_s$ (shared router for cross-modality alignment)

Each sub-router consists of a fully connected layer applied to temporally aggregated features, emitting softmax-normalized weights across the $N$ decomposers.

Weighted outputs from decomposers are calculated as: $$\begin{aligned} \mathbf{z}^m_u &= \sum_{i=1}^N w^m_{u,i} D_{m,i}(\mathbf{z}^m), \qquad (m\in\{r,o\}) \ \mathbf{z}^m_s &= \sum_{i=1}^N w_{s,i} D_{m,i}(\mathbf{z}^m) \end{aligned}$$ where $w^m_u$ and $w_s$ are the gating vectors from the respective sub-routers, and $\mathbf{z}^m$ are post-self-attention features. These disentangled representations are processed separately by subsequent layers, supporting independent adaptation and alignment of unique and shared subspaces (Wanyan et al., 24 Nov 2025).

3. Mathematical Formulation and Constraints

Core routing weights are computed as: $$\begin{aligned} w^r_u &= \mathsf{softmax}(R^r_u(\mathrm{TAP}(\mathbf{z}^r))) \ w^o_u &= \mathsf{softmax}(R^o_u(\mathrm{TAP}(\mathbf{z}^o))) \ w_s &= \mathsf{softmax}(R_s(\mathrm{TAP}([\mathbf{z}^r;\mathbf{z}^o]))) \end{aligned}$$ where $\mathrm{TAP}(\cdot)$ is temporal average pooling. The orthogonality among decomposers and decorrelation among sub-routers are crucial for effective disentanglement:

• Decomposer output orthogonality is enforced via $L_{dd}$, minimizing pairwise cosine similarity of decomposer activations.
• Sub-router decorrelation is enforced via $L_{rd}$, penalizing alignment between unique and shared routing weights.

Additionally, cross-domain activation consistency ($L_{ac}$) ensures that source and target domain samples of the same class employ similar gating patterns, promoting consistent alignment and mitigating overfitting to domain-unique spurious correlations. The total loss function for joint optimization is: $L = L_{cls} + L_{dd} + L_{rd} + L_{ac} + L_{ada}$ where $L_{cls}$ is the usual classification loss, and $L_{ada}$ is an adversarial domain alignment loss (e.g., via gradient reversal) (Wanyan et al., 24 Nov 2025).

4. Comparison to Related Routing Mechanisms

While MDR belong to the broader class of mixture-of-experts (MoE) routers, several distinguishing features are salient:

• MDR target fine-grained feature disentanglement within each modality, not only between experts but also between the different semantic “types” (shared vs. unique).
• Each MDR contains multiple gating heads ("sub-routers" for shared and modality-unique features), unlike standard MoE routers that jointly parameterize selection for all experts without explicit disentanglement.
• MDR are closely associated with banked low-rank decomposers tailored for modality interaction and domain adaptation, as opposed to generic expert layers.

A plausible implication is that MDR generalize conventional router networks by supporting structured and conditional routing patterns informed by modality composition and domain characteristics (Wanyan et al., 24 Nov 2025).

5. Empirical Impact and Ablation Studies

Experiments on public few-shot video domain adaptation benchmarks substantiate the efficacy of MDR. Removal of decomposer orthogonality ($L_{dd}$), router decorrelation ($L_{rd}$), or activation consistency ($L_{ac}$) individually degrades mean 1-shot accuracy on EPIC-Kitchens by 2.3%, 2.0%, and 1.8%, respectively. Sub-router ablations show that omitting the RGB-unique or shared router pathways results in performance drops of roughly 2.2% and 3.1%, confirming the complementary utility of distinct gating heads.

Unaligned models exhibit higher MMD (maximum mean discrepancy) in modality-unique subspaces relative to shared ones, which supports the design choice to treat these subspaces heterogeneously. Visualization shows that shared subspace weights ($w_s$) tend to concentrate on central decomposers, whereas unique router weights ($w^r_u$, $w^o_u$) polarize toward extremes, illustrating substantial disentanglement. Latent-space t-SNE projections further reveal improved alignment for shared subspace features and retained class separability for unique representations (Wanyan et al., 24 Nov 2025).

6. Application Scenarios and Integration

MDR are integrated as reusable modules for any scenario requiring multimodal domain adaptation—especially settings with heterogeneous modality reliability shifts. The MC-LRD application encompasses both clip-level and video-level feature routing for action recognition tasks. MDR can also be generalized to other multimodal tasks where it is crucial to dynamically modulate the use of transferable versus domain-unique structures.

The gating and regularization framework and the parameter sharing structure in decomposers support scalable extension to more modalities or alternative low-rank adaptation strategies, as required by the complexity of multimodal fusion or the granularity of domain-specific variation (Wanyan et al., 24 Nov 2025).

7. Significance in the Context of Multimodal Adaptation

MDR address key limitations in existing multimodal domain adaptation frameworks by jointly optimizing for granularity-aware decomposer selection, subspace orthogonality, and domain-consistent activation. Together, these properties facilitate a robust interplay between invariant and variant features, supporting the transferability and discrimination required by few-shot adaptation under complex modal interaction and domain shift.

The results establish MDR within MC-LRD as an empirically validated approach for orchestrating flexible, structured multimodal adaptation, yielding substantial improvements over prior art in the video domain adaptation setting (Wanyan et al., 24 Nov 2025).