Modality-Aware Projection & Routing

• Modality-aware projection and routing is a framework that uses modality-specific transformations to map inputs into dedicated representation spaces, ensuring expert specialization.
• It employs routing mechanisms that dynamically allocate tokens to expert networks based on modality, balancing accuracy with computational efficiency.
• This approach is validated in domains like remote sensing and speech-text modeling, yielding improved performance and notable FLOPs savings.

Modality-aware projection and routing refers to a family of architectural and algorithmic techniques that explicitly leverage knowledge of data modality (e.g., text, images, audio, sensor channels) in the projection of inputs and the routing of intermediate representations to specialized computational experts or operators within large, typically deep, neural models. This approach aims to enforce or exploit modality specialization, balance cross-modal information-sharing, and control parametrization or computational cost. Modality-awareness is operationalized through conditional projection layers, learned or rule-based routing functions, and associated regularization designed to promote coverage, confidence, and efficiency. The method is now established across domains such as remote sensing, speech-text language modeling, recommendation, and parameter-efficient fine-tuning.

1. Principles of Modality-Aware Projection

Modality-aware projection decomposes the standard input embedding process so that each modality—whether image patch, speech segment, or text token—is mapped to representation space via a modality-specific (often linear) transformation. This specialization is realized through either distinct projection matrices per modality or shared embeddings augmented with explicit modality tokens or type embeddings. For example, in remote sensing foundation modeling (MAPEX) and language-image early-fusion pre-training (MoMa), each modality $m$ provides an input $x^m$ that is projected as

$$t_i^m = E_m \cdot \operatorname{vec}(x_i^m) + b_m,\quad t_i^m\in\mathbb{R}^D,$$

where $E_m$ is a learnable matrix for modality $m$ (Hanna et al., 10 Jul 2025, Lin et al., 2024). Modality-specific projections have also been extended to state-space models, replacing shared parameter matrices with modality-indexed ones for every linear transformation whose domain is a single-modal token group (Liang et al., 27 Jan 2025).

Augmentations such as fixed or learned positional embeddings, explicit [MODALITY] tokens, and type encodings are commonly combined with the core projection stage to preserve both positional context and modality identity (Hanna et al., 10 Jul 2025, Lin et al., 2024). In graph-based recommendation and hybrid graph-neural networks, analogous lightweight modality-specific linear encoders map content features into a shared space for subsequent multimodal fusion (Dai et al., 24 Feb 2026).

2. Routing Mechanisms and Expert Selection

At the core of modality-aware architectures is the routing function—a mechanism for dynamically (or statically) selecting a subset of experts, sub-networks, or fusion strategies for a given token, region, or sample, based on modality information. Routing may be learned, as in mixture-of-experts (MoE), or rule-based, as in strict token-type partitioning.

In contemporary models, routing typically combines the following components:

• Routing Network Input: In MAPEX, the router takes only the modality embedding as input, not the token itself: for modality $m$, logits $a_{e|m} = W_r \cdot m_{\text{emb}} + b_r$ parametrize expert scores (Hanna et al., 10 Jul 2025). In MoST and MoMa, routers process token representations post-projection and apply modality masking, so a token of type $m$ cannot be routed to experts of other modalities (Lou et al., 15 Jan 2026, Lin et al., 2024).
• Hard or Soft Routing: Routing may be “hard” (top-1 or one-hot, as in speech/text hard assignment) or “soft” (distributing a token over a small subset of experts through softmax or sigmoid-derived gates) (Lee et al., 13 Feb 2026, Lou et al., 15 Jan 2026).
• Masking and Partitioning: Most frameworks partition experts into modality-specific groups. Only experts of the relevant modality are considered for a particular token, achieved through masking operations post-softmax or, in rule-based SSM architectures, direct assignment via the modality mask (Liang et al., 27 Jan 2025).
• Auxiliary Routing Heads: Where main routing is non-causal, lightweight auxiliary routers are sometimes trained to reproduce main routing decisions for improved efficiency or inference compatibility (Lin et al., 2024).
• Routing Regularization: Various models implement auxiliary regularization. This includes load-balancing losses, such as $$\mathcal{L}_\text{load} = \sum_i^N \bar p_i \log(\bar p_i)$$ (MoST), or entropy-based regularizers to maintain expert utilization or regulate specialization (MAGNET, MoST, MAPEX) (Dai et al., 24 Feb 2026, Lou et al., 15 Jan 2026, Hanna et al., 10 Jul 2025).
• Adaptive Fusion: In tasks such as change detection (UniRoute), routing applies not to experts but to fusion operators. Here, per-pixel softmax gates select, via a hard one-hot, which fusion primitive (difference, concatenation, multiplication) is used for each spatial position (Shu et al., 21 Jan 2026).

3. Expert Specialization and Modal Interaction

Modal-aware design induces specialization at the expert level, resulting in experts that model the unique spectral, spatial, or sequential statistics of their assigned modality. In MoE-based architectures, experts are grouped and exclusively exposed to tokens of relevant modalities (Lin et al., 2024, Lou et al., 15 Jan 2026). Shared experts or modules (included in most large-scale and robust systems) serve as bridges, capturing cross-modal primitives (edges, textures, generic syntax) and facilitating information transfer between modalities (Hanna et al., 10 Jul 2025, Lou et al., 15 Jan 2026).

Some frameworks—such as the structured mixture-of-experts routing in MAGNET—initiate expert roles as dominant, balanced, or complementary, with a weighting scheme that allows explicit balancing and interpretability of multimodal fusion contributions (Dai et al., 24 Feb 2026). In vision-language parameter-efficient fine-tuning, route-aware bottlenecks (routing functions) enable direct manipulation (addition, scaling, projection) of low-rank representations based on the other modality, yielding bottom-up and top-down alignment effects (Qu et al., 2024).

In state-space models, modality-awareness operates by decoupling all “projection” components (input, intermediate, output) such that each modality has a private set of matrices, while the core state-transition remains shared. This ensures that computational specialization is applied only where modalities differ in distribution, not in state evolution mechanics (Liang et al., 27 Jan 2025).

4. Training Protocols and Regularization

Training strategies are tailored to promote coverage, balance, and robust specialization:

• Modality Dropout: As in MAPEX, entire modalities are randomly zeroed out during pre-training, which has the effect of compelling shared experts to learn more general-purpose, cross-modal features, while discouraging spurious reliance on cross-modal co-occurrence (Hanna et al., 10 Jul 2025).
• Load-Balancing and Entropy Schedule: Explicit loss terms such as $\mathcal{L}_\text{load}$ for expert usage uniformity (Hanna et al., 10 Jul 2025, Lou et al., 15 Jan 2026) or staged entropy weighting (MAGNET), which transitions routing behavior from early-stage “coverage” (explore experts) to late-stage “confidence” (commit to stable specializations) (Dai et al., 24 Feb 2026), are widely used to stabilize expert utilization.
• Auxiliary Supervisory Losses: Multi-level consistency regularization (CASD) ensures stability under heterogeneous data and promotes pixel- or instance-level agreement between alternative routing or fusion choices, crucial for weakly supervised, data-scarce tasks (Shu et al., 21 Jan 2026).
• Balanced Downstream Pruning: MAPEX demonstrates explicit modality-aware pruning, where non-relevant experts and projections are removed for a given downstream modality subset, reducing model capacity, improving efficiency, and sometimes enhancing accuracy by removing modality-mismatched interference (Hanna et al., 10 Jul 2025).

5. Empirical Impact and Efficiency Gains

Empirical analyses consistently demonstrate that modality-aware projection and routing yield substantial improvements in parameter efficiency, FLOPs savings, and task-specific accuracy:

Model/Paper	Task(s)	Reported Gains
MAPEX (Hanna et al., 10 Jul 2025)	Remote sensing	+2–5% accuracy vs. monolithic models; 4× size ↓
MoMa (Lin et al., 2024)	Vision-language pretraining	3.7×–5.3× FLOPs saving vs. dense/mixed MoE
MoST (Lou et al., 15 Jan 2026)	Speech-text LLM	+7–12 points: ASR, TTS, A-LM, SQA over baselines
Mixture-of-Mamba (Liang et al., 27 Jan 2025)	Multimodal SSM	Baseline loss at 25–45 % FLOPs; >2–7% final loss ↓
MAGNET (Dai et al., 24 Feb 2026)	Multimodal recom.	Consistently outperforms strong multi-modal baselines
UniRoute (Shu et al., 21 Jan 2026)	Remote-sensing CD	High accuracy-efficiency trade-off, robust fusion
Routing-LoRA/Adapter (Qu et al., 2024)	VL PEFT	+10–30% rel. gain; VQA, captioning, multi-task sets

General patterns observed include: (1) specialists outperform monolithic/shared experts when data is heterogeneous or imbalanced; (2) modality-aware routing preserves or improves accuracy at reduced computational cost; (3) masking and balanced capacity allocation prevent expert collapse or starvation; (4) both cross-modal and modality-specific paths are critical, as confirmed by ablation studies.

6. Representative Architectures and Applications

Prominent architectures employing modality-aware projection and routing include:

• MAPEX (Hanna et al., 10 Jul 2025): Mixture-of-Modality Experts for remote sensing; ViT backbone, modality-conditioned router, expert pruning, shared cross-modal expert, load balancing and dropout.
• MoST (Lou et al., 15 Jan 2026): Speech-text LLM, MAMoE architecture; modality-masked routing, parallel expert groups, shared expert, uniform-load regularization, staged pretraining/fine-tuning on ASR, TTS, and mixed-instruction data.
• MoMa (Lin et al., 2024): Early-fusion LLM with 4+4 modality-specific experts, expert-choice routing, auxiliary router heads, significant FLOPs reduction for pretraining.
• Decoder-only Conformer (Lee et al., 13 Feb 2026): Speech-text Conformer with disjoint expert pools, hard top-1 routing, hybrid-causality blocks.
• MAGNET (Dai et al., 24 Feb 2026): Multimodal recommendation, three modalities, three expert types (dominant, balanced, complementary), entropy-triggered routing schedule.
• UniRoute (Shu et al., 21 Jan 2026): Remote sensing change detection, adaptive receptive field routing, modality-aware difference routing MoE, per-pixel operator selection, consistency self-distillation.
• Mixture-of-Mamba (Liang et al., 27 Jan 2025): SSM with modality-gated decoupled projections, rule-based routing, superadditive synergy of decoupling all parameter spaces touched by single-modal tokens.
• Routing Functions in PEFT (Qu et al., 2024): Vision-language fine-tuning, four linear routing functions in low-rank adapter/LoRA bottlenecks, no new parameters, direct cross-modal interaction.

Applications span remote sensing (land-cover/flood/wildfire detection), multimodal ASR/TTS, multimodal and hybrid recommendation, VL PEFT benchmarks (VQA, COCO, GQA, NLVR²), and generic multi-modal language and vision-language modeling.

7. Limitations and Future Directions

Known limitations include:

• Sensitivity to router accuracy, especially in causal/depth-variable early-fusion models (Lin et al., 2024).
• Potential expert collapse or congestion in absence of appropriate entropy/load balancing (Dai et al., 24 Feb 2026, Lou et al., 15 Jan 2026).
• Generalization to open-set, non-enumerated modality mixes remains investigational.
• The optimal degree of parameter decoupling and routing granularity is context-dependent; some gains are synergistic only when all projection components are decoupled (Liang et al., 27 Jan 2025).
• Routing methods in low-param PEFT are currently limited to linear functions; more expressive or nonlinear routing remains to be systematically explored (Qu et al., 2024).

Despite these challenges, modality-aware projection and routing has established itself as an effective principle for scalable, efficient, and interpretable multimodal learning, with demonstrated benefits in efficiency, performance, and model specialization across diverse domains.