MoMAE: Mixture-of-Modality Adapter Experts

• The paper introduces a modular expert design that replaces full FFNs with lightweight adapters, achieving a performance gain of +0.28% on ScienceQA.
• MoMAE offers explicit, one-hot token routing to modality-specific experts, ensuring efficient fine-tuning for mixed-modality integration.
• The architecture preserves frozen foundation models by isolating modality adaptations, effectively mitigating catastrophic forgetting and interference.

The Mixture-of-Modality-Adapter-Expert (MoMAE) is a modular architectural mechanism within transformer-based LLMs and vision-LLMs (VLMs) that assigns dedicated adapter "experts" to tokens according to their modality. This approach enforces principled disentanglement and adaptation for processing mixed-modality sequences, such as combined visual and textual data, by routing each token type to a modality-specific lightweight adapter. As instantiated in the PILL framework for efficient vision-language adaptation, MoMAE enables parameter-efficient fine-tuning, explicit specialization, and mitigates catastrophic forgetting, while maintaining the integrity of the pre-trained foundation model (Zhang et al., 2023). MoMAE also provides a conceptual foundation that is reflected in related modality-aware mixture-of-expert architectures for other mixed-modality settings (Lin et al., 2024, Chopra et al., 10 Jun 2025).

1. Motivation and Concept

MoMAE is motivated by challenges in mixed-modality model adaptation, where a single model must integrate and process non-homogeneous input types (e.g., language, vision, audio). In conventional strategies, modality fusion often leads to parameter sharing across modalities, which may result in catastrophic forgetting and suboptimal adaptation of each input type. MoMAE instead introduces a scheme where every transformer block replaces its feed-forward layer with a pair (or more, for additional modalities) of small bottleneck adapters—each dedicated to a modality. Routing is performed via a hard, one-hot gating mechanism: visual tokens select the visual adapter, text tokens select the text adapter, and so on. This specialization preserves the contextual capacity of the base transformer while ensuring that modality-specific adaptation proceeds independently (Zhang et al., 2023).

2. Mathematical Structure and Routing

Given a sequence of multimodal embeddings $H = \{h_1, ..., h_N, v_{11}, ..., v_{jK}\}$, with $h_i \in \mathbb{R}^d$ (text) and $v_{jk} \in \mathbb{R}^d$ (visual, after projection to model space), MoMAE splits $H$ into $H^t$ (text) and $H^v$ (visual). For each token $x$, a routing gate

$$g_v(x) = \begin{cases} 1, & x \in H^v \ 0, & x \in H^t \end{cases}, \quad g_t(x) = 1 - g_v(x)$$

ensures that MoMAE applies only the expert $E_v$ (visual) for visual tokens and $E_t$ (text) for text tokens:

$h_i \in \mathbb{R}^d$0

Each adapter expert is a bottleneck residual module (often a SwiGLU variant for expressivity), with parameters orders of magnitude smaller than a full FFN (e.g., adapter width $h_i \in \mathbb{R}^d$1 vs. FFN typically $h_i \in \mathbb{R}^d$2). The routing logic is explicit, requiring no optimization of gating or balancing objectives. This mechanism readily extends to more than two modalities by adding additional experts and corresponding gates (Zhang et al., 2023).

3. Adapter Expert Architecture

Each modality expert $h_i \in \mathbb{R}^d$3 ($h_i \in \mathbb{R}^d$4) processes its input via:

• Down-projection: $h_i \in \mathbb{R}^d$5, $h_i \in \mathbb{R}^d$6 with $h_i \in \mathbb{R}^d$7, $h_i \in \mathbb{R}^d$8.
• Gated fusion: $h_i \in \mathbb{R}^d$9.
• Up-projection and residual: $v_{jk} \in \mathbb{R}^d$0, $v_{jk} \in \mathbb{R}^d$1, with $v_{jk} \in \mathbb{R}^d$2.

Typical instantiations (e.g., with $v_{jk} \in \mathbb{R}^d$3 and $v_{jk} \in \mathbb{R}^d$4) yield minimal parameter and computational overhead ($v_{jk} \in \mathbb{R}^d$50.8M parameters per layer for visual and text adapters combined, compared to the full model scale). The adapters operate at each transformer block, replacing the original FFN, and all base transformer weights remain frozen. Only adapter and projection weights are updated during fine-tuning (Zhang et al., 2023).

4. Integration into Transformer Architectures

In transformer-based LLMs (e.g., LLaMA), a PILL layer adapts the original transformer block sequence as follows:

1. Multi-head self-attention and residual: $v_{jk} \in \mathbb{R}^d$6, $v_{jk} \in \mathbb{R}^d$7
2. Adapter expert application: $v_{jk} \in \mathbb{R}^d$8
3. Final output: $v_{jk} \in \mathbb{R}^d$9

Crucially, all original self-attention and normalization operations remain unchanged, and only the MoMAE adapters are trained. Token-modality information is always directly accessible due to the architectural definition of visual and language inputs, enabling efficient and deterministic per-token expert selection (Zhang et al., 2023).

5. Training Regimes and Empirical Results

Training proceeds in two stages: stage 1 optimizes the visual adapter and projection layers to align visual features with the LLM embedding space; stage 2 jointly fine-tunes both adapters and other newly introduced modules (e.g., Modality-Attention-Gating). The sole training objective is the standard autoregressive cross-entropy loss over the LLM vocabulary, with no explicit expert-balance, entropy, or auxiliary regularization.

Empirically, MoMAE yields parameter-efficient adaptation with minimal computational cost relative to full FFNs. Ablation studies on ScienceQA with a 7B LLM backbone show that a vanilla MoMAE plus linear-adapter baseline achieves 89.20% accuracy, while adding advanced adapters and modality-attention gates yields 90.73%. Removing MoMAE but keeping advanced gates and adapters results in 90.45%, indicating a +0.28 percentage point gain attributable to MoMAE relative to prefix-tuning baselines (Zhang et al., 2023). Gains from MoMAE are preserved when further scaling with more modalities or larger model backbones.

6. Comparison to Related Modality-Aware MoE Variants

MoMAE shares conceptual underpinnings with other modality-specialized mixture-of-experts designs:

• MoMa (Lin et al., 2024) introduces modality-grouped MoEs for mixed-modal autoregressive transformers, where tokens are routed to text-specific or image-specific expert FFNs, yielding substantial FLOPs savings and improved sample efficiency in multimodal pretraining.
• MedMoE (Chopra et al., 10 Jun 2025) applies report-conditioned, modality-specialized MoEs within medical vision-language systems. Here, a feature-pyramid backbone is routed through modality experts via a learned gating MLP, with scale-level attention for visual feature fusion.

All such architectures underscore the trend of explicit per-modality specialization in transformer-based settings. MoMAE distinguishes itself by its focus on efficiency and preservation of frozen foundation models, with direct routing and parameter isolation that avoids interference between modalities.

7. Limitations and Extensions

MoMAE incurs only a small fraction of the FFN's compute and parameter budget and supports independent adaptation for heterogeneous modalities. However, the current design's efficacy for more than two modalities, instability under heavy class imbalance, or scaling to higher-order data types such as audio or video remains to be explored within this framework. No evidence of expert-balancing regularization is present in the foundational implementation, which may limit robustness in scenarios with highly variable token type distributions. A plausible implication is that routing and adapter design will need refinement to maintain per-modality specialization as application domains become more heterogeneous and as models scale to new datatypes.

In summary, Mixture-of-Modality-Adapter-Expert modules offer a rigorously defined, parameter-efficient, and extensible foundation for modality-specialized adaptation in large neural sequence models, supporting advancements in multimodal foundation model alignment and transfer (Zhang et al., 2023).