---
title: 'AsyMoE: Asymmetric Modality-Aware Experts'
url: https://www.emergentmind.com/topics/asymoe
type: topic
---

# AsyMoE: Asymmetric Modality-Aware Experts

AsyMoE is a Mixture-of-Experts architecture for large vision-language models that is designed around the claim that visual and linguistic processing are structurally asymmetric. In the paper, AsyMoE is introduced as “Asymmetric Modality-aware Experts” and is organized around three specialized expert groups: intra-modality experts for modality-specific computation, hyperbolic inter-modality experts for hierarchical cross-modal interaction, and evidence-priority language experts intended to suppress parametric bias and preserve contextual grounding. The method is presented as a sparse LVLM design that improves multimodal reasoning while using fewer activated parameters than dense counterparts [2509.12715].

## 1. Modal asymmetry as the organizing principle

The motivating premise of AsyMoE is that visual information is spatially complete, whereas language is sequential and context-dependent. In the paper’s framing, this asymmetry is not a superficial difference in token type but a structural property that affects how expert specialization should be organized. Dense LVLMs process both modalities through shared feed-forward capacity, which can blur modality-specific inductive structure and permit drift away from visual evidence. Vanilla multimodal MoE models typically use a shared expert pool inherited from pure language-model designs, which the paper argues ignores modality asymmetry. Modality-specific MoE improves intra-modal processing but can weaken cross-modal association, while Euclidean inter-modal designs are described as poorly matched to the view that language often describes a partial semantic subset of a complete visual scene [2509.12715].

A central diagnostic claim is that language experts in deeper layers progressively lose contextual grounding and increasingly rely on parametric memory rather than the provided visual and linguistic evidence. The paper describes this as a memory-priority shift. Early layers are said to establish task context, while deeper layers increasingly favor memorized statistical patterns, especially in cross-modal settings, ambiguous language, and long-tail expressions. This diagnosis is tied directly to AsyMoE’s evidence-priority language experts, which are meant to maintain evidence faithfulness rather than merely increase parameter count [2509.12715].

The paper also supports the asymmetry claim with layerwise analyses. Visual attention entropy is described as relatively stable, whereas language attention entropy increases with depth. Likewise, expert-level grounding analyses are used to distinguish evidence-grounded behavior from memory-driven behavior. These analyses are not auxiliary presentation devices; they are used to justify the architecture’s separation into modality-specific, cross-modal, and evidence-priority components [2509.12715].

## 2. Architecture and expert groups

AsyMoE is built as an LVLM using a pretrained SigLIP visual encoder, a two-layer MLP connector, and a language-model backbone such as Phi-3-mini or LLaMA3-8B. The multimodal input is written as
\[
\mathbf{x} = \{\mathbf{x}_v, \mathbf{x}_l\},
\]
where \(\mathbf{x}_v \in \mathbb{R}^{N_v \times d}\) is the visual token sequence and \(\mathbf{x}_l \in \mathbb{R}^{N_l \times d}\) is the language token sequence. The paper describes MoE layers as replacing the normal feed-forward behavior with gated expert computation in the language-model stack, although the exact layer indices are not specified in the visible text [2509.12715].

The generic MoE update is written as
\[
\mathbf{h}_{i+1} = \mathbf{h}_i + \sum_{k=1}^{K} g_k(\mathbf{h}_i) \cdot E_k(\mathbf{h}_i),
\]
where \(E_k\) is expert \(k\), \(g_k(\mathbf{h}_i)\) is its gating weight, and \(K\) is the number of activated experts. The expert set is denoted
\[
\mathcal{E} = \{\mathcal{E}_V, \mathcal{E}_{L-evd}, \mathcal{E}_S\},
\]
where \(\mathcal{E}_V\) are visual intra-modality experts, \(\mathcal{E}_{L-evd}\) are evidence-priority language experts, and \(\mathcal{E}_S\) are shared inter-modality experts [2509.12715].

The intra-modality experts are intended for modality-specific processing. For visual tokens, routing is explicitly defined as
\[
g_V(\mathbf{h}_v) = \text{Softmax}(\mathbf{W}_V \cdot \mathbf{h}_v).
\]
These experts are described as preserving patterns internal to a modality rather than forcing visual and language tokens to compete for a completely shared expert pool. The paper’s textual description repeatedly refers to intra-modality processing for both visual and linguistic features, though the visible equations formalize the visual path more directly [2509.12715].

The hyperbolic inter-modality experts are the geometric center of the architecture. They are designed to mediate image-text interaction under the claim that text often provides a partial description of a larger visual scene. Rather than using a flat Euclidean representation, AsyMoE maps visual and evidence-priority language features into a hyperbolic space and measures alignment with Lorentzian distance [2509.12715].

The evidence-priority language experts are the most distinctive part of the design. They are meant to counter the late-layer drift toward parametric memory by mixing memory-driven and evidence-driven computation:
\[
E_{L-evd}(\mathbf{h}) = \alpha \cdot F_{mem}(\mathbf{h}) + (1-\alpha) \cdot F_{evd}(\mathbf{h}, \mathbf{c}),
\]
where \(\alpha \in [0,1]\) is learnable. The intended behavior is that evidence-intensive cases emphasize \(F_{evd}\), while still retaining some benefit from parametric memory [2509.12715].

## 3. Routing, geometry, and analytical formalization

The language-side routing adds an evidence-aware bias:
\[
g_L(\mathbf{h}_l) = \text{Softmax}(\mathbf{W}_L \cdot \mathbf{h}_l + s_{evd} \cdot \mathbf{m}_{evd}),
\]
where \(\mathbf{m}_{evd}\) is a mask indicating evidence-priority experts. The evidence relevance score is defined as
\[
s_{evd} = \sigma(\mathbf{w}_{evd}^T \cdot \text{Attn}(\mathbf{h}_l, \mathbf{c})).
\]
This mechanism does not introduce a hard evidence-versus-memory switch; instead, it biases routing toward evidence-priority experts when the token’s contextual dependence is high [2509.12715].

The hyperbolic cross-modal relation is defined as
\[
D_{hyp}^{AsyMoE} = d_L^2(\Phi_V(\mathbf{x}_v), \Phi_{L-evd}(\mathbf{x}_l, \mathbf{c})),
\]
where \(d_L\) is the Lorentzian distance, \(\Phi_V(\mathbf{x}_v)\) is the visual mapping into hyperbolic space, and \(\Phi_{L-evd}(\mathbf{x}_l, \mathbf{c})\) is the evidence-priority language mapping. The paper also introduces a cone-based partial-order constraint,
\[
\mathcal{L}_{order} = \max(0, EA(\mathbf{v}, \mathbf{t}) - HA(\mathbf{v})),
\]
where \(EA(\mathbf{v}, \mathbf{t})\) is the exterior angle between visual and textual embeddings and \(HA(\mathbf{v})\) is the entailment cone aperture. The stated purpose is to preserve the asymmetry that text is semantically contained within the visual scene rather than being an equal and symmetric counterpart [2509.12715].

The paper’s diagnostic formalization includes an evidence grounding ratio
\[
\mathcal{F}(E_k) = \frac{\mathbb{E}[A_{evd}^k]}{\mathbb{E}[A_{evd}^k] + \mathbb{E}[A_{mem}^k]},
\]
where higher values indicate stronger evidence grounding. It also gives modality-wise attention entropy:
\[
H_V(l) = -\sum_{i,j} a_{ij}^{(l)} \log a_{ij}^{(l)}, \qquad
H_L(l) = H_{L,0} + \beta \cdot l.
\]
The interpretation is that visual attention entropy remains relatively stable, while language attention entropy grows with depth, signaling context dilution [2509.12715].

The visible manuscript also describes an evidence-dependence ratio \(R_k\), but the equation is malformed in the supplied text. This suggests a contrast between evidence-driven and memory-driven activation frequencies, yet the exact algebra should be verified against the original manuscript rather than reconstructed from the corrupted expression [2509.12715].

## 4. Training configuration and empirical results

The implementation uses a 4-expert configuration comprising intra-modality experts for vision and language, inter-modality experts, and evidence-priority language experts. Training is two-stage: first a connector optimization stage on Bunny-pretrain-LAION-2M, then full AsyMoE instruction tuning on visual instruction datasets. The reported optimization setup uses AdamW, cosine scheduling, a learning rate of \(2 \times 10^{-6}\) for the visual encoder, a learning rate of \(2 \times 10^{-5}\) for other components, 8 NVIDIA H20 GPUs, and DeepSpeed ZeRO-3. AsyMoE-Phi3 activates 4.1B parameters and AsyMoE-LLaMA3 activates 10.8B parameters. The paper states that AsyMoE activates 25.45% fewer parameters than dense models, although the exact benchmark protocol associated with that headline percentage is not fully specified in the visible text [2509.12715].

On the dense-backbone family in Table 1, AsyMoE-Phi3 improves over MoIIE-Phi3 while using fewer activated parameters. The reported scores are TextVQA 68.2, GQA 65.2, POPE 87.6, MMBench 75.9, MME 1569.4, MMVet 45.8, MMMU\(_{val}\) 42.3, SEED-IMG 72.5, and AI2D 67.8, compared with MoIIE-Phi3 at 5.5B activated parameters and lower scores on each listed metric. AsyMoE-LLaMA3 similarly improves over MoIIE-LLaMA3, reporting TextVQA 70.9, GQA 66.9, POPE 89.7, MMBench 76.8, MME 1589.9, MMVet 49.2, MMMU\(_{val}\) 43.4, SEED-IMG 73.2, and AI2D 71.7 at 10.8B activated parameters [2509.12715].

In the broader Table 2 comparisons, AsyMoE is reported as the best model among Dense, Vanilla MoE, Modality MoE, ContextMoE, and MoIIE for the settings shown. With 2M data and Phi-3-mini, AsyMoE reaches AVG 58.3; with 2.7M data and Phi-3-mini, AVG 61.2; and with LLaMA3-8B and 2M data, AVG 62.1. Figure 4 is described as showing that AsyMoE scales better as visual instruction data increases from 1.3M to 2.7M samples, while baselines begin to plateau [2509.12715].

The abstract reports 26.58% accuracy improvement over vanilla MoE and 15.45% improvement over modality-specific MoE. The supplied details note that these percentages are presented as headline gains, but the precise benchmark averaging protocol is not fully recoverable from the visible text. The empirically auditable tables, however, consistently show absolute improvements over Dense, Vanilla MoE, Modality MoE, ContextMoE, and MoIIE across the listed settings [2509.12715].

## 5. Ablations, specialization evidence, and limitations

The ablation study in Table 4 attributes the strongest degradation to removal of evidence-priority language experts and substantial degradation to removal of hyperbolic space. The reported full model scores are TextVQA 68.1, MMBench 76.8, GQA 65.3, and AVG 61.2. The ablated variants are: w/o Hyperbolic Space at 66.8, 75.1, 64.2, AVG 58.7; w/o Evidence-Priority Experts at 67.1, 74.9, 64.5, AVG 58.8; w/o Cross-Modal Experts at 67.5, 75.2, 64.8, AVG 59.2; and w/o Intra-modality Separation at 67.0, 74.8, 64.3, AVG 58.7. The text summarizes the gains as about 2.5% average gain from evidence-priority experts, about 1.8% gain from hyperbolic cross-modal experts, and about 1.2% gain from evidence-aware routing [2509.12715].

The \(\alpha\) sensitivity study identifies \(\alpha = 0.3\) as the best setting, reporting TextVQA 68.24, MMBench 75.94, GQA 65.18, MMMU\(_v\) 42.31, POPE 87.64, and MMVet 45.83. Larger \(\alpha\) values reduce performance, which is consistent with the design claim that excessive reliance on memory-driven processing harms contextual grounding [2509.12715].

The paper also presents qualitative specialization evidence. Figure 5 is described as showing that evidence-priority experts dominate on General QA and Hallucination tasks, while visual experts dominate on spatial reasoning tasks. Figure 6 is described as showing stable context attention together with increasing answer focus through depth. These observations are used to argue that expert groups are not merely partitioned architecturally but become functionally differentiated in use [2509.12715].

Several limitations are explicit. The visible text does not provide a complete global training loss decomposition beyond the hyperbolic order term. The exact top-\(k\) value used by AsyMoE is not explicitly specified in the visible text, although the method is described as a sparse routed expert model in standard top-\(k\) MoE style. The mathematical treatment of hyperbolic space is incomplete in the visible manuscript: the Lorentzian distance is named, but explicit manifold operations are omitted. One evidence-dependence equation is malformed. The abstract’s headline percentage improvements are also somewhat ambiguous in the supplied text because their exact averaging protocol is not fully tied to a visible benchmark summary [2509.12715].

## 6. Disambiguation and place within the MoE literature

AsyMoE should be distinguished from several neighboring MoE designs that address different problems. “Adaptive Shared Experts with LoRA-Based Mixture of Experts for Multi-Task Learning” introduces Adaptive Shared Experts (ASE) for STL-to-MTL transition in dense prediction, not AsyMoE, and the paper explicitly does not use the acronym “AsyMoE” [2510.00570]. “Toward Cost-Efficient Serving of Mixture-of-Experts with Asynchrony” proposes Asynchronous Expert Parallelism (AEP) and the AMoE system for asynchronous MoE inference serving, which is a systems design concerned with \(\mu\)-queuing, adaptive re-batching, and barrier removal rather than modality asymmetry in LVLMs [2505.08944]. “ASAP: A Disaggregated and Asynchronous Inference System for MoE Prefill” is likewise an asynchronous serving architecture for MoE prefill, focused on TTFT and synchronization barriers between attention DP groups and experts [2606.22541]. “Hierarchical MoE: Continuous Multimodal Emotion Recognition with Incomplete and Asynchronous Inputs” is a multimodal emotion-recognition framework for asynchronous and incomplete modalities, but its target task, routing structure, and continuous regression setting differ materially from AsyMoE’s LVLM formulation [2508.02133].

Within LVLM research proper, AsyMoE is positioned against dense models, vanilla shared-pool MoE, modality-specific MoE, ContextMoE, MoIIE, and larger activated-parameter baselines such as CuMo. Its specific novelty is not merely the insertion of experts into a multimodal transformer. The paper’s claim is that expert specialization should be determined by modal asymmetry: modality-specific experts preserve local structure, hyperbolic inter-modality experts model hierarchical partial-to-whole relations between text and image, and evidence-priority language experts reduce deep-layer drift toward parametric memory. In that sense, AsyMoE is best understood as a modality-asymmetric sparse LVLM whose main contribution lies in how it organizes expert roles rather than in MoE sparsity alone [2509.12715].

Source: https://www.emergentmind.com/topics/asymoe