Context-Aware Mixture of Experts (CMoE)

• Context-aware Mixture-of-Experts (CMoE) is a design pattern where expert activation, weighting, and fusion depend on contextual signals like scene, task, or hardware status.
• Architecturally, CMoE enhances traditional MoE models by integrating context into gate functions and expert outputs through strategies like coarse-to-fine routing and late fusion.
• Applications of CMoE span vision-language, long-context language modeling, and real-time systems, resulting in improved accuracy, throughput, and task-specific performance.

Context-aware Mixture-of-Experts (CMoE) denotes a family of mixture-of-experts systems in which expert activation, weighting, fusion, or execution depends on contextual information rather than on a context-agnostic fixed partition of computation. Across the literature, the term covers model-level routing conditioned on scene, instruction, task, or process context, as well as systems-level policies that use runtime context to decide where and how experts execute. The acronym itself is not fully standardized: in addition to Context-aware or Contextual Mixture-of-Experts, arXiv uses CMoE for Convolution-based Mixture of Experts, Continual Mixture of Experts, Contrastive Mixture of Experts, and Carved MoE, which indicates that the concept is better understood as a design pattern than as a single canonical architecture (Zhou et al., 2022, Souza et al., 2022, Xie et al., 2024, Pei et al., 6 Feb 2025, Zhao et al., 27 Mar 2025, Ma et al., 3 Mar 2026).

1. Scope and conceptual variants

In the broadest sense, CMoE extends ordinary MoE by making expert usage depend on information that identifies the current regime of computation. In model-centric papers, that regime may be the current scene, user instruction, dialogue state, task distribution, process phase, or long-context positional requirement. In systems papers, it may instead be request-specific expert popularity, hardware availability, quantization sensitivity, or server idleness (Zong et al., 2024, Fan et al., 4 Dec 2025, Li et al., 2024).

Representative work	Context source	Expert granularity
CAME (Zhou et al., 2022)	Scene context and predicate context	Relation experts
cMoE (Souza et al., 2022)	Process knowledge via possibility distributions	Local predictive experts
MoVA (Zong et al., 2024)	Image, instruction, and expert descriptions	Frozen vision encoders
MoICE (Lin et al., 2024)	Query-state-dependent positional needs	RoPE-angle experts
MCCRS (Zou et al., 18 Apr 2025)	Conversation, KG, and review evidence	Context-modality experts
GPU–NDP CMoE (Fan et al., 4 Dec 2025)	Prefill-stage routing statistics	Runtime expert placement

This breadth creates an important distinction. Some CMoE systems alter the semantic routing function of the model, while others leave the model’s semantic routing unchanged and instead make execution policy context-aware. A related distinction is between homogeneous experts, such as replicated FFN branches, and heterogeneous experts, such as pretrained vision backbones, graph modules, review encoders, or server-side learners (Zong et al., 2024, Zou et al., 18 Apr 2025, Li et al., 2024).

2. Architectural principles

The most common architectural pattern is to preserve the MoE decomposition

$$\hat{y}(\mathbf{x})=\sum_{c=1}^{C} g_c(\mathbf{x})\hat{y}_c(\mathbf{x}),$$

while enriching either the gate $g_c$, the expert outputs $\hat{y}_c$, or both with contextual signals (Souza et al., 2022). In industrial soft sensing, the contextual mixture of experts uses analyst-defined contexts encoded as possibility distributions $\pi_{ci}$, which reshape responsibilities and weighted regression updates during EM; the experts and gates remain linear for interpretability (Souza et al., 2022). In unbiased scene graph generation, CAME adds multiple relation experts and makes them context-aware through expert weighting and predicate weighting derived from built-in scene context and edge context modules (Zhou et al., 2022).

A second pattern is coarse-to-fine contextualization. MoVA exemplifies this structure. It first performs coarse-grained routing over heterogeneous frozen vision experts using the image, the user query, and textual descriptions of each expert’s expertise, and then performs fine-grained fusion with a Mixture-of-Vision-Expert Adapter that uses both visual and textual context to assign soft expert weights block by block (Zong et al., 2024). The selected set is denoted $\mathbf{G}$, its size $K$ is sample-dependent, and the fusion stage combines cross-attended expert-conditioned residuals using softmax-normalized multimodal weights (Zong et al., 2024). This separates “which experts should be active at all” from “how the selected experts should be combined.”

A third pattern is contextual reinterpretation of what counts as an expert. MoICE does not use FFN experts. Instead, it treats different RoPE bases or angle sets as “in-context experts,” inserts a router into every attention head, and performs per-head, per-token top-$K$ routing based on the current query vector $\mathbf{q}_n^h$ (Lin et al., 2024). The resulting attention is a weighted mixture over attention maps computed under different RoPE-induced positional geometries, which directly targets uneven long-context awareness rather than weight-space specialization (Lin et al., 2024).

A fourth pattern is late-fusion CMoE. In conversational recommendation, MCCRS assigns a conversation expert, a graph expert, and a review expert to different context sources, then lets a ChairBot compute importance weights $\lambda_C,\lambda_G,\lambda_R$ from the experts’ own hidden representations and prediction vectors before mixing recommendation distributions: $$P_{rec}(i)= \lambda_C P_C(i) + \lambda_G P_G(i) + \lambda_R P_R(i).$$ The gating is therefore example-dependent but decision-level rather than token-level (Zou et al., 18 Apr 2025).

3. Learning signals for context and specialization

A recurring difficulty in CMoE is that conditional routing alone does not guarantee meaningful specialization. Several papers therefore add explicit learning signals that align routing with contextual structure. In CAME for scene graph generation, the context-aware loss is

$g_c$0

where the scene-context-dependent expert weights $g_c$1 amplify learning for experts that are more relevant to the current scene; predicate weighting then modulates final expert aggregation at the class level (Zhou et al., 2022). The result is a divide-and-ensemble strategy for long-tailed predicate learning rather than a hard-coded head/body/tail partition (Zhou et al., 2022).

In cMoE for industrial processes, contextual knowledge enters through possibility distributions $g_c$2. The E-step responsibilities become

$g_c$3

so a context with low possibility for a sample receives little or no responsibility mass for that sample (Souza et al., 2022). This makes contextual knowledge part of the optimization, not just a post hoc interpretation layer.

Contrastive learning is another major mechanism. In humanoid control, CMoE introduces a SwAV-style contrastive loss between transformed gate activations $g_c$4 and terrain embeddings $g_c$5. Positive pairs come from the same trajectory and negative pairs from different trajectories, which encourages expert activations to be consistent within a terrain and separated across terrains (Ma et al., 3 Mar 2026). The contrastive objective is

$g_c$6

and its purpose is to prevent the nearly uniform expert activations observed in vanilla MoE locomotion policies (Ma et al., 3 Mar 2026).

CoMoE uses a different contrastive logic. It treats activated experts under top-$g_c$7 routing as positives and inactivated experts as negatives, and adds

$g_c$8

to maximize the mutual-information gap between the input and the activated versus inactivated experts (Feng et al., 23 May 2025). This is explicitly a specialization method for sparse MoE-based PEFT rather than a new router architecture, but it is directly relevant to CMoE because it makes context-conditioned routing semantically sharper on heterogeneous data (Feng et al., 23 May 2025).

Continual multimodal learning introduces yet another specialization mechanism. LLaVA-CMoE uses Probe-Guided Knowledge Extension to decide where new experts should be added, and a Probabilistic Task Locator to select a task-specific router when task labels are unknown at inference (Zhao et al., 27 Mar 2025). The layer-expansion criterion compares probe activation frequency to the mean and standard deviation of expert activation frequencies in that layer, while router selection is done by VAE-based reconstruction matching over task-distribution primitives (Zhao et al., 27 Mar 2025). This suggests that, in continual settings, context may need to determine not only expert choice but also which router should govern expert choice.

4. Runtime- and systems-aware CMoE

Several papers expand CMoE beyond model semantics into execution policy. In a GPU–CXL-NDP inference system, context-aware MoE uses prefill-stage activation count $g_c$9 and routing-score sum $\hat{y}_c$0 to define a per-layer expert importance score

$\hat{y}_c$1

then pins the hottest experts in GPU HBM and leaves the rest on CXL-attached near-data processing memory (Fan et al., 4 Dec 2025). The same prefill statistics also drive per-expert mixed-precision assignment on the NDP side, with bitwidths from 1 to 4 bits. On the evaluated GPU–NDP platform, this yields up to an 8.7-fold decoding throughput improvement over the state-of-the-art method, with only a 0.13% average accuracy drop for the 3-bit configuration (Fan et al., 4 Dec 2025). Here, “context-aware” refers to runtime adaptation based on the request’s own routing profile rather than a new semantic gate.

A related systems reinterpretation appears in heterogeneous analog-digital deployment for sparse MoE LLMs. This framework does not change MoE routing at all; instead it assigns dense modules and the most noise-sensitive experts to digital hardware, and executes the rest on analog in-memory computing. The selection metric is the expert maximum neuron norm score

$\hat{y}_c$2

which the paper argues is provably linked to analog noise sensitivity (Chowdhury et al., 3 Mar 2026). This is not context-aware routing in the usual architectural sense, but it is context-dependent execution in the sense of expert-role- and hardware-aware placement (Chowdhury et al., 3 Mar 2026).

Dense-to-sparse conversion offers another angle. Carved MoE converts a pretrained dense LLM into an MoE by profiling FFN neuron activations, selecting high-rate neurons as shared experts, clustering the remaining neurons into balanced routed experts, and constructing a router analytically from representative neurons (Pei et al., 6 Feb 2025). It uses absolute TopK activation markers, activation rates

$\hat{y}_c$3

and a balanced assignment procedure to partition neurons (Pei et al., 6 Feb 2025). The reported workflow converts a 7B dense model within five minutes, and optional lightweight LoRA fine-tuning recovers much of the dense model’s quality under an hour (Pei et al., 6 Feb 2025). This is a sparsity-conversion framework rather than a semantic context model, but it shows how CMoE-style expert structure can be introduced post hoc.

In on-device modeling, MoLKV adds context awareness to lookup-based experts by turning each expert into a key-value pair and allowing the current token’s query to retrieve from sequence-cached experts: $\hat{y}_c$4 The key point is that global storage lookup remains token-id-based, but context-aware retrieval occurs over experts already cached from the current sequence, preserving the storage-bandwidth advantages of lookup models while making expert usage sequence-dependent (Wang, 10 Dec 2025).

Finally, the MEC theory paper treats each edge server as an expert and makes routing depend jointly on feature-derived gate scores and availability masks: $\hat{y}_c$5 It proves a minimum expert count condition under stochastic delays and shows that, once this threshold is satisfied, the generalization error converges to $\hat{y}_c$6, whereas standard MEC offloading converges to the expected inter-task gap (Li et al., 2024). A notable theoretical result is that adding more experts than necessary delays convergence and worsens finite-horizon generalization (Li et al., 2024).

5. Major application domains

In vision-language modeling, MoVA is one of the clearest CMoE realizations. It treats CLIP ViT-L/336px as the base encoder and uses DINOv2, Co-DETR, SAM, Pix2Struct, Deplot, Vary, and BiomedCLIP as frozen task-specific experts (Zong et al., 2024). The empirical complementarity is explicit: CLIP is best on MMBench among single encoders, DINOv2 on GQA and REC, Co-DETR on POPE, SAM on RES, Pix2Struct on DocVQA, Deplot on ChartQA, and BiomedCLIP on SLAKE (Zong et al., 2024). The context-aware routing and fusion stack yields broad multimodal gains; for example, MoVA-7B reaches 70.4 on MMBench, 69.3 on QBench, 37.6 on MathVista, 88.6 on POPE, 81.3 on DocVQA, and 68.3 on ChartQA (Zong et al., 2024). Ablations also show that replacing context-aware routing with random routing drops DocVQA from 81.3 to 71.6 and ChartQA from 68.3 to 60.4, while removing routing and using all experts gives 73.7 and 62.5, respectively (Zong et al., 2024).

Long-context language modeling motivates a different form of CMoE. MoICE identifies uneven positional awareness in RoPE-based LLMs and treats RoPE angle sets as experts. On Llama2-7B-chat, it improves the L-Eval closed-ended average from 47.52 to 50.72, outperforming Positional Interpolation, Dynamic NTK, Ms-PoE, and Attention Buckets; on Mistral-7B-Instruct-8k it reaches 55.38 (Lin et al., 2024). The $\hat{y}_c$7-ablation is especially informative: $\hat{y}_c$8 yields 47.22, while $\hat{y}_c$9 yields 50.72, which indicates that learned multi-expert positional mixtures matter more than selecting a single alternative RoPE basis (Lin et al., 2024).

Recommendation and industrial prediction show a more interpretable, decision-level CMoE form. MCCRS improves over $\pi_{ci}$0-CRS on ReDial from 0.407 to 0.473 in Recall@50 and on INSPIRED from 0.399 to 0.497, while ablations show that removing the graph expert hurts the most but removing any expert degrades the full system (Zou et al., 18 Apr 2025). In process modeling, cMoE raises SRU $\pi_{ci}$1 prediction to $\pi_{ci}$2 versus 0.583 for MoLE and identifies $\pi_{ci}$3 as the variable most associated with peak versus non-peak context switching; in the polymerization process it reaches $\pi_{ci}$4 and uses context-specific coefficients to distinguish reaction and vacuum regimes (Souza et al., 2022).

Scene graph generation supplies a specialized visual-reasoning CMoE. On Visual Genome PredCls, Motifs-CAME attains $\pi_{ci}$5, $\pi_{ci}$6, and mean 47.7, compared with baseline Motifs at $\pi_{ci}$7, and improves tail $\pi_{ci}$8 from 2.0 to 39.9 in the targeted analysis (Zhou et al., 2022). The gain comes from context-aware expert weighting and predicate weighting rather than from a large new context encoder (Zhou et al., 2022).

Acoustic recognition and robotics stress hard regime partitioning. The convolution-based MoE for underwater acoustic target recognition uses a ResNet-AP backbone, a linear routing layer, hard top-1 routing, and a balance loss $\pi_{ci}$9 to mitigate expert undertraining (Xie et al., 2024). On Shipsear with STFT features, performance rises from 75.24 for the baseline to 86.21 for CMoE with balance (Xie et al., 2024). In humanoid locomotion, contrastive CMoE achieves success rates of 0.886 on stair up, 0.974 on gaps, 0.767 on mix1, and 0.747 on mix2, outperforming vanilla MoE on the most difficult terrains; in real-world trials on Unitree G1 it traverses 20 cm continuous steps and 80 cm gaps (Ma et al., 3 Mar 2026).

Continual multimodal learning adds task-distribution awareness. On the CoIN benchmark, LLaVA-CMoE attains a final average of 59.23 with backward transfer -3.58, compared with 44.24 / -17.86 for LLaVA without MoE, and its average trainable parameter count per task is 43.96M versus 62M for the compared regularization baselines (Zhao et al., 27 Mar 2025). PTL localization exceeds 80% on ScienceQA, ImageNet, VizWiz, Grounding, and OCR-VQA, but is much weaker on overlapping tasks such as VQAv2, where the paper reports about 47% localization (Zhao et al., 27 Mar 2025).

6. Limitations, misconceptions, and open directions

A common misconception is that CMoE denotes one specific architecture. The literature shows instead that it is a family resemblance term. Some papers use it for semantic expert routing over model components (Zong et al., 2024, Zhou et al., 2022), some for knowledge-constrained EM in process modeling (Souza et al., 2022), some for continual router selection (Zhao et al., 27 Mar 2025), and some for hardware- or runtime-aware expert placement (Fan et al., 4 Dec 2025, Chowdhury et al., 3 Mar 2026). A plausible implication is that future surveys should distinguish model-aware, task-aware, process-aware, and systems-aware CMoE rather than treating them as interchangeable.

Another misconception is that sparse expert routing automatically yields efficient inference. A controlled vision study finds that naively implemented SparseMoE heads do not deliver inference speedups on modern hardware, despite theoretical expert-FLOP reductions, because routing, selection, and dispatch overhead dominate at that scale (Rokah et al., 21 Jan 2026). This systems caveat is consistent with the need for explicit co-design in GPU–NDP CMoE (Fan et al., 4 Dec 2025) and with lookup- or cache-constrained retrieval designs such as MoLKV (Wang, 10 Dec 2025).

Several limitations recur across domains. MoVA relies on offline routing annotations generated by evaluating multiple expert-specific models per sample and caps the number of selected experts at three, which is computationally practical but not fully end-to-end (Zong et al., 2024). LLaVA-CMoE assumes task distributions are sufficiently separable for VAE-based router identification; overlapping tasks reduce PTL accuracy (Zhao et al., 27 Mar 2025). The industrial cMoE requires the analyst to define contexts and possibility distributions a priori (Souza et al., 2022). Contrastive CMoE for robotics depends on elevation-map quality and does not provide a full ablation over expert count or contrastive weighting (Ma et al., 3 Mar 2026). Dense-to-MoE carving relies on heuristic assumptions about neuron activation magnitude and provides stronger evidence on quality preservation than on end-to-end acceleration (Pei et al., 6 Feb 2025).

The broader research direction suggested by these papers is that “context” in CMoE is expanding. It can mean multimodal instruction context, positional context, process regime knowledge, inferred task distribution, runtime expert popularity, hardware noise sensitivity, or edge-server availability. This suggests that future CMoE systems may need hierarchical routing in which semantic context decides what expertise is needed, while systems context decides where and at what precision that expertise should run.