Semantic-Anchor Expert Routing

• Semantic-anchor-guided expert routing is a method that directs sparse MoE expert selection using meaning-bearing signals, ensuring semantically similar inputs share expert pathways.
• The approach employs controlled probes and chance-corrected metrics to demonstrate that semantic cues, rather than mere token properties, drive expert selection.
• It has broad applications across modalities such as VQA, multilingual LLMs, and graph learning, enhancing both interpretability and adaptive routing performance.

Semantic-anchor-guided expert routing refers to mixture-of-experts routing schemes and analyses in which expert selection is constrained, biased, or evaluated with respect to meaning-bearing signals rather than only token identity, position, node type, or other surface cues. Recent work uses related labels such as semantic routing, semantically guided expert routing, concept-guided routing, semantic-driven expert routing, semantically anchored routing alignment, and functionally guided expert routing, but the common premise is consistent: semantically similar inputs should traverse overlapping or aligned expert pathways, whereas semantically different inputs should not (Olson et al., 15 Feb 2025, Zeng et al., 18 Apr 2026, Dong et al., 24 Jun 2026).

1. Conceptual foundation

Mixture-of-Experts (MoE) architectures activate only a small number of experts per token, enabling very large total parameter counts at limited compute. Within that setting, semantic-anchor-guided routing addresses a longstanding question: whether routers partition computation by meaning, rather than merely by lexical form, position, or other shallow regularities. The issue was sharpened by studies arguing that routing is often token-dependent or tied to domains and topics, and by later work showing that context and semantic association can nevertheless modulate routing, especially in encoder layers (Arnold et al., 2024, Olson et al., 15 Feb 2025).

A canonical formulation is given in the probe of large open-domain MoEs. If two inputs activate routed expert sets $A$ and $B$, each of size $k$, expert overlap is $o = |A \cap B|$. Because models differ in total experts $N$ and active experts $k$, raw overlap is not directly comparable. The chance-corrected normalization introduced there uses

$$\mathbb{E}[o] = \frac{k^2}{N}, \qquad P_o = \frac{o}{k}, \qquad P_e = \frac{k}{N},$$

and defines

$$\text{score} = \frac{o - \mathbb{E}[o]}{k - \mathbb{E}[o]} = \frac{P_o - P_e}{1 - P_e}.$$

This makes overlap interpretable relative to chance and comparable across architectures (Olson et al., 15 Feb 2025).

Earlier evidence from sparsely gated Switch Transformer models already indicated that routing in encoder layers mainly depends on semantic associations, with contextual cues providing an additional layer of refinement, while decoder routing is more variable and markedly less sensitive to context. In that study, routing similarity correlated more strongly with WordSim association than with SimLex semantic similarity, context increased encoder correlation on SCWS from 0.2773 to 0.3785, and WiC produced a bimodal separation between identical-context and different-context uses of the same word (Arnold et al., 2024). Taken together, these results established the modern problem setting: semantic-anchor-guided routing is not simply about sparse selection, but about whether sparse selection tracks semantic structure.

2. Evidence for semantic routing in large language-model MoEs

Direct evidence for semantic routing in large MoEs comes from controlled probes that hold lexical form fixed while varying meaning, or hold context fixed while varying lexical substitution. In the Word-in-Context experiment, six MoE models from three families—DeepSeek-R1, DeepSeek-V2-Lite, Mixtral-8x7B, Mixtral-8x22B, Llama-4-Scout, and Llama-4-Maverick—were prompted to define a target word in context, and router overlap was compared between same-sense and different-sense sentence pairs. All models showed significantly higher overlap when the target word had the same sense than when it had different senses, with significance at $p < 0.001$ when averaged across layers. The semantic routing signal was strongest in intermediate layers, the difference in overlap tended to increase with model size, and Llama-4 Scout was noted as an exception because it showed a pronounced effect despite a smaller total parameter count (Olson et al., 15 Feb 2025).

A complementary substitution-based probe fixed the sentence context and replaced the target word with semantically similar or dissimilar alternatives using SWORDS. Here the key comparison was between the original sentence and an equivalent substitution versus the original sentence and a different substitution. The reported result was strong evidence to reject the null for all six models, with $p < .0001$ for every model under a paired $B$0-test whose alternative hypothesis was that the equivalent pair had higher overlap. This provided a second line of evidence that routing tracks semantic similarity rather than just lexical identity (Olson et al., 15 Feb 2025).

These results directly counter a strong token-identity account of expert selection. The paper’s formulation is explicit: if routing were random, overlap should be close to $B$1; if routing is semantically structured, same-meaning inputs should overlap more than chance and more than meaning-changing inputs. A plausible implication is that sparse expert selection can expose how models assign shared computational pathways to shared meaning.

3. Anchor construction and routing mechanisms

In intervention-oriented systems, semantic anchors are not merely diagnostic; they are injected into the routing process. In CoGR-MoE for multiple-choice VQA, the anchor is a cue-derived semantic direction built from positive and negative answer cues. Positive cues describe what should be visible if an option is correct, while negative cues describe what would contradict it. The correct answer’s cues define

$B$2

which is injected into teacher-router logits during training,

$B$3

A cue-free student router uses $B$4, and the teacher routing is distilled with

$B$5

Once a shared Top-$B$6 expert set is selected, each option uses its own cue-derived signal to reweight those same experts, yielding what the paper calls consistent selection and flexible reasoning (Zeng et al., 18 Apr 2026).

In multilingual sparse LLMs, SARA treats a high-resource language as the semantic anchor and aligns low-resource routing to it with a symmetric Jensen–Shannon divergence over dense routing distributions. For token representation $B$7, routing probabilities are computed as $B$8. Sequence-level routing profiles are averaged over non-padding tokens and aligned with

$B$9

The method differs from output-logit distillation because it directly aligns the internal routing distributions of MoE layers, encouraging what the paper terms mechanistic consistency across languages (Dong et al., 24 Jun 2026).

Other systems define anchors in task-specific ways. GLIDER builds a semantic global router from an LLM-generated task instruction, embeds that instruction with nomic-embed-text-v1.5, compares it to stored expert global routing vectors, and combines the resulting score with a learned local token-level router through

$k$0

This makes the task-level instruction a global semantic anchor for expert retrieval (Li et al., 2024). STFER for Any-Time Person ReID uses LVLM-generated identity-intrinsic semantic text as a stable identity reference, pools it into a global vector, concatenates it with a scenario CLS token, and feeds both into a gating network, so that routing is conditioned on semantic identity text rather than current visual features alone (Li et al., 16 Apr 2026). In Soft MoE vision transformers, a different anchor appears: a pseudo-foreground mask obtained from Grounding DINO and SAM supervises dispatch weights through an IoU-like auxiliary loss, explicitly aligning high-dispatch regions with semantic foreground (Min et al., 24 May 2025).

4. Modalities and application patterns

Semantic-anchor-guided routing is not limited to language-model probing. It now appears across VQA, multilingual language modeling, person re-identification, referring image segmentation, heterogeneous graph learning, EEG-based language decoding, and post-hoc model MoErging. The anchors differ—answer-option cues, high-resource language traces, identity text, referring expressions, stochastically masked type embeddings, anatomically defined brain regions, or generated task instructions—but the recurring design aim is to make expert selection reflect latent semantics rather than brittle surface labels.

Setting	Anchor or guidance signal	Routing role
Multiple-choice VQA	Positive and negative answer cues	Teacher routing and option-specific reweighting
Multilingual sparse LLMs	High-resource language routing prior	Cross-lingual routing alignment
Any-Time Person ReID	LVLM-generated identity text	Semantic prior for scenario-aware gating
Referring image segmentation	Referring expression	Spatio-semantic expert weighting
Heterogeneous graph transformers	Stochastically masked type embeddings	Discourage type-to-expert lookup
EEG language decoding	Anatomical and functional brain partition	Functionally guided expert fusion
Model MoErging	LLM-generated task instruction	Global expert retrieval

In VQA, CoGR-MoE achieved the best overall accuracy on both reported MRAG-Bench backbones, reaching 63.25 on MOE-LLaVA and 68.96 on Qwen3-VL-A3B-30B, with especially strong gains in Perspective and Transformative categories. The paper also reports more concentrated routing, with higher routing sharpness and lower routing variance than MoE-LLaVA (Zeng et al., 18 Apr 2026). In person re-identification, STFER’s ablation study reported 55.80 R1 / 41.38 mAP for a baseline visual-only model, 93.79 R1 / 92.80 mAP with text + SER, and 94.54 R1 / 93.46 mAP with all components, supporting the claim that semantic guidance improves routing under clothing change and RGB/IR shifts (Li et al., 16 Apr 2026).

In heterogeneous graph learning, HER replaces type-separated experts with a shared expert pool and stochastic type masking. On link prediction, best SharedMoE scores reached 92.63% ROC-AUC on IMDB at $k$1, 84.97% on ACM at $k$2, and 91.99% on DBLP at $k$3, outperforming both HGT and SeparatedMoE. The IMDB analysis further reported genre-biased experts such as Expert 9 with 60% Comedy and Expert 12 with 60% Drama, which the paper interprets as specialization by semantic pattern rather than node type (Shakirov et al., 10 Nov 2025). In EEG decoding, BrainStack uses seven regional experts plus one global expert and a learnable softmax gate over expert outputs; reported average accuracy was 41.87% for BrainStack, compared with 37.19% for BrainStack_RoI5 and 32.78% for BrainStack_Homo, indicating that functionally partitioned experts plus adaptive routing were beneficial in that benchmark (Zhao et al., 29 Jan 2026).

5. Interpretability, trajectories, and reasoning control

A major interpretability shift in this literature is the move from individual experts to routing trajectories. In the control-theoretic decomposition of MoEs, each residual state $k$4 is split exactly into a router-visible component $k$5 and a router-blind component $k$6 using the router matrix’s row space: $k$7 Only $k$8 can causally influence expert selection. Across six architectures, the visible component predicted the current top-1 expert at about $k$9 accuracy but predicted the next layer’s expert poorly, at about $o = |A \cap B|$0, while the blind component was the strongest predictor of the next layer’s routing, at roughly $o = |A \cap B|$1. The same work reported a Pearson correlation of about $o = |A \cap B|$2 between per-dimension hidden-state magnitude and router-weight magnitude, and found that the top 2\%-5\% highest-magnitude dimensions were sufficient to predict top-1 expert choice surprisingly well (Ye et al., 20 Apr 2026).

The interpretive consequence is that individual experts remain polysemantic, but multi-layer expert paths become much more semantically coherent. A path is defined as

$o = |A \cap B|$3

Tokens sharing a path clustered by semantic function across languages and surface forms, and the token “:” followed distinct trajectories depending on whether it served as a type annotation, an introductory colon, or a time separator. Clustering in the control subspace was substantially more monosemantic than clustering in the full representation: control clusters had about 4.3 unique token IDs per cluster, whereas content clusters had about 1.1–1.4 (Ye et al., 20 Apr 2026). This reframes semantic-anchor-guided routing: the natural unit of interpretability is often the trajectory, not the expert.

A related line of work uses routing states as a signal for reasoning control. The central claim is that the same token id does not imply the same router state. Holding anchor tokens fixed, routing still separated problem identity, trajectory history, and reasoning-effort mode. On Qwen EOS, the same <|im_end|> token on BRUMO25 clustered by problem identity with under 5% cross-problem leakage; on OSS boundary anchors, the same final-channel token triplet clustered by reasoning-effort mode with under 1% contamination across Low, Medium, and High (Chen et al., 22 Jun 2026). Routing Agreement Decoding (RAD) operationalizes this by locating a fixed anchor, averaging sparse routing vectors over a window,

$o = |A \cap B|$4

comparing rollouts with Weighted Jaccard, and selecting the densest route-basin center. On the pooled math+GPQA setting, the reported averages were Majority 73.6, RAD 73.9, and RAD+DC 74.2; on code, strict answer-content-free delimiter@marker still reached 65.3, which was +4.0 pp over Avg@64; and on SWE-bench Verified, boundary-anchored RAD improved best-of-16 patch selection by +4.7 pp, +4.8 pp, and +5.0 pp on the decidable subset for the three reported models (Chen et al., 22 Jun 2026).

6. Constraints, recurring trade-offs, and open questions

The literature does not present semantic-anchor-guided routing as a universal property of all MoEs. The large-scale probing study explicitly notes that it covers only a small set of publicly available MoE models, cannot fully disentangle architecture effects from training-regime effects, and probes only a particular slice of the MoE design space (Olson et al., 15 Feb 2025). The encoder–decoder contrast in Switch Transformer routing likewise shows that context sensitivity is architecture-dependent: encoder routing is semantically structured and context-refined, whereas decoder routing is weaker and more variable (Arnold et al., 2024).

A recurring design trade-off is between stable selection and adaptive differentiation. CoGR-MoE formulates this directly: unstable routing can assign different experts to semantically similar questions, while overly stable routing can make all options use nearly the same expert pool and thereby weaken option-level discrimination. Its solution is to keep the Top-$o = |A \cap B|$5 expert set semantically anchored while allowing option-specific reweighting inside that set (Zeng et al., 18 Apr 2026). HER reports an analogous balance in graph learning: moderate stochastic masking of type embeddings works best, but too much masking hurts, and full masking at $o = |A \cap B|$6 collapses performance badly (Shakirov et al., 10 Nov 2025).

Anchor quality is another persistent limitation. STFER notes that LVLMs may generate homogenized neutral descriptions for low-quality images, implying that semantic text can become too generic and routing can lose discriminative power (Li et al., 16 Apr 2026). The foreground-guided Soft MoE method depends on the quality of Grounding DINO + SAM masks, and reports that background guidance is worse and that auxiliary supervision works best on the last MoE layer (Min et al., 24 May 2025). SARA similarly shows that anchor reliability matters: English anchors were stronger than Chinese anchors in that study, and aligning all layers or random layers was worse than aligning selected intermediate layers where routing divergence exhibited a U-shaped depth profile (Dong et al., 24 Jun 2026).

The interpretability debate is also unsettled. One view treats semantic specialization as a property of experts; another, supported by control-subspace analyses, holds that experts are often polysemantic and that semantic coherence emerges primarily at the level of routes through experts (Ye et al., 20 Apr 2026). A plausible implication is that future semantic-anchor-guided routing research will continue to shift from single-expert descriptions toward layerwise path analysis, routing-level alignment objectives, and anchor designs that shape not only which experts are selected, but also how shared expert subsets are reused across contexts, languages, and modalities.