---
title: Entropy-Aware Routing (EAR)
url: https://www.emergentmind.com/topics/entropy-aware-routing-ear
type: topic
---

# Entropy-Aware Routing (EAR)

Entropy-Aware Routing (EAR) denotes a class of routing mechanisms in which uncertainty, typically quantified by entropy or by an explicitly stated proxy for entropy, governs how computation, memory access, or expert activation is allocated. Across recent work, EAR appears in several distinct forms: token-wise expert-count selection in Mixture-of-Experts (MoE) large vision-language models, difficulty-biased routing in heterogeneous xLSTM MoE language models, entropy-regularized model selection among experts of different capability and cost, entropy-triggered specialization in multimodal recommendation, uncertainty-aware fallback in hierarchical memory systems, and entropy-weighted blending with shared experts in multi-domain speech recognition [2606.26287], [2505.01459], [2606.29424], [2602.20723], [2603.10032], [2606.10454]. The common premise is that routing should not remain static when the input-dependent uncertainty of the router varies substantially.

## 1. Conceptual scope and formal definitions

In the broadest sense, EAR treats routing as an uncertainty-sensitive decision problem. A router receives an input $x$ and produces either a distribution over experts, models, or memory candidates, or a decision rule derived from that distribution. The entropy of that distribution is then used to decide whether to broaden computation, sharpen selection, invoke fallback mechanisms, or mix in shared capacity.

The most explicit formulation appears in GeMoE, where the router for an MoE layer produces logits $R(x)=Wx$ over $K$ experts, followed by a softmax routing distribution
$$
R_{\text{norm}}(x)_i=\frac{\exp(R(x)_i)}{\sum_{j=1}^K \exp(R(x)_j)}.
$$
If $p \equiv R_{\text{norm}}(x)=\{w_1(x),\dots,w_K(x)\}$, the token-level gating entropy is
$$
H(p)=H(x\mid R)=-\sum_{i=1}^K w_i(x)\log w_i(x),
$$
with log base 2 used in that paper’s experiments [2606.26287]. Low entropy corresponds to a peaked routing distribution and is interpreted as an “easy” or unambiguous token; high entropy corresponds to a flatter distribution and is interpreted as a “hard” or ambiguous token.

A related but distinct construction appears in MoxE, where entropy is not used to choose the number of active experts but to train a difficulty predictor aligned with the normalized entropy of the unbiased router distribution:
$$
\tilde H_t=\frac{H(\tilde p_t)}{\log E}\in[0,1].
$$
That difficulty estimate then biases routing toward mLSTM experts for hard or rare tokens [2505.01459].

EntroRouter generalizes EAR beyond MoE layers to model selection. Given a router policy $\pi_\theta(m\mid x)$ over experts $m\in E$, it introduces an entropy-aware objective
$$
J(\pi)=\mathbb E_{x\sim D}\mathbb E_{m\sim \pi(\cdot\mid x)}[R_\lambda(x,m)]
+\beta \,\mathbb E_x[H(\pi(\cdot\mid x))]
-\alpha \,\mathbb E_x[\mathrm{KL}(\pi(\cdot\mid x)\Vert q(\cdot\mid x))],
$$
where entropy and KL terms explicitly regulate exploration, exploitation, and trust-region behavior [2606.29424].

Not all systems labeled EAR compute Shannon entropy directly. HTM-EAR operationalizes uncertainty through a two-signal gate: the top-1 similarity score from L1 memory and entity coverage of the top candidate. The paper also gives an optional entropy formalization over normalized $k$-NN similarities, but states that the reported experiments use the simpler similarity-and-coverage gate [2603.10032]. This distinction is important: EAR refers to uncertainty-aware routing more generally, but explicit entropy computation is only one implementation.

## 2. Theoretical rationales for entropy-aware routing

Several papers supply distinct formal rationales for why entropy should guide routing. In GeMoE, dynamic routing is framed as a Minimum Description Length (MDL) problem. For token $x$ and selected subset of experts $E' \subseteq E$, the total description length is
$$
L(x,E')=L(E')+L(x\mid E'),
$$
with
$$
L(E')=k\cdot c,\qquad k=|E'|,
$$
and
$$
L(x\mid E')=-\log \sum_{i\in E'} w_i P_i(y\mid x).
$$
Adding an expert $E_m$ is justified when
$$
\log\!\left[\frac{P(y\mid x,E'_{\text{new}})}{P(y\mid x,E')}\right]>c.
$$
Because evaluating candidate experts directly is expensive, the paper argues that gating entropy is a practical proxy for the information gain term: when entropy is high, additional experts tend to have non-negligible router weights and are more likely to produce gain above the per-expert complexity penalty [2606.26287].

MoxE provides a different mechanism. There, the router learns a difficulty score
$$
d_t=\sigma(w_D^\top h_t+b_D),
$$
and applies an additive routing bias
$$
\delta_{t,i}=
\begin{cases}
+\gamma d_t,& i\in \text{mLSTM group}\\
-\gamma d_t,& i\in \text{sLSTM group}.
\end{cases}
$$
Under the approximation of balanced unbiased logits across groups, the probability ratio obeys
$$
\frac{P(\text{mLSTM}\mid d_t)}{P(\text{sLSTM}\mid d_t)}\approx \exp(2\gamma d_t),
$$
so routing to mLSTM grows exponentially with difficulty [2505.01459]. Here entropy serves as a supervisory target for difficulty estimation rather than a direct routing statistic at inference.

EntroRouter’s theoretical contribution is tied to a failure mode it terms Trust Region Collapse (TRC). In multi-round routing, if escalation to stronger experts has positive cost and sparse supervision causes out-of-distribution expert traces to be ignored, the advantage of escalation becomes strictly negative, and the effective trust region for increasing escalation probability becomes empty. Entropy regulation and KL anchoring are then used to preserve exploration initially and contract entropy only within a safe region later in training [2606.29424]. In this formulation, EAR is not merely an efficiency heuristic; it is a stabilizing principle for policy optimization under sparse rewards.

MAGNET uses entropy in yet another way: not as an input-level difficulty score but as a batch-level monitor of routing diversity. Let $\bar{\boldsymbol{\pi}}$ denote the batch-mean dense routing distribution; then
$$
H(\bar{\boldsymbol{\pi}})=-\sum_{e=1}^E \bar{\pi}(e)\log \bar{\pi}(e),\qquad
\tilde H=\frac{H(\bar{\boldsymbol{\pi}})}{\log E}\in[0,1].
$$
This normalized entropy drives a two-stage controller that delays specialization until routing diversity has stabilized [2602.20723].

## 3. EAR in MoE architectures and model routing

In MoE systems, EAR primarily modulates either the number of active experts or the type of experts chosen. GeMoE introduces an Expert Assignment Predictor (EAP), a lightweight linear layer that predicts the number of experts to activate per token. For an allowed set $N=\{k_{\text{low}},\dots,k_{\text{high}}\}$, it computes
$$
N(x)_i=\frac{\exp(P(x)_i)}{\sum_{j=1}^{|N|}\exp(P(x)_j)},\qquad
k_{\text{soft}}=\sum_{i=k_{\text{low}}}^{k_{\text{high}}} i\cdot N(x)_i,
$$
followed by
$$
k=\mathrm{Round}(k_{\text{soft}})
$$
with a straight-through estimator during training. A monotonicity loss $L_{\text{mono}}$ enforces that $k_{\text{soft}}$ increases with gating entropy, and the full training objective is
$$
L=L_{ce}+\alpha\cdot L_{\text{mono}}+\beta\cdot L_{lb}.
$$
Only the router and EAP are trained; experts remain native and are not fine-tuned [2606.26287].

Empirically, GeMoE is evaluated on MolmoE-1B-7B (64Top8) and DeepSeek-VL2-Tiny-1B-3B (64Top6), against DYNMoE, Top-p, AdaMoE, and MoE++. On MolmoE-1B-7B, static Top-8 gives Avg\_P $=63.99$, Avg\_k $=8.00$, and $N_A=1.58$B, whereas GeMoE gives Avg\_P $=63.61$, Avg\_k $=5.43$, and $N_A=1.32$B, corresponding to performance retention of about $99.4$–$99.5\%$ and $32.1\%$ fewer experts than Top-8. Across its reported settings, the paper states $99.5\%$ average performance retention with $36.5\%$ average expert activation sparsity improvement [2606.26287].

MoxE also belongs to the EAR family, but its mechanism is heterogeneous-expert selection rather than expert-count adaptation. It combines mLSTM and sLSTM experts in a sparse MoE with top-$k=2$ routing. High-entropy tokens are preferentially routed to mLSTM experts, which the paper associates with stronger recall via matrix memory, while low-entropy tokens are balanced across both expert types [2505.01459]. The total objective combines task loss with a difficulty loss $\mathcal L_d$, group-wise balancing loss $\mathcal L_{\text{group}}$, router Z-loss $\mathcal L_z$, and a load-balancing auxiliary loss:
$$
L_{\text{total}}=\mathcal L_{\text{task}}+\lambda_d \mathcal L_d+\lambda_{\text{group}}\mathcal L_{\text{group}}+\lambda_z \mathcal L_z+\lambda_{\text{aux}}\mathcal L_{\text{aux}}.
$$

A broader model-routing instantiation appears in EntroRouter. Its Stage I Soft Supervision fits the router to a high-entropy target
$$
Q(m\mid x)=g(x)\cdot \mathbf 1[m=m_{\text{best}}(x)]
+(1-g(x))\cdot \frac{\mathbf 1[m\in M_{\text{top}k}(x)]}{k},
$$
where $m_{\text{best}}(x)$ is the most economical qualified expert and $M_{\text{top}k}(x)$ are the $k$ cheapest qualified experts. Stage II then performs reinforcement learning with a soft anchor
$$
\pi_{\text{ref}}(m\mid x)=
\frac{\exp(\mu(m,x)/T)}{\sum_{m'}\exp(\mu(m',x)/T)},\qquad
\mu(m,x)=\kappa(m,x)-\alpha C(m),
$$
stabilizing training and controlling entropy contraction [2606.29424].

The following comparison summarizes these variants.

| System | Routing signal | Main routing action |
|---|---|---|
| GeMoE | Token-level gating entropy $H(p)$ | Adaptively choose expert count $k$ |
| MoxE | Difficulty aligned to normalized entropy $\tilde H_t$ | Bias hard tokens toward mLSTM experts |
| EntroRouter | Policy entropy with KL anchoring | Regulate exploration and model choice |
| MAGNET | Batch routing entropy $\tilde H$ | Trigger coverage-to-confidence transition |
| HTM-EAR | Similarity and entity-coverage gate; optional entropy formulation | Decide L1-only vs. L1+L2 fallback |
| Speech-LLM EAR | Normalized entropy over fine-domain probabilities | Blend domain-specific and shared experts |

## 4. EAR beyond standard MoE layers

EAR has also been applied to memory systems, recommendation, and speech recognition, where the routed objects are not merely feed-forward experts.

HTM-EAR introduces a hierarchical tiered memory substrate with HNSW-based working memory (L1, capacity $500$) and archival storage (L2, capacity $5000$). Queries first search L1 with $k_1=100$. Let $s_1$ be the top-1 similarity and $x_1$ the top candidate. If either $s_1<\tau_s$ with $\tau_s=0.84$ or entity coverage fails, retrieval also searches L2 with $k_2=200$, merges candidates, scores them by
$$
S_{\text{retrieve}}=\text{sim}^3+\lambda\cdot \text{overlap}+\gamma\cdot \text{importance},
$$
with $\lambda=0.8$ and $\gamma=0.1$, and re-ranks the top 20 with an MS MARCO cross-encoder [2603.10032]. The paper explicitly notes that although the title suggests entropy awareness, the implemented EAR uses similarity and entity-coverage proxies rather than explicit entropy. Under sustained saturation, the full model preserves active-query precision at MRR $=1.000$ and approaches oracle active performance $0.997\pm 0.003$; on BGL logs it achieves MRR $0.336$ versus oracle $0.370$, while LRU falls to $0.069$ [2603.10032].

In MAGNET for multimodal recommendation, EAR appears as “progressive Entropy-Triggered Routing.” The router produces dense expert probabilities
$$
\boldsymbol{\pi}_{ui}=\mathrm{Softmax}(\mathrm{Router}([z_u;z_i];\Theta_{\mathrm{rt}})),
$$
over a default expert pool of $E=9$ experts organized as $3$ modality anchors times $3$ expert families. Training begins in Stage 1 with a coverage regularizer
$$
\mathcal L_{\text{cov}}=\sum_{e=1}^{E}\left(\bar{\boldsymbol{\pi}}(e)-\frac{1}{E}\right)^2,
$$
weighted by
$$
\lambda_{\text{cov}}(H)=\mathbb I[\text{stage}=1]\cdot \lambda_r\cdot (1-\tilde H),
$$
and switches to Stage 2, where a confidence regularizer
$$
\mathcal L_{\text{conf}}=
\frac{1}{|\mathcal B|}\sum_{(u,i)\in\mathcal B} H(\boldsymbol{\pi}_{ui})
$$
is weighted by
$$
\lambda_{\text{conf}}(H)=\mathbb I[\text{stage}=2]\cdot \lambda_r\cdot \tilde H.
$$
The switch is triggered when normalized batch entropy exceeds a threshold $H^\ast$ for $W$ consecutive steps [2602.20723]. This formulation makes EAR a training-time controller for sparse routing stability and specialization.

In the Speech-LLM case study on unified child-adult ASR, EAR is applied to a classifier-based domain router with a coarse-to-fine structure. The coarse classifier outputs $p_c$, and the fine classifier over OGI-S age groups outputs $p_f$. Uncertainty is measured by normalized entropy
$$
H_{\text{norm}}(p)= -\frac{1}{\log n}\sum_{i=1}^n p(i)\log p(i),
$$
computed over the OGI-S fine-grained age-group probabilities with $n=3$. EAR then blends the routed output with a shared expert:
$$
\hat z_o=(1-H_{\text{norm}}(p))\cdot \hat z_o(\text{routing}) + H_{\text{norm}}(p)\cdot \hat z_o(\text{shared}).
$$
The same blending applies to both Mixture-of-Projectors and Mixture-of-LoRAs, while the encoder and LLM backbone remain frozen [2606.10454]. Reported gains are concentrated on OGI-S: under soft routing, OGI-S average WER improves from $11.31$ to $11.08$, while MyST remains $8.58\%$ and Libri-Clean remains $1.61\%$ [2606.10454].

## 5. Empirical regularities and efficiency trade-offs

Across domains, EAR is consistently presented as a way to improve the trade-off between performance and resource usage rather than as a mechanism for maximizing raw accuracy irrespective of cost.

In GeMoE, reducing the average number of active experts from $8.00$ to $5.43$ on MolmoE reduces inference FLOPs/token from $80.65$ to $68.45$ GFLOPs, increases throughput from $1709$ to $1828$ tokens/s, lowers wall-clock time per sample from $0.169$s to $0.152$s, and leaves memory usage unchanged at $32.95$ GB on A800-40G [2606.26287]. On DeepSeek 64Top6, GeMoE achieves Avg\_P $=70.90$ versus Top-6 $=71.29$ while using Avg\_k $=3.55$, a $40.8\%$ reduction in experts [2606.26287]. Notably, in generalization to OLMoE-1B-7B, GeMoE attains Avg\_P $=43.58$ with Avg\_k $=5.53$, exceeding Top-8 Avg\_P $=38.59$, and improves HumanEval from $28.66$ to $14.63$? No—the reported direction is GeMoE $28.66$ versus Top-8 $14.63$, indicating higher score under entropy-aware sparsity [2606.26287].

MoxE reports strong ablation sensitivity to the entropy-aware components. On Lambada perplexity, the full MoxE baseline gives $65{,}213.61$, whereas removing entropy bias ($\gamma=0$) yields $348{,}908.14$, removing group-wise loss yields $213{,}974.42$, using mLSTM-only experts gives $85{,}963.85$, and using sLSTM-only experts gives $191{,}161.87$ [2505.01459]. The paper therefore positions entropy-aware bias and balancing as central to stability and utilization.

EntroRouter reports a cost-accuracy result at the system level rather than within a single neural layer: it retains approximately $98.3\%$ of the strongest expert’s accuracy while reducing computational cost by approximately $48.25\%$ on math reasoning benchmarks [2606.29424]. Its ablations indicate that removing Soft Supervision causes a marked accuracy drop, removing the gate $g(x)$ increases cost, removing the Soft Anchor degrades the trade-off, and removing shaped rewards worsens stability and efficiency [2606.29424].

MAGNET reports that removing routing regularizers ($\lambda_r=0$) consistently reduces Recall@20 and NDCG@20 across Baby, Sports, Clothing, and Electronics. Coverage-only and confidence-only variants underperform the full two-stage regime, and a fixed-step switch lags the entropy-triggered switch [2602.20723]. The reported post-switch behavior is also structurally consistent: per-sample routing entropy decreases, top-1–top-2 probability gaps increase, and MaxLoad share remains controlled.

HTM-EAR and the speech-domain EAR show the same trade-off pattern in non-MoE settings. In HTM-EAR, the full system is slower than LRU but prevents uncontrolled forgetting: LRU has latency $21.08$ ms but permanently evicts $2416.4\pm 23.1$ essential facts, whereas the full system has latency $39.69$ ms and essential loss $0.0$ [2603.10032]. In Speech-LLM EAR, the added cost is small because entropy is computed over a 3-class vector and used only for a convex combination, while performance gains are concentrated on uncertain child-speech subdomains [2606.10454].

## 6. Limitations, misconceptions, and open directions

A recurrent limitation is calibration. If router probabilities are poorly calibrated, entropy may fail to represent true uncertainty. GeMoE identifies entropy miscalibration as a direct risk, noting that poor calibration may under- or over-allocate experts; its mitigation is joint router training with load balancing and a monotonicity loss using a dynamic margin $1.2\times$ the entropy gap [2606.26287]. The speech-domain EAR likewise depends on the quality of fine-grained classifier probabilities; if uncertainty is inflated, routing may overuse the shared expert [2606.10454].

A second misconception is that EAR always requires explicit Shannon entropy. HTM-EAR is the clearest counterexample: the reported implementation uses a similarity threshold and entity coverage gate, while an entropy gate over neighbors is presented only as a conceptual extension [2603.10032]. A plausible implication is that EAR is best understood as a design principle centered on uncertainty-aware routing, not as a single formula.

A third misconception is that more experts should always improve quality. GeMoE explicitly argues against this, and its OLMoE experiments are presented as evidence that entropy-aware sparsity can outperform denser activation [2606.26287]. EntroRouter makes the analogous point at the model-selection level: unrestricted exploration without staged entropy control can collapse to poor local optima, while controlled entropy contraction is needed for stable, decisive routing [2606.29424].

Several open directions are stated explicitly across the papers. EntroRouter suggests adaptive anchor scheduling, periodic re-profiling of offline capability estimates, hierarchical and multi-level routers, and theory for trust-region safety and convergence under entropy schedules [2606.29424]. MAGNET proposes learnable or adaptive expert templates and more robust handling of missing or noisy modalities [2602.20723]. HTM-EAR notes that fixed thresholds and heuristic parameters may misroute under domain shift, and that explicit entropy or calibration could better capture multimodal neighbor sets [2603.10032]. The speech-domain work points to extensions beyond clear hierarchies, where multiple ambiguous axes may require additional shared experts or richer routing structures [2606.10454].

In aggregate, EAR has become a unifying pattern across contemporary routing problems: use entropy, or a stated proxy for uncertainty, to allocate computation, memory access, or specialization adaptively rather than uniformly. The specific object being routed varies by domain, but the technical motif is stable: uncertainty should alter the routing policy itself.

Source: https://www.emergentmind.com/topics/entropy-aware-routing-ear