---
title: Input-Aware Extendable Expert Module
url: https://www.emergentmind.com/topics/input-aware-extendable-expert-module
type: topic
---

# Input-Aware Extendable Expert Module

Searching arXiv for the cited and closely related work on input-aware, extendable expert modules and MoE-style routing.
An input-aware extendable expert module is a modular computation scheme in which multiple experts are activated, fused, or selected according to properties of the current input, while the expert set can be expanded, replaced, merged, or pruned without redesigning the entire system. Across recent work, the term covers several distinct but related constructions: sparse experts embedded inside a seed large language model, inference-time rerouting layers for Mixture-of-Experts decode, manifold-defined experts in continual model merging, isolated per-domain heads on frozen representations, tool- or server-backed experts accessed through protocol layers, and variable-context or budget-aware experts whose active capacity changes with the sample [2506.12597][2511.02237][2604.22464][2607.01674][2505.01834][2606.24062][2511.18314].

## 1. Definition and conceptual scope

The defining property is input dependence. In "Automatic Expert Discovery in LLM Upcycling via Sparse Interpolated Mixture-of-Experts," SIMoE fine-tunes a dense pre-trained LLM into a MoE-style model that automatically identifies multiple specialized experts under a specified sparsity constraint, and simultaneously learns an input-dependent expert merging strategy via a router network [2506.12597]. In other systems, the input-aware signal is not a conventional neural gate: MADE-IT activates experts through feature–subspace alignment without a parameterized gating network, OEA reroutes token-to-expert assignments using the current batch-step, and DMoE triggers expert activation only when token-level uncertainty exceeds a threshold during autoregressive generation [2604.22464][2511.02237][2606.14243].

The second defining property is extendability. Some modules add experts by appending new masks or adapters to an existing model, some grow repositories of subspace experts over arriving tasks, and some register entirely external expert services through a protocol interface [2506.12597][2604.22464][2505.01834]. In the ECG setting, extendability is realized by freezing a backbone and incrementally appending per-source balanced-softmax linear heads; in multimodal MoE, it can take the form of per-token variable expert slots filled by real or virtual experts [2607.01674][2511.18314].

This suggests that the term denotes a design pattern rather than a single architecture. What remains invariant is the separation between a shared substrate and a set of specialized modules whose participation is conditioned on the input, current batch, uncertainty state, feature geometry, or deployment context.

## 2. Architectural forms of experts

Recent systems instantiate experts in markedly different parameterizations. SIMoE treats each expert as a structurally sparse subset of the seed LLM’s parameters, while DMoE uses independently trainable LoRA-style adapters attached only to the final-layer feed-forward network, and AnyExperts distinguishes between trainable real experts and identity virtual experts [2506.12597][2606.14243][2511.18314]. MADE-IT instead models each modular expert as a rank-$r$ principal subspace extracted from task-specific weight updates, and IRFE-ECG uses per-domain linear two-class heads over frozen 1024-dimensional ECGFounder features [2604.22464][2607.01674].

At the module level, experts may also be structurally heterogeneous. CBDES MoE replaces a static camera backbone with four ImageNet-pretrained backbones—Swin Transformer, ResNet, ConvNeXt, and PVT—selected by a lightweight Self-Attention Router [2508.07838]. In video-based person re-identification, each expert contains a mapping block, a relevance evaluator, and channel-wise spatial or temporal branches selected by Improved Semhash [2607.01353]. In MEPG, experts are diffusion branches specialized for style or realism and are activated per spatial region rather than per token [2509.04126].

The same pattern extends beyond parametric modules. In the MCP-based Internet of Experts, each expert is an external deterministic classifier exposed as an MCP tool with a name, description, input schema, and JSON-RPC response format; the LLM remains frozen and queries these expert servers at inference time [2505.01834]. This externalized construction places the expert boundary at the system interface rather than inside a single neural network.

| Work | Expert representation | Activation mechanism |
|---|---|---|
| SIMoE | Structurally sparse subsets of seed LLM parameters | Input-dependent router merges experts [2506.12597] |
| MADE-IT | Rank-$r$ principal subspaces from $\Delta W$ | Feature Projection Alignment and dependency graph [2604.22464] |
| IRFE-ECG | Balanced-softmax linear heads on frozen 1024-D features | MLP routing or top-2 margin fusion [2607.01674] |
| DMoE | Final-layer FFN LoRA adapters | BM25 retrieval triggered by uncertainty [2606.14243] |
| IoX | External MCP expert servers | LLM planning over tool metadata [2505.01834] |
| CBDES MoE | Heterogeneous camera backbones | Self-Attention Router, soft train / top-1 infer [2508.07838] |
| AnyExperts | Real experts plus virtual identity experts | Per-token slot allocation under budget bounds [2511.18314] |

## 3. Routing, fusion, and conditional computation

Routing mechanisms differ chiefly in what signal they condition on and what object they select. In classical token-centric MoE decode, OEA models MoE-layer latency as
$$
T_{decode} \approx b \cdot T + a \cdot Bk,
$$
where $T$ is the number of unique activated experts, $B$ is batch size, and $k$ is experts per token; the method then reduces latency by lowering $T$ through batch-aware piggybacking while keeping a per-token baseline set of top-$k_0$ experts [2511.02237]. This routing is input-aware at the batch-step level rather than through retraining.

In manifold-based continual merging, routing can be entirely training-free. MADE-IT projects the backbone feature $h_c$ onto each expert’s principal input subspace and scores compatibility by
$$
M(h_c, E_c^{(k)}) = \frac{h_c^\top \hat{h}_c^{(k)}}{\|h_c\|_2 \cdot \|\hat{h}_c^{(k)}\|_2},
$$
with top-1 selection per module and a dependency graph enforcing path consistency across modules [2604.22464]. In DMoE, routing is not continuous mixture weighting but uncertainty-triggered activation: token uncertainty is
$$
H_t = -\sum_i p_i \log p_i,
$$
and the router retrieves top-$k$ experts by BM25 only when $H_t > \tau$ [2606.14243].

Several systems use dynamic fusion rather than discrete selection. RAVEN scores patches in reverse chronological order, builds cumulative importance
$$
\Psi_i = \sum_{j=1}^{i} \tilde{s}_j,
$$
and derives nested prefix sets
$$
G_k = \{ i : \Psi_i \le \tau_k \},
$$
so that each scale-specialized expert receives a sample-specific look-back horizon; Correlation-Aware Weighting then penalizes redundant expert outputs before aggregation [2606.24062]. AnyExperts maps token importance to a variable number of expert slots,
$$
\hat{S}_i = K_{min} + (K_{max} - K_{min}) \cdot w_i,\qquad S_i = \mathrm{Round}(\hat{S}_i),
$$
and fills those slots with a bounded mixture of real and virtual experts under a virtual-share cap [2511.18314].

Routing can also be decentralized. DQRoute assigns no centralized gate; instead, each expert produces its own confidence or OOD score, and expert weights are obtained by normalizing those scores, while training is simultaneously modulated by class-wise difficulty estimates derived from entropy and historical accuracy [2508.19630]. In protocol-based systems, routing reduces to tool planning: the LLM uses expert metadata and the current query to decide which MCP tools to invoke, then folds returned confidences into its reasoning context [2505.01834].

## 4. Mechanisms of extension, growth, and maintenance

Extendability is realized through several distinct lifecycle operations. In SIMoE, upcycling reuses a single seed model, and adding experts does not duplicate the full parameter set; experts are formed as structured sparse views of the seed parameters and can be extended by introducing new masks and small router parameters [2506.12597]. DMoE similarly decouples expert growth from the base model: new knowledge is inserted by training a new LoRA adapter for a passage-level knowledge unit, saving its text surrogate, and inserting that surrogate into the BM25 index, with no change to shared base parameters [2606.14243].

MADE-IT formalizes expert evolution through module-wise subspace affinity and an adaptive threshold
$$
\gamma_A^{(t)} = \mu^{(t)} + \beta \sigma^{(t)}.
$$
If the maximal affinity of a candidate subspace is below $\gamma_A^{(t)}$, a new expert is spawned; otherwise the candidate is merged into the closest existing expert and re-truncated to rank $r$, which functions as effective pruning of redundant directions [2604.22464]. This combines growth, merge, and pruning in a single expert-management rule.

Other systems use simpler append-only schemes. IRFE-ECG adds one new balanced-softmax linear expert for each arriving ECG source, freezes all earlier experts and the backbone, and refits only the lightweight router on retained frozen training features and domain labels from sources observed so far [2607.01674]. In the Re-ID setting, each layer maintains a wait-list expert; when the wait-list expert exhibits higher relevance than the existing experts for input samples, it is appended to the constant expert set and a fresh wait-list expert is instantiated [2607.01353].

Protocol-based and modular-serving systems externalize extension into registration interfaces. IoX adds new experts by packaging an MCP tool endpoint and registering its name, description, input schema, and version in the MCP registry, after which the LLM can discover and invoke it at runtime without parameter updates [2505.01834]. CBDES MoE likewise remains modular at functional-module granularity: experts share an aligned interface to the camera-to-BEV projector, and the router output can be expanded to accommodate an additional backbone expert while preserving the downstream BEV pipeline [2508.07838].

## 5. Representative empirical outcomes

Because these modules optimize different objectives—accuracy, robustness, latency, continual retention, or compute efficiency—their reported gains are not directly comparable. They nevertheless show that input-aware expert selection is useful across domains.

| Setting | Representative outcome | Citation |
|---|---|---|
| LLM upcycling with sparse experts | Consistently achieves state-of-the-art on common instruction-tuning benchmarks and maintains an optimal performance-compute trade-off | [2506.12597] |
| Batch-aware MoE decode rerouting | Up to 39% reduction in MoE layer decode latency on Qwen3-30B and 15% on Qwen3-235B without statistically significant loss in accuracy | [2511.02237] |
| Continual model merging | On CLIP ViT-B/32 with 20 tasks, ACC = 81.4%, surpassing MINGLE by 4.3% and OPCM by 15.7%; reduction rates approach about 95% in generic modules | [2604.22464] |
| Replay-free continual ECG deployment | Source-aware expert selection reaches $0.7915\pm0.0036$ Macro-F1, while top-2 margin fusion without source IDs reaches $0.7782\pm0.0022$ | [2607.01674] |
| Wireless environment-aware LLM agents | 40%-50% improvements in classification tasks over LLM-only baselines across multiple mainstream LLMs | [2505.01834] |
| Financial time-series forecasting | Pearson correlation improves by 9.2% on HS300 and 20.2% on S&P500, and MSE is reduced by 18.2% on fund sales forecasting | [2606.24062] |
| Budget-aware multimodal MoE | Comparable accuracy with 40% fewer real expert activations on general image/video tasks, and about 10% reduction on OCR and NLP while maintaining performance | [2511.18314] |
| Autonomous driving BEV perception | 1.6-point increase in mAP and 4.1-point improvement in NDS over the strongest single-expert model | [2508.07838] |
| Video-based person Re-ID | MARS: mAP 87.0%, Rank-1 91.6%; LS-VID: mAP 81.0%, Rank-1 88.3% | [2607.01353] |

A further pattern is that some modules primarily improve systems efficiency rather than benchmark accuracy. DMoE reports average latency of 2.6656 s and GPU memory of 7.2352 GB, compared with 9.2643 s and 13.9718 GB for FLARE, while preserving KV-cache reuse by attaching experts only to the final-layer FFN [2606.14243]. Klotski operates one level lower, at the inference-engine layer, and uses expert-aware multi-batch scheduling, a constraint-sensitive planner, and correlation-aware prefetching to improve throughput by up to 85.12x under offloading constraints [2502.06888].

## 6. Limitations, misconceptions, and open problems

A common misconception is that an input-aware extendable expert module is simply a standard top-$k$ MoE layer. Recent work shows a much broader landscape: routing may be training-free and geometric, entirely inference-time and batch-aware, or mediated by protocol calls to external tools rather than by a learned gate inside a monolithic network [2604.22464][2511.02237][2505.01834]. Another misconception is that extendability implies no memory burden. IRFE-ECG is explicit that the method is replay-free at the raw-ECG level but not memory-free because frozen training features are retained for router updates [2607.01674]. DMoE likewise shows that modular knowledge banks can be storage-heavy: its example bank contains 27,613 experts with a disk footprint of about 13.08 GiB [2606.14243].

Theoretical and systems constraints remain substantial. "Learning-to-Defer with Expert-Conditioned Advice" argues that separated router/query surrogates are inconsistent even in the smallest non-trivial setting, and instead advocates learning over the composite expert–advice action space; the same work notes that training may require executed-pair cost tables and that complexity scales with $J(K+1)$ [2603.14324]. AnyExperts demonstrates that variable expert allocation must still be budgeted by hard bounds and a virtual-share cap, with $\rho_{max}=0.2$ emerging as a useful operating point and larger values harming some NLP settings [2511.18314]. DQRoute depends on calibratable confidence or OOD scores across experts, and its difficulty estimates assume that entropy and historical performance remain meaningful after EMA smoothing [2508.19630].

This suggests three open directions. First, expert modules increasingly need joint treatment of routing, calibration, and systems scheduling rather than architecture alone. Second, extension policies are moving from append-only growth toward adaptive growth–merge–prune cycles. Third, the boundary of an “expert” is becoming less stable: it may be a sparse parameter mask, a low-rank adapter, a linear head, a subspace, a whole backbone, or an external service. The modern input-aware extendable expert module is therefore best understood as a general conditional-computation framework for specialization under evolving data, compute, and deployment constraints.

Source: https://www.emergentmind.com/topics/input-aware-extendable-expert-module