Input-Aware Extendable Expert Module

• Input-aware extendable expert modules are a design pattern that conditionally activates specialized computational units based on input properties.
• They employ diverse routing mechanisms like uncertainty and feature alignment to select, merge, or prune experts without redesigning the entire system.
• Empirical outcomes indicate these modules improve efficiency, accuracy, and latency across applications from language model upcycling to autonomous perception.

Searching arXiv for the cited and closely related work on input-aware, extendable expert modules and MoE-style routing. An input-aware extendable expert module is a modular computation scheme in which multiple experts are activated, fused, or selected according to properties of the current input, while the expert set can be expanded, replaced, merged, or pruned without redesigning the entire system. Across recent work, the term covers several distinct but related constructions: sparse experts embedded inside a seed LLM, inference-time rerouting layers for Mixture-of-Experts decode, manifold-defined experts in continual model merging, isolated per-domain heads on frozen representations, tool- or server-backed experts accessed through protocol layers, and variable-context or budget-aware experts whose active capacity changes with the sample (Chen et al., 14 Jun 2025, Oncescu et al., 4 Nov 2025, Qiu et al., 24 Apr 2026, Lu et al., 2 Jul 2026, Liu et al., 3 May 2025, He et al., 23 Jun 2026, Gao et al., 23 Nov 2025).

1. Definition and conceptual scope

The defining property is input dependence. In "Automatic Expert Discovery in LLM Upcycling via Sparse Interpolated Mixture-of-Experts," SIMoE fine-tunes a dense pre-trained LLM into a MoE-style model that automatically identifies multiple specialized experts under a specified sparsity constraint, and simultaneously learns an input-dependent expert merging strategy via a router network (Chen et al., 14 Jun 2025). In other systems, the input-aware signal is not a conventional neural gate: MADE-IT activates experts through feature–subspace alignment without a parameterized gating network, OEA reroutes token-to-expert assignments using the current batch-step, and DMoE triggers expert activation only when token-level uncertainty exceeds a threshold during autoregressive generation (Qiu et al., 24 Apr 2026, Oncescu et al., 4 Nov 2025, Yue et al., 12 Jun 2026).

The second defining property is extendability. Some modules add experts by appending new masks or adapters to an existing model, some grow repositories of subspace experts over arriving tasks, and some register entirely external expert services through a protocol interface (Chen et al., 14 Jun 2025, Qiu et al., 24 Apr 2026, Liu et al., 3 May 2025). In the ECG setting, extendability is realized by freezing a backbone and incrementally appending per-source balanced-softmax linear heads; in multimodal MoE, it can take the form of per-token variable expert slots filled by real or virtual experts (Lu et al., 2 Jul 2026, Gao et al., 23 Nov 2025).

This suggests that the term denotes a design pattern rather than a single architecture. What remains invariant is the separation between a shared substrate and a set of specialized modules whose participation is conditioned on the input, current batch, uncertainty state, feature geometry, or deployment context.

2. Architectural forms of experts

Recent systems instantiate experts in markedly different parameterizations. SIMoE treats each expert as a structurally sparse subset of the seed LLM’s parameters, while DMoE uses independently trainable LoRA-style adapters attached only to the final-layer feed-forward network, and AnyExperts distinguishes between trainable real experts and identity virtual experts (Chen et al., 14 Jun 2025, Yue et al., 12 Jun 2026, Gao et al., 23 Nov 2025). MADE-IT instead models each modular expert as a rank-$r$ principal subspace extracted from task-specific weight updates, and IRFE-ECG uses per-domain linear two-class heads over frozen 1024-dimensional ECGFounder features (Qiu et al., 24 Apr 2026, Lu et al., 2 Jul 2026).

At the module level, experts may also be structurally heterogeneous. CBDES MoE replaces a static camera backbone with four ImageNet-pretrained backbones—Swin Transformer, ResNet, ConvNeXt, and PVT—selected by a lightweight Self-Attention Router (Xiang et al., 11 Aug 2025). In video-based person re-identification, each expert contains a mapping block, a relevance evaluator, and channel-wise spatial or temporal branches selected by Improved Semhash (Hui et al., 1 Jul 2026). In MEPG, experts are diffusion branches specialized for style or realism and are activated per spatial region rather than per token (Zhao et al., 4 Sep 2025).

The same pattern extends beyond parametric modules. In the MCP-based Internet of Experts, each expert is an external deterministic classifier exposed as an MCP tool with a name, description, input schema, and JSON-RPC response format; the LLM remains frozen and queries these expert servers at inference time (Liu et al., 3 May 2025). This externalized construction places the expert boundary at the system interface rather than inside a single neural network.

Work	Expert representation	Activation mechanism
SIMoE	Structurally sparse subsets of seed LLM parameters	Input-dependent router merges experts (Chen et al., 14 Jun 2025)
MADE-IT	Rank-$r$ principal subspaces from $\Delta W$	Feature Projection Alignment and dependency graph (Qiu et al., 24 Apr 2026)
IRFE-ECG	Balanced-softmax linear heads on frozen 1024-D features	MLP routing or top-2 margin fusion (Lu et al., 2 Jul 2026)
DMoE	Final-layer FFN LoRA adapters	BM25 retrieval triggered by uncertainty (Yue et al., 12 Jun 2026)
IoX	External MCP expert servers	LLM planning over tool metadata (Liu et al., 3 May 2025)
CBDES MoE	Heterogeneous camera backbones	Self-Attention Router, soft train / top-1 infer (Xiang et al., 11 Aug 2025)
AnyExperts	Real experts plus virtual identity experts	Per-token slot allocation under budget bounds (Gao et al., 23 Nov 2025)

3. Routing, fusion, and conditional computation

Routing mechanisms differ chiefly in what signal they condition on and what object they select. In classical token-centric MoE decode, OEA models MoE-layer latency as

$T_{decode} \approx b \cdot T + a \cdot Bk,$

where $T$ is the number of unique activated experts, $B$ is batch size, and $k$ is experts per token; the method then reduces latency by lowering $T$ through batch-aware piggybacking while keeping a per-token baseline set of top-$k_0$ experts (Oncescu et al., 4 Nov 2025). This routing is input-aware at the batch-step level rather than through retraining.

In manifold-based continual merging, routing can be entirely training-free. MADE-IT projects the backbone feature $h_c$ onto each expert’s principal input subspace and scores compatibility by

$r$0

with top-1 selection per module and a dependency graph enforcing path consistency across modules (Qiu et al., 24 Apr 2026). In DMoE, routing is not continuous mixture weighting but uncertainty-triggered activation: token uncertainty is

$r$1

and the router retrieves top-$r$2 experts by BM25 only when $r$3 (Yue et al., 12 Jun 2026).

Several systems use dynamic fusion rather than discrete selection. RAVEN scores patches in reverse chronological order, builds cumulative importance

$r$4

and derives nested prefix sets

$r$5

so that each scale-specialized expert receives a sample-specific look-back horizon; Correlation-Aware Weighting then penalizes redundant expert outputs before aggregation (He et al., 23 Jun 2026). AnyExperts maps token importance to a variable number of expert slots,

$r$6

and fills those slots with a bounded mixture of real and virtual experts under a virtual-share cap (Gao et al., 23 Nov 2025).

Routing can also be decentralized. DQRoute assigns no centralized gate; instead, each expert produces its own confidence or OOD score, and expert weights are obtained by normalizing those scores, while training is simultaneously modulated by class-wise difficulty estimates derived from entropy and historical accuracy (Wei et al., 27 Aug 2025). In protocol-based systems, routing reduces to tool planning: the LLM uses expert metadata and the current query to decide which MCP tools to invoke, then folds returned confidences into its reasoning context (Liu et al., 3 May 2025).

4. Mechanisms of extension, growth, and maintenance

Extendability is realized through several distinct lifecycle operations. In SIMoE, upcycling reuses a single seed model, and adding experts does not duplicate the full parameter set; experts are formed as structured sparse views of the seed parameters and can be extended by introducing new masks and small router parameters (Chen et al., 14 Jun 2025). DMoE similarly decouples expert growth from the base model: new knowledge is inserted by training a new LoRA adapter for a passage-level knowledge unit, saving its text surrogate, and inserting that surrogate into the BM25 index, with no change to shared base parameters (Yue et al., 12 Jun 2026).

MADE-IT formalizes expert evolution through module-wise subspace affinity and an adaptive threshold

$r$7

If the maximal affinity of a candidate subspace is below $r$8, a new expert is spawned; otherwise the candidate is merged into the closest existing expert and re-truncated to rank $r$9, which functions as effective pruning of redundant directions (Qiu et al., 24 Apr 2026). This combines growth, merge, and pruning in a single expert-management rule.

Other systems use simpler append-only schemes. IRFE-ECG adds one new balanced-softmax linear expert for each arriving ECG source, freezes all earlier experts and the backbone, and refits only the lightweight router on retained frozen training features and domain labels from sources observed so far (Lu et al., 2 Jul 2026). In the Re-ID setting, each layer maintains a wait-list expert; when the wait-list expert exhibits higher relevance than the existing experts for input samples, it is appended to the constant expert set and a fresh wait-list expert is instantiated (Hui et al., 1 Jul 2026).

Protocol-based and modular-serving systems externalize extension into registration interfaces. IoX adds new experts by packaging an MCP tool endpoint and registering its name, description, input schema, and version in the MCP registry, after which the LLM can discover and invoke it at runtime without parameter updates (Liu et al., 3 May 2025). CBDES MoE likewise remains modular at functional-module granularity: experts share an aligned interface to the camera-to-BEV projector, and the router output can be expanded to accommodate an additional backbone expert while preserving the downstream BEV pipeline (Xiang et al., 11 Aug 2025).

5. Representative empirical outcomes

Because these modules optimize different objectives—accuracy, robustness, latency, continual retention, or compute efficiency—their reported gains are not directly comparable. They nevertheless show that input-aware expert selection is useful across domains.

Setting	Representative outcome	Citation
LLM upcycling with sparse experts	Consistently achieves state-of-the-art on common instruction-tuning benchmarks and maintains an optimal performance-compute trade-off	(Chen et al., 14 Jun 2025)
Batch-aware MoE decode rerouting	Up to 39% reduction in MoE layer decode latency on Qwen3-30B and 15% on Qwen3-235B without statistically significant loss in accuracy	(Oncescu et al., 4 Nov 2025)
Continual model merging	On CLIP ViT-B/32 with 20 tasks, ACC = 81.4%, surpassing MINGLE by 4.3% and OPCM by 15.7%; reduction rates approach about 95% in generic modules	(Qiu et al., 24 Apr 2026)
Replay-free continual ECG deployment	Source-aware expert selection reaches $\Delta W$0 Macro-F1, while top-2 margin fusion without source IDs reaches $\Delta W$1	(Lu et al., 2 Jul 2026)
Wireless environment-aware LLM agents	40%-50% improvements in classification tasks over LLM-only baselines across multiple mainstream LLMs	(Liu et al., 3 May 2025)
Financial time-series forecasting	Pearson correlation improves by 9.2% on HS300 and 20.2% on S&P500, and MSE is reduced by 18.2% on fund sales forecasting	(He et al., 23 Jun 2026)
Budget-aware multimodal MoE	Comparable accuracy with 40% fewer real expert activations on general image/video tasks, and about 10% reduction on OCR and NLP while maintaining performance	(Gao et al., 23 Nov 2025)
Autonomous driving BEV perception	1.6-point increase in mAP and 4.1-point improvement in NDS over the strongest single-expert model	(Xiang et al., 11 Aug 2025)
Video-based person Re-ID	MARS: mAP 87.0%, Rank-1 91.6%; LS-VID: mAP 81.0%, Rank-1 88.3%	(Hui et al., 1 Jul 2026)

A further pattern is that some modules primarily improve systems efficiency rather than benchmark accuracy. DMoE reports average latency of 2.6656 s and GPU memory of 7.2352 GB, compared with 9.2643 s and 13.9718 GB for FLARE, while preserving KV-cache reuse by attaching experts only to the final-layer FFN (Yue et al., 12 Jun 2026). Klotski operates one level lower, at the inference-engine layer, and uses expert-aware multi-batch scheduling, a constraint-sensitive planner, and correlation-aware prefetching to improve throughput by up to 85.12x under offloading constraints (Fang et al., 9 Feb 2025).

6. Limitations, misconceptions, and open problems

A common misconception is that an input-aware extendable expert module is simply a standard top-$\Delta W$2 MoE layer. Recent work shows a much broader landscape: routing may be training-free and geometric, entirely inference-time and batch-aware, or mediated by protocol calls to external tools rather than by a learned gate inside a monolithic network (Qiu et al., 24 Apr 2026, Oncescu et al., 4 Nov 2025, Liu et al., 3 May 2025). Another misconception is that extendability implies no memory burden. IRFE-ECG is explicit that the method is replay-free at the raw-ECG level but not memory-free because frozen training features are retained for router updates (Lu et al., 2 Jul 2026). DMoE likewise shows that modular knowledge banks can be storage-heavy: its example bank contains 27,613 experts with a disk footprint of about 13.08 GiB (Yue et al., 12 Jun 2026).

Theoretical and systems constraints remain substantial. "Learning-to-Defer with Expert-Conditioned Advice" argues that separated router/query surrogates are inconsistent even in the smallest non-trivial setting, and instead advocates learning over the composite expert–advice action space; the same work notes that training may require executed-pair cost tables and that complexity scales with $\Delta W$3 (Montreuil et al., 15 Mar 2026). AnyExperts demonstrates that variable expert allocation must still be budgeted by hard bounds and a virtual-share cap, with $\Delta W$4 emerging as a useful operating point and larger values harming some NLP settings (Gao et al., 23 Nov 2025). DQRoute depends on calibratable confidence or OOD scores across experts, and its difficulty estimates assume that entropy and historical performance remain meaningful after EMA smoothing (Wei et al., 27 Aug 2025).

This suggests three open directions. First, expert modules increasingly need joint treatment of routing, calibration, and systems scheduling rather than architecture alone. Second, extension policies are moving from append-only growth toward adaptive growth–merge–prune cycles. Third, the boundary of an “expert” is becoming less stable: it may be a sparse parameter mask, a low-rank adapter, a linear head, a subspace, a whole backbone, or an external service. The modern input-aware extendable expert module is therefore best understood as a general conditional-computation framework for specialization under evolving data, compute, and deployment constraints.