Mixture-of-Basis-Experts (MoBE)

• Mixture-of-Basis-Experts (MoBE) is a modular design principle that reuses shared basis components to enable task-adaptive predictions and policies.
• It employs various mixture mechanisms such as latent-variable conditioning, convex combination, and adaptive gating to integrate basis models or weight matrices.
• Empirical studies demonstrate significant parameter reduction and minimal accuracy loss, supporting applications in compression, continual learning, and control.

Mixture-of-Basis-Experts (MoBE) denotes a family of modular mixture architectures in which a limited set of reusable basis components is combined, selected, or recomposed to form task-adaptive predictions, policies, or expert parameterizations. In the cited literature, the “basis” can be a set of task-agnostic predictive models, shared weight matrices reused across experts, explicitly aligned basis solvers, or an orthogonal skill basis in action space; correspondingly, the “mixture” can be implemented by latent-variable conditioning, convex combination, adaptive gating, or sparse routing (Xu et al., 2022, Chen et al., 7 Aug 2025, Pan et al., 24 Oct 2025, Hao et al., 29 Jan 2026). The term is therefore structural rather than uniform: it refers less to a single canonical architecture than to a design principle of reusing shared components while preserving expert specialization.

1. Conceptual scope and terminology

A useful way to organize the literature is to distinguish what constitutes the basis object and where mixture is applied. In continual learning, the basis object is a set of task-agnostic predictive models mixed by a latent task representation. In LLM compression, the basis object is a set of shared matrices that re-parameterize expert weights. In combinatorial optimization and robotics, the basis object is a library of specialized policies or skills whose outputs are adaptively fused. This suggests that MoBE is best understood as a reusable-component view of expert modularity rather than as a single parameterization.

Representative form	Basis object	Mixture mechanism
MoB for continual learning (Xu et al., 2022)	Basis models $b^{(i)}$	Task-dependent weights $w^i(Z_t)$
MoBE for MoE compression (Chen et al., 7 Aug 2025)	Shared basis matrices $\{B^j\}$	Expert-specific coefficients $\alpha^{i,j}$
MoSES for VRP (Pan et al., 24 Oct 2025)	Specialized LoRA experts / basis policies	Adaptive gating $G(h^{\mathrm{in}})$
SMP for manipulation (Hao et al., 29 Jan 2026)	Orthonormal skill basis $B(s_t)$	Sticky gates $g_t$ over skill coefficients

The literature also differs in how strongly it separates basis and expert identity. In the compression setting, each expert remains identifiable through its own $A^i$ and coefficient vector $\{\alpha^{i,j}\}$. In continual learning, the basis models are explicitly task-agnostic and all adaptation is expressed through the mixing weights. In MoSES, specialization is explicit and aligned with specific VRP constraints rather than emerging implicitly from generic MoE training (Chen et al., 7 Aug 2025, Xu et al., 2022, Pan et al., 24 Oct 2025).

2. Core mathematical patterns

A canonical predictive form appears in the continual-learning formulation “mixture of Basismodels”: $$P(Y_t|X_t, Z_t) = \sum_{i=1}^{K} w^{i}(Z_t)\,b^{(i)}(Y_t|X_t),$$ where $w^i(Z_t)$0 is a latent task variable and $w^i(Z_t)$1 lies on the simplex. Each basis $w^i(Z_t)$2 can itself be implemented as an ensemble,

$w^i(Z_t)$3

with point prediction

$w^i(Z_t)$4

Learning is performed as a variational sequence model with an ELBO over $w^i(Z_t)$5 (Xu et al., 2022).

In the LLM-compression formulation, MoBE acts directly in weight space. For the $w^i(Z_t)$6-th expert,

$w^i(Z_t)$7

and each up/gate matrix $w^i(Z_t)$8 is factorized as

$w^i(Z_t)$9

with an expressive version

$\{B^j\}$0

Here $\{B^j\}$1 is expert-specific, $\{B^j\}$2 are shared across experts within a layer, and $\{B^j\}$3 defines the expert’s combination over basis matrices (Chen et al., 7 Aug 2025).

In control-oriented variants, the mixture is performed in latent policy or action space. MoSES defines a latent mixture

$\{B^j\}$4

where $\{B^j\}$5 are basis embeddings produced by optimal basis policies. SMP instead decodes actions from a skill basis,

$\{B^j\}$6

with $\{B^j\}$7, simplex gates $\{B^j\}$8, and per-skill coefficients $\{B^j\}$9 (Pan et al., 24 Oct 2025, Hao et al., 29 Jan 2026).

Taken together, these formulations place basis structure in at least four locations: predictive distributions, expert weight parameterization, latent policy embeddings, and action subspaces. That breadth is a defining feature of the MoBE literature rather than an inconsistency.

3. Weight-space MoBE for compressing MoE-based LLMs

The most explicit use of the name “Mixture-of-Basis-Experts” appears in the compression method for MoE-based LLMs (Chen et al., 7 Aug 2025). Its motivation is deployment: large MoE-based LLMs such as DeepSeek-V3-0324 and Kimi-K2-Instruct present serious challenges due to substantial memory requirements, while prior MoE compression methods often suffer from considerable accuracy drops, including “7-14% relatively” even at modest compression rates.

The technical choice is highly specific. MoBE compresses the two largest expert matrices, the up and gate projections, while leaving the down projection uncompressed because the down matrix “holds critical knowledge.” The factorization is learned data-free by minimizing reconstruction error: $\alpha^{i,j}$0 The paper states that optimization is typically done with Adam, with optional Z-score normalization for stability, and that Tanh/SiLU after the basis combination boosts expressive power (Chen et al., 7 Aug 2025).

A central quantitative claim concerns parameterization efficiency. For an MoE layer with $\alpha^{i,j}$1 experts, input dimension $\alpha^{i,j}$2, hidden dimension $\alpha^{i,j}$3, factorization rank $\alpha^{i,j}$4, and $\alpha^{i,j}$5 shared bases, the MoBE parameter count is

$\alpha^{i,j}$6

with compression ratio

$\alpha^{i,j}$7

Under the stated regime $\alpha^{i,j}$8 and $\alpha^{i,j}$9, this yields “24–30% parameter reduction” in practice (Chen et al., 7 Aug 2025).

The empirical claims are unusually strong for model compression. On Qwen3-235B-A22B-2507, DeepSeek-V3-0324 (671B), and Kimi-K2-Instruct (1T), MoBE reduces total parameters by 24%–30% with only 1%–2% accuracy drop, “about 2% drops when measured relatively.” The same source states that MoBE achieves “50%+ lower per-layer MSE than prior methods” and notably lower accuracy drops than SVD-based approaches such as MoLAE and D$G(h^{\mathrm{in}})$0-MoE (Chen et al., 7 Aug 2025).

The paper also identifies an important limitation of naive low-rank compression: the effective rank of expert weight matrices is often higher than the threshold needed for useful compression. The proposed remedy is shared basis learning across experts within a layer, which “explains away” more variance at the same parameter budget. A practical caveat remains: activation parameters may increase because both basis matrices and $G(h^{\mathrm{in}})$1 must be loaded per activated expert. The paper mitigates this with fewer active experts per token at inference, denoted “MoBE$G(h^{\mathrm{in}})$2” (Chen et al., 7 Aug 2025).

4. Continual learning, latent task structure, and adaptive basis growth

A distinct MoBE lineage appears in continual learning under distribution shifts, where the objective is not parameter compression but reusable task structure (Xu et al., 2022). In this formulation, a small set of basis models is learned offline, and a VAE-style latent variable $G(h^{\mathrm{in}})$3 encodes the current task regime. The model is explicitly task-agnostic at the basis level: “all adaptation and modulation is via $G(h^{\mathrm{in}})$4.” This separation makes the latent task representation central to both prediction and interpretability.

The setting is semi-supervised and task-agnostic: the learner has access to both task-segmented and unsegmented data for offline training, while online data arrives as a stream with abrupt task distribution shifts whose timing is unknown. Learning uses stochastic variational inference with the reparameterization trick, and basis diversity is encouraged by pretraining basis models on individual segmented tasks using a meta-learned MAML prior for fast adaptation (Xu et al., 2022).

The paper’s second contribution is adaptive model growth via out-of-distribution detection. It introduces the combined score

$G(h^{\mathrm{in}})$5

designed to detect both covariate and concept shifts. If $G(h^{\mathrm{in}})$6, the observation is counted as out-of-distribution; when a buffering threshold is reached, a new basis model is instantiated on the buffered data via fast adaptation from the MAML prior. The same work reports that similar tasks tend to cluster in latent space and that latent representations shift at task boundaries when tasks are dissimilar (Xu et al., 2022).

Empirically, the method “attains better prediction error than existing methods in most cases while using fewer models than other multiple model approaches.” The reported examples include roughly 3.4 bases versus 12.0 models for MOLe on a regression partition and 14.8 versus 52.8 on a HalfCheetah ice-slope partition; the latent task representation is presented as interpretable and aligned with semantic task similarity (Xu et al., 2022).

Expert lifecycle management is extended further by work on merging experts in MoE for multi-domain task incremental learning (Park, 2024). That paper argues that shared layers can degrade performance and proposes a usage-based consolidation rule: track expert selection frequency, merge the two most frequently selected experts, and replace the least frequently selected expert with

$G(h^{\mathrm{in}})$7

After each task, the top-$G(h^{\mathrm{in}})$8 most selected experts are frozen. The paper reports improvements of 0.2% in transfer accuracy and 0.3% in average accuracy over the prior state of the art, with domain-specific gains such as 1.2% in Flowers and 0.4% in SUN397, and interprets the behavior as evidence for the emergence of more general “basis” experts (Park, 2024).

5. Feature sparsity, expert selection, and runtime optimization

A longstanding theme adjacent to MoBE is that experts should not only specialize functionally but also operate on distinct subspaces. A regularized MoE formulation embeds local feature selection through $G(h^{\mathrm{in}})$9 regularization in both experts and the gate function: $B(s_t)$0 It further introduces an expert-selector variable $B(s_t)$1 so that only a subset of experts is active for a given input. The paper explicitly states that this can be interpreted as related to MoBE because each expert becomes a specialist over a distinct feature subspace and the selector acts as data-driven activation of those basis components (Peralta, 2014).

From a systems standpoint, large sparse mixtures introduce a second class of difficulty: dynamic data flow. DynaMoE addresses this at the runtime level with dynamic recompilation, described as a controlled, on-the-fly re-construction of the computation graph that reallocates computational and memory resources only where needed. The mechanism is asynchronous, with planning overlapped on CPU while training or inference proceeds on GPU via a buffer of pre-launched tasks $B(s_t)$2, and it is driven by user-defined recompile policies (Kossmann et al., 2022).

Two recompile-enabled optimizations are especially relevant to basis-expert architectures that inherit MoE-style routing. The first is dynamic capacity factors, replacing the fixed expert-capacity formula

$B(s_t)$3

with expert-specific, dynamically adjusted capacities based on observed slot utilization. The second is sample assignment caching, activated when routing assignments converge between epochs so that experts and gating networks can execute in parallel rather than serially (Kossmann et al., 2022).

The reported performance gains are concrete: DynaMoE is “up to 1.8× faster than FastMoE” and “supports 2.3× larger models” before out-of-memory; dynamic capacity factors yield a 1.4× speedup over static capacity at the same accuracy, and sample assignment caching raises total speedup to 1.7×. The paper also reports linear scaling in speed and supported model size with more GPUs and states that specification loss outperforms cooperation loss markedly in MoE settings (Kossmann et al., 2022). A plausible implication is that MoBE variants with dynamic routing face the same memory, buffer sizing, and scheduling constraints, even when their basis structures differ.

6. Compositional policies and control-oriented MoBE

In combinatorial optimization, MoBE appears as explicit policy reuse rather than as expert compression. The Mixture-of-Specialized-Experts Solver (MoSES) for vehicle routing begins from a State-Decomposable MDP in which the state space is the Cartesian product of basis state spaces associated with basis VRP variants: $B(s_t)$4 Transition probabilities decompose accordingly, and the framework yields an optimal basis policy for each basis VRP variant. A latent-space extension then proves that, under mild assumptions, a learnable mixture function can recover the optimal unified policy of the SDMDP from basis embeddings (Pan et al., 24 Oct 2025).

For practical implementation, MoSES realizes basis policies through specialized LoRA experts and mixes them with an adaptive gating mechanism: $B(s_t)$5 The paper emphasizes explicit expert alignment with VRP attributes, dense versus variant-aware routing, and a residual LoRA branch for capacity not covered by the basis experts. It reports superiority over prior methods across 16 VRP variants, including roughly 10%–25% relative reduction in optimality gap versus prior state of the art, as well as improved out-of-distribution generalization to unseen constraint values (Pan et al., 24 Oct 2025).

In robotics, the Skill Mixture-of-Experts Policy (SMP) provides a diffusion-based MoBE formulation. It learns a compact orthogonal skill basis $B(s_t)$6, uses sticky routing to activate a small subset of experts at each step, and trains with a variational objective combining reconstruction, diffusion, and gate regularization. The routing process is modeled as a sticky Dirichlet-Markov process,

$B(s_t)$7

favoring sparse and temporally persistent expert activation (Hao et al., 29 Jan 2026).

At inference, SMP selects experts by top-$B(s_t)$8 or coverage, where expert mass is $B(s_t)$9, and only the selected experts are denoised. The paper reports higher or comparable average success than large diffusion baselines on multi-task bimanual manipulation, with examples including 0.54 versus 0.48 on RoboTwin-2 and 0.18 versus 0.17 on RLBench-2. It also reports lower inference cost: SMP has 258.9M total parameters but 80.2M active parameters and 107.3 ms inference time, compared with 1200M active parameters and 183.1 ms for RDT (Hao et al., 29 Jan 2026).

These control-oriented works illustrate a broader MoBE principle: basis experts are not merely compressed representations of a larger model, but reusable modules aligned with compositional task structure.

7. Statistical questions, Bayesian extensions, and recurring issues

Two recurring technical questions for MoBE-like systems are how many experts are needed and how uncertainty should be estimated once experts are present. For Gaussian-gated Gaussian Mixture of Experts, model selection is addressed by dendrograms of mixing measures. Starting from an overfitted estimator,

$g_t$0

the method defines a dissimilarity

$g_t$1

merges the closest pair iteratively, and uses the Dendrogram Selection Criterion

$g_t$2

to recover the true number of experts (Thai et al., 19 May 2025).

The paper states that the procedure consistently estimates the number of experts, achieves pointwise optimal convergence rates for parameter estimation in overfitted scenarios, and avoids retraining multiple models for different component counts. On synthetic data it outperforms AIC, BIC, and ICL in accurately recovering the number of experts, while the dendrogram also offers a hierarchical view of the fitted structure (Thai et al., 19 May 2025). Although this analysis is presented for GGMoE rather than for MoBE specifically, it is directly relevant to any basis-expert model with covariate-dependent gating and overfitting risk.

Uncertainty estimation has been studied in Bayesian-MoE for LLMs, which applies a structured Laplace approximation to the second linear layer of each expert. The posterior is block-diagonal across experts,

$g_t$3

and each expert’s curvature is approximated by a low-rank Kronecker factorization

$g_t$4

The method improves expected calibration error and negative log-likelihood on common-sense reasoning benchmarks for Qwen1.5-MoE and DeepSeek-MoE, outperforming MAP, MC dropout, checkpoint ensembles, deep ensembles, and Bayesian-LoRA in the reported tables (Dialameh et al., 12 Nov 2025).

Several recurrent misconceptions are clarified by the literature. First, “basis” is not restricted to matrix factorization; it may refer to models, policies, or skill directions. Second, more sharing is not always better: one paper reports that incorporating a shared layer in a mixture-of-experts can lead to performance degradation (Park, 2024). Third, low-rank compression is not automatically sufficient; effective-rank limitations can make pure SVD-style approximations either lossy or ineffective for compression (Chen et al., 7 Aug 2025). Fourth, sparse mixtures remain systems-limited even when only a subset of experts is active; expert capacity, routing stability, and asynchronous remapping remain central engineering constraints (Kossmann et al., 2022).

Across these strands, MoBE emerges as a general strategy for representational economy: specialize locally, reuse globally, and let routing or mixture coefficients determine how shared basis structure is instantiated for each example, task, or deployment constraint.