Supervised Mixture of Experts (S-MoE)

• Supervised Mixture of Experts (S-MoE) are conditional models that use input-dependent gating to combine specialized predictors based on labels and metadata.
• They leverage deterministic routing by incorporating task-specific signals like speech bandwidth or surgical phase labels, which reduces negative transfer and expert collapse.
• Optimization strategies such as EM, proximal Newton, and MM promote efficient parameter estimation, sparsity, and interpretability through local feature and expert selection.

Supervised Mixture of Experts (S-MoE) denotes a family of conditional mixture architectures that combine specialized predictors through a supervised routing or weighting mechanism. In the classical statistical literature, S-MoE is the supervised or conditional version of a mixture model: the response $Y$ is modeled given predictors $X$, and both the mixture weights and the component models depend on $X$. In more recent deep-learning usage, the same label has also been applied to architectures in which expert assignment is driven directly by known task structure—such as task tags, speech bandwidth labels, or surgical phase labels—rather than learned solely from input features (Huynh et al., 2019, Jin et al., 5 Aug 2025, Mazza et al., 29 Jan 2026).

1. Terminological scope and historical framing

In the classical formulation, supervised MoE is a predictive model for regression or classification in which a gate assigns input-dependent weights to expert models, and the final supervised prediction is their weighted combination. This usage appears explicitly in conditional mixture formulations for classification and generalized-linear experts, and in softmax-gated multinomial-logistic MoE for multiclass prediction (Peralta, 2014, Tran et al., 8 Feb 2026). More recent work uses “Supervised Mixture of Experts” for deterministic routing schemes that eliminate learned gating when the routing signal is already available from metadata or privileged annotations, as in multi-task speech-to-text and phase-structured surgical imitation learning (Jin et al., 5 Aug 2025, Mazza et al., 29 Jan 2026).

Regime	Supervision source	Representative paper
Classical conditional MoE	labeled pairs $(x,y)$	(Huynh et al., 2019)
Softmax-gated multinomial-logistic MoE	class labels in conditional likelihood	(Tran et al., 8 Feb 2026)
Deterministic supervised routing	special guiding tokens; speech bandwidth label	(Jin et al., 5 Aug 2025)
Phase-aware supervised routing	privileged phase labels	(Mazza et al., 29 Jan 2026)
Representation-disentangled supervised MoE	class labels and SNNL regularization	(Agarap et al., 20 Mar 2026)

This suggests that the term is not uniform across subfields. In statistics, “supervised” usually refers to conditional modeling of $Y\mid X$. In contemporary architectures, “supervised” may instead refer to direct supervision of the router, or to label-aware representation shaping before routing.

2. Core mathematical formulations

A canonical supervised MoE decomposes the conditional predictive distribution as

$$p(y\mid x)=\sum_{i=1}^{K} p(y\mid m_i,x)\,p(m_i\mid x),$$

where $p(m_i\mid x)$ is the gate and $p(y\mid m_i,x)$ is the expert-specific conditional model. In multinomial classification, both gate and experts are often parameterized by softmax or multinomial-logit functions, yielding input-dependent mixture weights and input-dependent class probabilities (Peralta, 2014). A closely related multiclass formulation writes

$$s_\theta(y \mid x)=\sum_{k=1}^K g_k(w(x))\, e_k(y;v(x)),$$

with softmax gate

$$g_k(w(x))=\frac{\exp(w_k(x))}{\sum_{\ell=1}^K \exp(w_\ell(x))}$$

and multinomial-logistic experts

$X$0

This is the SGMLMoE specialization for multiclass classification (Tran et al., 8 Feb 2026).

Regularized generalized-linear S-MoE uses the same conditional-mixture structure but specifies the experts as Gaussian, Poisson, or multinomial-logistic generalized linear models. In that setting,

$X$1

with a multinomial-logistic gate and expert families chosen according to the response type (Huynh et al., 2019). This preserves the probabilistic interpretation while extending S-MoE to heterogeneous regression and classification tasks.

A distinct contemporary formulation replaces the learned gate by a predefined gating function: $X$2 where $X$3 is not trainable. In multi-task speech-to-text, the routing function is supplied by supervision and deterministically selects the corresponding expert from task tags or bandwidth labels; in the reported configuration, the number of experts is fixed to $X$4 for both encoder and decoder (Jin et al., 5 Aug 2025). In surgical imitation learning, the gate is a phase classifier over $X$5 ordered phases, and the final action and gripper outputs are soft mixtures of phase experts,

$X$6

with one expert per phase (Mazza et al., 29 Jan 2026).

These formulations show that S-MoE can be probabilistic, deterministic, hard-routed, or soft-routed. What remains common is expert specialization guided by supervision rather than left entirely to unsupervised partitioning.

3. Estimation, optimization, and model selection

For probabilistic MoE models, one line of work emphasizes likelihood-based estimation and asymptotic theory. An overview of practical and theoretical MoE modeling proposes the maximum quasi-likelihood (MQL) estimator for estimating MoE model parameters from data, gives conditions under which MQL estimators are consistent and asymptotically normal, proposes the blockwise minorization-maximization (blockwise-MM) algorithm framework as an all-purpose method for constructing algorithms for obtaining MQL estimators, and presents information criteria for estimating the number of components together with justification for the classic Bayesian information criterion (BIC) (Nguyen et al., 2017).

In high-dimensional supervised MoE with generalized linear experts, estimation is typically formulated as regularized maximum likelihood and solved by EM with structured M-steps. The proximal-Newton EM algorithm of (Huynh et al., 2019) computes responsibilities

$X$7

then maximizes a penalized $X$8-function that decomposes into gating and expert subproblems. The M-step uses proximal Newton or proximal Newton-type quadratic approximations so that each update becomes a weighted Lasso problem solved by coordinate ascent with soft-thresholding. The paper emphasizes monotone maximization, exact zeros from the $X$9 penalty, and the avoidance of approximated penalties or post-hoc thresholding (Huynh et al., 2019).

For softmax-gated multinomial-logistic S-MoE, a more recent alternative replaces approximate EM-style M-steps with batch MM. The method derives an explicit quadratic surrogate for the observed-data log-likelihood, updates responsibilities

$X$0

and obtains coordinate-wise closed-form updates with monotone ascent of the objective and convergence to a stationary point in the standard MM sense (Tran et al., 8 Feb 2026). The same work adapts dendrograms of mixing measures to classification and introduces the dendrogram selection criterion

$X$1

a sweep-free selector of the number of experts after fitting one moderately over-specified model and merging redundant fitted atoms.

The resulting methodological picture is bifurcated. One branch treats S-MoE as a rigorously estimable conditional mixture model with MQL, EM, proximal Newton, MM, and BIC-type selection. Another branch treats supervision as a routing primitive and dispenses with learned gating entirely when expert assignment is already known.

4. Sparsity, local feature selection, and expert selection

A major statistical extension of S-MoE concerns sparsity in both the gate and the experts. In high-dimensional classification, the regularized expected complete log-likelihood

$X$2

encourages local feature selection by driving gate and expert coefficients to zero. Because a zero coefficient indicates that the corresponding feature is irrelevant for that local model, sparsity sharpens specialization and supports interpretability (Peralta, 2014).

The same work extends feature selection to sample-specific expert selection through an “expert selector” variable $X$3, modifying the gate so that irrelevant experts can be ignored for a given datum. The hard formulation imposes a $X$4-norm constraint, while the practical relaxation solves an $X$5-constrained quadratic program for each sample. This yields simultaneous feature selection and simultaneous expert selection within the standard supervised MoE probabilistic structure (Peralta, 2014).

Regularized generalized-linear S-MoE develops the same idea for Gaussian, Poisson, and multinomial-logistic experts. Its penalized likelihood

$X$6

produces sparse gate parameters and sparse expert parameters, so that predictor relevance can differ across experts and across response models. The proximal Newton-type strategy avoids expensive matrix inversion and is explicitly motivated by numerical instability and singularity issues in high dimensions (Huynh et al., 2019).

These sparse S-MoE variants formalize a recurring intuition: different regions of input space may be best explained by different low-dimensional subspaces, and not all experts should compete for every example. They therefore move beyond expert specialization alone to specialization in both parameter support and participation.

5. Supervised routing in contemporary architectures

In multi-task speech-to-text, S-MoE is a lightweight alternative to both standard MoE and hard-parameter sharing. The model keeps shared attention and backbone layers but replaces the task-sensitive FFN sublayer with task-specific experts. Decoder routing is driven by special guiding tokens prepended to the text sequence, such as <transcribe> for ASR and <translate> for ST, while encoder routing is based on the speech bandwidth label, with WB inputs processed by expert $X$7 and NB inputs by expert $X$8. The full encoder-decoder S-MoE uses four experts total and reports a 6.35% relative improvement in Word Error Rate when applied to both encoder and decoder; in the NB/WB fine-tuned setting it also reports 1.63% BLEU improvement in the NB environment and 2.39% relative WER improvement with 1.15% BLEU improvement in the WB environment, while maintaining the same active-parameter footprint during inference as the baseline (Jin et al., 5 Aug 2025).

In surgical imitation learning, MoE-ACT attaches a supervised MoE architecture to Action Chunking Transformer for a phase-structured bowel grasping and retraction task. The task is decomposed into five phases—idle, approach & grasp, hold, retract, and maintain tension—and the router is directly supervised with privileged phase labels. Training optimizes an ELBO-style objective with action $X$9 loss, phase cross-entropy, gripper binary cross-entropy, and KL regularization. On in-distribution evaluation, standard ACT reaches 10/20 end-to-end success, whereas ACT + MoE reaches 17/20; the paper summarizes this as an improvement from 50% to 85% while preserving 27 Hz inference (Mazza et al., 29 Jan 2026).

A third interpretation of supervised MoE emphasizes representation supervision rather than deterministic routing. The SNNL-augmented MoE of (Agarap et al., 20 Mar 2026) inserts a CNN feature extractor trained with Soft Nearest Neighbor Loss before the gate and experts so that the latent space is already organized by class before routing. Inference uses hard expert selection by argmax, and training optimizes a composite loss combining cross-entropy with SNNL. The paper reports statistically significant improvements on FashionMNIST and CIFAR100, and on CIFAR100 pairwise expert similarity is reduced from 0.20 to 0.10, roughly halving redundancy (Agarap et al., 20 Mar 2026). The paper does not use the exact acronym S-MoE as a formal name, but it explicitly characterizes the method as a supervised, representation-disentangled MoE.

Together, these systems demonstrate three distinct supervised-routing regimes: metadata-driven deterministic routing, privileged-label router supervision, and label-aware latent-space preconditioning. All three reduce negative transfer or expert collapse by constraining how specialization emerges.

6. Theory, limitations, and related ambiguities

Theoretical work on supervised regression with MoE shows that the regression function can be estimable even when individual experts are not equally well behaved. For least squares estimation in softmax-gating MoE regression, the fitted regression function satisfies

$(x,y)$0

but expert recovery depends sharply on the expert family. Strongly identifiable experts, including feedforward networks with activations such as $(x,y)$1, $(x,y)$2, and GELU, inherit fast rates, whereas polynomial experts exhibit surprisingly slow estimation rates and may be as slow as $(x,y)$3 (Nguyen et al., 2024). A plausible implication is that expert architecture is not merely an implementation detail: it alters the statistical geometry of supervised MoE estimation.

Related gradient-dynamics theory studies a supervised regression problem with latent cluster structure and proves that MoE can detect and learn that structure while a vanilla neural network fails to do so. The argument hinges on information exponent, weak recovery of cluster-specific functions by experts, router learning from that weak specialization, and subsequent decoupling of the task into easier subproblems (Kawata et al., 2 Jun 2025). This is not presented as a new S-MoE variant, but it gives a mechanistic explanation for why mixture-based supervised decomposition can outperform a monolithic model.

A further extension replaces fully supervised training with semi-supervised estimation when unlabeled covariates contain a noisy proxy for expert structure. The noisy semi-supervised MoE model distinguishes the latent cluster structure in unlabeled covariates $(x,y)$4 from the latent expert assignment $(x,y)$5 governing $(x,y)$6, estimates experts robustly with least trimmed squares, and proves near-parametric convergence under a majority-transfer condition (Kwon et al., 2024). This clarifies that the boundary between supervised and semi-supervised MoE is itself an active research topic.

Several adjacent methods are sometimes conflated with S-MoE but are conceptually different. S2MoE is described as a supervised sparse MoE augmented with stochastic representation learning rather than as a new expert-supervision paradigm (Do et al., 29 Mar 2025). USMoE is presented as a unified competitive-learning framework for sparse MoE routing, not a supervised routing method based on labels (Do et al., 29 Mar 2025). Spatial Mixture-of-Experts introduces a self-supervised routing classification loss and explicitly states that the routing is not fully supervised (Dryden et al., 2022). A common misconception is therefore that any MoE trained with labels is an S-MoE. The literature instead distinguishes classical conditional-mixture supervision, deterministic supervised routing, privileged-label routing, self-supervised routing, and semi-supervised latent-linkage models.

In summary, S-MoE is best understood as a family of supervised expert-combination models rather than a single architecture. Its unifying principle is specialization under supervision, but the supervision can target the conditional likelihood, the routing map, the latent representation, or an externally known task structure.