---
title: Dual-Stage Reweighted Mixture-of-Experts (DR-MoE)
url: https://www.emergentmind.com/topics/dual-stage-reweighted-mixture-of-experts-dr-moe
type: topic
---

# Dual-Stage Reweighted Mixture-of-Experts (DR-MoE)

Searching arXiv for the cited papers to ground the article and verify metadata.
Dual-Stage Reweighted Mixture-of-Experts (DR-MoE) denotes a class of architectures in which mixture-of-experts is applied in two sequential stages, with explicit reweighting at one or both stages to combine complementary experts, losses, or policy branches. In the literature represented here, the term refers most directly to a framework for long-tailed egocentric mistake detection that first mixes feature extractors and then mixes classifiers, using learnable coefficients \(\alpha\) and \(\beta_k\) to modulate the contribution of each component [2509.12990]. The same phrase also describes a broader architectural motif: one stage determines or constrains which experts or branches should be emphasized, and a second stage reweights the resulting representations or decisions. This broader interpretation is made explicit by probabilistic routing formulations such as DirMoE, where expert selection and contribution are separated into distinct random variables and combined through a normalized product [2602.09001], and by two-stage control systems such as CoRe-MoE, where a base policy is learned first and a residual MoE branch is added and fused later [2606.04718].

## 1. Definition and conceptual scope

In its narrowest documented sense, DR-MoE is a framework proposed for egocentric mistake detection from video, designed to address subtle and infrequent mistakes under a strongly long-tailed class distribution [2509.12990]. The task input consists of a coarse egocentric video \(V_{0:T}\) and a fine-grained action segment \(V_{t:t+\tau}\subseteq V_{0:T}\), and the model outputs a binary decision \(\hat{y}\in\{0,1\}\), where \(1\) denotes “mistake” [2509.12990]. The architecture is “dual-stage” because it introduces mixture-of-experts at two levels: a feature-level expert module and a classification-level expert module [2509.12990].

The feature stage combines a frozen ViViT backbone and a LoRA-tuned ViViT backbone, while the classification stage fuses three classifier heads trained with different objectives: reweighted cross-entropy, AUC loss, and label-aware loss with sharpness-aware minimization [2509.12990]. The “reweighted” designation refers both to explicit class-frequency-based reweighting in the losses and to learnable mixing coefficients over experts [2509.12990].

A broader reading of DR-MoE is supported by subsequent and adjacent work. DirMoE is described as being “explicitly designed around the same conceptual decomposition that ‘Dual-Stage Reweighted MoE (DR-MoE)’ suggests,” namely a first stage for expert selection and a second stage for expert contribution or reweighting [2602.09001]. CoRe-MoE similarly instantiates a two-stage structure in reinforcement learning: a base locomotion policy is learned first, and a terrain-aware residual MoE branch is introduced in the second stage, with the final action obtained by weighted fusion [2606.04718]. This suggests that DR-MoE is best understood not as a single fixed implementation, but as an architectural pattern centered on staged decomposition and explicit weighting.

## 2. Canonical two-stage formulation in egocentric mistake detection

The canonical DR-MoE formulation is organized as two sequential expert mixtures [2509.12990]. Stage 1 performs feature extraction and feature fusion; Stage 2 performs classification and classifier fusion.

At the feature level, DR-MoE uses two ViViT-based experts:

1. \(f_{\text{ori}}\): a frozen ViViT pretrained on Kinetics, applied to the coarse video \(V_{0:T}\).
2. \(f_{\text{lora}}\): a LoRA-tuned ViViT applied to the fine-grained segment \(V_{t:t+\tau}\) [2509.12990].

These are combined by the feature mixture-of-experts module:
\[
\mathbf{F}_{\text{joint}} = \alpha \cdot f_{\text{ori}}(V_{0:T}) + (1 - \alpha) \cdot f_{\text{lora}}(V_{t:t+\tau}),
\]
where \(\alpha\in[0,1]\) is a learnable mixing parameter [2509.12990].

At the classification level, three classifiers \(f^{(1)}, f^{(2)}, f^{(3)}\) operate on \(\mathbf{F}_{\text{joint}}\), each optimized for a different aspect of long-tailed binary classification [2509.12990]. Their outputs are fused by the classification mixture-of-experts module:
\[
\hat{y} = \sum_{k=1}^3 \beta_k \, f^{(k)}(\mathbf{F}_{\text{joint}}),
\]
where \(\beta_k\) are learned expert weights [2509.12990].

Within the report, \(\alpha\) and \(\beta_k\) are not described as input-dependent gating networks; rather, they are global learnable weights [2509.12990]. The first stage therefore decides how much the joint representation should rely on generic versus task-adapted video features, whereas the second stage decides how much to trust classifiers specialized for imbalance correction, ranking, or calibration [2509.12990]. This division of labor is central to the architecture’s definition.

## 3. Feature-stage mechanism and backbone specialization

The feature stage is built on ViViT-B/\(16\times 2\), using clips of 32 frames in RGB modality [2509.12990]. One branch remains frozen and serves as a source of generic spatiotemporal semantics; the other branch is adapted using LoRA to focus on fine-grained mistake-sensitive cues [2509.12990].

LoRA is applied with rank \(8\) to attention-layer query and value projection matrices [2509.12990]. For an attention projection, the effective weight is written as
\[
W_{\text{eff}} = W + AB,
\]
where \(A \in \mathbb{R}^{d \times r}\), \(B \in \mathbb{R}^{r \times d}\), and \(r=8\) [2509.12990]. Only \(A\) and \(B\) are trained, while \(W\) remains fixed [2509.12990]. The stated rationale is efficient training, reduced overfitting relative to full finetuning, avoidance of catastrophic forgetting, and increased sensitivity to detailed mistake patterns [2509.12990].

The feature-level MoE is therefore not a sparse token-level router in the language-model sense; it is a learned convex combination of two semantically distinct video experts. The frozen branch encodes coarse human action semantics from the full video, while the LoRA branch emphasizes mistake-relevant details from the localized segment [2509.12990]. Because \(\alpha\) is learnable, the architecture can shift globally between generic and task-specific cues without committing to a single backbone [2509.12990].

A plausible implication is that this stage performs representational bias management: the frozen model preserves broad action understanding inherited from Kinetics pretraining, and the LoRA-tuned model injects high-resolution adaptation to the mistake-detection domain. The report itself does not supply an ablation table for \(\alpha\), but it states that the dual-stage design is intended to balance generic semantics with subtle error cues [2509.12990].

## 4. Classification-stage mechanism and reweighting strategies

The second stage comprises three classifier heads whose differentiation lies primarily in their loss functions rather than in distinct architectural complexity [2509.12990]. The report states that they are typically implemented as linear layers, possibly with a small MLP, mapping \(\mathbf{F}_{\text{joint}}\) to logits or scores over \(\{0,1\}\) [2509.12990].

The three objectives are summarized below.

| Classifier | Objective | Stated role |
|---|---|---|
| \(f^{(1)}\) | Reweighted cross-entropy | Mitigate class imbalance |
| \(f^{(2)}\) | AUC loss | Improve ranking under skewed distribution |
| \(f^{(3)}\) | Label-aware loss with SAM | Enhance calibration and generalization |

For the reweighted cross-entropy head, the loss is
\[
\mathcal{L}_{WCE}
= - \sum_{y} w_y \cdot \mathbf{1}_{[y = \hat{y}]} \log p_y,
\qquad
w_y = \frac{1}{f_y},
\]
where \(f_y\) is the class frequency [2509.12990]. This explicitly upweights rare mistake examples and downweights frequent correct-action examples [2509.12990].

For the AUC head, the loss is written as
\[
\mathcal{L}_{AUC} = \frac{1}{n^+n^-}
 \sum_{i=1}^{n^+} \sum_{j=1}^{n^-} \ell\left(
 f^{(2)}(\mathbf{F}_{\text{joint}}^{i,+})
 - f^{(2)}(\mathbf{F}_{\text{joint}}^{j,-})
 \right),
\]
with \(n^+\) and \(n^-\) denoting positive and negative sample counts [2509.12990]. The head is optimized as a pairwise ranker rather than a direct probabilistic classifier [2509.12990].

For the label-aware head, the loss is
\[
\mathcal{L}_{LA} = \sum_{i=1}^{N} \ell\big( f^{(3)}(\mathbf{F}_{\text{joint}}^{i}) + \log \mathbf{p}, \, y_i \big),
\]
where \(\mathbf{p}=[f_1,\dots,f_C]\) is the class-frequency vector [2509.12990]. This is combined with sharpness-aware minimization:
\[
\min_{\theta} \max_{\Vert \epsilon \Vert \leq \rho}
 \mathcal{L}\left(f^{(3)}_{\theta+\epsilon}\right),
\]
with radius \(\rho=0.05\) [2509.12990].

These heads are then fused through C-MoE by learned weights \(\beta_k\) [2509.12990]. In documented form, the fusion is not sample-adaptive. Instead, DR-MoE integrates multiple long-tail-aware training criteria into a single prediction pipeline through learned global coefficients. This contrasts with sparse MoE routers in large language models, where gating is typically token-dependent.

## 5. Relation to probabilistic and dynamic-routing MoE formulations

The expression “dual-stage reweighted MoE” also describes a more general decomposition in which selection and weighting are explicitly separated. DirMoE provides the clearest formalization of this view [2602.09001]. It models routing using a spike-and-slab structure over experts: a binary selection mask \(z\in\{0,1\}^E\) and simplex-valued contribution weights \(\theta\in\Delta^{E-1}\), with final routing weights
\[
r(x) = \mathrm{normalize}\big(z(x) \odot \theta(x)\big),
\qquad
y(x) = \sum_{i=1}^E r_i(x)\, E_i(x).
\]
The paper states that this is “exactly a ‘dual-stage, reweighted’ mechanism: discrete selection \(z\), then continuous reweighting \(\theta\) among the selected experts, turned into final mixture weights \(r\)” [2602.09001].

In DirMoE, Stage 1 uses relaxed Bernoulli selection via Gumbel-Sigmoid,
\[
\tilde{z}_i = \sigma\left(\frac{\ell_i(x) + g_i}{T_z}\right), \quad g_i \sim \mathrm{Logistic}(0,1),
\]
and Stage 2 uses a Dirichlet-distributed contribution vector conditioned on \(\tilde{z}\) [2602.09001]. The whole router remains differentiable through Gumbel-Sigmoid relaxation and implicit reparameterization for the Dirichlet distribution [2602.09001]. The training objective is a variational ELBO with a direct sparsity penalty that targets exactly \(k\) active experts in expectation [2602.09001].

Dynamic-routing work offers a different but related perspective. “Harder Tasks Need More Experts: Dynamic Routing in MoE Models” does not use the name DR-MoE, but it presents a two-phase routing process: first compute a full expert distribution \(\mathbf{P}\) by softmax, then select as many experts as needed for the cumulative probability to exceed a threshold \(p\) [2403.07652]. The number of active experts is therefore a function of confidence rather than a fixed \(K\) [2403.07652]. The paper explicitly interprets the first phase as confidence estimation and the second as adaptive expert-count selection, making it closely related to a staged MoE viewpoint [2403.07652].

These formulations highlight a conceptual distinction. In egocentric mistake detection, DR-MoE uses two semantically different MoE modules at two points in the pipeline [2509.12990]. In differentiable MoE routing for language models, the same phrase can denote the separation of “which experts” from “how much they contribute” within a single routing layer [2602.09001]. This suggests that DR-MoE has both an application-specific meaning and a cross-domain architectural meaning.

## 6. Empirical results, applications, and cross-domain variants

In the documented egocentric-video application, DR-MoE is evaluated on HoloAssist for mistake detection [2509.12990]. Table 1 in the report lists results on the test set against random, TimeSformer baselines with different modalities, and the UNICT 2024 top-1 solution [2509.12990]. The reported DR-MoE system uses RGB only and attains:

- F-score: \(0.57\)
- Correct precision: \(0.97\)
- Correct recall: \(0.60\)
- Mistake precision: \(0.08\)
- Mistake recall: \(0.63\) [2509.12990]

The report states that this is the best F-score among the listed methods and the highest mistake recall, while using only RGB input [2509.12990]. By contrast, the UNICT 2024 top-1 solution is reported with F-score \(0.51\), correct precision \(0.95\), correct recall \(0.93\), and mistake recall \(0.09\) [2509.12990]. The interpretation offered in the report is that DR-MoE substantially improves detection of rare mistakes, which are the tail class in the benchmark [2509.12990].

Cross-domain work indicates that the two-stage reweighted pattern is not confined to video classification. CoRe-MoE presents a two-stage reinforcement learning framework for humanoid locomotion with gait adaptation [2606.04718]. Stage 1 learns a flat-ground locomotion policy with a 2-expert MoE actor for walking and running; Stage 2 adds an 8-expert terrain-aware residual MoE branch whose outputs are fused with the base policy:
\[
a_t = u_t + \epsilon \cdot \Delta a_t,
\qquad
\Delta a_t = \sum_{i=1}^{N} \text{softmax}(g_i) \cdot a_i.
\]
The residual is scaled by \(\epsilon=0.1\) for upper body and \(\epsilon=0.5\) for lower body [2606.04718]. The gating network in Stage 2 is shaped by a SwAV-style contrastive objective to encourage terrain-structured expert specialization [2606.04718].

This suggests that the essential DR-MoE template is portable across domains: a first stage constructs a stable base representation or behavior, and a second stage applies expert-conditioned corrections or decisions, with explicit weighting governing the integration.

## 7. Limitations, misconceptions, and future directions

A common misconception is that DR-MoE necessarily denotes sparse, input-dependent expert routing of the kind used in large-scale language models. The documented egocentric-video formulation does not implement such routing. Its F-MoE and C-MoE use globally learnable weights \(\alpha\) and \(\beta_k\), not input-dependent gating networks [2509.12990]. The architecture is therefore “mixture-of-experts” in the sense of weighted fusion of specialized modules, rather than token-wise sparse dispatch.

Another misconception is that “dual-stage” always means two training phases. In the egocentric mistake-detection report, the duality is architectural: feature-level MoE followed by classification-level MoE, with single-stage joint optimization implied by the use of Adam with one learning rate [2509.12990]. In CoRe-MoE, by contrast, the duality is both architectural and temporal: Stage 1 is trained and then frozen before Stage 2 is trained [2606.04718]. In DirMoE, “dual-stage” refers to the decomposition of routing decisions within a single forward pass [2602.09001].

The reported limitations of the egocentric DR-MoE design include computational cost from using two ViViT backbones and three classifier heads, dependence on a strong Kinetics-pretrained backbone, domain specificity to HoloAssist-style egocentric video, and the use of static gating parameters rather than input-specific gating [2509.12990]. The report identifies potential future directions including input-dependent gating for F-MoE and C-MoE, multimodal extensions using hands, eyes, or audio, more efficient AUC-loss sampling strategies, and transfer to other long-tailed video tasks such as event detection or anomaly detection [2509.12990].

The broader literature suggests additional directions. DirMoE points toward replacing heuristic mixture weighting with probabilistically calibrated routing, ELBO-based objectives, and analytical control over sparsity and mass allocation [2602.09001]. Dynamic-routing results indicate that difficulty-aware variation in the number of active experts can improve performance and efficiency, especially on harder tasks [2403.07652]. CoRe-MoE suggests that expert specialization can be improved by contrastive structuring of gating representations and by preserving a stable base model through residual fusion rather than full joint retraining [2606.04718].

Taken together, these works position DR-MoE as a family of staged expert-mixture designs unified by two principles: decomposition of functionally distinct decisions across stages, and explicit reweighting to combine specialized components under long-tailed, uncertain, or heterogeneous input conditions.

Source: https://www.emergentmind.com/topics/dual-stage-reweighted-mixture-of-experts-dr-moe