Dual-Stage Reweighted Mixture-of-Experts (DR-MoE)

• DR-MoE is a two-stage framework that first mixes distinct feature extractors and then fuses classifier outputs using global learnable weights.
• It combines a frozen ViViT backbone and a LoRA-tuned branch for robust feature fusion, addressing subtle mistakes in long-tailed egocentric video data.
• Empirical results on HoloAssist demonstrate improved mistake recall and balanced performance, with potential extensions to dynamic routing and reinforcement learning applications.

Searching arXiv for the cited papers to ground the article and verify metadata. Dual-Stage Reweighted Mixture-of-Experts (DR-MoE) denotes a class of architectures in which mixture-of-experts is applied in two sequential stages, with explicit reweighting at one or both stages to combine complementary experts, losses, or policy branches. In the literature represented here, the term refers most directly to a framework for long-tailed egocentric mistake detection that first mixes feature extractors and then mixes classifiers, using learnable coefficients $\alpha$ and $\beta_k$ to modulate the contribution of each component (Han et al., 16 Sep 2025). The same phrase also describes a broader architectural motif: one stage determines or constrains which experts or branches should be emphasized, and a second stage reweights the resulting representations or decisions. This broader interpretation is made explicit by probabilistic routing formulations such as DirMoE, where expert selection and contribution are separated into distinct random variables and combined through a normalized product (Vahidi et al., 9 Feb 2026), and by two-stage control systems such as CoRe-MoE, where a base policy is learned first and a residual MoE branch is added and fused later (Huang et al., 3 Jun 2026).

1. Definition and conceptual scope

In its narrowest documented sense, DR-MoE is a framework proposed for egocentric mistake detection from video, designed to address subtle and infrequent mistakes under a strongly long-tailed class distribution (Han et al., 16 Sep 2025). The task input consists of a coarse egocentric video $V_{0:T}$ and a fine-grained action segment $V_{t:t+\tau}\subseteq V_{0:T}$, and the model outputs a binary decision $\hat{y}\in\{0,1\}$, where $1$ denotes “mistake” (Han et al., 16 Sep 2025). The architecture is “dual-stage” because it introduces mixture-of-experts at two levels: a feature-level expert module and a classification-level expert module (Han et al., 16 Sep 2025).

The feature stage combines a frozen ViViT backbone and a LoRA-tuned ViViT backbone, while the classification stage fuses three classifier heads trained with different objectives: reweighted cross-entropy, AUC loss, and label-aware loss with sharpness-aware minimization (Han et al., 16 Sep 2025). The “reweighted” designation refers both to explicit class-frequency-based reweighting in the losses and to learnable mixing coefficients over experts (Han et al., 16 Sep 2025).

A broader reading of DR-MoE is supported by subsequent and adjacent work. DirMoE is described as being “explicitly designed around the same conceptual decomposition that ‘Dual-Stage Reweighted MoE (DR-MoE)’ suggests,” namely a first stage for expert selection and a second stage for expert contribution or reweighting (Vahidi et al., 9 Feb 2026). CoRe-MoE similarly instantiates a two-stage structure in reinforcement learning: a base locomotion policy is learned first, and a terrain-aware residual MoE branch is introduced in the second stage, with the final action obtained by weighted fusion (Huang et al., 3 Jun 2026). This suggests that DR-MoE is best understood not as a single fixed implementation, but as an architectural pattern centered on staged decomposition and explicit weighting.

2. Canonical two-stage formulation in egocentric mistake detection

The canonical DR-MoE formulation is organized as two sequential expert mixtures (Han et al., 16 Sep 2025). Stage 1 performs feature extraction and feature fusion; Stage 2 performs classification and classifier fusion.

At the feature level, DR-MoE uses two ViViT-based experts:

1. $f_{\text{ori}}$: a frozen ViViT pretrained on Kinetics, applied to the coarse video $V_{0:T}$.
2. $f_{\text{lora}}$: a LoRA-tuned ViViT applied to the fine-grained segment $V_{t:t+\tau}$ (Han et al., 16 Sep 2025).

These are combined by the feature mixture-of-experts module: $\beta_k$0 where $\beta_k$1 is a learnable mixing parameter (Han et al., 16 Sep 2025).

At the classification level, three classifiers $\beta_k$2 operate on $\beta_k$3, each optimized for a different aspect of long-tailed binary classification (Han et al., 16 Sep 2025). Their outputs are fused by the classification mixture-of-experts module: $\beta_k$4 where $\beta_k$5 are learned expert weights (Han et al., 16 Sep 2025).

Within the report, $\beta_k$6 and $\beta_k$7 are not described as input-dependent gating networks; rather, they are global learnable weights (Han et al., 16 Sep 2025). The first stage therefore decides how much the joint representation should rely on generic versus task-adapted video features, whereas the second stage decides how much to trust classifiers specialized for imbalance correction, ranking, or calibration (Han et al., 16 Sep 2025). This division of labor is central to the architecture’s definition.

3. Feature-stage mechanism and backbone specialization

The feature stage is built on ViViT-B/$\beta_k$8, using clips of 32 frames in RGB modality (Han et al., 16 Sep 2025). One branch remains frozen and serves as a source of generic spatiotemporal semantics; the other branch is adapted using LoRA to focus on fine-grained mistake-sensitive cues (Han et al., 16 Sep 2025).

LoRA is applied with rank $\beta_k$9 to attention-layer query and value projection matrices (Han et al., 16 Sep 2025). For an attention projection, the effective weight is written as

$V_{0:T}$0

where $V_{0:T}$1, $V_{0:T}$2, and $V_{0:T}$3 (Han et al., 16 Sep 2025). Only $V_{0:T}$4 and $V_{0:T}$5 are trained, while $V_{0:T}$6 remains fixed (Han et al., 16 Sep 2025). The stated rationale is efficient training, reduced overfitting relative to full finetuning, avoidance of catastrophic forgetting, and increased sensitivity to detailed mistake patterns (Han et al., 16 Sep 2025).

The feature-level MoE is therefore not a sparse token-level router in the language-model sense; it is a learned convex combination of two semantically distinct video experts. The frozen branch encodes coarse human action semantics from the full video, while the LoRA branch emphasizes mistake-relevant details from the localized segment (Han et al., 16 Sep 2025). Because $V_{0:T}$7 is learnable, the architecture can shift globally between generic and task-specific cues without committing to a single backbone (Han et al., 16 Sep 2025).

A plausible implication is that this stage performs representational bias management: the frozen model preserves broad action understanding inherited from Kinetics pretraining, and the LoRA-tuned model injects high-resolution adaptation to the mistake-detection domain. The report itself does not supply an ablation table for $V_{0:T}$8, but it states that the dual-stage design is intended to balance generic semantics with subtle error cues (Han et al., 16 Sep 2025).

4. Classification-stage mechanism and reweighting strategies

The second stage comprises three classifier heads whose differentiation lies primarily in their loss functions rather than in distinct architectural complexity (Han et al., 16 Sep 2025). The report states that they are typically implemented as linear layers, possibly with a small MLP, mapping $V_{0:T}$9 to logits or scores over $V_{t:t+\tau}\subseteq V_{0:T}$0 (Han et al., 16 Sep 2025).

The three objectives are summarized below.

Classifier	Objective	Stated role
$V_{t:t+\tau}\subseteq V_{0:T}$1	Reweighted cross-entropy	Mitigate class imbalance
$V_{t:t+\tau}\subseteq V_{0:T}$2	AUC loss	Improve ranking under skewed distribution
$V_{t:t+\tau}\subseteq V_{0:T}$3	Label-aware loss with SAM	Enhance calibration and generalization

For the reweighted cross-entropy head, the loss is

$V_{t:t+\tau}\subseteq V_{0:T}$4

where $V_{t:t+\tau}\subseteq V_{0:T}$5 is the class frequency (Han et al., 16 Sep 2025). This explicitly upweights rare mistake examples and downweights frequent correct-action examples (Han et al., 16 Sep 2025).

For the AUC head, the loss is written as

$V_{t:t+\tau}\subseteq V_{0:T}$6

with $V_{t:t+\tau}\subseteq V_{0:T}$7 and $V_{t:t+\tau}\subseteq V_{0:T}$8 denoting positive and negative sample counts (Han et al., 16 Sep 2025). The head is optimized as a pairwise ranker rather than a direct probabilistic classifier (Han et al., 16 Sep 2025).

For the label-aware head, the loss is

$V_{t:t+\tau}\subseteq V_{0:T}$9

where $\hat{y}\in\{0,1\}$0 is the class-frequency vector (Han et al., 16 Sep 2025). This is combined with sharpness-aware minimization: $\hat{y}\in\{0,1\}$1 with radius $\hat{y}\in\{0,1\}$2 (Han et al., 16 Sep 2025).

These heads are then fused through C-MoE by learned weights $\hat{y}\in\{0,1\}$3 (Han et al., 16 Sep 2025). In documented form, the fusion is not sample-adaptive. Instead, DR-MoE integrates multiple long-tail-aware training criteria into a single prediction pipeline through learned global coefficients. This contrasts with sparse MoE routers in LLMs, where gating is typically token-dependent.

5. Relation to probabilistic and dynamic-routing MoE formulations

The expression “dual-stage reweighted MoE” also describes a more general decomposition in which selection and weighting are explicitly separated. DirMoE provides the clearest formalization of this view (Vahidi et al., 9 Feb 2026). It models routing using a spike-and-slab structure over experts: a binary selection mask $\hat{y}\in\{0,1\}$4 and simplex-valued contribution weights $\hat{y}\in\{0,1\}$5, with final routing weights

$\hat{y}\in\{0,1\}$6

The paper states that this is “exactly a ‘dual-stage, reweighted’ mechanism: discrete selection $\hat{y}\in\{0,1\}$7, then continuous reweighting $\hat{y}\in\{0,1\}$8 among the selected experts, turned into final mixture weights $\hat{y}\in\{0,1\}$9” (Vahidi et al., 9 Feb 2026).

In DirMoE, Stage 1 uses relaxed Bernoulli selection via Gumbel-Sigmoid,

$1$0

and Stage 2 uses a Dirichlet-distributed contribution vector conditioned on $1$1 (Vahidi et al., 9 Feb 2026). The whole router remains differentiable through Gumbel-Sigmoid relaxation and implicit reparameterization for the Dirichlet distribution (Vahidi et al., 9 Feb 2026). The training objective is a variational ELBO with a direct sparsity penalty that targets exactly $1$2 active experts in expectation (Vahidi et al., 9 Feb 2026).

Dynamic-routing work offers a different but related perspective. “Harder Tasks Need More Experts: Dynamic Routing in MoE Models” does not use the name DR-MoE, but it presents a two-phase routing process: first compute a full expert distribution $1$3 by softmax, then select as many experts as needed for the cumulative probability to exceed a threshold $1$4 (Huang et al., 2024). The number of active experts is therefore a function of confidence rather than a fixed $1$5 (Huang et al., 2024). The paper explicitly interprets the first phase as confidence estimation and the second as adaptive expert-count selection, making it closely related to a staged MoE viewpoint (Huang et al., 2024).

These formulations highlight a conceptual distinction. In egocentric mistake detection, DR-MoE uses two semantically different MoE modules at two points in the pipeline (Han et al., 16 Sep 2025). In differentiable MoE routing for LLMs, the same phrase can denote the separation of “which experts” from “how much they contribute” within a single routing layer (Vahidi et al., 9 Feb 2026). This suggests that DR-MoE has both an application-specific meaning and a cross-domain architectural meaning.

6. Empirical results, applications, and cross-domain variants

In the documented egocentric-video application, DR-MoE is evaluated on HoloAssist for mistake detection (Han et al., 16 Sep 2025). Table 1 in the report lists results on the test set against random, TimeSformer baselines with different modalities, and the UNICT 2024 top-1 solution (Han et al., 16 Sep 2025). The reported DR-MoE system uses RGB only and attains:

• F-score: $1$6
• Correct precision: $1$7
• Correct recall: $1$8
• Mistake precision: $1$9
• Mistake recall: $f_{\text{ori}}$0 (Han et al., 16 Sep 2025)

The report states that this is the best F-score among the listed methods and the highest mistake recall, while using only RGB input (Han et al., 16 Sep 2025). By contrast, the UNICT 2024 top-1 solution is reported with F-score $f_{\text{ori}}$1, correct precision $f_{\text{ori}}$2, correct recall $f_{\text{ori}}$3, and mistake recall $f_{\text{ori}}$4 (Han et al., 16 Sep 2025). The interpretation offered in the report is that DR-MoE substantially improves detection of rare mistakes, which are the tail class in the benchmark (Han et al., 16 Sep 2025).

Cross-domain work indicates that the two-stage reweighted pattern is not confined to video classification. CoRe-MoE presents a two-stage reinforcement learning framework for humanoid locomotion with gait adaptation (Huang et al., 3 Jun 2026). Stage 1 learns a flat-ground locomotion policy with a 2-expert MoE actor for walking and running; Stage 2 adds an 8-expert terrain-aware residual MoE branch whose outputs are fused with the base policy: $f_{\text{ori}}$5 The residual is scaled by $f_{\text{ori}}$6 for upper body and $f_{\text{ori}}$7 for lower body (Huang et al., 3 Jun 2026). The gating network in Stage 2 is shaped by a SwAV-style contrastive objective to encourage terrain-structured expert specialization (Huang et al., 3 Jun 2026).

This suggests that the essential DR-MoE template is portable across domains: a first stage constructs a stable base representation or behavior, and a second stage applies expert-conditioned corrections or decisions, with explicit weighting governing the integration.

7. Limitations, misconceptions, and future directions

A common misconception is that DR-MoE necessarily denotes sparse, input-dependent expert routing of the kind used in large-scale LLMs. The documented egocentric-video formulation does not implement such routing. Its F-MoE and C-MoE use globally learnable weights $f_{\text{ori}}$8 and $f_{\text{ori}}$9, not input-dependent gating networks (Han et al., 16 Sep 2025). The architecture is therefore “mixture-of-experts” in the sense of weighted fusion of specialized modules, rather than token-wise sparse dispatch.

Another misconception is that “dual-stage” always means two training phases. In the egocentric mistake-detection report, the duality is architectural: feature-level MoE followed by classification-level MoE, with single-stage joint optimization implied by the use of Adam with one learning rate (Han et al., 16 Sep 2025). In CoRe-MoE, by contrast, the duality is both architectural and temporal: Stage 1 is trained and then frozen before Stage 2 is trained (Huang et al., 3 Jun 2026). In DirMoE, “dual-stage” refers to the decomposition of routing decisions within a single forward pass (Vahidi et al., 9 Feb 2026).

The reported limitations of the egocentric DR-MoE design include computational cost from using two ViViT backbones and three classifier heads, dependence on a strong Kinetics-pretrained backbone, domain specificity to HoloAssist-style egocentric video, and the use of static gating parameters rather than input-specific gating (Han et al., 16 Sep 2025). The report identifies potential future directions including input-dependent gating for F-MoE and C-MoE, multimodal extensions using hands, eyes, or audio, more efficient AUC-loss sampling strategies, and transfer to other long-tailed video tasks such as event detection or anomaly detection (Han et al., 16 Sep 2025).

The broader literature suggests additional directions. DirMoE points toward replacing heuristic mixture weighting with probabilistically calibrated routing, ELBO-based objectives, and analytical control over sparsity and mass allocation (Vahidi et al., 9 Feb 2026). Dynamic-routing results indicate that difficulty-aware variation in the number of active experts can improve performance and efficiency, especially on harder tasks (Huang et al., 2024). CoRe-MoE suggests that expert specialization can be improved by contrastive structuring of gating representations and by preserving a stable base model through residual fusion rather than full joint retraining (Huang et al., 3 Jun 2026).

Taken together, these works position DR-MoE as a family of staged expert-mixture designs unified by two principles: decomposition of functionally distinct decisions across stages, and explicit reweighting to combine specialized components under long-tailed, uncertain, or heterogeneous input conditions.