Dual Gating MoE in FedVLA Robotics

Updated 7 July 2026

Dual Gating Mixture-of-Experts (DGMoE) is a two-stage routing mechanism that uses token-side and expert-side gates to dynamically allocate experts.
It features residual gating across layers and self-aware experts with learned thresholds to enable sparse, efficient computation.
Integrated in FedVLA, DGMoE enhances adaptive federated aggregation and task-specific specialization for robust robotic manipulation.

Searching arXiv for the cited papers to ground the article in current preprints. Dual Gating Mixture-of-Experts (DGMoE) is a two-stage mixture-of-experts routing mechanism introduced within the federated vision-language-action framework FedVLA for robotic manipulation. In that formulation, DGMoE is designed for multi-modal, heterogeneous, and resource-constrained settings in which standard MoE routing with a fixed top- $k$ can be inefficient and inflexible. Its defining feature is bidirectional token-expert selection: a token-side gate proposes expert assignments, while “self-aware experts” apply learned thresholds to accept or reject those proposals, producing dynamic sparse activation with a variable number of active experts per token (Miao et al., 4 Aug 2025).

1. Definition and conceptual scope

DGMoE appears in FedVLA as a core mechanism for adaptive expert selection under task heterogeneity. The paper describes it as comprising $K$ experts and two gates: a token-side gate $G_t$ that produces routing probabilities with residual accumulation across layers, and an expert-side gate $G_e$ that filters token assignments using learned per-expert thresholds. In the overview and method descriptions, DGMoE is located in the model trunk after a multi-head attention block, and its activation statistics are later reused by the federated server for Expert-Driven Aggregation (EDA) (Miao et al., 4 Aug 2025).

The central idea is not merely sparse routing, but a specific form of bidirectional routing autonomy. Tokens choose experts through soft routing scores, and experts may accept or reject tokens through thresholding. The paper characterizes this as “self-aware experts” that decide their activation. Unlike a standard MoE with a fixed top- $k$ policy, DGMoE does not enforce a hard top- $k$ or capacity limit; sparsity emerges through the learned expert thresholds rather than through an explicit routing cap (Miao et al., 4 Aug 2025).

A useful contrast is provided by the universal gating literature on heterogeneous experts. “Towards a Universal Gating Network for Mixtures of Experts” studies expert-selection gates for independently trained heterogeneous models, including a universal gate over expert activations, but it does not explicitly implement dual gating. Its relevance to DGMoE is therefore conceptual rather than architectural: it shows how expert-selection gates can be learned from activation signatures, whereas DGMoE adds a second gate that operates at the expert side (Kang et al., 2020).

2. Routing formulation and forward pass

A DGMoE layer contains $K$ experts, $E=\{E_1,\ldots,E_K\}$ , where each expert is typically an FFN block, though the formulation is stated more generally. For an input token $x$ at DGMoE layer $j$ , the token-side gate combines current routing logits with residual gating information from the previous DGMoE layer:

$K$ 0

Here $K$ 1 denotes learnable weights for current-layer raw routing logits, and $K$ 2 aggregates prior gating scores. The token-to-expert routing probabilities are then obtained by softmax normalization:

$K$ 3

This residual term is explicitly described as enabling smooth knowledge transitions across layers and allowing tokens to carry forward prior routing preferences, which the paper associates with stabilized specialization across depth (Miao et al., 4 Aug 2025).

The second gate is expert-side thresholding. Each expert has a learnable gating value, collected in a vector $K$ 4, and applies a sign gate to determine whether to accept a token:

$K$ 5

The scalar $K$ 6 scales the threshold, and the paper sets $K$ 7 to balance activation versus filtering. Final routing is then defined by

$K$ 8

with the layer output given by

$K$ 9

This construction yields several behaviors that distinguish DGMoE from fixed-top- $G_t$ 0 MoE. The number of active experts varies per token; the expert gate can reject a token even when its token-side routing score is not negligible; and routing preferences can persist across DGMoE layers through the residual gating term. The paper’s concise summary captures the mechanism as: tokens propose; experts decide (Miao et al., 4 Aug 2025).

3. Self-aware experts and specialization

The term “self-aware expert” refers to the learned per-expert gating parameter that determines how selective an expert is for incoming tokens. Mechanically, an expert is active for token $G_t$ 1 only when the token’s routing score $G_t$ 2 exceeds the scaled threshold $G_t$ 3. A higher $G_t$ 4 raises the bar for activation, so experts regulate their own usage by learned selectivity rather than by a global hand-coded routing rule (Miao et al., 4 Aug 2025).

The self-aware parameters $G_t$ 5 are trained end-to-end with the rest of the model. The paper does not describe an additional explicit confidence loss for these parameters; instead, specialization is said to emerge through thresholded routing and task supervision. This is an important point because it separates DGMoE from approaches that enforce specialization through auxiliary balancing or confidence objectives. In FedVLA, no explicit load-balancing losses or noise-in-gating terms are reported (Miao et al., 4 Aug 2025).

The empirical analysis reports both sparsity and specialization. Across four tasks, the average number of experts activated per token is close to one: Open Drawer, 1.219; Sorting Pills, 1.229; Trash Collection, 1.225; Clean Up, 1.227. The same analysis also shows expert preference patterns by object category: Target Objects frequently activate experts $G_t$ 6; Surrounding Objects prefer experts $G_t$ 7; Background Objects favor $G_t$ 8. This indicates content-aware specialization and efficient compute allocation to task-relevant tokens (Miao et al., 4 Aug 2025).

A plausible implication is that DGMoE’s expert-side gate acts as a selective bottleneck that sharpens semantic partitioning beyond what soft token routing alone would provide. That interpretation is consistent with the reported object-category preference patterns, but the paper itself frames the observation descriptively rather than as a formal causal decomposition (Miao et al., 4 Aug 2025).

4. Integration into FedVLA

Within FedVLA, DGMoE is part of a larger vision-language-action architecture split into Stem, Trunk, and Head. The stem performs input preprocessing; the trunk performs multi-modal representation learning with attention and DGMoE; the head predicts robot actions. Only the trunk is federated and aggregated, while the stem and head remain client-personalized (Miao et al., 4 Aug 2025).

The upstream component feeding DGMoE is the Instruction-Oriented Scene-Parsing (IOSP) mechanism. IOSP uses CLIP and YOLOv8 to extract instruction-guided object-level tokens partitioned into Target Objects, Surrounding Objects, and Background Objects, selecting top-8 tokens per object group. Each group may also be refined via an MoE module to strengthen intra-group features. These group tokens are concatenated with proprioception, processed by a multi-head attention block, and then passed through DGMoE layers for adaptive expert processing (Miao et al., 4 Aug 2025).

This placement matters for understanding DGMoE’s function. IOSP produces instruction-aligned object tokens, and DGMoE performs adaptive computation over those structured tokens. Because routing probabilities accumulate across layers and experts self-filter through thresholds, the trunk can allocate more compute to task-relevant tokens and less to background tokens. The paper’s expert-preference analysis by object category is presented as evidence of synergy between IOSP’s object structuring and DGMoE’s adaptive routing (Miao et al., 4 Aug 2025).

The action prediction head is trained with Huber loss. The end-to-end training objective for the global trunk parameters $G_t$ 9 is

$G_e$ 0

Training uses Adam with batch size 256. The paper reports a simulation learning rate of $G_e$ 1, a real-world learning rate of $G_e$ 2, 1,000 rounds, and 5 local epochs per round (Miao et al., 4 Aug 2025).

5. Federated operation and Expert-Driven Aggregation

DGMoE is not only an in-model routing mechanism; in FedVLA it also produces statistics that guide federated aggregation. During local training, each client records an expert selection matrix $G_e$ 3 that counts how often each expert is activated at each layer. These activation counts summarize which experts were most responsible for the client’s data and tasks during a communication round (Miao et al., 4 Aug 2025).

At the server, EDA computes pairwise expert-selection similarity between clients and uses it to weight layer-wise aggregation. For layer $G_e$ 4, the client-specific expert-selection vector is

$G_e$ 5

The cosine similarity between clients $G_e$ 6 and $G_e$ 7 at layer $G_e$ 8 is

$G_e$ 9

The resulting aggregation weight for client $k$ 0 at layer $k$ 1 is

$k$ 2

and the server updates trunk layer $k$ 3 via

$k$ 4

The role of DGMoE in this federated design is explicit: its expert specialization patterns generate the activation statistics that EDA exploits. The paper argues that this improves aggregation under non-IID data by preferentially blending updates from clients that activate similar experts, while sharing only model updates and activation counts rather than raw data (Miao et al., 4 Aug 2025).

This federated usage differentiates DGMoE from generic MoE routing proposals. In FedVLA, routing is not confined to inference-time or representation-time computation; it becomes a signal for server-side coordination. A plausible implication is that DGMoE couples representational modularity and federated personalization more tightly than a conventional dense trunk would, although the paper states this operationally through EDA rather than as a general theory of modular federated learning (Miao et al., 4 Aug 2025).

The empirical results reported for FedVLA attribute substantial performance and efficiency gains to DGMoE. In simulation, FedVLA with DGMoE, IOSP, and EDA achieves 63.3% average success, compared with centralized training at 65.0% and FedAvg at 51.7%. In real-world experiments, FedVLA achieves 63.3% average success, compared with centralized training at 63.4% and FedAvg at 53.3%. The real-world table further reports that FedVLA matches centralized training on Trash Collection and Sorting Pills and outperforms FedAvg consistently (Miao et al., 4 Aug 2025).

Ablation results show that removing DGMoE sharply degrades performance. In real-world experiments, average success drops from 63.3% to 31.7% without DGMoE, and Clean Up drops from 53.3% to 20.0%. Validation loss curves are reported to be lower and more stable for FedVLA than for ablated variants, with interaction-heavy tasks exhibiting larger loss increases when DGMoE is removed (Miao et al., 4 Aug 2025).

The paper also emphasizes efficiency. Because the average expert density per token is approximately 1.22 rather than a fixed larger $k$ 5, DGMoE significantly improves computational efficiency compared to its vanilla counterpart. This efficiency claim is directly tied to sparse activation: learned thresholds often result in near-single-expert routing per token while preserving success rates near centralized training (Miao et al., 4 Aug 2025).

Several limitations are also stated. The paper does not report explicit DGMoE-specific failure cases or a detailed analysis of hyperparameter sensitivity, including sensitivity to $k$ 6. Communication overheads and scalability to very large expert pools are not discussed. Future work is framed broadly in federated robotic learning terms rather than as a set of enumerated DGMoE modifications such as explicit load balancing, capacity constraints, or communication-efficient expert statistics (Miao et al., 4 Aug 2025).

Two adjacent lines of MoE research help situate DGMoE without altering its specific definition. “Towards a Universal Gating Network for Mixtures of Experts” investigates reusable gates for heterogeneous pre-trained experts and suggests how expert-selection gates can be learned from expert activations, but it does not implement DGMoE itself (Kang et al., 2020). “Rethinking Multinomial Logistic Mixture of Experts with Sigmoid Gating Function” provides a theoretical analysis of sigmoid-gated MoE, including identifiability and sample-complexity comparisons against softmax gating, and develops theory for modified sigmoid gates with $k$ 7 weights, temperature effects, and Euclidean-affinity gating (Pham et al., 1 Feb 2026). These works are relevant as neighboring gating paradigms rather than as direct descriptions of FedVLA’s DGMoE.

Taken together, the available evidence defines DGMoE as a two-stage routing mechanism specialized for privacy-preserving, non-IID, multi-modal robotic manipulation: a token-side soft router with residual gating memory, an expert-side learned threshold gate, sparse adaptive activation, and federated reuse of expert-selection statistics for aggregation. Its documented strengths are adaptive sparsity, expert specialization by object category, and success rates close to centralized training in both simulation and real-world robotic manipulation (Miao et al., 4 Aug 2025).