EC2MoE: Adaptive End-Cloud MoE Inference

• EC2MoE is an adaptive framework for scalable MoE inference across heterogeneous end-cloud systems, leveraging both device-aware and cloud-based resources.
• The HL-GGN module efficiently filters local experts using hardware-aware selection and group-based dynamic routing to cut down computation and memory overhead.
• The PO-ECC component optimizes pipeline execution through low-rank feature compression and route-aware scheduling, achieving notable throughput gains and reduced latency.

EC2MoE denotes an adaptive framework for scalable Mixture-of-Experts inference via end-cloud pipeline collaboration. In the cited literature, the formal system named EC2MoE comprises a hardware-aware lightweight group gate network for efficient expert selection on heterogeneous end devices and a pipeline optimization mechanism based on end-cloud collaboration for reducing transmission and computation costs (Yang et al., 8 Aug 2025). A separate but adjacent line of work uses the phrase “EC2MoE-style” for deployment-oriented MoE serving problems, including cache-aware routing on memory-constrained mobile devices and cost-efficient asynchronous expert-parallel serving; these works are closely related in objective but are distinct in method and scope (Skliar et al., 2024).

1. Problem setting and conceptual scope

EC2MoE targets MoE inference in heterogeneous end-cloud environments, where computation is split across end devices such as smartphones, IoT nodes, edge servers, and resource-constrained terminals, and cloud servers with abundant compute (Yang et al., 8 Aug 2025). The motivating difficulty is not MoE sparsity per se, but the interaction of dynamic expert routing, end-device resource limits, communication overhead, and resource heterogeneity. In the problem formulation, different inputs activate different experts, naive routing requires evaluating many experts or a heavy gate, end devices cannot store or evaluate all experts locally, cloud-only execution incurs transmission overhead and bandwidth sensitivity, and compute and network conditions vary over time.

The framework therefore aims to make MoE inference scalable, latency-efficient, communication-efficient, and robust to heterogeneous and dynamic end-cloud conditions. Its architecture is divided into two major components. The first is HL-GGN, the Hardware-Aware Lightweight Group Gate Network, which performs efficient expert selection on the end side. The second is PO-ECC, Pipeline Optimization based on End-Cloud Collaboration, which addresses execution and communication efficiency across the end-cloud boundary.

The system workflow is described as follows: the end device observes hardware state, local expert candidates are filtered by capability, lightweight gating generates expert routing probabilities, selected intermediate features are compressed and sent to cloud, the cloud reconstructs features and continues inference, and route-aware scheduling assigns pipeline stages across end and cloud. This decomposition makes explicit that EC2MoE does not treat inference as purely local or purely cloud-based; instead, routing, communication, and scheduling are jointly optimized.

2. Hardware-aware lightweight group gate network

HL-GGN is the routing mechanism used for efficient MoE expert selection on the end side (Yang et al., 8 Aug 2025). It contains three elements: hardware-aware local expert selection, a lightweight group gate network, and a two-stage fusion of local and global routing.

The end device status is modeled by a device-state vector,

$$S_{\text{device}}=\{C_{\text{cpu}}, M_{\text{mem}}, P_{\text{power}}, B_{\text{bandwidth}}\},$$

where the components denote available CPU resources, memory status, battery or power budget, and network bandwidth condition. A hardware-aware function maps this state to a capability threshold,

$T_{\text{capability}} = H(S_{\text{device}}).$

Given an expert complexity vector $V_{\text{expert}}$, EC2MoE defines a local candidate set

$$\mathcal{E}_{\text{local}} = \left\{ e_i \mid f(V_{\text{expert}_i}, T_{\text{capability}})\le \epsilon,\ \forall e_i\in\mathcal{E} \right\}.$$

Only experts whose complexity fits current hardware conditions are considered locally; expensive experts are offloaded to cloud. This reduces the candidate set before routing and thereby lowers computation, memory pressure, and routing overhead.

The group-gating component is motivated by the cost of conventional MoE gating. The paper states that a standard gate with weight matrix $W_g$ scales with both the number of experts and the feature dimension, which leads to large parameter count, high matrix multiplication cost, and latency problems on constrained devices. EC2MoE instead partitions $M$ experts into $K$ groups,

$M=\sum_{k=1}^{K} M_k,$

and assigns each group a local gate,

$$g^{(k)}(x)=\operatorname{Softmax}(W_k x+b_k), \quad W_k\in\mathbb{R}^{M_k\times d},\ b_k\in\mathbb{R}^{M_k}.$$

Because each gate handles only a subset of experts, gate computation is reduced.

The final routing probability is produced by a two-stage dynamic fusion strategy. A small global gate first predicts group contribution weights,

$$p_{\text{group}}= \operatorname{Softmax}(W_{\text{global}}x+b_{\text{global}}),$$

with

$T_{\text{capability}} = H(S_{\text{device}}).$0

Within each chosen group $T_{\text{capability}} = H(S_{\text{device}}).$1, a local gate computes $T_{\text{capability}} = H(S_{\text{device}}).$2. For expert $T_{\text{capability}} = H(S_{\text{device}}).$3 in group $T_{\text{capability}} = H(S_{\text{device}}).$4, the final probability is

$T_{\text{capability}} = H(S_{\text{device}}).$5

The paper attributes the efficiency of this design to global coarse filtering, which narrows attention to promising groups, and local fine-grained selection, which compares experts only within small groups. The stated effect is lower gate computation, lower latency, less memory traffic, and still good expert selection quality.

3. End-cloud pipeline optimization

PO-ECC addresses execution and communication bottlenecks in end-cloud inference (Yang et al., 8 Aug 2025). It consists of an encoder-decoder structure based on low-rank compression and a route-aware heuristic pipeline scheduling algorithm.

For intermediate feature transmission, the feature tensor is defined as

$T_{\text{capability}} = H(S_{\text{device}}).$6

The end-side encoder compresses $T_{\text{capability}} = H(S_{\text{device}}).$7 into a low-rank representation,

$T_{\text{capability}} = H(S_{\text{device}}).$8

where $T_{\text{capability}} = H(S_{\text{device}}).$9, $V_{\text{expert}}$0, and $V_{\text{expert}}$1. The cloud-side decoder reconstructs the feature as

$V_{\text{expert}}$2

To balance reconstruction fidelity and downstream task quality, the joint objective is

$V_{\text{expert}}$3

The intended effect is to reduce transmitted feature size, lower bandwidth usage, reduce communication latency, and preserve sufficient reconstruction quality for downstream inference.

The pipeline component treats inference stages as allocable across end and cloud rather than as a fixed partition. This suggests that the framework’s distinctive systems contribution lies not merely in routing fewer experts, but in overlapping communication and computation across heterogeneous resources. The paper states that the key benefit is that the inference path can be split and overlapped, so computation and communication occur asynchronously rather than serially.

4. Scheduling objective and runtime adaptation

EC2MoE formulates scheduling over tasks

$V_{\text{expert}}$4

where each task has computation complexity $V_{\text{expert}}$5 and communication cost $V_{\text{expert}}$6 (Yang et al., 8 Aug 2025). The optimization target is

$V_{\text{expert}}$7

This objective explicitly balances expected execution time and communication cost.

The heuristic priority score is

$V_{\text{expert}}$8

where $V_{\text{expert}}$9 avoids division by zero. High-computation, low-communication tasks therefore receive higher priority. The location rule is given as

$$\mathcal{E}_{\text{local}} = \left\{ e_i \mid f(V_{\text{expert}_i}, T_{\text{capability}})\le \epsilon,\ \forall e_i\in\mathcal{E} \right\}.$$0

The scheduler uses task complexity, task communication cost, current end load, cloud load in the heuristic context, the end capability threshold $$\mathcal{E}_{\text{local}} = \left\{ e_i \mid f(V_{\text{expert}_i}, T_{\text{capability}})\le \epsilon,\ \forall e_i\in\mathcal{E} \right\}.$$1, and the priority threshold $$\mathcal{E}_{\text{local}} = \left\{ e_i \mid f(V_{\text{expert}_i}, T_{\text{capability}})\le \epsilon,\ \forall e_i\in\mathcal{E} \right\}.$$2.

The paper describes this mechanism as greedy and fast, intended to avoid expensive global optimization at runtime. It adapts to varying request rates, varying bandwidth, and changing compute load by recomputing priorities, re-evaluating local feasibility, dynamically shifting tasks between end and cloud, and exploiting parallelism when both sides are available. A plausible implication is that EC2MoE’s scheduling policy is designed for online reconfiguration rather than offline optimal placement.

5. Experimental characterization

The reported evaluation uses GLUE and SQuAD, with Switch Transformer as the MoE backbone, PyTorch with custom modules for expert partitioning, dynamic routing, and end-cloud collaboration, an end device represented by an Intel Xeon Silver 4214R CPU, and a cloud configuration with two NVIDIA A100 GPUs (Yang et al., 8 Aug 2025). Network conditions are simulated with Linux Traffic Control using a base bandwidth of 300 Mbps and fluctuation of 20%. Expert settings include 8, 16, 32, and 64 experts, sequence length 256 tokens, batch size 4, Top-1 expert selection, and a local expert cap of at most 40% of total experts. Reported baselines are BrownoutServe and EdgeMoE.

Across both datasets and all expert counts, EC2MoE is reported to achieve the best accuracy. The paper gives the following example values, ordered as EC2MoE, BrownoutServe, and EdgeMoE:

Experts	GLUE	SQuAD
8	81.6 / 81.2 / 77.6	82.5 / 82.2 / 78.4
16	82.3 / 81.7 / 77.9	82.3 / 82.1 / 77.9
32	80.5 / 80.3 / 76.2	83.1 / 82.7 / 78.1
64	81.4 / 80.8 / 77.8	82.6 / 82.3 / 77.3

The reported average accuracy improvement over EdgeMoE is 4.07% on GLUE and 4.1% on SQuAD, and the framework also slightly surpasses BrownoutServe. In throughput, the paper reports average gains of 2.2× over BrownoutServe and 5.1× over EdgeMoE. In latency, it reports average reductions of 67% relative to BrownoutServe and 53% relative to EdgeMoE. The stated reasons are better overlap of communication and computation, local expert selection that reduces redundant work, reduced communication via compression, local filtering of experts, route-aware scheduling that avoids blocking, and parallel execution across end and cloud.

The scalability experiments consider request rates of 2, 4, 6, 8, and 10 req/s and bandwidth fluctuation settings of 0%, 10%, 20%, 30%, and 40%. Under task-load changes, EC2MoE maintains near-linear throughput growth and latency grows more slowly than the baselines, whereas BrownoutServe saturates under high load and EdgeMoE bottlenecks due to limited local compute. Under dynamic network conditions, EC2MoE remains stable in throughput and latency, BrownoutServe is highly sensitive to bandwidth jitter, and EdgeMoE avoids network dependence but remains limited by local capacity.

6. Ablations, limitations, and relation to adjacent MoE deployment work

The ablation study removes two components, HL-GGN and PO-ECC, and attributes distinct degradations to each (Yang et al., 8 Aug 2025). Without HL-GGN, expert selection becomes less adaptive to device state, accuracy drops by 2.1%, and latency increases by about 23%. Without PO-ECC, inference stages no longer overlap asynchronously, communication and computation become blocked, throughput decreases by 38%, and latency increases by 45%. The paper’s conclusion is that HL-GGN is important for efficient and accurate expert routing and that PO-ECC is important for system-level pipeline efficiency.

The stated assumptions include a heterogeneous but relatively structured end-cloud setup, access to device state information for CPU, memory, power, and bandwidth, sufficiently accurate cloud-side reconstruction, and expert complexity that can be meaningfully compared to hardware capability. The listed likely limitations are dependence on accurate runtime profiling, the need for joint training because the compression encoder-decoder uses end-cloud joint training, a heuristic scheduler that may not always find the best possible allocation, an evaluation scope limited to GLUE and SQuAD with Switch Transformer, and a network model simulated at 300 Mbps with fixed fluctuation ranges.

A common source of confusion is that EC2MoE is not the same method as the training-free cache-aware routing proposed for memory-constrained mobile inference in “Mixture of Cache-Conditional Experts for Efficient Mobile Device Inference” (Skliar et al., 2024). That work modifies the ranking used for expert selection so cached experts are preferred, keeps the original logits for the MoE forward pass, evaluates DeepSeek-V2-Lite, Qwen1.5-MoE-A2.7B, Phi-3.5-MoE, and Mixtral-8x7B on WikiText-2-raw-v1, MMLU, and GSM8K, and reports over a $$\mathcal{E}_{\text{local}} = \left\{ e_i \mid f(V_{\text{expert}_i}, T_{\text{capability}})\le \epsilon,\ \forall e_i\in\mathcal{E} \right\}.$$3 speedup on-device. EC2MoE is likewise distinct from the asynchronous expert-parallel serving system AMoE, which introduces Asynchronous Expert Parallelism, $$\mathcal{E}_{\text{local}} = \left\{ e_i \mid f(V_{\text{expert}_i}, T_{\text{capability}})\le \epsilon,\ \forall e_i\in\mathcal{E} \right\}.$$4-queuing, dynamic re-batching, attention–expert disaggregation, and a defragging scheduler to improve throughput by up to 2.7x on one node and about 3x in a multi-node setting (Wang et al., 13 May 2025). The relationship among these works is methodological rather than nominal: all address MoE deployment efficiency, but they operate at different loci of the stack—cache-aware routing on mobile DRAM, asynchronous expert-parallel serving, and adaptive end-cloud pipeline collaboration, respectively.