ElasticMoE: Efficient MoE Autoscaling

• ElasticMoE is an elastic inference framework for MoE models that achieves fine-grained scaling without downtime or excessive memory overhead.
• It decouples inference execution from memory management using zero-copy remapping and high-bandwidth peer-to-peer transfers, significantly reducing scale-up latency.
• In ELMoE-3D, ElasticMoE also refers to tuning model complexity along expert and bit-width axes for faster, energy-efficient self-speculative decoding.

ElasticMoE denotes, in its primary usage, an elastic inference framework for serving large-scale Mixture-of-Experts (MoE) LLMs with fine-grained, low-latency, and zero-downtime scaling. It was introduced as a response to a specific systems problem in MoE serving: only a small subset of experts is activated per token, but parallelized inference pipelines make elastic serving difficult, especially under bursty, short-lived cloud traffic. In this framework, inference execution is decoupled from HBM management, weights and KV caches are reused via zero-copy remapping and peer-to-peer transfers, and expert parallelism is reconfigured in place through virtual-memory-based expert redistribution (Singh et al., 2 Oct 2025). In later literature, the same label is also used in a different sense: the intrinsic elasticity of MoE models along the number-of-experts and bit-width axes, exploited to construct a lightweight draft model for self-speculative decoding in ELMoE-3D (Choi et al., 16 Apr 2026).

1. Problem setting and design objective

The autoscaling formulation of ElasticMoE begins from the limitations of existing scaling strategies for MoE LLMs. Horizontal scaling provisions entire replicas of the current configuration, often tens to hundreds of accelerators, which produces coarse granularity, long provisioning delays, and costly overprovisioning. Vertical scaling offers finer adjustments, but naive forms typically require instance restarts and therefore incur downtime. The framework is explicitly targeted at cloud deployments in which traffic is bursty and short-lived, making both coarse replication and cold restart behavior operationally inefficient (Singh et al., 2 Oct 2025).

ElasticMoE is designed to add or remove just a few NPUs on the fly while restoring Service Level Objective attainment and minimizing cost. Its stated objective is a combination of exact scaling granularity, low scale-up latency, zero downtime, and low memory overhead. This positioning is important because MoE serving differs from dense-model serving not only in activation sparsity but also in the coupling between expert parallelism, memory residency, and runtime execution state.

A recurrent source of ambiguity is that “elasticity” in this context does not merely mean standard autoscaling. In ElasticMoE, elasticity is tied to live reconfiguration of expert-parallel MoE inference without tearing down model state. This suggests a narrower and more systems-specific meaning than generic autoscaling terminology.

2. Runtime architecture

ElasticMoE is organized around three components: an Inference Execution Engine (IMM), an HBM Management Module (HMM), and a virtual-memory-based expert redistribution mechanism (Singh et al., 2 Oct 2025).

The IMM hosts one active and multiple standby vLLM-compatible inference instances. It includes a Zero-Copy Loader that attaches to model weights and KV caches served by the HMM without disk I/O, and an Instance Manager that pre-initializes likely future configurations in CPU memory for rapid activation. The architectural implication is that configuration changes can be prepared ahead of demand spikes rather than synthesized only after a scale event occurs.

The HMM is a persistent daemon that holds all model weights and KV caches in device memory. It is divided into a global control plane, described as Ray-based, and per-device workers. The module exposes three primitives: zero-copy sharing via Ascend IPC, high-bandwidth P2P transfers via HCCL, and virtual page remapping. On a scale event, the control plane computes a minimal-cost remapping plan, and workers execute remapping or transfers on each NPU asynchronously.

The expert redistribution layer addresses a specific kernel constraint: expert weights are stored as non-contiguous physical pages but mapped to a contiguous virtual address range required by inference kernels. During expert-parallelism reconfiguration, only the virtual-to-physical page table is updated, so large buffer reallocations or copies are avoided. In systems terms, this converts EP reconfiguration from a bulk data-movement problem into a page-mapping problem.

3. Memory reuse, remapping, and expert migration

ElasticMoE’s memory path is built around zero-copy weight and KV-cache remapping. Tensors are created with an IPC-safe allocator, IpcSafeAllocator, so they can be shared across processes. The HMM exports a handle $H$ via rtIpcSetMemoryName, and the IMM imports it via rtIpcOpenMemory. Address translation is described by the mapping rule that if $VA_{base}$ is the virtual base address and $p_i$ the $i$-th physical page, then $VA_{base} + i\cdot P$ maps to $PA_i$. The remapping cost per page is $O(1)$, and the total remapping cost is $O(N_{pages})$ (Singh et al., 2 Oct 2025).

For newly added accelerators, ElasticMoE uses high-bandwidth peer-to-peer transfers rather than host-mediated copies. The p2p_copy primitive invokes HCCL isend/irecv or broadcast over the Ascend Unified Bus, bypassing host memory. Its latency model is given as

$T_{copy} \approx S/B,$

where $S$ is the tensor size and $VA_{base}$0 the effective bus bandwidth, with the example range $VA_{base}$1–$VA_{base}$2 GB/s. Shared NPUs are reused at zero cost, and P2P is applied only to tensors that must be placed on newly added accelerators.

Expert migration is expressed algorithmically as a sequence of four steps on each device: allocate physical pages for incoming experts, perform P2P copies of expert weights, remap virtual pages for the new expert placement, and unmap pages for dropped experts. The complexity is $VA_{base}$3 for $VA_{base}$4 total experts, again with per-page remap cost $VA_{base}$5. The peak-memory model is stated as

$VA_{base}$6

and

$VA_{base}$7

where $VA_{base}$8–$VA_{base}$9 extra for overlapping mappings, measured as approximately $p_i$0 GB versus approximately $p_i$1 GB for the baseline. This indicates that ElasticMoE treats memory overhead during reconfiguration as a bounded overlap effect rather than a duplicate-allocation event.

4. Scaling policy and execution semantics

ElasticMoE’s coordinator uses SLO-based triggers rather than raw resource thresholds. The paper defines

$p_i$2

and

$p_i$3

The coordinator monitors sliding windows of TTFT and TPOT and enforces thresholds $p_i$4 and $p_i$5. Scale-up is triggered when $p_i$6, with $p_i$7 given as an example threshold; scale-down is triggered when $p_i$8 and resources are idle (Singh et al., 2 Oct 2025).

The framework is positioned against two alternative execution semantics. In horizontal scaling, the increment is $p_i$9 and the cost function is

$i$0

with coarse granularity. In vertical cold restart, $i$1 is exact, but downtime is $i$2, and the new cost applies only after restart. In ElasticMoE, $i$3 is exact, $i$4, scale-up latency $i$5 is much smaller than both $i$6 and $i$7, and the resource cost is

$i$8

This formulation makes the framework’s intended contribution precise: it is not merely faster rescheduling, but exact-granularity vertical scaling without restart semantics. A plausible implication is that the method is especially suited to deployments where SLO violations emerge faster than conventional provisioning cycles can respond.

5. Empirical results, baselines, and stated limitations

The evaluation is conducted on a Huawei CloudMatrix384 supernode with 384 Ascend 910C NPUs and 192 Kunpeng CPUs. The three MoE models are DeepSeek V2 Lite (16B, 64 experts, activate 6), Qwen 3-30B-A3B (30.5B, 128 experts, activate 8), and DeepSeek V3 (671B, 256 experts, activate 8). Workloads include synthetic online and offline settings with fixed, variable, and patterned RPS, and the reported metrics include scaling latency, downtime, peak memory, TTFT, TPOT, SLO attainment, and SLO/XPU (Singh et al., 2 Oct 2025).

The key reported results are specific. For scale-up latency, ElasticMoE completes in $i$9 the best baseline, corresponding to approximately $VA_{base} + i\cdot P$0 speedup and about $VA_{base} + i\cdot P$1–$VA_{base} + i\cdot P$2 seconds versus $VA_{base} + i\cdot P$3–$VA_{base} + i\cdot P$4 seconds. During scaling, throughput is reported as $VA_{base} + i\cdot P$5 req/s for ElasticMoE versus $VA_{base} + i\cdot P$6 req/s for Cold Restart. Peak memory is approximately $VA_{base} + i\cdot P$7 GB versus $VA_{base} + i\cdot P$8 GB for the Extravagant baseline. For SLO attainment, ElasticMoE maintains at least $VA_{base} + i\cdot P$9 up to approximately $PA_i$0 RPS, while baselines collapse below $PA_i$1 by $PA_i$2 RPS. The speedups are reported as consistent across three runs, with standard deviation below $PA_i$3 in ablations.

The framework is compared with horizontal approaches such as Ray Serve and Replica, with Vertical Cold-Restart approaches such as Dynamollm et al., and with Vertical Concurrent approaches such as CoCoServe. Horizontal approaches are described as simple but coarse and as over-provisioning experts per instance, wasting memory. Vertical Cold-Restart offers finer granularity but suffers downtime and cold-start delays. Vertical Concurrent avoids downtime by overlapping instances, but doubles peak memory and throttles throughput. ElasticMoE is distinguished by the combination of zero-copy reuse, P2P transfers, and in-place expert remapping.

The paper also states explicit limitations. Tensor-parallel degree is kept fixed, which simplifies remapping but limits scaling granularity in high-TP models. There is a throughput dip during transition because the active instance pauses intake of new requests, slightly reducing batch sizes. The stated future directions are to relax the TP constraint for fully flexible TP reconfiguration and to design dual-instance serving so old and new configurations co-serve requests at full capacity.

6. Alternative usage in ELMoE-3D

In ELMoE-3D, “ElasticMoE” refers to the intrinsic elasticity of a sparse MoE model along two orthogonal axes: the number of experts consulted per token and the bit-width of each expert weight (Choi et al., 16 Apr 2026). This is a distinct usage from the autoscaling framework. Here the term is used to construct a lightweight draft sub-model for self-speculative decoding that fits in high-bandwidth hybrid-bonded DRAM and remains strongly aligned with the full model.

The motivation is explicitly memory-centric. On-premises MoE serving is described as fundamentally memory-bound because each token’s expert-FFN layer reads large weight matrices with low reuse. The arithmetic intensity is given as

$PA_i$4

where $PA_i$5 is batch size and $PA_i$6. The two elasticity axes are then defined as expert elasticity and bit-width elasticity. For expert elasticity, heavy-tailed expert-gating distributions imply that only a small subset of experts contributes most of the routing mass, and the model defines $PA_i$7. For bit-width elasticity, the full model uses INT8 weights, while the most significant four bits form a valid coarse approximation; truncating to $PA_i$8 bits reduces storage by $PA_i$9, and bit-nested quantization ensures that the $O(1)$0-bit MSB slice is itself a valid quantization of the full INT8 weight.

The draft and verify costs are modeled as

$O(1)$1

and

$O(1)$2

with $O(1)$3 the acceptance ratio. Overall speculative-decoding latency is written as

$O(1)$4

Operationally, the draft phase restricts routing to a fixed hot expert pool of size $O(1)$5 and loads only the $O(1)$6-bit MSB slice from hybrid-bonded on-die DRAM, using the same scale as the full $O(1)$7-bit model. The verify phase restores full routing over $O(1)$8 experts and combines cached MSB data with LSB data from external memory. The paper states that $O(1)$9 is often less than $O(N_{pages})$0 of a single full layer pass and that the acceptance rate remains high, above $O(N_{pages})$1.

ELMoE-3D couples this ElasticMoE formulation to a 3D-stacked architecture with a logic die, on-die HB DRAM, and off-package LPDDR5. Its roofline expressions are

$O(N_{pages})$2

and

$O(N_{pages})$3

The reported performance is an average $O(N_{pages})$4 speedup and $O(N_{pages})$5 energy efficiency gain over naive MoE serving on xPU across batch sizes $O(N_{pages})$6–$O(N_{pages})$7, together with $O(N_{pages})$8 speedup and $O(N_{pages})$9 energy efficiency gain over the best-performing prior accelerator baseline. In this literature, ElasticMoE therefore denotes not live autoscaling but a model-and-hardware co-design principle for exploiting MoE sparsity and quantization elasticity. The two usages are related by a common concern with dynamic efficiency in MoE serving, but they address different layers of the serving stack.