Fully Sharded Expert Parallelism

• Fully Sharded Expert Parallel (FSEP) is a parallelization paradigm that shards MoE expert parameters across devices, allowing dynamic reconstruction via All-to-All communication.
• It enables flexible expert re-layout and effective load balancing during both training and inference, improving throughput and reducing memory usage.
• Variants like LAER-MoE, Expert Streaming, and MoEShard adapt the FSEP approach to different hardware setups, achieving significant speedups with minimal communication overhead.

Fully Sharded Expert Parallel (FSEP) denotes a set of Mixture-of-Experts (MoE) parallelization schemes in which expert parameters—and, in some formulations, token activations—are sharded across all participating devices or chiplets rather than bound monolithically to one device. In LAER-MoE, FSEP is a training paradigm that fully partitions each expert parameter by the number of devices and restores partial experts at expert granularity through All-to-All communication during training. In "Expert Streaming," FSEP, also called Fully Sharded Expert Data-Parallelism (FSE-DP), is a fine-grained parallelization paradigm for low-batch MoE inference on multi-chiplet accelerators. In MoEShard, FSEP is an inference strategy that decomposes each expert by a column-wise split of $W_i$ and a row-wise split of $W_o$ so that every GPU performs exactly $\tfrac1P$ of the expert’s total FLOPs, regardless of routing skew (Liu et al., 12 Feb 2026, Ma et al., 29 Mar 2026, Balmau et al., 11 Mar 2025).

1. Terminological scope and shared abstraction

Recent usage does not restrict FSEP to a single execution pattern. The term appears in distributed training, multi-chiplet low-batch inference, and multi-GPU inference, but the recurring structural idea is global expert sharding combined with runtime reconstruction, streaming, or aggregation.

Work	Setting	Defining mechanism
LAER-MoE	MoE training	Fully partitions each expert parameter by the number of devices and restores partial experts at expert granularity through All-to-All communication
Expert Streaming	Low-batch MoE inference on multi-chiplet accelerators	Shards both token activations and expert weights across all chiplets and streams micro-slices over D2D links
MoEShard	Multi-GPU encoder-based MoE inference	Column-wise split of $W_i$ and row-wise split of $W_o$ across $P$ GPUs

Taken together, these formulations suggest that FSEP is best understood as a sharding principle rather than a single fixed runtime. What remains constant is that expert state is globally partitioned; what varies is whether the system reconstructs full experts on demand, streams micro-slices along trajectories, or aggregates partial outputs after shard-local computation (Liu et al., 12 Feb 2026, Ma et al., 29 Mar 2026, Balmau et al., 11 Mar 2025).

2. Training-oriented FSEP in LAER-MoE

In LAER-MoE, the global symbols are $N$ for the number of devices, $E$ for the total number of experts, $C$ for expert-capacity per device, and $\Psi_{\text{expert}}$ for the size of one expert. Every expert’s parameter tensor is fully sharded across all $W_o$0 devices, instead of binding each expert to a single device. Let

$W_o$1

be the flattened parameter vector of expert $W_o$2. FSEP splits $W_o$3 into $W_o$4 disjoint, equal-sized shards,

$W_o$5

where each shard $W_o$6 is stored on device $W_o$7 after the initial shard step. During a forward pass, each device $W_o$8 identifies a set of $W_o$9 experts it must invoke via the gating or routing mechanism; it then issues an All-to-All unshard so that, for each needed expert $\tfrac1P$0, device $\tfrac1P$1 collects the $\tfrac1P$2 shards $\tfrac1P$3 and locally reassembles

$\tfrac1P$4

Backward proceeds similarly: after computing gradients $\tfrac1P$5, each device holds full $\tfrac1P$6 for its $\tfrac1P$7 experts; a second All-to-All reshard splits these gradients back into $\tfrac1P$8 shards and scatter-reduces them so that each device $\tfrac1P$9 adds $W_i$0 to its local shard’s gradient (Liu et al., 12 Feb 2026).

The architectural consequence is explicit in the paper: FSEP exposes the same memory benefits as FSDP, because each device only permanently stores $W_i$1 of every expert, while also enabling dynamic per-iteration expert-placement, since any device can choose any $W_i$2 experts to reconstruct. This dynamic placement is the basis for LAER-MoE’s load-adaptive expert re-layout during training.

3. Communication model, overlap conditions, and fine-grained scheduling

In LAER-MoE, each unshard and each reshard is a balanced $W_i$3-way All-to-All among the $W_i$4 devices. Per device, if it restores $W_i$5 full experts, it must send out $W_i$6 shards and receive $W_i$7 shards on each All-to-All, giving

$W_i$8

bytes per device per All-to-All. By contrast, in a traditional FSDP + EP split where EP group size is $W_i$9 and FSDP group size is $W_o$0 with $W_o$1 and $W_o$2, an FSDP AllGather on $W_o$3 experts costs

$W_o$4

The ratio tends to $W_o$5 as cluster size grows. The paper further states that these models imply that FSEP adds only $W_o$6 extra communication overhead but gains full re-layout flexibility (Liu et al., 12 Feb 2026).

Communication hiding is integral rather than incidental. FSEP interleaves or shadow-launches the All-to-All on a dedicated CUDA stream and overlaps forward unshard for layer $W_o$7 during expert compute of layer $W_o$8, while backward delays reshard of layer $W_o$9 until the next MoE layer’s backward compute. Under a bfloat16 SwiGLU MLP with hidden size $P$0, intermediate $P$1, top-$P$2, and token count $P$3 per device, the compute–comm overlap condition is

$P$4

In practice on A100s and typical $P$5, this holds comfortably, letting the communication be “free.” The fine-grained schedule uses four CUDA streams: Stream S0 for Attention$P$6 compute, Stream S1 for MLP$P$7 expert compute and concurrently prefetch unshard$P$8, Stream S2 for delayed gradient reshard$P$9 scheduled at the start of backward$N$0 compute, and Stream S3 for token-dispatcher All-to-All$N$1. The stated effect is reduced idle gaps and channel contention compared to a single coarse communication phase (Liu et al., 12 Feb 2026).

4. Expert re-layout planning and load balancing

The principal benefit claimed for training-time FSEP is per-iteration freedom to choose exactly which $N$2 experts each device reconstructs. LAER-MoE introduces binary placement variables $N$3 indicating that device $N$4 restores expert $N$5, with $N$6 for all $N$7, and routing variables $N$8 giving the number of tokens originally on device $N$9 routed to expert $E$0 but sent to device $E$1, with $E$2. Total iteration time is modeled as $E$3, where

$E$4

and

$E$5

The joint integer program minimizes $E$6 subject to the placement and routing constraints (Liu et al., 12 Feb 2026).

Because this is a large nonlinear IP, LAER-MoE splits it into two heuristics. The token dispatcher, Alg. LiteRouting, evenly splits $E$7 among replicas of expert $E$8 while preferring intra-node replicas to reduce cross-node traffic. The expert layout tuner, Alg. ExpertLayout, first decides replica counts expert_rep$E$9 by priority-queue, repeatedly assigning extra replicas to the expert with highest load/replicas until $C$0; then places replicas greedily by sorting all $C$1 in descending order and assigning each replica to the device with minimum current total load, balanced across nodes to respect topology; and finally evaluates a small set $C$2 of schemes such as proportional, even, and random perturbations by computing $C$3 via lite routing and picking the best. The overall greedy complexity is $C$4, and in practice with $C$5–$C$6 it runs in $C$7 ms per layer even at $C$8 (Liu et al., 12 Feb 2026).

The reported outcome is a near-perfect balancing effect. In the case study, LAER-MoE keeps $C$9 within $\Psi_{\text{expert}}$0 of perfect balance, whereas the baselines are $\Psi_{\text{expert}}$1–$\Psi_{\text{expert}}$2. This suggests that, within the planner’s objective, FSEP’s sharding is valuable less because it reduces asymptotic communication volume than because it makes expert placement a runtime decision rather than a static assignment.

5. Inference-oriented FSEP variants

For low-batch inference on multi-chiplet accelerators, "Expert Streaming" defines FSEP, or FSE-DP, as a fine-grained parallelization paradigm whose core ideas are to shard both token activations and expert weights across all chiplets, stream micro-slices of expert weights point-to-point over high-bandwidth D2D links, and dynamically schedule expert trajectories across chiplets to balance compute, on-chip storage, and communication. Let $\Psi_{\text{expert}}$3 be the number of chiplets, $\Psi_{\text{expert}}$4 the tokens-per-iteration aggregated across requests, $\Psi_{\text{expert}}$5 the experts per layer, and $\Psi_{\text{expert}}$6 the per-expert weight shape. Expert weights $\Psi_{\text{expert}}$7 are sliced into $\Psi_{\text{expert}}$8 equal expert-slices of size $\Psi_{\text{expert}}$9, and each expert-slice is further cut into $W_o$00 micro-slices of size

$W_o$01

At each timestep, each chiplet computes micro-slice $W_o$02 from expert $W_o$03 on its local token subset, simultaneously receives micro-slice $W_o$04 of $W_o$05 from the upstream chiplet via D2D and sends $W_o$06 downstream, and pre-loads the next micro-slice from DDR when buffer space is free. By choosing $W_o$07 so that $W_o$08, effective per-micro-slice latency is approximately $W_o$09. The dataflow is defined by five simple, hardware-embeddable rules governing receive-compute-forward, local fallback, buffer release at the final chiplet, opportunistic DDR fill, and optional prioritization of the chiplet with the largest available buffer (Ma et al., 29 Mar 2026).

The same work couples that dataflow to a dynamic expert trajectory scheduler. Experts are sorted by activation counts $W_o$10 and paired highest with lowest, so that a compute-bound hot expert overlaps with a communication-bound cold one. The scheduler maximizes $W_o$11 so no compute resources stay idle, balances hot and cold workloads by pairing, and can defer extreme cold experts through token buffering when a request has QoS slack. The paper states that FSEP achieves $W_o$12–$W_o$13 end-to-end latency speedup over EP or Hydra in low-batch $W_o$14–$W_o$15 tokens$W_o$16 MoE benchmarks and saves up to $W_o$17 percent on-chip memory (Ma et al., 29 Mar 2026).

MoEShard implements another inference-time FSEP for encoder-based MoE models in a multi-GPU setting. Each expert consists of two dense weight matrices, $W_o$18 and $W_o$19, and a classic expert invocation is

$W_o$20

With $W_o$21 GPUs, FSEP replaces each expert by $W_o$22 shards through a column-wise split of $W_o$23 and a row-wise split of $W_o$24:

$W_o$25

and

$W_o$26

Each GPU $W_o$27 loads exactly $W_o$28 for all experts. The paper states that every GPU performs exactly $W_o$29 of the expert’s total FLOPs, regardless of how many tokens that expert actually received. Communication remains a scatter/gather pattern, with per-layer per-GPU volume

$W_o$30

To reduce kernel-launch overhead, MoEShard uses expert-level fusion via MegaBlocks, and optionally stacks all experts into one larger block-sparse matrix so that all expert shards can be invoked in a single kernel launch (Balmau et al., 11 Mar 2025).

6. Empirical results, limitations, and recurrent misconceptions

On training workloads, LAER-MoE reports experiments on $W_o$31 A100 $W_o$32 GB with NVLink $W_o$33 GB/s$W_o$34 and InfiniBand $W_o$35 Gb/s$W_o$36 using Mixtral-8×7B, Mixtral-8×22B, and Qwen-8×7B in both e8k2 and e16k4 configurations on WikiText and C4. The reported end-to-end throughput is up to $W_o$37 speedup vs Megatron-LM, $W_o$38 vs FSDP+EP, and $W_o$39 vs FlexMoE. With auxiliary loss $W_o$40, LAER-MoE’s loss matches Megatron’s to $W_o$41 relative error, confirming no numerical drift. The case study reports that the All-to-All fraction drops from $W_o$42 to $W_o$43, a $W_o$44 reduction in dispatch/collect time, lite routing is $W_o$45 of iteration time, and disabling communication scheduling increases end-to-end time by $W_o$46 (Liu et al., 12 Feb 2026).

On inference workloads, the two FSEP variants emphasize different operating points. "Expert Streaming" is explicitly conditioned on multi-chiplet accelerators with high-bandwidth, low-latency D2D links, and states that it is not suited to monolithic accelerators lacking MIMD-style independent chiplet controls; micro-slice granularity must balance pipelining gain vs. control overhead, empirically $W_o$47–$W_o$48 micro-slices per expert slice; and token buffering trades per-request latency for throughput, so QoS parameters must be tuned to workload. MoEShard assumes $W_o$49 identical GPUs, all-to-all connected, such as NVLink; requires a high-bandwidth interconnect because of full token replication; assumes divisible dimensions, though padding or uneven splits can be used; and notes slight memory overhead for token buffers, though in practice negligible on modern GPUs (Ma et al., 29 Mar 2026, Balmau et al., 11 Mar 2025).

A recurrent misconception is that fully sharding experts necessarily implies prohibitive communication or token loss. The available results argue against both conclusions, though in different ways. LAER-MoE states that the communication ratio to FSDP AllGather tends to $W_o$50 as cluster size grows and that FSEP adds only $W_o$51 extra communication overhead while enabling flexible per-iteration re-layout (Liu et al., 12 Feb 2026). MoEShard states that it achieves perfect load balancing through tensor sharding of MoE experts, ensures full token retention, and does not rely on heuristic capacity factors or drop tokens; in its reported evaluation on Switch-Base encoder with T5 + 256 experts on BookCorpus using $W_o$52 NVIDIA A100 80 GB, per-layer forward latency falls from $W_o$53–$W_o$54 ms/layer in DeepSpeed to $W_o$55–$W_o$56 ms/layer in MoEShard, and TTFT speedup reaches $W_o$57 at $W_o$58 experts and remains above $W_o$59 up to $W_o$60 experts (Balmau et al., 11 Mar 2025).

These results suggest that the main significance of FSEP is not a single universal algorithmic improvement, but a reorganization of where MoE imbalance is handled. Static expert ownership is replaced by globally sharded expert state, and imbalance is then addressed through reconstruction, re-layout, micro-slice streaming, or fused shard-local computation, depending on the hardware and workload regime.