---
title: Fully Sharded Expert Parallelism
url: https://www.emergentmind.com/topics/fully-sharded-expert-parallel-fsep
type: topic
---

# Fully Sharded Expert Parallelism

Fully Sharded Expert Parallel (FSEP) denotes a set of Mixture-of-Experts (MoE) parallelization schemes in which expert parameters—and, in some formulations, token activations—are sharded across all participating devices or chiplets rather than bound monolithically to one device. In LAER-MoE, FSEP is a training paradigm that fully partitions each expert parameter by the number of devices and restores partial experts at expert granularity through All-to-All communication during training. In "Expert Streaming," FSEP, also called Fully Sharded Expert Data-Parallelism (FSE-DP), is a fine-grained parallelization paradigm for low-batch MoE inference on multi-chiplet accelerators. In MoEShard, FSEP is an inference strategy that decomposes each expert by a column-wise split of $W_i$ and a row-wise split of $W_o$ so that every GPU performs exactly $\tfrac1P$ of the expert’s total FLOPs, regardless of routing skew [2602.11686][2603.27624][2503.08467].

## 1. Terminological scope and shared abstraction

Recent usage does not restrict FSEP to a single execution pattern. The term appears in distributed training, multi-chiplet low-batch inference, and multi-GPU inference, but the recurring structural idea is global expert sharding combined with runtime reconstruction, streaming, or aggregation.

| Work | Setting | Defining mechanism |
|---|---|---|
| LAER-MoE | MoE training | Fully partitions each expert parameter by the number of devices and restores partial experts at expert granularity through All-to-All communication |
| Expert Streaming | Low-batch MoE inference on multi-chiplet accelerators | Shards both token activations and expert weights across all chiplets and streams micro-slices over D2D links |
| MoEShard | Multi-GPU encoder-based MoE inference | Column-wise split of $W_i$ and row-wise split of $W_o$ across $P$ GPUs |

Taken together, these formulations suggest that FSEP is best understood as a sharding principle rather than a single fixed runtime. What remains constant is that expert state is globally partitioned; what varies is whether the system reconstructs full experts on demand, streams micro-slices along trajectories, or aggregates partial outputs after shard-local computation [2602.11686][2603.27624][2503.08467].

## 2. Training-oriented FSEP in LAER-MoE

In LAER-MoE, the global symbols are $N$ for the number of devices, $E$ for the total number of experts, $C$ for expert-capacity per device, and $\Psi_{\text{expert}}$ for the size of one expert. Every expert’s parameter tensor is fully sharded across all $N$ devices, instead of binding each expert to a single device. Let
$$
E_j \in \mathbb{R}^{\Psi_{\text{expert}}}
$$
be the flattened parameter vector of expert $j$. FSEP splits $E_j$ into $N$ disjoint, equal-sized shards,
$$
E_j = [E_j^{(0)}, E_j^{(1)}, \ldots, E_j^{(N-1)}],
$$
where each shard $E_j^{(d)} \in \mathbb{R}^{\Psi_{\text{expert}}/N}$ is stored on device $d$ after the initial shard step. During a forward pass, each device $i$ identifies a set of $C$ experts it must invoke via the gating or routing mechanism; it then issues an All-to-All unshard so that, for each needed expert $j$, device $i$ collects the $N$ shards $\{E_j^{(d)}\}_{d=0\ldots N-1}$ and locally reassembles
$$
E_j = \operatorname{concat}_d(E_j^{(d)}).
$$
Backward proceeds similarly: after computing gradients $\nabla E_j$, each device holds full $\nabla E_j$ for its $C$ experts; a second All-to-All reshard splits these gradients back into $N$ shards and scatter-reduces them so that each device $d$ adds $\nabla E_j^{(d)}$ to its local shard’s gradient [2602.11686].

The architectural consequence is explicit in the paper: FSEP exposes the same memory benefits as FSDP, because each device only permanently stores $\tfrac1N$ of every expert, while also enabling dynamic per-iteration expert-placement, since any device can choose any $C$ experts to reconstruct. This dynamic placement is the basis for LAER-MoE’s load-adaptive expert re-layout during training.

## 3. Communication model, overlap conditions, and fine-grained scheduling

In LAER-MoE, each unshard and each reshard is a balanced $N$-way All-to-All among the $N$ devices. Per device, if it restores $C$ full experts, it must send out $C$ shards and receive $C$ shards on each All-to-All, giving
$$
V_{\text{fsep}} = C \cdot (N-1) \cdot (\Psi_{\text{expert}}/N)
$$
bytes per device per All-to-All. By contrast, in a traditional FSDP + EP split where EP group size is $P_{\text{ep}}$ and FSDP group size is $P_{\text{fsdp}}$ with $P_{\text{ep}} \cdot P_{\text{fsdp}} = N$ and $C \cdot P_{\text{ep}} = E$, an FSDP AllGather on $C$ experts costs
$$
V_{\text{fsdp}} = C \cdot \Psi_{\text{expert}} \cdot (P_{\text{fsdp}}-1)/P_{\text{fsdp}}.
$$
The ratio tends to $1$ as cluster size grows. The paper further states that these models imply that FSEP adds only $O(1\%)$ extra communication overhead but gains full re-layout flexibility [2602.11686].

Communication hiding is integral rather than incidental. FSEP interleaves or shadow-launches the All-to-All on a dedicated CUDA stream and overlaps forward unshard for layer $L+1$ during expert compute of layer $L$, while backward delays reshard of layer $L$ until the next MoE layer’s backward compute. Under a bfloat16 SwiGLU MLP with hidden size $H$, intermediate $H'$, top-$k=K$, and token count $S$ per device, the compute–comm overlap condition is
$$
S > \frac{C \cdot V_{\text{comp}}}{K \cdot V_{\text{comm}}}.
$$
In practice on A100s and typical $S \approx 16K \cdot K$, this holds comfortably, letting the communication be “free.” The fine-grained schedule uses four CUDA streams: Stream S0 for Attention$(L)$ compute, Stream S1 for MLP$(L)$ expert compute and concurrently prefetch unshard$(L+1)$, Stream S2 for delayed gradient reshard$(L)$ scheduled at the start of backward$(L+1)$ compute, and Stream S3 for token-dispatcher All-to-All$(L)$. The stated effect is reduced idle gaps and channel contention compared to a single coarse communication phase [2602.11686].

## 4. Expert re-layout planning and load balancing

The principal benefit claimed for training-time FSEP is per-iteration freedom to choose exactly which $C$ experts each device reconstructs. LAER-MoE introduces binary placement variables $A_{i,j} \in \{0,1\}$ indicating that device $i$ restores expert $j$, with $\sum_i A_{i,j} = C$ for all $j$, and routing variables $S_{i,j,k} \in \mathbb{N}$ giving the number of tokens originally on device $i$ routed to expert $j$ but sent to device $k$, with $\sum_k S_{i,j,k} \cdot A_{k,j} = R_{i,j}$. Total iteration time is modeled as $T = T^{\text{comm}} + T^{\text{comp}}$, where
$$
T^{\text{comm}} = 4 \cdot V_{\text{comm}} \cdot \sum_{i,j,k}\left(\frac{S_{i,j,k}}{\operatorname{bw}(i,k)}\right)
$$
and
$$
T^{\text{comp}} = (3 + F_{\text{ckpt}})\cdot \max_i T_i^{\text{fw\_comp}}, \qquad
T_i^{\text{fw\_comp}} = V_{\text{comp}} \cdot \sum_{j,k}\left(\frac{S_{k,j,i}}{B_{\text{comp}}}\right).
$$
The joint integer program minimizes $T$ subject to the placement and routing constraints [2602.11686].

Because this is a large nonlinear IP, LAER-MoE splits it into two heuristics. The token dispatcher, Alg. LiteRouting, evenly splits $R_{i,j}$ among replicas of expert $j$ while preferring intra-node replicas to reduce cross-node traffic. The expert layout tuner, Alg. ExpertLayout, first decides replica counts expert\_rep$[j]$ by priority-queue, repeatedly assigning extra replicas to the expert with highest load/replicas until $\sum_j \text{expert\_rep}[j] = N \cdot C$; then places replicas greedily by sorting all $(\text{expert}, \text{load\_per\_replica})$ in descending order and assigning each replica to the device with minimum current total load, balanced across nodes to respect topology; and finally evaluates a small set $\varepsilon$ of schemes such as proportional, even, and random perturbations by computing $T$ via lite routing and picking the best. The overall greedy complexity is $O(|\varepsilon| \cdot N^2 \cdot C)$, and in practice with $|\varepsilon| = 2$–$3$ it runs in $<1$ ms per layer even at $N = 1024$ [2602.11686].

The reported outcome is a near-perfect balancing effect. In the case study, LAER-MoE keeps $\max_i \sum_{j,k} S_{k,j,i}$ within $1.01\times$ of perfect balance, whereas the baselines are $1.3$–$1.5\times$. This suggests that, within the planner’s objective, FSEP’s sharding is valuable less because it reduces asymptotic communication volume than because it makes expert placement a runtime decision rather than a static assignment.

## 5. Inference-oriented FSEP variants

For low-batch inference on multi-chiplet accelerators, "Expert Streaming" defines FSEP, or FSE-DP, as a fine-grained parallelization paradigm whose core ideas are to shard both token activations and expert weights across all chiplets, stream micro-slices of expert weights point-to-point over high-bandwidth D2D links, and dynamically schedule expert trajectories across chiplets to balance compute, on-chip storage, and communication. Let $M$ be the number of chiplets, $B$ the tokens-per-iteration aggregated across requests, $E$ the experts per layer, and $D_{\text{in}} \times D_{\text{out}}$ the per-expert weight shape. Expert weights $W_e$ are sliced into $M$ equal expert-slices of size $(D_{\text{out}}/M)\times D_{\text{in}}$, and each expert-slice is further cut into $R$ micro-slices of size
$$
S_\mu = \frac{(D_{\text{out}}/M)\times D_{\text{in}}}{R}.
$$
At each timestep, each chiplet computes micro-slice $\mu^{t-1}$ from expert $e$ on its local token subset, simultaneously receives micro-slice $\mu^t$ of $e$ from the upstream chiplet via D2D and sends $\mu^{t-1}$ downstream, and pre-loads the next micro-slice from DDR when buffer space is free. By choosing $R$ so that $T_{\text{comp}}(\mu) \approx T_{\text{D2D}}(\mu)$, effective per-micro-slice latency is approximately $\max(T_{\text{comp}}(\mu), T_{\text{comm}}(\mu))$. The dataflow is defined by five simple, hardware-embeddable rules governing receive-compute-forward, local fallback, buffer release at the final chiplet, opportunistic DDR fill, and optional prioritization of the chiplet with the largest available buffer [2603.27624].

The same work couples that dataflow to a dynamic expert trajectory scheduler. Experts are sorted by activation counts $B_e$ and paired highest with lowest, so that a compute-bound hot expert overlaps with a communication-bound cold one. The scheduler maximizes $C_{\text{idle}} \to \varnothing$ so no compute resources stay idle, balances hot and cold workloads by pairing, and can defer extreme cold experts through token buffering when a request has QoS slack. The paper states that FSEP achieves $1.22\times$–$2.00\times$ end-to-end latency speedup over EP or Hydra in low-batch $(16$–$1024$ tokens$)$ MoE benchmarks and saves up to $78.8$ percent on-chip memory [2603.27624].

MoEShard implements another inference-time FSEP for encoder-based MoE models in a multi-GPU setting. Each expert consists of two dense weight matrices, $W_i \in \mathbb{R}^{h \times k}$ and $W_o \in \mathbb{R}^{k \times h}$, and a classic expert invocation is
$$
y = x\,W_i\,W_o,\qquad x \in \mathbb{R}^{T \times h},\; y \in \mathbb{R}^{T \times h}.
$$
With $P$ GPUs, FSEP replaces each expert by $P$ shards through a column-wise split of $W_i$ and a row-wise split of $W_o$:
$$
W_i = [W_i^{(1)} \mid W_i^{(2)} \mid \cdots \mid W_i^{(P)}], \qquad W_i^{(p)} \in \mathbb{R}^{h \times (k/P)},
$$
and
$$
W_o =
\begin{bmatrix}
W_o^{(1)} \\
W_o^{(2)} \\
\vdots \\
W_o^{(P)}
\end{bmatrix},
\qquad
W_o^{(p)} \in \mathbb{R}^{(k/P)\times h}.
$$
Each GPU $p$ loads exactly $\{W_i^{(p)}, W_o^{(p)}\}$ for all experts. The paper states that every GPU performs exactly $\tfrac1P$ of the expert’s total FLOPs, regardless of how many tokens that expert actually received. Communication remains a scatter/gather pattern, with per-layer per-GPU volume
$$
V_{\text{total}}^{\text{FSEP}} = 4\,b\,s\,h\Bigl[(P-1) + (P-1)/P\Bigr].
$$
To reduce kernel-launch overhead, MoEShard uses expert-level fusion via MegaBlocks, and optionally stacks all experts into one larger block-sparse matrix so that all expert shards can be invoked in a single kernel launch [2503.08467].

## 6. Empirical results, limitations, and recurrent misconceptions

On training workloads, LAER-MoE reports experiments on $4 \times 8$ A100 $80$ GB with NVLink $(300$ GB/s$)$ and InfiniBand $(800$ Gb/s$)$ using Mixtral-8×7B, Mixtral-8×22B, and Qwen-8×7B in both e8k2 and e16k4 configurations on WikiText and C4. The reported end-to-end throughput is up to $1.69\times$ speedup vs Megatron-LM, $1.50\times$ vs FSDP+EP, and $1.39\times$ vs FlexMoE. With auxiliary loss $=1e{-4}$, LAER-MoE’s loss matches Megatron’s to $<1e{-3}$ relative error, confirming no numerical drift. The case study reports that the All-to-All fraction drops from $\approx 40\%$ to $<20\%$, a $2.68\times$ reduction in dispatch/collect time, lite routing is $<0.1\%$ of iteration time, and disabling communication scheduling increases end-to-end time by $\approx 8\%$ [2602.11686].

On inference workloads, the two FSEP variants emphasize different operating points. "Expert Streaming" is explicitly conditioned on multi-chiplet accelerators with high-bandwidth, low-latency D2D links, and states that it is not suited to monolithic accelerators lacking MIMD-style independent chiplet controls; micro-slice granularity must balance pipelining gain vs. control overhead, empirically $5$–$10$ micro-slices per expert slice; and token buffering trades per-request latency for throughput, so QoS parameters must be tuned to workload. MoEShard assumes $P$ identical GPUs, all-to-all connected, such as NVLink; requires a high-bandwidth interconnect because of full token replication; assumes divisible dimensions, though padding or uneven splits can be used; and notes slight memory overhead for token buffers, though in practice negligible on modern GPUs [2603.27624][2503.08467].

A recurrent misconception is that fully sharding experts necessarily implies prohibitive communication or token loss. The available results argue against both conclusions, though in different ways. LAER-MoE states that the communication ratio to FSDP AllGather tends to $1$ as cluster size grows and that FSEP adds only $O(1\%)$ extra communication overhead while enabling flexible per-iteration re-layout [2602.11686]. MoEShard states that it achieves perfect load balancing through tensor sharding of MoE experts, ensures full token retention, and does not rely on heuristic capacity factors or drop tokens; in its reported evaluation on Switch-Base encoder with T5 + 256 experts on BookCorpus using $4 \times$ NVIDIA A100 80 GB, per-layer forward latency falls from $177$–$180$ ms/layer in DeepSpeed to $41.5$–$43.5$ ms/layer in MoEShard, and TTFT speedup reaches $6.45\times$ at $128$ experts and remains above $2.39\times$ up to $256$ experts [2503.08467].

These results suggest that the main significance of FSEP is not a single universal algorithmic improvement, but a reorganization of where MoE imbalance is handled. Static expert ownership is replaced by globally sharded expert state, and imbalance is then addressed through reconstruction, re-layout, micro-slice streaming, or fused shard-local computation, depending on the hardware and workload regime.

Source: https://www.emergentmind.com/topics/fully-sharded-expert-parallel-fsep