FinDEP: Fine-Grained MoE Inference Scheduling

• FinDEP is a fine-grained scheduling algorithm for DEP that maximizes MoE inference throughput by partitioning tasks into micro-batches and slices.
• It mitigates memory and communication bottlenecks by optimizing pipeline granularities and task ordering under strict GPU resource constraints.
• The solver employs nested search and convex optimization techniques to dynamically adapt configurations, leading to significant performance gains.

FinDEP is a fine-grained scheduling algorithm for Disaggregated Expert Parallelism (DEP) that aims to maximize inference throughput in large-scale Mixture-of-Experts (MoE) models. It introduces an optimized task partitioning and scheduling framework to address memory and communication bottlenecks inherent in standard DEP architectures, supporting scenarios with shared experts and highly variable workloads typical of state-of-the-art MoE deployments (Pan et al., 25 Dec 2025).

1. Background and Problem Context

MoE architectures enable scaling model parameters to hundreds of billions with sublinear computational cost by activating only a small subset (top_k) of experts per token. Standard expert parallelism divides expert blocks among multiple GPUs. DEP advances this by grouping all attention and any shared experts into an "Attention Group" (AG) and non-shared experts into an "Expert Group" (EG), each allocated to separate GPU pools.

Naïve DEP requires AG to fully finish computation before transferring (A2E) to EG, and EG must complete computation before returning results (E2A), resulting in significant AG/EG idle times due to lock-step execution. Previous work, such as PPPipe, pipelines computation at the micro-batch level, only partially overlapping compute and communication phases but failing to handle shared experts or fine-grained pipelining. Additionally, prior scheduling and pipeline choices relied on hand-tuning and exhaustive search, which is infeasible at large system scales or when deployment demands rapid adaptation.

2. Fine-Grained Task Partitioning in FinDEP

FinDEP partitions MoE inference across multiple axes. Each model layer is composed of an attention block, optional shared-expert block, and a sparse expert block. FinDEP introduces two configurable pipeline granularities:

• $r_1$: divides AG and shared-expert computation into $r_1$ micro-batches.
• $r_2$: splits tokens routed to EG further into $r_2$ slices per micro-batch.

For each layer $t$:

• Attention subtasks $A(t,i)$ and shared-expert subtasks $S(t,i)$, $i \in [1, r_1]$ process workload slices in AG.
• Following each slice, tokens are grouped and sent as $C_{A2E}(t,i,j)$ to EG, $j \in [1, r_2]$.
• EG handles $r_1$0 (expert computation) and returns results via $r_1$1.

Token conservation is ensured via $r_1$2.

3. Scheduling Optimization and Constraint Formulation

FinDEP formulates inference throughput as an explicit scheduling optimization problem, seeking to maximize tokens-per-second by tuning (a) $r_1$3, the AG pipeline degree; (b) $r_1$4, tokens per AG subtask; (c) $r_1$5, the EG pipeline degree; (d) $r_1$6, tokens per EG subtask; (e) task order (AASS vs ASAS interleaving of AG attention and shared experts); and (f) individual subtask start times $r_1$7.

Each subtask type is modeled with linear performance models: for instance,

$r_1$8

with similar forms for other subtask timings.

Precedence and resource constraints prevent overlapping compute on the same GPUs or communications on the same links, and maintain data dependencies between pipeline stages. The scheduling objective is: $r_1$9 where $r_2$0 denotes tokens served per multi-batch, and $r_2$1 is the maximum of the final finish times across all critical subtasks.

This scheduling problem is an NP-hard job-shop variant. However, exploiting problem structure—monotonicity in $r_2$2 and $r_2$3 and convexity in $r_2$4—permits practical solution strategies.

4. Efficient Solver Algorithm

FinDEP’s solver employs nested loops over candidate micro-batch sizes ($r_2$5) and pipeline degrees ($r_2$6), constrained by GPU memory limits. The algorithm focus is as follows:

1. Sweep $r_2$7 downward, starting from the largest value fitting into AG memory.
2. For each $r_2$8, determine the maximum feasible $r_2$9 given AG+EG memory. By monotonicity, only Pareto-optimal $r_2$0 need consideration.
3. For each valid $r_2$1 and both AG orderings, solve for $r_2$2 to minimize makespan (via a convex 1-D search), then set $r_2$3 accordingly.
4. Retain the configuration yielding maximal throughput.

The overall complexity is $r_2$4, with $r_2$5 a constant and $r_2$6 the largest candidate $r_2$7. Empirically, solution time is below one second, allowing online adaptation to changing workload characteristics such as sequence length or expert fan-out.

5. System Integration and Practical Implementation

FinDEP is integrated into a PyTorch pipeline. AG-side computations (attention and shared experts) are executed via FlashInfer (v0.3.0); EG-side expert matrices are handled with GEMM kernels. Inter-group communication leverages NCCL-based point-to-point collectives on NVLink/PCIe.

An offline calibration phase micro-benchmarks all elementary operation types (attention, GEMM, communication) to estimate $r_2$8 performance parameters for each. These are fed into the online solver. For each new inference request (characterized at least by sequence length $r_2$9), the scheduler computes optimal pipeline configuration in under one second and sets system parameters accordingly.

FinDEP accommodates dynamic sequence lengths and expert fan-out without user hand-tuning. The token-conservation property ensures that split and assignment of tokens remains consistent across pipeline staging and system scale.

6. Empirical Performance and Evaluation

FinDEP was evaluated on several GPU testbeds, including single-node (8x NVIDIA RTX A6000, NVIDIA A10, NVIDIA H20) and multi-node (4x8 NVIDIA H20, 32 GPUs) clusters. Models tested include DeepSeek-V2 (236B parameters, with shared experts) and Qwen3-MoE (235B, without shared experts). Throughput (tokens/s) was reported as the average of three runs for sequence lengths $t$0.

Representative results:

Backbone	PPPipe	FinDEP	Speedup
DeepSeek, S=4096 (A)	44.21	51.47	1.16×
Qwen, S=8192 (B)	15.98	25.71	1.61×
DeepSeek, S=4096 (D)	120.83	132.07	1.10×
Qwen, S=4096 (D)	61.59	76.53	1.24×

FinDEP achieved up to 1.61× throughput gains over PPPipe, with 1.24× speedup persisting at 32-GPU scale and longer sequences. The fraction of non-overlapped communication during inference was reduced by up to 2× compared to PPPipe.

7. Limitations and Prospective Extensions

FinDEP’s pipeline granularity is bounded by launch overheads: excessive $t$1 or $t$2 can result in dominant fixed costs ($t$3), mitigated by the solver’s balancing. Communication bottlenecks persist at extreme cluster scale; optimization of network-topology mappings is a potential avenue. The current framework assumes homogeneous clusters and single-tenant workload—generalizing to heterogeneous or multi-tenant systems remains future work.

Dynamic load imbalance across experts due to non-uniform gating is not directly addressed; adaptive expert replication or work-stealing mechanisms represent potential future integrations. FinDEP does not optimize for variable resource availability or failures in the underlying cluster.

In summary, FinDEP advances DEP scheduling by introducing fine-grained task partitioning, a principled throughput optimization, and a low-latency solver, demonstrably increasing MoE inference efficiency across diverse hardware and model scenarios (Pan et al., 25 Dec 2025).