FaaSMoE: A Serverless Framework for Multi-Tenant Mixture-of-Experts Serving

Abstract: Mixture-of-Experts (MoE) models offer high capacity with efficient inference cost by activating a small subset of expert models per input. However, deploying MoE models requires all experts to reside in memory, creating a gap between the resource used by activated experts and the provisioned resources. This underutilization is further pronounced in multi-tenant scenarios. In this paper, we propose FaaSMoE, a multi-tenant MoE serving architecture built on Function-as-a-Service (FaaS) platforms. FaaSMoE decouples the control and execution planes of MoE by deploying experts as stateless FaaS functions, enabling on-demand and scale-to-zero expert invocation across tenants. FaaSMoE further supports configurable expert granularity within functions, trading off per-expert elasticity for reduced invocation overhead. We implement a prototype with an open-source edge-oriented FaaS platform and evaluate it using Qwen1.5-moe-2.7B under multi-tenant workloads. Compared to a full-model baseline, FaaSMoE uses less than one third of the resources, demonstrating a practical and resource-efficient path towards scalable MoE serving in a multi-tenant environment.

• The paper introduces FaaSMoE, a framework that decouples orchestration from expert execution for efficient, scalable multi-tenant Mixture-of-Experts serving.
• It leverages FaaS capabilities to enable elastic autoscaling and achieves notable reductions in CPU (up to 71%) and memory (up to 67%) usage.
• Configurable expert granularity allows balancing between invocation overhead and resource utilization across diverse deployment scenarios.

FaaSMoE: A Serverless Framework for Multi-Tenant Mixture-of-Experts Serving

Introduction and Motivation

Mixture-of-Experts (MoE) architectures provide high model capacity with efficient inference by activating only a sparse subset of experts per input. Despite this runtime efficiency, conventional MoE deployments are burdened with the requirement that all experts reside in memory, creating significant resource underutilization—especially pronounced in multi-tenant environments. Traditional strategies for MoE optimization focus primarily on computation and memory efficiency per model instance but fail to address the persistent gap between provisioned and activated resources across tenants. The paper introduces FaaSMoE (2604.26881), a serverless framework that leverages Function-as-a-Service (FaaS) paradigms to enable elastic, on-demand expert invocation and cross-tenant expert sharing, providing a scalable and resource-efficient solution for multi-tenant MoE serving.

Architectural Design and Mechanisms

FaaSMoE fundamentally decouples MoE inference into a lightweight control plane and an elastic compute plane. The control plane, termed the Orchestrator, contains non-expert logic (tokenization, attention, gating) and is responsible for micro-batching, request dispatching, and managing expert invocation. Expert computation is placed on the compute plane, where each expert (or configurable expert block) is mapped to a stateless FaaS function, thus leveraging autoscaling and scale-to-zero capabilities intrinsic to FaaS providers.

Figure 1: FaaSMoE architecture outlining the separation of orchestration from stateless expert execution using serverless FaaS functions.

The configuration of expert granularity within FaaS functions is a key tunable parameter, supporting a trade-off between invocation overhead and expert elasticity. By packaging multiple experts into a single function, invocation fan-out and container startup overhead are reduced at the expense of coarser scaling.

The orchestrator's placement flexibility—either per-tenant or shared—allows practitioners to optimize resource sharing and isolation based on workload characteristics. This design ensures correctness and compatibility with original MoE routing semantics regardless of deployment topology.

Deployment Strategies for Multi-Tenant Serving

The paper evaluates four deployment configurations:

• Baseline: Each tenant hosts a full MoE model, maximizing resource duplication.
• Local Distribution: Experts are offloaded to a centralized server, reducing memory footprint but lacking elasticity.
• FaaSMoE Private: Per-tenant orchestrators with serverless expert blocks.
• FaaSMoE Shared: Centralized orchestrator utilizing serverless expert blocks, enabling micro-batching across tenants.

  Figure 2: Comparative illustrations of deployment strategies, delineating resource sharing and orchestration granularity.

These strategies illustrate the spectrum from static, resource-heavy duplication to agile, elastic sharing using serverless primitives.

Empirical Evaluation and Resource Efficiency

Using the Qwen1.5-moe-2.7B model and the tinyFaaS platform, the authors conduct multi-tenant benchmarks with heterogeneous task loads. The results demonstrate significant resource savings for the FaaSMoE variants:

• CPU usage: Baseline consumes up to 1126.84%, while FaaSMoE-Shared is reduced to 326.40%; FaaSMoE-Private to 408.49%.
• Memory usage: Baseline peaks at 217.52 GB, reduced to 72.25 GB (FaaSMoE-Shared) and 90.98 GB (FaaSMoE-Private).

These improvements are underpinned by optimal expert pooling and cross-tenant sharing. FaaS framework overhead is shown to be minor compared to expert computation.

Figure 3: Average CPU and memory utilization for various deployment scenarios with expert block size 20, indicating resource efficiency gains.

Breakdown analyses reveal that resource consumption is dominated by expert execution; platform and gateway overhead remain modest.

Figure 4: FaaS consumption breakdown showing dominance of expert (worker) execution relative to platform overhead.

Block size experiments indicate a non-monotonic relationship between expert granularity and resource consumption, with optimal memory usage at intermediate block sizes (e.g., 20 experts/block). Overly fine or coarse granularity can introduce undesired overhead or memory bloat.

Figure 5: Effects of varying expert block sizes on CPU and memory utilization, highlighting a trade-off between invocation overhead and resident memory footprint.

Discussion, Limitations, and Future Directions

The FaaSMoE architecture generalizes well across MoE variants, as the underlying patterns of expert sparsity and stateless execution are universal. While evaluated using a CPU-only setup (reflecting the constraints of mainstream FaaS providers), GPU-backed serverless offerings remain limited; however, most experts are small, making CPU-allocated FaaS suitable.

Expert granularity within FaaS functions emerges as an important architectural design parameter, enabling practitioners to tune elasticity and overhead according to workload and cost constraints. Orchestrator placement (centralized vs distributed) introduces a trade-off between batching efficiency and tenant isolation, suggesting that adaptive or hybrid schemes may provide further optimization.

Latency and network overhead, though not the primary focus, are acknowledged as limitations due to remote expert invocation. Integrating low-latency communication mechanisms or co-locating orchestration logic may further enhance efficiency.

Conclusion

FaaSMoE presents a modular, serverless solution to the resource inefficiency endemic in multi-tenant MoE model serving. By decoupling expert computation into stateless FaaS functions and leveraging autoscaling, FaaSMoE achieves substantial reductions in CPU and memory usage compared to conventional deployments. The framework's architectural flexibility provides practical deployment options suitable for diverse workloads and advances the feasibility of multi-tenant, resource-efficient MoE inference. Responsible expert sharing and elastic provisioning fundamentally address the persistent resource gap associated with MoE architectures, suggesting future directions for optimized distributed AI serving infrastructures.