ExpertWeave: Efficient ESFT Adapter Serving

• ExpertWeave is a serving system for ESFT adapters over a shared MoE base model, integrating task-specific expert fine-tuning with a unified deployment strategy.
• It employs a virtual-memory-assisted expert weight manager and a fused kernel for batched rerouting, ensuring seamless integration with existing MoE pipelines.
• Quantitative evaluations indicate up to 18% throughput gains and a 94x KV cache increase, all while maintaining minimal latency overhead and resource efficiency.

Searching arXiv for the cited ExpertWeave paper and closely related MoE/adapter serving work. ExpertWeave is a serving system for Expert-Specialized Fine-Tuning (ESFT) adapters on Mixture-of-Experts (MoE) LLMs. ESFT adapts an MoE model by selectively tuning the top-activated experts for a task, but serving such fine-tuned models at scale is difficult because merged deployment is prohibitively resource-hungry and existing multi-adapter serving systems with LoRA-style additive updates are incompatible with ESFT’s expert-oriented paradigm. ExpertWeave addresses this by serving multiple ESFT adapters concurrently over a single shared MoE base model, with two core mechanisms: a virtual-memory-assisted expert weight manager and a fused kernel for batched rerouting. The design aims to integrate into existing MoE inference pipelines with non-intrusive modifications and minimal latency overhead while preserving model accuracy (Shi et al., 25 Aug 2025).

1. Definition and system positioning

ExpertWeave is presented as a multi-tenant deployment system for ESFT-fine-tuned adapters over a single shared MoE base model. In the ESFT paradigm, only a small subset of “top-activated” experts are updated per task. ExpertWeave makes it practical to serve many such adapters without replicating the entire model for each (Shi et al., 25 Aug 2025).

Its positioning is defined in contrast to two alternatives stated explicitly in the source description. First, “merge-and-serve” deployment of separately merged models is resource-intensive. Second, existing multi-adapter serving systems built around LoRA-style additive updates do not match the expert-oriented structure of ESFT. ExpertWeave therefore uses a unified weight tensor for base and adapter experts, avoids memory fragmentation via virtual memory, and introduces only a lightweight rerouting step at runtime (Shi et al., 25 Aug 2025).

At a high level, the system consists of a single MoE base model checkpoint with $M$ experts per layer residing on the accelerator, plus $K$ ESFT adapters stored off-device and loaded on demand. Each adapter fine-tunes a task-specific subset of experts $A_k \subseteq \{1 \ldots M\}$. The remainder of the MoE inference pipeline—router, dispatch, and Grouped MatMul (GMM)—remains unchanged. This architectural choice is significant because it localizes the serving innovation to expert storage and expert ID redirection rather than requiring wholesale changes to sparse inference execution (Shi et al., 25 Aug 2025).

A plausible implication is that ExpertWeave is best understood not as a new MoE model family, but as a systems layer for deploying ESFT artifacts efficiently. Its contribution lies in reconciling expert-specialized fine-tuning with practical multi-adapter inference.

2. Architectural organization and execution model

The system is organized around two core modules. The first is the Virtual-memory-assisted Expert Weight Manager, which co-locates base and adapter experts in one logical tensor. The second is the Fused Kernel for Batched Rerouting, which redirects tokens from base-model expert IDs to adapter-expert IDs during execution (Shi et al., 25 Aug 2025).

The architecture proceeds as follows. A single MoE base model remains resident on the accelerator. ESFT adapters are stored off-device and loaded when needed. Once loaded, adapter experts are integrated into a common logical weight space. After the router emits TopK expert IDs for each token, a rerouting operator maps those IDs to task-specific fine-tuned experts when an adapter has provided such replacements. The downstream dispatch and GMM operate over the resulting virtual tensor without modification (Shi et al., 25 Aug 2025).

This organization is important because it preserves the standard sparse execution path of MoE inference. The router still determines the top experts, token dispatch still groups inputs by expert, and grouped matrix multiplication still executes over expert weights. The only additional dynamic step is the translation from base expert IDs to adapter expert IDs. This suggests that ExpertWeave’s compatibility claim derives from its narrow insertion point in the inference stack rather than from new routing semantics.

The description also characterizes the modifications as minimal: reserve and map pages once at adapter-load time, insert a single rerouting operator immediately after the router and before dispatch and GMM, and then execute GMM unchanged on the virtual weight tensor (Shi et al., 25 Aug 2025). That narrow integration surface is central to the system’s stated low-overhead behavior.

3. Virtual-memory-assisted expert weight management

A central problem in multi-adapter ESFT serving is the memory layout of expert weights. The source description first states a naïve memory decomposition:

$$\mathrm{Memory}_{\mathrm{base}} = \sum_{e=1}^{M} \mathrm{size}(W_e^{\mathrm{base}})$$

and

$$\mathrm{Memory}_{\mathrm{adapter,naive}} = \sum_{k=1}^{K}\sum_{e\in A_k}\mathrm{size}(\Delta W_e^k).$$

If each adapter is padded to a worst-case per-layer expert count $E_{\max}$, then the allocated padding cost is

$$\mathrm{Memory}_{\mathrm{padding}} = \sum_{k=1}^{K} E_{\max}\cdot \mathrm{size}(\mathrm{expert}),$$

which can be much larger than $\mathrm{Memory}_{\mathrm{adapter,naive}}$ and leads to severe fragmentation (Shi et al., 25 Aug 2025).

ExpertWeave addresses this by reserving one contiguous virtual tensor of shape

$$[M + K \cdot E_{\max}, H_{\mathrm{out}}, H_{\mathrm{in}}],$$

while physically mapping pages only for the base experts and the actually present adapter experts, namely the $M$ base experts plus $K$0 adapter experts. In this arrangement, holes in virtual address space consume no device memory. The optimized total memory is therefore

$K$1

which the source describes as incurring no padding overhead (Shi et al., 25 Aug 2025).

The design further addresses sub-page wastage. Because expert sizes often do not align with whole memory pages, ExpertWeave tracks expert-to-page mappings and allows unused bytes in a page to be reused by adjacent experts via reference counting, so all requested pages are nearly fully utilized. The implementation details given are specific to AscendCL, using aclrtReserveMemAddress for virtual reservation, aclrtMallocPhysical and aclrtFreePhysical for physical allocation and release, and aclrtMapMem and aclrtUnmapMem for address mapping (Shi et al., 25 Aug 2025).

This memory-management scheme is one of the defining technical ideas of the system. The source makes the strong claim that fragmentation is zero at the physical-page level. Interpreted conservatively, this means that padding-induced waste in physical device memory is eliminated even though the logical tensor retains empty virtual regions.

4. Batched rerouting and runtime redirection

The second core mechanism is rerouting after the MoE router has produced TopK expert IDs. The problem is stated explicitly: after the router emits TopK expert IDs $K$2 for each token $K$3, the system must redirect $K$4 if adapter $K$5 contains a fine-tuned version of expert $K$6 (Shi et al., 25 Aug 2025).

The data structure used for this redirection is the ESFT expert map $K$7, defined layerwise by

$K$8

In addition, the runtime maintains a per-token AdapterID array $K$9, where $A_k \subseteq \{1 \ldots M\}$0 denotes the base model (Shi et al., 25 Aug 2025).

The described kernel logic is straightforward. For each token index $A_k \subseteq \{1 \ldots M\}$1 in the batch and each TopK slot $A_k \subseteq \{1 \ldots M\}$2, the kernel reads old_id = TopK[t,s], then reads the adapter identity i = AID[t]; if i >= 0, it replaces the expert ID with Π[i, old_id], otherwise it leaves the ID unchanged. The updated ID is written back to TopK[t,s] (Shi et al., 25 Aug 2025).

Implementation-wise, the system fuses broadcast of AID, index arithmetic, and gather into one custom Ascend NPU kernel. The overhead comparison given is specific: the fused implementation incurs less than 1% overhead versus a hand-written sequence of PyTorch operations that causes roughly 29% slowdown (Shi et al., 25 Aug 2025).

The significance of this fused rerouting stage is twofold. First, it keeps the semantics local: rerouting changes only expert identifiers, not router outputs or dispatch structure. Second, it avoids decomposing the operation into multiple tensor kernels whose cumulative overhead would undermine the system’s low-latency objective. This suggests that the serving efficiency of ExpertWeave depends not only on memory savings but also on preserving the tight kernel-level execution profile of standard MoE inference.

5. Quantitative evaluation

The evaluation reported for ExpertWeave emphasizes memory usage, KV cache capacity, throughput, scalability, and overhead. The paper states that ExpertWeave can simultaneously serve multiple adapters of a 16B MoE model on a single accelerator where the baseline runs out of memory, or provide up to 94x more KV cache capacity and achieve up to 18% higher throughput while using comparable resources, all without compromising model accuracy (Shi et al., 25 Aug 2025).

A condensed memory and KV cache comparison is given for a 64 GB NPU when serving up to three adapters:

#Adapters	Configuration	Result
1	vLLM-Ascend (merged) / ExpertWeave-Padding / ExpertWeave (virtual)	29 GB / 33.7 GB / 31.8 GB; KV cache capacity 810 K tokens
2	vLLM-Ascend (merged) / ExpertWeave-Padding / ExpertWeave (virtual)	58.6 GB / 63.3 GB / 60.1 GB; KV cache capacity 572 K tokens
3	vLLM-Ascend (merged) / ExpertWeave-Padding / ExpertWeave (virtual)	OOM / 89.1 GB (OOM) / 68.5 GB; KV cache capacity 477 K tokens

The same evaluation summary states that ExpertWeave saves 28–40% of the padding overhead versus a simple padding scheme and supports three adapters where merged or padded systems run out of memory. The 94x KV cache increase is reported relative to classic merging (Shi et al., 25 Aug 2025).

Latency and throughput are also quantified. Time-to-First-Token (TTFT) increases by 4–11% as the number of adapters grows from 5 to 20, and Time-Per-Output-Token (TPOT) shows a similar 4–11% increase. Prefill throughput drops by less than 2%. Under uniform load, TTFT overhead is approximately 8% for 5 adapters and approximately 11% for 20. Under skewed ESFT workloads, ExpertWeave achieves 7–14% higher prefill throughput and 14–18% higher decode throughput than two separately served merged models, despite having fewer NPUs per model. In scale-out settings, multi-NPU skewed workloads can produce up to 18% throughput gain versus multiple merged models due to better device utilization (Shi et al., 25 Aug 2025).

The evaluation further states that ExpertWeave maintains low overhead even when scaling to 20 adapters, with a 4–11% latency increase compared with serving the base model alone, and that model accuracy is identical to serving each merged model standalone (Shi et al., 25 Aug 2025).

These results collectively frame the system’s contribution as a resource-efficiency improvement rather than an accuracy tradeoff. The paper’s own emphasis is that memory savings translate directly into higher KV cache capacity and improved utilization, especially under skewed multi-tenant demand.

6. Integration boundaries, limitations, and extensions

ExpertWeave is designed for seamless insertion into existing MoE inference engines. The source description emphasizes that the rest of the MoE inference pipeline remains unchanged and that only one rerouting operator is inserted between router and dispatch/GMM. This sharply limits the integration boundary and helps explain the reported low overhead (Shi et al., 25 Aug 2025).

The system’s conclusions are stated in terms of four takeaways. First, multi-adapter ESFT serving becomes practical by sharing one MoE base model and dynamically mapping only the needed adapter experts. Second, virtual memory eliminates padding fragmentation, yielding 94x more KV cache and enabling 20-adapter scale with only 4–11% latency overhead. Third, a single fused rerouting kernel integrates seamlessly into existing MoE inference engines with less than 1% added kernel cost. Fourth, accuracy is identical to serving each merged model standalone (Shi et al., 25 Aug 2025).

The paper also outlines several potential extensions. These include applying the same virtual-tensor plus batched-rerouting design to other sparsity paradigms such as GShard-style MoE and product-key MoE; combining the approach with quantized expert weights such as INT8/4 or with further PEFT methods including prefix tuning and prompt tuning; extending virtual memory management to dynamic on-the-fly adapter eviction and load balancing in large multi-tenant clusters; and exploring automated $A_k \subseteq \{1 \ldots M\}$3 selection or hierarchical tiling to further reduce worst-case virtual size (Shi et al., 25 Aug 2025).

These forward-looking directions should be treated as prospective rather than established results. A plausible implication is that ExpertWeave represents a more general serving pattern for sparse model specialization: maintain a stable base execution graph, colocate specialized parameters in a unified virtual namespace, and perform lightweight runtime indirection at the expert-selection boundary. Within the scope of the reported work, however, the validated setting is concurrent ESFT adapter serving for MoE models on Ascend NPUs (Shi et al., 25 Aug 2025).