Shared Expert Isolation in ML Systems

• Shared Expert Isolation is a set of algorithmic techniques for efficiently managing multiple specialized experts in distributed ML models while preventing cross-contamination.
• It leverages virtual-memory management to dynamically map only active experts, reducing memory overhead (e.g., 1.51× improvement over padding) and achieving throughput gains up to 18% with minimal latency.
• The approach enables multi-tenant inference by combining strict expert isolation with shared knowledge distillation, fostering scalable, personalized, and resource-efficient ML systems.

Shared Expert Isolation refers to a set of algorithmic and systems techniques for managing expert-specialized models in large-scale distributed or modular machine learning architectures. The central principle is the ability to serve or distill numerous task- or cluster-specific "experts" while maintaining both memory and computational efficiency, and simultaneously preserving the isolation of task- or group-specific adaptivity and the sharing of globally beneficial knowledge. Key instantiations include multi-tenant inference for Mixture-of-Experts (MoE) models with expert-specialized fine-tuning (ESFT), subspace expert merging for MoE compression, and federated settings where cross-group knowledge is distilled into a "universal" expert. These approaches address the tension between scalability, personalization, and efficient resource utilization in domains with significant model heterogeneity.

1. Systemic Motivation and Problem Definition

Shared expert isolation arises in settings where multiple adaptations, specializations, or client clusters must either (a) be concurrently served on a single infrastructure (e.g., multi-adapter MoE inference), or (b) have their knowledge recombined to extract globally relevant information (e.g., federated learning with cross-cluster distillation). Naïvely partitioning resources for each expert yields prohibitive memory and compute costs and quickly scales out of reach for large models or high model counts. Moreover, typical additive adapter approaches (e.g., LoRA, residual fine-tuning) do not generalize to expert-oriented architectures, as in ESFT, or to nontrivial aggregation requirements across heterogeneous clusters. The objective is to:

• Serve, compress, or distill multiple specialized experts with minimal redundancy and maximal resource efficiency.
• Ensure strict isolation of the expert's unique information—no cross-contamination—while enabling shared access to the base or generalized representations.

2. Shared Expert Isolation in MoE Inference: The ExpertWeave Framework

ExpertWeave (Shi et al., 25 Aug 2025) exemplifies shared expert isolation in scalable MoE inference, particularly for expert-specialized fine-tuning. Its system overlays a virtual-memory-backed "virtual weight tensor" of shape $[M + N \cdot E_{max}, H_{out}, H_{in}]$, where $M$ is the number of base-model experts, $N$ the number of concurrent adapters, and $E_{max}$ the maximum number of fine-tuned experts per adapter/layer. This virtual address space is unified across base and adapter experts but only the needed subranges—the active experts—are physically mapped in RAM, managed by a physical memory pool and an expert weight manager.

A deterministic two-level mapping table ("ESFT expert map" $\Pi^{(\ell)}$) indexes, for each adapter and base-expert, the virtual location of the relevant expert copy, supporting both non-overridden and overridden cases. At inference, a lightweight batched rerouting operator replaces expert IDs on-the-fly, such that the rest of the MoE dispatch and grouped matrix multiplication pipeline remains entirely unchanged. This rerouting is executed via a fused kernel, yielding an amortized latency overhead of only 4–11% for up to 20 adapters.

Memory usage benefits are substantial: ExpertWeave avoids the fragmentation and padding overheads of naïve approaches, with a measured blow-up factor of $F_{mem} \approx 1.51$ for padding-based schemes (i.e., 51% overhead) compared to the joint virtual space approach, which only allocates physical memory for active experts. Compared to merged or siloed serving, ExpertWeave supports multi-adapter deployment while maintaining up to $94\times$ more key-value cache capacity and improving throughput by up to 18%, all with bitwise-accurate model outputs for each adapter.

3. Virtual-Memory Techniques and Resource Efficiency

The virtual-memory-assisted expert weight manager underpinning ExpertWeave reserves one large virtual address space for all potential experts (base and adapters), preventing address fragmentation and facilitating dynamic page mapping only for the experts actually instantiated. This strategy enables:

• Sublinear, near-minimal memory allocation relative to the total number of adapters and experts.
• Efficient adapter admission/eviction via mapped/unmapped pages, eliminating the quadratic waste of static padding.
• Compositional integration with existing MoE serving pipelines, as the main MoE computation (grouped matrix multiplication) sees a single contiguous, virtually indexed tensor.

In equations, total memory for $N$ adapters is:

$M_{shared} = M_{base} + \sum_{i=1}^N \Delta M_i$

where $M_{base}$ is base expert memory, $\Delta M_i$ is per-adapter fine-tuned expert memory, and $M_{overlap} = 0$ under strict per-adapter isolation (no expert sharing).

4. Algorithmic Realization of Batched Rerouting

For each batch of $B$ tokens and each MoE layer $\ell$, and top-$k$ arrays $\text{TopK}[B, k]$ and adapter IDs $\text{AID}[B]$, shared expert isolation uses the following kernel:

for b in range(B): for t in range(k): j = TopK[b, t] i = AID[b] TopK_prime[b, t] = Pi[i, j]

Tokens are then dispatched into the grouped matrix multiplication kernel using $\text{TopK}'$ as indices into the virtual tensor.

Performance evaluation shows that without batch fusion, naively gathering expert weights incurs 29% slowdown. With the fused kernel, the overhead per layer is under 1%, and the overall end-to-end latency increase remains below 11% even at maximal scale.

5. Preservation of Model Integrity and Empirical Evaluation

Empirical benchmarks (Shi et al., 25 Aug 2025) demonstrate that shared expert isolation as implemented in ExpertWeave introduces zero model accuracy degradation compared to offline-merged checkpoints. On GSM8K (math) and intent-recognition tasks, ExpertWeave achieves exactly the same per-task accuracy as merged baselines (GSM8K: 62.3%; intent: 78.8%), while the underlying MoE-only model scores considerably lower (GSM8K: 56.5%; intent: 18.6%), demonstrating that the performance gain is from ESFT adapters, not from the serving infrastructure. This is attributed to the deterministic, injective mapping of expert IDs and the unmodified core computational kernels, which maintain the precise per-adapter parameters. Shared expert isolation therefore enables large-scale, multi-tenant inference without accuracy compromise or task-identity leakage.

6. Related Paradigms: MoE Compression and Federated Aggregation

In MoE compression, Sub-MoE (Li et al., 29 Jun 2025) pursues expert merging via subspace alignment, effectively isolating per-expert specialization in the $V$ components of joint SVD, while retaining shared input structure in a common $U$-matrix. Whereas ExpertWeave maintains strict expert separation at inference, Sub-MoE emphasizes parameter efficiency but must account for conflicts between merged expert functions. Expert clustering by output similarity (cosine distance) and frequency-weighted merging reduces these conflicts but does not enforce hard isolation between experts.

In federated learning, shared expert isolation appears in frameworks that distill a global "universal expert" from cluster-specialized experts (Leng et al., 25 Jun 2025). The process alternates between local client training, cluster-specific model aggregation, and knowledge distillation into a shared model using a synthetic data generator. Distillation of logits isolates global information in the universal model while heterogeneity (idiosyncratic knowledge) remains with per-cluster experts. This enables both effective personalization and cross-cluster generalization with empirical results demonstrating increased accuracy and faster convergence under severe non-IID conditions.

7. Theoretical and Practical Implications

Shared expert isolation provides a principled framework for modularity in large, distributed, or multi-user machine learning systems where both shared and isolated knowledge components are crucial. Benefits include:

• Resource-efficient, high-throughput serving of multi-domain or multi-task models without parameter or memory explosion.
• Deterministic control over adapter activation and routing with minimal system overhead.
• Full preservation of task-specific accuracy and model integrity via bitwise-accurate dispatch.
• Flexible extension to federated and compressed-model settings, where it forms the backbone for robust personalization and cross-task or cross-client knowledge transfer.

A plausible implication is that shared expert isolation techniques will become foundational primitives in future ML systems supporting multi-tenant, on-demand adaptation scenarios. The systematic separation and efficient co-location of experts ensure scalability, performance, and privacy guarantees as model and deployment heterogeneity increases.