Multiple Experts Fine-Tuning Framework (MEFF)

• MEFF is a framework that leverages specialized lightweight experts with dynamic gating to efficiently fine-tune frozen foundation models.
• It employs techniques like LoRA submodules and prompt tokens to optimize resource allocation, reduce compute overhead, and mitigate catastrophic forgetting.
• Its design supports continual and federated learning by enabling modular expansion and fine-grained task adaptation with minimal retraining.

A Multiple Experts Fine-tuning Framework (MEFF) encompasses a family of architectures and methodologies that leverage a pool of specialized, lightweight “experts”―typically parameter-efficient modules such as low-rank adapters or prompt tokens―in conjunction with frozen backbone models to efficiently adapt foundation models to diverse tasks, domains, and user constraints. MEFFs deploy dynamic or static routing (gating) mechanisms to activate (sparsely or densely) subsets of these experts on a per-input or per-task basis, optimizing for compute, memory, adaptability, and mitigation of catastrophic forgetting. This paradigm supports continual or federated adaptation, task composition, multi-domain learning, and knowledge reuse with minimal retraining or communication burden.

1. Core Principles and Motivation

The MEFF paradigm addresses the intractability of full-parameter fine-tuning for large models in resource- or privacy-constrained scenarios, as well as the inefficiencies and interference resulting from coarse-grained module reuse or naive merging strategies (Wu et al., 30 Nov 2025). Rather than training new adapters for each new task or client from scratch, MEFF methodologies exploit a modular design where pretrained or previously fine-tuned lightweight experts encode prior semantic knowledge. This approach enables:

• Scalability: Experts can be incrementally expanded for new domains, tasks, or users with negligible interference.
• Resource Awareness: Sparse activation and elastic expert budgeting enable fine-tuning under stringent compute, memory, and communication constraints.
• Mitigation of Catastrophic Forgetting: By avoiding destructive interference between experts or with the frozen backbone, original task performance is preserved during continual adaptation (Wang et al., 9 Apr 2025, Cardona-Giraldo et al., 27 Mar 2026).
• Fine-Grained Adaptation: Rank-wise decomposition (e.g., in LoRA modules) or residual structures enable more precise reuse of adaptation capacity (Wu et al., 30 Nov 2025, Zeng et al., 8 Apr 2025).

2. Structural Variants of MEFF

2.1 Rank-wise and Fine-Grained Expert Decomposition

MEFF often decomposes traditional adaptation modules, such as LoRA matrices $\Delta W = BA$, into individual rank-one experts $E_i(\mathbf{x}) = (B_{:,i}A_{i,:})\mathbf{x}$, so that the update becomes a sum over $M$ experts drawn from multiple LoRA modules:

$$\mathbf{h}' = W_0 \mathbf{x} + \sum_{m=1}^M \tilde{g}_m E_m(\mathbf{x})$$

This fine-grained decomposition (MoRE) enhances expressivity and enables input- or resource-conditioned gating (Wu et al., 30 Nov 2025).

2.2 Mixture-of-Experts Prompting and Decoding

In the vision domain, prompt-based MEFF variants define sets of expert-specific prompt tokens and dynamically combine them via a learnable dispatcher at each transformer layer. For each forward pass, the prompt input to a frozen encoder is a convex combination of the $K$ experts’ prompts, with gating determined by local context and task requirements (Mo et al., 26 Feb 2026). Dense prediction architectures may further split experts intra-task and inter-task (fine-grained, shared, and global), selectively routing among them for each spatial or semantic location in the decoder (Xu et al., 25 Jul 2025).

2.3 Modular Expansion and Continual Learning

In static or continual settings, MEFFs assemble a sequential or parallel ensemble of experts with expert-specific gating. In some frameworks (e.g., SEE), each expert encompasses both a familiarity classifier (via indicator tokens) and its own adaptation module, removing the need for a centralized router (Wang et al., 9 Apr 2025). Models may expand by adding new experts without retraining previously learned components, preserving original task behaviors (Cardona-Giraldo et al., 27 Mar 2026).

2.4 Graph-based and Hierarchical Routing

More advanced MEFFs incorporate graph-based routers (e.g., GNNs over the expert set), which enable information sharing and load balancing through expert–expert neighborhood interactions (Bai et al., 2024). Structural or hierarchical mixtures (S'MoRE) arrange low-rank residuals in multilayer trees, allowing tokens to traverse custom sub-trees and yielding exponential expressivity under constant parameter budgets (Zeng et al., 8 Apr 2025).

3. Routing, Gating, and Resource Allocation

MEFFs rely on diverse gating mechanisms:

• Input-conditioned softmax routers: A gating matrix $W_{\mathrm{router}}$ transforms the current hidden state into routing scores, with top-$K$ masking for compute control (Wu et al., 30 Nov 2025, Mo et al., 26 Feb 2026, Liu et al., 4 Aug 2025).
• Task-motivated or prompt-driven gates: Softmax gates over task embeddings or natural-language prompts enable direct correlation-based expert assignment (Lin et al., 2024, Liu et al., 2023).
• Sequential or distributed routing: Experts autonomously decide to claim or reject an input, enabling OOD detection and robust continual learning without explicit routers (Wang et al., 9 Apr 2025).
• Elastic expert quota allocation: Global and per-layer expert quotas are dynamically reallocated based on estimated marginal performance contribution, leveraging group importance scores and multi-phase resource filling (Wu et al., 30 Nov 2025).
• Load-balancing and regularization: Auxiliary losses encourage balanced expert utilization (e.g., Poisson and Normal target distributions for specialization and balance) and prevent routing collapse or expert starvation (Bai et al., 2024, Li et al., 2024).
• Hierarchical top-down routers: Structural frameworks employ multi-level selection of child experts at each routing layer, building dynamic trees per token (Zeng et al., 8 Apr 2025).

4. Parameter Efficiency and Optimization Workflows

The MEFF design imposes strict parameter efficiency:

• Frozen Backbone: All original backbone weights are fixed, so adaptation modules alone are trainable.
• Expert Budgeting: The aggregate number of trainable parameters can be kept at or below a single LoRA module via sharing or decomposition (e.g., $N$ experts of rank $r/N$) (Liu et al., 2023).
• Trainable Router/Dispatcher: Either a lightweight per-layer gating matrix or small MLP governs expert selection. In some settings, the router is the only globally updated parameter during fine-tuning rounds (not full experts) (Wu et al., 30 Nov 2025).
• PEFT Module Choices: LoRA (Wu et al., 30 Nov 2025, Liu et al., 2023), adapters (Lin et al., 2024), prompt-tuning (Mo et al., 26 Feb 2026, Zeng et al., 31 Aug 2025), or hybrid structures are supported as expert modules.
• Federated and Continual Learning: Devices or tasks can independently add or specialize experts without communication or retraining, ensuring privacy and scalability (Wu et al., 30 Nov 2025, Cardona-Giraldo et al., 27 Mar 2026, Wang et al., 9 Apr 2025).

Typical objectives combine standard cross-entropy (or task-specific) losses for prediction with sparsity or balance regularization for routing (Wu et al., 30 Nov 2025, Li et al., 2024, Bai et al., 2024). Training proceeds via standard optimizers (e.g., AdamW), with group-wise or task-wise updates, and all backbone weights strictly frozen.

5. Empirical Outcomes and Efficiency Gains

MEFF instantiations consistently report marked improvements in efficiency and performance relative to single-expert or classical PEFT methods:

Framework	Main Efficiency/Perf Gains	Reference
SmartFed (MoRE)	3–10 point accuracy gain, 3.9× convergence speed, 31.5× comm reduction	(Wu et al., 30 Nov 2025)
pMoE (Prompt)	+2–5% avg accuracy, <5% runtime/FLOPs cost, broad task span	(Mo et al., 26 Feb 2026)
FGMoE (MTL)	Outperforms MoE baselines, Am=+0.39–+21.0% using <5% params	(Xu et al., 25 Jul 2025)
S'MoRE	+1–2 pp vs. MixLoRA/HydraLoRA, 10–50% fewer params, exponential flexibility	(Zeng et al., 8 Apr 2025)
SEE (Continual)	BWT≈0, AR matches/beats MTL, >99.5% OOD recall, no forgetting	(Wang et al., 9 Apr 2025)
MoFE (Frozen)	>90% reduction trainables, train time halved, <3% accuracy trade-off	(Seo et al., 9 Mar 2025)
GMoE (Graph-MoE)	Best accuracy in 10/12 tasks, halves instability, <50% trainable reduction	(Bai et al., 2024)
PEMT (MTL MoE)	+4.0 pts over FT (GLUE/SuperGLUE), few-shot robustness	(Lin et al., 2024)

In federated adaptation, MEFFs like SmartFed accomplished data efficiency (small-data fine-tune matches full-data baselines), and communication savings by reusing pool experts. In dense prediction or multi-task learning, MEFFs with structured or fine-grained experts maintain or improve task accuracy with <5% extra parameters (Xu et al., 25 Jul 2025). In continual learning, frameworks such as SEE ensure nearly zero forgetting with linearly scaling parameter cost (Wang et al., 9 Apr 2025).

6. Specializations, Extensions, and Limitations

• Task Diversity and Seesaw Effects: Task-motivated, gate-weighted mixtures (e.g., MOELoRA) address imbalanced data and mitigate performance collapse on minority tasks (Liu et al., 2023).
• Knowledge Reuse and Modular Transfer: Prompt- and adapter-based MEFFs support flexible transfer by integrating soft prompt-based task similarity into gating, yielding robust adaptation even for unrelated downstream tasks (Lin et al., 2024, Wu et al., 30 Nov 2025).
• Structured and Collaborative Routing: Graph-based routers extend independence assumptions, supporting expert collaboration and reducing inter-expert interference (Bai et al., 2024).
• Empirical Scalability: Most studies focus on backbones up to the ~10B parameter scale; scalability to >100B parameter models with thousands of experts remains to be demonstrated (Bai et al., 2024).
• Inference and Memory: The end-to-end cost is dominated by the number of activated experts per forward pass; efficient batching, kernel fusion, and dynamic quota allocation are essential for deployment (Li et al., 2024, Wu et al., 30 Nov 2025).
• Potential for Structural Hybrids: Hierarchical or GNN-based mixtures realize exponential numbers of effective “expert pathways” without incurring linear parameter growth (Zeng et al., 8 Apr 2025).

7. Representative Algorithmic Flows

The MEFF paradigm can be summarized by the following algorithmic stages:

1. Expert Pool Construction: Gather or learn a repository of experts—fine-grained (rank-one), task/domain-specific adapters, or prompt modules—across prior domains/tasks.
2. Router/Dispatcher Training: On new data, train lightweight routers to sparsely select, combine, or weight experts on a per-input, per-task, or per-location basis, subject to compute and memory constraints.
3. Resource-Aware Allocation: Employ elastic quota allocation or balancing losses to focus resources on highly impactful experts and avoid overload or starvation.
4. Inference: For each input, dynamically activate the top-K experts per group, aggregate their outputs according to the learned routing weights, and combine with frozen backbone predictions.
5. Continual/Incremental Expansion: New experts and quota are added modularly as needed, with minimal or no retraining of existing modules.

Many instantiations, such as SmartFed’s workflow or FGMoE’s decoder-only adaptation, optimize only routers and adapters, never the backbone, and allow persistent expansion and efficient scaling for federated, multi-task, and continual learning settings (Wu et al., 30 Nov 2025, Xu et al., 25 Jul 2025, Seo et al., 9 Mar 2025).

---

Multiple Experts Fine-tuning Frameworks provide a general, powerful infrastructure for parameter-efficient, modular, and scalable fine-tuning across modalities, task compositions, institutions, and users. State-of-the-art approaches leverage rank-wise, structured, prompt-based, and graph-coordinated experts, coupled with elastic routing, to realize substantial efficiency and performance gains across large-scale language, vision, and scientific modeling applications. These architectures offer robust solutions for privacy-preserving, resource-constrained, and continually evolving adaptation workflows in the era of foundation models (Wu et al., 30 Nov 2025, Wang et al., 9 Apr 2025, Zeng et al., 8 Apr 2025, Bai et al., 2024, Lin et al., 2024, Liu et al., 2023).