Expert-Specialized Fine-Tuning (ESFT)

Updated 1 March 2026

Expert-Specialized Fine-Tuning (ESFT) is a parameter-efficient paradigm that selectively adapts only task-relevant MoE experts to optimize performance.
It uses metrics such as routing gate scores and token activation frequency to select and update specialized subnetworks while freezing the remainder.
ESFT achieves strong per-task performance and mitigates catastrophic forgetting with minimal parameter tuning, enabling scalable domain adaptation.

Expert-Specialized Fine-Tuning (ESFT) is a parameter-efficient training paradigm that fully exploits the modularity and sparsity of Mixture-of-Experts (MoE) architectures and expert-defined adapter pools, allowing precise adaptation of large pretrained models to diverse tasks or domains by tuning only highly relevant, specialized subnetworks. ESFT combines insights from modular routing, low-rank adaptation, parameter sensitivity, and dynamic expert gating, yielding strong per-task performance, robust resistance to catastrophic forgetting, and substantial reductions in memory and computation.

1. Foundations and Core Principles

The central tenet of ESFT is the restriction of trainable parameters to a subset of modules—either MoE experts, specialized adapters, or parameter blocks—that are empirically determined to be most relevant to the downstream task. For each task, only these "task-relevant experts" are updated, while all others (including the backbone and unrelated experts) remain frozen. Selection is typically based on metrics such as routing gate scores, token activation frequency, or gradient sensitivity measured with task-specific samples (Wang et al., 2024, Shi et al., 25 Aug 2025, Xu et al., 6 May 2025).

This selective adaptation avoids cross-task interference and catastrophic forgetting that typically arise when unrelated specialties are co-adapted, and enables efficient handling of very large models with modular expert banks. ESFT is particularly well suited to sparse MoE LLMs but also generalizes to settings with expert adapters, as in PHATGOOSE (Muqeeth et al., 2024) and XFT (Ding et al., 2024).

2. Selection and Specialization of Experts

The first stage of ESFT consists of expert specialization and selection. The methodology entails:

Scoring Expert Relevance: For each expert in every MoE layer (or each candidate adapter), a task-specific score is computed over a small validation set. Common metrics include average router gating probabilities, token-level selection ratios, or cumulative gradient sensitivity (Wang et al., 2024, Xu et al., 6 May 2025).
Thresholding and Subset Selection: Experts are sorted by relevance; a minimal subset per layer is selected to cover a threshold percentage of total gating/activity, e.g., the top $p$ fraction covering 90% of mass (Shi et al., 25 Aug 2025). Alternatively, per-block sensitivity analyses convert scores into a constrained allocation problem solved by knapsack optimization (Xu et al., 6 May 2025).
Freezing the Remainder: All parameters outside the selected experts (including router weights, shared modules, and unrelated experts) are frozen, guaranteeing isolation and minimal memory footprint.

In the context of multiple domain adaptation or multi-tasking, ESFT can be extended with dynamic expert-domain correlation mapping, where the mapping $M_{d,e}$ between each domain and expert is learned online to account for overlapping subspace usage, as in DES-MoE (Li et al., 21 Sep 2025).

3. Training Procedures and Mathematical Formulations

Once the expert subset is determined, ESFT fine-tunes only these modules using task-specific loss objectives. General steps include:

Empirical Loss Functions: Standard objectives such as cross-entropy for autoregressive or classification outputs, mean squared error for regression or embedding similarity, and regularization (e.g., $\ell_2$ norm) on adaptation weights (Chen et al., 2023, Lei et al., 10 Nov 2025, Lu et al., 2024).
Two-Stage Schemes: In modular adapter frameworks (e.g., TT-LoRA MoE), expert adapters are trained independently and sequentially, with base model and other experts always frozen. Later, a lightweight router is trained for task-agnostic selection at inference, with no interference possible due to frozen expert parameters (Kunwar et al., 29 Apr 2025).
Explicit Routing: For architectures with dynamic or learned router modules, ESFT typically keeps the original router fixed, but in cases such as DES-MoE or CEFT, router adaptation precedes or interleaves with expert tuning to ensure context alignment (Li et al., 21 Sep 2025, Bai et al., 27 Aug 2025).
Parameter Efficiency: ESFT achieves substantial reduction in trainable parameters. For example, tuning only 5–15% of experts per layer matches full-parameter fine-tuning (FFT) on domain tasks while drastically reducing memory, as shown in DeepSeek-V2-Lite experiments (Wang et al., 2024).

Example: ESFT Training Algorithm (MoE LLM)

def ESFT_train(model, data, expert_selector, fine_tune_steps):
    selected_experts = expert_selector(model, data)
    freeze_all_except(model, selected_experts)
    for step in range(fine_tune_steps):
        x, y = next_batch(data)
        loss = model.compute_loss(x, y)
        update_params(loss, params=selected_experts)

This procedure is specialized with architectural details (e.g., tensor adapters, routing kernels) according to framework.

4. Variants: Modular Adapter ESFT, Dynamic Routing, and Post-Hoc Composition

Several architectural and training extensions of ESFT further improve adaptability and scalability:

Tensorized and Hierarchical Experts: TuckA organizes experts as slices of compact decomposition tensors, enabling hierarchical batch-level routing and extreme parameter efficiency across adapted layers (Lei et al., 10 Nov 2025).
Contrastive and Expert-Ensembled Labeling: In embedding models, ESFT leverages soft supervision by aggregating multiple expert similarity scores into soft targets, balancing generality and task specificity (Lu et al., 2024).
Wavelet and Feature-conditioned Experts: Visual ESFT frameworks utilize independent feature extractors (e.g., wavelet experts) and injection adapters for fine-grained, image-specific specialization, as in WEFT (Sun et al., 14 Jan 2026).
Post-hoc Routing and Composition: PHATGOOSE demonstrates zero-shot transfer improvements via post-hoc gating over a large pool of frozen expert adapters, with adaptive per-token and per-layer gating learned after all experts are trained. This enables modular accumulation of expertise (including from decentralized data) (Muqeeth et al., 2024).
Scalable Serving: ExpertWeave introduces virtual-memory-assisted colocation and fused rerouting kernels, enabling concurrent, largescale serving of many ESFT adapters over a shared MoE backbone with negligible latency and no accuracy degradation (Shi et al., 25 Aug 2025).

5. Empirical Results and Comparative Analyses

ESFT consistently matches or exceeds full fine-tuning on specialized tasks, while preserving general ability and reducing catastrophic forgetting:

Model/Method	Specialized Task (%)	General Retention	Params Tuned (%)	Speedup/Mem
Full Fine-Tuning (FFT)	51.0	58.8	100	1×
ESFT-Token (DeepSeek-V2)	49.4	61.5	5–15	~1.4×
TT-LoRA MoE	79.04 (avg acc)	–	0.03	1.1–1.9×
LoRA-SMoE-S (60% budget)	83.2 (avg acc)	–	1.6	1.2×
WEFT (vision)	0.896 (IoU)	–	4.5	↑14.7%
PHATGOOSE (T0 HO)	56.9 (acc)	–	–	–
DES-MoE (multi-domain)	51.6 (avg acc)	~98% retention	domain-specific	68% faster

Tabulated metrics collated from (Kunwar et al., 29 Apr 2025, Wang et al., 2024, Sun et al., 14 Jan 2026, Xu et al., 6 May 2025, Muqeeth et al., 2024, Li et al., 21 Sep 2025, Shi et al., 25 Aug 2025).

Key empirical findings include:

ESFT reduces storage and memory overhead by up to 90%, allowing tractable adaptation of very large models on commodity hardware.
Specialist-only fine-tuning (ESFT) can achieve equivalent or superior performance to dense adapter strategies or FFT on core tasks, especially when expert granularity is high.
Routing and specialization substantially mitigate catastrophic forgetting, promote modularity, and enable flexible incremental insertion of new experts (Kunwar et al., 29 Apr 2025, Li et al., 21 Sep 2025).
In the context of multi-expert financial and retrieval models, ESFT outperforms flat LoRA or AdapterFusion baselines by significant margins on downstream metrics (Chen et al., 2023).
For code models (XFT), upcycled MoE fine-tuning and merging delivers substantial improvements in functional benchmarks over single/dense-trained baselines (Ding et al., 2024).

6. Applications, Adaptations, and Broader Implications

ESFT has been instantiated in a wide range of domains and architectures:

Natural Language and Multitask LLMs: Specialist tasks in finance, mathematical reasoning, code generation, and domain-specific consulting (Chen et al., 2023, Lei et al., 10 Nov 2025, Ding et al., 2024).
Vision: Expert feature extractors and conditional adapters for segmentation tasks in remote sensing and medical imaging (WEFT) (Sun et al., 14 Jan 2026).
Retrieval and Embedding Models: Textual similarity and semantic search improved via contrastive, expert-augmented supervision (Lu et al., 2024).
Dynamic and Continual Adaptation: Adaptive routing and multi-phase fine-tuning (DES-MoE) for catastrophic-forgetting-free multi-domain adaptation (Li et al., 21 Sep 2025).
Inference-Scale Serving: Highly concurrent, memory-efficient deployment of many specialist adapters for diverse client tasks (ExpertWeave) (Shi et al., 25 Aug 2025).

The modular, compositional structure of ESFT frameworks enables the accumulation and serving of ever-growing pools of domain-, task-, or context-specialized “skills” in a single shared foundation, yielding a practical path toward lifelong and federated learning (Muqeeth et al., 2024).

7. Limitations, Open Questions, and Future Directions

Despite substantial gains, several aspects of ESFT remain areas of active research:

Expert Pool Scaling: As the number of specialized experts increases, routing and storage costs may require further algorithmic innovation (e.g., hierarchical routers, sub-linear selection) (Muqeeth et al., 2024).
Expert Granularity: Coarse grouping of MoE units reduces the benefits of selective fine-tuning, pointing to the necessity of fine-grained MoE architectures for maximal ESFT efficiency (Wang et al., 2024).
Dynamic vs. Static Selection: Adaptive, per-batch or per-input routing (as in CEFT and DES-MoE) can further mitigate interference but introduces new complexity in router design and training stability (Bai et al., 27 Aug 2025, Li et al., 21 Sep 2025).
Combination with Other PEFT Methods: Integrating ESFT with LoRA, tensor adapters, and quantized adaptation promises further compression and democratization (Kunwar et al., 29 Apr 2025, Lei et al., 10 Nov 2025, Huang, 25 Sep 2025).
Theoretical Analysis: Understanding the intrinsic dimension, routing entropy, and combinatorial capacity of expert sets remains an open challenge (Wang et al., 2024, Muqeeth et al., 2024).
Automatic Selection and Online Adaptation: Automated thresholding, continual incorporation of new experts, and seamless extension to multimodal or structured domains are under exploration.

References

(Wang et al., 2024) Let the Expert Stick to His Last: Expert-Specialized Fine-Tuning for Sparse Architectural LLMs
(Shi et al., 25 Aug 2025) ExpertWeave: Efficiently Serving Expert-Specialized Fine-Tuned Adapters at Scale
(Xu et al., 6 May 2025) A Sensitivity-Driven Expert Allocation Method in LoRA-MoE for Efficient Fine-Tuning
(Kunwar et al., 29 Apr 2025) TT-LoRA MoE: Unifying Parameter-Efficient Fine-Tuning and Sparse Mixture-of-Experts
(Sun et al., 14 Jan 2026) Small but Mighty: Dynamic Wavelet Expert-Guided Fine-Tuning
(Chen et al., 2023) DISC-FinLLM: A Chinese Financial LLM based on Multiple Experts Fine-tuning
(Lei et al., 10 Nov 2025) TuckA: Hierarchical Compact Tensor Experts for Efficient Fine-Tuning
(Lu et al., 2024) Improving embedding with contrastive fine-tuning on small datasets with expert-augmented scores
(Muqeeth et al., 2024) Learning to Route Among Specialized Experts for Zero-Shot Generalization
(Huang, 25 Sep 2025) Fine-tuning of LLMs for Domain-Specific Cybersecurity Knowledge
(Ding et al., 2024) XFT: Unlocking the Power of Code Instruction Tuning by Simply Merging Upcycled Mixture-of-Experts
(Li et al., 21 Sep 2025) Dynamic Expert Specialization: Towards Catastrophic Forgetting-Free Multi-Domain MoE Adaptation
(Liu et al., 4 Aug 2025) Parameter-Efficient Routed Fine-Tuning: Mixture-of-Experts Demands Mixture of Adaptation Modules
(Bai et al., 27 Aug 2025) Understanding and Leveraging the Expert Specialization of Context Faithfulness in Mixture-of-Experts LLMs

Collectively, these works establish ESFT as a robust, scalable, and efficient paradigm for modular specialization, task adaptation, and domain retention in large pre-trained models across language, vision, and retrieval applications.