Mixture of Expert Prompt Tuning (MEPT)

• Mixture of Expert Prompt Tuning (MEPT) is a modular adaptation technique that conditions a frozen backbone model with multiple prompt experts via a dynamic routing mechanism.
• It employs diverse architectural forms, including attentional prompt interpolation and low-rank prompt factorization, to balance efficiency and task adaptability.
• Empirical analyses show that MEPT methods significantly reduce parameter counts while improving metrics such as perplexity, F1 scores, and adversarial robustness.

Searching arXiv for recent and foundational papers on MEPT and closely related prompt-mixture methods. Mixture of Expert Prompt Tuning (MEPT) denotes a family of parameter-efficient adaptation methods in which a frozen backbone model is conditioned by multiple prompt experts and a routing mechanism that selects or weights those experts per input. In the formulation made explicit in later work, MEPT methods combine three elements: a shared backbone model, a set of prompt parameters partitioned into multiple experts, and a routing mechanism that selects or weights those prompt-experts per input (Wang et al., 5 Nov 2025). Earlier and closely related instantiations already appeared as attentional mixtures of soft prompts for LLMs, prompt-group mixtures for heterogeneous task adaptation, and prompt-expert mixtures for multimodal fusion (Asai et al., 2022, Dun et al., 2023, Jiang et al., 2024).

1. Conceptual development

The idea emerged from the observation that a single soft prompt is often too rigid for heterogeneous tasks, domains, or input regimes. In ATTEMPT, source-task soft prompts are treated as reusable modules, and an attention module interpolates them with a newly initialized target prompt for every instance in the target task (Asai et al., 2022). In “Mixture of Prompts” for LLM task adaptation, the core claim is stated even more directly: prompts are the experts, the base LLM is frozen, and a smart gate dynamically assigns combined experts based on the target task (Dun et al., 2023).

Later work generalized the pattern across modalities. Multimodal fusion methods replaced a single global prompt with prompt experts selected by a conditional router (Jiang et al., 2024). Graph transfer methods attached expert-specific prompt vectors to frozen graph experts and routed them with structure-aware gating (Wang et al., 5 Nov 2025). The explicit MEPT formulation casts prompt experts as adaptive manifold-mapping modules that activate different neural pathways for different regions of the data distribution (Zeng et al., 31 Aug 2025).

This suggests that MEPT is best understood as a design pattern rather than a single architecture. Across the literature, the recurring premise is that prompt-space modularity can recover some of the conditional capacity of mixture-of-experts systems without updating the backbone weights.

2. Architectural forms

At the most abstract level, MEPT inherits the classical MoE template

$y(x) = \sum_{i=1}^{K} g_i(x)\, e_i(x),$

but replaces network experts with prompt experts. In prompt-centric formulations, the effective conditioning signal becomes a learned combination of prompt modules rather than a learned combination of FFN outputs (Dun et al., 2023).

Different papers instantiate this pattern differently. ATTEMPT forms an instance-wise prompt

$$\mathbf{P}_{\text{instance}}(\mathbf{X}) = \mathbf{P}_{\text{target}} + \sum_{j=1}^{t+1} a_j \mathbf{P}_j,$$

where source prompts and a target prompt are mixed by an attention module over pooled prompt and input representations (Asai et al., 2022). PT-MoE instead factorizes each prompt expert as a low-rank product and routes over decomposed prompts,

$P = \sum_{i=1}^{N} w_i A_i B,$

with expert-specific $A_i$, a shared basis $B$, and a router over the mean input embedding (Li et al., 14 May 2025). In MoPs for decoder-only LLMs, the mixture is realized inside attention rather than by directly summing prompt embeddings: after prompts are concatenated with hidden states, attention columns corresponding to prompt tokens are reweighted by gate outputs,

$$\mathbf{A}^h[:, 0:K] = \mathbf{A}^h[:, 0:K] \odot \mathbf{G},$$

so the gate modulates attention to prompt groups instead of additively mixing expert network outputs (Dun et al., 2023).

A second architectural axis concerns expert granularity. Some methods define experts purely as prompt blocks attached to a shared frozen model. Others couple prompts to backbone experts. GMoPE associates each graph expert $E_m$ with its own prompt vector $\mathbf{p}_m$, so the effective expert is the pair of frozen encoder and expert-specific prompt (Wang et al., 5 Nov 2025). In visual adaptation, pMoE uses multiple frozen visual experts, each equipped with expert-specific prompt tokens, and a dispatcher mixes prompts across experts at several layers (Mo et al., 26 Feb 2026). These variations preserve the MEPT principle: adaptation is concentrated in prompt-space modules and their routing logic.

3. Routing, optimization, and regularization

Routing is the main differentiator among MEPT systems. In MoPs, the gate is a small MLP over the mean hidden representation of the question and context at a middle layer of a frozen decoder-only transformer. Training proceeds in two stages: a supervised gate pretraining stage with an initial 1:1 task-to-expert mapping, followed by joint optimization of gate and prompts under the language-modeling loss (Dun et al., 2023). In PEMT, routing is task-level rather than instance-level: a target task prompt $\mathbf{Q}$ attends over source task prompts $\mathbf{P}_k$ to form a correlation-augmented task representation $$\mathbf{P}_{\text{instance}}(\mathbf{X}) = \mathbf{P}_{\text{target}} + \sum_{j=1}^{t+1} a_j \mathbf{P}_j,$$0, and layer-specific softmax gates over source adapters are computed from $$\mathbf{P}_{\text{instance}}(\mathbf{X}) = \mathbf{P}_{\text{target}} + \sum_{j=1}^{t+1} a_j \mathbf{P}_j,$$1 (Lin et al., 2024).

Other systems use structurally different gates. GMoPE performs structure-aware routing by scoring experts through their training losses on the current graph batch, applying top-$$\mathbf{P}_{\text{instance}}(\mathbf{X}) = \mathbf{P}_{\text{target}} + \sum_{j=1}^{t+1} a_j \mathbf{P}_j,$$2 selection, and combining that routing with a soft orthogonality loss over prompt vectors to prevent expert collapse (Wang et al., 5 Nov 2025). TAME uses a routing network over image-token representations to generate mixture weights for a bank of expert prompts at test time, then updates prompts and router under a combined unsupervised objective consisting of multi-view entropy minimization, alignment to clean and adversarial visual token statistics, and MoE regularization for balanced expert utilization and prompt diversity (Wang et al., 17 May 2026).

Regularization strategies vary accordingly. PEMT introduces a Task Sparsity Loss to increase sparsity in the gated unit and emphasize source experts whose outputs are most aligned with the final hidden representation (Lin et al., 2024). MoPE employs an importance loss based on the coefficient of variation of batch-level expert importance to discourage routing collapse (Jiang et al., 2024). GMoPE regularizes prompt geometry directly through a soft orthogonality loss rather than through explicit load balancing on router outputs (Wang et al., 5 Nov 2025). PT-MoE uses selective top-1 routing with straight-through estimation and Gaussian noise on router logits, and its ablations indicate that selective and probationary routing outperforms non-selective alternatives (Li et al., 14 May 2025).

A common misconception is that MEPT necessarily implies sparse, token-level routing. The literature includes dense prompt interpolation, hard nearest-centroid routing over discrete textual prompt experts, loss-based top-$$\mathbf{P}_{\text{instance}}(\mathbf{X}) = \mathbf{P}_{\text{target}} + \sum_{j=1}^{t+1} a_j \mathbf{P}_j,$$3 routing, and per-layer top-1 expert selection (Asai et al., 2022, Wang et al., 2024, Wang et al., 5 Nov 2025, Zeng et al., 31 Aug 2025).

4. Representative domains and systems

The literature spans language, multimodal fusion, graphs, vision, prompt optimization, and even inference-time behavior steering. Representative instances are summarized below.

System	Domain	Prompt-expert / routing pattern
ATTEMPT (Asai et al., 2022)	NLP multi-task transfer	Source soft prompts as experts; attention module interpolates them with a target prompt per instance
MoPs (Dun et al., 2023)	LLM task adaptation	Prompt groups as experts; smart gate reweights attention to prompt tokens
“Conditional Prompt Tuning for Multimodal Fusion” (Jiang et al., 2023)	Multimodal fusion	Conditional prompts from one modality guide another; MoPE dynamically routes prompt experts
MoPE (Jiang et al., 2024)	Multimodal fusion	Static, dynamic, and mapped prompts; dynamic prompt is a dense mixture of prompt experts
ConstitutionalExperts (Petridis et al., 2024)	Natural-language prompt optimization	Principle-based prompt experts specialized to semantic regions; nearest-centroid routing
“One Prompt is not Enough” (Wang et al., 2024)	In-context prompt optimization	Each expert contains an instruction and demo set; routing by embedding-space region
GMoPE (Wang et al., 5 Nov 2025)	Graph foundation models	Expert-specific graph prompts with structure-aware MoE routing
PT-MoE (Li et al., 14 May 2025)	LLM PEFT	Low-rank decomposed prompt experts with selective routing
pMoE (Mo et al., 26 Feb 2026)	Visual adaptation	Expert-specific prompt tokens for multiple frozen visual experts with dispatcher-based mixing
TAME (Wang et al., 17 May 2026)	Vision-language adversarial defense	Prompt bank with input-dependent routing updated at test time
MoTE (Dahlke et al., 16 Feb 2025)	Inference-time MoE steering	Prompt-based profiling identifies behavior-specific experts; routing is modified at inference without training
MoDEM (Simonds et al., 2024)	Domain-routed LLM systems	Hard domain routing to specialized models; not prompt-level MEPT, but directly relevant to router design

These systems differ in what counts as an expert, what is routed, and when routing is applied. In some formulations the expert is a soft prompt tensor; in others it is an expert-specific prompt attached to a frozen encoder, a natural-language prompt-template expert, or an inference-time expert subset identified through prompt-based profiling. This suggests that MEPT is less about a single parameterization than about a common modularization principle: decouple backbone knowledge from prompt-conditioned, selectively activated adaptation modules.

5. Empirical behavior, efficiency, and specialization

Across LLM adaptation, MEPT-style methods are repeatedly reported to improve over single-prompt baselines under strict PEFT budgets. In MoPs, final perplexity decreases range from $$\mathbf{P}_{\text{instance}}(\mathbf{X}) = \mathbf{P}_{\text{target}} + \sum_{j=1}^{t+1} a_j \mathbf{P}_j,$$4 up to $$\mathbf{P}_{\text{instance}}(\mathbf{X}) = \mathbf{P}_{\text{target}} + \sum_{j=1}^{t+1} a_j \mathbf{P}_j,$$5 relative to baselines in the federated scenario and from $$\mathbf{P}_{\text{instance}}(\mathbf{X}) = \mathbf{P}_{\text{target}} + \sum_{j=1}^{t+1} a_j \mathbf{P}_j,$$6 up to $$\mathbf{P}_{\text{instance}}(\mathbf{X}) = \mathbf{P}_{\text{target}} + \sum_{j=1}^{t+1} a_j \mathbf{P}_j,$$7 in the centralized scenario (Dun et al., 2023). ATTEMPT updates 2,300 times fewer parameters than full fine-tuning and, across 21 diverse NLP datasets, significantly outperforms prompt tuning while matching or outperforming methods that use over ten times more parameters (Asai et al., 2022). PT-MoE improves F1 score by 1.49 points over PT and 2.13 points over LoRA in QA tasks, while enhancing mathematical accuracy by 10.75 points over PT and 0.44 points over LoRA, all while using 25% fewer parameters than LoRA (Li et al., 14 May 2025). The explicit MEPT paper reports notable improvements in mean accuracy on SuperGLUE, for example 1.94%, while significantly reducing activated prompts by 79.25% (Zeng et al., 31 Aug 2025).

In multimodal and visual settings, the same pattern holds. MoPE matches or even surpasses fine-tuning while requiring only 0.8% of the trainable parameters across six multimodal datasets spanning four modalities (Jiang et al., 2024). The earlier conditional prompt tuning formulation reports state-of-the-art results over three multimodal datasets while necessitating 0.7% of the trainable parameters (Jiang et al., 2023). pMoE evaluates 47 adaptation tasks and reports consistent gains over strong prompt-tuning baselines, typically in the 1–4 point range, while keeping parameter multipliers close to standard visual prompt tuning settings (Mo et al., 26 Feb 2026). TAME improves the zero-shot adversarial robustness of the original CLIP by at least 49.1% under AutoAttack while largely preserving generalization on clean samples, and reports an average robustness gain of at least 30.2% over adversarial prompt-tuning baselines (Wang et al., 17 May 2026).

Empirical analyses also emphasize specialization rather than only aggregate performance. Gate analysis in MoPs shows that a pretraining stage with a 1:1 task-to-expert mapping is not a hard constraint: after full training, related tasks converge to shared experts or similar expert mixtures (Dun et al., 2023). In prompt-optimization variants, experts specialize to semantic regions such as celebrity versus country categorization, or distinct movie preference subspaces, and nearest-centroid routing exploits that structure at inference (Wang et al., 2024, Petridis et al., 2024). The explicit MEPT work adds neural activation pathway visualizations showing that semantically similar tasks produce more similar expert-selection trajectories across layers, which the authors interpret as evidence for prompt-expert manifold mapping (Zeng et al., 31 Aug 2025).

6. Theory, limitations, and open problems

Theoretical work on contaminated mixture-of-experts provides a statistical lens for MEPT. In that model, a frozen pre-trained expert is mixed with a trainable prompt expert by a scalar proportion $$\mathbf{P}_{\text{instance}}(\mathbf{X}) = \mathbf{P}_{\text{target}} + \sum_{j=1}^{t+1} a_j \mathbf{P}_j,$$8, and the analysis identifies two fundamental pathologies: prompt vanishing, where the prompt’s mixture proportion converges to zero, and prompt merging, where the prompt expert becomes statistically indistinguishable from the pre-trained expert (Yan et al., 2024). The paper introduces a distinguishability condition to control parameter interaction and shows that convergence rates degrade with small $$\mathbf{P}_{\text{instance}}(\mathbf{X}) = \mathbf{P}_{\text{target}} + \sum_{j=1}^{t+1} a_j \mathbf{P}_j,$$9 or near-merging regimes. This suggests that MEPT performance is constrained not only by optimization quality but also by identifiability of prompt contributions.

A second recurring limitation is expert collapse or under-utilization. GMoPE shows that without prompt-space diversity regularization, experts can become insufficiently specialized; too large an orthogonality weight can in turn destabilize training (Wang et al., 5 Nov 2025). PT-MoE shows that the number of experts is not monotonic: one expert is insufficient, but too many experts can reduce performance because router quality and data per expert become limiting factors (Li et al., 14 May 2025). The explicit MEPT paper likewise notes that expert count introduces a new hyperparameter and that mixture training over imbalanced tasks can under-train some experts (Zeng et al., 31 Aug 2025).

A third issue is router quality and computational placement. Loss-based routing in graphs requires forward passes through all experts to score them (Wang et al., 5 Nov 2025). Test-time adaptation schemes such as TAME require many augmentations and online updates at inference, which improves robustness but introduces deployment overhead (Wang et al., 17 May 2026). External domain routers, as in MoDEM, achieve 97% average in-distribution accuracy and 81.00% overall on MMLU subject-to-domain mapping, but also show that broad fallback categories such as “Other” are intrinsically harder to route (Simonds et al., 2024). This suggests that MEPT router design remains a primary systems bottleneck.

A final misconception is that MEPT is purely an efficiency technique. Adjacent work on Mixture-of-Tunable-Experts demonstrates that prompt-based profiling can identify behavior-specific experts and that suppressing or stimulating those experts at inference can sharply alter refusal behavior and reasoning language without retraining (Dahlke et al., 16 Feb 2025). A plausible implication is that MEPT is also a steerability mechanism. That broader capability raises safety questions: prompt-expert routing can improve specialization and robustness, but it can also be used to weaken refusal behavior or otherwise alter alignment.

Open directions recur across the surveyed systems. These include hierarchical expert structures, top-$P = \sum_{i=1}^{N} w_i A_i B,$0 or otherwise sparser routing, fully federated or unlabeled gate pretraining, integration with other PEFT modules such as LoRA or adapters, scaling to larger expert counts, and extensions from classification or retrieval settings to harder generative tasks (Dun et al., 2023, Wang et al., 5 Nov 2025, Li et al., 14 May 2025). Taken together, the literature indicates that MEPT is evolving from a prompt-efficiency trick into a general framework for conditional adaptation, modular transfer, and controllable routing in frozen foundation models.