PerM-MoE: Parameter-Efficient Mixture of Experts

• PerM-MoE is a term for routing modifications in MoE architectures, improving parameter efficiency in both language and neuroimaging tasks.
• In instruction tuning, it replaces full MoE experts with lightweight PEFT modules like (IA)³ and LoRA, yielding up to a 14.57% accuracy boost on T5 models.
• In neuroimaging, PerM-MoE uses modality-specific routers to enhance Alzheimer’s disease progression prediction, especially under severe modality missingness.

PerM-MoE denotes two distinct Mixture-of-Experts constructions in recent arXiv literature. In instruction tuning, it refers to an “extremely parameter-efficient Mixture of Experts” recipe that replaces conventional MoE experts with lightweight parameter-efficient fine-tuning modules, specifically $(\mathrm{IA})^3$ vectors or LoRA adapters, yielding the variants Mixture of Vectors (MoV) and Mixture of LoRA (MoLORA) on frozen T5 backbones (Zadouri et al., 2023). In multimodal neuroimaging, it refers to a sparse multimodal MoE for Alzheimer’s disease progression prediction that replaces a single shared router with independent routers for each modality in order to improve robustness under severe modality missingness (Burns et al., 8 Sep 2025). The shared label therefore identifies a research theme centered on MoE under constraints, but the concrete architecture, objective, and domain depend on the specific paper.

1. Scope of the term

In the cited literature, “PerM-MoE” is not a single standardized architecture. It is used for both a PEFT-based MoE recipe for LLM instruction tuning and a per-modality routed sparse MoE for multimodal neuroimaging. The common element is that both works alter the expert-routing interface rather than simply scaling conventional MoE experts.

Usage	Core mechanism	Evaluation setting
Instruction tuning PerM-MoE	Replace full MoE experts with $(\mathrm{IA})^3$ or LoRA experts; freeze dense backbone; train router plus lightweight experts	P3 instruction tuning with held-out zero-shot tasks on T5 v1.1+LM adaptation models (Zadouri et al., 2023)
Neuroimaging PerM-MoE	Replace one shared router with one router per modality in a sparse multimodal MoE	ADNI two-year CDR-SB change prediction with T1 MRI, FLAIR, amyloid beta PET, and tau PET (Burns et al., 8 Sep 2025)

A common misconception is that PerM-MoE always refers to PEFT for LLMs. The neuroimaging paper uses the same label for a different architectural intervention: modality-specific routing rather than lightweight PEFT experts. Conversely, the 2023 instruction-tuning work is not a standard sparse MoE with full feed-forward experts; its defining move is to make the experts themselves tiny.

2. Parameter-efficient PerM-MoE for instruction tuning

In the 2023 formulation, standard MoE is written as a router $R$ and experts $E_1,\dots,E_n$. For an input representation $x$, the router produces

$s_i = R(x)_i = \mathrm{softmax}(W_g^T x),$

with $W_g \in \mathbb{R}^{d_m \times n}$, and the MoE output is

$y = \sum_{i=1}^n s_i \cdot E_i(x).$

PerM-MoE changes the meaning of $E_i$: each expert is no longer a full feed-forward subnetwork but a PEFT module, while the pretrained Transformer backbone remains frozen and only the router and expert parameters are updated during fine-tuning (Zadouri et al., 2023).

The paper instantiates this idea in two forms. In Mixture of Vectors (MoV), each expert is an $(\mathrm{IA})^3$ module, where the learned parameters are multiplicative scaling vectors applied to key, value, and feed-forward activations: $(\mathrm{IA})^3$0 In Mixture of LoRA (MoLORA), each expert is a low-rank LoRA update: $(\mathrm{IA})^3$1 with $(\mathrm{IA})^3$2 and $(\mathrm{IA})^3$3. The defining point is that expert multiplicity is added without unfreezing the dense backbone.

The training setup follows the T0/T5 instruction-tuning protocol on the P3 multitask prompt collection. Training is performed on a subset of tasks and evaluated zero-shot on held-out tasks. The base models are T5 v1.1+LM adaptation models at 770M, 3B, and 11B parameters. The optimization uses Adafactor with learning rate $(\mathrm{IA})^3$4, sequence length 1024 for inputs and 256 for targets, batch size 32, and 500K fine-tuning steps for the PEFT MoE methods. Evaluation is on ANLI, HellaSwag, WinoGrande, and 5 SuperGLUE tasks, with performance reported as median accuracy across prompt templates and then averaged across tasks.

3. Soft merging, routing, and empirical behavior in the PEFT formulation

A central implementation detail of the 2023 method is soft merging rather than discrete sparse top-$(\mathrm{IA})^3$5 routing. Because both $(\mathrm{IA})^3$6 and LoRA are linear transformations, the method forms a weighted average of the experts before applying the combined adapter: $(\mathrm{IA})^3$7 The paper’s JAX-style pseudocode expresses this as router probabilities over experts, an einsum-based weighted combination of expert parameters, and a single application of the merged adapter to the input. This avoids a separate full forward pass per expert while still allowing multiple experts to contribute (Zadouri et al., 2023).

The reported parameter budgets are the basis for the paper’s “extremely parameter-efficient” characterization. On T5-3B, single-expert $(\mathrm{IA})^3$8 updates 0.018% of the model and LoRA with rank 4 updates 0.3%. The mixture variants remain small despite adding experts: MoV-10 updates 0.32%, MoV-30 updates 0.68%, and MoV-60 updates 1.22%. On T5-11B, MoV-10 updates 0.143% and MoV-60 updates 0.862%.

Empirically, mixtures of PEFT experts outperform the corresponding single PEFT baselines and approach full fine-tuning. On T5-3B, $(\mathrm{IA})^3$9 attains 52.90 average median accuracy, while MoV-30 reaches 60.61, described as a 14.57% improvement over $R$0. MoV-10 reaches 59.93, essentially matching the replicated T0-3B baseline at 60.06, while using only 0.32% of parameters. On T5-11B, MoV-60 reaches 64.08, compared with the $R$1 baseline at 59.12 and the replicated T0-11B baseline around 65.03, while updating only 0.862% of parameters. The paper’s stated interpretation is that increasing base model size makes the mixture-of-PEFT-experts approach increasingly competitive with full fine-tuning.

The routing ablations are equally important. On MoV-10 at 3B, soft merging yields 59.93, while discrete routing falls to 57.45 for top-2 and 54.92 for top-1. Load balancing, often useful in standard MoE, is reported to hurt in this regime. The paper attributes this to the extreme parameter limitation: using all available lightweight expert capacity via soft merging is more effective than sparsifying to a few experts. It also reports that token-based routing outperforms sentence-embedding routing at all model sizes, with especially large gains at 3B, which supports the claim that prior task identity is unnecessary and that the router can infer useful specialization from hidden token representations alone.

The expert analysis further indicates that different tasks activate different experts with different magnitudes and rank orderings, including on unseen tasks. This suggests transferable specialization rather than simple task memorization. The paper also reports hyperparameter sensitivity: larger batch sizes can cause collapse to a single expert, and smaller learning rates such as $R$2 help stabilize expert diversity.

4. Per-modality PerM-MoE for multimodal Alzheimer’s disease progression prediction

The 2025 neuroimaging work uses the same name for a different problem: robust multimodal prediction under missing modalities. The task is to predict the patient’s two-year change in CDR-SB from baseline neuroimaging and baseline CDR-SB, rather than binary conversion classification. The paper emphasizes that multimodal methods often assume full modality availability, but in the cited ADNI cohort only 5.9% of subjects have all four modalities, while 60.8% have exactly one modality available. This is the stated motivation for replacing a single router with one router per modality (Burns et al., 8 Sep 2025).

PerM-MoE is built on Flex-MoE and retains its main components: modality-specific CNN encoders, a missing modality bank, a self-attention layer over modality embeddings, a sparse MoE layer, an MLP prediction head, and the expert specialization loss. The architectural novelty is the routing layer. Standard sparse MoE and Flex-MoE use one router for the joint input representation. PerM-MoE instead uses per-modality routing: each modality has its own router, and each router selects experts independently for that modality’s embedding.

The multimodal inputs are $R$3 for T1 MRI, $R$4 for FLAIR, $R$5 for amyloid beta PET, $R$6 for tau PET, together with baseline CDR-SB $R$7. After self-attention, the embedding for modality $R$8 is routed through a per-modality sparse MoE whose output is expressed as a weighted sum of expert outputs,

$R$9

where the paper states that $E_1,\dots,E_n$0 is the learned gating network for the $E_1,\dots,E_n$1 modality and routing is top-$E_1,\dots,E_n$2, so only the top $E_1,\dots,E_n$3 gating values are retained. The paper notes that the displayed equation in the PDF is visually corrupted in places, but the intended top-$E_1,\dots,E_n$4 sparse gating semantics are described explicitly.

Missing modalities are handled through the inherited missing modality bank. For available modalities $E_1,\dots,E_n$5, the corresponding encoder outputs are used; for missing modalities $E_1,\dots,E_n$6, the embedding is replaced by a learned vector $E_1,\dots,E_n$7 conditioned on the available set $E_1,\dots,E_n$8. Those available or imputed embeddings pass through self-attention and then through their modality-specific routers. The resulting routed outputs are concatenated with baseline CDR-SB and passed to the final MLP head. The paper’s claim is that this decouples expert selection from modality interactions and is particularly useful when only one modality is present.

The training objective retains Flex-MoE’s expert specialization loss in addition to the conventional sparse-MoE balancing loss from Shazeer et al. The main prediction task is evaluated by RMSE on two-year CDR-SB change. The regression setup is therefore aligned with minimizing prediction error rather than a classification loss.

5. Dataset, preprocessing, and results in the neuroimaging formulation

The neuroimaging data come from ADNI. Subjects are formed from appointment pairs spaced two years apart, retaining only pairs with CDR-SB available at both time points, at least one imaging modality among T1 MRI, FLAIR, AB PET, and tau PET, and baseline CDR-SB above 1. The earlier appointment supplies the inputs and baseline CDR-SB, and the two-year change in CDR-SB is the label. This yields 837 subjects from 469 participants, split 70% training, 15% validation, and 15% testing, with all pairs from a participant kept in the same split to avoid leakage (Burns et al., 8 Sep 2025).

The test set is further augmented to stress severe missingness by systematically withholding modalities. For example, a subject with T1 MRI and AB PET is duplicated into two additional test cases containing only T1 MRI or only AB PET. PET images are reported as already preprocessed by ADNI and resampled to $E_1,\dots,E_n$9 voxels at $x$0 resolution. MRI is linearly registered to a standard template, skull-stripped, bias-corrected with N4, and z-score normalized, yielding $x$1 voxels at $x$2. All MRI and PET images are then downsampled by 50%. Results are averaged over three runs with different random initializations, and hyperparameters are tuned by grid search.

The central quantitative result is that PerM-MoE outperforms Flex-MoE in 11 of 15 modality combinations, including all one-modality cases. Representative one-modality RMSEs from Table 1 are: T1 MRI only, Flex-MoE 1.95 versus PerM-MoE 1.94; FLAIR only, 2.53 versus 2.33; AB PET only, 2.42 versus 2.32; tau PET only, 2.56 versus 2.47. The paper highlights percentage improvements over Flex-MoE for single-modality inputs of 8.6% with only FLAIR, 4.3% with only AB PET, and 3.6% with only tau PET.

For two-modality inputs, PerM-MoE improves over Flex-MoE in 5 of 6 cases, with a 7.4% average improvement. The stated exception is AB PET + tau PET, which the authors suggest may be an edge case where a single generalist router better captures interactions between those similar modalities. When three modalities are available, PerM-MoE remains competitive with a 2.4% average improvement, and with all four modalities it still shows a 1.3% average improvement. The paper’s strongest summary claim is that the largest gains occur exactly where missingness is most severe.

Performance is also broken down by the observed two-year $x$3CDR-SB. PerM-MoE is reported to be especially strong when $x$4, where it achieves a 21.5% average RMSE improvement over Flex-MoE. It also improves by 6.1% for negative $x$5CDR-SB. The expert-utilization analysis reports more balanced expert activation and more uniform routing across modality combinations than Flex-MoE, while preserving meaningful specialization: experts 6 and 11, for example, still receive many inputs from their intended modality combinations. The stated interpretation is that better-balanced expert usage, without loss of specialization, explains the missing-modality gains.

6. Relation to broader MoE fine-tuning research

The 2023 instruction-tuning formulation of PerM-MoE sits within a broader line of work that treats MoE adaptation as a routing-aware PEFT problem rather than ordinary dense-model PEFT. PERFT introduces a unified framework for integrating PEFT modules directly into the MoE mechanism, organized by a functional strategy and a compositional strategy. Its main routed variant, PERFT-R, adds an independent router over PEFT experts on top of a frozen MoE backbone, while PERFT-E reuses the pretrained router and PERFT-D and PERFT-S remove routing. On OLMoE-1B-7B and Mixtral-8×7B, the paper reports that PERFT-R is generally the strongest variant, with up to 17.2% improvement on commonsense and 12.3% improvement on arithmetic relative to the MoE-agnostic baseline at equivalent activated trainable parameter levels (Liu et al., 2024).

EPnG extends the same broader agenda by making expert-wise adapter allocation dynamic. Rather than assigning equal LoRA rank to every expert, it uses router gate probabilities to estimate downstream expert importance, prunes under-utilized experts, and grows the rank of high-importance experts while maintaining a fixed parameter budget. On OLMoE-1B-7B-0125 and Qwen1.5-MoE-A2.7B, EPnG updates only 0.72% and 0.55% of parameters, respectively, and is reported to outperform static LoRA under the same budget while achieving performance comparable to full fine-tuning (Lee et al., 2 Jul 2026).

Taken together, these works clarify the landscape around PerM-MoE. In the language-model setting, the core recurring idea is that MoE adaptation should respect routing structure and should not allocate adaptation capacity uniformly or densely. In the neuroimaging setting, the corresponding principle is that routing should reflect modality structure rather than forcing one shared gating policy across all modality combinations. A plausible implication is that “PerM-MoE” is best understood not as a single fixed architecture, but as a label attached to MoE variants that restructure routing and expert capacity to operate effectively under a severe constraint, whether that constraint is trainable-parameter budget or modality availability.