---
title: 'PerM-MoE: Parameter-Efficient Mixture of Experts'
url: https://www.emergentmind.com/topics/perm-moe
type: topic
---

# PerM-MoE: Parameter-Efficient Mixture of Experts

PerM-MoE denotes two distinct Mixture-of-Experts constructions in recent arXiv literature. In instruction tuning, it refers to an “extremely parameter-efficient Mixture of Experts” recipe that replaces conventional MoE experts with lightweight parameter-efficient fine-tuning modules, specifically \((\mathrm{IA})^3\) vectors or LoRA adapters, yielding the variants Mixture of Vectors (MoV) and Mixture of LoRA (MoLORA) on frozen T5 backbones [2309.05444]. In multimodal neuroimaging, it refers to a sparse multimodal MoE for Alzheimer’s disease progression prediction that replaces a single shared router with independent routers for each modality in order to improve robustness under severe modality missingness [2509.12234]. The shared label therefore identifies a research theme centered on MoE under constraints, but the concrete architecture, objective, and domain depend on the specific paper.

## 1. Scope of the term

In the cited literature, “PerM-MoE” is not a single standardized architecture. It is used for both a PEFT-based MoE recipe for large language model instruction tuning and a per-modality routed sparse MoE for multimodal neuroimaging. The common element is that both works alter the expert-routing interface rather than simply scaling conventional MoE experts.

| Usage | Core mechanism | Evaluation setting |
|---|---|---|
| Instruction tuning PerM-MoE | Replace full MoE experts with \((\mathrm{IA})^3\) or LoRA experts; freeze dense backbone; train router plus lightweight experts | P3 instruction tuning with held-out zero-shot tasks on T5 v1.1+LM adaptation models [2309.05444] |
| Neuroimaging PerM-MoE | Replace one shared router with one router per modality in a sparse multimodal MoE | ADNI two-year CDR-SB change prediction with T1 MRI, FLAIR, amyloid beta PET, and tau PET [2509.12234] |

A common misconception is that PerM-MoE always refers to PEFT for language models. The neuroimaging paper uses the same label for a different architectural intervention: modality-specific routing rather than lightweight PEFT experts. Conversely, the 2023 instruction-tuning work is not a standard sparse MoE with full feed-forward experts; its defining move is to make the experts themselves tiny.

## 2. Parameter-efficient PerM-MoE for instruction tuning

In the 2023 formulation, standard MoE is written as a router \(R\) and experts \(E_1,\dots,E_n\). For an input representation \(x\), the router produces
\[
s_i = R(x)_i = \mathrm{softmax}(W_g^T x),
\]
with \(W_g \in \mathbb{R}^{d_m \times n}\), and the MoE output is
\[
y = \sum_{i=1}^n s_i \cdot E_i(x).
\]
PerM-MoE changes the meaning of \(E_i\): each expert is no longer a full feed-forward subnetwork but a PEFT module, while the pretrained Transformer backbone remains frozen and only the router and expert parameters are updated during fine-tuning [2309.05444].

The paper instantiates this idea in two forms. In **Mixture of Vectors (MoV)**, each expert is an \((\mathrm{IA})^3\) module, where the learned parameters are multiplicative scaling vectors applied to key, value, and feed-forward activations:
\[
\mathrm{softmax}\left(\frac{Q(l_k \odot K^T)}{\sqrt{d_k}}\right)(l_v \odot V); \quad (l_{ff} \odot \gamma(W_1x))W_2.
\]
In **Mixture of LoRA (MoLORA)**, each expert is a low-rank LoRA update:
\[
h = W_0 + \Delta W_x = W_0 + BAx,
\]
with \(A \in \mathbb{R}^{r \times d_m}\) and \(B \in \mathbb{R}^{d_p \times r}\). The defining point is that expert multiplicity is added without unfreezing the dense backbone.

The training setup follows the T0/T5 instruction-tuning protocol on the **P3** multitask prompt collection. Training is performed on a subset of tasks and evaluated zero-shot on held-out tasks. The base models are **T5 v1.1+LM adaptation** models at **770M**, **3B**, and **11B** parameters. The optimization uses **Adafactor** with learning rate \(3\times 10^{-4}\), sequence length **1024** for inputs and **256** for targets, batch size **32**, and **500K** fine-tuning steps for the PEFT MoE methods. Evaluation is on **ANLI, HellaSwag, WinoGrande, and 5 SuperGLUE tasks**, with performance reported as median accuracy across prompt templates and then averaged across tasks.

## 3. Soft merging, routing, and empirical behavior in the PEFT formulation

A central implementation detail of the 2023 method is **soft merging** rather than discrete sparse top-\(k\) routing. Because both \((\mathrm{IA})^3\) and LoRA are linear transformations, the method forms a weighted average of the experts before applying the combined adapter:
\[
E_{mix} = \sum_{i=1}^n s_i \cdot E_i; \qquad y = E_{mix}(x).
\]
The paper’s JAX-style pseudocode expresses this as router probabilities over experts, an einsum-based weighted combination of expert parameters, and a single application of the merged adapter to the input. This avoids a separate full forward pass per expert while still allowing multiple experts to contribute [2309.05444].

The reported parameter budgets are the basis for the paper’s “extremely parameter-efficient” characterization. On **T5-3B**, single-expert \((\mathrm{IA})^3\) updates **0.018%** of the model and LoRA with rank 4 updates **0.3%**. The mixture variants remain small despite adding experts: **MoV-10 updates 0.32%**, **MoV-30 updates 0.68%**, and **MoV-60 updates 1.22%**. On **T5-11B**, **MoV-10 updates 0.143%** and **MoV-60 updates 0.862%**.

Empirically, mixtures of PEFT experts outperform the corresponding single PEFT baselines and approach full fine-tuning. On **T5-3B**, \((\mathrm{IA})^3\) attains **52.90** average median accuracy, while **MoV-30** reaches **60.61**, described as a **14.57%** improvement over \((\mathrm{IA})^3\). **MoV-10** reaches **59.93**, essentially matching the replicated **T0-3B** baseline at **60.06**, while using only **0.32%** of parameters. On **T5-11B**, **MoV-60** reaches **64.08**, compared with the \((\mathrm{IA})^3\) baseline at **59.12** and the replicated **T0-11B** baseline around **65.03**, while updating only **0.862%** of parameters. The paper’s stated interpretation is that increasing base model size makes the mixture-of-PEFT-experts approach increasingly competitive with full fine-tuning.

The routing ablations are equally important. On **MoV-10 at 3B**, soft merging yields **59.93**, while discrete routing falls to **57.45** for top-2 and **54.92** for top-1. Load balancing, often useful in standard MoE, is reported to hurt in this regime. The paper attributes this to the extreme parameter limitation: using all available lightweight expert capacity via soft merging is more effective than sparsifying to a few experts. It also reports that token-based routing outperforms sentence-embedding routing at all model sizes, with especially large gains at **3B**, which supports the claim that prior task identity is unnecessary and that the router can infer useful specialization from hidden token representations alone.

The expert analysis further indicates that different tasks activate different experts with different magnitudes and rank orderings, including on unseen tasks. This suggests transferable specialization rather than simple task memorization. The paper also reports hyperparameter sensitivity: larger batch sizes can cause collapse to a single expert, and smaller learning rates such as \(3\times 10^{-4}\) help stabilize expert diversity.

## 4. Per-modality PerM-MoE for multimodal Alzheimer’s disease progression prediction

The 2025 neuroimaging work uses the same name for a different problem: robust multimodal prediction under missing modalities. The task is to predict the patient’s **two-year change in CDR-SB** from baseline neuroimaging and baseline CDR-SB, rather than binary conversion classification. The paper emphasizes that multimodal methods often assume full modality availability, but in the cited ADNI cohort only **5.9%** of subjects have all four modalities, while **60.8%** have exactly one modality available. This is the stated motivation for replacing a single router with one router per modality [2509.12234].

PerM-MoE is built on **Flex-MoE** and retains its main components: modality-specific CNN encoders, a missing modality bank, a self-attention layer over modality embeddings, a sparse MoE layer, an MLP prediction head, and the expert specialization loss. The architectural novelty is the routing layer. Standard sparse MoE and Flex-MoE use one router for the joint input representation. PerM-MoE instead uses *per-modality routing*: each modality has its own router, and each router selects experts independently for that modality’s embedding.

The multimodal inputs are \( \mathbf{x}^{M} \) for **T1 MRI**, \( \mathbf{x}^{F} \) for **FLAIR**, \( \mathbf{x}^{A} \) for **amyloid beta PET**, \( \mathbf{x}^{T} \) for **tau PET**, together with baseline CDR-SB \( \mathbf{x}^{s} \). After self-attention, the embedding for modality \(i\) is routed through a per-modality sparse MoE whose output is expressed as a weighted sum of expert outputs,
\[
\mathbf{o}^i = \sum_{j=1}^E f_j(\mathbf{h}^i) w_{ij},
\]
where the paper states that \(g_i(\cdot)\) is the learned gating network for the \(i^{\text{th}}\) modality and routing is top-\(k\), so only the top \(k\) gating values are retained. The paper notes that the displayed equation in the PDF is visually corrupted in places, but the intended top-\(k\) sparse gating semantics are described explicitly.

Missing modalities are handled through the inherited **missing modality bank**. For available modalities \(p \in P\), the corresponding encoder outputs are used; for missing modalities \(m \in M\), the embedding is replaced by a learned vector \(\mathbf{B}^P_m\) conditioned on the available set \(P\). Those available or imputed embeddings pass through self-attention and then through their modality-specific routers. The resulting routed outputs are concatenated with baseline CDR-SB and passed to the final MLP head. The paper’s claim is that this decouples expert selection from modality interactions and is particularly useful when only one modality is present.

The training objective retains Flex-MoE’s expert specialization loss in addition to the conventional sparse-MoE balancing loss from Shazeer et al. The main prediction task is evaluated by **RMSE** on two-year CDR-SB change. The regression setup is therefore aligned with minimizing prediction error rather than a classification loss.

## 5. Dataset, preprocessing, and results in the neuroimaging formulation

The neuroimaging data come from **ADNI**. Subjects are formed from appointment pairs spaced two years apart, retaining only pairs with CDR-SB available at both time points, at least one imaging modality among T1 MRI, FLAIR, AB PET, and tau PET, and baseline CDR-SB above **1**. The earlier appointment supplies the inputs and baseline CDR-SB, and the two-year change in CDR-SB is the label. This yields **837 subjects from 469 participants**, split **70%** training, **15%** validation, and **15%** testing, with all pairs from a participant kept in the same split to avoid leakage [2509.12234].

The test set is further augmented to stress severe missingness by systematically withholding modalities. For example, a subject with T1 MRI and AB PET is duplicated into two additional test cases containing only T1 MRI or only AB PET. PET images are reported as already preprocessed by ADNI and resampled to \(160 \times 160 \times 96\) voxels at \(6\text{ mm}^3\) resolution. MRI is linearly registered to a standard template, skull-stripped, bias-corrected with N4, and z-score normalized, yielding \(193 \times 229 \times 193\) voxels at \(1\text{ mm}^3\). All MRI and PET images are then downsampled by **50%**. Results are averaged over **three runs** with different random initializations, and hyperparameters are tuned by grid search.

The central quantitative result is that PerM-MoE outperforms **Flex-MoE** in **11 of 15** modality combinations, including **all one-modality cases**. Representative one-modality RMSEs from Table 1 are: **T1 MRI only**, Flex-MoE **1.95** versus PerM-MoE **1.94**; **FLAIR only**, **2.53** versus **2.33**; **AB PET only**, **2.42** versus **2.32**; **tau PET only**, **2.56** versus **2.47**. The paper highlights percentage improvements over Flex-MoE for single-modality inputs of **8.6%** with only FLAIR, **4.3%** with only AB PET, and **3.6%** with only tau PET.

For two-modality inputs, PerM-MoE improves over Flex-MoE in **5 of 6** cases, with a **7.4%** average improvement. The stated exception is **AB PET + tau PET**, which the authors suggest may be an edge case where a single generalist router better captures interactions between those similar modalities. When three modalities are available, PerM-MoE remains competitive with a **2.4%** average improvement, and with all four modalities it still shows a **1.3%** average improvement. The paper’s strongest summary claim is that the largest gains occur exactly where missingness is most severe.

Performance is also broken down by the observed two-year \(\Delta\)CDR-SB. PerM-MoE is reported to be especially strong when \(0 < \Delta\text{CDR-SB} \le 1\), where it achieves a **21.5%** average RMSE improvement over Flex-MoE. It also improves by **6.1%** for negative \(\Delta\)CDR-SB. The expert-utilization analysis reports more balanced expert activation and more uniform routing across modality combinations than Flex-MoE, while preserving meaningful specialization: experts **6** and **11**, for example, still receive many inputs from their intended modality combinations. The stated interpretation is that better-balanced expert usage, without loss of specialization, explains the missing-modality gains.

## 6. Relation to broader MoE fine-tuning research

The 2023 instruction-tuning formulation of PerM-MoE sits within a broader line of work that treats MoE adaptation as a routing-aware PEFT problem rather than ordinary dense-model PEFT. **PERFT** introduces a unified framework for integrating PEFT modules directly into the MoE mechanism, organized by a functional strategy and a compositional strategy. Its main routed variant, **PERFT-R**, adds an independent router over PEFT experts on top of a frozen MoE backbone, while **PERFT-E** reuses the pretrained router and **PERFT-D** and **PERFT-S** remove routing. On **OLMoE-1B-7B** and **Mixtral-8×7B**, the paper reports that **PERFT-R** is generally the strongest variant, with up to **17.2% improvement** on commonsense and **12.3% improvement** on arithmetic relative to the MoE-agnostic baseline at equivalent activated trainable parameter levels [2411.08212].

**EPnG** extends the same broader agenda by making expert-wise adapter allocation dynamic. Rather than assigning equal LoRA rank to every expert, it uses router gate probabilities to estimate downstream expert importance, prunes under-utilized experts, and grows the rank of high-importance experts while maintaining a fixed parameter budget. On **OLMoE-1B-7B-0125** and **Qwen1.5-MoE-A2.7B**, EPnG updates only **0.72%** and **0.55%** of parameters, respectively, and is reported to outperform static LoRA under the same budget while achieving performance comparable to full fine-tuning [2607.01789].

Taken together, these works clarify the landscape around PerM-MoE. In the language-model setting, the core recurring idea is that MoE adaptation should respect routing structure and should not allocate adaptation capacity uniformly or densely. In the neuroimaging setting, the corresponding principle is that routing should reflect modality structure rather than forcing one shared gating policy across all modality combinations. A plausible implication is that “PerM-MoE” is best understood not as a single fixed architecture, but as a label attached to MoE variants that restructure routing and expert capacity to operate effectively under a severe constraint, whether that constraint is trainable-parameter budget or modality availability.

Source: https://www.emergentmind.com/topics/perm-moe