Task Mode: Mixture of Dyadic Experts

• Task Mode (MoDE) is a multi-task, parameter-efficient fine-tuning architecture that reformulates LoRA adaptation using dyadic decomposition.
• It employs a shared down-projection matrix combined with per-rank expert-specific up-projections, enabling finer routing and reduced redundancy.
• Empirical results on the Supernatural Instructions benchmark indicate that MoDE achieves higher ROUGE-L scores with improved expressivity and parameter efficiency.

Searching arXiv for the specified paper and closely related PEFT/MoE work to ground the article. MoDE, short for Mixture of Dyadic Experts, is a multi-task parameter-efficient fine-tuning architecture for LLMs that reformulates LoRA adaptation at the granularity of rank-one dyadic components and combines this with shared down-projections and per-rank routing (Ning et al., 2024). It is motivated by an empirical observation about multi-task LoRA-style mixtures: when independently trained LoRA modules are compared across tasks, the down-projection matrices appear highly similar while the up-projection matrices differ, suggesting that existing LoRA-MoE designs repeatedly learn redundant structure (Ning et al., 2024). On the Supernatural Instructions benchmark using 756 English tasks and a frozen Gemma-2B backbone, MoDE improves over single multitask LoRA and over prior LoRA-mixture baselines while preserving a parameter-efficient adaptation regime (Ning et al., 2024).

1. Redundancy in multi-task LoRA mixtures

Standard LoRA augments a base weight matrix $W_0$ with a low-rank update

$$\Delta W = A B^T,\quad A\in\mathbb{R}^{P\times r},\ B\in\mathbb{R}^{Q\times r},$$

so that a linear layer computes

$y = x W_0 + x A B^T.$

In multi-task and MoE-style parameter-efficient fine-tuning methods, multiple LoRA experts are typically instantiated,

$E^i = A^i {B^i}^T,\quad i=1,\dots,m,$

and mixed by a router $\mathcal{R}$,

$$y = x W_0 + \sum_{i=1}^m \mathcal{R}^i(x)\, x A^i {B^i}^T.$$

The central empirical result motivating MoDE is that the down-projection matrices $A^i$ learned across tasks are highly similar, whereas the up-projection matrices $B^i$ are task-specific (Ning et al., 2024).

This result was established by training 15 single-task LoRAs from a shared initialization on 15 diverse Supernatural Instructions tasks, slicing $A$ and $B$ along the rank dimension into column vectors, and applying PCA to those vectors. The down vectors clustered tightly by rank index across tasks, while the up vectors did not cluster. The authors interpret this as evidence that multi-task LoRA-MoE architectures redundantly relearn similar down-projection structure while the useful specialization resides primarily in the up-projection side (Ning et al., 2024).

A plausible implication is that the low-dimensional adapter subspace defined by $$\Delta W = A B^T,\quad A\in\mathbb{R}^{P\times r},\ B\in\mathbb{R}^{Q\times r},$$0 is largely task-agnostic, while task variability is better modeled as different ways of decoding or routing within that subspace. MoDE is built directly around that interpretation.

2. Dyadic decomposition and the MoDE architecture

A LoRA update can be written as a sum of dyadic rank-one terms,

$$\Delta W = A B^T,\quad A\in\mathbb{R}^{P\times r},\ B\in\mathbb{R}^{Q\times r},$$1

where $$\Delta W = A B^T,\quad A\in\mathbb{R}^{P\times r},\ B\in\mathbb{R}^{Q\times r},$$2 and $$\Delta W = A B^T,\quad A\in\mathbb{R}^{P\times r},\ B\in\mathbb{R}^{Q\times r},$$3 are the $$\Delta W = A B^T,\quad A\in\mathbb{R}^{P\times r},\ B\in\mathbb{R}^{Q\times r},$$4-th columns of $$\Delta W = A B^T,\quad A\in\mathbb{R}^{P\times r},\ B\in\mathbb{R}^{Q\times r},$$5 and $$\Delta W = A B^T,\quad A\in\mathbb{R}^{P\times r},\ B\in\mathbb{R}^{Q\times r},$$6. MoDE treats each dyadic product $$\Delta W = A B^T,\quad A\in\mathbb{R}^{P\times r},\ B\in\mathbb{R}^{Q\times r},$$7 as an atomic expert (Ning et al., 2024).

The design has three defining elements. First, the down-projection matrix $$\Delta W = A B^T,\quad A\in\mathbb{R}^{P\times r},\ B\in\mathbb{R}^{Q\times r},$$8 is shared across all experts. Second, specialization is shifted to the up-projection vectors $$\Delta W = A B^T,\quad A\in\mathbb{R}^{P\times r},\ B\in\mathbb{R}^{Q\times r},$$9, with $y = x W_0 + x A B^T.$0 candidate vectors per rank slot $y = x W_0 + x A B^T.$1. Third, routing is performed independently for each rank slot rather than over the full rank-$y = x W_0 + x A B^T.$2 adapter as a block (Ning et al., 2024).

For each rank index $y = x W_0 + x A B^T.$3, MoDE defines $y = x W_0 + x A B^T.$4 rank-one experts

$y = x W_0 + x A B^T.$5

The router has separate parameters for each rank slot,

$y = x W_0 + x A B^T.$6

and for token hidden state $y = x W_0 + x A B^T.$7,

$y = x W_0 + x A B^T.$8

The resulting layer output is

$y = x W_0 + x A B^T.$9

Operationally, each rank slot first projects the token onto the shared input direction $E^i = A^i {B^i}^T,\quad i=1,\dots,m,$0, then forms multiple candidate output directions via the expert-specific $E^i = A^i {B^i}^T,\quad i=1,\dots,m,$1, and finally mixes them with token-dependent router weights (Ning et al., 2024).

This factorization gives MoDE a much finer routing granularity than standard LoRA-MoE. The paper argues that with $E^i = A^i {B^i}^T,\quad i=1,\dots,m,$2 expert choices for each of $E^i = A^i {B^i}^T,\quad i=1,\dots,m,$3 rank slots, the number of effective compositions grows roughly as $E^i = A^i {B^i}^T,\quad i=1,\dots,m,$4, whereas conventional LoRA-MoE exposes only $E^i = A^i {B^i}^T,\quad i=1,\dots,m,$5 full-matrix choices at a layer (Ning et al., 2024). This suggests substantially higher combinatorial expressivity for comparable parameter budgets.

3. Generalized formulation and structural comparison

MoDE is embedded in a broader family parameterized as MoDE $E^i = A^i {B^i}^T,\quad i=1,\dots,m,$6, where $E^i = A^i {B^i}^T,\quad i=1,\dots,m,$7 is the number of expert choices per group, $E^i = A^i {B^i}^T,\quad i=1,\dots,m,$8 is the total LoRA rank, and $E^i = A^i {B^i}^T,\quad i=1,\dots,m,$9 is the rank per expert group (Ning et al., 2024). Assuming $\mathcal{R}$0 is divisible by $\mathcal{R}$1, the $\mathcal{R}$2 dyadic terms are grouped into $\mathcal{R}$3 groups: $\mathcal{R}$4 and the generalized output becomes

$\mathcal{R}$5

Rank-one experts correspond to the default form of MoDE; higher $\mathcal{R}$6 recovers coarser expert groupings (Ning et al., 2024).

Two special cases connect MoDE to earlier PEFT formulations. MoDE $\mathcal{R}$7 is equivalent to a single rank-$\mathcal{R}$8 LoRA, while MoDE $\mathcal{R}$9 is equivalent to a LoRA-MoE with shared down-projection, termed LoRA-MoE-SD in the paper (Ning et al., 2024). This places MoDE on a continuum between standard LoRA and blockwise shared-$$y = x W_0 + \sum_{i=1}^m \mathcal{R}^i(x)\, x A^i {B^i}^T.$$0 expert mixtures.

The structural distinction from standard LoRA, LoRA-MoE, and LoRA-MoE-SD is summarized below.

Method	Core structure	Parameters per layer
LoRA	One rank-$$y = x W_0 + \sum_{i=1}^m \mathcal{R}^i(x)\, x A^i {B^i}^T.$$1 adapter, no routing	$$y = x W_0 + \sum_{i=1}^m \mathcal{R}^i(x)\, x A^i {B^i}^T.$$2
LoRA-MoE	$$y = x W_0 + \sum_{i=1}^m \mathcal{R}^i(x)\, x A^i {B^i}^T.$$3 experts, each with its own $$y = x W_0 + \sum_{i=1}^m \mathcal{R}^i(x)\, x A^i {B^i}^T.$$4, single router	$$y = x W_0 + \sum_{i=1}^m \mathcal{R}^i(x)\, x A^i {B^i}^T.$$5
LoRA-MoE-SD	Shared $$y = x W_0 + \sum_{i=1}^m \mathcal{R}^i(x)\, x A^i {B^i}^T.$$6, expert-specific $$y = x W_0 + \sum_{i=1}^m \mathcal{R}^i(x)\, x A^i {B^i}^T.$$7, single router	$$y = x W_0 + \sum_{i=1}^m \mathcal{R}^i(x)\, x A^i {B^i}^T.$$8
MoDE	Shared $$y = x W_0 + \sum_{i=1}^m \mathcal{R}^i(x)\, x A^i {B^i}^T.$$9, per-rank expert-specific $A^i$0, per-rank routers	$A^i$1

The main architectural reallocation is therefore from redundant down-projection matrices into additional router capacity and finer expert specialization (Ning et al., 2024).

4. Training procedure and inference behavior

The experimental training setup uses a frozen Gemma-2B backbone and a single multitask corpus formed by mixing examples from the 756 English tasks in Supernatural Instructions (Ning et al., 2024). Tasks are not represented by explicit task IDs or task embeddings; instead, routing is conditioned only on token hidden states derived from the concatenated instruction and input. This means task-specific behavior is implicit, arising from how different instructions and inputs induce different router activations (Ning et al., 2024).

Only adapter and router parameters are optimized: the shared $A^i$2, the expert-specific up-vectors $A^i$3, and the router parameters $A^i$4. Optimization uses Adafactor with learning rate $A^i$5, sequence length 1024, batch size 128, and 20k training steps (Ning et al., 2024). The objective is standard next-token cross-entropy over the mixed multitask corpus,

$A^i$6

with no auxiliary routing loss, no load-balancing term, and no explicit sparsity penalty (Ning et al., 2024).

At inference time, MoDE uses dense soft routing. For each token and each MoDE-enabled layer, it computes $A^i$7 router projections $A^i$8, applies softmax over the $A^i$9 experts per rank slot, and forms a weighted mixture over all $B^i$0 rank-one experts. There is no sampling: routing is continuous and deterministic given the hidden state (Ning et al., 2024).

The paper notes that MoDE introduces more router computations than standard LoRA because it performs $B^i$1 router matmuls rather than one. However, it also notes that exact FLOP formulas are not derived and qualitatively argues that the dominant costs remain the linear projections for large $B^i$2 and $B^i$3, with router overhead relatively small (Ning et al., 2024). This suggests that MoDE’s main trade-off is architectural rather than one of dramatically increased compute.

5. Experimental setting and quantitative results

All reported main experiments use Supernatural Instructions. The full benchmark contains 1,616 instruction-following tasks over 76 task types, but the multitask experiments use the 756 English-only tasks from the default train split, with each task split into 90% training and 10% evaluation (Ning et al., 2024). The principal metric is ROUGE-L, which is the benchmark default (Ning et al., 2024).

The study compares MoDE against a single multitask LoRA, a LoRA-MoE baseline denoted MoLORA, and a shared-down-projection variant MoLORA-SD. The main 756-task results are:

Model	ROUGE-L	Added params (% of Gemma-2B)
LoRA rank 64	56.11	6.31%
MoLORA 16×4	57.77	7.62%
MoLORA-SD 16×4	58.28	2.71%
MoDE 16×4	60.00	6.64%
MoDE 8×4	59.00	3.48%
MoDE 6×4	60.91	2.69%
MoDE 4×4	60.18	1.90%
MoDE 4×6	60.53	2.86%
MoDE 4×8	58.92	3.81%
MoDE 4×16	60.04	7.62%

Several comparisons are especially central to the paper’s argument. Sharing the down-projection in MoLORA-SD reduces added parameters from 7.62% to 2.71% while improving ROUGE-L from 57.77 to 58.28, which supports the redundancy hypothesis directly (Ning et al., 2024). Relative to the single multitask LoRA baseline, MoDE 16×4 improves from 56.11 to 60.00 at a similar parameter budget, a gain of +3.89 ROUGE-L (Ning et al., 2024). Relative to MoLORA 16×4, MoDE 16×4 improves by +2.23 ROUGE-L at essentially the same effective expert-rank setting (Ning et al., 2024).

The paper also reports broad task-level advantages. Across the 756 tasks, MoDE wins on 78% of tasks versus single LoRA, 73% versus MoLORA, and 68% versus MoLORA-SD, with all win rates statistically significant versus 50% at 0.99 confidence (Ning et al., 2024). This is presented as evidence that MoDE’s gains are not confined to a narrow subset of tasks.

6. Ablations, generalized MoDE, and interpretation

Ablations over the number of experts $B^i$4 at fixed rank $B^i$5 show a non-monotonic trend. MoDE 4×4 achieves 60.18 ROUGE-L, 6×4 achieves 60.91, 8×4 drops to 59.00, and 16×4 yields 60.00 (Ning et al., 2024). The paper interprets this as diminishing returns from adding too many experts, plausibly because router capacity and dataset size limit useful specialization.

Varying rank at fixed $B^i$6 also yields non-monotonic behavior: 4×4 gives 60.18, 4×6 gives 60.53, 4×8 drops to 58.92, and 4×16 gives 60.04 (Ning et al., 2024). This suggests that modest rank increases can help, but larger ranks may introduce overparameterization or optimization difficulty when not matched by corresponding expert diversity.

The generalized rank-$B^i$7 formulation further clarifies the trade-off between fine-grained routing and parameter cost. At fixed $B^i$8, decreasing $B^i$9 toward rank-one experts improves performance but also increases parameters. For $A$0, performance rises from 58.51 at $A$1 to 59.93 at $A$2; for $A$3, it rises from 58.97 at $A$4 to 59.91 at $A$5 (Ning et al., 2024). The paper interprets this as evidence that finer decomposition increases expressivity.

Under an iso-parametric budget of roughly 6.5% added parameters, the best generalized configuration reported is 12×16×8, reaching 60.94 ROUGE-L (Ning et al., 2024). This indicates that the most parameter-efficient design within a fixed budget is not always the finest rank-one decomposition. A plausible implication is that MoDE defines a three-way trade-off among total LoRA rank, number of expert choices, and expert granularity, rather than a universally dominant operating point.

A separate 15-task fixed-budget study reinforces the multitask advantage. Under roughly 6M adapter parameters, a single multitask MoDE model achieves around 63.37–63.39 ROUGE-L, outperforming 15 separate single-task LoRAs, a single multitask LoRA, MoLORA, and MoLORA-SD (Ning et al., 2024). This result suggests that MoDE’s shared representation and routing scheme can outperform even an oracle-like collection of small task-specific adapters under a matched budget.

7. Conceptual significance, limitations, and position in the literature

MoDE’s main conceptual contribution is to recast LoRA as a sum of rank-one dyadic updates and to treat those atomic terms as the natural granularity for mixture-of-experts routing (Ning et al., 2024). This yields a separation between a shared adapter feature space, represented by the common down-projection $A$6, and task-dependent decoding behavior, represented by routed up-projection vectors $A$7. The design is tightly aligned with the empirical structure observed in independently trained single-task LoRAs.

The paper is primarily empirical and does not present formal theorems. Its expressivity claims are therefore conceptual rather than deductive: standard LoRA-MoE offers $A$8 alternative low-rank updates, whereas MoDE offers roughly $A$9 possible compositions from $B$0 choices per rank slot (Ning et al., 2024). This suggests a large increase in representable effective updates for similar parameter counts, though the paper does not formalize this as a complexity theorem.

Several limitations are explicit. The routing strategy is simple: token-level softmax over a linear transformation of the hidden state, with no load balancing, no sparsity, and no top-$B$1 selection (Ning et al., 2024). Hyperparameter sensitivity remains only partially mapped, especially over the joint space of $B$2, $B$3, and $B$4 (Ning et al., 2024). The evaluation focuses on Supernatural Instructions, so broader claims about multilingual, code, or other multi-domain settings remain open (Ning et al., 2024). The added per-rank router computation may also matter in low-latency deployments, although the paper does not quantify this precisely (Ning et al., 2024).

Within the broader PEFT landscape, MoDE is positioned as a mixture-of-adapters method rather than a prompt-based or prefix-based scheme. It is directly descended from LoRA and most closely related to LoRA-MoE variants such as MoLORA, MixLoRA, SiRA, and related multi-task adapter-composition approaches listed in the paper’s positioning discussion (Ning et al., 2024). Its distinguishing features are the empirical identification of down-projection redundancy, shared $B$5, and dyadic-level routing. In that sense, MoDE proposes not merely a more compact mixture, but a different view of where multitask specialization should reside in low-rank adaptation.