--- title: 'Task Mode: Mixture of Dyadic Experts' url: https://www.emergentmind.com/topics/task-mode type: topic --- # Task Mode: Mixture of Dyadic Experts Searching arXiv for the specified paper and closely related PEFT/MoE work to ground the article. MoDE, short for **Mixture of Dyadic Experts**, is a multi-task parameter-efficient fine-tuning architecture for large language models that reformulates LoRA adaptation at the granularity of rank-one dyadic components and combines this with shared down-projections and per-rank routing [2408.01505]. It is motivated by an empirical observation about multi-task LoRA-style mixtures: when independently trained LoRA modules are compared across tasks, the down-projection matrices appear highly similar while the up-projection matrices differ, suggesting that existing LoRA-MoE designs repeatedly learn redundant structure [2408.01505]. On the Supernatural Instructions benchmark using 756 English tasks and a frozen Gemma-2B backbone, MoDE improves over single multitask LoRA and over prior LoRA-mixture baselines while preserving a parameter-efficient adaptation regime [2408.01505]. ## 1. Redundancy in multi-task LoRA mixtures Standard LoRA augments a base weight matrix \(W_0\) with a low-rank update \[ \Delta W = A B^T,\quad A\in\mathbb{R}^{P\times r},\ B\in\mathbb{R}^{Q\times r}, \] so that a linear layer computes \[ y = x W_0 + x A B^T. \] In multi-task and MoE-style parameter-efficient fine-tuning methods, multiple LoRA experts are typically instantiated, \[ E^i = A^i {B^i}^T,\quad i=1,\dots,m, \] and mixed by a router \(\mathcal{R}\), \[ y = x W_0 + \sum_{i=1}^m \mathcal{R}^i(x)\, x A^i {B^i}^T. \] The central empirical result motivating MoDE is that the down-projection matrices \(A^i\) learned across tasks are highly similar, whereas the up-projection matrices \(B^i\) are task-specific [2408.01505]. This result was established by training 15 single-task LoRAs from a shared initialization on 15 diverse Supernatural Instructions tasks, slicing \(A\) and \(B\) along the rank dimension into column vectors, and applying PCA to those vectors. The down vectors clustered tightly by rank index across tasks, while the up vectors did not cluster. The authors interpret this as evidence that multi-task LoRA-MoE architectures redundantly relearn similar down-projection structure while the useful specialization resides primarily in the up-projection side [2408.01505]. A plausible implication is that the low-dimensional adapter subspace defined by \(A\) is largely task-agnostic, while task variability is better modeled as different ways of decoding or routing within that subspace. MoDE is built directly around that interpretation. ## 2. Dyadic decomposition and the MoDE architecture A LoRA update can be written as a sum of dyadic rank-one terms, \[ \Delta W = A B^T = [a_1,\dots,a_r][b_1,\dots,b_r]^T = \sum_{j=1}^r (a_j \otimes b_j), \] where \(a_j\in\mathbb{R}^P\) and \(b_j\in\mathbb{R}^Q\) are the \(j\)-th columns of \(A\) and \(B\). MoDE treats each dyadic product \(a_j \otimes b_j\) as an atomic expert [2408.01505]. The design has three defining elements. First, the down-projection matrix \(A\) is shared across all experts. Second, specialization is shifted to the up-projection vectors \(b_j^i\), with \(m\) candidate vectors per rank slot \(j\). Third, routing is performed independently for each rank slot rather than over the full rank-\(r\) adapter as a block [2408.01505]. For each rank index \(j\in\{1,\dots,r\}\), MoDE defines \(m\) rank-one experts \[ a_j \otimes {b_j^i}^T,\quad i=1,\dots,m. \] The router has separate parameters for each rank slot, \[ \mathbf{W}_{\mathcal{R}}\in\mathbb{R}^{r\times P\times m},\qquad \mathbf{W}_{\mathcal{R};j}\in\mathbb{R}^{P\times m}, \] and for token hidden state \(x\in\mathbb{R}^{1\times P}\), \[ \mathcal{R}_j(x)=\text{softmax}(x\,\mathbf{W}_{\mathcal{R};j})\in\mathbb{R}^{1\times m}. \] The resulting layer output is \[ y = x W_0 + \sum_{i=1}^m \sum_{j=1}^r \mathcal{R}_j^i(x)\,\big(x(a_j \otimes {b_j^i}^T)\big). \] Operationally, each rank slot first projects the token onto the shared input direction \(a_j\), then forms multiple candidate output directions via the expert-specific \(b_j^i\), and finally mixes them with token-dependent router weights [2408.01505]. This factorization gives MoDE a much finer routing granularity than standard LoRA-MoE. The paper argues that with \(m\) expert choices for each of \(r\) rank slots, the number of effective compositions grows roughly as \(m^r\), whereas conventional LoRA-MoE exposes only \(m\) full-matrix choices at a layer [2408.01505]. This suggests substantially higher combinatorial expressivity for comparable parameter budgets. ## 3. Generalized formulation and structural comparison MoDE is embedded in a broader family parameterized as **MoDE \(m\times r\times p\)**, where \(m\) is the number of expert choices per group, \(r\) is the total LoRA rank, and \(p\) is the rank per expert group [2408.01505]. Assuming \(r\) is divisible by \(p\), the \(r\) dyadic terms are grouped into \(r/p\) groups: \[ \mathbf{A}_k \mathbf{B}_k^{iT} = \sum_{j=1}^p \big(a_{j + p(k-1)}\otimes b_{j + p(k-1)}^{iT}\big), \] and the generalized output becomes \[ y = x W_0 + \sum_{i=1}^m \sum_{k=1}^{r/p} \mathcal{R}_k^i(x)\, x \mathbf{A}_k \mathbf{B}_k^{iT}. \] Rank-one experts correspond to the default form of MoDE; higher \(p\) recovers coarser expert groupings [2408.01505]. Two special cases connect MoDE to earlier PEFT formulations. MoDE \(1\times r\times r\) is equivalent to a single rank-\(r\) LoRA, while MoDE \(m\times r\times r\) is equivalent to a LoRA-MoE with shared down-projection, termed LoRA-MoE-SD in the paper [2408.01505]. This places MoDE on a continuum between standard LoRA and blockwise shared-\(A\) expert mixtures. The structural distinction from standard LoRA, LoRA-MoE, and LoRA-MoE-SD is summarized below. | Method | Core structure | Parameters per layer | |---|---|---| | LoRA | One rank-\(r\) adapter, no routing | \(Pr + Qr\) | | LoRA-MoE | \(m\) experts, each with its own \(A^i,B^i\), single router | \(m(Pr + Qr) + Pm\) | | LoRA-MoE-SD | Shared \(A\), expert-specific \(B^i\), single router | \(Pr + mQr + Pm\) | | MoDE | Shared \(A\), per-rank expert-specific \(b_j^i\), per-rank routers | \(Pr + mQr + rPm\) | The main architectural reallocation is therefore from redundant down-projection matrices into additional router capacity and finer expert specialization [2408.01505]. ## 4. Training procedure and inference behavior The experimental training setup uses a frozen **Gemma-2B** backbone and a single multitask corpus formed by mixing examples from the 756 English tasks in Supernatural Instructions [2408.01505]. Tasks are not represented by explicit task IDs or task embeddings; instead, routing is conditioned only on token hidden states derived from the concatenated instruction and input. This means task-specific behavior is implicit, arising from how different instructions and inputs induce different router activations [2408.01505]. Only adapter and router parameters are optimized: the shared \(A\), the expert-specific up-vectors \(\{b_j^i\}\), and the router parameters \(\mathbf{W}_{\mathcal{R};j}\). Optimization uses Adafactor with learning rate \(1\times 10^{-3}\), sequence length 1024, batch size 128, and 20k training steps [2408.01505]. The objective is standard next-token cross-entropy over the mixed multitask corpus, \[ \mathcal{L}(\theta) = - \mathbb{E}_{(x, y) \sim \text{SNI}} \sum_t \log p_\theta(y_t \mid y_{