---
title: 'LLaVA-DyMoE: Dynamic MoE for Continual LVLMs'
url: https://www.emergentmind.com/topics/llava-dymoe
type: topic
---

# LLaVA-DyMoE: Dynamic MoE for Continual LVLMs

Searching arXiv for the specified paper and closely related continual LVLM/MoE context.
LLaVA-DyMoE is a dynamic Mixture of Experts framework for Multimodal Continual Instruction Tuning in Large Vision Language Models. It is introduced in "On Token's Dilemma: Dynamic MoE with Drift-Aware Token Assignment for Continual Learning of Large Vision Language Models" [2603.27481]. The method addresses a specific failure mode in continual LVLM adaptation: even when old experts are frozen and new experts are appended incrementally, the router can undergo routing-drift, causing old-task tokens to be attracted to newly added experts and thereby degrading prior-task performance. LLaVA-DyMoE formulates this phenomenon at the token level, identifies a token’s dilemma in which old and ambiguous tokens contribute little to new-task learning yet induce forgetting if routed to new experts, and mitigates the problem through drift-aware token assignment together with routing score regularization [2603.27481].

## 1. Problem formulation and continual learning setting

The method is studied under **Multimodal Continual Instruction Tuning (MCIT)** for LVLMs such as LLaVA [2603.27481]. In this setting, the model is exposed to a stream of tasks,
\[
\{ \mathcal{D}_1, \mathcal{D}_2, \dots, \mathcal{D}_T \},
\]
where each task dataset \(\mathcal{D}_t\) contains multimodal instruction-response samples
\[
\mathbf{X} = (\mathbf{x}_v, \mathbf{x}_q, \mathbf{x}_a),
\]
with image tokens, instruction or question tokens, and answer tokens [2603.27481]. The objective is the standard continual learning objective: learn new tasks, retain old tasks, and avoid catastrophic forgetting.

Within this setting, Mixture of Experts is used with **LoRA experts**. Each task can add new LoRA experts, old experts are frozen, and a router determines which experts each token uses [2603.27481]. The design is motivated by parameter isolation, scalability, sparse routing through top-\(K\) activation, and token-level flexibility. However, the central argument of LLaVA-DyMoE is that incremental expert addition alone does not prevent forgetting, because the routing policy itself can change during new-task training [2603.27481].

A plausible implication is that LLaVA-DyMoE reframes continual LVLM adaptation as not merely a parameter preservation problem, but also a routing policy preservation problem. This interpretation is explicitly supported by the paper’s emphasis on router behavior rather than expert overwriting [2603.27481].

## 2. Routing-drift and the token’s dilemma

The paper defines **routing-drift** as corruption of the router’s learned policy for old tasks during training on a new task [2603.27481]. Even if old experts are frozen, training the new router and new experts on incoming data can alter routing behavior such that old-task tokens are later sent to newly added experts at inference time. This leads to forgetting despite expert isolation.

The analysis is performed at the token level. The paper distinguishes three token types based on their interaction with old and new expert groups [2603.27481]. **New tokens** strongly prefer the new expert group and are helpful for learning the new task. **Old tokens** mostly match old-task patterns, offer little benefit to new-task learning, but can harmfully bias the router if routed to new experts. **Ambiguous tokens** have similar affinity to old and new expert groups; they contribute little to learning while strongly destabilizing routing because their assignments are uncertain.

This leads to the paper’s formulation of the **token’s dilemma**: old and ambiguous tokens provide little new-task benefit, but if assigned to new experts during training, they induce routing-drift and future forgetting [2603.27481]. The method’s main insight is therefore that token assignment during training is the immediate source of drift.

This suggests that, in dynamic MoE continual learning, forgetting can arise even without direct modification of historical expert parameters. The destabilizing factor is the interaction between incoming tokens and the expanded routing space.

## 3. Dynamic MoE architecture and incremental expansion

LLaVA-DyMoE uses a MoE formulation in which each expert is a LoRA module. For layer \(l,m\) in the LLaVA Transformer, the LoRA update is
\[
\mathbf{W}^{l,m} = \mathbf{W}_0^{l,m} + \mathbf{B}^{l,m}\mathbf{A}^{l,m},
\]
and expert \(e_i^{l,m}\) is defined as
\[
e_i^{l,m}(\mathbf{h}) = \mathbf{B}_i^{l,m}\mathbf{A}_i^{l,m}\mathbf{h}.
\]
For token representation \(\mathbf{h}^{l,m}\), the MoE output is
\[
\mathbf{h}^{l,m}_{\text{out} = \mathbf{W}_0^{l,m}\mathbf{h}^{l,m} + \sum_{i=1}^{N} w_i\, \mathbf{B}_i^{l,m}\mathbf{A}_i^{l,m}\mathbf{h}^{l,m}.
\]
Routing is sparse top-\(K\):
\[
w'_i = \exp(s_i)\,\mathbbm{1}[i \in \mathrm{TopK}_K(\mathbf{s})], \qquad w_i = \frac{w'_i}{\sum_{j=1}^{N} w'_j},
\]
where \(\mathbf{s}=R^{l,m}(\mathbf{h}^{l,m})\) is the router logit vector [2603.27481].

The framework expands dynamically across tasks. At task \(t>1\), old experts from task \(t-1\) are frozen, new experts \(\{e^{l,m}_i\}_{i=1}^{N_t}\) are added, the router output dimension is expanded to include the new experts, and only the new experts and new router parameters are trainable [2603.27481]. The expert set thus grows as
\[
\mathcal{E}_t = \mathcal{E}_{t-1} \cup \mathcal{E}_{t,\text{new}.
\]

The significance of this design lies in the distinction between **parameter isolation** and **routing stability**. The architecture secures the former by freezing old experts, but the paper shows that this alone does not secure the latter, which motivates the additional mechanisms of LLaVA-DyMoE [2603.27481].

## 4. Token characterization and drift-aware token assignment

Token types are characterized from **routing score distributions** over expert groups. For a token, router logits are partitioned into old-group logits and new-group logits. The paper defines
\[
c_{\text{old} = \max(\mathbf{s}_{t-1}), \qquad c_{\text{new} = \max(\mathbf{s}_{t,\text{new}),
\]
and the relative difference
\[
D_{\text{rel} = \frac{|c_{\text{new} - c_{\text{old}|} {\max(|c_{\text{new}|, |c_{\text{old}|) + \epsilon},
\]
with \(\epsilon\) for numerical stability [2603.27481]. Using threshold \(\tau\), a token is **ambiguous** if \(D_{\text{rel} \le \tau\); otherwise it is assigned according to which group has larger confidence.

On the basis of this characterization, the method introduces **Token Assignment Guidance (TAG)**. A token is routed to the new expert group only if it is not ambiguous and is new-dominant:
\[
M_{\text{new} = \mathbbm{1}\left((c_{\text{new} > c_{\text{old})\land(D_{\text{rel}>\tau)\right),
\]
\[
M_{\text{old} = 1 - M_{\text{new}.
\]
Each expert \(i\) receives a binary mask
\[
m_{t,i} = \mathbbm{1}(i \in \mathcal{E}_{t-1})\, M_{\text{old} + \mathbbm{1}(i \in \mathcal{E}_{t,\text{new})\, M_{\text{new},
\]
and masked routing scores are
\[
s'_{t,i} = \begin{cases} s_{t,i}, & \text{if } m_{t,i}=1,\ -\infty, & \text{otherwise.} \end{cases}
\]
[2603.27481]

The operational consequence is explicit: new tokens are allowed to train new experts, old tokens are forced to old experts, and ambiguous tokens are conservatively routed to old experts [2603.27481]. The paper describes TAG as a hard routing correction during training. Its role is to suppress misleading gradients that would otherwise encourage the router to overfit new experts to old or ambiguous patterns.

A plausible implication is that TAG imposes a conservative bias in favor of routing continuity. Rather than attempting to resolve uncertainty aggressively, it preserves established routing patterns when the score gap is insufficiently decisive.

## 5. Routing score regularization and training objective

LLaVA-DyMoE complements TAG with **Routing Score Regularization (RSR)**, which softly shapes router behavior [2603.27481]. For a token at one layer, the total routing probability mass on old and new expert groups is defined as
\[
g_{\text{old} = \sum_{i\in \mathcal{E}_{t-1} w_i, \qquad g_{\text{new} = \sum_{i\in \mathcal{E}_{t,\text{new} w_i.
\]

The first regularizer is the **exclusivity loss**,
\[
\mathcal{L}_{\text{exc} = g_{\text{old}\, g_{\text{new},
\]
which encourages a token to activate only one expert group rather than both [2603.27481]. If both group masses are large, the product is large; minimizing it enforces separation between old and new expert groups and reduces mixed assignments that can corrupt routing.

The second regularizer is the **specialization loss**. The paper defines a soft target
\[
y \triangleq 1 - \tilde{g}_{\text{old},
\]
where
\[
\tilde{g}_{\text{old} = \max\{w_i\}_{i\in\mathcal{E}_{t-1}.
\]
The loss is then
\[
\mathcal{L}_{\text{spe} = -y\log g_{\text{new} -(1-y)\log(1-g_{\text{new}).
\]
This is described as a BCE-style loss on the total new-group gate mass [2603.27481]. If old experts are not active, \(y\) approaches 1 and encourages new-expert usage; if old experts are active, \(y\) is smaller and discourages unnecessary new-group activation.

The full objective is
\[
\mathcal{L} = \mathcal{L}_{\text{NTP} + \lambda \mathcal{L}_{\text{aux} + \alpha\left(\mathcal{L}_{\text{exc} + \mathcal{L}_{\text{spe}\right),
\]
where \(\mathcal{L}_{\text{NTP}\) is instruction-tuning or next-token prediction loss, \(\mathcal{L}_{\text{aux}\) is MoE load balancing loss, \(\lambda\) is the load-balancing weight, and \(\alpha\) is the weight for the proposed routing regularizers [2603.27481]. The paper uses \(\alpha=10^{-3}\) by default.

Taken together, TAG and RSR divide responsibility between hard assignment control and soft router shaping. The paper’s interpretation is that hard token guidance prevents drift, while soft losses stabilize and specialize routing [2603.27481].

## 6. Training procedure, implementation details, and evaluation protocol

The training procedure begins from **LLaVA-v1.5-7B**, with 13B also tested in the appendix [2603.27481]. Backbone components are frozen except for newly added modules. For each incoming task \(t\), the method adds \(N_t\) new LoRA experts per MoE layer, expands router outputs accordingly, freezes old parameters, computes token routing logits over old and new groups, applies TAG assignment, applies RSR, and optimizes the full objective [2603.27481]. After training, old experts remain frozen and the newly trained experts become part of the old group for future tasks.

The appendix reports the following default implementation details: **16 rank-4 LoRA experts** added per task, top-\(K=16\), batch size 128, learning rate \(2\times10^{-4}\), warmup ratio 0.03, AdamW, BF16, DeepSpeed ZeRO-2, and both \(\lambda\) and \(\alpha\) set to \(10^{-3}\) [2603.27481].

Evaluation is conducted on **CoIN**, a continual instruction tuning benchmark with **8 VQA-style tasks**: ScienceQA (SQA), TextVQA, ImageNet, GQA, VizWiz, RefCOCO (REF), VQAv2, and OCR-VQA [2603.27481]. The dataset sizes are **569k** for training and **261k** for test. The backbone uses **LLaVA-v1.5-7B**, with visual encoder CLIP ViT-L/14 and a Vicuna backbone [2603.27481].

The paper adopts the standard CoIN metrics:

\[
\mathrm{MFN} = \frac{1}{T}\sum_{i=1}^{T} A_{T,i},
\]

\[
\mathrm{MAA} = \frac{1}{T}\sum_{j=1}^{T}\frac{1}{j}\sum_{i=1}^{j} A_{j,i},
\]

\[
\mathrm{BWT} = \frac{1}{T}\sum_{i=1}^{T}(A_{T,i}-A_{i,i}).
\]

Here, MFN is Mean Final Accuracy, MAA is Mean Average Accuracy, and BWT is Backward Transfer, where less negative BWT indicates less forgetting [2603.27481].

## 7. Empirical results, ablations, and limitations

On CoIN, LLaVA-DyMoE achieves **MFN = 57.03**, **MAA = 57.70**, and **BWT = -4.67** [2603.27481]. Compared with **IncMoELoRA**, which attains **MFN = 49.68**, **MAA = 49.50**, and **BWT = -16.67**, the paper reports **+7.35% MFN improvement**, **+8.20% MAA improvement**, and **12.00% reduction in forgetting** [2603.27481]. These values correspond to the abstract’s statement that the method yields over a 7% gain in mean final accuracy and a 12% reduction in forgetting.

The paper further reports particularly large per-task improvements on several tasks: **ImageNet: 68.42 → 95.80**, **VizWiz: 39.46 → 52.35**, **REF: 4.56 → 9.25**, and **OCR-VQA: 60.95 → 62.00** [2603.27481]. The interpretation offered is that the method improves retention while also improving final accuracy on new tasks.

Ablation results identify the role of each component. Starting from IncMoELoRA, adding \(\mathcal{L}_{aux}\) gives a small gain, with **MFN 50.76** and **BWT -15.44**. Adding TAG yields **MFN 54.44** and **BWT -7.04**. Adding \(\mathcal{L}_{exc}\) gives **MFN 55.18** and **BWT -6.83**. Adding \(\mathcal{L}_{spe}\), i.e., the full model, reaches **MFN 57.03** and **BWT -4.67** [2603.27481]. The paper’s interpretation is that TAG is the most important component because it directly addresses the token-level source of routing drift.

Threshold and loss-weight ablations further characterize the design. For ambiguity threshold \(\tau\), MFN is **56.87** at 10%, **57.03** at 20%, **56.27** at 30%, and **55.32** at 50%, with the main setting using **20%** [2603.27481]. For \(\alpha\), the paper reports that \(10^{-2}\) gives worse forgetting, \(5\times10^{-3}\) is good, \(10^{-3}\) is best, and \(5\times10^{-4}\) is slightly worse [2603.27481].

The paper also identifies several limitations. The ambiguity score based on the relative max-logit gap is effective but remains heuristic, and tokens near the threshold may be unstable [2603.27481]. If \(\tau\) is too large, too many tokens are forced to old experts, which can suppress useful new-task learning and reduce MFN and MAA. Qualitative examples indicate that difficult or visually subtle cases can still be mistaken, especially with small objects or similar categories. The paper also notes the need for future work on larger models, more realistic continual settings, and broader task heterogeneity. Finally, it states that the method targets routing-drift, a major source of forgetting in dynamic MoE, but not necessarily all causes of forgetting in LVLM continual learning [2603.27481].

The appendix further shows that the method is **orthogonal** to replay-based methods, ASD-style data-based methods, and task-level router approaches [2603.27481]. It reports that adding LLaVA-DyMoE on top of SEFE improves BWT from \(-10.45\) to \(-4.75\), and that replay yields stronger results with BWT of \(-1.55\), \(-1.00\), and \(-0.64\) for replay buffer sizes 200, 500, and 1000 respectively [2603.27481]. This suggests that the method functions as a micro-level routing fix that can compose with macro-level continual learning strategies.

In summary, LLaVA-DyMoE is defined by a specific diagnosis of forgetting in dynamic MoE-based LVLM continual learning: the decisive problem is not only preserving old expert parameters, but controlling which tokens are permitted to influence new experts and the expanded router. By characterizing tokens through routing-score ambiguity, forcing old and ambiguous tokens away from new experts during training, and regularizing expert-group separation and specialization, the framework substantially improves continual instruction tuning performance on CoIN [2603.27481].

Source: https://www.emergentmind.com/topics/llava-dymoe