LLaVA-DyMoE: Dynamic MoE for Continual LVLMs

• The paper introduces a dynamic MoE approach that uses drift-aware token assignment and routing score regularization to tackle routing drift and forgetting in LVLMs.
• LLaVA-DyMoE is a framework for multimodal continual instruction tuning that preserves old-task performance while facilitating new task learning through incremental expert expansion.
• The method achieves significant gains, including over 7% improvement in mean final accuracy and a 12% reduction in catastrophic forgetting on the CoIN benchmark.

Searching arXiv for the specified paper and closely related continual LVLM/MoE context. LLaVA-DyMoE is a dynamic Mixture of Experts framework for Multimodal Continual Instruction Tuning in Large Vision LLMs. It is introduced in "On Token's Dilemma: Dynamic MoE with Drift-Aware Token Assignment for Continual Learning of Large Vision LLMs" (Zhao et al., 29 Mar 2026). The method addresses a specific failure mode in continual LVLM adaptation: even when old experts are frozen and new experts are appended incrementally, the router can undergo routing-drift, causing old-task tokens to be attracted to newly added experts and thereby degrading prior-task performance. LLaVA-DyMoE formulates this phenomenon at the token level, identifies a token’s dilemma in which old and ambiguous tokens contribute little to new-task learning yet induce forgetting if routed to new experts, and mitigates the problem through drift-aware token assignment together with routing score regularization (Zhao et al., 29 Mar 2026).

1. Problem formulation and continual learning setting

The method is studied under Multimodal Continual Instruction Tuning (MCIT) for LVLMs such as LLaVA (Zhao et al., 29 Mar 2026). In this setting, the model is exposed to a stream of tasks,

$$\{ \mathcal{D}_1, \mathcal{D}_2, \dots, \mathcal{D}_T \},$$

where each task dataset $\mathcal{D}_t$ contains multimodal instruction-response samples

$$\mathbf{X} = (\mathbf{x}_v, \mathbf{x}_q, \mathbf{x}_a),$$

with image tokens, instruction or question tokens, and answer tokens (Zhao et al., 29 Mar 2026). The objective is the standard continual learning objective: learn new tasks, retain old tasks, and avoid catastrophic forgetting.

Within this setting, Mixture of Experts is used with LoRA experts. Each task can add new LoRA experts, old experts are frozen, and a router determines which experts each token uses (Zhao et al., 29 Mar 2026). The design is motivated by parameter isolation, scalability, sparse routing through top-$K$ activation, and token-level flexibility. However, the central argument of LLaVA-DyMoE is that incremental expert addition alone does not prevent forgetting, because the routing policy itself can change during new-task training (Zhao et al., 29 Mar 2026).

A plausible implication is that LLaVA-DyMoE reframes continual LVLM adaptation as not merely a parameter preservation problem, but also a routing policy preservation problem. This interpretation is explicitly supported by the paper’s emphasis on router behavior rather than expert overwriting (Zhao et al., 29 Mar 2026).

2. Routing-drift and the token’s dilemma

The paper defines routing-drift as corruption of the router’s learned policy for old tasks during training on a new task (Zhao et al., 29 Mar 2026). Even if old experts are frozen, training the new router and new experts on incoming data can alter routing behavior such that old-task tokens are later sent to newly added experts at inference time. This leads to forgetting despite expert isolation.

The analysis is performed at the token level. The paper distinguishes three token types based on their interaction with old and new expert groups (Zhao et al., 29 Mar 2026). New tokens strongly prefer the new expert group and are helpful for learning the new task. Old tokens mostly match old-task patterns, offer little benefit to new-task learning, but can harmfully bias the router if routed to new experts. Ambiguous tokens have similar affinity to old and new expert groups; they contribute little to learning while strongly destabilizing routing because their assignments are uncertain.

This leads to the paper’s formulation of the token’s dilemma: old and ambiguous tokens provide little new-task benefit, but if assigned to new experts during training, they induce routing-drift and future forgetting (Zhao et al., 29 Mar 2026). The method’s main insight is therefore that token assignment during training is the immediate source of drift.

This suggests that, in dynamic MoE continual learning, forgetting can arise even without direct modification of historical expert parameters. The destabilizing factor is the interaction between incoming tokens and the expanded routing space.

3. Dynamic MoE architecture and incremental expansion

LLaVA-DyMoE uses a MoE formulation in which each expert is a LoRA module. For layer $l,m$ in the LLaVA Transformer, the LoRA update is

$$\mathbf{W}^{l,m} = \mathbf{W}_0^{l,m} + \mathbf{B}^{l,m}\mathbf{A}^{l,m},$$

and expert $e_i^{l,m}$ is defined as

$$e_i^{l,m}(\mathbf{h}) = \mathbf{B}_i^{l,m}\mathbf{A}_i^{l,m}\mathbf{h}.$$

For token representation $\mathbf{h}^{l,m}$, the MoE output is

$\mathbf{h}^{l,m}_{\text{out} = \mathbf{W}_0^{l,m}\mathbf{h}^{l,m} + \sum_{i=1}^{N} w_i\, \mathbf{B}_i^{l,m}\mathbf{A}_i^{l,m}\mathbf{h}^{l,m}.$

Routing is sparse top-$\mathcal{D}_t$0: $\mathcal{D}_t$1 where $\mathcal{D}_t$2 is the router logit vector (Zhao et al., 29 Mar 2026).

The framework expands dynamically across tasks. At task $\mathcal{D}_t$3, old experts from task $\mathcal{D}_t$4 are frozen, new experts $\mathcal{D}_t$5 are added, the router output dimension is expanded to include the new experts, and only the new experts and new router parameters are trainable (Zhao et al., 29 Mar 2026). The expert set thus grows as

$\mathcal{D}_t$6

The significance of this design lies in the distinction between parameter isolation and routing stability. The architecture secures the former by freezing old experts, but the paper shows that this alone does not secure the latter, which motivates the additional mechanisms of LLaVA-DyMoE (Zhao et al., 29 Mar 2026).

4. Token characterization and drift-aware token assignment

Token types are characterized from routing score distributions over expert groups. For a token, router logits are partitioned into old-group logits and new-group logits. The paper defines

$\mathcal{D}_t$7

and the relative difference

$\mathcal{D}_t$8

with $\mathcal{D}_t$9 for numerical stability (Zhao et al., 29 Mar 2026). Using threshold $$\mathbf{X} = (\mathbf{x}_v, \mathbf{x}_q, \mathbf{x}_a),$$0, a token is ambiguous if $$\mathbf{X} = (\mathbf{x}_v, \mathbf{x}_q, \mathbf{x}_a),$$1; otherwise it is assigned according to which group has larger confidence.

On the basis of this characterization, the method introduces Token Assignment Guidance (TAG). A token is routed to the new expert group only if it is not ambiguous and is new-dominant: $$\mathbf{X} = (\mathbf{x}_v, \mathbf{x}_q, \mathbf{x}_a),$$2

$$\mathbf{X} = (\mathbf{x}_v, \mathbf{x}_q, \mathbf{x}_a),$$3

Each expert $$\mathbf{X} = (\mathbf{x}_v, \mathbf{x}_q, \mathbf{x}_a),$$4 receives a binary mask

$$\mathbf{X} = (\mathbf{x}_v, \mathbf{x}_q, \mathbf{x}_a),$$5

and masked routing scores are

$$\mathbf{X} = (\mathbf{x}_v, \mathbf{x}_q, \mathbf{x}_a),$$6

(Zhao et al., 29 Mar 2026)

The operational consequence is explicit: new tokens are allowed to train new experts, old tokens are forced to old experts, and ambiguous tokens are conservatively routed to old experts (Zhao et al., 29 Mar 2026). The paper describes TAG as a hard routing correction during training. Its role is to suppress misleading gradients that would otherwise encourage the router to overfit new experts to old or ambiguous patterns.

A plausible implication is that TAG imposes a conservative bias in favor of routing continuity. Rather than attempting to resolve uncertainty aggressively, it preserves established routing patterns when the score gap is insufficiently decisive.

5. Routing score regularization and training objective

LLaVA-DyMoE complements TAG with Routing Score Regularization (RSR), which softly shapes router behavior (Zhao et al., 29 Mar 2026). For a token at one layer, the total routing probability mass on old and new expert groups is defined as

$$\mathbf{X} = (\mathbf{x}_v, \mathbf{x}_q, \mathbf{x}_a),$$7

The first regularizer is the exclusivity loss,

$$\mathbf{X} = (\mathbf{x}_v, \mathbf{x}_q, \mathbf{x}_a),$$8

which encourages a token to activate only one expert group rather than both (Zhao et al., 29 Mar 2026). If both group masses are large, the product is large; minimizing it enforces separation between old and new expert groups and reduces mixed assignments that can corrupt routing.

The second regularizer is the specialization loss. The paper defines a soft target

$$\mathbf{X} = (\mathbf{x}_v, \mathbf{x}_q, \mathbf{x}_a),$$9

where

$K$0

The loss is then

$K$1

This is described as a BCE-style loss on the total new-group gate mass (Zhao et al., 29 Mar 2026). If old experts are not active, $K$2 approaches 1 and encourages new-expert usage; if old experts are active, $K$3 is smaller and discourages unnecessary new-group activation.

The full objective is

$K$4

where $K$5 is instruction-tuning or next-token prediction loss, $K$6 is MoE load balancing loss, $K$7 is the load-balancing weight, and $K$8 is the weight for the proposed routing regularizers (Zhao et al., 29 Mar 2026). The paper uses $K$9 by default.

Taken together, TAG and RSR divide responsibility between hard assignment control and soft router shaping. The paper’s interpretation is that hard token guidance prevents drift, while soft losses stabilize and specialize routing (Zhao et al., 29 Mar 2026).

6. Training procedure, implementation details, and evaluation protocol

The training procedure begins from LLaVA-v1.5-7B, with 13B also tested in the appendix (Zhao et al., 29 Mar 2026). Backbone components are frozen except for newly added modules. For each incoming task $l,m$0, the method adds $l,m$1 new LoRA experts per MoE layer, expands router outputs accordingly, freezes old parameters, computes token routing logits over old and new groups, applies TAG assignment, applies RSR, and optimizes the full objective (Zhao et al., 29 Mar 2026). After training, old experts remain frozen and the newly trained experts become part of the old group for future tasks.

The appendix reports the following default implementation details: 16 rank-4 LoRA experts added per task, top-$l,m$2, batch size 128, learning rate $l,m$3, warmup ratio 0.03, AdamW, BF16, DeepSpeed ZeRO-2, and both $l,m$4 and $l,m$5 set to $l,m$6 (Zhao et al., 29 Mar 2026).

Evaluation is conducted on CoIN, a continual instruction tuning benchmark with 8 VQA-style tasks: ScienceQA (SQA), TextVQA, ImageNet, GQA, VizWiz, RefCOCO (REF), VQAv2, and OCR-VQA (Zhao et al., 29 Mar 2026). The dataset sizes are 569k for training and 261k for test. The backbone uses LLaVA-v1.5-7B, with visual encoder CLIP ViT-L/14 and a Vicuna backbone (Zhao et al., 29 Mar 2026).

The paper adopts the standard CoIN metrics:

$l,m$7

$l,m$8

$l,m$9

Here, MFN is Mean Final Accuracy, MAA is Mean Average Accuracy, and BWT is Backward Transfer, where less negative BWT indicates less forgetting (Zhao et al., 29 Mar 2026).

7. Empirical results, ablations, and limitations

On CoIN, LLaVA-DyMoE achieves MFN = 57.03, MAA = 57.70, and BWT = -4.67 (Zhao et al., 29 Mar 2026). Compared with IncMoELoRA, which attains MFN = 49.68, MAA = 49.50, and BWT = -16.67, the paper reports +7.35% MFN improvement, +8.20% MAA improvement, and 12.00% reduction in forgetting (Zhao et al., 29 Mar 2026). These values correspond to the abstract’s statement that the method yields over a 7% gain in mean final accuracy and a 12% reduction in forgetting.

The paper further reports particularly large per-task improvements on several tasks: ImageNet: 68.42 → 95.80, VizWiz: 39.46 → 52.35, REF: 4.56 → 9.25, and OCR-VQA: 60.95 → 62.00 (Zhao et al., 29 Mar 2026). The interpretation offered is that the method improves retention while also improving final accuracy on new tasks.

Ablation results identify the role of each component. Starting from IncMoELoRA, adding $$\mathbf{W}^{l,m} = \mathbf{W}_0^{l,m} + \mathbf{B}^{l,m}\mathbf{A}^{l,m},$$0 gives a small gain, with MFN 50.76 and BWT -15.44. Adding TAG yields MFN 54.44 and BWT -7.04. Adding $$\mathbf{W}^{l,m} = \mathbf{W}_0^{l,m} + \mathbf{B}^{l,m}\mathbf{A}^{l,m},$$1 gives MFN 55.18 and BWT -6.83. Adding $$\mathbf{W}^{l,m} = \mathbf{W}_0^{l,m} + \mathbf{B}^{l,m}\mathbf{A}^{l,m},$$2, i.e., the full model, reaches MFN 57.03 and BWT -4.67 (Zhao et al., 29 Mar 2026). The paper’s interpretation is that TAG is the most important component because it directly addresses the token-level source of routing drift.

Threshold and loss-weight ablations further characterize the design. For ambiguity threshold $$\mathbf{W}^{l,m} = \mathbf{W}_0^{l,m} + \mathbf{B}^{l,m}\mathbf{A}^{l,m},$$3, MFN is 56.87 at 10%, 57.03 at 20%, 56.27 at 30%, and 55.32 at 50%, with the main setting using 20% (Zhao et al., 29 Mar 2026). For $$\mathbf{W}^{l,m} = \mathbf{W}_0^{l,m} + \mathbf{B}^{l,m}\mathbf{A}^{l,m},$$4, the paper reports that $$\mathbf{W}^{l,m} = \mathbf{W}_0^{l,m} + \mathbf{B}^{l,m}\mathbf{A}^{l,m},$$5 gives worse forgetting, $$\mathbf{W}^{l,m} = \mathbf{W}_0^{l,m} + \mathbf{B}^{l,m}\mathbf{A}^{l,m},$$6 is good, $$\mathbf{W}^{l,m} = \mathbf{W}_0^{l,m} + \mathbf{B}^{l,m}\mathbf{A}^{l,m},$$7 is best, and $$\mathbf{W}^{l,m} = \mathbf{W}_0^{l,m} + \mathbf{B}^{l,m}\mathbf{A}^{l,m},$$8 is slightly worse (Zhao et al., 29 Mar 2026).

The paper also identifies several limitations. The ambiguity score based on the relative max-logit gap is effective but remains heuristic, and tokens near the threshold may be unstable (Zhao et al., 29 Mar 2026). If $$\mathbf{W}^{l,m} = \mathbf{W}_0^{l,m} + \mathbf{B}^{l,m}\mathbf{A}^{l,m},$$9 is too large, too many tokens are forced to old experts, which can suppress useful new-task learning and reduce MFN and MAA. Qualitative examples indicate that difficult or visually subtle cases can still be mistaken, especially with small objects or similar categories. The paper also notes the need for future work on larger models, more realistic continual settings, and broader task heterogeneity. Finally, it states that the method targets routing-drift, a major source of forgetting in dynamic MoE, but not necessarily all causes of forgetting in LVLM continual learning (Zhao et al., 29 Mar 2026).

The appendix further shows that the method is orthogonal to replay-based methods, ASD-style data-based methods, and task-level router approaches (Zhao et al., 29 Mar 2026). It reports that adding LLaVA-DyMoE on top of SEFE improves BWT from $e_i^{l,m}$0 to $e_i^{l,m}$1, and that replay yields stronger results with BWT of $e_i^{l,m}$2, $e_i^{l,m}$3, and $e_i^{l,m}$4 for replay buffer sizes 200, 500, and 1000 respectively (Zhao et al., 29 Mar 2026). This suggests that the method functions as a micro-level routing fix that can compose with macro-level continual learning strategies.

In summary, LLaVA-DyMoE is defined by a specific diagnosis of forgetting in dynamic MoE-based LVLM continual learning: the decisive problem is not only preserving old expert parameters, but controlling which tokens are permitted to influence new experts and the expanded router. By characterizing tokens through routing-score ambiguity, forcing old and ambiguous tokens away from new experts during training, and regularizing expert-group separation and specialization, the framework substantially improves continual instruction tuning performance on CoIN (Zhao et al., 29 Mar 2026).