TRGE: Two-Level Routing for Grouped Mixture-of-Experts

• The paper introduces TRGE as a parameter-efficient solution for multi-domain continual learning, effectively mitigating catastrophic and forward forgetting through two-level routing and dynamic fusion.
• TRGE employs task-specific grouped experts with a fixed expert count, leveraging LoRA adapters to ensure intra-group specialization and controlled inter-group collaboration.
• Empirical results across 11 datasets demonstrate that TRGE outperforms existing PEFT methods with up to +1.8% improvement and 61% fewer parameters.

Two-Level Routing Grouped Mixture-of-Experts (TRGE) is a method for Multi-Domain Continual Learning (MDCL) that expands a frozen pre-trained CLIP model with task-specific grouped experts, routes inputs through a two-level mechanism that separates intra-group specialization from inter-group collaboration, uses Multimodal LLMs (MLLMs) for semantic task identification, and dynamically fuses adapter outputs with the frozen backbone to mitigate both catastrophic forgetting and forward forgetting (Zhou et al., 11 Aug 2025).

1. Problem setting and design objective

TRGE is formulated for MDCL under sequential tasks with shifting class sets and distribution. In this setting, the paper identifies two central failure modes: catastrophic forgetting, defined as the loss of previously learned knowledge as new tasks or domains are learned, and forward forgetting, defined as degradation of zero-shot pretrained model capabilities, including knowledge necessary for unseen future tasks (Zhou et al., 11 Aug 2025).

The method is motivated by limitations of existing Parameter-Efficient Fine-Tuning (PEFT) approaches such as prompt tuning and adapter methods. These approaches freeze pre-trained models and train a small number of additional parameters for each new task, but they are described as struggling to maintain a balance between knowledge retention and cross-task transfer or accumulation, especially when both the class set and input distribution shift. The paper also situates TRGE against existing MoE-style approaches whose growing number of experts increases router complexity and risks routing overfitting and representational collapse as more tasks are added (Zhou et al., 11 Aug 2025).

Within that problem formulation, TRGE addresses a specific question: how to dynamically expand capacity for new task or domain knowledge without losing prior knowledge or impairing zero-shot forward generalization, while still enabling effective cross-task knowledge transfer without overfitting or inefficiency in the routing mechanism. The architecture is therefore organized around separation of task-specific knowledge and controlled collaboration across tasks rather than around a single flat expert pool (Zhou et al., 11 Aug 2025).

2. Architectural organization of grouped experts

TRGE operates on a frozen pre-trained CLIP backbone. For each new task $t$, it adds a new task-specific expert group as a plug-in adapter module, and only the current task’s group is trainable during task $t$ training; previous expert groups and the CLIP backbone are frozen. The paper presents this freezing policy as the core mechanism for isolating task knowledge and mitigating catastrophic forgetting (Zhou et al., 11 Aug 2025).

Each task-specific group $G_t$ contains $N_e$ LoRA experts $E_i^t$ together with a small intra-group router $R_t^I$. LoRA adapters are used to limit parameter overhead. The architecture thus combines dynamic expansion across tasks with parameter-efficient adaptation inside each task-specific module. The expert count within each group is kept static even as the number of tasks grows. In the paper, $N_e$ is fixed and is typically set to $3$, which is explicitly described as limiting routing complexity within each group and preventing overfitting as more groups are added over time (Zhou et al., 11 Aug 2025).

This grouped decomposition gives TRGE two distinct forms of modularity. First, the addition of a new group for each task makes knowledge accumulation incremental and localized. Second, the fixed expert cardinality within a group ensures that the growth of the continual-learning system occurs primarily at the group level rather than through unbounded expansion of a single router’s action space. A plausible implication is that TRGE treats continual learning not as repeated retuning of one shared adapter, but as structured accumulation of narrowly trainable modules plus controlled reuse of older modules.

3. Two-level routing: intra-group specialization and inter-group collaboration

The defining mechanism of TRGE is its two-level routing structure. The first level is intra-group routing, which selects among experts inside a task group. The second level is inter-group routing, which selects and combines outputs from multiple task groups. This decomposition is introduced specifically to preserve task separation while still allowing collaboration across tasks (Zhou et al., 11 Aug 2025).

For an input sample, the feature vector $x_C$ is the $[CLS]$ token from CLIP. Inside group $t$0, the intra-group router computes gating weights over the $t$1 experts as

$t$2

where $t$3 is learnable. The resulting group output is

$t$4

This stage handles specialization within a task-local parameter budget: the group already corresponds to a task, and the intra-group router resolves which experts inside that task group should contribute (Zhou et al., 11 Aug 2025).

The inter-group stage enables collaboration across tasks. For each task $t$5, TRGE stores a task prototype

$t$6

Given input $t$7, it computes Euclidean distances to all prototypes,

$t$8

and converts them into relevance scores

$t$9

where $G_t$0. After softmax normalization,

$G_t$1

with $G_t$2, the routing policy performs weight scaling to distinguish a main group from assistant groups. The group matched by the task identifier receives weight $G_t$3; assistant groups with score above threshold $G_t$4 receive $G_t$5; the remaining groups are suppressed:

$G_t$6

The final group weights are then normalized,

$G_t$7

and the overall adapter output is

$G_t$8

Functionally, the main group preserves task-specificity while assistant groups provide cross-task collaboration. The paper explicitly states that the static expert count within each group is introduced to alleviate routing overfitting caused by increasing routing complexity as the number of experts grows over time (Zhou et al., 11 Aug 2025).

4. Semantic task recognition and dynamic fusion with the frozen backbone

TRGE makes the inter-group router depend on task identity, and the paper treats reliable task recognition as a central requirement rather than as a secondary heuristic. To obtain correct task identifiers, the method leverages MLLMs with multimodal comprehension capabilities to generate semantic task descriptions and recognize the correct task identifier (Zhou et al., 11 Aug 2025).

For each task, the class set $G_t$9 is summarized into a natural-language description

$N_e$0

At inference time, the MLLM ranks the task descriptions $N_e$1 against the input image $N_e$2 and returns

$N_e$3

The output may also indicate an unseen task, $N_e$4. In the technical summary, this mechanism is named Semantics-based Task Recognition (STR) and is described as highly robust, near error-free, and substantially more precise than feature-based task recognition under task or domain shifts (Zhou et al., 11 Aug 2025).

TRGE also addresses forward forgetting by dynamically fusing the outputs of frozen CLIP and the TRGE adapter. For seen tasks, the output is

$N_e$5

For unseen tasks, the adapter contribution grows with the number of learned tasks via hyperparameter $N_e$6:

$N_e$7

The stated rationale is that continual finetuning, even with PEFT, tends to degrade the zero-shot capability of pre-trained CLIP, while learned knowledge becomes increasingly useful for zero-shot prediction as training progresses. TRGE therefore retains the pretrained model’s zero-shot ability while incrementally leveraging new knowledge as the adapter becomes more capable (Zhou et al., 11 Aug 2025).

5. Empirical results, ablations, and parameter efficiency

The reported evaluation spans 11 datasets—Aircraft, Caltech101, CIFAR100, DTD, EuroSAT, Flowers, Food, MNIST, OxfordPet, StanfordCars, and SUN397—under both MTIL and MCIL settings. Across these settings, TRGE is reported to outperform ZSCL, MoE-Adapter, iCaRL, LwF, and various non-PEFT and PEFT baselines on the three metrics used in the paper: Transfer for unseen tasks, Average, and Last for previously learned tasks (Zhou et al., 11 Aug 2025).

In the MTIL setting, Table 1 reports an Average score of 76.7% for MoE-Adapter and 78.5% for TRGE, a +1.8% improvement, with 61% fewer parameters. In the MCIL setting, Table 2 reports an Average score of 76.1% for MoE-Adapter and 77.3% for TRGE, a +1.2% improvement, again with 61% fewer parameters (Zhou et al., 11 Aug 2025).

The ablation results attribute specific gains to the grouped and routed design. Adding grouping and inter-group routing improves Last and Average performance by up to +2.6% and +1.8%, respectively. Dynamic fusion further improves zero-shot Transfer accuracy. The task-recognition study shows that STR using an MLLM matches almost oracle levels of task identification for both seen and unseen tasks and outperforms cluster or feature-reconstruction methods. The expert-number study reports that setting the number of experts per group to 3 provides the best balance for specialization and routing robustness (Zhou et al., 11 Aug 2025).

These findings support two points that are central to the method’s identity. First, TRGE is not only a forgetting-mitigation framework but also a parameter-efficient one. Second, the paper’s gains are tied to the interaction of grouped expansion, two-level routing, MLLM-based task recognition, and dynamic fusion rather than to any one component in isolation.

6. Position within grouped and hierarchical MoE research

TRGE belongs to a broader family of grouped or hierarchical MoE designs, but its problem setting and routing semantics are distinct. In AT-MoE, task-specific experts are trained via LoRA and a layer-wise adaptive grouped routing module first performs overall weight allocation from the dimension of the expert group and then local weight normalization adjustments within the group; the stated goals are multi-dimensional balance, controllability, interpretability, and task-specific fusion for complex instructions (Li et al., 2024). TRGE also uses two-level grouped routing and LoRA experts, but it couples them to continual task expansion, task identifiers, task prototype distance, and forward-forgetting mitigation (Zhou et al., 11 Aug 2025).

In MoGE, introduced in the Pangu Pro MoE work, experts are divided into non-overlapping groups, and each token activates an equal number of experts within each predefined group. That design guarantees that every group receives exactly the same number of token-assigned expert computations per input, yielding an Imbalance Score of $N_e$8 by construction and targeting distributed load balance and throughput on Ascend NPUs (Tang et al., 27 May 2025). TRGE’s grouped structure is instead organized around tasks rather than device-balanced execution. A common misconception is that two-level grouped routing is primarily a systems-level load-balancing device; TRGE shows a different use case in which grouped routing is primarily a continual-learning mechanism for separation and collaboration.

Hi-MoE decomposes routing control into inter-group balancing and intra-group specialization, with hierarchical regularizers that target fair traffic across expert groups and complementary expert behaviors within groups; on 58B tokens, Hi-MoE-7B reports a 5.6% perplexity reduction and a 40% improvement in expert balance over OLMoE-7B (Molodtsov et al., 8 May 2026). MoHGE also adopts two-level routing, but in a heterogeneous-expert setting where groups differ in size, combining group gating, intra-group expert gating, a Group-Wise Auxiliary Loss, an Intra-Group Experts Auxiliary Loss, and an All-size Group-decoupling Allocation strategy to maintain balanced GPU utilization while reducing total parameters by approximately 20% (Ma et al., 25 Apr 2026). GQE, by contrast, places MoE routing inside grouped-query attention: within each GQA group, a router selects $N_e$9 query-head experts per token while all key-value heads remain dense and unchanged, and it does not route between groups (Tripathi et al., 18 Jun 2026).

This comparison suggests that TRGE is best understood as a continual-learning-specific instance of grouped MoE design. Its distinctive contribution lies in combining task-specific grouped expansion, two-level routing, MLLM-based semantic task recognition, and dynamic fusion with a frozen backbone to address catastrophic forgetting and forward forgetting simultaneously (Zhou et al., 11 Aug 2025).