Tea-MOELoRA: Adaptive MoE-LoRA Framework

• Tea-MOELoRA is a parameter-efficient Mixture-of-Experts system using Low-Rank Adaptation to combine multiple specialized experts with a frozen backbone via metadata or token-level routing.
• It features two formulations: a Chinese information extraction framework with task-era-aware routing and a MeteoRA-grounded design enabling autonomous token-level expert selection.
• The framework mitigates interference and enhances multi-task performance while addressing efficiency challenges through novel gate training and optimized kernel strategies.

Searching arXiv for Tea-MOELoRA and closely related MoE-LoRA work to ground the article in current papers. Tea-MOELoRA denotes a parameter-efficient class of Mixture-of-Experts (MoE) systems built on Low-Rank Adaptation (LoRA), in which a frozen backbone is augmented by multiple low-rank experts and a learned router that allocates expert contributions. In the provided literature, the term refers most directly to a multi-task information extraction framework for Classical and Modern Chinese, where a task-era-aware router mitigates interference across heterogeneous tasks and temporally distant domains (Tang et al., 1 Sep 2025). Closely related descriptions also use Tea-MOELoRA for a MeteoRA-grounded design that embeds a library of task-specific LoRA adapters into all linear layers of a base LLM via a full-mode MoE architecture, enabling autonomous task sensing and timely adapter switching within a single inference pass (Xu et al., 2024). This suggests that Tea-MOELoRA is best understood as a design pattern within the broader MoE-LoRA literature rather than as a single immutable architecture.

1. Scope and terminological usage

Within the provided sources, Tea-MOELoRA appears in two technically distinct but conceptually aligned forms. One is a Chinese information extraction framework that jointly models relation extraction and event-oriented tasks across Classical and Modern Chinese by combining multiple LoRA experts with a task-era-aware router (Tang et al., 1 Sep 2025). The other is a MeteoRA-based multi-LoRA framework in which multiple task-specific adapters are embedded into every adapted linear layer of a base LLM through a full-mode MoE architecture, with token-level expert selection driven by hidden states rather than explicit metadata (Xu et al., 2024).

Usage in the literature	Routing signal	Mixture pattern
Chinese IE Tea-MOELoRA	Task identity and era label	Dense mixture over experts
MeteoRA-grounded Tea-MOELoRA	Token-level hidden states	Top-1 or top-k per-layer routing

The common denominator is the replacement of a single shared low-rank update with a set of specialized low-rank experts. In both forms, the base model remains frozen and only lightweight adaptation components are trained. The principal difference is the source of routing information: metadata-conditioned routing in the Chinese IE setting versus hidden-state-conditioned routing in the MeteoRA-style formulation. A frequent misconception is therefore to treat Tea-MOELoRA as synonymous with a single routing policy. The literature instead supports both dense metadata-aware mixtures and sparse token-level routing.

2. Mathematical basis: LoRA, expertization, and routing

Tea-MOELoRA inherits the standard LoRA premise that a pretrained weight is adapted by a low-rank update rather than full dense fine-tuning. In the Chinese IE formulation, for a frozen linear map $W_0 \in R^{d_{in}\times d_{out}}$, the adapted weight is written as

$W' = W_0 + \alpha BA,$

where $B \in R^{d_{in}\times r}$ and $A \in R^{r\times d_{out}}$, with $r \ll \min(d_{in}, d_{out})$; the paper denotes the scaling factor by $\lambda$ (Tang et al., 1 Sep 2025). In the MeteoRA formulation, the same idea is expressed as

$$W' = W + \Delta W,\qquad \Delta W = (\alpha / r)BA,$$

with per-token forward decomposition into a frozen base term and a low-rank correction (Xu et al., 2024). The notational difference reflects paper-specific conventions rather than a substantive architectural divergence.

The MoE step replaces one low-rank branch with multiple low-rank experts. In the MeteoRA-style formulation, each layer exposes $n$ experts $\{(A_i,B_i)\}$ and computes

$$y(x) = xW_{base} + \sum_{i \in I(x)} p_i(x) f_i(x),$$

where $W' = W_0 + \alpha BA,$0, $W' = W_0 + \alpha BA,$1 is the top-$W' = W_0 + \alpha BA,$2 expert set for token $W' = W_0 + \alpha BA,$3, and $W' = W_0 + \alpha BA,$4 are softmax-normalized gate weights over selected experts (Xu et al., 2024). In the Chinese IE Tea-MOELoRA, the corresponding dense mixture is

$W' = W_0 + \alpha BA,$5

where $W' = W_0 + \alpha BA,$6 and $W' = W_0 + \alpha BA,$7 are task- and era-aware weights for expert $W' = W_0 + \alpha BA,$8 (Tang et al., 1 Sep 2025).

These formulations expose the two central axes along which Tea-MOELoRA variants differ. First, the router may be content-driven, as in token-level hidden-state gating, or metadata-driven, as in task- and era-conditioned expert weighting. Second, the mixture may be sparse, with only top-$W' = W_0 + \alpha BA,$9 experts active, or dense, with all experts contributing through soft weights. A plausible implication is that the name now covers a family of MoE-LoRA mechanisms whose shared goal is interference reduction under PEFT constraints.

3. Task-era-aware Tea-MOELoRA for Chinese information extraction

The 2025 Tea-MOELoRA paper introduces a parameter-efficient multi-task framework for Chinese information extraction across Classical and Modern documents, motivated by two forms of heterogeneity: task heterogeneity and temporal domain shift (Tang et al., 1 Sep 2025). The experimental setting spans relation extraction and event tasks using CHisRE and CHED for Classical Chinese, and DUIE and ACE2005 Chinese for Modern Chinese. Era labels are assigned accordingly: CHED and CHisRE are labeled Classical, while ACE2005 and DUIE are labeled Modern.

The architectural core is a frozen instruction-tuned LLM backbone augmented with multiple LoRA experts and a factorized task-era-aware router. The visualization indicates eight experts, indexed $B \in R^{d_{in}\times r}$0–$B \in R^{d_{in}\times r}$1, and the forward pass uses a dense mixture rather than top-$B \in R^{d_{in}\times r}$2 sparsification. Task identity features are obtained from learned task embeddings $B \in R^{d_{in}\times r}$3 and era identity features from learned era embeddings $B \in R^{d_{in}\times r}$4. Separate linear maps followed by softmax produce task and era weights,

$B \in R^{d_{in}\times r}$5

and the expert contribution for expert $B \in R^{d_{in}\times r}$6 is modulated multiplicatively by $B \in R^{d_{in}\times r}$7 (Tang et al., 1 Sep 2025). The paper reports that factorized routing with both signals is superior to task-only or era-only routing.

The training objective is deliberately simple. Tea-MOELoRA uses the main task loss alone,

$B \in R^{d_{in}\times r}$8

without auxiliary router regularization such as load-balancing or entropy penalties (Tang et al., 1 Sep 2025). This is an important point of contrast with several other MoE-LoRA architectures. The intended effect is to let expert specialization emerge end-to-end from mixed-task supervision while keeping the backbone frozen and training only adapter and router parameters.

The framework is explicitly positioned against the negative transfer observed in joint LoRA baselines. The reported comparison shows that joint LoRA across all datasets degrades relative to single-task LoRA, whereas Tea-MOELoRA recovers or improves performance by separating semantic task effects from diachronic era effects. Expert usage visualizations further indicate structured specialization: Classical samples favor Experts $B \in R^{d_{in}\times r}$9 and $A \in R^{r\times d_{out}}$0, Modern samples favor Experts $A \in R^{r\times d_{out}}$1 and $A \in R^{r\times d_{out}}$2, CHisRE emphasizes Expert $A \in R^{r\times d_{out}}$3, DUIE emphasizes Experts $A \in R^{r\times d_{out}}$4 and $A \in R^{r\times d_{out}}$5, CHED emphasizes Expert $A \in R^{r\times d_{out}}$6, and ACE2005 emphasizes Expert $A \in R^{r\times d_{out}}$7 (Tang et al., 1 Sep 2025).

4. MeteoRA-grounded Tea-MOELoRA and full-mode coverage

A second, broader formulation describes Tea-MOELoRA as an MoE-integrated multi-LoRA framework grounded in MeteoRA, whose central idea is to reuse a library of task-specific LoRA adapters by embedding them into all linear layers of a base LLM through a full-mode MoE architecture (Xu et al., 2024). “Full-mode” means that every adapted linear layer in both attention and MLP blocks receives its own gate. The seven adapted linear layers per Transformer block are $A \in R^{r\times d_{out}}$8, $A \in R^{r\times d_{out}}$9, $r \ll \min(d_{in}, d_{out})$0, $r \ll \min(d_{in}, d_{out})$1, $r \ll \min(d_{in}, d_{out})$2, $r \ll \min(d_{in}, d_{out})$3 (SiLU), and $r \ll \min(d_{in}, d_{out})$4.

In this design, each adapted linear layer contains an MoE over intact LoRA adapters, and only the gating networks are trained. The base LLM and all LoRA adapters remain frozen (Xu et al., 2024). For LLaMA3-8B, the paper notes that 32 decoder layers with 7 gates per layer yield 224 total MeteoRA modules. This arrangement is intended to support autonomous task sensing and timely adapter switching at token granularity: as hidden states evolve layer by layer, different gates may route the current token to different experts.

The MeteoRA-grounded version also emphasizes systems engineering. Naive token-level routing across many experts produces poor GPU utilization and memory thrashing, so the implementation introduces two accelerated forward paths adopted by Tea-MOELoRA: bmm-torch, which turns expert evaluation into large batched matrix multiplies but increases memory through per-token duplication of expert matrices, and bmm-triton, a fused Triton kernel that avoids such duplication while retaining most of the throughput benefit (Xu et al., 2024). The result is a practical distinction within Tea-MOELoRA itself: the conceptual model is multi-expert token routing, but deployability depends heavily on the kernel strategy.

Training in this formulation uses either a top-1 gate-selection objective or a joint objective combining causal language modeling and per-layer gate classification. The paper gives

$r \ll \min(d_{in}, d_{out})$5

and also describes a top-1 variant based on gate classification alone (Xu et al., 2024). A salient property is adapter reuse: all task-specific LoRA adapters are reused as-is, their $r \ll \min(d_{in}, d_{out})$6 weights are frozen, and the gating layers alone are fit on balanced gate-training data.

5. Empirical results, routing behavior, and efficiency

In the Chinese information extraction setting, Tea-MOELoRA is reported to outperform both joint LoRA baselines and other LoRA+MoE baselines across the mixed Classical/Modern benchmark suite (Tang et al., 1 Sep 2025). The reported scores for Tea-MOELoRA are CHisRE F1 $r \ll \min(d_{in}, d_{out})$7, DUIE F1 $r \ll \min(d_{in}, d_{out})$8, CHED Tri-I/Tri-C $r \ll \min(d_{in}, d_{out})$9, and ACE2005 Tri-I/Tri-C/Arg-I/Arg-C $\lambda$0. By comparison, the MOELoRA baseline reports CHisRE F1 $\lambda$1, DUIE F1 $\lambda$2, CHED Tri-I/Tri-C $\lambda$3, and ACE2005 Tri-I/Tri-C/Arg-I/Arg-C $\lambda$4, while joint LoRA (“LoRA Mix”) underperforms single-task LoRA. The ablation results also show that using both task and era signals is better than task-only or era-only routing, and that four task IDs at dataset granularity are better than collapsing the problem into only two task IDs, RE versus EE.

In the MeteoRA-grounded setting, evaluation on LLaMA2-13B and LLaMA3-8B with 28 LoRA adapters shows single-task performance close to conventional PEFT while also improving composite-task execution (Xu et al., 2024). For single-task averages across 28 tasks, Tea-MOELoRA (MeteoRA top-1-1k) reports accuracy $\lambda$5, BLEU $\lambda$6, and ROUGE-L $\lambda$7 for LLaMA2-13B/LLaMA3-8B. The same system is also reported to solve ten sequential problems in a single inference pass with timely adapter switching, and gate visualizations show dominant weights on correct adapters with smooth switching at task boundaries across both input understanding and output generation phases.

The efficiency profile is equally central. In the MeteoRA line, naive loop-based routing can be up to $\lambda$8 slower than single-adapter forward when $\lambda$9 and $$W' = W + \Delta W,\qquad \Delta W = (\alpha / r)BA,$$0 in autoregressive inference, whereas bmm-torch yields $$W' = W + \Delta W,\qquad \Delta W = (\alpha / r)BA,$$1 speedup over loop-original and bmm-triton retains roughly $$W' = W + \Delta W,\qquad \Delta W = (\alpha / r)BA,$$2 of that speed with memory footprint comparable to loop-original (Xu et al., 2024). Gate fitting on the 28-task setup is reported to converge in $$W' = W + \Delta W,\qquad \Delta W = (\alpha / r)BA,$$3 hours on $$W' = W + \Delta W,\qquad \Delta W = (\alpha / r)BA,$$4 H800 (80 GB), and few-shot gate training with 100 or even 5 samples per task still works reasonably. These results clarify that Tea-MOELoRA is not merely a representational device; it is also a routing-and-kernel co-design problem.

6. Relation to the broader MoE-LoRA literature and unresolved issues

Tea-MOELoRA belongs to a rapidly expanding MoE-LoRA lineage. MoELoRA frames LoRA modules themselves as experts, uses Top-2 routing by default, and adds both a Switch-style load-balancing loss and an InfoNCE-style experts contrastive loss to reduce random routing; under equal trainable parameter budgets, it reports a $$W' = W + \Delta W,\qquad \Delta W = (\alpha / r)BA,$$5 average accuracy improvement over LoRA on math reasoning and a $$W' = W + \Delta W,\qquad \Delta W = (\alpha / r)BA,$$6 improvement on common-sense reasoning (Luo et al., 2024). MixLoRA inserts multiple LoRA-based experts within FFN blocks of a frozen dense model, pairs them with a top-$$W' = W + \Delta W,\qquad \Delta W = (\alpha / r)BA,$$7 router and an auxiliary load-balance loss, and reports about $$W' = W + \Delta W,\qquad \Delta W = (\alpha / r)BA,$$8 accuracy improvement in multi-task learning scenarios, alongside a high-throughput framework that reduces GPU memory consumption by $$W' = W + \Delta W,\qquad \Delta W = (\alpha / r)BA,$$9 and token computation latency by $n$0 during training and inference (Li et al., 2024). LLaVA-MoLE transfers the idea to multimodal instruction finetuning, using top-1 sparse routing in FFN LoRA adapters to mitigate data conflicts across mixed image-text instruction datasets (Chen et al., 2024). Medical MOELoRA instead uses a task-motivated gate and per-task parameter recovery, emphasizing multi-task medical applications while keeping total trainable parameters equal to LoRA (Liu et al., 2023).

More recent variants move in different directions. TT-LoRA MoE decouples expert training from router training and selects exactly one frozen TT-LoRA expert per input, reporting that TT-LoRA experts use only $n$1 of LoRA parameters and that the router uses $n$2 of AdapterFusion parameters (Kunwar et al., 29 Apr 2025). CoMoL relocates expert diversity into compact $n$3 core matrices and routes in the same low-rank space, aiming for token-level adaptivity with parameter efficiency comparable to standard LoRA (Cao et al., 28 Feb 2026). TalkLoRA adds a communication-aware “Talking Module” before routing, thereby relaxing the assumption that experts are independent and empirically improving routing balance under comparable parameter budgets (Mu et al., 7 Apr 2026). HLoRA extends hierarchical LoRA-MoE design to CTC-based multilingual ASR with LID-posterior-driven routing, while MoRAL applies FFN-level LoRA experts with top-2 routing to lifelong learning over QA pairs (Zheng et al., 2 Jan 2026, Yang et al., 2024).

Against this background, Tea-MOELoRA is notable for exposing a fundamental design bifurcation inside MoE-LoRA research. One branch, exemplified by the Chinese IE formulation, uses explicit supervision signals such as task and era metadata to stabilize routing and disentangle structured sources of interference (Tang et al., 1 Sep 2025). The other, exemplified by MeteoRA-style full-mode routing, attempts autonomous task sensing directly from hidden states and thereby targets dynamic multi-subtask execution without explicit task selection (Xu et al., 2024). This distinction also marks the main unresolved issues. The Chinese IE formulation relies on accurate task and era labels and does not use explicit load balancing; the MeteoRA formulation incurs heavier routing and batched-matmul cost than single-adapter inference, requires gate retraining when adapters change, and may exhibit OOD miscalibration or negative interference under top-$n$4 blending. A plausible implication is that future Tea-MOELoRA-style systems will continue to trade off metadata supervision, autonomous routing, sparsity, calibration, and kernel efficiency rather than converging on a single universally dominant recipe.