Continual Retrieval-Augmented MoE LLM

• C‑RAG‑LLM is a dynamic LLM combining continual retrieval, sparse Mixture-of-Experts, and ongoing adaptation to handle evolving data.
• It employs a modular architecture with dynamic database, retrieval, integration, and adaptation layers for efficient knowledge updates.
• The design balances parametric memory with external retrieval, enabling adaptive expert routing and effective long-term knowledge management.

Searching arXiv for papers on continual retrieval-augmented MoE-based LLMs and closely related adaptive RAG/MoE mechanisms. Search query: "continual retrieval augmented mixture of experts LLM ExpertRAG UrbanMind adaptive RAG MoE" Continual Retrieval-Augmented MoE-based LLM (C‑RAG‑LLM) denotes a class of LLM architectures that combine retrieval-augmented generation, sparse Mixture-of-Experts computation, and continual adaptation of retrieval, memory, and model components. In UrbanMind, C‑RAG‑LLM is the core model architecture for urban general intelligence, with continual retrieval, continual knowledge base and index updates, and continual model adaptation operating on different time scales; in parallel, ExpertRAG supplies a closely aligned theoretical formulation in which retrieval and expert selection are treated as latent decisions mediated by gating (Yang et al., 7 Jul 2025, Gumaan, 23 Mar 2025).

1. Conceptual scope and relation to adjacent paradigms

C‑RAG‑LLM sits at the intersection of three established lines of work. From retrieval-augmented generation it inherits the use of external, non-parametric knowledge stores; from Mixture-of-Experts it inherits sparse routing among specialized feed-forward sub-networks; from continual learning it inherits repeated adaptation under distribution shift rather than one-shot static training. UrbanMind makes this explicit by placing C‑RAG‑LLM inside a four-layer stack consisting of a dynamic database layer, a retrieval layer, an integration layer, and an adaptation layer, with retriever strategy updates occurring at the fastest time scale, knowledge base updates at an intermediate time scale, and LLM or expert fine-tuning at the slowest time scale (Yang et al., 7 Jul 2025).

This positioning distinguishes C‑RAG‑LLM from standard RAG, pure MoE, and long-context memory. Standard RAG generally retrieves top-$k$ documents for every query and uses a single monolithic generator; pure MoE stores knowledge parametrically and requires re-training or fine-tuning to update it; long-context approaches push large memory directly into the prompt and are reported to be stronger on “big picture” queries while being more expensive, whereas RAG is more competitive on specific queries when keywords can be explicitly matched (Gumaan, 23 Mar 2025, Wei et al., 20 Mar 2025). C‑RAG‑LLM is therefore not simply “RAG plus experts.” Its defining property is that the balance between parametric memory and retrieved knowledge is itself adaptive and, in the continual case, repeatedly recalibrated as the corpus, query distribution, and task mix evolve.

A recurrent theme in the literature is modularity. ExpertRAG emphasizes that the retrieval module can be updated continuously, experts can be added or extended over time, and the gating mechanism can be retrained or adapted online to reflect changing data distributions (Gumaan, 23 Mar 2025). A plausible implication is that C‑RAG‑LLM is best understood less as a single fixed architecture than as a modular systems pattern for long-lived LLM deployment.

2. Core architecture and probabilistic formulation

ExpertRAG gives the clearest generic architectural decomposition for a C‑RAG‑LLM. It consists of a retrieval module, a Mixture-of-Experts generator, and a gating mechanism that decides whether to retrieve and routes tokens to internal experts. Given query $q$, retrieval gating computes

$P_{\text{ret}}(q)=\sigma(w^\top h_q+b),$

samples $Z_{\text{ret}}\in\{0,1\}$, retrieves top-$k$ documents $D_q=\{d_1,\dots,d_k\}$ if $Z_{\text{ret}}=1$, forms the model input as either $X_{\text{inp}}=[q;d_1;\dots;d_k]$ or $X_{\text{inp}}=[q]$, and then runs a Transformer whose subset $L_{\text{MoE}}$ contains MoE layers with sparse top-$q$0 routing (Gumaan, 23 Mar 2025).

Within each MoE layer, the router computes expert scores

$q$1

and selects either top‑1 or top‑2 experts. For top‑1 routing, $q$2 and the output is $q$3; for top‑2 routing the model forms a weighted combination over the two selected experts. ExpertRAG explicitly characterizes both retrieval and expert selection as latent decisions, with retrieval latent $q$4 and per-layer, per-token expert latents $q$5, leading to a conceptual marginalization over retrieval choice, retrieved documents, and expert routes in $q$6 (Gumaan, 23 Mar 2025).

UrbanMind provides the continual systems view of the same structure. For query $q$7 at time $q$8 and task descriptor $q$9, the task-aware retriever returns

$P_{\text{ret}}(q)=\sigma(w^\top h_q+b),$0

the integration layer forms

$P_{\text{ret}}(q)=\sigma(w^\top h_q+b),$1

and the MoE-based adaptation layer routes $P_{\text{ret}}(q)=\sigma(w^\top h_q+b),$2 through sparse experts before producing $P_{\text{ret}}(q)=\sigma(w^\top h_q+b),$3 (Yang et al., 7 Jul 2025). This formalization makes the continual aspect explicit: the corpus $P_{\text{ret}}(q)=\sigma(w^\top h_q+b),$4, retriever $P_{\text{ret}}(q)=\sigma(w^\top h_q+b),$5, and parameters $P_{\text{ret}}(q)=\sigma(w^\top h_q+b),$6 are all time-indexed.

A notable architectural idea in ExpertRAG is that retrieval may be conceptualized as a first-class “pathway” analogous to an expert, even if the paper’s main formulation keeps a separate retrieval gate. The paper explicitly notes that retrieval could be unified into a single MoE-like router that routes between internal parametric experts and a retrieval “expert” (Gumaan, 23 Mar 2025). This suggests a direct path from conditional RAG to a unified MoE-with-tools formulation.

3. Retrieval control, expert behavior, and internal signals

The control problem in C‑RAG‑LLM is deciding when to trust internal experts, when to retrieve, and how strongly to use retrieved context once it is available. ExpertRAG proposes an explicit binary retrieval gate trained either with supervised labels, reinforcement learning, straight-through estimators, or heuristics that compare generator loss with and without retrieval; it also recommends a retrieval cost penalty,

$P_{\text{ret}}(q)=\sigma(w^\top h_q+b),$7

to discourage always-on retrieval (Gumaan, 23 Mar 2025). The gate may use query embeddings and optional uncertainty signals such as output entropy, and the paper explicitly frames this as a trade-off between parametric knowledge and non-parametric retrieval.

Other work refines this control layer by reading internal MoE or representation signals rather than relying only on external retrieval scores. “Unveiling and Consulting Core Experts in Retrieval-Augmented MoE-based LLMs” identifies three core expert types inside Mixtral and Qwen-MoE systems: cognizant experts indicating whether internal knowledge is sufficient, quality experts tracking whether retrieved documents are high-quality or low-quality, and in-context experts governing the degree to which external context is used. Its CEAI method computes contrastive activation $P_{\text{ret}}(q)=\sigma(w^\top h_q+b),$8 between positive and negative behavioral scenarios and aggregates those activations into a scenario score

$P_{\text{ret}}(q)=\sigma(w^\top h_q+b),$9

which can then be thresholded for retrieval control or document filtering (Zhou et al., 2024). On Mixtral‑8x22B, cognizant experts reached F1 75.40 for knowledge-sufficiency prediction on RGBqa, and quality experts reached 93.00 accuracy for distinguishing high-quality from unrelated documents (Zhou et al., 2024).

CtrlA develops a complementary representation-centric approach that is not MoE-specific but is directly relevant to C‑RAG‑LLM. It extracts honesty and confidence directions from hidden states using Representation Engineering, uses honesty steering to bias activations toward admitting ignorance rather than fabricating, and uses confidence monitoring to trigger retrieval when newly generated information is internally low-confidence (Liu et al., 2024). Its retrieval trigger is activated if there exists a new-information token with normalized confidence score $Z_{\text{ret}}\in\{0,1\}$0, and it combines this with query reformulation strategies such as Context-Augmented Querying and Targeted Validation Querying. On PopQA, CtrlA raised accuracy to 61.8 compared with 51.9 for single-time RAG and 25.7 without retrieval; on FreshQA it achieved 48.4 relaxed accuracy and 43.8 strict accuracy (Liu et al., 2024).

CRAG adds an explicit retrieval evaluator that classifies retrieval outcomes into Correct, Incorrect, or Ambiguous, triggering either internal document refinement, web search, or both (Yan et al., 2024). This design is especially relevant for continual settings because evolving corpora make retrieval quality non-stationary. Taken together, these works indicate that the controller in C‑RAG‑LLM can be implemented through at least four signal families: explicit retrieval gates, internal expert-activation statistics, latent representation probes, and external retrieval evaluators. A plausible implication is that robust continual systems will combine these signals rather than depend on only one.

4. Continual memory, evidence accumulation, and knowledge transfer

The “continual” component of C‑RAG‑LLM concerns both the external knowledge substrate and the way retrieved evidence is accumulated over time. UrbanMind defines incremental corpus updating as repeated ingestion, indexing, pruning, and validation of the evolving knowledge base $Z_{\text{ret}}\in\{0,1\}$1, with timestamping, task relevance scores, uncertainty estimates, temporal decay functions, redundancy detection, and lightweight validation filters before new entries become retrievable (Yang et al., 7 Jul 2025). This is the clearest explicit treatment of long-lived corpus maintenance.

RFM‑RAG contributes a stateful multi-round retrieval mechanism based on a dynamic evidence pool $Z_{\text{ret}}\in\{0,1\}$2. Starting from $Z_{\text{ret}}\in\{0,1\}$3, each retrieval round retrieves passages $Z_{\text{ret}}\in\{0,1\}$4, curates them with an LLM into $Z_{\text{ret}}\in\{0,1\}$5, and appends them to the evidence pool; an R‑Feedback model then uses syntactic coverage $Z_{\text{ret}}\in\{0,1\}$6 and semantic relevance $Z_{\text{ret}}\in\{0,1\}$7 to predict whether the current evidence is sufficient and whether retrieval should stop (Li et al., 25 Aug 2025). The final answer is generated from $Z_{\text{ret}}\in\{0,1\}$8 and the accumulated $Z_{\text{ret}}\in\{0,1\}$9, not just the latest retrieval. This turns retrieval from a stateless top‑$k$0 lookup into stateful continuous knowledge management.

RAL pushes the same idea toward autonomous external memory construction. It organizes learning into three databases—Hypothesis DB $k$1, Validation DB $k$2, and Experience DB $k$3—with RAG-style retrieval and storage primitives. Hypotheses are proposed from transitions, validated against later observations, and consolidated into reusable experience entries, so learning occurs through accumulation and retrieval of validated textual knowledge rather than parameter updates (Li et al., 2 May 2025). This is not described as lifelong cross-session memory in the paper, but it provides a concrete template for persistent validated-knowledge stores inside a continual retrieval system.

These proposals imply several levels of memory in C‑RAG‑LLM. There is immediate retrieved context, session-level evidence accumulation, and longer-lived external knowledge bases that can be updated independently of the model. ExpertRAG explicitly notes a future direction in which knowledge frequently outsourced to retrieval could be shifted into parametric experts through periodic fine-tuning, while knowledge that becomes volatile or recent could remain external (Gumaan, 23 Mar 2025). This suggests a recurrent cycle of retrieval, accumulation, consolidation, and re-routing between non-parametric and parametric memory.

5. Optimization, efficiency, and parametric–non-parametric balance

ExpertRAG formulates the core computational trade-off of C‑RAG‑LLM. If $k$4 is the fraction of queries that invoke retrieval, $k$5 is retrieval cost over index size $k$6, and $k$7 is the cost of processing retrieved context, then expected query cost is

$k$8

compared with always-on RAG cost $k$9 on every query (Gumaan, 23 Mar 2025). Because MoE activates only $D_q=\{d_1,\dots,d_k\}$0 experts per token while storing many more total parameters, capacity can grow through sparse experts without linear growth in per-token computation.

UrbanMind casts continual C‑RAG‑LLM as a multilevel optimization problem. At the outer level, retriever parameters are optimized against validation loss; at the middle level, generator or expert parameters are optimized on retrieval-conditioned inputs; and, in a distributionally robust extension, domain weights $D_q=\{d_1,\dots,d_k\}$1 are optimized subject to $D_q=\{d_1,\dots,d_k\}$2 and a KL constraint to prevent overconcentration on one domain (Yang et al., 7 Jul 2025). The framework allows end-to-end training or partial layer-wise optimization, which is central for resource-constrained cloud–edge deployment.

ExpertRAG describes a practical training path consisting of pretraining a base MoE LLM, pretraining a retriever, and then jointly fine-tuning generator, retriever, and retrieval gate with answer loss, optional gate supervision or policy gradient, load-balancing loss for experts, and retrieval sparsity penalties (Gumaan, 23 Mar 2025). RFM‑RAG, by contrast, keeps retrieval iterative and uses a lightweight feedback model with binary cross-entropy to learn when evidence is sufficient (Li et al., 25 Aug 2025). RAL goes further toward train-free continual adaptation, treating retrieval as a module for organizing intermediate knowledge without model training (Li et al., 2 May 2025).

A related but distinct line is parametric consolidation. “Tuning LLMs by RAG Principles: Towards LLM-native Memory” uses GraphRAG to generate local and global query–answer data, then fine-tunes Qwen‑2‑7B‑Instruct with LoRA so that, at inference time, the model answers without external retrieval (Wei et al., 20 Mar 2025). For C‑RAG‑LLM, this does not replace continual retrieval, but it provides a mechanism for transferring stable external knowledge into experts or adapters. A plausible implication is that continual systems can alternate between fast non-parametric updates for freshness and slower parametric distillation for latency and recurrent knowledge.

6. Evaluation, limitations, and open research directions

The evaluation literature around C‑RAG‑LLM spans knowledge-intensive QA, reasoning, factuality, efficiency, and distribution shift. ExpertRAG proposes KILT-style benchmarks including NQ, HotpotQA, 2WikiMultiHopQA, WikiHop, TriviaQA, FEVER, ELI5, MMLU, BBH, and ARC, with metrics covering accuracy or exact match, factuality and hallucination rate, FLOPs per query, latency, fraction of queries with retrieval, and expert-usage distributions (Gumaan, 23 Mar 2025). It also explicitly recommends corpus update experiments and distribution-shift experiments, both of which are central to continual retrieval systems.

Several papers provide quantitative evidence for adaptive control rather than always-on retrieval. In the core-expert line, Expert-RAG on Mixtral‑8x7B reached 59.00 accuracy and 64.70 R‑Score on BalanceQA, while the Mixtral‑8x22B version reached 65.75 accuracy and 69.25 R‑Score; on PubHealth with Mixtral‑8x22B, Expert-RAG (C+Q) achieved 80.95 accuracy with 76,455 R‑Token, while Always RAG reached 57.54 accuracy (Zhou et al., 2024). CRAG reports that, with SelfRAG-LLaMA2-7b as generator, Self-CRAG reached 61.8 on PopQA versus 54.9 for Self-RAG, and CRAG raised PopQA from 40.3 for RAG to 59.3 while also improving Biography, PubHealth, and ARC-Challenge (Yan et al., 2024). RFM‑RAG reports average EM 32.7 and ACC 50.7 with Mistral‑7B, compared with 19.2 and 42.7 for Vanilla RAG, and shows that R‑Feedback reduces average retrieval steps by 12–35% (Li et al., 25 Aug 2025).

UrbanMind presents system-level evidence on real-world urban tasks, reporting that continual RAG produced the most accurate and satisfactory responses across task levels relative to LLM-only and static RAG baselines, and that the tool-enhanced variant generated more detailed, context-aware travel plans by invoking weather, time, and traffic tools (Yang et al., 7 Jul 2025). These results are qualitative rather than fully tabulated, but they are important because they move C‑RAG‑LLM from question-answering benchmarks to long-lived operational settings.

The literature also converges on a set of limitations. ExpertRAG emphasizes training complexity, variance from RL-based gate learning, gate failures that either skip needed retrieval or over-retrieve, and brittleness or imbalance in MoE routing (Gumaan, 23 Mar 2025). Core-expert methods are limited by MoE-only applicability, static-dataset evaluation, partial controllability, imperfect quality experts, and hyperparameter sensitivity in CEAI (Zhou et al., 2024). UrbanMind adds system-level concerns: multilevel optimization is computationally heavy, retrieval and tool use add latency, expert imbalance remains a standard MoE problem, and dynamic knowledge bases may still be incomplete or biased (Yang et al., 7 Jul 2025). RFM‑RAG notes that memory is only per-session, not cross-session, and that evidence-pool growth is not aggressively controlled (Li et al., 25 Aug 2025).

Open directions follow directly from these constraints. Retrieval may be unified as a standard expert inside the MoE router, multi-step retrieval may be planned during generation rather than applied only once, and external tool selection may be extended beyond search to calculator, code interpreter, or vision experts (Gumaan, 23 Mar 2025). Cross-session consolidation of session-level evidence pools, automated recalibration of retrieval thresholds and expert roles under drift, and scalable memory hierarchies that separate per-turn, per-episode, and global knowledge are repeatedly suggested by the supporting literature (Li et al., 25 Aug 2025, Li et al., 2 May 2025). The cumulative picture is that C‑RAG‑LLM is not a settled architecture but an emerging systems framework for adaptive allocation between sparse parametric expertise and continually updated external knowledge.