MoshiRAG: Real-Time Duplex Speech LM

• MoshiRAG is a modular full-duplex speech language model framework that combines a compact streaming core with asynchronous retrieval of up-to-date knowledge.
• It minimizes hallucinations by selectively retrieving and injecting concise factual references during conversational dialogue.
• Empirical evaluations show significant improvements in factual accuracy and latency over traditional large, turn-based speech models.

MoshiRAG is a modular full-duplex speech LLM framework that enables real-time factuality by combining a compact streaming core with asynchronous retrieval of external knowledge. The system is designed to overcome the challenge of hallucinations and outdated information in end-to-end speech models while preserving the low latency and continuous interactivity required for natural conversational dialogue. Unlike large parameter models, which suffer from prohibitive computational costs and increased inference delay, MoshiRAG uses selective, plug-and-play external retrieval to ground its responses in up-to-date information, all within stringent real-time constraints (Chien et al., 14 Apr 2026).

1. Problem Context and Motivation

Full-duplex speech LLMs, such as Moshi, are characterized by their simultaneous listening and speaking capabilities. This architecture supports conversational phenomena including pauses, interruptions, and backchannels, producing highly interactive systems. However, these models, being trained end-to-end on limited speech data, tend to hallucinate or provide outdated content—especially for knowledge-intensive queries. Previous approaches have attempted to mitigate this via turn-based or cascaded retrieval-augmented pipelines; however, such designs introduce turn-taking delays, compromising the seamless nature of duplex dialogue.

Scaling model size can—in principle—increase factuality, as seen in large text-only LMs (e.g., from 7B to 70B parameters), but this approach is infeasible for streaming speech due to excessive acoustic-to-audio latency, prohibitive memory and compute requirements, and poor energy efficiency in real-time applications. MoshiRAG addresses these limitations with a modular asynchronous retrieval layer, allowing the deployment of a lightweight duplex speech core (7B parameters) capable of accessing more powerful knowledge sources on demand without the computational burden of a monolithic large model.

2. System Architecture

MoshiRAG comprises two loosely coupled pipelines orchestrated for real-time streaming and asynchronous evidence integration:

• Front-end (Real-time):
  • Streaming ASR (1B parameters) produces a live transcript of user speech with approximately 0.5 s latency.
  • The compact Moshi full-duplex LM (7B parameters) processes both user codec tokens and its generated output concurrently, emitting early-stage ("lead") responses.
  • At knowledge-sensitive junctures, the model emits a special retrieval trigger token, $\langle\mathrm{RAG}\rangle$.
• Back-end (Asynchronous):
  • Upon emitting $\langle\mathrm{RAG}\rangle$, the current transcript is passed to a retrieval engine, which may employ an LLM or web search backend.
  • The retrieval engine is constrained to a $\leq$2 s budget and returns a concise reference text.
  • The ARC-Encoder compresses this reference into a short embedding stream, which is then injected additively into the model’s temporal Transformer input at the correct streaming timesteps. This enables the model to continue generating output while asynchronously awaiting external knowledge, maintaining the conversational flow.

3. Query Demand Detection

MoshiRAG obviates the need for a separate classifier by training the duplex LM to emit $\langle\mathrm{RAG}\rangle$ when external retrieval is warranted. Formally, given user text tokens $u_{1:M}$ and context $c$, the model estimates

$$f(u_{1:M},c) = P(\langle\mathrm{RAG}\rangle \mid u_{1:M},c)$$

Retrieval is triggered when $f(u_{1:M},c) > \tau$ with $\tau \approx 0.5$. Synthetic RAG training data is constructed, annotating each knowledge-intensive turn by replacing the first token of the “lead” segment with $\langle\mathrm{RAG}\rangle$. The model retains the original architecture, adding only the embedding and softmax row for the special token, and optimizes a standard cross-entropy loss targeting precise RAG triggering.

4. Asynchronous Retrieval Scheduling

MoshiRAG exploits the natural latency between user utterance and response “body” in typical speech LMs, termed end-to-end keyword delay (E2EKD), which often exceeds 3 s. The retrieval subsystem is engineered to return the external evidence within $\langle\mathrm{RAG}\rangle$0 s, ensuring that the knowledge context is injected before the model emits core content. The asynchronous scheduling operates as follows:

1. The model emits $\langle\mathrm{RAG}\rangle$1 at time $\langle\mathrm{RAG}\rangle$2.
2. A short (0.5 s) ASR buffer ensures transcript stability prior to retrieval.
3. Retrieval executes in a parallel thread, while the model continues streaming pre-RAG content.
4. Upon completion, the compressed reference embedding is streamed into the Transformer over the next $\langle\mathrm{RAG}\rangle$3 frames at 12.5 Hz.

Training data simulates varying retrieval delays $\langle\mathrm{RAG}\rangle$4 to expose the system to both fast and slow retrieval edge cases, enhancing its robustness.

5. Retrieval Back-ends and Scoring

MoshiRAG’s retrieval subsystem is agnostic to the backend, supporting seamless integration of various methods:

• LLM-based retrievers: e.g., Gemma 3 27B and GPT-4.1, which are prompted to generate concise factual references.
• Search-based retrievers: e.g., Tavily, which scrape and summarize web content.

Dense-retrieval is, in principle, supported via typical scoring functions such as

$\langle\mathrm{RAG}\rangle$5

but in practice, retrieval is delegated to external APIs.

6. Knowledge Injection and Response Synthesis

When retrieval concludes, the ARC-Encoder’s compressed reference embedding $\langle\mathrm{RAG}\rangle$6 is additively injected into the Moshi temporal Transformer input for subsequent timesteps:

$\langle\mathrm{RAG}\rangle$7

where $\langle\mathrm{RAG}\rangle$8 is the original input embedding and $\langle\mathrm{RAG}\rangle$9 Hz. This operation can be conceptualized as a mixture-of-experts augmentation of the token generation process but is executed by directly streaming the reference into self-attention.

7. Algorithmic Flow

The following high-level pseudocode outlines MoshiRAG’s control flow:

$\leq$1 The asynchronous retrieval must complete within $\leq$0 s to meet the overall end-to-end latency budget.

8. Empirical Results and Analysis

MoshiRAG exhibits significant factuality improvements on standard spoken QA benchmarks (LlamaQ, WebQ, TriviaQA, HaluEval). The baseline Moshi (7B) achieves 26–62% accuracy; with a Gemma3 retrieval backend, MoshiRAG attains 74–80%, approaching GPT-4o Audio performance. Swapping to GPT-4.1 or Tavily further increases accuracy by 5–15%, confirming retrieval-agnostic gains.

Latency metrics indicate time-to-first-audio-token (TTFAT) near zero, keyword delay around 3.1 s, and end-to-end keyword delay (E2EKD) ≈ 3.1 s including retrieval. Compute cost is approximately 0.37 TFLOPs/s, on par with vanilla Moshi (0.22 TFLOPs/s) and substantially below large turn-based models (>4 TFLOPs/s).

On the Full-Duplex-Bench v1.0 (Table 2), MoshiRAG achieves lower takeover rates (TOR) and maintains or improves backchannel frequencies, turn-taking, and interruption recovery metrics relative to the baseline. For out-of-domain mathematical reasoning (AddSub, MultiArith, SingleEq, SVAMP, GSM8K), MoshiRAG increases response accuracy from ≤20% (vanilla) to 33–69% (Gemma3) and up to 76% with GPT-4.1. For comparison, STITCH-S (math-specialized) achieves 91% but at the cost of much higher model complexity.

Sensitivity and ablation results show that moderate ASR WERs (≤11%) have minimal impact on RAG triggering, but retrieval delay beyond 1.5 s degrades accuracy sharply. Evaluations of additive vs. insertive embedding injection, ARC-Encoder compression ratio, and backend swapping affirm the robustness of the chosen design.

In sum, MoshiRAG establishes that a compact full-duplex speech LM can attain near state-of-the-art factuality through asynchronous retrieval-driven evidence integration, all while retaining the low-latency, highly interactive features essential for fluid spoken dialogue (Chien et al., 14 Apr 2026).