MoshiRAG: Asynchronous Knowledge Retrieval for Full-Duplex Speech Language Models

Abstract: Speech-to-speech LLMs have recently emerged to enhance the naturalness of conversational AI. In particular, full-duplex models are distinguished by their real-time interactivity, including handling of pauses, interruptions, and backchannels. However, improving their factuality remains an open challenge. While scaling the model size could address this gap, it would make real-time inference prohibitively expensive. In this work, we propose MoshiRAG, a modular approach that combines a compact full-duplex interface with selective retrieval to access more powerful knowledge sources. Our asynchronous framework enables the model to identify knowledge-demanding queries and ground its responses in external information. By leveraging the natural temporal gap between response onset and the delivery of core information, the retrieval process can be completed while maintaining a natural conversation flow. With this approach, MoshiRAG achieves factuality comparable to the best publicly released non-duplex speech LLMs while preserving the interactivity inherent to full-duplex systems. Moreover, our flexible design supports plug-and-play retrieval methods without retraining and demonstrates strong performance on out-of-domain mathematical reasoning tasks.

• The paper introduces an asynchronous RAG system that lets a compact 7B Moshi speech model retrieve information while speaking, using a 1B streaming ASR model and a pluggable back end within a roughly 2-second retrieval target.
• MoshiRAG raises TriviaQA accuracy from 22.8% to 69.6% with Gemma and up to 78.2% with GPT-4.1, while reaching 51.3% on HaluEval, using 0.37 TFLOPs/sec and preserving a 0.18-second interruption-response latency.
• The paper shows that retrieval latency, ASR errors, and lossy reference integration remain major bottlenecks, motivating adaptive triggering, stronger reference encoding, and robust multi-turn evaluation for real-time spoken agents.

MoshiRAG extends the full-duplex speech LLM Moshi with retrieval-augmented generation (RAG), addressing a tension that has limited native audio models: they lag text LLMs in factuality, yet scaling them to close the gap would break real-time inference. The system keeps a compact 7B full-duplex front end and delegates knowledge-intensive content to an asynchronous retrieval back end, exploiting the natural delay between response onset and the delivery of key information. The result is factuality comparable to the strongest publicly released non-duplex speech models while preserving duplex interactivity (2604.12928).

Timing constraints as the design principle

The central observation is that spoken responses contain a "keyword delay": the interval between the start of the model's speech and the appearance of the word that directly answers the user's query. The paper defines time-to-first-audio-token (TTFAT), keyword delay (KD), end-to-end keyword delay (E2EKD = TTFAT + KD), and retrieval delay. For retrieved information to be usable, retrieval must complete before the keyword is spoken; since E2EKD of existing speech LMs often exceeds 3 seconds, the authors target a retrieval delay of at most 2 seconds. This constraint governs both training data construction and back-end selection.

System architecture

MoshiRAG consists of three components communicating in text: a 7B Moshi model fine-tuned for RAG, a 1B streaming ASR model with 0.5-second latency for transcribing user speech, and a pluggable retrieval back end. Moshi's RQ-Transformer operates at 12.5 Hz over Mimi codec tokens; the only architectural changes are a special retrieval trigger token and a reference text encoder. When the trigger is predicted, the conversation transcript is sent to the back end while Moshi continues speaking "pre-RAG" content — filler phrases and coarse answers. Once retrieval completes, the reference document is encoded by ARC-Encoder (a 4× sequence compression network) and its embeddings are projected through a linear layer and summed into the temporal Transformer input over successive frames. Two back ends are evaluated: a locally hosted Gemma 3 27B prompted to produce concise factual references, and Tavily web search for open-domain QA.

An appendix ablation justifies these choices: insertive injection of references outperforms additive injection but inflates input sequence length, and with T5 encoders additive injection collapses (e.g., 28.5% vs. 82.0% on TriviaQA) because long encoded sequences arrive too late. ARC-Encoder with compression ratio four and additive injection is adopted as the default, trading some integration effectiveness for sustainable long-form conversation length — a trade-off the authors acknowledge remains unresolved.

Synthetic data and training

Training relies on approximately 1.9M synthetic conversations (~47.8k hours). Topics come from Natural Questions, HotpotQA, and TriviaQA training splits (~474k topics) plus 111 LLM-curated expert domains. Three role-played Gemma 3 27B agents generate multi-turn scripts: a user agent without access to references (preventing leakage), a Moshi agent, and a reference agent. RAG-enabled turns follow a lead–body–tail structure so references can be injected before the body during training. Scripts are converted to audio with a dual-speaker TTS model, and forced alignment places the trigger token; retrieval delays are sampled to guarantee at least one second of buffer before key content, with a 0.2 fallback probability covering edge cases. Training runs 100k updates at learning rate $2 \times 10^{-6}$, with 0.2 dropout on references replaced by a learnable embedding when dropped.

Factuality results

Evaluated with LLM judges on OpenAudioBench-style protocols (Llama Questions, Web Questions, TriviaQA) plus a synthesized HaluEval-audio set, MoshiRAG improves dramatically over vanilla Moshi: TriviaQA rises from 22.8% to 69.6% and HaluEval from 10.5% to 36.3% with the Gemma back end. A vanilla Moshi fine-tuned on the same RAG data without actual retrieval gains little (TriviaQA 29.7%), attributing the improvement to retrieval rather than data exposure. With GPT-4.1 or Tavily back ends — swapped in without retraining — MoshiRAG reaches 78.2% and 77.5% on TriviaQA and 51.3% and 47.0% on HaluEval respectively, exceeding all compared speech LMs except GPT-4o Audio. Notably, this is achieved at 0.37 TFLOPs/sec, lower than most competitors, and with an E2EKD of 3.1 seconds versus 4.3–10.4 seconds for comparable systems.

A consistent finding is a roughly 5-point gap between reference accuracy and final response accuracy, indicating information loss during knowledge integration; the reference score also marks a hard upper bound set by the back end. Sensitivity analysis shows the pipeline is highly sensitive to ASR errors: substituting ground-truth transcriptions improves response accuracy by up to 15% on TriviaQA, and providing HaluEval ground-truth references raises response accuracy to 65.1% while widening the ref–resp gap to over 30 points — evidence that integration, not retrieval, is the dominant bottleneck under ideal conditions.

Interactivity

On Full-Duplex-Bench, MoshiRAG shows consistently lower takeover rates than vanilla Moshi across pause, backchannel, turn-taking, and interruption tracks, an effect traced to the training distribution (longer knowledge-intensive turns induce conservative turn-taking). It maintains interruption-response latency of 0.18s and achieves a GPT interruption score of 3.75 versus 0.77 for vanilla Moshi, attributed to adversarial v2/v3 training subsets. On Full-Duplex-Bench v1.5, MoshiRAG responds appropriately to genuine interruptions at rates similar to vanilla Moshi while resuming more gracefully under backchannels and background speech. These results indicate the RAG mechanism does not degrade the duplex behaviors that motivate full-duplex modeling.

Out-of-domain generalization

Although trained only on QA-style data, MoshiRAG generalizes to mathematical reasoning benchmarks used by STITCH. With the Gemma back end it reaches 61.7% on AddSub and 33.9% on GSM8K, far above vanilla Moshi (8.3% and 2.1%) and GLM-4-Voice, though below STITCH-S, which was explicitly trained for math. An instructive detail: summarizing the LLM-generated reference before injection improves accuracy, because raw references contain excessive numerical detail and symbolic expressions that hinder integration. With GPT-4.1, GSM8K rises to 43.2%. The authors frame this as early evidence of tool use in full-duplex models, where the LLM back end functions as an external solver.

Limitations and open questions

The paper concedes several limitations. Retrieval triggering depends entirely on learned behavior from synthetic data; trigger rates decline as user speech WER increases, and no adaptive or RL-based triggering mechanism is explored. Accuracy drops sharply when retrieval latency exceeds 1.5 seconds, making back-end speed critical — a dependency on infrastructure the framework cannot itself control. The ~5-point ref–resp gap and the enlarged gap under ground-truth references show knowledge integration remains lossy. Benchmarks evaluate primarily single-turn QA; multi-turn RAG quality is not directly measured. Finally, the choice between insertive injection (better integration) and additive injection (bounded sequence length) is left as an open design question, as is robustness to retrieval errors and failures.

Conclusion

MoshiRAG demonstrates that asynchronous retrieval can be reconciled with full-duplex speech interaction by exploiting the temporal structure of spoken responses. The approach yields large factual gains over the base model, competitive performance against much larger non-duplex systems, preserved interactivity, plug-and-play back-end flexibility, and unexpected generalization to mathematical reasoning. Its main open problems — reliable retrieval triggering, tighter knowledge integration, and robustness to pipeline errors — define a concrete agenda for extending tool use in real-time spoken dialogue systems.