Speech-to-Speech RAG

• Speech-to-Speech RAG is an end-to-end framework that combines raw audio processing, shared embedding alignment, and retrieval-based generation to deliver knowledge-grounded spoken responses.
• It employs modality-aligned encoders, contrastive and distillation losses, and ANN search to integrate audio and text in a common embedding space for efficient retrieval.
• The framework shows significant improvements in latency and robustness over traditional ASR cascades, enabling real-time applications such as open-domain QA and dialogue systems.

Speech-to-Speech Retrieval-Augmented Generation (S2S-RAG) is an integrative framework for open-domain spoken question answering, dialogue modeling, and knowledge-intensive speech applications that combines end-to-end speech processing with knowledge retrieval and neural generation. Distinct from traditional ASR → text-RAG → TTS cascades, S2S-RAG frameworks directly map raw speech queries into a shared embedding space for retrieval and use speech-aware generation models to synthesize knowledge-grounded spoken responses, substantially reducing latency and error propagation while enabling richer paralinguistic and multimodal interaction.

1. Architectural Foundations and Pipeline Variants

The canonical S2S-RAG pipeline replaces the sequential ASR→text-retrieval→LLM→TTS architecture with an end-to-end sequence composed of:

• Speech Encoder: Processes raw audio input $x_s$, generating a dense vector $q$ via a stack of Transformer blocks (e.g., Whisper-large-v3, HuBERT-large), temporal convolution, and a modality alignment adapter (Sun et al., 26 Jan 2025, Min et al., 2024).
• Shared Embedding Space: Speech and text representations are aligned into a joint space by a trainable linear “scaling layer” or projection module—typically, $z = W u + b$ where $W$ and $b$ are shared weights across modalities (Sun et al., 26 Jan 2025, Chen et al., 20 Feb 2025).
• Retriever and Index: Document embeddings $k_i$ (from text, audio, or both) are precomputed and indexed (FAISS or similar), enabling fast nearest-neighbor search with cosine similarity in $\mathbb{R}^d$ (Sun et al., 26 Jan 2025, Chen et al., 20 Feb 2025, Rackauckas et al., 22 May 2025).
• Retrieval-Augmented Speech Generation: Retrieved knowledge (textual passages, audio segments, or hybrid content) is fused with the speech query embedding as conditioning input for a neural speech generator (e.g., SLLM, SLM, GLM-4-Voice, TTS/vocoder), which outputs a waveform or mel-spectrogram (Sun et al., 26 Jan 2025, Feng et al., 27 Apr 2025, Min et al., 2024).

Two principal S2S-RAG design patterns emerge:

• Text-Retrieval Grounded (S2T-to-S2S): Speech input is embedded and retrieves text, which is then used by a speech-aware LLM to condition generation (Sun et al., 26 Jan 2025, Feng et al., 27 Apr 2025).
• Audio/Hybrid Retrieval Grounded (S2S-to-S2S): Speech queries directly retrieve audio segments (sometimes with text), supporting full end-to-end speech retrieval and generation, possibly bypassing transcription entirely (Chen et al., 20 Feb 2025, Min et al., 2024, Rackauckas et al., 22 May 2025).

2. Embedding Alignment and Retrieval Mechanisms

Unified cross-modal embeddings are essential for transcription-free S2S-RAG. Architectures implement:

• Modality-Aligned Encoders: Separate encoder networks for speech ($f_s$) and text ($f_t$), both aligned by a shared scaling/projection layer to ensure $q=f_s(x_s)$ and $k=f_t(d)$ live in a common $\mathbb{R}^d$ space (Sun et al., 26 Jan 2025, Chen et al., 20 Feb 2025, Min et al., 2024).
• Contrastive and Distillation Loss Functions: Training objectives include mean squared error (MSE) alignment loss, InfoNCE contrastive loss

$$\mathcal{L}_{\text{retrieval}} = - \log \frac{\exp(\text{sim}(q, k^+)/\tau)}{\sum_{i} \exp(\text{sim}(q, k_i)/\tau)}$$

and cross-modal cosine distillation losses

$$\mathcal{L}_{\text{align}}(e_s, e_t) = 1 - \cos(e_s, e_t)\ .$$

These objectives force speech and text embeddings with similar content to be close under cosine similarity, enabling retrieval without intermediate transcription (Sun et al., 26 Jan 2025, Min et al., 2024, Chen et al., 20 Feb 2025).

• Retrieval over Hybrid Indices: ANN search is conducted over a database of normalized embeddings for passages that may be text, audio, or multimodal (hybrid). Scores are typically:

$$\mathrm{score}(q, k) = \frac{\langle q, k \rangle}{\|q\|\|k\|}$$

with probabilistic softmax for retrieval distributions (Chen et al., 20 Feb 2025).

• Token and Sequence-Level Retrieval: Some variants use token-level embeddings and kNN retrieval (e.g., LA-RAG), aggregating token-level matches to score sequence relevance and guide in-context correction (Li et al., 2024).

3. End-to-End Generation and Fusion Strategies

After retrieval, S2S-RAG systems employ one of several generation mechanisms:

• Speech-Centric LLMs (SLLMs): Condition directly on concatenated speech-query and retrieval embeddings (either as fixed vectors or sequences), allowing direct waveform or spectrogram generation without intermediate text (Sun et al., 26 Jan 2025, Feng et al., 27 Apr 2025, Min et al., 2024).
• Chain-of-Thought and Reasoned Generation: Some systems (e.g., WavRAG) prompt the generator with chain-of-thought (CoT) reasoning, using magic prompts and universal self-consistency, yielding more robust, grounded spoken responses (Chen et al., 20 Feb 2025).
• Streaming and Incremental Generation: Streaming RAG architectures predict tool queries and initiate retrieval while a query is still being spoken, fusing retrieved results with ongoing audio encodings, and generating waveform in streaming mode for responsiveness (Arora et al., 2 Oct 2025).
• Integration of Symbolic and Acoustic Knowledge: Retrievals from KBs containing both audio and text enable multimodal fusion; retrieved audio passages can be transcribed or captioned for LLM-based reasoning, or input directly to an audio-capable generation model (Chen et al., 20 Feb 2025, Min et al., 2024).

4. Systemic Challenges and Performance Benchmarks

A range of empirical and system-level analyses illuminate the strengths and limitations of S2S-RAG:

• Latency and Robustness: End-to-end S2S-RAG pipelines halve or quadruple latency compared to cascaded ASR-based systems (e.g., 0.31s vs. 0.67s; 0.08s vs. 0.4s), and maintain stable performance under acoustic variation or added noise, exhibiting less error propagation due to the absence of ASR (Sun et al., 26 Jan 2025, Min et al., 2024, Rackauckas et al., 22 May 2025, Feng et al., 27 Apr 2025).
• Retrieval and Generation Quality: Although S2S-RAG can achieve retrieval accuracy and answer quality close to transcript-based RAG in clean regimes, a 1–9 percentage point drop is still typical relative to strong text baselines. For instance, end-to-end RAG versus ASR RAG yields F1 0.24 vs. 0.28 and EM 0.43 vs. 0.52 on HotpotQA (Feng et al., 27 Apr 2025).
• Hybrid and Audio-Native Retrieval:
  • VoxRAG and SpeechRAG demonstrate that direct audio-to-audio retrieval is feasible, with Recall@10 for very relevant (VR) segments at 0.34 and somewhat relevant (SR) at 0.60; nDCG for VR is low, highlighting ongoing precision bottlenecks (Rackauckas et al., 22 May 2025).
  • WavRAG reports 8–14x retrieval speedup over ASR-RAG at similar Recall@10, and improved generation (EM) scores with CoT prompting (Chen et al., 20 Feb 2025).
• Interactional Friction: Modular S2S-RAG systems are prone to three systemic frictions—temporal misalignment (multi-second latencies), expressive flattening (loss of prosody and affect), and repair rigidity (inability to interrupt or correct system output mid-turn) (Mairittha et al., 12 Dec 2025).

Example Performance Table

System	Latency (s)	Retrieval F1	EM (HotpotQA)	Recall@10 (VoxRAG)
SEAL S2S-RAG	0.31	–	–	–
ASR+Text RAG	0.67	0.28	0.52	–
E2E S2S-RAG	0.08	0.24	0.43	–
VoxRAG (VR,S2S)	–	–	–	0.34
WavRAG	0.23	–	0.40	0.72*

*On SLUE-SQA-5 (Sun et al., 26 Jan 2025, Chen et al., 20 Feb 2025, Min et al., 2024, Rackauckas et al., 22 May 2025, Feng et al., 27 Apr 2025).

5. Specialized Variants and Extensions

S2S-RAG encompasses a spectrum of research directions:

• Streaming S2S-RAG: Real-time, tool-augmented dialogue systems proactively retrieve and integrate external knowledge during ongoing speech, minimizing user-perceived latency (by ≈20%) while increasing QA accuracy up to 207% over closed-book baselines (Arora et al., 2 Oct 2025).
• Fine-Grained Speech Retrieval and Correction: Frameworks like LA-RAG exploit token-level speech embeddings and datastore kNN retrieval to correct ASR outputs via LLM in-context learning, demonstrating gains on accented and dialectal speech (Li et al., 2024).
• Transcription-Free Pipelines: VoxRAG, SpeechRAG, and WavRAG eliminate intermediate transcripts, instead aligning speech and text in shared embedding spaces and employing audio-native retrievers and SLMs to directly process and generate spoken content (Chen et al., 20 Feb 2025, Min et al., 2024, Rackauckas et al., 22 May 2025).

6. Open Challenges and Future Directions

Key limitations and frontiers for S2S-RAG are under active investigation:

• Semantic and Paralinguistic Grounding: Most current approaches focus on semantic retrieval; modeling expressive speech properties (prosody, emotion) and preserving them across retrieval and generation remains an unresolved issue (Chen et al., 20 Feb 2025, Mairittha et al., 12 Dec 2025).
• Hybrid and Multimodal Retrieval: Extending RAG to retrieve and fuse multimodal evidence (audio, text, video) and advanced reranking of retrieved content is an open direction (Sun et al., 26 Jan 2025, Chen et al., 20 Feb 2025).
• End-to-End Joint Optimization: Joint training of retrieval and generation modules for tighter modality integration, as well as graph-based and self-refinement retrieval strategies, is suggested but not yet realized in most S2S-RAG frameworks (Feng et al., 27 Apr 2025, Min et al., 2024).
• Interactional Design: Addressing friction points requires infrastructure-level advances, including full-duplex, incremental processing, prosody-aware LLMs, adaptive orchestration, and user feedback integration (Mairittha et al., 12 Dec 2025).
• Evaluation: LLM-based metrics dominate current assessment; calibration with human judgments and more discriminative task-specific metrics are vital for progress, particularly on precision and factuality (Rackauckas et al., 22 May 2025).

7. Significance and Impact

S2S-RAG marks a decisive shift in spoken AI by enabling transcription-free, low-latency, and knowledge-augmented speech-to-speech systems. While empirical results demonstrate latency gains, robustness to acoustic variability, and competitive retrieval/generation metrics relative to traditional pipelines, moderate deficits in answer accuracy, fine-grained factuality, and expressive output persist. Progress in embedding alignment, audio-native generation, and interactional fluency will define the next generation of S2S-RAG architectures, with broad applicability across real-time dialogue, conversational QA, and multimodal knowledge agents (Sun et al., 26 Jan 2025, Chen et al., 20 Feb 2025, Min et al., 2024, Feng et al., 27 Apr 2025, Arora et al., 2 Oct 2025, Mairittha et al., 12 Dec 2025, Rackauckas et al., 22 May 2025, Li et al., 2024).