---
title: 'PlanRAG-Audio: Retrieval-Augmented Audio Understanding'
url: https://www.emergentmind.com/topics/planrag-audio
type: topic
---

# PlanRAG-Audio: Retrieval-Augmented Audio Understanding

Searching arXiv for the target paper and a few closely related audio-RAG works to ground the article with current citations.
PlanRAG-Audio is a planning-based retrieval-augmented generation framework for long-form audio understanding that treats long recordings as a structured retrieval problem rather than a brute-force long-context modeling problem. It is designed for settings in which relevant evidence is distributed over time and across heterogeneous acoustic cues—speech content, speaker identity, emotion, and sound events—and where directly processing entire recordings with large audio language models is inefficient or unstable. Instead of asking a model to ingest a full recording, PlanRAG-Audio first plans which modalities and temporal spans are needed for a given query, retrieves only the relevant records from a structured multimodal database, and then performs answer generation over that compact evidence set [2605.20414].

## 1. Problem setting and conceptual reframing

PlanRAG-Audio addresses long-form audio understanding for large audio language models under three stated difficulties: extreme sequence length, heterogeneous acoustic cues distributed over time, and the need for joint reasoning across modalities and time [2605.20414]. The motivating example is that a one-hour lecture is roughly **12k text tokens** but **100k+ speech tokens** under Gemini-style tokenization, which creates severe memory and compute bottlenecks for direct end-to-end inference [2605.20414].

The framework is grounded in the observation that long-form audio understanding is not reducible to transcript extraction. Relevant evidence may lie in speech content, speaker identity, emotional expression, non-speech sound events, and temporal ordering across those signals [2605.20414]. This places it in contrast both to transcript-first pipelines and to brute-force long-context multimodal inference. The paper explicitly argues that simply feeding a full recording or a full structured dump to a model is inefficient and often ineffective because the model must process large amounts of irrelevant context, performance degrades as duration grows, non-text evidence is drowned out, formatting failures increase, and temporal structure remains implicit rather than externalized [2605.20414].

This positioning differentiates PlanRAG-Audio from several adjacent audio-RAG paradigms. “VoxRAG” retrieves semantically relevant spoken segments directly from speech but remains text-mediated at answer generation time, with transcripts passed to GPT-4o rather than a planner operating over structured multimodal evidence [2505.17326]. “Speech Retrieval-Augmented Generation without Automatic Speech Recognition” aligns speech passages into a frozen text retrieval space and retrieves audio directly from text queries, but its main contribution is ASR-free retrieval rather than explicit retrieval planning over multiple modality streams [2412.16500]. “WavRAG” generalizes RAG to a text-audio hybrid knowledge base and emphasizes audio-native retrieval, yet it does not introduce an explicit planner over time-aligned modality streams [2502.14727]. This suggests that PlanRAG-Audio’s distinctive contribution is not merely audio retrieval, but planning-conditioned retrieval over a structured long-audio database.

## 2. System architecture and retrieval plan

The architecture has four stated stages: audio and speech processing, retrieval planning, structured retrieval, and answer generation [2605.20414]. In the first stage, raw audio is transformed into a structured database \(D(a)\) for recording \(a\) using four preprocessing modules: speaker diarization and segmentation, speech recognition, emotion recognition, and audio event detection [2605.20414]. These yield modality-specific streams stored as time-aligned records.

The planning stage is the central mechanism. Given a user question \(q\), a planning LLM produces a structured retrieval plan \(\mathcal{P}(q)\) via constrained decoding under a fixed schema, which the paper says makes planning “effectively deterministic” and prevents invalid plans [2605.20414]. The plan specifies: which modality-specific streams are required, what filters apply to each stream, how streams are joined or fused, which fields are returned, and what answer schema the final generation model must follow [2605.20414].

The simplified plan contract given in the paper includes fields such as `"streams"`, `"filters"`, `"fusion"`, `"output"`, and `"answer_schema"` [2605.20414]. In the worked example, the plan selects transcript and speaker streams, applies a keyword filter `"employment"` and a speaker filter `"SPEAKER_02"`, anchors fusion on the transcript stream, returns start/end/speaker/text fields, and constrains the final answer to a fixed JSON schema [2605.20414]. This is a concrete operationalization of query decomposition into retrieval subgoals.

The retrieved result is formalized as
\[
R(q, a) = \mathrm{Exec}(Q(\mathcal{P}(q)), D(a)), \tag{1}
\]
where \(Q(\mathcal{P}(q))\) is the SQL query compiled from the plan and \(D(a)\) is the structured database for recording \(a\) [2605.20414]. This formulation makes the evidence bottleneck explicit: the answer model does not operate on raw audio length, but on executed retrieval results.

A plausible implication is that PlanRAG-Audio externalizes part of the reasoning burden into plan generation and deterministic execution. This differs from systems such as “AudioRAG,” which use an agentic Think-Call-Answer loop with an audio tool and web search but do not expose a comparable fixed-schema retrieval contract over a structured long-audio database [2602.10656], and from “LongAudio-RAG,” which also performs time-aware structured retrieval over event logs but is specialized to timestamped acoustic event detections rather than a four-stream transcript/speaker/emotion/sound-event database [2602.14612].

## 3. Structured multimodal database and temporal fusion

The database representation is one of the framework’s defining technical features. The paper describes time-aligned records for at least four streams: `transcript`, `speaker`, `emotion`, and `sound_event` [2605.20414]. Transcript, speaker, and emotion share boundaries defined by speaker diarization, while sound events are generated independently with a sliding-window tagging approach because non-speech events do not naturally align to dialogue turns or speaker boundaries [2605.20414].

This representation supports direct association of what was said, by whom, with what emotional label, while also allowing separately aligned sound-event evidence to be fused later [2605.20414]. Emotion and sound-event labels are stored as label-score pairs in JSONB fields [2605.20414]. The framework therefore treats long-form audio as a multimodal relational store rather than a monolithic sequence.

Temporal fusion is implemented by timestamp-based joining. In Appendix F, for each base segment \(b\) and target segment \(t\), the midpoint distance is computed from segment midpoints, and among overlapping candidates within tolerance \(T\), the target whose midpoint is closest to the base segment midpoint is selected [2605.20414]. The default tolerance is
\[
T = 2.5 \text{ seconds}.
\]
This is a simple nearest-neighbor temporal fusion strategy rather than a learned temporal reasoning module [2605.20414].

The use of explicit timestamps and symbolic joins is a central reason the system handles long recordings more robustly. Temporal overlap drives fusion, event ordering is computed from onset times, speaker counting is done over windows, and summarization is performed for specific intervals [2605.20414]. In this sense, temporal structure is not inferred implicitly inside the LLM; it is externalized into retrieved evidence.

This design places PlanRAG-Audio in closer conceptual proximity to “LongAudio-RAG,” which also converts long audio into timestamped event records and then answers queries from retrieved structured evidence [2602.14612]. However, LongAudio-RAG grounds retrieval in event detections with columns such as event name, start time, end time, confidence, and loudness, whereas PlanRAG-Audio supports transcript, speaker, emotion, and sound-event streams and uses planner-generated SQL to join them [2602.14612][2605.20414]. By contrast, “VoxRAG” indexes diarization- and VAD-derived spoken segments but does not elevate temporal fusion into a planner-executable structured query abstraction [2505.17326].

## 4. Retrieval mechanism, answer generation, and efficiency

The paper states that retrieval in the main experiments uses a simple keyword-based mechanism, and an appendix comparison reports that vector search does not consistently improve results [2605.20414]. The interpretation offered is that planning contributes more than retriever sophistication in this setting [2605.20414]. This is an important methodological choice: the system is designed to isolate the effect of planning-conditioned retrieval rather than maximize retrieval performance through a complex ANN or multimodal retriever.

Once retrieved evidence \(R(q,a)\) is obtained, it is passed to a generation LLM together with the output schema, and the final answer is generated in the requested form, including short answers, JSON lists of diarization segments, event localization JSON, integer counts, ordered event lists, and abstention responses [2605.20414]. The generation model used in Table 3 is `Qwen3-4B-Instruct-2507` [2605.20414]. The long-context baseline is `Gemini 2.5 Flash`, and the audio LALM baseline is `Voxtral-Mini-3B-2507` [2605.20414].

The efficiency claim is one of the paper’s most concrete findings. Because the LLM sees only retrieved evidence rather than the full recording, generation-stage cost depends mostly on retrieved evidence size rather than raw audio duration [2605.20414]. Table 4 for MCQA with 60-minute recordings reports average input lengths of **115.2k** tokens for full-context Gemini, **0.9k** for Gemini + PlanRAG-Audio, and **1.2k** for Qwen + PlanRAG-Audio [2605.20414]. The paper explicitly states that PlanRAG-Audio **“decouples inference cost from raw audio length”** [2605.20414].

The tradeoff is preprocessing overhead. Appendix E reports total preprocessing times of **37.25 sec** for 10 minutes, **93.80 sec** for 30 minutes, **184.18 sec** for 60 minutes, **1036.34 sec** for 300 minutes, and **1986.00 sec** for 540 minutes, with real-time factor around **0.05–0.06** [2605.20414]. The authors state that this cost is amortized across queries, making the design particularly suitable when many queries are asked over the same long recording [2605.20414].

A plausible implication is that PlanRAG-Audio is best matched to archive-style, post-hoc, or repeated-query settings rather than strict real-time interaction. This contrasts with “Stream RAG,” which focuses on pre-utterance tool triggering to reduce perceived latency in speech-in/speech-out dialogue systems [2510.02044], and with “VoiceAgentRAG,” which uses a dual-agent semantic-cache architecture to prefetch likely future knowledge for low-latency voice interaction [2603.02206]. Those works address real-time conversational bottlenecks; PlanRAG-Audio instead addresses long-form evidence selection and multimodal reasoning stability.

## 5. Experimental evaluation and empirical findings

The evaluation spans base tasks and advanced tasks across recordings of **10, 30, 60, 300, and 540 minutes** [2605.20414]. Base tasks include semantic understanding / QA from LibriSpeech with LibriSQA questions, speaker diarization on AMI, summarization on AMI, emotion recognition on MSP-Podcast, and sound event detection on VoxPopuli recordings with inserted AudioSet event clips [2605.20414]. Advanced tasks include speaker count, event ordering, and speaker-constrained MCQA with abstention [2605.20414].

The central empirical finding is that PlanRAG-Audio stabilizes performance as audio duration increases, whereas direct long-context baselines degrade [2605.20414]. The paper states that without retrieval planning, performance drops as duration increases for Qwen and Gemini, and that degradation is especially strong for audio-only or non-text-centric tasks such as speaker diarization, emotion recognition, and sound event detection [2605.20414]. Voxtral performs reasonably on text-centric tasks but fails on non-text-based ones [2605.20414].

Several quantitative results illustrate the benefits of retrieval planning. In advanced single-modality reasoning, speaker count accuracy is **69.40** for Gemini + PlanRAG-Audio versus **14.20** for Gemini and **9.17** for Voxtral, while event ordering Spearman correlation is **0.68** for Gemini + PlanRAG-Audio versus **0.30** for Gemini and **-0.10** for Voxtral [2605.20414]. For Qwen, speaker count improves from **35.16** to **36.66**, and event ordering from **0.11** to **0.34** with PlanRAG-Audio [2605.20414].

The speaker-constrained MCQA task is especially revealing because it requires content understanding, speaker selection, temporal alignment, and abstention. Gemini without PlanRAG-Audio achieves unconstrained QA accuracy **58.83**, constrained QA accuracy **68.13**, and abstention accuracy **0.54**; with PlanRAG-Audio these become **65.00**, **70.96**, and **94.90** [2605.20414]. Qwen with PlanRAG-Audio reaches unconstrained QA accuracy **65.09**, constrained QA accuracy **67.59**, and abstention accuracy **82.20** [2605.20414]. The paper’s interpretation is that evidence filtered to the target speaker before generation greatly improves speaker-conditioned retrieval and abstention [2605.20414].

There is also evidence that planning matters more than retriever choice in this setup. Appendix G compares keyword search and vector search and reports that vector search does not consistently improve results [2605.20414]. This aligns with the framework’s core claim that the main leverage comes from deciding what to retrieve and how to fuse streams, not merely from replacing one nearest-neighbor backend with another.

A plausible implication is that PlanRAG-Audio’s gains arise primarily from structured problem decomposition and evidence bottlenecking rather than raw retriever power. This differs from works like “WavRAG,” where contrastive retriever adaptation is central to obtaining a unified audio-text retrieval space [2502.14727], or “SpeechRAG,” where direct speech retrieval matching transcript-oracle performance is the primary achievement [2412.16500].

## 6. Relation to adjacent audio-RAG paradigms and limitations

PlanRAG-Audio sits within a broader landscape of audio retrieval-augmented methods, but it is distinguished by the explicitness of its planning layer. “SonicRAG” demonstrates retrieval-augmented audio composition over a curated sound library using an LLM-generated Mixer Script, yet its orchestration is limited to asset selection and signal-processing control rather than long-form understanding with structured query planning [2505.03244]. “Audiobox TTA-RAG” and “Feedback-driven Retrieval-augmented Audio Generation with Large Audio Language Models” retrieve audio exemplars to improve text-to-audio generation, but they focus on generation-time conditioning and deficiency repair rather than retrieval planning over time-aligned evidence for QA [2411.05141][2511.01091]. “RAG-Boost” and “LA-RAG” integrate retrieval into ASR correction, targeting lexical recovery under noisy or accented speech rather than multimodal reasoning over long recordings [2508.14048][2409.08597].

The closest conceptual neighbors are systems that restructure audio into retrievable units. “LongAudio-RAG” converts multi-hour audio into SQL-indexed event logs and answers time-aware queries from retrieved event records [2602.14612]. “AudioRAG” evaluates multi-hop questions that require extracting latent attributes from audio and then retrieving external web knowledge, using an agentic Think-Call-Answer loop [2602.10656]. “OmniRAG-Agent” adds an agentic retrieval loop over image and transcript-based audio banks and jointly optimizes tool use and answer quality with GRPO [2602.03707]. Relative to these, PlanRAG-Audio’s main technical specificity lies in its planner-generated, schema-constrained retrieval plan over transcript, speaker, emotion, and sound-event streams [2605.20414].

The paper is explicit about several limitations. First, the framework depends on pretrained perception modules—ASR, speaker diarization, emotion recognition, and sound event detection—so its upper bound is constrained by upstream model accuracy [2605.20414]. Second, retrieval in the main experiments is deliberately simple keyword search, which limits recall even if it isolates the planning effect [2605.20414]. Third, preprocessing overhead is nontrivial and may hinder real-time deployment [2605.20414]. Fourth, Gemini experiments were affected by API instability and malformed outputs [2605.20414]. Fifth, the paper does not claim to remove ethical or robustness risks inherited from the component models [2605.20414].

Further limitations follow from what is absent. There is no dedicated supervised planner training loss, no learned temporal reasoning model, no explicit planner/executor separation, and no end-to-end optimization comparable to GRPO-style agent training in “OmniRAG-Agent” [2602.03707]. The retrieval mechanism is not a strong audio-native retriever of the kind explored in “WavRAG” or “SpeechRAG” [2502.14727][2412.16500]. This suggests that PlanRAG-Audio should be understood less as a universal audio-RAG solution than as a concrete argument for a particular abstraction: long-form audio should be queried through planned, structured retrieval over time-aligned multimodal records rather than by asking a single model to absorb the entire recording.

In that sense, PlanRAG-Audio’s main contribution is conceptual and architectural. It reframes long-form audio understanding as planning-conditioned multimodal retrieval, demonstrates that this stabilizes performance as duration increases, and shows that explicit stream selection, temporal filtering, and answer-schema control can reduce input size by roughly two orders of magnitude while improving reasoning on long recordings [2605.20414]. For long-audio QA, speaker-aware reasoning, event-aware reasoning, and affect-aware reasoning over hours of content, that reframing is the central significance of the framework.

Source: https://www.emergentmind.com/topics/planrag-audio