PlanRAG-Audio: Retrieval-Augmented Audio Understanding

• PlanRAG-Audio is a planning-based retrieval framework that treats long recordings as structured multimodal databases instead of brute-force long-context inputs.
• It uses a deterministic planning LLM to decompose queries into modality-specific streams, filtering evidence from speech, speaker, emotion, and sound-event cues.
• The system decouples inference cost from raw audio length, stabilizing performance and enhancing efficiency for extensive audio recordings.

Searching arXiv for the target paper and a few closely related audio-RAG works to ground the article with current citations. PlanRAG-Audio is a planning-based retrieval-augmented generation framework for long-form audio understanding that treats long recordings as a structured retrieval problem rather than a brute-force long-context modeling problem. It is designed for settings in which relevant evidence is distributed over time and across heterogeneous acoustic cues—speech content, speaker identity, emotion, and sound events—and where directly processing entire recordings with large audio LLMs is inefficient or unstable. Instead of asking a model to ingest a full recording, PlanRAG-Audio first plans which modalities and temporal spans are needed for a given query, retrieves only the relevant records from a structured multimodal database, and then performs answer generation over that compact evidence set (Masao et al., 19 May 2026).

1. Problem setting and conceptual reframing

PlanRAG-Audio addresses long-form audio understanding for large audio LLMs under three stated difficulties: extreme sequence length, heterogeneous acoustic cues distributed over time, and the need for joint reasoning across modalities and time (Masao et al., 19 May 2026). The motivating example is that a one-hour lecture is roughly 12k text tokens but 100k+ speech tokens under Gemini-style tokenization, which creates severe memory and compute bottlenecks for direct end-to-end inference (Masao et al., 19 May 2026).

The framework is grounded in the observation that long-form audio understanding is not reducible to transcript extraction. Relevant evidence may lie in speech content, speaker identity, emotional expression, non-speech sound events, and temporal ordering across those signals (Masao et al., 19 May 2026). This places it in contrast both to transcript-first pipelines and to brute-force long-context multimodal inference. The paper explicitly argues that simply feeding a full recording or a full structured dump to a model is inefficient and often ineffective because the model must process large amounts of irrelevant context, performance degrades as duration grows, non-text evidence is drowned out, formatting failures increase, and temporal structure remains implicit rather than externalized (Masao et al., 19 May 2026).

This positioning differentiates PlanRAG-Audio from several adjacent audio-RAG paradigms. “VoxRAG” retrieves semantically relevant spoken segments directly from speech but remains text-mediated at answer generation time, with transcripts passed to GPT-4o rather than a planner operating over structured multimodal evidence (Rackauckas et al., 22 May 2025). “Speech Retrieval-Augmented Generation without Automatic Speech Recognition” aligns speech passages into a frozen text retrieval space and retrieves audio directly from text queries, but its main contribution is ASR-free retrieval rather than explicit retrieval planning over multiple modality streams (Min et al., 2024). “WavRAG” generalizes RAG to a text-audio hybrid knowledge base and emphasizes audio-native retrieval, yet it does not introduce an explicit planner over time-aligned modality streams (Chen et al., 20 Feb 2025). This suggests that PlanRAG-Audio’s distinctive contribution is not merely audio retrieval, but planning-conditioned retrieval over a structured long-audio database.

2. System architecture and retrieval plan

The architecture has four stated stages: audio and speech processing, retrieval planning, structured retrieval, and answer generation (Masao et al., 19 May 2026). In the first stage, raw audio is transformed into a structured database $D(a)$ for recording $a$ using four preprocessing modules: speaker diarization and segmentation, speech recognition, emotion recognition, and audio event detection (Masao et al., 19 May 2026). These yield modality-specific streams stored as time-aligned records.

The planning stage is the central mechanism. Given a user question $q$, a planning LLM produces a structured retrieval plan $\mathcal{P}(q)$ via constrained decoding under a fixed schema, which the paper says makes planning “effectively deterministic” and prevents invalid plans (Masao et al., 19 May 2026). The plan specifies: which modality-specific streams are required, what filters apply to each stream, how streams are joined or fused, which fields are returned, and what answer schema the final generation model must follow (Masao et al., 19 May 2026).

The simplified plan contract given in the paper includes fields such as "streams", "filters", "fusion", "output", and "answer_schema" (Masao et al., 19 May 2026). In the worked example, the plan selects transcript and speaker streams, applies a keyword filter "employment" and a speaker filter "SPEAKER_02", anchors fusion on the transcript stream, returns start/end/speaker/text fields, and constrains the final answer to a fixed JSON schema (Masao et al., 19 May 2026). This is a concrete operationalization of query decomposition into retrieval subgoals.

The retrieved result is formalized as

$$R(q, a) = \mathrm{Exec}(Q(\mathcal{P}(q)), D(a)), \tag{1}$$

where $Q(\mathcal{P}(q))$ is the SQL query compiled from the plan and $D(a)$ is the structured database for recording $a$ (Masao et al., 19 May 2026). This formulation makes the evidence bottleneck explicit: the answer model does not operate on raw audio length, but on executed retrieval results.

A plausible implication is that PlanRAG-Audio externalizes part of the reasoning burden into plan generation and deterministic execution. This differs from systems such as “AudioRAG,” which use an agentic Think-Call-Answer loop with an audio tool and web search but do not expose a comparable fixed-schema retrieval contract over a structured long-audio database (Lin et al., 11 Feb 2026), and from “LongAudio-RAG,” which also performs time-aware structured retrieval over event logs but is specialized to timestamped acoustic event detections rather than a four-stream transcript/speaker/emotion/sound-event database (Vakada et al., 16 Feb 2026).

3. Structured multimodal database and temporal fusion

The database representation is one of the framework’s defining technical features. The paper describes time-aligned records for at least four streams: transcript, speaker, emotion, and sound_event (Masao et al., 19 May 2026). Transcript, speaker, and emotion share boundaries defined by speaker diarization, while sound events are generated independently with a sliding-window tagging approach because non-speech events do not naturally align to dialogue turns or speaker boundaries (Masao et al., 19 May 2026).

This representation supports direct association of what was said, by whom, with what emotional label, while also allowing separately aligned sound-event evidence to be fused later (Masao et al., 19 May 2026). Emotion and sound-event labels are stored as label-score pairs in JSONB fields (Masao et al., 19 May 2026). The framework therefore treats long-form audio as a multimodal relational store rather than a monolithic sequence.

Temporal fusion is implemented by timestamp-based joining. In Appendix F, for each base segment $b$ and target segment $t$, the midpoint distance is computed from segment midpoints, and among overlapping candidates within tolerance $a$0, the target whose midpoint is closest to the base segment midpoint is selected (Masao et al., 19 May 2026). The default tolerance is

$a$1

This is a simple nearest-neighbor temporal fusion strategy rather than a learned temporal reasoning module (Masao et al., 19 May 2026).

The use of explicit timestamps and symbolic joins is a central reason the system handles long recordings more robustly. Temporal overlap drives fusion, event ordering is computed from onset times, speaker counting is done over windows, and summarization is performed for specific intervals (Masao et al., 19 May 2026). In this sense, temporal structure is not inferred implicitly inside the LLM; it is externalized into retrieved evidence.

This design places PlanRAG-Audio in closer conceptual proximity to “LongAudio-RAG,” which also converts long audio into timestamped event records and then answers queries from retrieved structured evidence (Vakada et al., 16 Feb 2026). However, LongAudio-RAG grounds retrieval in event detections with columns such as event name, start time, end time, confidence, and loudness, whereas PlanRAG-Audio supports transcript, speaker, emotion, and sound-event streams and uses planner-generated SQL to join them (Vakada et al., 16 Feb 2026, Masao et al., 19 May 2026). By contrast, “VoxRAG” indexes diarization- and VAD-derived spoken segments but does not elevate temporal fusion into a planner-executable structured query abstraction (Rackauckas et al., 22 May 2025).

4. Retrieval mechanism, answer generation, and efficiency

The paper states that retrieval in the main experiments uses a simple keyword-based mechanism, and an appendix comparison reports that vector search does not consistently improve results (Masao et al., 19 May 2026). The interpretation offered is that planning contributes more than retriever sophistication in this setting (Masao et al., 19 May 2026). This is an important methodological choice: the system is designed to isolate the effect of planning-conditioned retrieval rather than maximize retrieval performance through a complex ANN or multimodal retriever.

Once retrieved evidence $a$2 is obtained, it is passed to a generation LLM together with the output schema, and the final answer is generated in the requested form, including short answers, JSON lists of diarization segments, event localization JSON, integer counts, ordered event lists, and abstention responses (Masao et al., 19 May 2026). The generation model used in Table 3 is Qwen3-4B-Instruct-2507 (Masao et al., 19 May 2026). The long-context baseline is [Gemini](https://www.emergentmind.com/topics/gemini) 2.5 Flash, and the audio LALM baseline is Voxtral-Mini-3B-2507 (Masao et al., 19 May 2026).

The efficiency claim is one of the paper’s most concrete findings. Because the LLM sees only retrieved evidence rather than the full recording, generation-stage cost depends mostly on retrieved evidence size rather than raw audio duration (Masao et al., 19 May 2026). Table 4 for MCQA with 60-minute recordings reports average input lengths of 115.2k tokens for full-context Gemini, 0.9k for Gemini + PlanRAG-Audio, and 1.2k for Qwen + PlanRAG-Audio (Masao et al., 19 May 2026). The paper explicitly states that PlanRAG-Audio “decouples inference cost from raw audio length” (Masao et al., 19 May 2026).

The tradeoff is preprocessing overhead. Appendix E reports total preprocessing times of 37.25 sec for 10 minutes, 93.80 sec for 30 minutes, 184.18 sec for 60 minutes, 1036.34 sec for 300 minutes, and 1986.00 sec for 540 minutes, with real-time factor around 0.05–0.06 (Masao et al., 19 May 2026). The authors state that this cost is amortized across queries, making the design particularly suitable when many queries are asked over the same long recording (Masao et al., 19 May 2026).

A plausible implication is that PlanRAG-Audio is best matched to archive-style, post-hoc, or repeated-query settings rather than strict real-time interaction. This contrasts with “Stream RAG,” which focuses on pre-utterance tool triggering to reduce perceived latency in speech-in/speech-out dialogue systems (Arora et al., 2 Oct 2025), and with “VoiceAgentRAG,” which uses a dual-agent semantic-cache architecture to prefetch likely future knowledge for low-latency voice interaction (Qiu et al., 2 Mar 2026). Those works address real-time conversational bottlenecks; PlanRAG-Audio instead addresses long-form evidence selection and multimodal reasoning stability.

5. Experimental evaluation and empirical findings

The evaluation spans base tasks and advanced tasks across recordings of 10, 30, 60, 300, and 540 minutes (Masao et al., 19 May 2026). Base tasks include semantic understanding / QA from LibriSpeech with LibriSQA questions, speaker diarization on AMI, summarization on AMI, emotion recognition on MSP-Podcast, and sound event detection on VoxPopuli recordings with inserted AudioSet event clips (Masao et al., 19 May 2026). Advanced tasks include speaker count, event ordering, and speaker-constrained MCQA with abstention (Masao et al., 19 May 2026).

The central empirical finding is that PlanRAG-Audio stabilizes performance as audio duration increases, whereas direct long-context baselines degrade (Masao et al., 19 May 2026). The paper states that without retrieval planning, performance drops as duration increases for Qwen and Gemini, and that degradation is especially strong for audio-only or non-text-centric tasks such as speaker diarization, emotion recognition, and sound event detection (Masao et al., 19 May 2026). Voxtral performs reasonably on text-centric tasks but fails on non-text-based ones (Masao et al., 19 May 2026).

Several quantitative results illustrate the benefits of retrieval planning. In advanced single-modality reasoning, speaker count accuracy is 69.40 for Gemini + PlanRAG-Audio versus 14.20 for Gemini and 9.17 for Voxtral, while event ordering Spearman correlation is 0.68 for Gemini + PlanRAG-Audio versus 0.30 for Gemini and -0.10 for Voxtral (Masao et al., 19 May 2026). For Qwen, speaker count improves from 35.16 to 36.66, and event ordering from 0.11 to 0.34 with PlanRAG-Audio (Masao et al., 19 May 2026).

The speaker-constrained MCQA task is especially revealing because it requires content understanding, speaker selection, temporal alignment, and abstention. Gemini without PlanRAG-Audio achieves unconstrained QA accuracy 58.83, constrained QA accuracy 68.13, and abstention accuracy 0.54; with PlanRAG-Audio these become 65.00, 70.96, and 94.90 (Masao et al., 19 May 2026). Qwen with PlanRAG-Audio reaches unconstrained QA accuracy 65.09, constrained QA accuracy 67.59, and abstention accuracy 82.20 (Masao et al., 19 May 2026). The paper’s interpretation is that evidence filtered to the target speaker before generation greatly improves speaker-conditioned retrieval and abstention (Masao et al., 19 May 2026).

There is also evidence that planning matters more than retriever choice in this setup. Appendix G compares keyword search and vector search and reports that vector search does not consistently improve results (Masao et al., 19 May 2026). This aligns with the framework’s core claim that the main leverage comes from deciding what to retrieve and how to fuse streams, not merely from replacing one nearest-neighbor backend with another.

A plausible implication is that PlanRAG-Audio’s gains arise primarily from structured problem decomposition and evidence bottlenecking rather than raw retriever power. This differs from works like “WavRAG,” where contrastive retriever adaptation is central to obtaining a unified audio-text retrieval space (Chen et al., 20 Feb 2025), or “SpeechRAG,” where direct speech retrieval matching transcript-oracle performance is the primary achievement (Min et al., 2024).

6. Relation to adjacent audio-RAG paradigms and limitations

PlanRAG-Audio sits within a broader landscape of audio retrieval-augmented methods, but it is distinguished by the explicitness of its planning layer. “SonicRAG” demonstrates retrieval-augmented audio composition over a curated sound library using an LLM-generated Mixer Script, yet its orchestration is limited to asset selection and signal-processing control rather than long-form understanding with structured query planning (Guo et al., 6 May 2025). “Audiobox TTA-RAG” and “Feedback-driven Retrieval-augmented Audio Generation with Large Audio LLMs” retrieve audio exemplars to improve text-to-audio generation, but they focus on generation-time conditioning and deficiency repair rather than retrieval planning over time-aligned evidence for QA (Yang et al., 2024, Zhao et al., 2 Nov 2025). “RAG-Boost” and “LA-RAG” integrate retrieval into ASR correction, targeting lexical recovery under noisy or accented speech rather than multimodal reasoning over long recordings (Wang et al., 5 Aug 2025, Li et al., 2024).

The closest conceptual neighbors are systems that restructure audio into retrievable units. “LongAudio-RAG” converts multi-hour audio into SQL-indexed event logs and answers time-aware queries from retrieved event records (Vakada et al., 16 Feb 2026). “AudioRAG” evaluates multi-hop questions that require extracting latent attributes from audio and then retrieving external web knowledge, using an agentic Think-Call-Answer loop (Lin et al., 11 Feb 2026). “OmniRAG-Agent” adds an agentic retrieval loop over image and transcript-based audio banks and jointly optimizes tool use and answer quality with GRPO (Zhu et al., 3 Feb 2026). Relative to these, PlanRAG-Audio’s main technical specificity lies in its planner-generated, schema-constrained retrieval plan over transcript, speaker, emotion, and sound-event streams (Masao et al., 19 May 2026).

The paper is explicit about several limitations. First, the framework depends on pretrained perception modules—ASR, speaker diarization, emotion recognition, and sound event detection—so its upper bound is constrained by upstream model accuracy (Masao et al., 19 May 2026). Second, retrieval in the main experiments is deliberately simple keyword search, which limits recall even if it isolates the planning effect (Masao et al., 19 May 2026). Third, preprocessing overhead is nontrivial and may hinder real-time deployment (Masao et al., 19 May 2026). Fourth, Gemini experiments were affected by API instability and malformed outputs (Masao et al., 19 May 2026). Fifth, the paper does not claim to remove ethical or robustness risks inherited from the component models (Masao et al., 19 May 2026).

Further limitations follow from what is absent. There is no dedicated supervised planner training loss, no learned temporal reasoning model, no explicit planner/executor separation, and no end-to-end optimization comparable to GRPO-style agent training in “OmniRAG-Agent” (Zhu et al., 3 Feb 2026). The retrieval mechanism is not a strong audio-native retriever of the kind explored in “WavRAG” or “SpeechRAG” (Chen et al., 20 Feb 2025, Min et al., 2024). This suggests that PlanRAG-Audio should be understood less as a universal audio-RAG solution than as a concrete argument for a particular abstraction: long-form audio should be queried through planned, structured retrieval over time-aligned multimodal records rather than by asking a single model to absorb the entire recording.

In that sense, PlanRAG-Audio’s main contribution is conceptual and architectural. It reframes long-form audio understanding as planning-conditioned multimodal retrieval, demonstrates that this stabilizes performance as duration increases, and shows that explicit stream selection, temporal filtering, and answer-schema control can reduce input size by roughly two orders of magnitude while improving reasoning on long recordings (Masao et al., 19 May 2026). For long-audio QA, speaker-aware reasoning, event-aware reasoning, and affect-aware reasoning over hours of content, that reframing is the central significance of the framework.