MusT-RAG: Retrieval-Augmented Generation for Music

• MusT-RAG is a retrieval-augmented generation framework specifically designed for music question answering, integrating curated musical knowledge with generative models.
• It employs a dual pipeline for text-only and multi-modal tasks, using BM25 for precise retrieval and iterative re-ranking to optimize evidence alignment.
• The architecture enhances factual accuracy and efficiency for music queries by leveraging domain-specific databases and retrieval-conditioned fine-tuning.

MusT-RAG is a retrieval-augmented generation (RAG) framework designed for music question answering (MQA), both in text-only and multi-modal (text and vision) forms. It combines specialized knowledge retrieval with LLM or large vision-LLM (LVLM) generative reasoning, leveraging music-focused data curation, domain-adapted retrieval strategies, and context-grounded answer generation. The architecture was introduced to address the limited music-specific knowledge inherent to general-purpose foundation models and adapts retrieval-augmented generation with domain-specific innovations to achieve state-of-the-art results on MQA benchmarks (Kwon et al., 31 Jul 2025, Hu et al., 29 May 2025).

1. Definition and Motivation

MusT-RAG extends the RAG paradigm for MQA by explicitly injecting curated musical knowledge into the answer generation pipeline. Standard LLMs and LVLMs possess limited music-domain competence due to the small representation of music-specific data in their training corpora, resulting in suboptimal performance for specialized question answering. By integrating domain-focused retrieval (via MusWikiDB) and fine-tuned or retrieval-conditioned generative models, MusT-RAG enables both factual accuracy and up-to-date reasoning for music-related queries (Kwon et al., 31 Jul 2025).

A related, multi-modal pipeline—also named MusT-RAG and expounded in (Hu et al., 29 May 2025)—extends these principles to vision-language tasks, unifying retrieval, re-ranking, and generation with an "agentic" loop that reduces hallucination and improves evidence alignment in answer generation.

2. System Architecture

The MusT-RAG pipeline consists of a closely-coupled retriever and generator:

• Retriever: Encodes the user query $q$ into a vector representation $E(q)$ and ranks candidate documents $p \in D$ from MusWikiDB using cosine similarity:

$$\textrm{sim}(q, p) = \frac{E(q) \cdot E(p)}{||E(q)|| \, ||E(p)||}$$

The top-$k$ passages $c$ are then selected:

$c = R(q, D)$

• Generator: Receives the original question and retrieved passages and generates the answer token-by-token, grounding each next-token prediction on $[q, c]$:

$$p(x_1 \dots x_n \mid q, c) = \prod_i p_\theta(x_i \mid [q, c; x_{<i}])$$

• Knowledge Base Construction (MusWikiDB): Wikipedia articles are filtered to seven high-level music categories (artists, genres, instruments, history, technology, theory, forms), pruned (sections < 60 tokens removed), chunked into 128-token overlapping passages (10% overlap), and indexed with BM25. Contriever and CLAP dense representations were evaluated, but BM25 yielded best factual/contextual recall per cost (Kwon et al., 31 Jul 2025).

Text-Only vs. Multi-Modal Variants

• Text-Only MusT-RAG: Uses only textual queries and passages; evaluations focus on MQA (artist metadata, historical, theoretical questions).
• Multi-Modal MusT-RAG: Employs image and text retrieval using a dual-encoder CLIP-style retriever (EVA-CLIP). Modality configurations include image-only queries against image+text documents (I ↔ IT), image+question (IQ ↔ IT), and caption variants. The scoring function fuses visual and textual embeddings:

$$S(q, d) = \langle \phi_{\text{vis}}(I_q), \phi_{\text{vis}}(I_d) \rangle + \langle \phi_{\text{txt}}(T_q), \phi_{\text{txt}}(T_d) \rangle$$

where $E(q)$0 and $E(q)$1 are the visual and textual encoders, respectively (Hu et al., 29 May 2025).

3. Retrieval, Re-ranking, and Generation Strategies

Retrieval: MusT-RAG prioritizes fast, high-precision entity retrieval with BM25 on a tightly focused passage set. For multi-modal retrieval, approximate nearest neighbor search (FAISS) is used with K=5.

Re-ranking: To counteract LVLM positional biases (strongest attention to early prompt positions—"Lost in the Middle") and optimize evidence order, three strategies are evaluated:

• Pointwise: Score each doc with a relevance function and sort.
• Pairwise: LVLM chooses the more relevant document between each pair.
• Listwise: LVLM emits a re-ordered list via prompt, simulating listwise loss:

$E(q)$2

Empirically, listwise zero-shot prompting achieves highest Recall@1 (e.g., InfoSeek: 64.4% without rerank vs. 65.9% with listwise) (Hu et al., 29 May 2025).

Generation: The pipeline forms the input as a question followed by the most relevant document: $E(q)$7 Best results are achieved by fusing only the top-1 document. Providing more retrieved evidence ($E(q)$3) degrades answer accuracy, due to LVLMs' strong positional bias and limited context window (Hu et al., 29 May 2025).

4. Training Objectives and Adaptation Modes

MusT-RAG supports two principal model adaptation strategies:

• RAG Inference Only: The base LLM or LVLM remains frozen; retrieved passages are prepended to the prompt at inference. No parameter updates occur.
• RAG-Style Fine-Tuning: The generator is fine-tuned with (q, c, answer) triples, learning to explicitly condition answer generation on both the query and external context. The objective extends standard QA next-token loss,

$E(q)$4

to a retrieval-grounded loss,

$E(q)$5

• Optional Retriever Fine-Tuning: A retrieval loss trains the retriever to distinguish gold passages from negatives:

$E(q)$6

In practice, gold passages from MusWikiDB are always used for positive supervision (Kwon et al., 31 Jul 2025).

5. Evaluation, Metrics, and Performance

MusT-RAG was evaluated on both in-domain (ArtistMus, focusing on artist metadata) and out-of-domain (TrustMus, derived from The Grove Dictionary) MQA benchmarks. Architectures were built on Llama 3.1 8B Instruct with LoRA adapters.

Key Results:

Method	ArtistMus Factual	ArtistMus Context	TrustMus
Zero-shot Llama3	39.0%	–	35.8%
QA Fine-Tuned	40.0%	–	32.0%
RAG Inference	82.0%	88.8%	40.8%
RAG Fine-Tuned	82.4%	92.0%	41.5%
GPT-4o Zero-shot	67.4%	–	53.0%

MusT-RAG inference delivered a 42% absolute factual improvement over closed-book Llama 3.1 on ArtistMus (82.0% vs. 39.0%) and outperformed GPT-4o by 14.6% in factual accuracy. On TrustMus, RAG fine-tuning added a 0.7% absolute gain over RAG inference and an overall 5.0% improvement over zero-shot Llama3 (Kwon et al., 31 Jul 2025). Retrieval over MusWikiDB was ten times faster than querying the full Wikipedia index (≈20 ms vs. ≈200 ms), while raising accuracy by 5.9%.

For the multi-modal pipeline (Hu et al., 29 May 2025), use of unified agentic self-reflection (iterative re-rank + verification) boosted GPT-4 judge accuracy by ~4 points on E-VQA and ~2 points on InfoSeek benchmarks, with up to 5% relative improvement compared to non-agentic pipelines. Evaluation metrics included Recall@K, MRR for retrieval, ROUGE-L, and judge-verified semantic precision.

6. Design Insights and Trade-Offs

Curating a laser-focused, domain-centered retrieval database (MusWikiDB) significantly reduces noise versus general Wikipedia, resulting in more accurate and efficient retrieval. BM25 with 128-token chunking balances factual recall and manageable context length. In multi-modal variants, using a single, top-ranked document for answer generation is superior to multi-evidence fusion due to LVLMs' positional token bias.

Zero-shot listwise LVLM re-ranking outperforms pointwise and pairwise strategies, even rivaling some supervised alternatives. Agentic pipelines that allow the model to reflect and verify answers on retrieved evidence further suppress hallucinations.

A plausible implication is that for highly specialized domains, compact yet comprehensive and efficiently indexed domain-specific KBs, coupled with retrieval-grounded generation (and, where feasible, dynamic agentic feedback), consistently outperform closed-book and general-purpose approaches in both accuracy and computational cost.

7. Significance and Applicability

MusT-RAG demonstrates that RAG frameworks, when methodically configured with domain-specialized retrieval, tailored passage chunking, fine-grained re-ranking, and context-grounded generation, can approach or surpass human-level performance on specialized MQA tasks. The principles are extensible beyond music: similar architecture, pipeline design, and knowledge engineering strategies are directly applicable to other domains demanding high-precision, current, and context-aware knowledge integration (Kwon et al., 31 Jul 2025, Hu et al., 29 May 2025).

The outcome substantiates that a targeted, efficient knowledge base combined with retrieval-conditioned reasoning not only elevates factual performance but also ensures scalable inference, tractable fine-tuning, and robust generalization to both in-domain and out-of-domain questions.