MusiCRS: Audio-Driven Conversational Music Benchmark

• MusiCRS is an audio-centric conversational music recommendation benchmark designed to evaluate models that integrate dialogue, audio grounding, and multi-stakeholder fairness.
• It constructs a corpus from 477 high-quality multi-turn conversations across seven major music subreddits, linking authentic Reddit dialogues with corresponding audio tracks.
• The benchmark reveals current systems’ reliance on textual cues, highlighting their struggle with nuanced audio reasoning and the need for improved cross-modal integration.

Searching arXiv for MusiCRS and closely related work. MusiCRS designates an audio-centric conversational music recommendation benchmark and, in a broader systems sense, a music recommender setting in which recommendation quality depends on jointly modeling dialogue, audio grounding, and stakeholder-sensitive evaluation. In its most specific usage, MusiCRS is introduced as “the first benchmark for audio-centric conversational recommendation that links authentic user conversations from Reddit with corresponding audio tracks,” enabling evaluation under audio-only, query-only, and audio+query conditions; in the surrounding music recommender literature, a system of this kind also sits within a multi-stakeholder ecosystem involving listeners, artists, and platforms, with fairness implications for recommendation quality, exposure, and governance (Surana et al., 23 Sep 2025, Dinnissen et al., 2022).

1. Problem setting and domain specificity

MusiCRS is motivated by a domain-specific difficulty: conversational recommendation has advanced rapidly with LLMs, but music remains “a uniquely challenging domain where effective recommendations require reasoning over audio content beyond what text or metadata can capture” (Surana et al., 23 Sep 2025). In music, similarity judgments frequently depend on rhythm, timbre, instrumentation and arrangement, production style, and vocal style. The underlying claim is not merely that music metadata is incomplete, but that many salient properties are “not reliably described, or even describable, in text metadata or tags.”

This problem setting distinguishes music from domains in which titles, plot summaries, tags, or product descriptions are often adequate surrogates for user intent. A music recommender can fail even when genre labels match, because tracks with the same label may differ substantially in sound. The benchmark therefore targets cases in which a request such as “the glitchy hi-hats” or “the droning synth under the vocals” requires grounding abstract descriptors in actual recordings rather than in genre or popularity heuristics.

MusiCRS is also positioned against the limitations of existing conversational recommenders for music. The dominant pattern in prior systems is text-centric processing based on titles, artist names, genres, tags, and descriptions. This creates recurring failure modes: recommending tracks with similar genre labels but very different sound, ignoring constraints such as “similar vocal timbre but more minimal arrangement,” and struggling when users reference audio-only phenomena. The benchmark is designed precisely to test whether a model can understand dialogue, ground that understanding in actual audio content, and rank candidate tracks accordingly.

2. Corpus construction and dataset composition

MusiCRS is constructed from authentic Reddit conversations drawn from seven major music subreddits: r/classicalmusic, r/hiphopheads, r/electronicmusic, r/metal, r/popheads, r/indieheads, and r/jazz. The collection pipeline begins with 2.7M Reddit submissions and 28.5M comments, followed by a multi-stage filtering process: 46,218 threads are retained after YouTube link filtering; 10,167 multi-turn discussions remain after conversation-quality filtering; and 1,623 conversations are manually validated by four graduate students before the final corpus of 477 high-quality music-focused conversational threads is selected (Surana et al., 23 Sep 2025).

The data model couples conversational text, extracted musical entities, and audio grounding. Qwen2.5-7B is used for music entity extraction, identifying songs, artists, and albums from post titles, post bodies, and top-level comments. Gemma-3-12b-it generates concise, context-preserving queries summarizing user intent. Each conversation is then associated with a candidate pool containing, on average, 100 candidates: the top 10 ground truth entities ranked by comment upvotes and approximately 90 additional entities randomly sampled from the same subreddit and then shuffled. Ground truth entities and linked YouTube URLs are validated and processed into audio clips, with a 300-second “audio budget” for each query across up to 10 clips.

Statistic	Value
Conversations	477
Unique musical entities	3,589
Average ground truth per conversation	8
Candidates per conversation	100

The genre distribution is given via subreddit provenance: Classical: 25.8%, Hip-hop: 21.0%, Indie: 12.8%, Electronic: 11.7%, Metal: 10.3%, Pop: 9.4%, and Jazz: 9.0%. Conversations are retained only when they contain 2–30 musical entities, and comments are limited to max 5 entities to reduce verbosity and topic drift. This yields a corpus that, in the paper’s characterization, covers both mainstream and niche content, diverse conversational styles and knowledge levels, and a broad spectrum of musical aesthetics.

3. Task formulation, modalities, and evaluation protocol

The MusiCRS task is ranking-based. Given a conversation and, depending on the configuration, a reference audio signal or a text query or both, the system ranks a candidate set so that ground-truth recommendations appear near the top (Surana et al., 23 Sep 2025). The paper’s conceptual formalization treats a conversation as

$C = \{u_t\}_{t=1}^T$

where $u_t$ denotes the utterance at turn $t$, and a distilled query as

$q = g(C).$

Each candidate item $i \in \mathcal{I}$ may have an audio representation $x_i$ and metadata $m_i$, and a recommender learns a ranking function

$f(I; \theta) : \mathcal{I} \rightarrow \mathbb{R}.$

Three input-modality configurations are defined. In audio-only, the input is reference audio without text; this tests pure audio understanding and similarity. In query-only, the input is the LLM-generated text query without audio; this tests text-based conversational recommendation. In audio+query (multimodal), both are provided; this tests multimodal fusion and whether a model can combine what is said with what the music sounds like.

Evaluation uses Recall@k, nDCG@k, and Mean Reciprocal Rank (MRR), aggregated both overall and per subreddit or genre. The standard ranking definitions are employed: $$\text{Recall@}k = \frac{\left|\{ i \in G_C : i \text{ appears in top }k\}\right|}{|G_C|}$$ for a ground-truth set $G_C$,

$u_t$0

and

$u_t$1

where $u_t$2 is the rank of the first relevant item for conversation $u_t$3. The benchmark is designed so that different conversations and entities can be used across splits to avoid leakage and to mimic realistic recommendation settings.

4. Baselines, architectures, and empirical findings

MusiCRS benchmarks three broad classes of systems: generative audio-LLMs and multimodal LLMs, retrieval-based models, and traditional baselines (Surana et al., 23 Sep 2025). The generative set includes Qwen2-Audio-7B, Qwen2.5-Omni-7B, SALMONN-7B / SALMONN-13B, FUTGA, Audio Flamingo 3, and Phi-4-Multimodal. Retrieval baselines include CLAP and CoLLAP, which operate in joint text-audio embedding spaces and rank candidates with cosine similarity,

$u_t$4

Traditional baselines consist of subreddit-specific popularity ranking and neighborhood-based collaborative recommendation.

The reported results reveal a central asymmetry between textual and audio evidence. From Table 1, CLAP (query-only) achieves 22.71 / 15.90 on Recall@20 / nDCG@20, which is the best overall performance. Qwen2.5-Omni (audio+query) reaches 21.93 / 16.21, the best result among generative models. At the same time, only 3 of 9 models achieve their best performance in the multimodal (audio+query) setting. Traditional baselines perform substantially lower: Popularity: 16.51 / 11.09 overall and Neighborhood: 14.72 / 9.30.

The benchmark’s main empirical conclusion is explicit: current systems “rely heavily on textual signals and struggle with nuanced audio reasoning,” exposing “fundamental limitations in cross-modal knowledge integration where models excel at dialogue semantics but cannot effectively ground abstract musical concepts in actual audio content” (Surana et al., 23 Sep 2025). This is reinforced by the modality breakdown for CLAP, where audio-only yields 21.15 / 14.90 and query-only yields 22.71 / 15.90, and by the observation that multimodal fusion often fails to produce consistent gains over the best single modality.

The qualitative case studies sharpen this diagnosis. In a prompt asking for “songs that evoke overwhelming sorrow and emptiness, similar to Mozart’s Lacrimosa or the opening of Elfen Lied,” the query-only top-5 includes Wake Up, Numb, My Immortal, One More Light, and Fix You; the audio-only top-5 includes Ave Maria, Clair de Lune, Moonlight Sonata, Canon in D, and Adagio for Strings; and the combined top-5 includes Lamentations of Jeremiah, Daphnis et Chloé, Elfen Lied, Requiem, and Miserere. The paper’s interpretation is that models often default to genre-level heuristics such as “sad classical piece” and still struggle with finer distinctions in vocal timbre, harmonic language, production aesthetics, and dynamic trajectory.

5. Multi-stakeholder fairness and governance context

Within the broader literature on music recommender systems, MusiCRS belongs to a domain in which fairness cannot be reduced to a single user-centric notion of relevance (Dinnissen et al., 2022). A stakeholder-centered view distinguishes three principal groups: users / listeners (consumers/customers), item providers / artists (producers/suppliers), and the platform (music streaming service). The fairness criteria attached to these groups differ. For users, fairness means similar recommendation quality and meaningful personalization across groups defined by attributes such as gender, age, country, mainstream versus beyond-mainstream taste, or personality traits. For artists and other providers, fairness concerns exposure, discovery, and opportunity across gender, country, genre, popularity level, and related attributes. For platforms, fairness is operationalized through compliance with laws, alignment with organizational mission, and responsiveness to regulators and advocacy groups.

The review emphasizes that music recommender fairness research is still relatively sparse and largely diagnostic. The main issues identified include popularity bias, genre / taste / country bias, gender bias, and exposure and feedback loops. In the summary provided, collaborative filtering approaches disproportionately favor popular items in a long-tail economy; users with beyond-mainstream tastes, users from countries whose musical preferences diverge from the global mainstream, minority gender groups, niche genres, and smaller markets can all be disadvantaged. The review also reports that women and multi-gender formations are under-represented in listening logs and recommendations, with organic streaming: ~21.75% of tracks from women or multi-gender formations and programmed/recommended streaming: ~23.55%.

For a system like MusiCRS, the data block operationalizes group disparity in a metric $u_t$5 as

$u_t$6

and ranking exposure for an item $u_t$7 as

$u_t$8

Group-level provider fairness then depends on how exposure is distributed across artist groups relative to group size or target representation. The open questions identified in the review are directly relevant to MusiCRS: limited public data for fairness research in music, and a field in which “the large majority of works analyze the current situation of MRS fairness, whereas only few works propose approaches to improve it.” A plausible implication is that conversational, audio-grounded recommendation should be evaluated not only for relevance and cross-modal grounding, but also for disparities in recommendation quality and exposure across stakeholder groups.

6. Adjacent resources, symbolic extensions, and research directions

Two adjacent resources described in the source material are especially relevant to MusiCRS research, though they are not part of the MusiCRS benchmark proper. JamendoMaxCaps provides 362,238 audio tracks (after cleaning) from the Jamendo public API, all instrumental, all freely licensed for research, and all equipped with captions generated by Qwen2-Audio together with imputed metadata for genre, instrumentation, variable tags, and speed (Roy et al., 11 Feb 2025). Its retrieval-and-imputation pipeline combines MERT-v1-330M audio features with Flan-T5-small metadata embeddings in a joint space,

$u_t$9

and retrieves the top-10 most similar songs for metadata completion. This suggests a concrete route for MusiCRS-style systems to address cold start, catalog enrichment, text-to-music retrieval, and explanation generation, particularly when rich natural-language descriptions of audio content are needed.

GraphMuse provides a different but complementary direction: a graph-processing framework and library for symbolic music, built around heterogeneous score graphs

$t$0

with note, beat, and measure nodes; temporal relations such as onset, during, follow, and silence; and musically informed neighbor sampling for GNN training (Karystinaios et al., 2024). Its hierarchical models—NoteGNN, BeatGNN, MeasureGNN, MetricalGNN, and HybridGNN—improve performance on pitch spelling and cadence detection, and the accompanying discussion explicitly connects these abstractions to MusiCRS-style systems that need “rich symbolic representations of pieces, sections, or motifs and must scale to large catalogs.” This suggests a multimodal extension in which audio-grounded conversational recommendation is augmented with symbolic structure, harmonic context, cadence patterns, or measure-level similarity.

The future directions identified for MusiCRS itself remain centered on multimodal grounding and benchmark growth. The paper proposes richer annotations, including emotional labels, musical structure labels, and instrument tags tied to segments; more genres and subcultures; longer-term preference traces by linking multiple conversations from the same user; and improved cross-modal training objectives that align conversational intent with audio-based recommendation outcomes (Surana et al., 23 Sep 2025). From the fairness review, the complementary agenda includes improved data sharing and benchmarks, synthetic data and simulation, privacy-preserving methods, and a shift from diagnosis to intervention (Dinnissen et al., 2022). Taken together, these directions frame MusiCRS as both a benchmark for present-day audio-centric conversational recommendation and a focal point for broader research on multimodal grounding, fairness-sensitive optimization, and structure-aware music understanding.