MusicSimpleQA Benchmark

• MusicSimpleQA is a domain-specific benchmark designed to measure LLMs' factual knowledge of music facts including artists, albums, and awards.
• It employs an automated agreement scoring methodology using DeepSeek-v3 to ensure rapid, reproducible, and precise evaluations with minimized ambiguity.
• The benchmark demonstrates that domain-specific pretraining significantly improves factual accuracy over general models, addressing scalability and precision challenges.

MusicSimpleQA is a domain-specific benchmark for evaluating factual music knowledge in LLMs via a short-form, single-answer question–answering (QA) suite with fully automated agreement scoring. Developed as part of the MuCPT project, MusicSimpleQA addresses limitations of general-purpose LLMs in the music-entertainment domain by providing a reproducible, efficient, and precise tool for benchmarking models’ knowledge of musical facts, including details about artists, albums, awards, and release dates (Tian et al., 18 Nov 2025).

1. Design Motivation and Evaluation Objectives

MusicSimpleQA was conceived to address a specialized factuality gap: while LLMs like GPT-4 demonstrate linguistic fluency, they lack up-to-date and deep factual knowledge specific to music domain entities and relationships. Existing manual QA processes are labor-intensive and scale poorly, motivating a design that enables reproducible, efficient, and fully automated evaluation. The benchmark is tailored to mirror real-world deployment requirements such as music recommendation systems, chatbots, and music encyclopedia applications, with an explicit focus on precise, verifiable, single-fact knowledge.

2. Dataset Construction and Content

The MusicSimpleQA benchmark contains 500 question–answer pairs, constructed as follows:

• Composition:
  • 300 questions about “popular artists” reflecting current user interest.
  • 200 questions uniformly sampled across eras and genres to ensure coverage of long-tail artists and works.
• Sourcing:
  • Initial questions automatically extracted using DeepSeek-v3 from an in-house encyclopedic corpus focused on singers and songs.
  • Questions are subjected to manual and automated filtering for uniqueness and answer clarity.
• Validation Process:

1. Generation of candidate QAs with DeepSeek-v3.
2. Consistency checks—combining rule-based and LLM-assisted protocols—to eliminate ambiguity or questions with multiple legitimate answers.
3. Final manual spot-checking to achieve high answer precision.

• Answer Format: Each answer consists of a single, unambiguous, verifiable entity such as an album name, birthplace, or award title, minimizing interpretive variation.

3. Prompt and Template Characteristics

MusicSimpleQA adopts a uniform prompt structure for QA:

• Style: Prompts are concise, direct, and restricted to a single sentence ending with a question mark.
• Expected Responses: Answers are required as single-token or single-span spans, with case normalization enforced.
• Illustrative Examples:

Question	Answer
Where is Jay Chou from?	Taipei, Taiwan
What is the name of Jay Chou’s first solo album?	Jay
Which film did Jay Chou win the Best Newcomer Award at the Golden Horse Awards?	The Green Hornet
Which talent show did Zhou Shen debut in?	Super Boy

These conventions are designed to minimize answer ambiguity and support reliable automation of scoring.

4. Automated Agreement Scoring Methodology

To facilitate large-scale benchmarking without human annotation, MusicSimpleQA employs an automated agreement scorer based on DeepSeek-v3:

• Scoring Protocol: For each question $i$ in $N$ total items,

$$\hat{a}_i = \text{model’s answer to question }i, \quad a_i = \text{reference answer}$$

The per-instance agreement indicator:

$$\mathbf{1}_i = \begin{cases} 1, & \text{if DeepSeek-v3 judges } \hat{a}_i \text{ and } a_i \text{ to match} \ 0, & \text{otherwise} \end{cases}$$

The overall agreement score,

$$S_{\mathrm{agree}} = \frac{1}{N} \sum_{i=1}^{N} \mathbf{1}_i$$

• Automation: Each match is determined by DeepSeek-v3 responding to a fixed prompt (“Does ‘$\hat{a}_i$’ match the reference ‘$a_i$’?”), and returning a binary verdict.

A plausible implication is that this fully automated protocol ensures rapid, reproducible evaluation but may miss partial correctness (see Section 7).

5. Benchmark Protocol and Baselines

MusicSimpleQA is released as a fixed 500-question evaluation set without a public train/test split; it is intended solely for evaluation.

• Metric: Accuracy ($S_{\mathrm{agree}}$) is the sole evaluation criterion.
• Evaluator Models: The following LLMs are assessed as reference points:
  • GPT-4o (multimodal system card results)
  • DeepSeek-v3 (in-house)
  • Qwen3-235B-A22B-Instruct (large-scale LLM)
  • Qwen2.5-32B-Instruct (general instructed baseline)
  • Qwen2.5-32B-MuCPT (music-domain continued-pretraining)

Models are ranked by accuracy; no explicit pass/fail threshold is defined.

6. Experimental Results and Comparative Analyses

Performance results on the 500-question set are summarized as follows:

Model	Accuracy ($S_{\mathrm{agree}}$)
GPT-4o	0.6632
DeepSeek-v3	0.7539
Qwen3-235B-A22B-Instruct	0.6719
Qwen2.5-32B-Instruct	0.3599
Qwen2.5-32B-MuCPT	0.7759

Key findings:

• Qwen2.5-32B-MuCPT achieves the highest accuracy, outperforming GPT-4o and other larger, more general models.
• Continued pretraining on music-domain data yields larger factuality gains than simply increasing model scale.
• General instruction finetuning, absent domain-specific data, underperforms substantially (e.g., Qwen2.5-32B-Instruct: 0.3599), emphasizing the critical role of domain corpus construction.

Additional analyses include:

• Token-level quality control: Comparing next-token prediction (no RM), RHO-1 (hard filtering), and MuCPT (soft token down-weighting), soft scoring yields superior performance across both 1.5B and 7B parameter models.
• Data recipe effects: Targeted music corpora (e.g., in-house WeChat-music) support proportionally higher accuracy than broader entertainment domain text, confirming the significance of domain purity and distributional alignment.

7. Strengths, Limitations, and Future Directions

MusicSimpleQA offers several strengths:

• Fully automated, scalable, and reproducible evaluation.
• Minimization of ambiguity through short-form, single-answer prompts.
• Partitioning of questions to cover both popular and long-tail music entities.

Limitations and potential enhancements include:

• Limited dataset size (500 QAs) restricts fine-grained statistical analyses; future expansions (≥1,000 QAs) may improve robustness.
• The binary scoring protocol discards partial correctness; integrated graded or embedding-based similarity metrics could capture nuanced answer similarities.
• Single-turn, factoid-only structure does not probe multi-hop or compositional reasoning; future benchmarks could extend scope to complex chains (e.g., shared collaborators or producer queries).
• Automated scoring’s reliance on a single LLM (DeepSeek-v3) may introduce systematic biases; periodic human audits or ensembling of multiple scorers could enhance reliability.

MusicSimpleQA fills a critical gap in evaluating factual music knowledge in LLMs. By combining a targeted corpus, verifiable single-answer prompts, and automated, agreement-based scoring, it provides an efficient resource for benchmarking and guiding the continued pretraining and alignment of music-domain LLMs (Tian et al., 18 Nov 2025).