MusicJudge: Comparative Music QA

• MusicJudge is a protocol for evaluating music QA by emphasizing semantic and relational comparison between two tracks rather than relying on lexical overlap.
• It employs a four-stage LLM-assisted pipeline—including Music Flamingo and GPT variants—to generate, curate, and validate comparative QA items across diverse musical attributes.
• The protocol addresses challenges like comparative collapse and attribute hallucination through strict quality control, using both human calibration and automated judge scoring metrics.

Searching arXiv for the cited papers and closely related work to ground the article. MusicJudge denotes a judge-style evaluation protocol for music question answering in which semantic correctness and comparative soundness are assessed by a language-capable judge rather than by lexical overlap alone. In the Jamendo-MT-QA benchmark, the term refers specifically to the use of LLM-as-a-Judge for both dataset curation and model evaluation in multi-track comparative music question answering, where systems must compare two tracks, integrate evidence across them, and express the comparison in natural language (Koh et al., 8 Apr 2026). Related judge-model research in speech evaluation, notably AudioJudge, provides a closely aligned methodological frame: pairwise audio comparison, prompt-sensitive judge behavior, aspect decomposition, and correlation with human preferences at the system level (Manakul et al., 17 Jul 2025).

1. Conceptual scope

MusicJudge arises from a limitation in prior Music-QA benchmarks: most of them evaluate single-track understanding, whereas musical description is often relational. Jamendo-MT-QA formalizes this relational setting by requiring explicit comparison of two tracks rather than independent analysis of one clip at a time. The benchmark is therefore designed to measure whether a model can compare two tracks, integrate evidence across them, and produce a comparative answer in one of three output formats: yes/no, short-answer, and sentence-level (Koh et al., 8 Apr 2026).

In this setting, MusicJudge is not merely a scoring heuristic. It is part of the benchmark’s operational definition of quality. The judge protocol is used twice: first, to filter generated comparative QA items during benchmark construction; second, to score sentence-level model outputs during benchmarking. The paper explicitly states that MusicJudge in this context is a judge-style evaluation protocol that scores semantic correctness and comparative soundness, not just lexical overlap (Koh et al., 8 Apr 2026).

A common misconception is to treat music evaluation of this kind as equivalent to caption similarity. The benchmark argues against that reduction. The central target is well-grounded comparative explanation, which requires comparison over attributes such as genre, tempo or energy, vocal characteristics, production style, and mood, and not merely surface-form agreement with a reference answer (Koh et al., 8 Apr 2026).

2. Benchmark substrate and data design

The immediate substrate for MusicJudge is Jamendo-MT-QA, introduced as a dataset and benchmark for multi-track comparative music question answering and constructed on top of Jamendo-QA using Creative Commons-licensed audio tracks on Jamendo (Koh et al., 8 Apr 2026). The licensing model is operationally important: the release includes only annotations and metadata, while respecting the original licensing conditions.

The released benchmark contains 12,173 track pairs and 36,519 comparative QA items, with exactly 3 questions per pair. Each pair yields one item for each of the benchmark’s three question types: yes/no, short-answer, and sentence-level (Koh et al., 8 Apr 2026). The design is intentionally comparative rather than descriptive. The generation prompt requires that every question compare both tracks and explicitly forbids single-track questions.

The pair distribution emphasizes cross-genre comparison. The appendix reports that 91.25% of pairs are different-genre pairs and 8.75% are same-genre pairs (Koh et al., 8 Apr 2026). This design choice shifts the benchmark away from trivial within-genre discrimination and toward more fine-grained reasoning over instrumentation, tempo, mood, production, and vocals. A plausible implication is that genre-label shortcuts are structurally disfavored, although the benchmark itself does not claim that such shortcuts are eliminated.

The benchmark’s scale and controlled output structure make MusicJudge possible as a reproducible evaluation regime. Because each pair produces exactly one item of each type, the benchmark supports separate analysis of binary comparative recognition, track selection, and free-form comparative explanation within a single shared framework (Koh et al., 8 Apr 2026).

3. Construction pipeline and curation logic

Jamendo-MT-QA uses a four-stage LLM-assisted pipeline. In the first stage, each track receives a rich caption generated by Music Flamingo, intended to describe genre, tempo, key, instrumentation, vocal characteristics, production style, mood, and lyrical themes. These captions are then validated by human raters who listen to the audio and judge whether the captions align with the musical content (Koh et al., 8 Apr 2026).

In the second stage, the captions are expanded into single-track QA pairs using GPT-5.1, producing a richer textual representation of each track. In the third stage, two track descriptions at a time—caption, metadata, and single-track QA—are supplied to GPT-5 mini, which generates exactly three comparative QA items per track pair. The three required forms are fixed: a YES/NO question comparing an attribute, a short-answer question whose answer is the audio name, and a SENTENCE question requiring a complete comparative sentence (Koh et al., 8 Apr 2026).

The generation schema is correspondingly explicit. For each pair, the JSON structure contains audio1, audio2, and qa_pairs, and each QA item records type, question, reasoning, and answer. This explicit schema matters because it operationalizes not only the answer but also the intermediate comparative rationale used during generation (Koh et al., 8 Apr 2026).

The fourth stage applies quality control through both human evaluation and LLM-as-a-Judge. The authors first sample 300 items and have four annotators score each item on Correctness, Comparative Validity, Reasoning Quality, and Difficulty. They then compare these human mean scores to the scores produced by GPT-5 mini as judge. The reported human mean versus GPT-5 mini mean is 4.79 vs. 4.87 for Correctness, 4.83 vs. 4.61 for Comparative Validity, 4.78 vs. 4.37 for Reasoning Quality, and 2.25 vs. 2.17 for Difficulty (Koh et al., 8 Apr 2026).

The filtering rule is intentionally strict. The final benchmark retains only those QA groups for which all three QA items receive perfect 5/5/5 scores on the three semantic dimensions: Correctness, Comparative Validity, Reasoning Quality. Difficulty is used for analysis rather than filtering. This conservative rule reduces the generated collection from 13,097 track pairs to 12,173 final track pairs (Koh et al., 8 Apr 2026). The paper explicitly states that this filtering prioritizes semantic reliability rather than coverage.

4. Judge protocol and scoring criteria

MusicJudge functions as an evaluation layer over model outputs, especially for sentence-level comparative answers. The benchmark applies task-specific metrics: accuracy for yes/no items, exact-match accuracy on the correct track identifier for short-answer items, and BLEU, ROUGE-1/2/L, BERTScore, and an LLM-as-a-Judge score for sentence-level answers (Koh et al., 8 Apr 2026).

For sentence-level evaluation, the judge uses a 0–5 scale. The rubric is defined as follows: 0 for “Completely wrong or irrelevant,” 1 for “Mostly wrong with very minor correct elements,” 2 for “Partially correct but missing key points,” 3 for “Correct on main points but missing details or minor errors,” 4 for “Mostly correct with only minor omissions,” and 5 for “Fully correct, capturing all key comparative information” (Koh et al., 8 Apr 2026).

The judge prompt requires comparison between the prediction and the ground-truth answer and instructs the model to evaluate semantic similarity and correctness without requiring exact wording. It explicitly directs attention to genre comparison, tempo or energy, vocal characteristics, production style, and mood, and constrains the output to a JSON object with score and explanation (Koh et al., 8 Apr 2026). This is the operational core of MusicJudge: the evaluation target is semantic adequacy of comparative music reasoning, not n-gram overlap.

A central implication of this protocol is methodological rather than merely metric. The benchmark reports that surface-form metrics like BLEU and ROUGE are poor indicators of quality for this task, remaining low even when BERTScore and LLM-judge scores are much higher (Koh et al., 8 Apr 2026). This suggests that comparative music answers exhibit high lexical variability while preserving evaluable semantic structure.

5. Benchmarked systems and empirical profile

The benchmark evaluates two modeling setups. The first consists of multi-audio end-to-end models, which take two audio inputs plus the question. The second consists of caption-based baselines, in which each track is captioned independently and an LLM compares the resulting textual descriptions (Koh et al., 8 Apr 2026). The evaluated models are GPT-4o Audio, GPT-4o-mini Audio, Qwen2-Audio, Qwen3-Omni, Music Flamingo, and MU-LLaMA. Because of cost and compatibility, some models are run on the full 12,173-pair benchmark, whereas GPT-style audio models are evaluated on a subset of 2,010 track pairs.

The reported findings are structurally consistent across models. Yes/no and short-answer questions are easier than sentence-level comparative generation. Caption-based models can be surprisingly competitive, and Music Flamingo is described as notably strong, at times outperforming direct multi-audio baselines on sentence-level metrics. The benchmark further reports that direct access to two audio clips does not guarantee strong comparative explanation quality (Koh et al., 8 Apr 2026).

These findings delimit what MusicJudge is actually measuring. Performance on constrained outputs does not imply robust comparative explanation. The benchmark identifies the main bottleneck for current models as well-grounded comparative explanation, not merely identifying the correct track or the direction of comparison (Koh et al., 8 Apr 2026). In other words, the hard part is generating a semantically faithful comparative account rather than making a coarse binary discrimination.

The error analysis isolates three recurrent failure modes. Comparative Collapse occurs when a model avoids making a real comparison and instead gives a generic summary. Attribute Hallucination occurs when unsupported musical attributes are invented. Granularity Mismatch occurs when the comparison is made at the wrong level of detail (Koh et al., 8 Apr 2026). These categories clarify why MusicJudge cannot be reduced to answer matching: the central failure is often structural misreasoning rather than literal contradiction.

6. Relation to judge-model research beyond music

The nearest general analogue is AudioJudge, which studies Large Audio Model (LAM) as a Judge for speech evaluation and asks whether a single prompted judge can replace fragmented specialized evaluators across tasks such as pronunciation, speaking rate, speaker identification, speech quality, and human preference simulation (Manakul et al., 17 Jul 2025). Although AudioJudge is about speech rather than music, several design principles map directly onto MusicJudge.

First, AudioJudge finds that pairwise evaluation is more reliable than pointwise scoring (Manakul et al., 17 Jul 2025). MusicJudge already adopts this pairwise comparative format at the data level, since every question is grounded in a two-track comparison. Second, AudioJudge shows that prompt engineering matters substantially, with audio concatenation and in-context learning improving performance across fine-grained comparison and human preference simulation tasks. The best fine-grained configuration is reported as “Examples+Test Concat” with 4-shot examples (Manakul et al., 17 Jul 2025). A plausible implication is that music judges may also benefit from continuous audio context rather than heavily fragmented audio-text alternation.

Third, AudioJudge introduces a multi-aspect ensemble composed of a lexical judge, a paralinguistic judge, and a speech-quality judge, with majority voting used for aggregation (Manakul et al., 17 Jul 2025). The paper itself states that the idea is directly transferable to music. This suggests a possible MusicJudge decomposition into content or lyrics, expressive or performance-related features, and sonic production quality, although that decomposition is proposed as an analogy rather than reported as an implemented music benchmark.

At the system level, AudioJudge reports up to 0.91 Spearman correlation with human preferences on SpeakBench, specifically 0.912 for the multi-aspect ensemble using Gemini-2.5-Flash in the zero-shot setting (Manakul et al., 17 Jul 2025). It also reports up to 0.931 on ChatbotArena-Spoken using GPT-4o-Audio with 4-shot Examples+Test Concatenation. These results do not establish the same correlation for music, but they indicate that judge-style audio evaluation can serve as a realistic automated benchmarker for system ranking when prompting and aggregation are carefully designed.

7. Biases, limitations, and open technical issues

MusicJudge inherits several limitations from the broader judge-model paradigm. Within Jamendo-MT-QA, the use of LLM-as-a-Judge is central to both curation and evaluation, but the benchmark also acknowledges the necessity of human anchoring through the initial 300-item evaluation and comparison between annotator means and judge scores (Koh et al., 8 Apr 2026). This indicates that judge reliability is established by calibration, not assumed a priori.

AudioJudge makes the bias problem more explicit. Its robustness analysis reports that under acoustic noise, GPT-4o-Audio remains stable even at 1 dB signal-to-noise ratio, with unchanged predictions for 85% of chosen examples, 93% of not-chosen examples, and 73% of ties (Manakul et al., 17 Jul 2025). At the same time, it identifies verbosity bias and positional bias, and notes a recency bias in speaking-rate judgments for GPT-4o-Audio. It further reports that positional bias increases when the discrimination task becomes harder (Manakul et al., 17 Jul 2025).

For MusicJudge, these observations are best read as methodological cautions rather than direct empirical claims about music. This suggests that any judge comparing two tracks or two generated answers should consider position randomization, explicit tie handling, and robustness checks. A plausible implication is that comparative music evaluation may be especially sensitive to order effects when the contrast between tracks is subtle or when models must justify a fine-grained difference in mood, production, or vocal delivery.

Another limitation concerns the relation between semantic evaluation and lexical metrics. Jamendo-MT-QA explicitly reports that BLEU and ROUGE are poor indicators of quality for sentence-level comparative music QA, even when BERTScore and judge scores are much higher (Koh et al., 8 Apr 2026). This does not render lexical metrics useless, but it confines their interpretive value. In MusicJudge-style settings, lexical overlap is at most a secondary proxy for a target that is fundamentally semantic, relational, and attribute-grounded.