AnyAudio-Judge Bench: Bilingual Audio Evaluation

• AnyAudio-Judge Bench is a bilingual benchmark that measures whether audio clips satisfy complex, multi-constraint instructions spanning speech, sound, music, and mixed audio.
• The evaluation employs a dynamic rubric-based paradigm that decomposes instructions into atomic, verifiable items judged with binary yes/no decisions.
• Empirical results highlight that dynamic rubric prompting boosts classification accuracy by over 10% compared to holistic scoring, supporting robust evaluation.

AnyAudio-Judge Bench is a bilingual benchmark for evaluating audio instruction following through a dynamic rubric-based judging paradigm rather than a single holistic match score. It was introduced together with the AnyAudio-Judge Corpus and the AnyAudio-Judge model to assess whether an audio clip satisfies the fine-grained, compositional constraints of an instruction across speech, sound, music, and mixed audio. The benchmark comprises 7,920 curated samples organized into seven subsets, maintains a strict 1:1 ratio of positive and negative samples, and uses deliberately constructed hard negatives intended to preserve much of the original caption while perturbing a small but critical attribute (Li et al., 2 Jun 2026).

1. Evaluation target and problem formulation

AnyAudio-Judge Bench addresses a specific weakness in automated evaluation for instruction-guided audio generation: existing methods can often detect whether an audio clip is globally similar to a prompt, but they are substantially weaker at determining whether the clip satisfies all of the prompt’s fine-grained, compositional constraints. The benchmark is motivated by three limitations that the paper identifies in prior practice. First, embedding-based metrics such as CLAP similarity provide coarse global semantic closeness and are not sensitive to subtle but important attribute mismatches. Second, human evaluation is expensive, unscalable, and subjective. Third, surrogate LLM or LALM judges typically reduce the task to a single holistic match or mismatch decision, which makes them difficult to interpret and weak at diagnosing which part of a complex instruction failed (Li et al., 2 Jun 2026).

The intended evaluation target is therefore not generic audio similarity. It is instruction alignment under multi-constraint prompts that may specify speaker identity, emotion, dialect, chronology of sound events, instrumentation, background layers, scene composition, and foreground/background mixing. In this setting, a single scalar similarity score obscures partial correctness: an output may satisfy some constraints and violate others. AnyAudio-Judge Bench operationalizes this problem as balanced binary alignment detection, with positive samples that match the instruction and hard negatives that differ in narrow but perceptually meaningful ways.

The benchmark covers four audio domains—speech, sound, music, and mixed audio—and seven subsets: Speech-Real, Speech-Gen, Sound-Real, Sound-Gen, Music-Real, Music-Gen, and Mix. The Mix subset contains only real-world samples because the authors state that high-fidelity mixed-audio synthesis remains challenging. The benchmark is bilingual in English and Chinese, and for every sample the instruction is translated in both directions to create parallel English and Chinese evaluation sets while the audio itself is unchanged and spoken content is not translated (Li et al., 2 Jun 2026).

2. Dynamic rubric-based evaluation paradigm

The central methodological contribution is a dynamic rubric-based evaluation paradigm. For an audio-instruction pair $(a, i)$, where $a$ is the audio and $i$ is the instruction, the instruction is decomposed into a variable number $n$ of rubric items $\{p_1, p_2, \dots, p_n\}$. These items are intended to be independent, faithful to the caption, and verifiable by listening. The decomposition is generated by Qwen3-30B-A3B-Instruct-2507 using a structured prompt that requires atomicity, verifiability, faithfulness, completeness, and objectivity. It also enforces a semantic convention: every item is phrased so that answering “yes” means the audio matches the caption on that aspect; if the caption contains a negative requirement, the rubric question is written in negative form so that “yes” still indicates correctness (Li et al., 2 Jun 2026).

Each rubric item is then judged as a binary yes/no decision. The paper states that the item-level score is obtained by comparing the logits for “yes” and “no” with a two-way softmax, and that the final alignment score is the average yes probability across all rubric items:

$$p_j^{\mathrm{yes}} = \frac{\exp(z_j^{\mathrm{yes}})}{\exp(z_j^{\mathrm{yes}}) + \exp(z_j^{\mathrm{no}})}$$

$s = \frac{1}{n}\sum_{j=1}^{n} p_j^{\mathrm{yes}}$

This scoring rule is paired with a structured output format. The decompose-judge prompt instructs the evaluator to rely only on perceivable content, treat all questions as aligned so that “yes = match,” and answer “no” under ambiguity. The judge output is a JSON array with one record per rubric item:

$a$2

The rubric construction procedure is itself filtered. After decomposition, a separate rubric-filtering prompt checks whether each generated question is fully grounded in the caption or introduces hallucinated information; each item is marked keep or remove. This filtering step is important because decomposition quality is a potential failure point. The allowed decomposition dimensions include audio components, speech dimensions per speaker, speech content, sound-effect type and temporal sequence, music style and instrumentation, and mixing relationships such as foreground/background hierarchy and dynamic changes (Li et al., 2 Jun 2026).

3. Benchmark composition and data construction

AnyAudio-Judge Bench contains 7,920 curated samples across the four audio domains and seven subsets. The benchmark is evaluated with classification accuracy because it enforces a strict 1:1 ratio of positive and negative samples. The paper does not explicitly state whether 7,920 counts one language view or both language views combined; the wording “a bilingual benchmark comprising 7,920 meticulously curated samples” strongly suggests that the released benchmark is bilingual and totals 7,920 samples overall. The exact per-subset counts are not recoverable from the text dump, but the seven-subset composition is explicit (Li et al., 2 Jun 2026).

Construction differs by subset and combines strong positive-pair verification with adversarial hard-negative generation.

Subset	Basis	Construction note
Speech-Real	InstructTTSEval	Positives from audio-description pairs; negatives from swapping and perturbation
Speech-Gen	Qwen3TTS-12Hz-1.7B-VD, MOSS-VoiceGenerator, MiMo-Audio-7B-Instruct	Gemini dual-pass mines genuine synthesis failures as hard negatives
Sound-Real	Clotho v2 test	Positives filtered by CLAP with similarity $\ge 0.6$
Sound-Gen	AudioGen, AudioLDM2, Stable Audio	CLAP selects aligned outputs before negative construction
Music-Real	Song Describer Dataset	Gemini with music-specific prompts filters positives and verifies negatives
Music-Gen	MusicGen, ACE-Step, Stable Audio	Gemini forms initial positive and negative sets
Mix	Cinematic clips	Real-only subset with Gemini-generated comprehensive captions

Negatives are produced by two main strategies. Instruction swapping exchanges instructions between samples to create semantic mismatches. Attribute perturbation uses Qwen3-30B-A3B-Instruct-2507 to alter a small number of perceptually meaningful attributes in a caption while preserving fluency and plausibility. These perturbations are tailored to each domain. For speech, the prompt modifies 1–2 attributes such as dynamic changes, emotion intensity, terminal intonation, accent/dialect, timbre, age, or pitch. For sound, it simulates TTA failures such as secondary sound omission, spurious sound addition, temporal sequence enrichment, dynamic variation enrichment, acoustic environment enrichment, semantic category drift, and count mismatch. For music, it simulates TTM failures such as instrument substitution, genre/style drift, tempo/rhythm mismatch, mood/energy mismatch, texture enrichment, vocal character mismatch, and adding subtle background layers (Li et al., 2 Jun 2026).

The bilingual design is explicitly symmetric. For each sample, the instruction is translated in both directions to create English and Chinese evaluation views. Because the audio itself is unchanged, this design probes whether alignment evaluation is stable across prompt language while holding acoustic evidence fixed. A plausible implication is that the benchmark tests not only audio understanding, but also the cross-lingual robustness of the evaluation interface.

4. Corpus, evaluator, and training pipeline

The benchmark is paired with the AnyAudio-Judge Corpus, a 105K-sample training corpus that is disjoint from the benchmark and therefore out-of-distribution relative to benchmark evaluation. Its composition is explicit: 30K Speech, 30K Sound, 30K Music, and 15K Mixed, with a strict 1:1 positive-to-negative ratio within each modality. Positive audio-caption pairs come from internal high-quality speech assets, AudioCaps for sound, the MusicBench training split for music, and newly collected movie audio clips with Gemini-verified captions for mixed audio. Negatives are created using the same swapping and perturbation pipeline as in the benchmark. Each sample carries two additional supervision layers: per-rubric binary labels and explicit Chain-of-Thought rationales (Li et al., 2 Jun 2026).

The annotation pipeline has three stages. First, the original caption is decomposed into $n$ binary rubric items. Second, the audio is paired either with its true caption or with a constructed negative caption. Third, per-rubric yes/no labels and CoT rationales are generated by comparing the original true caption with the target caption. Positive samples receive all-“yes” rubric labels; negative samples are only partially contradicted, because a text-only LLM decides item by item which rubric statements remain true and which are contradicted.

The evaluator, AnyAudio-Judge, is built by starting from Qwen3-Omni-30B-A3B-Captioner and training in two stages: Supervised Fine-Tuning followed by Group Relative Policy Optimization. SFT teaches the model to take an audio-instruction pair plus decomposed rubric items and output binary judgments with CoT rationales. GRPO then focuses on difficult cases. The paper states that four rollouts are first performed on the corpus and that samples consistently answered correctly are removed, leaving 8,454 harder samples for GRPO optimization. The reward is a weighted sum of three terms: format consistency with weight 0.1, global accuracy with weight 0.2, and balanced rubric accuracy with weight 0.7. The final model outputs a JSON array in rubric-aligned format, with rubric ID, binary answer, and supporting evidence (Li et al., 2 Jun 2026).

Training details are fully specified in prose. Both stages use 16 H20 GPUs with 96GB memory. SFT is full-parameter fine-tuning for one epoch with batch size 4 per device, gradient accumulation 1, and learning rate $1\times10^{-5}$. GRPO uses LoRA with rank 16 and alpha 32 for one epoch, 8 generations per prompt, batch size 8 per device, gradient accumulation 1, and learning rate $a$0. The benchmark, the 105K corpus, and the model are available at the project repository: https://github.com/CuCl-2/AnyAudio-Judge.

5. Evaluation protocol and empirical results

Benchmark evaluation uses ACC and is reported separately on Chinese and English. The baselines are Audio-Flamingo3, MiDashengLM, Kimi-Audio-7B-Instruct, Qwen2.5-Omni-7B, Qwen3-Omni-30B-A3B-Instruct, Qwen3-Omni-30B-A3B-Captioner, and Gemini-2.5-Pro. For all LALM baselines, both holistic prompting and dynamic rubric prompting are tested. The main result is that dynamic rubric prompting almost always improves over holistic prompting for general-purpose audio-LLMs, often substantially. MiDashengLM rises from 50.75 to 68.35 on Chinese and from 49.71 to 67.94 on English; Qwen2.5-Omni-7B rises from 51.76 to 71.93 on Chinese and from 51.76 to 72.24 on English. Gemini-2.5-Pro is the notable exception, with 80.01 holistic versus 78.31 dynamic rubric on Chinese and 77.72 versus 77.27 on English (Li et al., 2 Jun 2026).

AnyAudio-Judge is the strongest reported evaluator on the benchmark. Its average ACC is 85.26 on Chinese and 84.45 on English, outperforming all prompt-only baselines. The strongest baseline average is Gemini-2.5-Pro at 80.01 on Chinese and 77.72 on English under holistic prompting, or 78.31 and 77.27 under dynamic rubric prompting. The benchmark is therefore not near ceiling even for strong proprietary models.

The per-subset scores of AnyAudio-Judge show that real-sample subsets are easier than generated subsets, and that mixed audio remains demanding but tractable.

Subset	Chinese ACC	English ACC
Speech-Real	93.33	91.42
Speech-Gen	80.15	78.85
Sound-Real	77.90	78.70
Sound-Gen	82.50	84.67
Music-Real	92.22	91.25
Music-Gen	80.13	79.87
Mix	90.60	86.40

The ablation study isolates the effect of decomposition and training. For Qwen3-Omni-30B-A3B-Captioner, holistic judgment yields 65.33 on Chinese and 64.24 on English. Dynamic rubric prompting alone raises this to 76.66 and 76.77. Adding SFT raises performance to 84.02 and 83.78. Adding SFT and GRPO yields 85.26 and 84.45. This indicates that the rubric decomposition itself accounts for roughly an 11–12 point gain over holistic evaluation, while supervised judge training contributes another large gain and GRPO adds a smaller final improvement (Li et al., 2 Jun 2026).

The paper also evaluates external transfer on PAM, where AnyAudio-Judge achieves LCC 0.614, SRCC 0.601, and KTAU 0.435, outperforming CLAPScore and AQAScore variants on all three reported metrics. In downstream use, the rubric-aggregated scalar score $a$1 serves as a reward model for GRPO-based reinforcement learning of DiTAR. The authors report that the reward rises steadily during training and that the AnyAudio-Judge-optimized model outperforms the base model on InstructTTSEval under both Gemini-based scores and human preference judgments. The evaluator is also used to score several InstructTTS systems, with reported AnyAudio-Judge scores of 87.5 for Gemini 2.5-Pro, 84.8 for Qwen3-TTS-12Hz-1.7B-VD, 80.6 for MOSS-VoiceGenerator, and 81.1 for MiMo-Audio-7B-Instruct.

6. Limitations and comparative position

The benchmark’s main stated limitations concern decomposition quality and inference cost. The approach depends on the quality of rubric decomposition; if decomposition misses an implicit constraint or splits one attribute too finely, evaluation quality may degrade. The decomposition stage also adds inference latency. These are the principal failure cases noted by the authors. A further caveat is that exact per-subset counts are not recoverable from the text dump, and the paper does not explicitly resolve whether the 7,920 total is counted per language view or across the bilingual release as a whole (Li et al., 2 Jun 2026).

Within the broader literature on audio judging, AnyAudio-Judge Bench occupies a distinct position. Relative to "ParaPairAudioBench" (Jeon et al., 23 Jun 2026), it is broader in audio domain coverage but less speech-diagnostic: ParaPairAudioBench focuses on pairwise A/B/Tie judgments over five paralinguistic speech dimensions, transcript control, and calibration via abstention, whereas AnyAudio-Judge Bench targets instruction-audio alignment across speech, sound, music, and mixed audio. Relative to "AudioCapBench" (Qiu et al., 27 Feb 2026), it is also different in task formulation: AudioCapBench evaluates free-form caption generation quality with an LLM-as-Judge over accuracy, completeness, and hallucination, whereas AnyAudio-Judge Bench evaluates whether an audio clip satisfies a compositional instruction through dynamic rubric verification. Relative to "Judge Anything" (Pu et al., 21 Mar 2025), AnyAudio-Judge Bench is narrower in modality coverage but more audio-specific: JudgeAnything spans text, image, video, and audio across 15 any-to-any categories, while AnyAudio-Judge Bench is dedicated to audio instruction following and hard-negative alignment detection.

A common misconception is to treat the benchmark as a conventional caption-audio matching dataset. Its design is more specific. It is constructed around dynamic decomposition of instructions, bilingual symmetry, subset-specific negative mining, and rubric-level evidence generation. Another common misconception is to interpret it as a benchmark for audio generation alone. The benchmark evaluates evaluators: it measures whether a judge can discriminate subtle matched and mismatched instruction-audio pairs, localize failure sources through rubric items, and provide a reward signal usable in downstream reinforcement learning.

In that sense, AnyAudio-Judge Bench serves as a benchmark, an evaluation protocol, and a training target for rubric-conditioned audio alignment assessment. Its most characteristic properties are cross-domain audio coverage, bilingual prompting, hard-negative construction, interpretable rubric-level scoring, and the pairing of benchmark evaluation with a dedicated trained judge model (Li et al., 2 Jun 2026).