Multi-Aspect Ensemble AudioJudge
- The paper presents a prompt-only framework that decomposes speech evaluation into three specialized judges for lexical content, speech quality, and paralinguistic properties.
- It leverages pairwise comparisons, in-context learning, and audio concatenation to achieve human-aligned results on benchmarks like SpeakBench.
- The framework enhances comparative judgment in complex speech tasks while exposing limitations such as verbosity and positional biases.
Searching arXiv for the specified paper and closely related context. Multi-Aspect Ensemble AudioJudge is a prompt-only, judge-style framework for speech evaluation that decomposes assessment into three specialized subjudges and aggregates their pairwise decisions by majority voting. Introduced in "AudioJudge: Understanding What Works in Large Audio Model Based Speech Evaluation" (Manakul et al., 17 Jul 2025), it is designed to make evaluation more general-purpose and more aligned with human preferences than a single large audio model judge. The framework targets settings in which speech must be assessed simultaneously along lexical, acoustic, and expressive dimensions, particularly the SpeakBench benchmark, where content correctness, speech quality, and paralinguistic appropriateness must all be considered (Manakul et al., 17 Jul 2025).
1. Conceptual basis
The central premise of Multi-Aspect Ensemble AudioJudge is that speech evaluation is not a single-dimensional problem. The framework distinguishes three axes that are explicitly named in the source work: lexical content, speech quality, and paralinguistic properties (Manakul et al., 17 Jul 2025). Lexical content concerns whether the response says the right thing; speech quality concerns whether it sounds clear, fluent, natural, and high quality; paralinguistic properties concern tone, accent, emotion, style, prosody, speed, and related attributes.
The method is motivated by the observation that a single judge prompt tends to blur these aspects together. The paper states that even strong large audio models can perform well on coarse content judgments while struggling on finer paralinguistic distinctions (Manakul et al., 17 Jul 2025). The ensemble therefore uses specialization as a mechanism for reducing interference between criteria. This suggests that the framework treats evaluation error partly as a prompt allocation problem: if distinct perceptual and semantic criteria are conflated in one instruction, the judge may attend unevenly across them.
The framework is described as general-purpose rather than task-specific. Its goal is not to train separate scoring models for each evaluation target, but to retain the flexibility of a prompt-only evaluation scheme while imposing structural separation among dimensions of judgment (Manakul et al., 17 Jul 2025).
2. Ensemble structure and decision rule
The framework introduces three specialized judges, implemented by the same underlying large audio model but differentiated by prompt content (Manakul et al., 17 Jul 2025). Each judge is assigned a narrowly defined scope.
| Judge | Primary focus | Explicitly ignored dimensions |
|---|---|---|
| Lexical Judge | textual content; accuracy, completeness, organization, language use | pronunciation, voice quality, style, tone, speed, accent, and other audio features |
| Paralinguistic Judge | tone, prosody, expressiveness, accent/pronunciation patterns when relevant | lexical content |
| Speech Quality Judge | clarity, naturalness, fluency, pronunciation correctness, artifacts/noise | content and expressive features |
Each specialized judge produces a pairwise decision over two candidate audios. The output label space is
where $1$ denotes that Audio 1 is better, $2$ denotes that Audio 2 is better, and denotes no clear winner (Manakul et al., 17 Jul 2025).
Final aggregation is performed by majority voting across the three specialized judges. The paper writes the ensemble rule as
with the final output determined by agreement among the three subjudges (Manakul et al., 17 Jul 2025). More concretely, if two or more judges agree, that becomes the final output; when votes are split or include ties, the majority rule still determines the final label as described in the system design.
This architecture is notable because specialization is achieved purely through prompting. No new task-specific model components, auxiliary losses, or retraining procedures are introduced in the description. A plausible implication is that the framework is intended to isolate the contribution of evaluation decomposition itself rather than confound it with architectural modification.
3. Prompting, concatenation, and in-context learning
The Multi-Aspect Ensemble AudioJudge is built around pairwise comparison. For system-level evaluation on SpeakBench, the input consists of the instruction audio, Audio 1, and Audio 2. For fine-grained tasks, the input is typically just Audio 1 and Audio 2 (Manakul et al., 17 Jul 2025). This pairwise format is significant because the same paper later argues that pointwise MOS-style scoring is weaker than relative comparison.
A major design variable in the study is how audio is presented to the judge. The best-performing configuration is Examples + Test Concatenation, in which all in-context example audios are concatenated into one long audio stream and the test audios are also concatenated into one stream (Manakul et al., 17 Jul 2025). The paper emphasizes that concatenation helps because large audio models may understand continuous audio streams better than fragmented, alternating audio-text turns. For the multi-aspect ensemble, the prompt structure remains the same across judges; only the judge-specific system prompt changes.
The framework also uses in-context learning (ICL). In the best setup, the model is given 4-shot examples together with Examples + Test Concatenation (Manakul et al., 17 Jul 2025). The study compares this with no concatenation, pair example concatenation, examples concatenation, and test concatenation. Among these alternatives, Examples + Test Concatenation with 4 shots is reported as the best general configuration.
The three prompts are explicitly written to exclude irrelevant criteria. The lexical judge is instructed to focus exclusively on lexical content and ignore audio qualities; the paralinguistic judge focuses on tone, prosody, expressiveness, and accent/pronunciation; the speech quality judge focuses on clarity, naturalness, fluency, pronunciation, and audio artifacts (Manakul et al., 17 Jul 2025). This prompt design is central to the ensemble’s logic: decomposition is not merely analytical, but operationally enforced through instruction-level scope control.
4. Evaluation tasks, benchmarks, and metrics
The source paper evaluates AudioJudge in two broad regimes: audio characteristic detection and human preference correlation / system ranking (Manakul et al., 17 Jul 2025).
For audio characteristic detection, the evaluated tasks are pronunciation, speaking rate, speaker identification, and speech quality. The datasets and pair counts are given explicitly:
- Pronunciation: 200 pairs
- Speaking Rate: 187 pairs
- Speaker Identification: 200 pairs
- SOMOS: 200 pairs
- TMHINTQ: 200 pairs
- ThaiMOS: 200 pairs (Manakul et al., 17 Jul 2025)
Under the best configuration, Examples + Test Concatenation, 4-shot, the reported accuracies are:
- Pronunciation: 66.5%
- Speaking rate: 55.3%
- Speaker ID: 70.0%
- SOMOS: 71.0%
- TMHINTQ: 74.5%
- ThaiMOS: 64.0%
- Average: 66.9% (Manakul et al., 17 Jul 2025)
Human performance on the same tasks is reported as:
- Pronunciation: 69.5%
- Speaking Rate: 77.8%
- Speaker ID: 83.0%
- SOMOS: 71.0%
- TMHINTQ: 78.0%
- ThaiMOS: 77.5%
- Average: 76.2% (Manakul et al., 17 Jul 2025)
For system-level evaluation, the main metric is Spearman correlation between model judgments and human preferences:
This is used to measure how well the model preserves human ordering over systems (Manakul et al., 17 Jul 2025).
Two datasets are used for ranking evaluation. ChatbotArena-Spoken is a speech version of two-turn ChatbotArena conversations and is used for lexical content evaluation; it comprises 1K sampled evaluation items from 7.8K synthesized datapoints. SpeakBench is a multi-aspect speech-in/speech-out benchmark with 82 instructions, 13 systems, 508 human judgments, and 1,066 total datapoints. SpeakBench instructions cover pronunciation, speaking style / emotion / tone, prosody / delivery, and non-linguistic sound effects (Manakul et al., 17 Jul 2025).
5. Empirical results
On ChatbotArena-Spoken, the paper reports the following Spearman correlations:
- GPT-4o-Audio baseline: 0.902
- GPT-4o-Audio + ICL: 0.931
- Gemini-2.5 baseline: 0.805
- Gemini-2.5 + ICL: 0.877 (Manakul et al., 17 Jul 2025)
On SpeakBench, the reported baseline and ICL results are:
- GPT-4o-Audio baseline: 0.731
- GPT-4o-Audio + ICL: 0.775
- Gemini-2.5 baseline: 0.846
- Gemini-2.5 + ICL: 0.857 (Manakul et al., 17 Jul 2025)
For the Multi-Aspect Ensemble specifically on SpeakBench, the reported correlations are:
- GPT-4o-Audio: 0.802
- Gemini-2.5-Flash: 0.912
With ICL:
- GPT-4o-Audio: 0.846
- Gemini-2.5-Flash: 0.857 (Manakul et al., 17 Jul 2025)
These values establish several points. First, the ensemble improves over the single-judge baseline on SpeakBench for GPT-4o-Audio in both zero-shot and ICL settings. Second, the paper explicitly states that the ensemble gives the best zero-shot result for Gemini-2.5-Flash on the multi-aspect benchmark (Manakul et al., 17 Jul 2025). Third, the interaction between prompting strategies is not monotonic: combining ensemble and ICL helps GPT-4o-Audio but slightly hurts Gemini-2.5-Flash. This is an important methodological caveat, because it indicates that decomposition and example conditioning do not combine additively in a model-independent way.
The study also compares AudioJudge with specialized models for speech quality evaluation. AudioJudge uses <0.2 hours of annotation/in-context data and obtains SOMOS: 71.0, TMHINTQ: 74.5, and ThaiMOS: 64.0 (Manakul et al., 17 Jul 2025). The paper states that UTMOS, MOSANET+, and SALMONN-FT outperform AudioJudge on in-domain tasks, but AudioJudge is more flexible and more robust cross-domain, especially on ThaiMOS (Manakul et al., 17 Jul 2025). This comparison situates the ensemble within a trade-off between domain-specialized optimization and prompt-based generality.
6. Robustness, biases, and limitations
The paper reports robustness experiments in zero-shot mode without audio concatenation in order to isolate inherent behavior (Manakul et al., 17 Jul 2025). Under added white Gaussian noise on ChatbotArena-Spoken, GPT-4o-Audio remains stable. At 1 dB SNR, unchanged predictions are:
- 85% for chosen responses
- 93% for not-chosen responses
- 73% for ties (Manakul et al., 17 Jul 2025)
The paper interprets this as strong robustness to acoustic corruption. However, robustness to noise does not eliminate systematic judgment biases.
One reported issue is verbosity bias. On tie-rated examples where one response is at least 5 tokens longer, all models prefer the longer response, and this bias is statistically significant with (Manakul et al., 17 Jul 2025). The correlation metric remains high partly because tie cases are only about 20% of the data and because the bias does not consistently favor one system over another. Still, the result indicates that judge outputs can reflect superficial response length cues rather than the intended evaluation target.
A second issue is positional bias. GPT-4o-Audio and Gemini-2.5-Flash show a first-position bias on lexical evaluation, while Gemini-1.5-Flash exhibits stronger instability (Manakul et al., 17 Jul 2025). On non-lexical tasks, biases are even more pronounced; speaking rate shows a recency bias, and positional bias becomes stronger when discrimination is harder, especially when MOS differences are small (Manakul et al., 17 Jul 2025). This challenges any simplistic interpretation of large audio model judges as purely content-sensitive evaluators.
The paper also lists broader limitations:
- Paralinguistic tasks remain hard, especially speaking rate and speaker identification.
- Cost and scalability are constrained because the best prompting setup uses multiple in-context examples and multiple judges.
- Benchmark size is limited, since SpeakBench has 82 instructions and 13 systems.
- Biases persist, including verbosity and positional bias.
- Absolute scoring remains weak, with pointwise MOS-style scoring performing worse than pairwise judgment (Manakul et al., 17 Jul 2025).
These limitations are methodologically significant. They imply that the framework improves system-level alignment with human preferences without resolving deeper calibration problems in audio evaluation. A plausible implication is that the ensemble is best understood as a stronger comparative judge, not as a fully reliable absolute scorer.
7. Significance within speech evaluation
The importance of Multi-Aspect Ensemble AudioJudge lies in how it reframes prompt-based speech evaluation. Rather than treating a large audio model as a monolithic evaluator, it decomposes speech assessment into specialized subjudgments and then recombines them through a simple aggregation rule (Manakul et al., 17 Jul 2025). The source paper characterizes the contrast succinctly: a single judge is broader but noisier, whereas the multi-aspect ensemble is more structured, more interpretable, and more aligned with human evaluation on complex speech tasks (Manakul et al., 17 Jul 2025).
The framework is especially relevant for settings such as SpeakBench, where evaluation requires balancing what was said, how it was said, and how good it sounded. By assigning each judge a narrower goal, making prompt instructions more precise, and allowing different dimensions to be evaluated independently before majority-vote aggregation, the method aims to reduce criterion conflation (Manakul et al., 17 Jul 2025). This suggests a broader methodological lesson for judge-model research: prompt specialization can function as a lightweight substitute for task-specific evaluator design when the target construct is inherently multi-factor.
At the same time, the reported results show that decomposition is not a universal remedy. The gains are strongest in multi-aspect evaluation, but paralinguistic discrimination remains difficult, bias effects persist, and model-specific interactions with ICL can be unfavorable (Manakul et al., 17 Jul 2025). For that reason, Multi-Aspect Ensemble AudioJudge is best viewed as a structured evaluation protocol for large audio models rather than a settled solution to speech assessment. Its main contribution is to demonstrate that prompt-only specialization and majority-vote aggregation can materially improve alignment with human preferences on complex speech benchmarks while preserving the flexibility of general-purpose audio judges (Manakul et al., 17 Jul 2025).