- The paper introduces a preference-aware ASR benchmark that integrates explicit user instructions and a custom normalizer to evaluate transcription adherence.
- The paper demonstrates that instruction-following significantly alters error metrics, exposing model strengths and issues such as entity hallucination.
- The paper emphasizes the need for advanced evaluation metrics capturing nuanced transcription outputs for more reliable ASR deployments.
Preference-ASR: Design and Implications of a Preference-Aware ASR Benchmark
Motivation and Background
The adoption of SpeechLLMs has altered both the expectations and capabilities of modern automatic speech recognition (ASR) systems. Unlike traditional systems constrained by static output formats, SpeechLLMs permit users to specify desired transcription styles and formatting through natural-language instructions. However, standard ASR benchmarks and evaluation practices remain largely format-agnostic, with fixed references and conventional normalizers obscuring the degree to which systems comply with user preferences. Moreover, inconsistencies across reference corpora—including handling of numeric expressions, entities, disfluencies, and casing—make it difficult to disambiguate transcription performance from annotation artifacts.
Construction of Preference-ASR
Preference-ASR was designed to address these gaps by providing explicit user instructions for transcription format and evaluating ASR output adherence to those instructions. The benchmark encompasses four preference categories: normalization, entities, disfluencies, and case. Each is further subdivided to enable fine-grained assessment. The dataset construction pipeline is two-stage and LLM-assisted:
- Preference Classification: Samples drawn from seven publicly available ASR test corpora are first tagged, via LLM and human verification, for relevant preference categories (e.g., numbers that can be formatted variably, utterances with named entities, regions containing disfluencies, case ambiguities).
- Instruction and Reference Generation: The LLM then generates explicit instructions matching the detected preferences and produces corresponding reference transcripts. A human review finalizes each triple (audio, instruction, reference).
After deduplication, 3,210 unique instruction-annotated evaluation samples are obtained, well-distributed across normalization subtypes, entity categories, disfluency types, and casing styles. This resource enables systematic, controlled measurement of instruction-following.
Evaluation Methodology
Traditional WER-based evaluation applies blanket normalization rules to reference and predicted transcripts, masking differences in formatting that may be significant in user-critical applications. Preference-ASR introduces a preference-aware normalizer that disables the normalizer's relevant transformation steps according to the active instruction. This makes it possible to directly quantify compliance: for instance, “twenty-two” versus “22” is scored appropriately under alternate normalization instructions, and the presence or removal of fillers is assessed only under corresponding prompts.
The benchmark encompasses a diverse set of models:
- Parakeet-TDT-0.6B-v3: A standard non-LLM ASR model incapable of instruction-following.
- Canary-Qwen-2.5B: A hybrid with LLM backend but no explicit preference training.
- Phi-4-Multimodal: A Mixture-of-LoRAs, instruction-capable SpeechLLM.
- Qwen3-Omni-30B: A large instruction-aware, multimodal SpeechLLM with contextual biasing.
All models are evaluated both with and without explicit preference instructions, and under both standard and preference-aware normalization.
Key Experimental Findings
Several strong and nontrivial results are highlighted:
- Under standard normalization, models such as Canary-Qwen-2.5B and Qwen3-Omni-30B are virtually indistinguishable (both ~5.6% WER) but diverge markedly in instruction-following under preference-aware evaluation.
- Qwen3-Omni-30B, when given normalization/disfluency/case instructions, achieves preference-aware WER notably lower than baselines, confirming effective capability for controlled output.
- Qwen3-Omni-30B exhibits prompt-driven entity hallucination: entity error rate more than doubles under entity-biased instructions due to insertion of prompted but acoustically absent names—a behavior invisible without preference-aware metrics.
- Canary-Qwen-2.5B, despite an LLM backend, is almost completely insensitive to preference instructions in practice, confirming that LLM integration alone does not confer instruction-following without targeted training.
- Phi-4's extreme reduction in disfluency WER under explicit prompt (e.g., ∼50% to ∼10%) exposes default, non-instruction-aware aggressive filtering aggressively, while failing outright on casing.
- Non-LLM ASR baselines show strong performance under traditional normalization but high error under preference-aware metrics whenever default output does not align with instruction.
These observations reveal that standard references and blanket normalization not only under-scrutinize modern models' strengths (instruction-following, context adaptation) but also hide weaknesses (hallucination, overfitting to prompt context).
Theoretical and Practical Implications
Preference-ASR demonstrates the inadequacy of conventional ASR evaluation regimes in the context of instruction-responsive models. This has several important consequences:
- For Benchmarking: Model rankings can shift substantially depending on the evaluated instruction type and normalization, fundamentally impacting conclusions regarding model capability.
- For Deployment: In user-facing or workflow-dependent ASR systems (meeting transcription, database entry, entity-aware voice interfaces), controlled output style and entity grounding become essential, and errors in these facets are underreported by word-based metrics alone.
- For Model Development: The entity hallucination observed in Qwen3-Omni-30B indicates that SpeechLLMs must better balance prompt-based biasing with acoustic evidence, preventing prior-driven confabulation—a challenge with broader applicability in LLM-integrated systems.
- For Evaluation Metrics: The release of a preference-aware normalizer and corresponding evaluation pipeline constitutes a step towards rigorous, capability-focused assessment, informing future standards as the field shifts towards instruction-based ASR.
Limitations and Future Directions
The dataset is English-exclusive and limited to single-speaker segments. LLM-based reference generation, especially in normalization, still requires nontrivial human curation due to ambiguity inherent in spoken audio. Preference annotations could be extended to richer domains (e.g., speaker diarization preferences, multilingual settings). Manual verification remains a crucial component of reference curation.
Future directions include proceduralized, less labor-intensive reference generation, expansion beyond English, and inclusion of multi-preference and multi-speaker settings. As instruction-following models become more universally capable, the evaluation infrastructure must likewise adopt fine-grained, context-sensitive methodologies.
Conclusion
Preference-ASR (2606.29534) provides a comprehensive, instruction-annotated benchmark and preference-aware metric for assessing instruction-following in ASR. It exposes limitations in current models and standard metrics, demonstrating that both strengths and critical failures are often invisible in traditional WER evaluation. This work thus clarifies the evaluation frontier for SpeechLLMs and establishes a necessary baseline for assessing context-controlled ASR output, informing future model design and benchmark methodology.