PreferenceASR: Instruction-Aware ASR Benchmark

• PreferenceASR is a benchmark that pairs audio with explicit natural-language instructions and preference-conditioned references to assess transcription style compliance.
• It employs a selective, preference-aware normalization process to retain important stylistic distinctions often removed by standard ASR normalizers.
• The benchmark spans four categories—normalization, entities, disfluencies, and case—across diverse corpora, revealing strengths and weaknesses in model instruction-following.

Searching arXiv for papers on Preference-ASR and related instruction-following ASR benchmarks. PreferenceASR is a preference-aware automatic speech recognition benchmark designed to evaluate whether ASR systems follow natural-language instructions about output style rather than merely minimizing conventional word error under a fixed normalization regime. It was introduced to address a specific benchmarking gap: popular ASR test sets use inconsistent conventions for numbers, disfluencies, entities, and casing, while standard normalizers erase the format distinctions that matter to users. PreferenceASR therefore pairs audio with explicit instructions and preference-conditioned references, and evaluates systems with a preference-aware normalizer that selectively disables only those normalization steps that would conflict with the active instruction (Koluguri et al., 28 Jun 2026).

1. Problem setting and benchmark objective

PreferenceASR targets a failure mode in standard ASR evaluation: benchmark references and post-processing pipelines often collapse stylistic distinctions that are semantically or operationally important. In the formulation of the benchmark, current test sets cannot determine whether a model follows user preferences for output style because standard normalizers may remove the very differences under evaluation, such as digitized versus spoken forms of numbers, retention versus removal of fillers, or cased versus uncased transcription (Koluguri et al., 28 Jun 2026).

The benchmark frames ASR as an instruction-conditioned generation task across four categories: normalization, entities, disfluencies, and case. Each test item consists of audio, a natural-language preference instruction, and a reference transcription conditioned on that instruction. This design makes compliance with user-specified output conventions measurable rather than incidental. A plausible implication is that PreferenceASR shifts evaluation from pure lexical recovery toward a joint assessment of transcription fidelity and controllable formatting behavior.

The benchmark also distinguishes default transcription from instructed transcription. For each model, the reported evaluation includes a default condition without instruction and an instructed condition in which a preference instruction is appended. This makes it possible to isolate whether a model’s performance depends on explicit conditioning rather than on its latent formatting bias alone (Koluguri et al., 28 Jun 2026).

2. Corpus composition and construction pipeline

PreferenceASR is built from seven open-source corpora selected to span the four preference axes in realistic acoustic and discourse conditions. The source corpora are AMI, Common Voice, Earnings-22, GigaSpeech, LibriSpeech, SPGISpeech, and VoxPopuli. Their stated rationale is coverage of conversational meeting speech, crowd-sourced read speech, financial audio rich in numbers and symbols, multi-domain spontaneous dialogue, clean audiobook speech, professional financial audio with fully formatted transcripts, and named-entity–rich parliamentary speech (Koluguri et al., 28 Jun 2026).

The construction pipeline is explicitly two-stage and LLM-assisted, preceded by a baseline human-correction phase. In Stage 0, approximately 600 samples are pulled from each corpus, yielding 3,545 total, and human annotators correct errors in the original references. In Stage 1, termed preference classification, Qwen3-30B-A3B is used in “classify” mode to tag each transcript according to which of the four categories it exhibits, and humans spot-check roughly 10% of the outputs. In Stage 2, instruction and reference generation, Qwen-3 produces one or two natural-language instructions together with the corresponding preference-conditioned reference text, and human annotators review every resulting triple of audio, instruction, and generated reference, correcting mistakes in alignment or formatting (Koluguri et al., 28 Jun 2026).

The final dataset contains 3,210 unique (audio, instruction, reference) triples plus 335 baseline cases without instructions. After priority deduplication in the order normalization > entities > disfluencies > case, the category counts are 409 normalization samples, 680 entity samples, 539 disfluency samples, and 1,582 case samples. By source corpus, the benchmark contains 407 triples from AMI, 352 from Common Voice, 337 from Earnings-22, 519 from GigaSpeech, 594 from LibriSpeech, 416 from SPGISpeech, and 585 from VoxPopuli (Koluguri et al., 28 Jun 2026).

Dimension	Values
Final size	3,210 unique triples + 335 baseline cases
Categories	Normalization 409; Entities 680; Disfluencies 539; Case 1,582
Source corpora	AMI 407; Common Voice 352; Earnings-22 337; GigaSpeech 519; LibriSpeech 594; SPGISpeech 416; VoxPopuli 585

The benchmark inherits the diversity of the underlying corpora, ranging from conversational meeting rooms to audiobooks and noisy broadcast conditions. This matters because formatting preferences interact with genre: disfluencies are prevalent in conversational material, financial audio is dense in numbers and symbols, and parliamentary speech is rich in named entities (Koluguri et al., 28 Jun 2026).

3. Preference taxonomy and instruction design

PreferenceASR organizes evaluation into four top-level categories, each defined by explicit instruction patterns and, in some cases, directional variants.

Normalization covers numbers, symbols, and website links. The numbers subcategory contrasts inverse text normalization forms such as “22” with spoken-word forms such as “twenty-two.” The symbols subcategory contrasts symbolic renderings such as “\$100” or “20 %” with verbal forms such as “one hundred dollars” or “twenty percent.” The website-link subcategory contrasts literal URLs such as “www.google.com” with spoken sequences such as “w w w dot google dot com.” An example instruction in the spoken-form direction is: “Normalize all numbers and symbols into spoken form. E.g. ’12 %’ → ’twelve percent’. Keep fillers/punctuation as in source.” (Koluguri et al., 28 Jun 2026)

Entities includes company, product, person, location, organization, event, and drug names. Some instructions deliberately include false-positive entities that do not appear in the audio, specifically to measure hallucination under entity prompting. The example given is: “Transcribe the audio. Audio may contain: Craig Larson, KKR, (false positives: Michael Smith, Goldman Sachs).” (Koluguri et al., 28 Jun 2026)

Disfluencies includes filler words such as “um,” “uh,” and “ah,” repetitions such as “I I think,” false starts such as “I was go-- going,” and wrong-grammar correction versus literal retention. The benchmark defines two directional instructions: a keep direction, exemplified by “Keep all filler and repeated words verbatim in the transcription,” and a remove direction, exemplified by “Remove hesitation markers (uh, um, ah) and repeated words.” (Koluguri et al., 28 Jun 2026)

Case contrasts lowercase transcription without punctuation against transcription with proper capitalization and punctuation. The example lowercase instruction is “Output the transcription entirely in lowercase, no punctuation,” and the cased instruction is “Transcribe with correct sentence capitalization and punctuation.” (Koluguri et al., 28 Jun 2026)

These categories operationalize preference following as a controlled-output problem rather than as a latent property of a model’s decoding defaults. This suggests that PreferenceASR is as much a benchmark for instruction compliance as for recognition accuracy.

4. Preference-aware normalization and scoring

A central methodological contribution of PreferenceASR is the preference-aware normalizer. Its design premise is that standard ASR normalizers apply blanket steps such as lowercasing, number-to-word conversion, and symbol stripping, thereby erasing the distinctions the benchmark intends to evaluate. The benchmark instead applies a selective normalization policy: it skips only those steps that conflict with the active instruction, while still applying non-conflicting cleanup such as whitespace normalization (Koluguri et al., 28 Jun 2026).

The operational logic is given as pseudocode for a function $f_{pref}(h, P)$, where $h$ is the hypothesis and $P$ is the active preference. The full normalizer $N$ is described as having steps such as lowercase, TN, ITN, strip_symbols, and strip_fillers. If the preference requires lowercase, lowercase is applied; otherwise that step is skipped. If the preference requires TN, TN is applied; if it requires ITN, ITN is applied; otherwise number and symbol conversion is skipped. If the preference does not keep fillers, filler removal is applied; if it requires keeping fillers, filler removal is skipped. Punctuation and case are treated analogously, and non-conflicting cleanup is always applied (Koluguri et al., 28 Jun 2026).

The benchmark states the basic two-way normalization case as

$\hat{h} = f_{pref}(h, P) = \begin{cases} N(h)\,, & \text{if %%%%0%%%% requires standard normalization} \ h\,, & \text{otherwise.} \end{cases}$

and then generalizes this by stating that $f_{pref}$ composes only the subset of normalization sub-functions not disabled by $P$ (Koluguri et al., 28 Jun 2026).

Preference-aware word error rate is then computed after applying $f_{pref}$ to both hypothesis and reference. The metric is

$WER_{pref} \;=\; \frac{S + D + I}{N},$

where $S$, $h$0, and $h$1 are substitutions, deletions, and insertions, and $h$2 is the number of words in the normalized reference. The benchmark contrasts $h$3, computed under a standard normalizer, with $h$4, computed under the selective preference-aware normalizer (Koluguri et al., 28 Jun 2026).

A common misconception is that this normalization simply removes evaluation rigor by making scoring more permissive. The benchmark’s stated procedure does not relax scoring indiscriminately; it preserves all non-conflicting cleanup while disabling only the transformations that would collapse the instructed distinction. In that sense, the procedure changes the target of measurement rather than lowering the bar.

5. Benchmark models and empirical findings

PreferenceASR reports benchmarking results for four models: Parakeet-TDT-0.6B-v3, described as a non-LLM baseline with no instruction support; Canary-Qwen-2.5B, described as FastConformer plus a Qwen 2.5 LLM backbone with no preference tuning; Phi-4-Multimodal, described as a 5.6B model with native multimodal and instruction support; and Qwen3-Omni-30B, described as the strongest instruction-aware model in the reported comparison (Koluguri et al., 28 Jun 2026).

For normalization, the reported preference-aware WER values are 11.16% for Parakeet, 10.56% to 11.32% for Canary-Qwen from default to instructed, 10.76% to 11.10% for Phi-4, and 10.90% to 9.84% for Qwen3-Omni, with Qwen3-Omni improving under instruction. For entities, the values are 4.97% for Parakeet, 4.78% to 4.90% for Canary-Qwen, 5.26% to 5.28% for Phi-4, and 4.84% to 12.68% for Qwen3-Omni, where the sharp degradation is attributed to hallucinated false positives under instruction. For disfluencies, the values are 10.93% for Parakeet, 10.49% to 10.63% for Canary-Qwen, 49.88% to 10.79% for Phi-4, and 10.83% to 9.90% for Qwen3-Omni. For case, the values are 9.40% for Parakeet, 10.08% to 10.04% for Canary-Qwen, 5.88% to 27.23% for Phi-4, and 10.01% to 9.82% for Qwen3-Omni (Koluguri et al., 28 Jun 2026).

Category	Reported outcome
Normalization	Qwen3-Omni improves under instruction: 10.90% → 9.84%
Entities	Qwen3-Omni degrades sharply: 4.84% → 12.68%
Disfluencies	Phi-4 shifts from 49.88% to 10.79%
Case	Phi-4 shifts from 5.88% to 27.23%

The benchmark’s reported overall pattern is that standard WER rankings shift substantially once preference-following is evaluated directly. Specifically, the summary states that rankings under standard WER—Parakeet > Canary-Qwen ≈ Qwen3-Omni > Phi-4—change once models are rewarded or penalized for following formatting instructions correctly. Qwen3-Omni leads on normalization, disfluencies, and case, but performs poorly on entity prompting because of hallucination; Phi-4 shows large gains for instructed disfluency retention but a severe failure mode for case prompting (Koluguri et al., 28 Jun 2026).

These results underscore that strong instruction-following capacity is not uniform across preference types. A plausible implication is that ASR models with multimodal or LLM-based interfaces may require category-specific alignment rather than a single generic instruction-tuning recipe.

6. Interpretation, limitations, and research directions

PreferenceASR argues that traditional benchmarks can be misleading because blanket normalization obscures whether models preserve or transform user-salient formatting distinctions. The benchmark gives the example that a model which always converts “22” to “twenty-two” may be penalized on an ITN test set despite doing what the user requested, and likewise notes that when filler words are uniformly stripped in evaluation, one cannot determine whether the model actually understands “keep” versus “remove fillers” (Koluguri et al., 28 Jun 2026).

By conditioning references on explicit instructions and aligning evaluation with those instructions through selective normalization, PreferenceASR exposes behaviors that standard WER conceals. The paper specifically highlights hallucinated false-positive entities for Qwen3-Omni, default overzealous filler removal for Phi-4, and brittle casing support reflected in Phi-4’s case-prompt WER spike (Koluguri et al., 28 Jun 2026).

The reported limitations are precise. The benchmark is only in English. It supports only single-speaker preferences and does not yet include explicit multi-speaker labeling or speaker-selection instructions. Stage 2 requires heavy manual review for normalization references because acoustic context is needed. Proposed future extensions include multilingual preference evaluation, multi-speaker instructions such as “transcribe only speaker A,” extension to other modalities, and a version for real-time streaming compliance (Koluguri et al., 28 Jun 2026).

In summary, PreferenceASR defines a benchmark in which ASR quality is evaluated jointly with controllable output formatting. Its contribution lies in combining instruction-conditioned ground truth with a matching evaluation protocol that preserves the distinctions users specify. This suggests a broader methodological point: in speech systems that expose natural-language control surfaces, evaluation must be instruction-conditional if it is to measure the actual interface contract rather than only latent transcript accuracy (Koluguri et al., 28 Jun 2026).