DoWhatISay (DOWIS): Dataset for SLLM Evaluation
- DoWhatISay (DOWIS) is a multilingual dataset offering parallel human-written and recorded speech prompts, enabling evaluation of instruction-following in speech large language models.
- The dataset covers 9 tasks across 11 languages with 5 distinct prompt styles, allowing comprehensive comparisons between text and speech modalities and diverse interaction conditions.
- Empirical results show that text prompts generally outperform spoken prompts for text outputs, while spoken prompts are competitive for speech outputs, revealing modality-specific performance gaps.
Searching arXiv for the specified paper and closely related benchmarks to ground the article. DoWhatISay (DOWIS) is a multilingual dataset of parallel spoken and written prompts for evaluating the instruction-following behavior of Speech LLMs (SLLMs) under spoken interaction conditions. It was introduced to address a persistent evaluation mismatch: SLLMs are commonly benchmarked with text instructions even when the intended deployment setting is speech-based human–computer interaction. DOWIS provides human-authored text prompts and human-recorded spoken renditions that are decoupled from downstream task inputs, allowing the same prompt inventory to be paired with existing benchmarks across multiple speech and text tasks. The dataset spans 9 tasks, 11 languages, 5 prompt styles, and 10 prompt variants per task–language pair, and is designed to expose the interaction between prompt modality, prompt style, language, and task type in realistic instruction-following evaluation (Züfle et al., 10 Mar 2026).
1. Motivation and problem formulation
DOWIS is situated in the evaluation of SLLMs that accept speech, text, or both as input and produce speech and/or text as output. In this setting, instruction-following is the capability by which a model infers the requested task and desired output behavior from a prompt such as “Please summarise this talk” or “Translate what the other person is saying.” The DOWIS paper identifies a central gap between this use case and prevailing evaluation practice: speech instruction-following is usually assessed with text prompts rather than spoken ones, even when the task content itself is audio (Züfle et al., 10 Mar 2026).
The paper isolates two shortcomings in existing practice. The first is prompt modality mismatch. A model that performs well when prompted in text may still degrade substantially when the same instruction is delivered as speech, particularly because spoken instructions introduce prosody, disfluencies, accent variation, recording variability, and possible front-end recognition or encoding errors. The second is limited stylistic diversity. Benchmarks often rely on one canonical instruction per task, whereas real users employ heterogeneous formulations ranging from formal to colloquial and from concise to detailed. DOWIS was constructed to make both factors measurable within a controlled framework (Züfle et al., 10 Mar 2026).
The paper also positions DOWIS against prior spoken instruction benchmarks such as SpeechInstructBench and URO-Bench. The limitations it identifies in that prior landscape include synthetic rather than human-recorded prompts, restricted language coverage, pre-concatenation of instructions with task input that reduces reuse, emphasis on reasoning or QA rather than a broader set of speech-centric tasks, and limited support for cross-lingual evaluation. This suggests that DOWIS is intended less as a replacement for downstream benchmarks than as a reusable prompt layer for retrofitting them with spoken instruction conditions.
2. Dataset architecture and scope
DOWIS is explicitly task-agnostic: prompts are separated from task input data so that they can be paired with any existing benchmark dataset. Its basic unit is a parallel prompt pair consisting of a written instruction and a human-recorded spoken version of that same instruction. The dataset contains 990 unique text prompts, obtained from the product
with corresponding speech recordings by human speakers (Züfle et al., 10 Mar 2026).
The speech component was recorded by 19 speakers in total, with 9 male and 10 female speakers. Total audio duration is 3 hours 17 minutes, average recording time per speaker is approximately 8 minutes 35 seconds, and average prompt duration is typically 4–5 seconds, except for Audio Chapter Generation, whose prompts average approximately 16 seconds because of their more elaborate formatting and segmentation instructions (Züfle et al., 10 Mar 2026).
DOWIS is released under CC-BY and distributed via Hugging Face as maikezu/dowis and via GitHub at https://github.com/MaikeZuefle/DOWIS. The dataset is intended primarily for evaluation rather than model training, and it does not define train, validation, or test splits. Researchers instead sample prompt variants according to their own experimental design (Züfle et al., 10 Mar 2026).
3. Tasks, languages, and prompt styles
The dataset was designed to span the major input–output patterns relevant to SLLMs, including speech-to-text, text-to-speech, speech-to-speech, text-to-text, and mixed settings. The nine tasks are Automatic Speech Recognition (ASR), Text-to-Speech (TTS), Speech Translation (ST), Machine Translation (MT), Speech-to-Speech Translation (S2ST), Speech Summarisation (SSUM), Text Summarisation (TSUM), Audio Chapter Generation (ACHAP), and Spoken Question Answering (SQA) (Züfle et al., 10 Mar 2026).
| Component | Coverage | Notes |
|---|---|---|
| Tasks | 9 | ASR, TTS, ST, MT, S2ST, SSUM, TSUM, ACHAP, SQA |
| Languages | 11 | en, de, it, es, fr, pt, nl, sv, cs, hu, ru |
| Styles | 5 | basic, formal, informal, detailed, short |
| Variants | 10 per task–language pair | 2 prompts per style |
The language set consists of English, German, Italian, Spanish, French, Portuguese, Dutch, Swedish, Czech, Hungarian, and Russian. The paper characterizes this inventory as combining high-resource European languages with mid- or low-resource European languages that are less commonly benchmarked. All prompts originate in English and are translated and adapted by native speakers to sound natural in the target language. For cross-lingual tasks such as MT, ST, and S2ST, the prompt is always in the target language, reflecting the assumption that the user issues the instruction in their own language (Züfle et al., 10 Mar 2026).
Prompt variation is organized into five styles: basic, formal, informal, detailed, and short. Each task–language pair includes two distinct prompts per style, yielding ten variants. The styles are intended to capture natural phrasing, polished professional register, conversational colloquiality, more explicit task descriptions, and maximally concise but unambiguous instructions, respectively. This design supports systematic ablations over wording while holding task identity fixed (Züfle et al., 10 Mar 2026).
The acoustic duration statistics underscore that most prompts are short single-turn instructions. Average spoken prompt duration by task is 4.3 seconds for ASR, 5.0 for SQA, 15.8 for ACHAP, 4.4 for TTS, 4.1 for MT, 4.5 for ST, 4.6 for TSUM, 4.6 for SSUM, and 4.8 for S2ST. ACHAP is the major exception because its prompts must specify structured outputs such as chapter boundaries and titles (Züfle et al., 10 Mar 2026).
4. Data collection, recording conditions, and processing
Prompt authoring began with seed prompts written by task specialists familiar with the respective tasks. For each task, they produced two basic English prompts they would actually use, and then two rephrasings for each of the other styles: formal, informal, detailed, and short. These English prompts were translated into the target languages by native speakers, who were explicitly instructed to prioritize naturalness rather than literal translation (Züfle et al., 10 Mar 2026).
Speech recording was carried out by native or highly proficient speakers using personal phones or laptops in realistic conditions intended to simulate a meeting or everyday environment. Speakers were instructed to read the prompts as if giving them to an AI model. Some speakers were bilingual, and four recorded in two languages. Per-language recording times reported in the paper are approximately 33 minutes for German, 31 for English, 35 for Italian, 17 for Czech, 17 for Spanish, 18 for French, 11 for Hungarian, 10 for Dutch, 8 for Portuguese, 9 for Russian, and 8 for Swedish (Züfle et al., 10 Mar 2026).
All recordings were converted to WAV and trimmed to remove leading and trailing silence. Voice activity detection used a 10 ms sliding window, with regions marked as non-silent if loudness exceeded dBFS. The retained segment was cropped between the first and last non-silent regions, with 500 ms padding preserved at both ends to avoid abrupt onset or offset artifacts (Züfle et al., 10 Mar 2026).
Prompt intelligibility was assessed automatically by transcribing all prompt audios with Whisper-large-v3 and computing Word Error Rate against the reference prompt text. The overall average WER across all prompts was 12.72%. Language-wise examples given in the paper include 16% for Czech, 26% for Dutch, 18% for Portuguese, and 13% for Swedish. The paper interprets these values as evidence that audio quality is generally sufficient and that major downstream performance drops cannot be reduced to unintelligible prompt recordings alone (Züfle et al., 10 Mar 2026).
5. Integration with external benchmarks and evaluation protocol
Because DOWIS is prompt-only, it is paired with external datasets rather than used as a standalone benchmark. The paper demonstrates this with FLEURS for ASR, MT, ST, S2ST, and TTS; MCIF for TSUM, SSUM, and SQA; and YTSeg for ACHAP. The general pattern is to select a benchmark instance with input content and reference output , prepend or otherwise provide an appropriate DOWIS prompt in text or speech, run the SLLM to produce , and evaluate with task-appropriate metrics (Züfle et al., 10 Mar 2026).
Two SLLMs are evaluated: Qwen2.5-Omni-7B and Phi-4-multimodal-instruct. Qwen2.5-Omni-7B supports both speech input and speech output and is used across all tasks, including TTS and S2ST. Phi-4-multimodal-instruct supports speech input but not speech output and is therefore evaluated only on tasks with text outputs: ASR, MT, ST, TSUM, SSUM, SQA, and ACHAP. Inference uses default provider parameters, batch size 1, and a single NVIDIA A100-SXM4-40GB GPU (Züfle et al., 10 Mar 2026).
The evaluation metrics are matched to task output type. WER, computed with jiwer, is used for ASR and for TTS content accuracy after automatic transcription:
where denotes substitutions, deletions, insertions, and 0 the number of reference words. CometKiwi is used for MT, ST, and S2ST content quality. Normalized BERTScore with deberta-xlarge-mnli is used for TSUM, SSUM, SQA, and ACHAP titles, with GC-BERTScore additionally used for concatenated ACHAP chapter titles. UTMOS measures perceived speech quality for TTS and S2ST. ACHAP segmentation is measured with Collar-F1 using a 1 second tolerance and the chunkseg library under the protocol of Retkowski et al. (2026) (Züfle et al., 10 Mar 2026).
The core experimental comparisons are text versus speech prompting for the same model and task, style-wise comparisons across the five prompt styles, male versus female spoken prompts where both are available, and multilingual evaluation across all supported languages. No external model-family baselines are introduced; the emphasis is on condition sensitivity rather than leaderboard ranking (Züfle et al., 10 Mar 2026).
6. Empirical findings, interpretation, and limitations
The central empirical result is that text prompts consistently outperform spoken prompts for tasks with text outputs. For ASR, Qwen records a WER of 31.21 with text prompts versus 35.96 with speech prompts, while Phi records 35.93 with text prompts versus 347.43 with speech prompts. For MT, Qwen records Comet 81.41 with text prompts versus 70.97 with speech prompts, and Phi 77.23 versus 46.99. For ST, Qwen records 80.21 versus 68.57, and Phi 75.82 versus 57.79. TSUM, SSUM, SQA, and ACHAP show the same directional effect, although the magnitude varies by task and model. The paper therefore argues that text-only prompting gives an overly optimistic estimate of instruction-following performance for these settings (Züfle et al., 10 Mar 2026).
By contrast, the gap narrows sharply for tasks with speech outputs. For Qwen TTS, UTMOS is 4.33 with text prompts and 4.34 with speech prompts, while WER derived from ASR transcription is 36.09 versus 34.52, slightly favoring spoken prompts. For S2ST, UTMOS is approximately 4.35 under both prompt modalities, and CometKiwi via transcription is 72.10 for text prompts and 72.08 on average for speech prompts. The paper also reports that, when S2ST spoken prompts are broken down by gender, male and female prompts both outperform the text average in CometKiwi, with 75.99 and 76.18 respectively. This indicates that spoken prompting becomes competitive, and can marginally outperform text prompting, when the output modality is itself speech (Züfle et al., 10 Mar 2026).
Language-wise analyses reveal larger text–speech gaps for some lower-resource or less commonly benchmarked languages, especially Czech, Dutch, Portuguese, and Swedish in ASR, MT, and ST. The paper notes that in some cases the model already performs poorly under text prompting, but in others the model performs adequately with text prompts and then degrades substantially under spoken prompting despite only modest prompt WER under Whisper. The authors interpret this as evidence that the bottleneck lies in understanding spoken instructions rather than in basic acoustic intelligibility alone (Züfle et al., 10 Mar 2026).
Prompt style also matters. Informal prompts are reported as consistently worst across tasks and models, whereas formal and detailed prompts generally perform best. Short prompts are often more difficult than basic, formal, or detailed prompts, although typically less harmful than informal prompts. The paper interprets this pattern as evidence that current models benefit from structured and explicit instructions, while colloquial phrasings are harder to parse robustly (Züfle et al., 10 Mar 2026).
The paper further reports small but consistent gender effects in languages for which both male and female speakers recorded the prompts. Qwen performs better with male spoken prompts on TSUM and SSUM, but better with female spoken prompts on TTS, MT, ST, and S2ST. Because Whisper WER is similar across genders for the relevant prompts, the paper suggests that these differences may reflect speaker-related biases in the models rather than simple intelligibility differences (Züfle et al., 10 Mar 2026).
The authors’ interpretation centers on three factors: training data imbalance in favor of text instructions, a modality transfer gap introduced by audio front-ends or latent speech encoders, and greater vulnerability of short or informal spoken prompts to misinterpretation. In that reading, current SLLMs remain text-centric in instruction-following behavior. The paper also identifies clear limitations in DOWIS itself: all languages are European, speaker diversity is limited, recordings are realistic but relatively clean, only 9 tasks are covered, the total audio scale is modest at 3h17m, and the prompts are generally short single-turn instructions rather than complex multi-turn interactions. Future extensions suggested or implied in the paper include broader language and speaker coverage, additional tasks such as dialogue and paralinguistic reasoning, noisier and more disfluent prompt types, richer error taxonomies, and fine-tuning strategies that exploit spoken prompts to narrow the speech–text gap (Züfle et al., 10 Mar 2026).