Quantum Uncloneable Encryption Overview

• Quantum Uncloneable Encryption is a method that uses the no‐cloning theorem to prevent the unauthorized reproduction of quantum information.
• It leverages principles such as superposition and entanglement to enhance data security against both classical and quantum attacks.
• Practical applications include secure communication in quantum networks and safeguarding sensitive digital assets from quantum-enabled threats.

1. Motivation and Problem Setting

Speech LLMs (SLLMs)

The paper studies Speech LLMs (SLLMs), i.e., LLMs that can process and/or generate speech in addition to text. Examples include Qwen2.5-Omni and Phi‑4 Multimodal. These models support:

• speech‑in / text‑out (e.g., ASR, spoken QA),
• text‑in / speech‑out (e.g., TTS),
• speech‑in / speech‑out (e.g., speech‑to‑speech translation),
• as well as standard text‑in / text‑out tasks.

A key capability is instruction-following (IF): instead of issuing API flags like “mode=ASR”, a user can say a natural-language instruction such as “Please transcribe this audio” or “Translate what the speaker says into German.” The model must understand the instruction, map it to the correct task, and execute it.

Why text-based evaluation is insufficient

Most existing evaluation of SLLMs uses text prompts even when the underlying tasks involve speech. Benchmarks such as AIR‑Bench, MMSU, SIFT‑50M/EvalSIFT, AudioBench, Speech‑IFEval and others generally feed text instructions while the model processes speech inputs. This is problematic for two main reasons:

1. Prompt modality mismatch. Real users will often speak their instructions (“Summarise this meeting”, “Translate what the other person said”). Performance under text prompts can overestimate how well models will work in such realistic speech‑based interaction.
2. Lack of prompt diversity. Many benchmarks rely on one or a handful of instruction templates, often LLM‑generated. But robust instruction‑following requires handling a variety of styles—formal vs informal, verbose vs concise, high-level vs very explicit. Without testing this variability, we do not know how brittle models are to the way people naturally phrase instructions.

Moreover, the few benchmarks with spoken prompts (e.g., SpeechInstructBench, URO‑Bench, some QA setups) have limitations:

• Instructions are synthetic (text‑to‑speech) rather than human‑recorded;
• They focus on English and Chinese only;
• Instructions are pre‑concatenated with task inputs, which makes them hard to reuse with arbitrary new downstream datasets;
• They mostly target general instruction-following or QA, not the broad spectrum of speech tasks (e.g., speech translation, audio chaptering, summarisation);
• They usually evaluate monolingual scenarios, not cross‑lingual tasks like speech translation.

Instruction-following with spoken prompts

In this context, instruction-following means: given a (spoken or written) instruction and an input (speech or text), the model must infer:

• which task to perform (e.g., ASR vs ST vs summarisation),
• the relevant arguments (languages, styles, degree of detail, output format),
• and then produce the correct output in the correct modality.

Spoken prompts are crucial to:

• test whether the speech pathway can convey task intent as reliably as text;
• identify gaps between speech vs text instruction performance;
• evaluate robustness to prosody, disfluencies, and acoustic variability.

This is the gap that DoWhatISay (DOWIS) is designed to fill.

---

2. Overall Description of DOWIS

What is DOWIS?

DoWhatISay (DOWIS) is a multilingual prompt dataset consisting of:

• parallel spoken and written instructions,
• for 9 different tasks in speech and language processing,
• across 11 languages,
• with 10 instruction variants per task–language pair,
• covering 5 distinct prompt styles.

Crucially, instructions are kept separate from task inputs. The dataset provides standalone spoken/written prompts that can be paired with any existing benchmark (e.g., FLEURS, MCIF, YTSeg, etc.). This design allows researchers to evaluate instruction-following in SLLMs under realistic spoken prompting while using any corpus they care about for the underlying tasks.

Goals and intended use

DOWIS is designed to:

• enable realistic evaluation of SLLMs under spoken instructions;
• capture prompt modality (text vs speech),
• prompt style (formal, informal, detailed, short, basic),
• language (11 typologically and geographically varied languages),
• and task type (speech‑to‑text, text‑to‑speech, speech‑to‑speech, and text‑to‑text).

The main intended use cases:

• Pair DOWIS prompts with downstream datasets to evaluate how well SLLMs follow spoken vs written instructions, holding task and underlying data constant.
• Compare performance across languages, including higher‑resource vs lower‑resource settings.
• Study how prompt style and spoken realization affect instruction-following.

Scale and structure

• Tasks: 9 (listed in detail below).
• Languages: 11.
• Prompt variants: 10 per task–language pair.
• Total unique text prompts:

$$10 \text{ prompts} \times 9 \text{ tasks} \times 11 \text{ languages} = 990$$

• Speakers: 19 (9 male, 10 female), with 4 bilingual individuals who recorded in two languages.
• Audio duration: total of 3 hours 17 minutes of spoken prompts.
• Average audio per speaker: ~8m 35s.
• Average duration per prompt by task (\cref{tab:task_durations} in the paper):
  • ASR: 4.3 s
  • SQA: 5.0 s
  • ACHAP (audio chaptering): 15.8 s (longest, due to complex formatting instructions)
  • TTS: 4.4 s
  • MT: 4.1 s
  • ST: 4.5 s
  • TSUM: 4.6 s
  • SSUM: 4.6 s
  • S2ST: 4.8 s

Monolingual vs cross‑lingual designation: some tasks (TSUM, SSUM) can be monolingual or cross‑lingual depending on which dataset they are paired with.

---

3. Tasks, Languages, and Styles

3.1 Tasks

DOWIS covers 9 tasks, spanning all SLLM input/output modality combinations:

1. Automatic Speech Recognition (ASR)
   • Input: speech; Output: text.
   • Instruction-following aspect: “transcribe this audio” type prompts, sometimes specifying normalization or punctuation.
2. Spoken Question Answering (SQA)
   • Input: speech (content) + spoken or written question; Output: text answer.
   • Requires comprehension and reasoning over speech, plus instruction-following.
3. Audio Chapter Generation (ACHAP)
   • Input: long speech (e.g., talks, videos); Output: a sequence of chapters with timestamps and titles.
   • Prompts describe how to segment the audio and format titles; these are longer (≈15.8 s) and more structured.
4. Text-to-Speech (TTS)
   • Input: text plus instruction; Output: speech.
   • Prompts specify that the model should “read” or “speak” the text, sometimes style / voice constraints.
5. Machine Translation (MT)
   • Input: text; Output: text (another language).
   • Instructions specify source and target languages and what to do (“Translate the following sentence into …”).
6. Speech Translation (ST)
   • Input: speech; Output: text (another language).
   • Instructions like “Translate the following audio into English” in the target language.
7. Text Summarization (TSUM)
   • Input: text; Output: text summary.
   • Prompts vary from generic (“Summarise this talk”) to detailed constraints (length, focus, etc.).
8. Speech Summarization (SSUM)
   • Input: speech; Output: text summary.
   • Instructions specify summarisation of audio content.
9. Speech-to-Speech Translation (S2ST)
   • Input: speech in source language; Output: speech in target language.
   • Instructions define the translation direction and sometimes style or fidelity.

Together, these tasks cover:

• speech‑to‑text: ASR, SQA, ST, SSUM, ACHAP,
• text‑to‑text: MT, TSUM,
• text‑to‑speech: TTS,
• speech‑to‑speech: S2ST.

This gives a broad testbed for how well SLLMs handle instruction-following across modalities.

3.2 Languages

DOWIS includes 11 languages:

• English (en)
• German (de)
• Italian (it)
• Spanish (es)
• French (fr)
• Portuguese (pt)
• Dutch (nl)
• Swedish (sv)
• Czech (cs)
• Hungarian (hu)
• Russian (ru)

Prompts were first authored in English, then translated and adapted by native speakers into the other 10 languages, aiming for naturalness rather than literal translation. The set balances:

• High‑resource European languages (en, de, es, fr, it, pt),
• Medium/less-resourced languages in common benchmarks (cs, hu, nl, sv, ru),
• Geographic and typological diversity within Europe.

This allows studying how instruction-following varies between high‑ and lower‑resource languages.

3.3 Prompt styles and variants

Each task–language pair has 10 prompt variants, structured as 5 styles with 2 variants each:

1. Basic
   • Natural, everyday phrasing a task expert would genuinely use in their research.
   • E.g., “Please transcribe the audio” or “Summarise the talk”.
2. Formal
   • Professional, polished language; more official tone.
   • E.g., “Please provide an accurate verbatim transcription of the following speech segment.”
3. Informal
   • Casual, conversational style, often with colloquial phrases.
   • E.g., “Hey, can you write out what’s being said in this audio?”
4. Detailed
   • Explicit, precise instructions about how to perform the task: formatting, length constraints, etc.
   • E.g., for ACHAP: instructions on how to produce chapters, timestamps and titles.
5. Short
   • As concise as possible while unambiguous.
   • E.g., “Transcribe this audio.”

“10 prompt variants per task–language pair” concretely means:

• For each of the 9 tasks,
• for each of the 11 languages,
• there are 2 versions of basic, 2 formal, 2 informal, 2 detailed, and 2 short prompts.

These 10 distinct instructions can be spoken or used in written form, enabling multi‑style, multi‑modality evaluation.

---

4. Data Collection and Annotation

4.1 Written prompt creation and translation

1. Seed prompts from experts. For each task, researchers specializing in that task wrote:

• 2 basic English prompts that they themselves would use in research,
• 2 rephrasings for each of the remaining styles (formal, informal, detailed, short).

This yields 10 English prompts per task.

2. Translation and adaptation. Native speakers translated and adapted these prompts into the other 10 languages. Instructions to translators:

• preserve the intended style,
• make the prompts sound natural in the target language (not literal),
• preserve task semantics.

Total: 9 tasks × 11 languages × 10 variants = 990 text prompts.

4.2 Recording the spoken prompts

• Speakers: 19 people across the 11 languages (9 male, 10 female), including 4 bilinguals recording in two languages.
• Per-language counts (from \cref{tab:annotation_overview}):
  • Example rows:
  • German (de): 2 male, 2 female, total 33 minutes
  • English (en): 2 male, 2 female, total 31 minutes
  • Italian (it): 2 male, 2 female, total 35 minutes
  • Others have 1–2 speakers of each gender with total 8–18 minutes per language, and Hungarian, Dutch, Portuguese, Russian, Swedish have only one gender represented.
• Recording conditions:
  • Speakers were asked to read out the 90 prompts of their language (9 tasks × 10 prompts), speaking as if giving instructions to an AI model.
  • They used their own phones or laptops, without studio equipment, to simulate realistic meeting or personal assistant scenarios.
  • This introduces realistic variation in microphones, environments, and background noise.

4.3 Audio processing and quality control

All recordings were:

• converted to .wav format;
• trimmed using a loudness-based voice activity detection:
  • a sliding window of 10 ms;
  • frames marked non-silent if loudness > $-40\,\text{dBFS}$;
  • audio cropped between first and last non-silent frames, with 500 ms padding at start and end to preserve natural onset/offset.

Prompt intelligibility was later checked by transcribing all prompts with whisper‑large‑v3 and computing Word Error Rate (WER) against the reference text prompts. The overall WER across all languages is 12.72%, with per-language WERs of 16% (cs), 26% (nl), 18% (pt), 13% (sv), showing that:

• audio is generally clear,
• acoustic noise is not the main driver of model failures with spoken instructions.

No additional complex filtering pipelines are described; the corpus is small and carefully curated, so manual design and VAD trimming were sufficient.

4.4 Length and acoustic statistics

From the paper’s statistics:

• Average duration per prompt (by task) is 4–5 seconds for most tasks, except ACHAP (≈15.8 s).
• Total prompt audio: 3h17m over all languages and tasks.
• Average per speaker: 8m35s.
• Distribution of male/female speakers and durations per language is in Table 1 (in the paper).

Exact distributions in tokens or speech rate are not tabulated, but the durations give a good sense of prompt length and complexity.

---

5. Dataset Format and Accessibility

5.1 Data format

The paper does not give a full schema, but from description:

• Audio:
  • Format: .wav
  • Sampling: standard speech-quality (exact rate not specified, but typical 16 kHz or higher).
  • One file per spoken prompt variant and speaker.
• Text:
  • Written prompts: plain text files or JSON fields, containing the instruction in each language.
  • Transcriptions: parallel text corresponding to each spoken prompt.
• Metadata (implied/typical for such datasets, and necessary for the experiments):
  • Language code (e.g., en, de…),
  • Task (ASR, MT, ST, etc.),
  • Prompt style (basic, formal, informal, detailed, short),
  • Prompt variant index (1 or 2 per style),
  • Speaker ID and gender (used for the male/female analysis),
  • Path to audio file,
  • Prompt text.

5.2 Linking to external benchmarks

DOWIS does not include task inputs (e.g., the audio to transcribe or the text to translate). Instead, it is designed to be paired with existing benchmarks, which provide the inputs and references.

In experiments, the authors pair DOWIS with:

• FLEURS \cite{conneau2022fleursfewshotlearningevaluation} for:
  • ASR (11 languages),
  • MT and ST (en→{de, it, es, fr, pt, nl, ru, sv, cs, hu}),
  • S2ST (source = those 10 languages, target = English),
  • TTS (text in English).
• MCIF \cite{papi2026mcif} for:
  • TSUM and SSUM (en→{en, de, it}),
  • SQA (en→en).
• YTSeg \cite{retkowski2024ytseg} for:
  • ACHAP (en).

The evaluation pipeline is:

• Select a dataset example (input + reference).
• Select a DOWIS prompt for the corresponding task and target language.
• Feed them jointly to the SLLM (instruction + input).
• Compare model outputs against dataset references using task-specific metrics.

Because prompts are kept separate, DOWIS can extend to any new corpus without modification.

5.3 Splits, licensing, and ethics

• The paper does not define train/dev/test splits; DOWIS is intended as an evaluation resource, not a training set.
• License: CC‑BY (Creative Commons Attribution). From the paper:

  “DOWIS is available under CC-BY license at maikezu/dowis and https://github.com/MaikeZuefle/DOWIS.”

• Ethical aspects:

  • Prompts are read by volunteers (native or highly proficient speakers) who consented to recording.
  • Data is limited to instructions, without personal content or sensitive topics.
  • The paper highlights gender-related performance differences (possible model bias) and encourages evaluating both male and female speakers as part of responsible development.
  • Potential misuse would be similar to other speech corpora (e.g., voice spoofing); the dataset is intended for research on SLLMs and instruction-following.

---

6. Benchmarking and Evaluation Protocol

6.1 Models evaluated

Two SLLMs are benchmarked:

• Qwen2.5‑Omni‑7B (Qwen/Qwen2.5-Omni-7B): supports speech input and speech output, used on all tasks.
• Phi‑4‑multimodal‑instruct (microsoft/Phi-4-multimodal-instruct): supports speech input but no audio generation, so evaluated only on tasks with text output.

Both are run with default inference parameters, batch size 1, on a single NVIDIA A100 40GB GPU.

6.2 Prompt modalities and setup

For each task, language, and prompt style, the authors evaluate:

• Text prompts: written prompt in the target language.
• Speech prompts: corresponding spoken version (chosen randomly among speakers, or conditioned on gender in gender-specific analyses).

For all tasks, including cross-lingual ones, the prompt is written/spoken in the target language (the language of the expected output).

Input/output modalities:

• Text‑output tasks:
  • ASR: (speech input + prompt) → text transcription.
  • MT: (text input + prompt) → text translation.
  • ST: (speech input + prompt) → text translation.
  • SQA: speech + (spoken/text) question → text answer.
  • TSUM: text + prompt → text summary.
  • SSUM: speech + prompt → text summary.
  • ACHAP: speech + prompt → predicted chapter boundaries and titles (text).
• Speech‑output tasks:
  • TTS: (text input + prompt) → output speech.
  • S2ST: (speech input + prompt) → output speech translation.

For speech‑output tasks, Qwen is the only model used.

Two types of prompting pipelines are implicitly compared:

1. Direct speech-in prompting: The SLLM receives speech as an input channel (for content and/or instructions).
2. Text prompts vs speech prompts: For each task, compare text instruction vs spoken instruction, while keeping the task data identical.

6.3 Metrics

Standard metrics:

• ASR: Word Error Rate (WER) computed with jiwer. WER is the standard:

$\mathrm{WER} = \frac{S + D + I}{N}$

where $S$ = substitutions, $D$ = deletions, $I$ = insertions, $N$ = total words in reference.

• MT / ST: CometKiwi \cite{rei-etal-2022-cometkiwi}, a reference-free quality estimation metric. Higher scores are better.
• TSUM / SSUM / SQA: normalized BERTScore \cite{bert_score} with microsoft/deberta-xlarge-mnli:

$$\text{BERTScore}(c, r) = F_1\big(\text{sim}(\text{emb}(c), \text{emb}(r))\big),$$

where $c$ is candidate output, $r$ reference, and similarity is computed in contextual embedding space. Values are normalized to an interpretable scale (numbers in the tables look like percentage × 100).

• ACHAP:
  • Collar‑F1 ($-40\,\text{dBFS}$0 s): F1 score for boundary detection with a 3‑second temporal collar.
  • BERTScore$-40\,\text{dBFS}$1 (Global Concatenation): BERTScore between concatenated predicted titles and concatenated reference titles.
• TTS, S2ST:
  • Content accuracy: model output is transcribed with whisper-large-v3 and evaluated by:
  • WER for TTS,
  • CometKiwi (via ASR transcripts) for S2ST.
  • Speech quality: UTMOS \cite{saeki22c_interspeech} (predicted MOS-like score; higher is better).

The paper presents tables with these metrics, e.g., Table \ref{tab:modality} showing averaged scores for text vs speech prompts and male vs female speakers.

6.4 Baselines and comparison settings

The central comparisons are:

• Text prompts vs speech prompts, averaged over all prompt types and languages.
• Male vs female speaker prompts, controlling for languages with both genders represented.
• Prompt style vs performance, averaged over modality and languages.
• Heatmaps showing language-wise differences between text and speech prompts (for ASR, MT, ST) and prompt-type-wise differences (Figure \ref{fig:ext_vs_speech_qwen-heatmaps} and Figure \ref{fig:prompt_types_heatmap}).

There is no separate non‑SLLM baseline; the baseline is essentially “same SLLM under text prompts”.

---

7. Key Experimental Results and Findings

7.1 Overall text vs speech prompt performance

Table \ref{tab:modality} summarises average model performance (over languages and prompt types) under text vs speech prompts.

Key pattern: For tasks with text output, text prompts substantially outperform speech prompts, especially for Phi:

• ASR:
  • Phi (subset of languages):
  • Text prompt WER: $-40\,\text{dBFS}$2
  • Speech prompt WER: $-40\,\text{dBFS}$3 (≫100, essentially failure)
  • Qwen:
  • Text: $-40\,\text{dBFS}$4
  • Speech: $-40\,\text{dBFS}$5 (gap but not catastrophic).
• MT (Comet; higher is better):
  • Phi: text $-40\,\text{dBFS}$6 vs speech $-40\,\text{dBFS}$7 (large drop).
  • Qwen: text $-40\,\text{dBFS}$8 vs speech $-40\,\text{dBFS}$9.
• ST:
  • Phi: text $\mathrm{WER} = \frac{S + D + I}{N}$0 vs speech $\mathrm{WER} = \frac{S + D + I}{N}$1.
  • Qwen: text $\mathrm{WER} = \frac{S + D + I}{N}$2 vs speech $\mathrm{WER} = \frac{S + D + I}{N}$3.
• TSUM:
  • Phi: text $\mathrm{WER} = \frac{S + D + I}{N}$4 vs speech $\mathrm{WER} = \frac{S + D + I}{N}$5.
  • Qwen: text $\mathrm{WER} = \frac{S + D + I}{N}$6 vs speech $\mathrm{WER} = \frac{S + D + I}{N}$7 (small gap).
• SSUM:
  • Phi: text $\mathrm{WER} = \frac{S + D + I}{N}$8 vs speech $\mathrm{WER} = \frac{S + D + I}{N}$9.
  • Qwen: text $S$0 vs speech $S$1.
• SQA:
  • Phi: text $S$2 vs speech $S$3 (very large gap).
  • Qwen: text $S$4 vs speech $S$5 (almost no difference).
• ACHAP:
  • Collar‑F1 for Qwen: text $S$6 vs speech $S$7.
  • GC‑BERTScore for Qwen: text $S$8 vs speech $S$9.

Tasks with speech output: gap is much smaller.

• TTS (Qwen):
  • UTMOS: text $D$0 vs speech $D$1 (virtually identical).
  • WER (of ASR on generated speech): text $D$2 vs speech $D$3 (slight advantage for speech prompts).
• S2ST (Qwen):
  • UTMOS: identical (~4.35) for both text and speech prompts.
  • CometKiwi via ASR transcripts: text $D$4 vs speech $D$5 (negligible difference overall); female prompts sometimes slightly better.

Summarizing:

• Text prompts “overestimate” SLLM capabilities for text-output tasks compared to more realistic spoken instructions.
• For speech-output tasks, models are almost equally good under text and speech prompts; in some settings speech even slightly edges out text.

7.2 Language-wise differences: high vs low resource

Figure \ref{fig:ext_vs_speech_qwen-heatmaps} shows, for Qwen:

• per-language differences between text and speech prompts for ASR, MT, ST.

Observations:

• Languages like Czech (cs), Dutch (nl), Portuguese (pt), Swedish (sv) show a strong preference for text prompts in ASR (large positive differences).
• In MT and ST, cs, nl, sv similarly show strong text-over-speech gaps.

For some languages (cs, sv), Qwen’s ASR performance is already poor with text prompts (WER > 100), so modality makes little difference there. But for nl and pt:

• ASR WER with text prompts is reasonably good (31–37),
• MT/ST Comet scores with text prompts are high (76–82),
• yet performance drops substantially with spoken instructions.

Since prompt intelligibility (ASR WER of the prompts themselves) is moderate (per-language ~16–26%), these drops are attributed to difficulty in mapping spoken instructions to tasks, not to noisy audio.

Thus, lower-resource or less well‑supported languages are more vulnerable to performance degradation under spoken instructions, even when text‑prompt performance is solid.

7.3 Prompt style and length effects

Table \ref{tab:prompt_type} presents performance by prompt type, averaged over modalities and languages.

Key patterns:

• Informal and short prompts are consistently the hardest across tasks (worst metrics in many rows).
• Formal and detailed prompts generally yield the best performance, especially for Qwen.

Examples (Qwen):

• ASR (WER; lower better):
  • Basic: 37.81
  • Formal: 29.93 (best)
  • Informal: 35.11
  • Detailed: 31.99
  • Short: 39.44 (worst)
• MT (Comet; higher better):
  • Basic: 76.55 (best)
  • Formal: 74.75
  • Informal: 73.98
  • Detailed: 75.56
  • Short: 72.08 (worst)
• TSUM (BERTScore; higher better, Qwen):
  • Basic: 45.98 (best)
  • Informal: 38.62 (by far worst)
  • Short: 45.31 (strong).

Thus, even with text prompts, informal phrasing like “Hey, could you…?” hurts models; structured, explicit instructions help them interpret task demands.

7.4 Interplay between prompt type and modality

Figure \ref{fig:prompt_types_heatmap} shows, for Qwen, text vs speech difference per prompt type:

• For ASR, MT, ST, text prompts win across all styles.
• For TTS, a nuanced pattern emerges:
  • Formal and detailed instructions perform better with speech prompts than text.
  • Basic, informal, short instructions do better with text prompts.

This suggests that, for generation tasks like TTS, rich spoken instructions—with more prosodic cues and explicit specification of what to do—can sometimes be an advantage.

7.5 Gender effects

From the last two columns of Table \ref{tab:modality}:

• Some tasks show small but consistent differences in performance between prompts recorded by male vs female speakers (for languages with both genders).
• Example trends (Qwen):
  • TSUM and SSUM: better with male prompts.
  • TTS, MT, ST, S2ST: better with female prompts.

To test whether this is due to acoustic clarity, the authors transcribe all prompts with Whisper and compute WER. They find:

• No clear relationship between prompt WER and downstream performance.
• E.g., TSUM prompts have ~12% WER for both genders, yet TSUM BERTScore differs (43.88 vs 42.93).

They interpret this as possible speaker‑related bias in the models, consistent with prior work on gender gaps in speech recognition. This underscores the value of including both male and female speakers in evaluation.

---

8. Analysis and Discussion

8.1 Why text prompts outperform spoken prompts

The authors argue that:

• SLLMs are largely trained and tuned on text instructions, so they are “prompt‑aligned” primarily in the text modality.
• The mapping from speech waveform → instruction representation is still noisy:
  • internal speech encoders may lose some details,
  • speech pathway may not be fully aligned with the text instruction space.
• Synthetic or limited speech data used during training may not cover the wide variety of styles, accents, and environments represented in DOWIS.

Thus, when the instruction is spoken, semantic information in the prompt is either degraded or underutilized, causing errors in task selection and configuration (e.g., mixing ASR vs ST, ignoring specified language) and resulting in worse downstream performance.

Differences are especially severe for Phi, whose architecture seems less robust to spoken instructions; in some tasks (ASR with speech prompts) it collapses completely.

8.2 Interplay of modality, style, language, and task

DOWIS enables analysis along four axes:

1. Modality (text vs speech),
2. Style (basic, formal, informal, detailed, short),
3. Language (11 languages),
4. Task type (9 tasks, with different input/output modalities).

Concrete interplay examples:

• Informal speech:
  • Spoken informal prompts, such as “Hey, can you write out what’s being said in this audio?”, consistently perform worse than formal/detailed text prompts across tasks.
  • This shows that colloquial speech is harder for SLLMs to parse into an instruction than clean, structured text.
• Low-resource languages with speech prompts:
  • For Dutch and Portuguese ASR and MT/ST, Qwen performs well with text prompts but degrades sharply with speech instructions.
  • This suggests that the speech encoder’s representations for these languages are weaker or less aligned to the instruction space.
• Speech-output tasks:
  • For TTS and S2ST, speech prompts close the gap, sometimes match or slightly exceed text prompts.
  • Possibly because these tasks inherently require robust speech–speech mapping, and training data for speech generation already includes some speech conditioning.
• Prompt style vs modality for TTS:
  • Formal/detailed spoken prompts → better TTS than text prompts.
  • Informal/short text prompts → better than their spoken counterparts.
  • Suggests that rich, explicit spoken instructions help generative tasks, while brief speech instructions may be misinterpreted.

8.3 Error modes and robustness

Although the paper does not provide a long qualitative error analysis, the patterns imply common failure modes for spoken instructions:

• Misinterpreting the task: treating “translate” as “transcribe”, or vice versa.
• Ignoring details about target language, summarisation length, or formatting instructions (especially for ACHAP).
• Degradation in languages with weaker speech encoders (cs, nl, pt, sv).
• Sensitivity to colloquial phrasing and filler words in informal prompts.

Robustness to noise and devices:

• Given the moderate WER (~12.7%) of Whisper on the prompts, the dataset is moderately noisy but intelligible.
• The high gap between text and speech performance cannot be fully explained by acoustic corruption; it reflects real modality-induced performance gaps in SLLMs.

---

9. Implications, Limitations, and Future Work

9.1 Implications for SLLMs and speech interaction

DOWIS demonstrates that:

• Text-only evaluation is overly optimistic: models that appear strong with text prompts can fail badly when interacting via speech, particularly in lower-resource languages and for complex tasks.
• Prompt style matters: informal and short phrasing can harm performance; formal/detailed prompts help, across both speech and text.
• Speech-output tasks are less affected by the modality gap—an encouraging sign for speech assistants and conversational agents, but still warrant careful evaluation.

For future SLLMs, this implies:

• The need to explicitly align speech and text instruction pathways (e.g., via training objectives like in InSerter).
• Training on diverse, human-recorded spoken instructions, not just synthetic TTS.
• Benchmarking across languages, styles, and modalities rather than relying on a few English text prompts.

9.2 Limitations of DOWIS

The authors note several limitations:

• Scale of audio: 3h17m is relatively small; DOWIS is for evaluation, not training. It cannot cover the full variability of the real world.
• Language coverage: while 11 European languages are a good start, many language families and scripts (e.g., non‑European, tonal languages) are absent.
• Speaker diversity: 19 speakers, mostly European, with some languages having only one gender represented (e.g., no male Dutch speaker). Accents and age ranges are not systematically varied.
• Recording conditions: though realistic, they may not include extreme noise or highly challenging acoustics.
• Task coverage: 9 tasks are broad but still omit many speech tasks (e.g., emotion recognition, paralinguistics, multi-turn dialogues).

9.3 Future directions

The paper suggests that DOWIS is a first step and could be extended by:

• Adding more languages, especially low‑resource and typologically diverse ones.
• Expanding speaker diversity (accents, age, gender, geography).
• Introducing more prompt styles, including adversarial or ambiguous instructions.
• Collecting more challenging audio (e.g., noisy, overlapping speakers, long conversational prompts).
• Improving evaluation methodologies, including:
  • better metrics for speech‑to‑speech tasks,
  • more human evaluation for nuanced tasks like summarisation and chaptering.
• Using DOWIS as a benchmark for speech instruction alignment methods, tracking improvements in modality gaps.

---

10. Practical Guidance for Users of DOWIS

10.1 Integrating DOWIS with existing benchmarks

To use DOWIS in your experiments:

1. Choose a downstream task and dataset, e.g.:
   • ASR or ST on FLEURS,
   • Summarisation or QA on MCIF,
   • Chaptering on YTSeg, etc.
2. Identify the target language(s) for that dataset.
3. Select DOWIS prompts:
   • Use prompts of the appropriate task label (ASR, ST, etc.),
   • in the target language,
   • choose a prompt style (or cycle over all 5 styles),
   • choose text vs speech modality.
4. Construct the model input:
   • For text prompt: [prompt text] + [task input].
   • For speech prompt: feed the prompt audio plus the task input (speech or text) via the model’s multimodal interface.
5. Evaluate with the same metric used by the original dataset (WER, BLEU/Comet, BERTScore, Collar-F1, etc.).

This gives a direct comparison between text and speech prompting, under controlled task and data conditions.

10.2 Designing fair text vs speech experiments

Recommendations based on the paper’s methodology:

• Same content, different modality. Use DOWIS’s parallel text and speech prompts so that differences are due to modality, not content.
• Control for style. Either:
  • evaluate each style separately to see style‑specific effects, or
  • average over styles to get a robust overall estimate.
• Control for language. Report per-language scores where possible; gaps may be large in lower-resource languages.
• Use consistent evaluation metrics. For speech-output tasks, follow the paper:
  • transcribe with Whisper,
  • compute WER/Comet on transcripts,
  • compute UTMOS for quality.

10.3 Pitfalls to avoid

• Misaligned instructions and tasks. Ensure you pick the DOWIS task label that matches your downstream dataset (e.g., don’t use ST prompts for an ASR dataset).
• Ignoring target language assumptions. DOWIS prompts are phrased in the target language. If your dataset uses a different target language, you should either retranslate prompts or adapt your setup.
• Over‑reliance on a single prompt style. Using only one style (e.g., basic English text) can mask robustness issues. The paper shows big differences between informal vs formal or short vs detailed prompts.
• Averaging away language differences. Strong text advantages over speech in aggregate may hide nearly catastrophic failures in some particular languages; report per-language results.
• Assuming speech-as-ASR-then-text is equivalent to true speech prompting. The paper shows that even when speech is intelligible (low WER), SLLMs may still mishandle spoken instructions via their internal speech pathways.

10.4 Code, scripts, and examples

The dataset and accompanying code are available at:

• Hugging Face: maikezu/dowis (CC‑BY),
• GitHub: https://github.com/MaikeZuefle/DOWIS.

While the paper does not detail every script, they at least include:

• tools to load prompts and metadata,
• example pipelines for pairing DOWIS prompts with benchmarks like FLEURS and MCIF,
• plotting scripts used for the heatmaps and tables.

These examples demonstrate how to:

• select prompt variants by task, style, language, and modality;
• feed them into models like Qwen and Phi;
• compute evaluation metrics and aggregate results.

---

In summary

DoWhatISay (DOWIS) is the first multilingual, human-recorded dataset of spoken and written instruction prompts explicitly designed to evaluate instruction-following in SLLMs. It decouples instructions from task inputs, supports 9 tasks and 11 languages, and offers rich style variation. Experiments with Qwen2.5‑Omni and Phi‑4 Multimodal show that:

• text-only evaluations significantly overestimate SLLM performance,
• modality, style, language, and task type interact in nontrivial ways,
• spoken prompts are particularly challenging in low-resource and cross‑lingual settings,
• speech-output tasks show smaller modality gaps.

DOWIS thus provides a necessary, reusable tool for realistic, nuanced assessment of speech-enabled LLMs in the instruction-following regime.