WHISMA: Unified Speech-Language Model

• WHISMA is a unified speech-LLM that integrates a 32-layer Whisper encoder, a modality aligner, and a Llama-3 decoder to address multiple SLU tasks in zero-shot settings.
• The system leverages parameter-efficient fine-tuning with LoRA and multi-task instruction tuning, significantly enhancing performance in intent classification and slot filling.
• It demonstrates robust results on benchmarks such as SLURP and SLU-GLUE using innovative inference strategies like speech chain-of-thought and multi-round prompting.

Searching arXiv for the WHISMA paper and closely related models mentioned in the provided data. WHISMA is a speech LLM tailored for spoken language understanding (SLU) and designed to operate in zero-shot settings. It combines the speech encoder from Whisper with the Llama-3 LLM, and is fine-tuned in a parameter-efficient manner on a comprehensive collection of SLU-related datasets. The model is positioned as a unified speech-LLM for tasks including intent classification, slot filling, spoken question answering, and instruction-based spoken interaction, with evaluation spanning seen-task/seen-corpus, seen-task/unseen-corpus, and unseen-task/unseen-corpus regimes (Li et al., 2024).

1. Definition and system composition

WHISMA is constructed from three principal components: a speech encoder, a modality aligner, and an LLM decoder. The speech encoder is Whisper-large-v2, described here as a 32-layer Transformer with CNN downsamplers that outputs 375 speech embeddings per input after down-sampling by a factor of 8. The LLM decoder is Llama-3-8B-Instruct. Between them sits a modality aligner composed of two 1D CNN layers with stride 2, a bottleneck adaptor with dimension 320, and a linear layer that matches the dimensions of the LLM input (Li et al., 2024).

The modality aligner follows a design inspired by WavLLM. In operational terms, input audio is processed by Whisper into dense speech embeddings, the modality aligner projects these embeddings into the Llama-3 embedding space, and a task-specific text prompt is also input to Llama-3. The decoder then produces the task output, which may be SLU inference, transcription, or another prompt-conditioned response (Li et al., 2024).

Parameter-efficient adaptation is achieved with LoRA, using trainable adapters with rank $8$ and alpha $16$, inserted into all 32 Transformer layers’ attention modules of Llama-3. During fine-tuning, only the modality aligner and the inserted LoRA weights are trainable, while Whisper and Llama-3 remain frozen. This design places WHISMA within the class of end-to-end speech-LLMs that retain the pretrained capacities of both the acoustic encoder and the text decoder while learning a lightweight cross-modal interface (Li et al., 2024).

2. Fine-tuning regime and inference strategies

WHISMA is fine-tuned by multi-task learning on multiple tasks simultaneously, using harmonized prompt templates. The task set includes automatic speech recognition (ASR), intent classification (IC), slot filling (SF), spoken question answering (SQA) in open-ended and multi-choice forms, Spoken Query Instruction Tuning (SQIT), and Spoken Instruction Tuning (SIT). ASR is included for modality alignment rather than as SLU per se (Li et al., 2024).

Prompting is task-specific. For IC and SF, prompts enumerate candidate labels or slots in each instance; for other tasks, question-based or instruction-based prompts are used. Ten distinct prompt templates are randomly sampled to increase training diversity and robustness. This suggests that prompt variability is treated not merely as formatting, but as a regularizer for zero-shot transfer (Li et al., 2024).

A central aspect of WHISMA is its use of ASR-integrated inference to mitigate a known limitation of end-to-end SLU models, namely difficulty with semantic extraction, especially for slot filling. Two end-to-end inference variants are defined.

• Speech Chain-of-Thought (SCoT): concatenates ASR and SLU instructions in a single prompt, so the system performs both tasks sequentially in one step.
• Multi-Round (MR): the first round produces a transcript, and the second uses that transcript in the dialogue history as auxiliary context for SLU.

Both variants remain within the same model pipeline, so the ASR stage functions as an auxiliary reasoning scaffold rather than as an external modular subsystem. A common misconception is that such designs cease to be end-to-end once transcription is introduced. Here, the formulation explicitly maintains end-to-end decoding because both steps occur inside the same speech-LLM pipeline and leverage the LLM’s reasoning (Li et al., 2024).

3. Training corpora, released resources, and evaluation regimes

The fine-tuning corpus covers approximately 2000 hours of speech and spans several task families. The paper lists GigaSpeech (Medium) for ASR, SLURP (zero-shot split) for IC/SF, LibriSQA part I and part II for SQA, and Spoken-Alpaca for SQIT and SIT (Li et al., 2024).

Task	Source	Hours / Samples
ASR	GigaSpeech (Medium)	1000 hours, 910k samples
IC/SF	SLURP (zero-shot split)	80 hours, 94k samples
SQA	LibriSQA-partI, partII	720 hours, 208k samples
SQIT	Spoken-Alpaca	30 hours, 29k samples
SIT	Spoken-Alpaca	14 hours, 14k samples

Two resources are particularly salient. Spoken-Alpaca is a speech-synthesized version of the Alpaca instruction-tuning dataset, produced via an in-house VITS TTS using LibriSpeech speaker data, with filtering to avoid unsuitable content such as mathematical material. In SIT, speech is used for input fields with instruction as textual context; in SQIT, speech is used for the instruction field with no textual context (Li et al., 2024).

The second resource is SLU-GLUE, introduced as a new task-agnostic benchmark for zero-shot SLU. Its tasks include Sentiment Analysis using SST-2, Semantic Equivalence Recognition using QQP and QNLI, and Spoken-Textual Entailment Recognition using RTE and SciTail. The benchmark contains approximately 14,000 samples and is described as containing multiple speakers, real-world noise, and diverse domains (Li et al., 2024).

Evaluation is organized into three zero-shot scenarios.

• STSC (Seen-task, seen-corpus): exemplified by the SLURP test split, where slot types in test are not seen in training.
• STUC (Seen-task, unseen-corpus): exemplified by FSC and SmartLight, where IC/SF tasks are transferred to new domains.
• UTUC (Unseen-task, unseen-corpus): exemplified by SLU-GLUE, where both tasks and domains are absent from training.

The metrics are Word Error Rate for ASR, Intent Classification Accuracy for IC, SLU-F1 for SF, and Binary Accuracy for SLU-GLUE tasks (Li et al., 2024).

4. Zero-shot performance across SLU settings

On the SLURP benchmark in the STSC setting, WHISMA substantially improves zero-shot slot filling. The reported SLU-F1 is $50.0\%$ for ZS-Whisper-SLU, $46.8\%$ for WHISMA without ASR, and $63.1\% / 63.3\%$ for WHISMA with SCoT and MR, respectively. The paper reports a relative gain of $26.6\%$ over the current state-of-the-art model on this benchmark (Li et al., 2024).

On FSC in the STUC setting, WHISMA reaches $97.3\%$ IC accuracy with both SCoT and MR. This is compared in the paper to $77.5\%$ for BLSP, $95.0\%$ for ZS-Whisper-SLU, and $89.3\%$ for a modular Whisper-Llama-3 baseline. The result indicates strong transfer for seen tasks under domain shift (Li et al., 2024).

On SmartLight, also in the STUC setting, WHISMA with SCoT obtains $16$0 intent classification accuracy, $16$1 slot-filling SLU-F1, and $16$2 perfect parsing. The paper compares these with $16$3, $16$4, and $16$5, respectively, for ZS-Whisper-SLU. The mixed outcome on slot filling and perfect parsing indicates that superiority is not uniform across all submetrics, even when intent classification improves (Li et al., 2024).

On SLU-GLUE in the UTUC setting, WHISMA is evaluated for generalization to unseen tasks and domains. The reported average accuracy over five tasks is $16$6 for Qwen-Audio, $16$7 for Qwen-Audio+MR, $16$8 for modular Whisper-Llama-3, and $16$9 for WHISMA with SCoT and MR, respectively. The relative improvement over Qwen-Audio+MR is reported as $50.0\%$0, and the relative improvement over the modular system as $50.0\%$1 (Li et al., 2024).

Taken together, these results present WHISMA as strongest in regimes where zero-shot generalization is central, especially slot filling on SLURP and task-agnostic transfer on SLU-GLUE. A plausible implication is that the model’s gains are tied not only to architectural integration, but also to its prompt-conditioned multi-task training mixture.

5. Generalization, instruction tuning, and comparative position

The paper emphasizes generalization to unseen domains and unseen tasks as a defining property of WHISMA. On SLU-GLUE, tasks and domains such as textual entailment, sentiment, and science are explicitly outside the fine-tuning set, yet WHISMA+MR reaches $50.0\%$2 average accuracy. This exceeds Qwen-Audio+MR at $50.0\%$3 and modular Whisper-Llama-3 at $50.0\%$4 (Li et al., 2024).

A key reported factor is Spoken-Alpaca. Without Spoken-Alpaca, the paper reports a substantial drop in generalization; for example, the no-ASR average on SLU-GLUE drops from $50.0\%$5 to $50.0\%$6, corresponding to a relative decrease of $50.0\%$7. The paper also states that adding only $50.0\%$8 more data via the Spoken-Alpaca instruction-tuning dataset yields a significant increase in generalization to new zero-shot SLU tasks (Li et al., 2024).

WHISMA is compared with several baselines. These include supervised models such as SNLU and Finstreder, zero-shot systems such as ZS-Whisper-SLU, BLSP, and Qwen-Audio, and a modular ASR-LLM pipeline in which Whisper ASR is followed by a Llama-3 text LLM. Other speech-LLMs mentioned include Qwen-Audio and WavLLM (Li et al., 2024).

The comparison with Qwen-Audio is particularly important because both support similar inference setups, including multi-round prompting. The paper states that WHISMA+MR reaches $50.0\%$9 average accuracy on SLU-GLUE versus $46.8\%$0 for Qwen-Audio+MR, and interprets this as strong evidence that WHISMA’s architectural and training advances produce better generalization and task transfer (Li et al., 2024). The comparison with the modular Whisper+Llama-3 system is narrower: on SLU-GLUE, WHISMA+MR is higher at $46.8\%$1 versus $46.8\%$2, while the paper also argues that the end-to-end design reduces pipeline complexity and allows more effective prompt-based reasoning.

6. Limitations, caveats, and significance for SLU

WHISMA is not presented as uniformly superior on every axis. The paper notes that its ASR performance, measured by WER in SCoT and MR settings, lags behind some strong baselines because the multi-task fine-tuning uses limited ASR data. However, the paper also states that joint SLU performance is not impaired thanks to the end-to-end adaptation (Li et al., 2024).

The work also identifies a limitation in reasoning behavior. Speech-LLMs, including WHISMA, do not automatically learn advanced reasoning such as chain-of-thought or multi-round behavior unless these are explicitly integrated into fine-tuning. This is relevant because SCoT and MR are not merely inference-time wrappers; they are part of the model’s operational logic for extracting semantics from speech (Li et al., 2024).

Another caveat concerns dataset coverage. Although the fine-tuning tasks are described as highly diverse, they do not cover all possible SLU scenarios, and the paper argues that further work should expand both real and synthetic speech-instruction diversity (Li et al., 2024). This suggests that WHISMA should be interpreted as a strong but not exhaustive template for zero-shot SLU.

Within the field, WHISMA’s significance lies in demonstrating a unified instruction-prompted interface for multiple spoken language tasks in zero-shot settings. The paper presents it as an end-to-end SLU system combining foundation models of speech and language through a modality aligner and lightweight adaptation, while also releasing Spoken-Alpaca and SLU-GLUE to support reproducibility and follow-on research (Li et al., 2024). A plausible implication is that WHISMA marks a shift from task-specific spoken understanding pipelines toward general-purpose speech-LLMs that are optimized for cross-task and cross-domain transfer rather than for a single narrowly supervised benchmark.