Whisper-Medium: Speech Model Analysis

• Whisper-Medium is a pretrained encoder–decoder speech model characterized by consistent 1024-dimensional representations and varying architectural descriptions, making it a stable substrate for downstream tasks.
• It demonstrates exceptional capabilities in dysarthric speech probing and low-resource language adaptation, with mid-layer encoder states achieving optimal accuracy and reduced WER.
• The model supports advanced adaptations including automated layer selection, audiovisual fusion for noise robustness, and structured pruning to balance compute efficiency with performance.

Whisper-Medium is a pretrained encoder–decoder speech model that has been studied not only as an automatic speech recognition backbone, but also as a source of intermediate representations for pathological-speech analysis, a substrate for self-supervised low-resource adaptation, a target for end-to-end language-specific fine-tuning, a base model for audiovisual fusion, and a candidate for structured compression. Across recent reports, its most consequential properties are not limited to headline transcription accuracy: they include layer-wise representational structure, sensitivity to domain adaptation, amenability to multimodal fusion, and nontrivial compute–accuracy trade-offs under pruning (Yue et al., 5 Oct 2025, Wang et al., 9 Jun 2026, Rijal et al., 2024, Li et al., 26 Jan 2026, Mudi et al., 14 Oct 2025).

1. Reported model identity and architecture

Whisper-Medium is consistently treated as a sequence-to-sequence speech model with an encoder–decoder backbone, but the architectural descriptions reported in application papers are not fully uniform. One study describes it as a standard “encoder–decoder” Transformer with a 24-layer encoder and a 6-layer decoder; each encoder layer has hidden dimension $d=1024$, $H=16$ attention heads, and a position-wise feed-forward inner dimension of approximately $4096$, with total parameter count “about 769 M parameters” and encoder/decoder parameters each approximately $385$ M. A pruning study instead describes Whisper-medium as having a Conformer-style encoder of 24 layers and a Transformer-style decoder of 24 layers, again with $d=1024$, $H=16$, and inner dimension $4096$. An audiovisual adaptation paper reports a base Whisper Medium with a Transformer encoder of $N^{ENC}=36$ blocks and decoder of $N^{DEC}=32$ blocks. A Nepali ASR fine-tuning study characterizes the model as having “roughly 580 M parameters” in discussing capacity (Yue et al., 5 Oct 2025, Mudi et al., 14 Oct 2025, Li et al., 26 Jan 2026, Rijal et al., 2024).

Source	Reported encoder/decoder description	Reported scale
Dysarthria probing	24-layer encoder, 6-layer decoder	about 769 M
Structured pruning	24-layer Conformer-style encoder, 24-layer Transformer-style decoder	769 M
AV-ASR fusion	36-block encoder, 32-block decoder	not summarized as total parameters
Nepali fine-tuning	end-to-end Whisper Medium	roughly 580 M

Two reports agree on the internal dimensionality of major sublayers, specifically 1024-dimensional hidden representations, 16 attention heads, and 4096-dimensional feed-forward expansions. This suggests that, within the cited application literature, Whisper-Medium is more stable as a representational substrate than as a uniformly reported architectural specification. For technical reading, that distinction matters: layer selection, probing, or pruning results can be portable across studies even when secondary summaries disagree on exact block counts or parameter totals.

2. Layer-wise representations and dysarthric-speech probing

A detailed layer-wise analysis of Whisper-Medium for pathological speech focuses on dysarthria detection and severity assessment. In that setup, 80-dimensional log-mel filterbanks are fed into the encoder, each encoder layer $\ell \in \{1,\dots,24\}$ produces a sequence of 1024-dimensional token embeddings $H=16$0, and frame-level embeddings are averaged across time to produce an utterance-level vector $H=16$1. For each layer, a linear probe is trained as

$H=16$2

with $H=16$3 for detection and $H=16$4 for severity classification. Single-task probing uses one head per task, whereas multi-task probing uses two heads sharing the same $H=16$5 and optimizing $H=16$6. Evaluation uses five-fold Stratified-KFold on TORGO, AdamW with learning rate $H=16$7, 20 epochs per fold, and batch size 32. The same study complements probe accuracy and $H=16$8 with Silhouette score

$H=16$9

and mutual information

$4096$0

These analyses converge on the same conclusion: encoder layers 13–15 are the most informative for dysarthric speech, while the earliest layers emphasize low-level spectral and phonetic cues, layers 7–12 become increasingly text-aligned and contextual, and layers 16–24 become more ASR-specialized while still preserving dysarthria-relevant subspaces (Yue et al., 5 Oct 2025).

The quantitative pattern is unusually consistent across metrics. Independent linear probes for both detection and severity achieve maximal accuracy, $4096$1, and mutual information at layers 13–15. Silhouette scores for dysarthric versus typical speech peak at approximately layer 13. Even the final encoder layer remains “good enough,” with greater than 90% accuracy, which was interpreted as evidence that Whisper does not collapse all speakers into a single ASR-centric manifold. The best single-task results are reported around layer 14: detection accuracy of $4096$2 for the pretrained model and $4096$3 after fine-tuning on TORGO ASR; severity accuracy of $4096$4 pretrained and $4096$5 fine-tuned. Mutual information between pretrained and fine-tuned embeddings is greater than 0.9 in lower layers $4096$6, approximately $4096$7–$4096$8 in the mid layers 13–15, and approximately $4096$9–$385$0 above layer 16, indicating modest adaptation rather than representational replacement.

The practical implication is that dysarthria-related signal in Whisper-Medium is concentrated in mid-level encoder states rather than in the final encoder output or in task-specific fine-tuning alone. The same report states that a frozen encoder up to layer 15 plus a single $385$1 or $385$2 linear head, amounting to approximately $385$3 K–$385$4 K parameters, can achieve greater than 94% accuracy. It also reports little to no benefit from heavier multi-task heads. A plausible implication is that, for pathological-speech assessment, the central design question is not merely whether to fine-tune Whisper-Medium, but which layer’s geometry should be exposed to the downstream estimator.

3. Automated layer selection in low-resource adaptation

In low-resource domain adaptation, Whisper-Medium has also been used as the backbone for a trainable layer-selection mechanism rather than a fixed-depth feature extractor. Gumbel-BEARD introduces a learnable logit vector $385$5 over the $385$6 encoder layers of Whisper-medium and samples a discrete layer index using hard Gumbel-Softmax with a straight-through estimator. Once a one-hot selector $385$7 is produced, the student representation is

$385$8

The selected layer is trained with a BEST-RQ masked-prediction objective. A frozen random projection and quantizer with codebook size $385$9 generates target indices, and the quantization loss on masked frames is

$d=1024$0

Two cosine-distillation terms preserve pre-trained knowledge:

$d=1024$1

yielding the stage-one objective

$d=1024$2

with $d=1024$3 and $d=1024$4 (Wang et al., 9 Jun 2026).

The training pipeline is two-stage. First, self-supervised adaptation runs on unlabeled target-domain audio using the Whisper encoder only, with the decoder frozen or detached, $d=1024$5 initialized to zero, temperature $d=1024$6 annealed linearly from 5.0 to 0.1 over a single epoch, Adam at learning rate $d=1024$7, batch size 32, and one adaptation epoch. Second, supervised fine-tuning reattaches the decoder and uses limited in-domain transcripts. The method is designed to replace manual intermediate-layer choice with end-to-end trainable routing.

For Whisper-medium, the reported gains are substantial in child-speech and dialectal settings. On MyST, using the full labeled data, supervised fine-tuning gives 8.86% WER, while Gumbel-BEARD reaches 8.21% WER. The abstract further states that with 10 h of labeled data, the method matches a fully supervised baseline trained on the complete 133 h labeled set. On CORAAL, zero-shot Whisper-medium yields 17.09% WER, the supervised fine-tuning baseline gives 9.81%, and Gumbel-BEARD reaches 9.25%, reported as statistically significant with $d=1024$8. The replication notes emphasize that the same SSL hyperparameters remain unchanged across model sizes and corpora, and adaptation is reported on a single NVIDIA RTX 5090 GPU. This suggests that Whisper-Medium’s encoder depth is not merely descriptive structure: it is an optimization variable that can be learned to match target-domain acoustics.

4. End-to-end language adaptation: the Nepali case

Whisper-Medium has also been adapted through conventional end-to-end fine-tuning for an underrepresented language. In the Nepali study, the model is trained on a 42.9-hour corpus formed by merging four open-source collections—Fleurs (10.38 h), Common Voice (1.28 h), SLR43 (2.82 h), and SLR143 (1.25 h)—with a custom 13.58-hour manually curated speech set. The custom portion spans speakers aged 25–60, includes male, female, and unknown gender tags, covers clean-room, white-noise, and crowded acoustic conditions, and contains happy, sad, neutral, and angry sentiment. Audio is chunked into segments up to 30 seconds long with silences greater than 1 second removed. A single augmentation is used: 8000 Hz white noise added through torchaudio at approximately $d=1024$9 dB, expanding the custom portion from 13.58 h to 27.17 h and contributing to the 42.9 h “all_combined” set. Data are shuffled and split 80%/20% into train and hold-out (Rijal et al., 2024).

The reported fine-tuning recipe leaves the Medium architecture unaltered and updates the full encoder and decoder stack end-to-end. Training uses Adam with initial learning rate $H=16$0, linear warm-up over the first 10% of steps, and linear decay thereafter. Batch size is 16, with no gradient accumulation required on an RTX 3090 with 24 GB memory. The “all_combined” set is trained for 4000 epochs; smaller corpora use early stopping at 1500 epochs when validation loss shows no improvement for 100 consecutive epochs. Checkpoints are saved every 250 epochs.

The performance summary is anchored in Word Error Rate,

$H=16$1

where $H=16$2, $H=16$3, and $H=16$4 denote substitutions, deletions, and insertions. For Whisper-Medium on “all_combined,” training cross-entropy declines from roughly 0.45 to 0.10 and validation loss from about 0.65 to 0.15. The final held-out WER is 23.8%, compared with a Whisper baseline of 54.4% on Fleur’s Nepali test set. The same summary attributes the improvement to greater variation in speaker age, gender, sentiment, dialect, acoustic environment, longer 15–30 second segments, and manual curation. A plausible implication is that, when Whisper-Medium is adapted to a low-resource language, corpus construction and utterance design can be at least as consequential as optimizer choice.

5. Audiovisual fusion and noise robustness

Whisper-Medium has also been extended beyond audio-only ASR through visual conditioning. In a noise-robust AV-ASR system, the base audio branch uses a 26-dimensional log-filterbank frontend over 16 kHz audio with 25 ms window and 10 ms hop, while the visual branch uses AV-HuBERT large operating on 25 Hz grayscale lip crops of size $H=16$5. The proposed “dual-use” strategy injects visual features both before the Whisper encoder and inside the decoder. Encoder-side fusion upsamples visual latents to the audio frame rate, projects them to the audio model dimension, scales them with a trainable scalar $H=16$6 initialized at zero, and adds them to acoustic features:

$H=16$7

Decoder-side fusion inserts Flamingo blocks before each decoder block, using cross-modal attention from decoder queries onto visual latents:

$H=16$8

followed by gated residual updates

$H=16$9

with trainable gating vectors $4096$0 and $4096$1 initialized at zero (Li et al., 26 Jan 2026).

Training uses standard sequence cross-entropy without auxiliary alignment losses. Fine-tuning data total 1929 hours, combining LRS3 (433 h), LRS2 (224 h), and an English subset of VoxCeleb2 pseudo-labels (1326 h). The optimization schedule is explicitly two-stage: Whisper medium is first fine-tuned on audio only for 90 k steps with 1 k warm-up and peak learning rate $4096$2, then the AV-Whisper medium is trained for 120 k steps with 5 k warm-up and peak learning rate $4096$3. MUSAN babble noise at 0 dB SNR is added during second-stage AV fine-tuning, and evaluation also uses unseen NoiseX babble at $4096$4 dB.

The reported gains are large under severe noise. On LRS3 test with MUSAN babble at 0 dB, audio-only fine-tuned Whisper medium yields 8.90% WER, a middle-fusion baseline gives 9.53%, and dual-use fusion reaches 4.07%, described as a 57% relative improvement over middle fusion. Across $4096$5 dB and clean conditions on MUSAN babble, average WER is 4.08%; under NoiseX babble, average WER is 4.43%. The authors state that this establishes a new state of the art on noisy LRS3 AV-ASR. The central methodological point is that Whisper-Medium can be modified both at the encoder input and at decoder attention interfaces, allowing the model to learn audiovisual interactions at depth while still performing context-dependent modality weighting at decoding time.

6. Structured sparsity, pruning, and methodological tensions

Whisper-Medium has also been studied as a target for compute and memory reduction through structured sparsity. The pruning framework adds a Sparse Group LASSO penalty to the cross-entropy loss,

$4096$6

with $4096$7 and $4096$8. Column-wise group penalties are applied to attention- and MLP-related matrices, after which a two-pass weight-statistics-aware pruning algorithm first removes nearly zero columns and then prunes remaining small weights according to layer-dependent thresholds. For Whisper-medium’s fully connected layers, the report states that the first five encoder layers were Type A and the remaining encoder plus all decoder layers were Type B, leading to threshold choice $4096$9 for layers 0–4 and $N^{ENC}=36$0 for layers 5–23. Fine-tuning on Common Voice 11.0 Hindi uses approximately 13 hours of training data, batch size 8, 5000 steps, AdamW with weight decay 0.01, and a cosine learning-rate schedule with 20% warm-up (Mudi et al., 14 Oct 2025).

The detailed numbers for Whisper-medium are as follows. The baseline fine-tuned model has 769 M parameters, 1557 MB memory, 4.95 $N^{ENC}=36$1 FLOPs, and 15.86 WER. After 31.0% pruning, the model has 530 M parameters, 1319 MB memory, 4.11 $N^{ENC}=36$2 FLOPs, and 19.58 WER. At 38.9% pruning, WER rises to 21.15; at 51.5%, to 27.80. The same summary states that only regularizing fully connected layers, rather than all layers, gave the best trade-off, and that the trade-off curve becomes steep beyond roughly 40% pruning. It also reports that the method substantially outperforms an Iterative Magnitude Pruning baseline by pruning more parameters with a much smaller WER increase.

A notable tension appears inside the reported summary itself. The abstract states that on Whisper-medium the method obtains “31% reduction in model parameters, 15.29% lower memory consumption and 16.95% fewer FLOPs” “without degrading WER,” whereas the detailed Net-3 table reports a shift from 15.86 WER to 19.58 WER at 31.0% pruning. This does not invalidate the compression result, but it does require careful reading of summary claims versus tabulated metrics. More broadly, the cited literature suggests three recurrent corrections to oversimplified views of Whisper-Medium. First, the final encoder layer is not necessarily the most informative representation. Second, full end-to-end fine-tuning is not always required to extract high-value domain signal. Third, compression, multimodal fusion, and self-supervised adaptation affect different parts of the model and therefore should not be evaluated as interchangeable “improvements.”