Frozen Whisper Encoder Features

• Frozen Whisper Encoder Features are fixed, non-adaptive activations extracted from a pre-trained Whisper ASR model, capturing rich multi-lingual, phonetic, and semantic details.
• These features are aggregated from various encoder layers using techniques like weighted-sum pooling, serving as versatile inputs for tasks such as speech coding and classification.
• Empirical studies show that they deliver high isotropy and state-of-the-art results in speech tasks, including low-bitrate coding, speaker verification, and continual learning.

Frozen Whisper encoder features are the fixed activations extracted from a pre-trained Whisper automatic speech recognition (ASR) model, employed as non-adaptive representations in a variety of downstream neural pipelines. These features are typically either directly used for tasks such as speech coding, assessment, or classification, or they serve as the bottleneck representations in parameter-efficient adaptation schemes. Freezing the encoder ensures strong text-alignment and preserves the rich multi-lingual, phonetic, and semantic information captured during Whisper’s large-scale ASR pre-training. Rigorous empirical studies demonstrate that, when selected and processed appropriately, frozen Whisper encoder features yield state-of-the-art performance across a broad spectrum of speech tasks, from low-bitrate coding (Zhang et al., 23 Oct 2025) to dysarthria detection (Yue et al., 5 Oct 2025), continual learning (Wang et al., 2 Jun 2025), speech quality prediction (Close et al., 4 Aug 2025), and more.

1. Whisper Encoder Structure and Frozen Feature Extraction

Whisper models consist of a convolutional frontend followed by multiple stacked Transformer layers: Whisper-base uses 6 layers, Whisper-small has 12, Whisper-medium 24, and Whisper-large-v2 up to 32. Each Transformer block consists of multi-head self-attention, feed-forward layers, residual connections, and per-block LayerNorm. The encoder accepts either 80-channel log-Mel spectrogram frames or raw waveforms (pre-converted), producing a sequence of hidden-state tensors $h_\ell \in \mathbb{R}^{T \times d}$ at each layer $\ell$, where $T$ is the number of frames and $d$ the hidden size.

Frozen feature extraction involves running an input utterance through the pre-trained (and unmodified) Whisper encoder and extracting activations from selected layers. Some approaches use only the final-layer embeddings, while others apply learned or fixed weights to combine outputs across all or a subset of layers ("weighted-sum" pooling) (Yang et al., 2023, Wang et al., 2 Jun 2025, Yue et al., 5 Oct 2025, Ma et al., 10 Sep 2025). No further weight updates or gradients are propagated through the frozen encoder during downstream task training, except when parameter-efficient adapters or external heads are introduced.

2. Layer Selection, Aggregation, and Task-Specificity

Empirical analyses consistently show that the informativeness of Whisper encoder features is highly layer-dependent and task-dependent:

• Content/ASR tasks: Final encoder layers specialize in phonetic and lexical information essential for transcription and keyword spotting (Yang et al., 2023, Kwon et al., 9 Aug 2025, Li et al., 2023).
• Speaker/Paralinguistic tasks: Speaker identity and emotion cues are concentrated in intermediate or mid-late layers (e.g., layers 13–15 in Whisper-Medium for dysarthria (Yue et al., 5 Oct 2025), layers 17–24 in Whisper-Large-v2 for speaker verification (Zhao et al., 28 Aug 2024)).
• Fusion approaches: Weighted sums or gated-fusion of all per-layer outputs (learned gating matrices or softmax-weighted layer pooling) achieve superior cross-task generalization and allow continual learning without encoder retraining (Wang et al., 2 Jun 2025, Close et al., 4 Aug 2025, Li et al., 2023).

Table: Typical layer selection strategies

Task domain	Layer selection	Rationale
ASR, content, KWS	Last layer or weighted last N	Maximal text/phoneme encoding
Speaker, emotion	Intermediate/mid-late layers	Maximal speaker/paralinguistic cue
Fusion, continual learn	Weighted all-layer pooling	Cross-task trade-off, flexibility

For all strategies, per-frame activations are typically pooled via mean, attention, or statistics pooling to yield fixed-size utterance embeddings, which are fed to downstream heads (linear, MLP, CNN, or decoder input).

3. Encapsulation and Usage in Downstream Pipelines

Frozen Whisper encoder features serve as versatile inputs to a range of downstream architectures:

• Speech Codec Bottlenecks: SimWhisper-Codec deploys a frozen (architecturally simplified) Whisper encoder as the semantic-acoustic bottleneck in a generative, low-bitrate universal codec. Features are downsampled, projected via residual convolutions, quantized with finite scalar quantization (FSQ), and decoded through a symmetric Transformer and vocoder. Crucially, convolutional front-end GELUs and absolute positional encodings are stripped to improve acoustic detail retention, yielding state-of-the-art balance between semantic preservation (WER) and acoustic quality (PESQ/STOI/SIM) at 1.1 kbps (Zhang et al., 23 Oct 2025).
• Multi-Task, Continual, and Plug-in Scenarios: In continual speech learning (Wang et al., 2 Jun 2025), a Gated-Fusion Layer learns task-specific softmax weights over all encoder layers, dynamically aggregating features for six generative speech tasks with no encoder training. For open-vocabulary keyword spotting or contextual biasing, a frozen encoder’s weighted-sum features participate in a similarity-matching CNN without ever updating ASR parameters (Li et al., 2023).
• Proxy/Adapter/LoRA Approaches: Sample-specific encoder perturbations leverage a frozen encoder; a small proxy MLP predicts WER for each utterance, and its gradient is used to compute a per-sample encoder perturbation, reducing WER by up to 0.8% absolute with negligible overhead (Fathullah et al., 1 May 2024). LoRA adapts only low-rank updates, leaving the main encoder path frozen, and delivers competitive results with orders-of-magnitude fewer trainable parameters in both speaker verification (Zhao et al., 28 Aug 2024) and SER (Ma et al., 10 Sep 2025).
• Domain Adaptation/Self-Supervision: Frameworks such as BEARD freeze a pre-trained Whisper-small encoder (teacher) and adapt a copy (student) on unlabeled target-domain data, using a BEST-RQ self-supervised objective and cosine distillation loss on all intermediate and final layer features, before reattaching the pre-trained decoder (Bagat et al., 28 Oct 2025).

4. Empirical Properties and Quantitative Performance

Across a variety of domains, frozen Whisper encoder features consistently demonstrate:

• High isotropy: Whisper encoder representations are nearly isotropic, unlike the severe anisotropy of wav2vec2 or WavLM (isotropy: $\sim10^{-2}$ for Whisper vs $\sim10^{-14}$ to $\sim10^{-300}$ for others) (Yang et al., 2023). This facilitates easy linear separation and stable downstream “head” training.
• Universality and clustering: t-SNE and other analyses confirm that content (keywords, intents) cluster tightly in the frozen feature space, explaining rapid convergence and few-shot effectiveness. Speaker-specific information is less concentrated, but accessible through appropriate layer selection (Yang et al., 2023, Zhao et al., 28 Aug 2024).
• Quantitative SOTA: Frozen encoders combined with simple downstream heads outperform (WER, F1, accuracy, EER) much larger or fully finetuned models in low- and moderate-data regimes, e.g., SimWhisper-Codec beats Mimi and XCodec2.0 for both semantics and acoustics (Zhang et al., 23 Oct 2025); WhiSQA yields highest MOS correlation for non-intrusive speech quality prediction (Close et al., 4 Aug 2025); Whisper-PMFA achieves a 1.42% EER against the best ECAPA-TDNN and ResNet34 (Zhao et al., 28 Aug 2024).

5. Architectural Simplifications and Processing for Feature Utility

Architectural manipulations are often applied to further enhance the informativeness and parsimony of frozen Whisper features for bottleneck use:

• Removal of convolutional GELUs: Replacing convolution+GELU with linear convolutions at the front end substantially restores spectral detail (PESQ-NB increases from ≈1.24 to ≈3.67), as shown empirically in SimWhisper-Codec (Zhang et al., 23 Oct 2025).
• Dropping absolute positional encodings: Stripping PEs from self-attention eliminates position-specific biases detrimental for acoustic reconstruction and repeated pattern handling.
• Downsampling and projection: Features are typically stacked and projected from their original high dimension (e.g., 768) to a lower latent space (e.g., 32–256) before quantization or further processing (Zhang et al., 23 Oct 2025, Wang et al., 2 Jun 2025).
• Task-specific gating or fusion: Gated-fusion layers or learnable pooling (softmax over layer weights) enables the system to dynamically select the most instructive combination of layer-wise embeddings for a given task without modifying the frozen encoder itself (Wang et al., 2 Jun 2025, Yue et al., 5 Oct 2025, Close et al., 4 Aug 2025).

6. Recommended Usage Strategies and Limitations

The optimal scheme for using frozen Whisper encoder features is strongly task and data regime dependent:

• Low-resource settings: Freezing the entire encoder and training only a downstream classifier or decoder generally yields highest performance on content and semantic tasks with small data (Yang et al., 2023).
• Transfer/parameter efficiency: For domain adaptation or continual learning, prefer adapters, gating, or student-teacher distillation frameworks, which retain backbone invariance and avoid catastrophic forgetting (Wang et al., 2 Jun 2025, Bagat et al., 28 Oct 2025).
• Task-specific selection: Use a single (final) layer for content/ASR, intermediate layers or all-layer weighted sums for paralinguistic and cross-task adaptation.
• Practical notes: Mean pooling remains an effective aggregation for utterance-level tasks. When data size increases (>10%), unfreezing upper encoder layers or fine-tuning adapters can yield further gains in semantic transfer, though at increased compute cost and risk of overfitting (Ameer et al., 2023).

7. Representative Benchmarks and Comparisons

System/Feature Approach	Key Task(s)	Main Metric/Result	Reference
SimWhisper-Codec (frozen, simplified Whisper encoder)	Low-bitrate speech coding	WER 3.10, PESQ-NB 2.98, SIM 0.83 @1.1kbps	(Zhang et al., 23 Oct 2025)
Gated-fusion (all frozen layers)	Continual learning (multi-task)	SID Acc 83.93, ER Acc 68.39, MR 2.50	(Wang et al., 2 Jun 2025)
Layer 13–15 frozen features	Dysarthria detection/assessment	Acc 94.4%/94.1%, MI=0.32 peak	(Yue et al., 5 Oct 2025)
Whisper-PMFA mid-late block concat	Speaker verification	EER 1.42% VoxCeleb1, EER 8.23% CN-Celeb1	(Zhao et al., 28 Aug 2024)
Sample-specific proxy perturbation	ASR (WER reduction)	WER↓ by up to 0.8% (AMI IHM)	(Fathullah et al., 1 May 2024)
WhiSQA (frozen encoder + attention pooling)	Speech quality prediction	MOS correlation 0.94 (LIVETALK/P501/FOR)	(Close et al., 4 Aug 2025)

These results establish that frozen Whisper encoder features, when carefully selected and appropriately aggregated, provide robust, information-rich representations that excel as bottleneck, pretext, or plug-in features for both speech recognition and a wide array of downstream paralinguistic and generative tasks.