Target Speaker ASR with Whisper

• Target speaker ASR with Whisper is a framework that refines the foundational model by integrating speaker conditioning via prompt tuning, diarization, and modular adaptations in overlapping audio.
• It employs parameter-efficient techniques like LoRA and adapter modules to achieve significant WER reductions and robust adaptation to diverse speaker attributes.
• The approach underpins applications from conversational transcription to voice command execution while addressing challenges in diarization accuracy and low-resource settings.

Target speaker automatic speech recognition (ASR) using Whisper denotes the suite of algorithms, architectural adaptations, and conditioning strategies intended to restrict transcription to the speech of a pre-identified target speaker within overlapping and multi-talker audio environments. This capability is critical for meeting transcription, conversational AI, and device control use cases where ASR must discriminate and focus on speaker-attributed content for accurate dialogue reconstruction, command execution, or privacy. Whisper, as a large-scale foundational ASR model, has prompted substantial research into extending its single-speaker paradigm to effective target speaker ASR via prompt tuning, diarization conditioning, joint optimization, and modular adaptation techniques.

1. Architectures and Conditioning Strategies

Several design genres have emerged for integrating target speaker focus into Whisper:

Prompt Tuning and Adapter Modules:

Prompt tuning methods prepend trainable soft prompt tokens and/or speaker projections to the input of Whisper's encoder and/or decoder (Ma et al., 2023). The speaker embedding, after projection, is processed either as part of the input feature sequence or as a direct condition on intermediate activations. LoRA-based adaptation modules—injected selectively per speaker or language—provide low-rank, parameter-efficient updates to key layers, enabling tailored adaptation without full retraining (Song et al., 7 Jun 2024, Zhao et al., 7 Aug 2024). Conditional gating, as used in DistilWhisper, routes tokens through expert or generic paths depending on input language (or potentially speaker) characteristics (Ferraz, 2 May 2024).

Diarization-Conditioned Transformations:

Frame-level diarization dependent conditioning, as in the FDDT and QKb (Query-Key bias) techniques (Polok et al., 14 Sep 2024, Polok et al., 30 Dec 2024), foregoes explicit speaker embeddings. Instead, the model accesses per-frame probabilities that a segment belongs to silence, the target speaker, a non-target speaker, or overlapping speech (the STNO mask). Affine transformations (or bias terms) are applied to hidden states in the encoder, selecting or suppressing features according to diarization output. Query-key biasing further modulates attention scores to focus on the target speaker. These methods demonstrate that relative labels (STNO per frame) simplify model adaptation for speaker-attributed ASR, outperforming separation-plus-diarization cascades.

Sidecar Separation and Speaker-Querying:

For joint multi-talker and target-talker ASR, a Sidecar separator is interposed in early encoder stages, decomposing mixed embeddings into talker-specific branches via temporal convolution networks (Meng et al., 13 Jul 2024). Speaker-querying models introduce explicit trainable query vectors to extract speaker-specific prompts from the mixture and enroLLMent samples (Guo et al., 7 Dec 2024). These prompts condition both encoder and decoder representations, jointly optimized with contrastive losses and cross-entropy.

Serialized Output Training (SOT):

SOT serializes speaker-role tokens and word tokens in the output sequence, yielding transcriptions with embedded role labels in one pass. The decoder's vocabulary is expanded to include dedicated tokens for speaker roles, and decoding is initialized with tokens such as <startoftranscript> <en> <transcribe> <notimestamps> (Xu et al., 12 Jun 2025).

2. Training Paradigms and Adaptation Techniques

Full Fine-Tuning vs. Parameter-Efficient Strategies:

While full fine-tuning of Whisper on target speaker data is possible, it is inefficient and prone to overfitting on limited data (Ma et al., 2023, Ferraz, 2 May 2024). Parameter-efficient approaches—soft prompt tuning, LoRA modules, modular experts, CNN-based profile libraries—allow fast adaptation to speakers or speaker attributes (accent, gender, age) with only 1–10% additional parameters and minimal computational cost.

Hybrid and Incremental Learning:

Frameworks such as PI-Whisper and AS-ASR enable incremental and attribute-aware adaptation. By maintaining libraries of LoRA profiles for each speaker characteristic group, new profiles can be trained and merged dynamically as new data is acquired (Nassereldine et al., 21 Jun 2024, Bao et al., 6 Jun 2025). Mixing pathological and fluent speech data (e.g., aphasia-specific) in various ratios ensures robust generalization across populations (Bao et al., 6 Jun 2025).

Auxiliary Losses for Speaker Discriminativeness:

Contrastive losses, such as NT-Xent (Emon et al., 13 Mar 2025) and speaker vs. non-target similarity constraints (Guo et al., 7 Dec 2024), are introduced to encourage speaker-discriminative embedding spaces in the encoder. Joint loss optimization with hard triplet mining supports robust identification and separation.

3. Performance Metrics, Validation, and Benchmarks

Most studies benchmark WER (word error rate, %), cpWER (concatenated permutation WER), and other specialized metrics such as multi-talker WER (mtWER), Attribution Error Rate (AER), or DNSMOS for perceptual quality.

• Prompt tuning and deep prompt insertion achieve WER reductions exceeding 60% versus zero-shot Whisper in overlapping speech (Ma et al., 2023).
• Diarization-conditioned approaches demonstrate absolute ORC-WER improvements over separation–diarization cascades (e.g., 12.9% on NOTSOFAR-1) (Polok et al., 14 Sep 2024, Polok et al., 30 Dec 2024).
• SQ-Whisper achieves state-of-the-art WERs of 14.6% (Libri2Mix Test) and 4.4% (WSJ0-2Mix Test) with data augmentation (Guo et al., 7 Dec 2024).
• PI-Whisper attains up to 13.7% relative WER improvement compared to single LoRA profiles, with measurable benefits in fairness metrics (SPD, DIR) regarding speaker group equity (Nassereldine et al., 21 Jun 2024).
• Joint ASR and role SOT training reduces mtWER by over 10% compared to SSL baselines (Xu et al., 12 Jun 2025).

4. Computational Efficiency and Scalability

Whisper extensions for target speaker ASR consistently prioritize efficiency:

• LoRA, prompt tokens, and deep modular routing ensure low parameter overhead (~1–10%) relative to full model size (Ma et al., 2023, Song et al., 7 Jun 2024).
• Quantized models (P4Q, using NF4 format) compress the base model by factors of 7×, while LoRA-based speaker adaptation recovers accuracy, yielding 15–24% WER reductions for target speakers even with quantized parameters (Zhao et al., 7 Aug 2024).
• Edge deployments (PI-Whisper, AS-ASR) leverage Whisper-tiny variants and dynamic LoRA merging to maintain real-time operation and linear scaling with resource availability (Nassereldine et al., 21 Jun 2024, Bao et al., 6 Jun 2025).

5. Applications and Generalization

Target speaker ASR systems using Whisper accommodate:

• Meeting and conversational transcription (speaker-attributed ASR, role-aware outputs) (Polok et al., 30 Dec 2024, Xu et al., 12 Jun 2025).
• Voice command execution, privacy-sensitive scenarios (hands-free, social acceptability, wearable computing) (Rekimoto, 2022).
• Clinical speech (aphasia-specific recognition, equity across disorders) (Bao et al., 6 Jun 2025).
• Linguistically diverse environments (multilingual spoken content, code-switching) (Ferraz, 2 May 2024, Emon et al., 13 Mar 2025).
• Speaker identification and diarization across languages via Whisper’s robust encoder (Emon et al., 13 Mar 2025).

Most conditioning schemes (diarization, modular prompts, profile merging) generalize to unseen speakers and new acoustic conditions. They minimize dependence on fixed speaker embeddings and reduce catastrophic forgetting when adding speakers or languages.

6. Limitations, Challenges, and Future Research

• Reliance on diarization quality is a bottleneck; diarization errors may propagate and reduce ASR accuracy in fully overlapped or ambiguous segments (Polok et al., 14 Sep 2024, Polok et al., 30 Dec 2024).
• EnroLLMent-based methods (Speaker-Querying, Sidecar separation) require reliable target speaker samples; mismatched or missing enroLLMents can degrade performance (Guo et al., 7 Dec 2024, Meng et al., 13 Jul 2024).
• Low-resource speaker/language adaptation remains nontrivial; knowledge distillation and modular adaptation are preferred, yet sufficient in-domain supervision is still required (Ferraz, 2 May 2024).
• Full system generality to multi-channel audio and multi-modal (audio-visual) configurations has yet to be demonstrated at scale; future work aims to extend diarization-conditioning and speaker-prompting to these contexts (Guo et al., 7 Dec 2024).
• Further investigation into error correction, training schedule optimization, robustness to highly variable speech, and integration with LLMs is ongoing (Ma et al., 2023).

7. Summary Table: Whisper-based Target Speaker ASR Adaptations

Approach	Conditioning Method	Parameter Overhead	Key Metric/Improvement
Prompt Tuning (soft)	Speaker embedding, prompts	~1%	60%+ WER reduction vs. baseline (Ma et al., 2023)
Diarization Conditioning	STNO probabilities, FDDT, QKb	4 bias terms minimum	12.9% absolute ORC-WER gain (Polok et al., 14 Sep 2024, Polok et al., 30 Dec 2024)
Speaker-Querying (SQ)	Trainable queries, enroLLMent	10% compute	15% WER reduction, SoTA on WSJ0-2Mix (Guo et al., 7 Dec 2024)
Modular LoRA	Per-speaker/attribute LoRA	1–10%	Efficient adaptation, fairness gains (Nassereldine et al., 21 Jun 2024, Song et al., 7 Jun 2024)
Sidecar Separation + TTI	TCN separator + identifier	1–3%	<8% WER on LibriMix target-talker (Meng et al., 13 Jul 2024)
SOT Role Tagging	Serialization, role tokens	Full fine-tune	10–15% mtWER reduction (Xu et al., 12 Jun 2025)

Approaches are selected based on data availability, required generalization, deployment constraints, and specific downstream applications. Diarization-conditioned methods are prominent for generalization to unseen speakers and minimal enroLLMent requirements, while modular low-rank adapters and profile libraries support incremental adaptation across speaker or language groups.