Target Speaker ASR: Techniques and Trends

• Target Speaker ASR is a speech recognition method that uses speaker-specific information to isolate and transcribe a designated speaker's words from overlapping or noisy inputs.
• It integrates various conditioning techniques like enrollment i-vectors, d-vectors, TitaNet embeddings, spatial cues, and prompt tuning within diverse architectures from front-end separation to integrated ASR models.
• Recent research focuses on joint optimization and adaptive conditioning strategies that balance extraction quality with recognition accuracy in complex, multi-speaker scenarios.

Target-speaker automatic speech recognition (TS-ASR) denotes ASR conditioned on speaker-specific side information so that the recognizer outputs only the words of a designated speaker from a noisy or overlapping mixture. A canonical formulation is to decode a target transcript $W_{\rm tgt}$ from acoustic frames ${\bf X}$ given a fixed-dimensional target-speaker embedding $e_{\rm tgt}$, $$\tilde W_{\rm tgt}=\arg\max_{W_{\rm tgt}}P(W_{\rm tgt}\mid e_{\rm tgt},{\bf X})$$ (Kanda et al., 2019). Across the literature, the conditioning signal may be a short enrollment utterance, a speaker embedding, speaker activity curves, or even spatial cues, and the implementation ranges from cascaded extraction-plus-ASR pipelines to encoder-integrated streaming transducers and foundation-model adaptations (Moriya et al., 2022, Ma et al., 2023, Guo et al., 2024).

1. Problem definition and task boundaries

TS-ASR is distinct from ordinary single-speaker ASR because the input may contain multiple simultaneous speakers or severe interference, and the system must suppress non-target speech at the recognition stage rather than merely improve acoustic quality. In the monaural dialogue setting, the task has been formulated as simultaneous speech recognition and speaker diarization by conditioning an acoustic model on a target embedding and decoding only the designated speaker’s words; an iterative variant alternates speaker-embedding estimation and TS-ASR decoding when oracle speaker queries are unavailable (Kanda et al., 2019).

The task is also distinct from generic speaker-attributed ASR (SA-ASR). In comparative studies on AliMeeting, TS-ASR is evaluated alongside frame-level diarization with serialized output training (FD-SOT) and word-level diarization with SOT (WD-SOT). The stated advantage of TS-ASR in that setting is that it avoids explicit frame- or word-timestamp alignment by first producing a speaker-specific signal and then running a single-speaker recognizer on that signal (Yu et al., 2022). Multichannel SA-ASR extends the same idea with multichannel fusion and neural beamforming, yielding MC-TS-ASR as one branch of the broader SA-ASR design space (Shi et al., 2022).

A further boundary appears in practical overlap transcription. A 2025 streaming system combines a speaker-independent model for standard operation with a speaker-conditioned model used only on detected overlap segments, so the target-speaker mechanism is activated selectively rather than continuously (Pražák et al., 25 Jun 2025). This shows that TS-ASR can be embedded within continuous ASR services without requiring the entire pipeline to operate in target-conditioned mode at all times.

2. Target specification and speaker representation

Most TS-ASR systems rely on enrollment-based speaker conditioning. Early LF-MMI systems use a 100-dimensional i-vector extracted from a short enrollment utterance and repeated across frames in a CNN-TDNN-LSTM acoustic model, sometimes with dual target and interference branches (Kanda et al., 2019, Kanda et al., 2019). Meeting-oriented TS-ASR replaces i-vectors with d-vectors extracted by a Res2Net speaker-embedding network, and conditions the target-speaker separation front-end by Feature-wise Linear Modulation (FiLM), $\gamma=\mathrm{MLP}_1(v_n)$, $\beta=\mathrm{MLP}_2(v_n)$, followed by $\gamma\odot h+\beta$ at each layer (Yu et al., 2022). Single-channel Conformer-based TS-ASR uses TitaNet to produce a 192-dimensional embedding that is linearly projected and injected into both masking and ASR modules (Zhang et al., 2023). Practical streaming overlap transcription likewise uses TitaNet-S embeddings and FiLM inside a wav2vec 2.0-based speaker-conditioned model (Pražák et al., 25 Jun 2025).

Foundation-model adaptations preserve the same basic conditioning principle while changing the injection mechanism. Prompt-tuned Whisper prepends a projected x-vector and learned encoder/decoder prompts to a frozen Whisper backbone (Ma et al., 2023). SQ-Whisper instead learns trainable speaker queries that interact with enrollment features and mixture features to produce speaker prompts inserted into both the encoder and decoder (Guo et al., 2024).

Several papers examine alternatives to conventional enrollment embeddings. A unified cross-task study reports that, for TS-ASR, a one-hot speaker code outperforms enrollment-based embeddings in speaker-closed conditions, that speaker verification performance is “somewhat unrelated” to TS task performance, and that the optimal embedding depends on the input mixture (Ashihara et al., 2024). Another line of work replaces speaker embeddings with geometry: a 3D spatial feature computed from azimuth, elevation, and distance is concatenated with spectral features and used either in a pipelined separation-plus-ASR system or in an “All-In-One” Conformer ASR without explicit separation (Shao et al., 2021). More recently, self-speaker adaptation removes explicit speaker queries altogether and injects speaker-specific kernels derived from speaker-wise speech activity curves into a streaming multi-talker encoder (Wang et al., 27 Jun 2025).

Conditioning signal	Representative use	Characteristic mechanism
Enrollment i-vector	(Kanda et al., 2019, Kanda et al., 2019)	Frame-level conditioning of LF-MMI acoustic models
d-vector / Res2Net embedding	(Yu et al., 2022, Shi et al., 2022)	FiLM-conditioned separator or beamforming front-end
TitaNet embedding	(Zhang et al., 2023, Pražák et al., 25 Jun 2025)	Conformer masking or FiLM-conditioned SC decoding
Prompt/query conditioning	(Ma et al., 2023, Guo et al., 2024)	Soft prompts or speaker-query prompts in Whisper
Spatial or activity cues	(Shao et al., 2021, Wang et al., 27 Jun 2025)	3D spatial feature or speaker-wise activity-driven adaptation

3. Architectural paradigms

A major architectural axis is whether target-speaker filtering is implemented as an explicit front-end or absorbed into the recognizer. In “Speaker Reinforcement Using Target Source Extraction for Robust Automatic Speech Recognition,” the front-end is a single-channel, time-domain SpeakerBeam network that estimates a target waveform $s'(n)$ from a noisy mixture $y(n)$ and enrollment utterance $r(n)$, after which ASR is run with no acoustic-model retraining (Zorila et al., 2022). In AliMeeting TS-ASR, the target-speaker separation module may be Conformer-based or CRN-based, and its output waveform is passed to a Conformer encoder-decoder ASR back-end under joint training (Yu et al., 2022).

A second paradigm is end-to-end joint masking or beamforming with ASR. CONF-TSASR comprises a TitaNet speaker encoder, an 18-layer Conformer MaskNet that predicts a real element-wise mask over time-frequency features, and an 18-layer Conformer-CTC recognizer, optimized jointly for target-speaker transcription (Zhang et al., 2023). In multichannel meetings, MC-TS-ASR combines an embedding module and a beamforming module, known jointly as EaBNet, to produce a single enhanced STFT ${\bf X}$0 before Conformer recognition (Shi et al., 2022).

A third paradigm integrates speaker conditioning directly inside the ASR encoder. TS-RNNT computes a target embedding from enrollment speech and applies feature-wise gating at the first Conformer-RNNT encoder layer, ${\bf X}$1, thereby incorporating target speech extraction functionality into a streaming transducer without an external front-end (Moriya et al., 2022). META-CAT uses frame-wise speaker probabilities from a frozen diarizer to weight ASR encoder activations and concatenate speaker-informed sub-embeddings before RNN-T decoding (Wang et al., 2024). The Czech broadcast system of 2025 keeps a speaker-independent backbone for ordinary regions and “fans out” overlap windows through a speaker-conditioned model using the ${\bf X}$2 most recent speaker embeddings, which are decoded in parallel (Pražák et al., 25 Jun 2025).

A fourth paradigm adapts large pretrained encoder-decoder models. Prompt-tuned Whisper freezes the backbone and optimizes only a speaker projection and soft prompts, while retaining Whisper’s existing decoding format, inverse text normalization, and timestamp tagging (Ma et al., 2023). SQ-Whisper fine-tunes Whisper with an SQ-Former that learns speaker prompts from enrollment speech and mixture representations (Guo et al., 2024). This foundation-model branch places TS-ASR within the broader trend of parameter-efficient adaptation rather than task-specific model construction.

The 3D spatial-feature work adds a related contrast between pipelined and “All-In-One” systems. The pipelined model uses a separation module followed by a Conformer ASR module, whereas the “All-In-One” model removes explicit separation and relies wholly on the 3D spatial feature plus log-Mel filterbanks (Shao et al., 2021). This provides an explicit example of TS-ASR moving from signal-level separation toward representation-level target guidance.

4. Training objectives, auxiliary criteria, and inference control

The training objectives used in TS-ASR are heterogeneous but usually combine speaker selection with transcription supervision. In the LF-MMI line, the target branch is trained with a sequence-discriminative objective conditioned on a target embedding, and an auxiliary branch may model the interfering speaker. The combined loss is ${\bf X}$3, with ${\bf X}$4 as the auxiliary-loss weight (Kanda et al., 2019). “Auxiliary Interference Speaker Loss for Target-Speaker Speech Recognition” shows that explicitly maximizing interference-speaker ASR accuracy during training regularizes the shared encoder and reduces target-speaker WER (Kanda et al., 2019).

Joint separation-and-recognition criteria are central in meeting and end-to-end TS-ASR. Both SC-TS-ASR and MC-TS-ASR on AliMeeting use a two-term loss,

${\bf X}$5

with negative SI-SNR for separation and joint Attention+CTC for ASR; a typical setting is ${\bf X}$6 during joint pre-training, followed by ASR-only fine-tuning for MC-TS-ASR (Shi et al., 2022). The single-channel AliMeeting TS-ASR system uses

${\bf X}$7

with ${\bf X}$8 based on SI-SNR and ${\bf X}$9 as cross-entropy or joint CTC+attention, and reports that joint fine-tuning is materially better than separated training (Yu et al., 2022).

Conformer-based single-channel TS-ASR employs a non-autoregressive CTC objective together with a scale-invariant spectrogram reconstruction loss,

$e_{\rm tgt}$0

where $e_{\rm tgt}$1 compares the up-sampled masked spectrogram with the oracle target spectrogram (Zhang et al., 2023). SQ-Whisper combines cross-entropy on decoder outputs with a speaker contrastive loss that pulls pooled prompt vectors toward pooled enrollment vectors and away from negatives in the same batch (Guo et al., 2024). Streaming TS-RNNT and self-speaker adaptation instead optimize the standard transducer objective without an extra target-speech extraction loss, training the target focus implicitly through conditioning (Moriya et al., 2022, Wang et al., 27 Jun 2025).

Inference-time control is also a recurring theme. The speaker-reinforcement method addresses the mismatch between extraction quality and recognition robustness by remixing the extracted signal with a scaled copy of the original mixture,

$e_{\rm tgt}$2

with $e_{\rm tgt}$3 chosen from a target power ratio $e_{\rm tgt}$4 (Zorila et al., 2022). The paper reports that this “wet/dry” reinforcement alleviates artifacts without retraining the acoustic model. In the TS-SUPERB benchmark, multi-task optimization across TSE and TS-ASR further illustrates that better extraction metrics do not automatically guarantee lower WER unless the SSL encoder is fine-tuned jointly (Peng et al., 10 May 2025).

5. Benchmarks, corpora, and empirical performance

Empirical evaluation spans monaural dialogues, far-field meetings, synthetic mixture corpora, noisy speech benchmarks, and streaming broadcast or conversational audio. In heavily overlapped monaural dialogue recordings with overlap ratio over 20%, iterative TS-ASR with estimated i-vectors reaches a WER of 40.03% after three iterations, only 2.1% absolute worse than TS-ASR with oracle speaker embeddings, and achieves a DER of 24.63%, better than a conventional clustering-based i-vector diarization baseline (Kanda et al., 2019). On two-speaker LibriSpeech-based mixtures, adding the auxiliary interference speaker loss reduces WER from 18.06% to 16.87%, a 6.6% relative reduction over a strong TS-AM baseline (Kanda et al., 2019).

In noisy single-channel CHiME-4, pure target-speaker enhancement can be harmful to recognition because of artifacts, but speaker reinforcement changes that trade-off. Without acoustic-model retraining, unprocessed WER of approximately 13.6% on simulated and 12.3% on real single-channel CHiME-4 drops to approximately 10.4% and 9.6% at $e_{\rm tgt}$5 dB, corresponding to 23% and 25% relative reductions (Zorila et al., 2022).

Meeting corpora show the value of TS-ASR for speaker-attributed recognition. On AliMeeting, FD-SOT yields an average SD-CER of 41.2%, WD-SOT 36.8%, TS-ASR with a Conformer front-end 34.7%, and TS-ASR with a CRN front-end 34.4%; the latter corresponds to a 16.5% relative average SD-CER reduction over FD-SOT (Yu et al., 2022). Multichannel TS-ASR further reduces average SD-CER from 34.4% for SC-TS-ASR to 28.3% for MC-TS-ASR, approximately a 17.7% relative reduction (Shi et al., 2022). With explicit 3D spatial cues, the proposed feature yields about 31% relative CERR over 1D directional information, while the “All-In-One” model approaches pipelined accuracy and halves inference time (Shao et al., 2021).

Single-channel benchmark mixtures remain a dominant test bed. CONF-TSASR reports target-speaker WERs of 4.2% on WSJ0-2mix-extr, 12.4% on WSJ0-3mix-extr, 4.2% on LibriSpeech2Mix, and 7.6% on LibriSpeech3Mix (Zhang et al., 2023). Prompt-tuned Whisper reaches performance comparable to state-of-the-art full-training approaches while tuning about 1% or less of model parameters, and preserves natural text formatting and timestamps (Ma et al., 2023). SQ-Whisper reports 20.1% on Libri2Mix with full fine-tuning, 14.6% on Libri2Mix Test with data augmentation, and 4.4% on WSJ0-2Mix Test with speed perturbation, while also improving over other adaptation methods on AMI (Guo et al., 2024).

The TS-SUPERB benchmark makes the difficulty of TS-ASR explicit: on Libri2Mix-max clean, WavLM Large yields 17.97% WER with LM and 22.62% without LM, whereas the paper notes that single-speaker ASR baselines on LibriSpeech are around 3.4% WER (Peng et al., 10 May 2025). This benchmark also reports that frozen-SSL joint TSE+TS-ASR improves extraction metrics but degrades WER relative to TS-ASR alone, while fine-tuning SSL recovers both sides (Peng et al., 10 May 2025).

Streaming and practical-overlap systems shift attention from accuracy alone to compute and latency. TS-RNNT attains an average offline CER of 15.8% on CSJ two-speaker mixtures with CHiME-4 noise, slightly better than a TSE+RNNT cascade at 16.5%, while keeping the same RTF of 0.40 instead of 1.22 for the cascade (Moriya et al., 2022). On ten 1-hour Czech TV debates with about 16% overlap, a lightweight SI+SC system reduces overlap WER from 68.00% to 35.78% with a relative hardware factor of 1.44 (Pražák et al., 25 Jun 2025). Query-less self-speaker adaptation reports cpWER $e_{\rm tgt}$6 on LibriSpeechMix 1/2/3-mix in the offline setting and competitive low-latency streaming results, including 26.21% cpWER on CH109 at 1120 ms latency (Wang et al., 27 Jun 2025).

Setting	Reported result	Paper
CHiME-4 single-channel, no AM retraining	WER $e_{\rm tgt}$7 at $e_{\rm tgt}$8 dB	(Zorila et al., 2022)
AliMeeting single-channel SA-ASR	Avg. SD-CER $e_{\rm tgt}$9 with TS-ASR (CRN FE)	(Yu et al., 2022)
AliMeeting multichannel SA-ASR	Avg. SD-CER $$\tilde W_{\rm tgt}=\arg\max_{W_{\rm tgt}}P(W_{\rm tgt}\mid e_{\rm tgt},{\bf X})$$0 from SC-TS-ASR to MC-TS-ASR	(Shi et al., 2022)
WSJ0-2mix-extr	TS-WER 4.2% with CONF-TSASR	(Zhang et al., 2023)
Libri2Mix Test	WER 14.6% with SQ-Whisper + augmentation	(Guo et al., 2024)
Czech TV debates, streaming overlap	Overlap WER $$\tilde W_{\rm tgt}=\arg\max_{W_{\rm tgt}}P(W_{\rm tgt}\mid e_{\rm tgt},{\bf X})$$1 with HW factor 1.44	(Pražák et al., 25 Jun 2025)

6. Conceptual issues, common misconceptions, and current directions

A recurrent misconception is to equate TS-ASR with speech enhancement. The CHiME-4 reinforcement study shows the opposite failure mode: pure speaker extraction at $$\tilde W_{\rm tgt}=\arg\max_{W_{\rm tgt}}P(W_{\rm tgt}\mid e_{\rm tgt},{\bf X})$$2 degrades WER relative to unprocessed input, while remixing the extracted signal with the original mixture improves recognition markedly (Zorila et al., 2022). This indicates that target extraction and ASR robustness are related but not identical objectives.

A second misconception is that TS-ASR is merely diarization followed by ordinary ASR. Comparative meeting studies argue that TS-ASR reduces dependence on timestamp alignment by producing speaker-specific waveforms before recognition (Yu et al., 2022). Conversely, the monaural dialogue work shows that TS-ASR can also yield diarization as a by-product through iterative embedding estimation and speaker-conditioned decoding (Kanda et al., 2019). The relationship is therefore bidirectional but not reducible to a single pipeline template.

A third issue concerns speaker representation itself. Cross-task analysis finds that ASV equal error rate does not reliably predict TS-ASR WER, that one-hot speaker codes can surpass enrollment-derived embeddings in speaker-closed conditions, and that the optimal embedding depends on the mixture (Ashihara et al., 2024). TS-SUPERB generalizes this point by showing that performance in target-speaker scenarios cannot be inferred from related single-speaker tasks and that joint optimization across target-speaker tasks can be beneficial (Peng et al., 10 May 2025).

Recent work points toward broader integration rather than isolated target extraction modules. One direction extends TS-ASR to joint target and non-target transcription by recursively generating text tokens together with target/non-target markers (Masumura et al., 2023). Another direction uses Meta-Cat to support both MS-ASR and TS-ASR within one end-to-end architecture (Wang et al., 2024). Self-speaker adaptation eliminates explicit speaker queries in streaming multi-talker ASR (Wang et al., 27 Jun 2025), while a later large-audio-language-model approach introduces chain-of-thought supervision and reinforcement learning, reporting an average WER of 8.33% on LibriSpeech, Libri2Mix, and Libri3Mix after CoT and RL training (Zhang et al., 19 Sep 2025). This suggests a shift from explicit front-end separation toward speaker-informed reasoning and speaker-conditioned decoding inside unified end-to-end models.