Target-Speaker ASR

• Target-Speaker ASR is an automatic speech recognition method that extracts and transcribes only a designated speaker’s voice from mixed, overlapping audio signals.
• It employs a two-stage streaming pipeline combining speaker-independent processing with speaker-conditioned rescoring using FiLM-based speaker embeddings.
• The approach significantly reduces word error rates in overlap scenarios while maintaining low latency and computational efficiency for real-world applications.

Target-Speaker Automatic Speech Recognition (TS-ASR) is an ASR paradigm that extracts and transcribes only the utterances of a designated speaker (the "target") from single-channel or multi-channel mixtures containing multiple, often overlapping, speakers. Given a short enrollment utterance from the target speaker or corresponding diarization labels, TS-ASR systems condition the transcription on speaker-specific cues to provide robust recognition in overlapped and real-world conversational scenarios. This approach addresses the fundamental limitations of conventional speaker-independent ASR in high-overlap or multi-talker environments and is increasingly deployed in broadcast media, meetings, and streaming services (Pražák et al., 25 Jun 2025).

1. Core System Architecture

Modern TS-ASR frameworks are most commonly structured around a two-stage streaming pipeline integrating both speaker-independent (SI) and speaker-conditioned (SC) models (Pražák et al., 25 Jun 2025). The canonical workflow is as follows:

1. Speaker-Independent (SI) Model: The incoming audio is processed in fixed windows (e.g., 15 s, with a 3 s look-back) by a standard streaming ASR such as wav2vec 2.0, equipped with a speaker-change detection head. This SI ASR handles single-speaker regions and performs initial feature extraction shared with downstream modules.
2. Overlap Detection: A lightweight overlap detector, implemented as a small binary classifier head (≤769 parameters), analyzes frame-level representations to flag multi-speaker/overlap regions (Pražák et al., 25 Jun 2025). The classifier is trained using binary cross-entropy on synthetic multi-talker mixtures and achieves F₁ > 85% with negligible runtime overhead (<1% compute).
3. Speaker-Conditioned (SC) Model: For detected overlap segments, the audio is re-scored with an SC variant of the ASR, typically the same base model augmented through a feature-wise linear modulation (FiLM) mechanism that incorporates the target speaker embedding (Pražák et al., 25 Jun 2025, Zhang et al., 2023, Shi et al., 2022). Each overlap is processed once per candidate speaker, making the system scalable to N concurrent talkers.
4. Speaker Embedding Extraction: Speaker embeddings (d-vectors) are derived from enrollment utterances using a pretrained or domain-adapted network (e.g., TitaNet-S) (Pražák et al., 25 Jun 2025). The streaming system tracks the most recent embeddings for active speakers, typically using only the embedding from the last non-overlap segment for efficiency.
5. Dynamic Decoder Management: A set of N streaming decoders is dynamically allocated, each corresponding to a tracked speaker. Decoders maintain individual CTC prefix states, and output streams are sentence-wise merged and post-punctuated for final transcriptions.

This modular design enables high-accuracy target-speaker transcription while preserving low latency and computational efficiency for streaming inference.

2. Speaker Conditioning and FiLM Integration

Conditioning the ASR model on speaker identity is implemented via Feature-wise Linear Modulation (FiLM) or equivalent affine conditioning at the input to the encoder stack (Pražák et al., 25 Jun 2025, Zhang et al., 2023). Specifically, given a speaker embedding $e\in\mathbb{R}^d$, FiLM computes per-feature scaling $\gamma(e)$ and bias $\beta(e)$:

$\mathrm{FiLM}(h) = \gamma(e) \odot h + \beta(e)$

where $h$ denotes the latent features after the shared CNN front end and before the first (or multiple) transformer layers of the ASR encoder.

This mechanism serves to bias the internal representations toward spectral–temporal patterns characteristic of the target speaker, effectively suppressing interference from non-target speakers in overlapped speech. The FiLM parameters (scalers and shifters) are typically small and easily adapted during training (Zhang et al., 2023).

The SC model is trained on synthetically mixed multi-talker data: for each batch, the model is exposed to both genuine speaker embeddings and "non-speaker" distractors, penalized using a normalized CTC loss to prevent dominance by absent-speaker runs (Pražák et al., 25 Jun 2025). This protocol robustifies the model against both speaker confusion and false positives in overlap.

3. Overlap Detection and Dynamic Speaker Tracking

TS-ASR requires robust detection of overlapping speech regions, as these are where standard SI models fail catastrophically (e.g., SI WER 68% on Czech TV overlap segments) (Pražák et al., 25 Jun 2025). The overlap detector is a compact classifier operating on the shared SI encoder's output, trained using per-frame binary supervision derived from synthetic mixes and refined with post-processing heuristics:

• Segments <1 s are relabeled to handle short-lived overlaps and avoid annotation artifacts.
• A collar-aware loss ensures boundary robustness.

Dynamic speaker tracking is facilitated by a streaming diarization module (SI model's speaker-change detector + TitaNet embeddings) (Pražák et al., 25 Jun 2025). Upon overlap, the system collects the most recent N speaker tracks and applies SC rescoring for each. Decoders are resynchronized with speaker entry and exit, enabling real-time adaptation to conversational turn-taking with minimal compute overhead.

4. Computational Efficiency and Scaling

The design achieves high computational efficiency:

• Overlap detection incurs negligible overhead (<1% of base compute) due to parameter sharing with the SI model (Pražák et al., 25 Jun 2025).
• Each additional active speaker in an overlap segment increases compute by a single forward pass of only the transformer stack (CNN computations are cached).
• For N=2,3,4 active speakers, total load is 1.29×, 1.40×, and 1.44× the SI-only baseline, respectively (Pražák et al., 25 Jun 2025). Further increases show diminishing WER improvements as >4-way overlap is rare in practice.

The approach supports uninterrupted streaming operation: all modules operate on 15 s sliding windows, induce no extra latency, and preserve the single-speaker performance in non-overlap segments.

5. Quantitative Results and Performance

TS-ASR architectures as described achieve major WER reductions in multi-talker broadcast scenarios:

System	Overall WER (%)	Single-Speaker WER (%)	Overlap WER (%)
SI-only baseline	19.80	3.74	68.00
SI + SC, N=4 (TS-ASR)	11.75	3.74	35.78

On real TV debates with 16% overlap, the TS-ASR approach more than halves the error rate on overlap segments, from 68% to 36%, with only a 40% increase in compute (Pražák et al., 25 Jun 2025). There is no degradation in single-speaker regions, and the overall system remains fully streaming-compatible.

Similar performance gains are observed across other domains and architectures, such as Conformer-based masking (Zhang et al., 2023), SSL-based joint extraction/recognition (Peng et al., 10 May 2025), and prompt-tuned Whisper models using diarization masks instead of embeddings (Polok et al., 2024, Polok et al., 2024).

6. Limitations, Open Challenges, and Extensions

TS-ASR's principal limitations include:

• Speaker embedding quality: Noisy or misaligned embeddings from the diarization module can impair SC model accuracy, propagating diarization errors to the ASR (Pražák et al., 25 Jun 2025).
• Finite-N constraint: The system can only process up to N tracked speakers in overlap. In settings with rare, higher-order overlap, coverage degrades.
• Heuristic overlap handling: Very short overlaps (<1 s) pose challenges for both the overlap detector and the underlying model, possibly requiring higher time resolution or loss modifications.
• Generalization: Transfer to high-SNR, polyphonic, or heavily reverberant conditions may require additional separation front-ends or spatial features, especially in multi-channel settings (Shao et al., 2023, Shi et al., 2022).

Potential extensions include:

• Joint SI+SC training for improved transfer and end-to-end optimization.
• Incorporating small streaming separation modules for extremely dense overlap.
• Adaptive N: Scaling the number of concurrently processed speakers as a function of overlap complexity or model confidence.

7. Impact, Integration with Foundation Models, and Future Directions

TS-ASR is now a foundational building block for robust meeting and broadcast transcription, streaming ASR, and personalized speech services. With the integration of parameter-efficient fine-tuning strategies such as prompt tuning and frame-wise diarization-based conditioning, large foundation models (e.g., Whisper) can be converted to TS-ASR with minimal added parameters and no loss in single-speaker accuracy (Ma et al., 2023, Polok et al., 2024, Polok et al., 2024). Frame-level conditioning (via diarization outputs or speaker masks) is often preferable to direct embedding integration, simplifying the representation space and enhancing generalization to unseen speakers.

The trend in recent research is toward architectures that natively support TS-ASR alongside multi-speaker ASR, diarization, and speaker-attributed transcription, allowing a single model to provide flexible functionality via dynamic speaker prompts or masks (Wang et al., 2024, Peng et al., 10 May 2025, Wang et al., 27 Jun 2025).

Remaining open questions concern efficient multi-speaker decoding without linear scaling in K, robust handling of diarization noise in the absence of ground truth, and the integration of multi-modal and spatial cues for far-field and real-world deployments.