Target-Speaker Diarization

• Target-Speaker Diarization is a framework that conditions speaker detection on target speaker embeddings, enabling precise frame-wise activity estimation and native overlap handling.
• It leverages speaker representations like i-vectors, x-vectors, and multimodal prompts to refine initial diarization hypotheses, surpassing traditional clustering methods.
• Recent advancements include transformer-based, sequence-to-sequence, and online approaches that dynamically update speaker profiles for robust performance in complex multi-speaker scenarios.

Target-speaker diarization is a family of diarization methods that reformulates the classical “who spoke when” problem as target-conditioned activity estimation: given a multi-speaker recording and one or more target-speaker representations, the system predicts for each target speaker whether that speaker is active at each frame. In the supplied literature, the most recurrent formulation is target-speaker voice activity detection (TS-VAD), which directly predicts an activity of each speaker on each time frame and is designed to address overlapping speech more effectively than clustering-only pipelines (Medennikov et al., 2020). Subsequent work generalized the paradigm to sequence-to-sequence prediction, variable-length speaker axes, enrollment-conditioned end-to-end neural diarization, audio-visual conditioning, online target-speaker tracking, and joint diarization-separation or diarization-ASR systems (Cheng et al., 2022, Wang et al., 2022, Wang et al., 2023). A related extension, target speech diarization, further broadens the conditioning signal from speaker identity to semantic or multimodal prompts, replacing “who spoke when” with “when target event occurs” (Jiang et al., 2024).

1. Task definition and formalization

In its canonical form, target-speaker diarization assumes an acoustic sequence and a set of target-speaker profiles, then estimates a frame-wise posterior for each target. One representative formulation writes, for speaker $s$ and frame $t$, the posterior $P(y_s(t)=1 \mid X, I)$, where $X$ is the acoustic feature sequence and $I$ is the set of per-speaker embeddings (Medennikov et al., 2020). A later end-to-end formulation states the same objective as estimating, for each speaker $i$, the frame-wise posterior probability of that speaker’s activity conditioned on both the acoustic input and the speaker’s enrollment embedding, with outputs arranged as a multi-label speaker-by-time tensor rather than a single speaker label per frame (Chen et al., 2023).

This target-conditioned view differs from conventional clustering-based diarization in an important way. Instead of discovering speaker structure only by segmenting and clustering embeddings, the model is explicitly told which speaker to detect through an auxiliary representation such as an i-vector, x-vector, d-vector, z-vector, lip anchor, enrollment embedding, or attractor. This is why the literature repeatedly treats target-speaker diarization as especially suitable for strongly overlapping speech: multiple target posteriors can be high at the same frame, so overlap is handled natively rather than by post hoc correction (Medennikov et al., 2020, Cheng et al., 2022).

The same formal pattern has also been extended beyond speaker identity. “Target Speech Diarization with Multimodal Prompts” defines target speech diarization as detecting “when target event occurs” according to the semantic characteristics of speech, and proposes prompts including semantic language description, pre-enrolled speech, pre-registered face image, and audio-language logical prompts (Jiang et al., 2024). This suggests a broader interpretation of target-speaker diarization as one instance of prompt-conditioned temporal event detection in conversational audio.

2. Canonical TS-VAD pipeline

The classical TS-VAD workflow is two-stage. A conventional diarization system first produces rough speaker segments and corresponding speaker profiles; TS-VAD then refines those hypotheses by predicting each target speaker’s activity on each frame (Cheng et al., 2024). In the original 2020 formulation, the model takes conventional speech features such as MFCCs together with i-vectors for each speaker as inputs, and a set of binary classification output layers produces activities of each speaker (Medennikov et al., 2020).

Architecturally, the original single-channel TS-VAD model uses a speaker-detection block followed by a fusion BLSTMP. For $S=4$ speakers, each speaker stream concatenates frame features with that speaker’s i-vector, processes the result through a 2-layer BLSTMP, and then concatenates all per-speaker hidden states into a fusion BLSTMP that produces per-speaker speech posteriors (Medennikov et al., 2020). The training objective is binary cross-entropy over speakers and frames:

$$L = - \sum_{s=1}^S \sum_{t=1}^T \left[y_s(t)\log P(y_s(t)=1\mid X,I) + (1-y_s(t))\log P(y_s(t)=0\mid X,I)\right].$$

This formulation is directly multi-label, so it does not force a single active speaker per frame (Medennikov et al., 2020).

Several practical mechanisms became standard around this design. One is iterative profile refinement: speaker i-vectors can be estimated iteratively, starting with a strong clustering-based diarization, then recomputed from improved TS-VAD outputs (Medennikov et al., 2020). Another is post-processing. The original work investigated both median-threshold smoothing and an HMM+Viterbi decoder whose states represent silence, single-speaker activity, and exactly two-speaker overlap (Medennikov et al., 2020). A further extension to TS-VAD-MC applied a simple attention mechanism across multiple microphones, aggregating channel-specific hidden representations before the final fusion block (Medennikov et al., 2020).

Empirically, this design was motivated by failure modes of clustering on realistic overlap-heavy recordings. On CHiME-6 unsegmented data, the TS-VAD approach outperformed the baseline x-vector-based system by more than 30% Diarization Error Rate abs, and the final fusion of six TS-VAD variants with Viterbi decoding reached 32.84% DER on dev and 36.02% DER on eval (Medennikov et al., 2020). Those results established the central claim that direct frame-level target-speaker activity estimation can outperform purely clustering-based diarization in difficult multi-speaker conditions.

3. Scaling the paradigm: sequence-to-sequence, transformers, and dynamic speaker axes

A major limitation of early TS-VAD was the entanglement of time and speaker dimensions, which constrained speaker capacity and output resolution. Seq2Seq-TSVAD replaced the conventional encoder-only TS-VAD architecture by a true sequence-to-sequence model that factorizes time and speaker axes, reducing memory from $\mathcal O(T\cdot N)$ to $\mathcal O(T)+\mathcal O(N)$ and enabling decoding of up to $t$0 speakers in a single 12 GB GPU (Cheng et al., 2022). The same model made temporal resolution a parameter of the final output layer, with reported experiments at $t$1 and $t$2; on VoxConverse test, the best single Seq2Seq-TSVAD with $t$3 and $t$4 achieved 4.55% DER, while on DIHARD-III Track 1 it achieved 10.77% DER (Cheng et al., 2022).

Transformerization addressed a different constraint: order sensitivity and variable speaker count. “Target Speaker Voice Activity Detection with Transformers and Its Integration with End-to-End Neural Diarization” introduced a joint speaker detection module that alternates Transformer layers along the speaker axis and sequential layers along the time axis, with no positional encoding on the speaker axis so that the model output is insensitive to the order of the speaker profiles (Wang et al., 2022). On VoxConverse, using transformers for cross-speaker modeling reduced DER by 11.3% relative over the original TS-VAD, reaching 4.57%, and the same speaker-conditioned matching module reduced CALLHOME DER to 11.18% when integrated into EEND-EDA (Wang et al., 2022).

A complementary line of work removed the fixed-speaker assumption by making the detector itself operate on a dynamic speaker axis. MTEAD combines a z-vector extractor with a detector composed of a Feature Mixer and stacked Time-Speaker Contextualizers; the across-speaker BiLSTM runs along an axis whose length varies from session to session, so the model can handle any number of speakers at inference time (Cheng et al., 2022). On CALLHOME, MTEAD(z-vector) achieved 14.31% DER and yielded relative reductions over several baselines (Cheng et al., 2022).

AED-EEND addressed target conditioning from the end-to-end diarization side. It calculates attractors from enrollment embeddings, including three special embeddings for non-speech, single-speaker speech, and overlap speech, and replaces the LSTM attractor decoder of EEND-EDA with an attention-based model (Chen et al., 2023). With only $t$5 of enrollment data, AED-EEND reached 7.75% DER on 2-speaker CALLHOME and 14.22% overall in the flexible-speaker setting (Chen et al., 2023). Taken together, these models show that target-speaker diarization is no longer confined to the original fixed-profile, BLSTM-only TS-VAD template.

4. Modalities, prompts, and online operation

The addition of visual information produced a distinct audio-visual branch of target-speaker diarization. MIMO-TSVAD takes audio-visual input and uses either the speaker’s acoustic footprint or lip track to conduct audio-based, video-based, and audio-visual speaker diarization in a unified sequence-to-sequence framework (Cheng et al., 2024). Its architecture combines an audio extractor, a lip-feature extractor, a shared multi-modal Conformer encoder, and a tri-branch decoder with audio, video, and mixed heads. The design explicitly addresses the modality-missing problem through model-level masking, data-level masking of anchors, and a Speaker Alignment step based on the Hungarian algorithm when lip anchors are completely missing (Cheng et al., 2024). Reported DERs are 4.18% on VoxConverse, 10.10% on DIHARD-III, and 8.15% on MISP 2022 (Cheng et al., 2024).

The same conditioning logic has been generalized from audio-visual identity cues to prompt-based control. MM-TSD proposes a Multimodal Target Speech Diarization framework that accommodates semantic language description, pre-enrolled speech, pre-registered face image, and audio-language logical prompts, and includes a voice-face aligner module that projects human voice and face representation into a shared space (Jiang et al., 2024). The framework is also described as versatile enough to perform speaker diarization and overlap speech detection using task-specific prompts (Jiang et al., 2024). Although this is formally target speech diarization rather than target-speaker diarization, it extends the same prompt-conditioned temporal detection paradigm.

Online operation required a further change: the target profiles themselves had to be generated and updated during inference. Online TS-VAD used a ResNet-based front end to extract frame-level speaker embeddings block by block, then updated target-speaker embeddings through a Target Speaker Embedding Buffer while predicting frame-wise presence probabilities (Wang et al., 2022). With oracle VAD on AliMeeting and a block size of $t$6 with $t$7 shift, the reported online single-channel TS-VAD reached 8.14% DER on Eval and 11.42% DER on Test (Wang et al., 2022). The later “End-to-end Online Speaker Diarization with Target Speaker Tracking” removed the clustering prerequisite entirely by self-generating embeddings and updating them with either a buffer-based or accumulate-sum rule; it reported 13.31% DER on DIHARD III with 0.8 s latency and 4.50% DER for the multi-channel version on AliMeeting Test (Wang et al., 2023). A recurrent misconception is therefore that target-speaker diarization must be offline and initialization-heavy; the online literature shows that self-generated, incrementally updated profiles are viable, although often at the cost of additional thresholds, oracle or separate VAD, and predefined maximum speaker count (Wang et al., 2022, Wang et al., 2023).

5. Joint diarization with separation, extraction, and ASR

Because target-speaker diarization already estimates speaker-conditioned activity, it is a natural control signal for source separation and target-speaker extraction. TS-SEP extends TS-VAD by replacing the final frame-level speaker activity estimation layer with a network that produces speaker activity estimates at a time-frequency resolution; the resulting masks can be used directly for source extraction or as initialization for beamforming (Boeddeker et al., 2023). On LibriCSS full 10 min sessions, the best reported system, TS-SEP $t$8 GSS (TF-init) $t$9 WavLM, achieved 5.06% cpWER and 3.26% DI-cpWER, with DER 14.7% (Boeddeker et al., 2023).

USED makes the integration even tighter by sharing a TCN backbone for speaker extraction and diarization and adding an Embedding Assignment Module, a Scenario-Aware Differentiated extraction loss, and a Multi-Task Interaction module (Ao et al., 2023). On LibriMix max mode, USED-F reported 4.75% DER and 12.70 dB SI-SDRi; on CALLHOME, with 452 h pre-training on LibriMix, USED-F achieved 13.16% DER (Ao et al., 2023). The paper explicitly frames this as a response to output inconsistency and scenario mismatch between diarization and extraction (Ao et al., 2023).

Continuous Target Speech Extraction pushes the same idea into long recordings with target absence. Its A-TSVAD directly generates timestamps of the target speaker rather than merely refining existing diarization, and the best integration strategy is a sequential cascade in which A-TSVAD first segments the target-active regions and pBSRNN then performs extraction (Zhao et al., 2024). In the reported comparison, Cascade 1 achieved 26.5 DER, 16.2 JER, 41.3 INT, 10.4 SI-SNR, and 2.91 PESQ (Zhao et al., 2024).

Target-speaker conditioning has also been used to couple diarization with recognition. The 2019 TS-ASR framework alternates between estimating speaker embeddings and decoding each segment with a target-speaker acoustic model, producing diarization as a by-product of frame-level alignments (Kanda et al., 2019). On two-speaker CSJ dialogue mixtures, three EM-style iterations reduced total WER to 40.03%, only 2.07% absolute behind the oracle TS-AM, while DER improved to 24.63% (Kanda et al., 2019). More recently, DiCoW replaced speaker embeddings with diarization outputs as conditioning information for target-speaker ASR, introducing frame-level diarization-dependent transformations and query-key biasing inside Whisper (Polok et al., 2024). On AMI-sdm with oracle diarization, DiCoW reported 17.2/16.5 cpWER/ORC-WER, and on LibriCSS it reported 5.6/5.5 (Polok et al., 2024). This progression indicates that target-speaker diarization is not only an endpoint but also a latent control interface for downstream extraction and recognition.

6. Empirical trends, limitations, and open directions

Across the supplied literature, three empirical trends recur. First, overlap handling is the principal advantage of target-speaker conditioning. The original TS-VAD literature explicitly attributes the weakness of clustering systems to limited ability to handle overlapping speech, while target-conditioned frame-level posteriors naturally allow simultaneous activity (Medennikov et al., 2020). Second, better speaker representations and better initialization matter. For unknown speaker counts, output masking and improved i-vector initialization via spectral clustering, VBx, and region proposal networks reduced DER by up to 50% relative across LibriCSS variants, and fusion of initialization systems yielded a further 3.9% relative DER gain over the best single system (He et al., 2021). Third, robustness increasingly depends on training the conditioning mechanism itself to tolerate noise or imperfection. “Robust Target Speaker Diarization and Separation via Augmented Speaker Embedding Sampling” uses automatic identification of target speaker embeddings within mixtures and an overlapping spectral loss; on LibriCSS it reduced DER from 14.69% for the cited baseline to 4.21%, corresponding to a 71.3% relative improvement, and reduced cpWER to 9.52% (Jalal et al., 8 Aug 2025).

Reported numbers, however, should not be read as a single leaderboard. The cited works use different collars, different overlap policies, different VAD assumptions, and different datasets, including VoxConverse, DIHARD-III, CALLHOME, CHiME-6, LibriCSS, AliMeeting, MISP 2022, AMI, and CSJ dialogue mixtures (Medennikov et al., 2020, Cheng et al., 2022, Wang et al., 2022, Wang et al., 2023). This suggests that cross-paper comparison is only meaningful after protocol normalization.

Several limitations are persistent. The original TS-VAD requires a fixed, known number of speakers and reasonably accurate initial clustering to estimate i-vectors (Medennikov et al., 2020). Audio-visual systems face partial or full occlusion, low resolution, fast motion blur, and complete lip loss (Cheng et al., 2024). Online systems often still require oracle or separate VAD, heuristic thresholds, and a predefined maximum number of speakers (Wang et al., 2023). Continuous target speech extraction assumes matched acoustic conditions between enrollment and mixture (Zhao et al., 2024). Transformer-based TS-VAD and related hybrids still rely on pre-extracted profiles or attractors, and joint end-to-end learning of profiles and diarization remains open (Wang et al., 2022).

A further misconception is that target-speaker diarization is synonymous with enrollment of fixed known speakers. The literature already contains multiple departures from that assumption: masking-based support for unknown speaker counts (He et al., 2021), dynamic speaker-axis models (Cheng et al., 2022), self-generated online profiles (Wang et al., 2023), and prompt-conditioned multimodal target speech diarization (Jiang et al., 2024). The present research frontier therefore lies less in the existence of target conditioning than in the source, modality, and reliability of the target representation. Concrete future directions named in the supplied papers include incorporating multi-channel audio and beamforming features, stronger semi-supervised pre-training, graph-based anchor fusion, joint VAD plus online TS-VAD training, dynamic speaker-count estimation, and integrating profile extraction into the model graph itself (Cheng et al., 2024, Wang et al., 2023, Wang et al., 2022).