---
title: Target-Speaker Diarization
url: https://www.emergentmind.com/topics/target-speaker-diarization
type: topic
---

# Target-Speaker Diarization

Target-speaker diarization is a family of diarization methods that reformulates the classical “who spoke when” problem as target-conditioned activity estimation: given a multi-speaker recording and one or more target-speaker representations, the system predicts for each target speaker whether that speaker is active at each frame. In the supplied literature, the most recurrent formulation is target-speaker voice activity detection (TS-VAD), which directly predicts an activity of each speaker on each time frame and is designed to address overlapping speech more effectively than clustering-only pipelines [2005.07272]. Subsequent work generalized the paradigm to sequence-to-sequence prediction, variable-length speaker axes, enrollment-conditioned end-to-end neural diarization, audio-visual conditioning, online target-speaker tracking, and joint diarization-separation or diarization-ASR systems [2210.16127][2208.13085][2310.08696]. A related extension, target speech diarization, further broadens the conditioning signal from speaker identity to semantic or multimodal prompts, replacing “who spoke when” with “when target event occurs” [2406.07198].

## 1. Task definition and formalization

In its canonical form, target-speaker diarization assumes an acoustic sequence and a set of target-speaker profiles, then estimates a frame-wise posterior for each target. One representative formulation writes, for speaker $s$ and frame $t$, the posterior $P(y_s(t)=1 \mid X, I)$, where $X$ is the acoustic feature sequence and $I$ is the set of per-speaker embeddings [2005.07272]. A later end-to-end formulation states the same objective as estimating, for each speaker $i$, the frame-wise posterior probability of that speaker’s activity conditioned on both the acoustic input and the speaker’s enrollment embedding, with outputs arranged as a multi-label speaker-by-time tensor rather than a single speaker label per frame [2305.10704].

This target-conditioned view differs from conventional clustering-based diarization in an important way. Instead of discovering speaker structure only by segmenting and clustering embeddings, the model is explicitly told which speaker to detect through an auxiliary representation such as an i-vector, x-vector, d-vector, z-vector, lip anchor, enrollment embedding, or attractor. This is why the literature repeatedly treats target-speaker diarization as especially suitable for strongly overlapping speech: multiple target posteriors can be high at the same frame, so overlap is handled natively rather than by post hoc correction [2005.07272][2210.16127].

The same formal pattern has also been extended beyond speaker identity. “Target Speech Diarization with Multimodal Prompts” defines target speech diarization as detecting “when target event occurs” according to the semantic characteristics of speech, and proposes prompts including semantic language description, pre-enrolled speech, pre-registered face image, and audio-language logical prompts [2406.07198]. This suggests a broader interpretation of target-speaker diarization as one instance of prompt-conditioned temporal event detection in conversational audio.

## 2. Canonical TS-VAD pipeline

The classical TS-VAD workflow is two-stage. A conventional diarization system first produces rough speaker segments and corresponding speaker profiles; TS-VAD then refines those hypotheses by predicting each target speaker’s activity on each frame [2401.08052]. In the original 2020 formulation, the model takes conventional speech features such as MFCCs together with i-vectors for each speaker as inputs, and a set of binary classification output layers produces activities of each speaker [2005.07272].

Architecturally, the original single-channel TS-VAD model uses a speaker-detection block followed by a fusion BLSTMP. For $S=4$ speakers, each speaker stream concatenates frame features with that speaker’s i-vector, processes the result through a 2-layer BLSTMP, and then concatenates all per-speaker hidden states into a fusion BLSTMP that produces per-speaker speech posteriors [2005.07272]. The training objective is binary cross-entropy over speakers and frames:
$$
L = - \sum_{s=1}^S \sum_{t=1}^T \left[y_s(t)\log P(y_s(t)=1\mid X,I) + (1-y_s(t))\log P(y_s(t)=0\mid X,I)\right].
$$
This formulation is directly multi-label, so it does not force a single active speaker per frame [2005.07272].

Several practical mechanisms became standard around this design. One is iterative profile refinement: speaker i-vectors can be estimated iteratively, starting with a strong clustering-based diarization, then recomputed from improved TS-VAD outputs [2005.07272]. Another is post-processing. The original work investigated both median-threshold smoothing and an HMM+Viterbi decoder whose states represent silence, single-speaker activity, and exactly two-speaker overlap [2005.07272]. A further extension to TS-VAD-MC applied a simple attention mechanism across multiple microphones, aggregating channel-specific hidden representations before the final fusion block [2005.07272].

Empirically, this design was motivated by failure modes of clustering on realistic overlap-heavy recordings. On CHiME-6 unsegmented data, the TS-VAD approach outperformed the baseline x-vector-based system by more than 30% Diarization Error Rate abs, and the final fusion of six TS-VAD variants with Viterbi decoding reached 32.84% DER on dev and 36.02% DER on eval [2005.07272]. Those results established the central claim that direct frame-level target-speaker activity estimation can outperform purely clustering-based diarization in difficult multi-speaker conditions.

## 3. Scaling the paradigm: sequence-to-sequence, transformers, and dynamic speaker axes

A major limitation of early TS-VAD was the entanglement of time and speaker dimensions, which constrained speaker capacity and output resolution. Seq2Seq-TSVAD replaced the conventional encoder-only TS-VAD architecture by a true sequence-to-sequence model that factorizes time and speaker axes, reducing memory from $\mathcal O(T\cdot N)$ to $\mathcal O(T)+\mathcal O(N)$ and enabling decoding of up to $N=30$ speakers in a single 12 GB GPU [2210.16127]. The same model made temporal resolution a parameter of the final output layer, with reported experiments at $R=80\,\mathrm{ms}$ and $R=10\,\mathrm{ms}$; on VoxConverse test, the best single Seq2Seq-TSVAD with $L=30$ and $R=10\,\mathrm{ms}$ achieved 4.55% DER, while on DIHARD-III Track 1 it achieved 10.77% DER [2210.16127].

Transformerization addressed a different constraint: order sensitivity and variable speaker count. “Target Speaker Voice Activity Detection with Transformers and Its Integration with End-to-End Neural Diarization” introduced a joint speaker detection module that alternates Transformer layers along the speaker axis and sequential layers along the time axis, with no positional encoding on the speaker axis so that the model output is insensitive to the order of the speaker profiles [2208.13085]. On VoxConverse, using transformers for cross-speaker modeling reduced DER by 11.3% relative over the original TS-VAD, reaching 4.57%, and the same speaker-conditioned matching module reduced CALLHOME DER to 11.18% when integrated into EEND-EDA [2208.13085].

A complementary line of work removed the fixed-speaker assumption by making the detector itself operate on a dynamic speaker axis. MTEAD combines a z-vector extractor with a detector composed of a Feature Mixer and stacked Time-Speaker Contextualizers; the across-speaker BiLSTM runs along an axis whose length varies from session to session, so the model can handle any number of speakers at inference time [2203.16007]. On CALLHOME, MTEAD(z-vector) achieved 14.31% DER and yielded relative reductions over several baselines [2203.16007].

AED-EEND addressed target conditioning from the end-to-end diarization side. It calculates attractors from enrollment embeddings, including three special embeddings for non-speech, single-speaker speech, and overlap speech, and replaces the LSTM attractor decoder of EEND-EDA with an attention-based model [2305.10704]. With only $0.5\,\mathrm{s}$ of enrollment data, AED-EEND reached 7.75% DER on 2-speaker CALLHOME and 14.22% overall in the flexible-speaker setting [2305.10704]. Taken together, these models show that target-speaker diarization is no longer confined to the original fixed-profile, BLSTM-only TS-VAD template.

## 4. Modalities, prompts, and online operation

The addition of visual information produced a distinct audio-visual branch of target-speaker diarization. MIMO-TSVAD takes audio-visual input and uses either the speaker’s acoustic footprint or lip track to conduct audio-based, video-based, and audio-visual speaker diarization in a unified sequence-to-sequence framework [2401.08052]. Its architecture combines an audio extractor, a lip-feature extractor, a shared multi-modal Conformer encoder, and a tri-branch decoder with audio, video, and mixed heads. The design explicitly addresses the modality-missing problem through model-level masking, data-level masking of anchors, and a Speaker Alignment step based on the Hungarian algorithm when lip anchors are completely missing [2401.08052]. Reported DERs are 4.18% on VoxConverse, 10.10% on DIHARD-III, and 8.15% on MISP 2022 [2401.08052].

The same conditioning logic has been generalized from audio-visual identity cues to prompt-based control. MM-TSD proposes a Multimodal Target Speech Diarization framework that accommodates semantic language description, pre-enrolled speech, pre-registered face image, and audio-language logical prompts, and includes a voice-face aligner module that projects human voice and face representation into a shared space [2406.07198]. The framework is also described as versatile enough to perform speaker diarization and overlap speech detection using task-specific prompts [2406.07198]. Although this is formally target speech diarization rather than target-speaker diarization, it extends the same prompt-conditioned temporal detection paradigm.

Online operation required a further change: the target profiles themselves had to be generated and updated during inference. Online TS-VAD used a ResNet-based front end to extract frame-level speaker embeddings block by block, then updated target-speaker embeddings through a Target Speaker Embedding Buffer while predicting frame-wise presence probabilities [2207.05920]. With oracle VAD on AliMeeting and a block size of $16\,\mathrm{s}$ with $2\,\mathrm{s}$ shift, the reported online single-channel TS-VAD reached 8.14% DER on Eval and 11.42% DER on Test [2207.05920]. The later “End-to-end Online Speaker Diarization with Target Speaker Tracking” removed the clustering prerequisite entirely by self-generating embeddings and updating them with either a buffer-based or accumulate-sum rule; it reported 13.31% DER on DIHARD III with 0.8 s latency and 4.50% DER for the multi-channel version on AliMeeting Test [2310.08696]. A recurrent misconception is therefore that target-speaker diarization must be offline and initialization-heavy; the online literature shows that self-generated, incrementally updated profiles are viable, although often at the cost of additional thresholds, oracle or separate VAD, and predefined maximum speaker count [2207.05920][2310.08696].

## 5. Joint diarization with separation, extraction, and ASR

Because target-speaker diarization already estimates speaker-conditioned activity, it is a natural control signal for source separation and target-speaker extraction. TS-SEP extends TS-VAD by replacing the final frame-level speaker activity estimation layer with a network that produces speaker activity estimates at a time-frequency resolution; the resulting masks can be used directly for source extraction or as initialization for beamforming [2303.03849]. On LibriCSS full 10 min sessions, the best reported system, TS-SEP $\rightarrow$ GSS (TF-init) $\rightarrow$ WavLM, achieved 5.06% cpWER and 3.26% DI-cpWER, with DER 14.7% [2303.03849].

USED makes the integration even tighter by sharing a TCN backbone for speaker extraction and diarization and adding an Embedding Assignment Module, a Scenario-Aware Differentiated extraction loss, and a Multi-Task Interaction module [2309.10674]. On LibriMix max mode, USED-F reported 4.75% DER and 12.70 dB SI-SDRi; on CALLHOME, with 452 h pre-training on LibriMix, USED-F achieved 13.16% DER [2309.10674]. The paper explicitly frames this as a response to output inconsistency and scenario mismatch between diarization and extraction [2309.10674].

Continuous Target Speech Extraction pushes the same idea into long recordings with target absence. Its A-TSVAD directly generates timestamps of the target speaker rather than merely refining existing diarization, and the best integration strategy is a sequential cascade in which A-TSVAD first segments the target-active regions and pBSRNN then performs extraction [2401.15993]. In the reported comparison, Cascade 1 achieved 26.5 DER, 16.2 JER, 41.3 INT, 10.4 SI-SNR, and 2.91 PESQ [2401.15993].

Target-speaker conditioning has also been used to couple diarization with recognition. The 2019 TS-ASR framework alternates between estimating speaker embeddings and decoding each segment with a target-speaker acoustic model, producing diarization as a by-product of frame-level alignments [1909.08103]. On two-speaker CSJ dialogue mixtures, three EM-style iterations reduced total WER to 40.03%, only 2.07% absolute behind the oracle TS-AM, while DER improved to 24.63% [1909.08103]. More recently, DiCoW replaced speaker embeddings with diarization outputs as conditioning information for target-speaker ASR, introducing frame-level diarization-dependent transformations and query-key biasing inside Whisper [2501.00114]. On AMI-sdm with oracle diarization, DiCoW reported 17.2/16.5 cpWER/ORC-WER, and on LibriCSS it reported 5.6/5.5 [2501.00114]. This progression indicates that target-speaker diarization is not only an endpoint but also a latent control interface for downstream extraction and recognition.

## 6. Empirical trends, limitations, and open directions

Across the supplied literature, three empirical trends recur. First, overlap handling is the principal advantage of target-speaker conditioning. The original TS-VAD literature explicitly attributes the weakness of clustering systems to limited ability to handle overlapping speech, while target-conditioned frame-level posteriors naturally allow simultaneous activity [2005.07272]. Second, better speaker representations and better initialization matter. For unknown speaker counts, output masking and improved i-vector initialization via spectral clustering, VBx, and region proposal networks reduced DER by up to 50% relative across LibriCSS variants, and fusion of initialization systems yielded a further 3.9% relative DER gain over the best single system [2108.03342]. Third, robustness increasingly depends on training the conditioning mechanism itself to tolerate noise or imperfection. “Robust Target Speaker Diarization and Separation via Augmented Speaker Embedding Sampling” uses automatic identification of target speaker embeddings within mixtures and an overlapping spectral loss; on LibriCSS it reduced DER from 14.69% for the cited baseline to 4.21%, corresponding to a 71.3% relative improvement, and reduced cpWER to 9.52% [2508.06393].

Reported numbers, however, should not be read as a single leaderboard. The cited works use different collars, different overlap policies, different VAD assumptions, and different datasets, including VoxConverse, DIHARD-III, CALLHOME, CHiME-6, LibriCSS, AliMeeting, MISP 2022, AMI, and CSJ dialogue mixtures [2005.07272][2210.16127][2208.13085][2310.08696]. This suggests that cross-paper comparison is only meaningful after protocol normalization.

Several limitations are persistent. The original TS-VAD requires a fixed, known number of speakers and reasonably accurate initial clustering to estimate i-vectors [2005.07272]. Audio-visual systems face partial or full occlusion, low resolution, fast motion blur, and complete lip loss [2401.08052]. Online systems often still require oracle or separate VAD, heuristic thresholds, and a predefined maximum number of speakers [2310.08696]. Continuous target speech extraction assumes matched acoustic conditions between enrollment and mixture [2401.15993]. Transformer-based TS-VAD and related hybrids still rely on pre-extracted profiles or attractors, and joint end-to-end learning of profiles and diarization remains open [2208.13085].

A further misconception is that target-speaker diarization is synonymous with enrollment of fixed known speakers. The literature already contains multiple departures from that assumption: masking-based support for unknown speaker counts [2108.03342], dynamic speaker-axis models [2203.16007], self-generated online profiles [2310.08696], and prompt-conditioned multimodal target speech diarization [2406.07198]. The present research frontier therefore lies less in the existence of target conditioning than in the source, modality, and reliability of the target representation. Concrete future directions named in the supplied papers include incorporating multi-channel audio and beamforming features, stronger semi-supervised pre-training, graph-based anchor fusion, joint VAD plus online TS-VAD training, dynamic speaker-count estimation, and integrating profile extraction into the model graph itself [2401.08052][2310.08696][2208.13085].

Source: https://www.emergentmind.com/topics/target-speaker-diarization