Target-Speaker Voice Activity Detection

• TS-VAD is a speaker-conditioned, frame-level detection method that uses target-speaker profiles to accurately label overlapping speech in recordings.
• It refines initial clustering-based diarization by employing techniques like BLSTM, transformers, and convolutional front-ends for enhanced activity prediction.
• Recent advancements include multi-modal, online, and scalable architectures that improve robustness and detection accuracy in challenging acoustic scenarios.

Target-Speaker Voice Activity Detection (TS-VAD) is a speaker-conditioned, frame-level activity estimation paradigm in which an input recording and one or more target-speaker profiles are used to predict whether each target speaker is active at each frame. In speaker diarization, TS-VAD is typically used as a refinement stage after a first-pass clustering-based system supplies hypothesized speakers and speaker profiles, and its main technical advantage is that it can represent overlapping speech naturally because multiple target speakers can be marked active at the same time (Cheng et al., 2022, Wang et al., 2023). Across the literature, the term also covers close single-target relatives such as “personal VAD” and “speaker-dependent VAD,” which retain the same target-conditioned framewise detection idea but narrow the task to one enrolled speaker, often for streaming or on-device gating rather than full diarization (Ding et al., 2019, Chen et al., 2020).

1. Definition and task variants

In its diarization-oriented form, TS-VAD takes acoustic features from a recording together with a set of speaker profiles and predicts frame-level activity for each target speaker. The original CHiME-6 formulation “directly predicts the activity of each speaker on each time frame,” takes conventional speech features together with i-vectors for each speaker, and uses a set of binary classification output layers to produce activities of each speaker (Medennikov et al., 2020). Transformer-based work later described TS-VAD as a speaker-conditioned multi-label VAD: for every frame and every target speaker, the model estimates whether that speaker is active, making it especially suitable as a resegmentation or refinement stage after a first-pass diarization system has already identified speakers and extracted profiles (Wang et al., 2022).

Single-target variants narrow this formulation. “Personal VAD” detects, at the frame level, whether the current frame is non-speech, target-speaker speech, or non-target-speaker speech, and was explicitly motivated by gating a streaming on-device ASR system so that it triggers only for the target user (Ding et al., 2019). “Speaker-dependent VAD” defines the target as one designated speaker and treats silence, noise, and speech from non-target speakers as negative, which makes it an early precursor and special case of later target-speaker VAD formulations (Chen et al., 2020).

A useful way to distinguish these formulations is by their target definition and output structure.

Formulation	Conditioning	Output structure
Canonical diarization TS-VAD	Set of speaker profiles	One activity stream per target speaker
Personal VAD	Single enrolled target embedding or score	$\{\tt ns, tss, ntss\}$ multiclass output
Speaker-dependent VAD	Single target i-vector	Binary target-speech vs everything-else output

This division also clarifies a common misconception. Not every target-speaker VAD system is a canonical diarization TS-VAD system. Personal VAD and speaker-dependent VAD are target-conditioned frame-level detectors, but they do not attempt simultaneous multi-speaker attribution for all speakers in a recording, and they are typically not designed for overlap-aware diarization in the full multi-speaker sense (Ding et al., 2019, Chen et al., 2020).

2. Output spaces, labels, and probabilistic formulations

The literature contains two main output geometries. The first is the single-target formulation, where the model predicts whether the enrolled speaker is active. In “Personal VAD,” for each frame $t$, the system predicts one of three labels: non-speech $(\tt ns)$, target-speaker speech $(\tt tss)$, or non-target-speaker speech $(\tt ntss)$. Under embedding-conditioned training, the input is

$$\hat{\mathbf{x}}_t = [\mathbf{x}_t, \mathbf{e}^{\mathrm{target}}],$$

and the network produces

$$\mathbf{z}_t = [z_t^{\tt tss}, z_t^{\tt ntss}, z_t^{\tt ns}].$$

The same paper also defines a weighted pairwise loss to deemphasize confusion between $\tt ns$ and $\tt ntss$ while keeping target-speech confusions maximally important (Ding et al., 2019).

The second is the canonical multi-speaker formulation. Transformer-based TS-VAD writes the acoustic input as a sequence of log mel-filterbank features and the speaker profiles as

$$P = [p_1,\ldots,p_S]^T \in \mathbb{R}^{S \times P},$$

with one output stream per supplied profile. In this view, TS-VAD is a profile-conditioned multi-speaker activity detector whose output is a frame-by-speaker matrix of speech activity posteriors (Wang et al., 2023). Sequence-to-sequence TS-VAD makes this explicitly multi-label: $t$0 where each row corresponds to a target speaker and each column to an output time step; because the final projection uses sigmoid, several speakers can be simultaneously active at the same output time step, which is how overlapping speech is represented (Cheng et al., 2022).

Earlier online and CHiME-6 systems implemented the same idea with per-speaker binary heads. The CHiME-6 TS-VAD used four output layers, one per speaker, implemented in Kaldi as four 2-class softmax outputs, yielding an 8-dimensional training target vector corresponding to four pairs of silence and speech probabilities (Medennikov et al., 2020). The online AliMeeting system similarly estimated per-frame presence probabilities for all tracked speakers and used binary cross-entropy over the frame-by-speaker activity matrix (Wang et al., 2022).

This output design is central to what TS-VAD changes relative to clustering-based diarization. Clustering systems generally assume one speaker per segment. TS-VAD instead predicts parallel frame-level activity streams anchored to specific speaker profiles, so overlap becomes an ordinary multi-label prediction problem rather than an exceptional case (Cheng et al., 2022, Wang et al., 2023).

3. Architectures and conditioning mechanisms

The original diarization-oriented TS-VAD architecture in the CHiME-6 dinner-party work processed each target speaker with the same speaker-detection subnetwork and then combined the per-speaker hidden representations. The decisive architectural choice was to process each speaker by the same 2-layer BLSTMP speaker-detection block and then combine outputs for all speakers by one more BLSTMP layer, with parameters of the per-speaker block shared across speakers (Medennikov et al., 2020).

Subsequent work focused on variable speaker count, profile order invariance, memory scaling, and online conditioning. Transformer-based TS-VAD retained the encoder–ISD–JSD decomposition but replaced speaker-axis concatenation with an explicit tensor of shape $t$1, then applied transformer layers without positional encoding over the speaker axis. This made the model naturally handle a variable number of speakers and insensitive to the order of the supplied speaker profiles; the best reported configuration used BLSTM on the time axis and transformer on the speaker axis (Wang et al., 2022).

Sequence-to-sequence TS-VAD factorized acoustic-time modeling and speaker modeling even further. Its front end is a ResNet-34 operating on 80-dimensional log Mel-filterbank energies, followed by segmental statistical pooling, a Conformer encoder, and a speaker-wise decoder inspired by DAB-DETR. The decoder uses target-speaker embeddings as auxiliary queries, self-attention across target speakers, and a final projection that can emit voice activity at either 80 ms or 10 ms resolution. The paper’s central architectural claim is that this changes memory growth from $t$2 to $t$3 when feature dimensions are treated as constants (Cheng et al., 2022).

Online TS-VAD for AliMeeting made the conditioning state itself causal. A ResNet34 front end produced frame-level speaker embeddings for each incoming block, a Transformer encoder processed each speaker-conditioned sequence, and a BiLSTM predicted per-speaker presence. The target-speaker embeddings were not supplied by an offline clustering system; instead, a target speaker embedding buffer stored running means and counts and was updated block by block from high-confidence, non-overlap frame assignments (Wang et al., 2022).

Several papers extended TS-VAD beyond the single-channel audio-only setting. The M2MeT system from DKU_DukeECE used a ResNet speaker front end, per-speaker encoders, a BiLSTM for cross-speaker interaction, and then a multi-channel extension in which channel-wise speaker-conditioned representations were arranged as a $t$4 tensor and fused by a 2-layer, 2-head cross-channel Transformer encoder before average pooling over channels (Wang et al., 2022). MIMO-TSVAD generalized the same principle to audio-visual diarization with a ResNet-34 audio extractor, a modified ResNet18-3D lip-video extractor, a Conformer-based multimodal encoder, and three decoder branches for audio-based, video-based, and mixed audio-visual target-speaker activity estimation (Cheng et al., 2024).

Conditioning itself has also diversified. Personal VAD compared score-conditioned training, embedding-conditioned training, and combined score-plus-embedding conditioning, with embedding-conditioned training offering the clearest “TS-VAD-style” formulation for a single target speaker (Ding et al., 2019). A later noise-robust compact TS-VAD study compared concatenation, addition, multiplication, FiLM, and FiLM with embedding preprocessing in a small causal Conformer with two encoder layers and sub-150k parameter counts, and reported that FiLM provided the best overall performance (Bovbjerg et al., 6 Jan 2025).

4. Supervision, data construction, and optimization

A recurring difficulty is that public corpora rarely contain exactly the frame-level target-speaker labels and target enrollments that TS-VAD requires. Much of the literature therefore relies on synthetic conversations, overlap simulation, and label generation from alignments. Personal VAD built training examples from LibriSpeech by concatenating utterances from multiple speakers, sampling the number of concatenated utterances as

$t$5

choosing one speaker as the target, and relabeling frame-level speech as target-speaker or non-target-speaker speech according to the selected target. Forced alignment with a pretrained ASR model provided frame-level ground truth (Ding et al., 2019).

Large-scale diarization TS-VAD training follows the same pattern at greater scale. Seq2Seq-TSVAD generated synthetic diarization training data from VoxCeleb2 and LibriSpeech by removing non-speech with WebRTC VAD and then randomly mixing 1–4 speakers online, with MUSAN noise and RIR augmentation; training then proceeded in three stages, from frozen front end to mixed simulated-plus-real data to real-data fine-tuning, all with BCE supervision on per-speaker activity sequences (Cheng et al., 2022). PET-TSVAD pre-trained on 21k hours of simulated conversations with 1–10 speakers and overlap ratio up to $t$6, then replaced oracle speaker profiles by clustering-derived profiles from AHC and NME-SC to reduce the train/test mismatch caused by imperfect first-pass diarization. Because speaker-profile errors and pseudo-speaker slots make fixed speaker-to-output assignments inappropriate, PET-TSVAD used permutation invariant training, with the best permutation found by the Hungarian algorithm (Wang et al., 2023).

Objective functions depend on the formulation. Multi-speaker diarization systems typically use binary cross-entropy over time and speakers (Cheng et al., 2022, Wang et al., 2023). Personal VAD used standard multiclass cross-entropy

$t$7

but also proposed the weighted pairwise loss

$t$8

with $t$9 and $(\tt ns)$0 varied in $(\tt ns)$1, where the best performance occurred at $(\tt ns)$2 (Ding et al., 2019).

More recent work has introduced task-specific pretraining and multimodal masking. The noise-robust compact TS-VAD paper proposed causal DN-APC pretraining, in which noisy or reverberant input features are used to predict clean log-Mel features $(\tt ns)$3 frames ahead, thereby learning a causal denoising representation before full TS-VAD fine-tuning (Bovbjerg et al., 6 Jan 2025). MIMO-TSVAD used four-stage training, profile shuffling, model-level attention masking, and data-level masking of speaker and lip profiles so that the system could function under audio-only, video-only, and mixed-modality conditions (Cheng et al., 2024). In continuous target speech extraction, A-TSVAD used BCE supervision and then converted posterior activity estimates into timestamps through thresholding and morphological smoothing,

$(\tt ns)$4

with $(\tt ns)$5 (Zhao et al., 2024).

5. Applications and reported empirical performance

TS-VAD has been deployed across at least three major application regimes: overlap-aware diarization, streaming target-speaker gating, and target-conditioned extraction. In CHiME-6, the original multi-speaker TS-VAD reduced DER from 63.42 to 32.84 on the development set and from 68.20 to 36.02 on the evaluation set relative to the x-vector+AHC baseline, a reduction of more than 30 absolute DER points (Medennikov et al., 2020). Transformer-based TS-VAD later reported 4.57% DER on VoxConverse and 11.18% DER for EDA-TS-VAD on CALLHOME, framing TS-VAD as both a strong diarization backend in its own right and a stronger matching module for end-to-end neural diarization with attractors (Wang et al., 2022).

Sequence-to-sequence TS-VAD then emphasized scalability in speaker count and temporal resolution. Its best system achieved 4.55% DER on VoxConverse and 10.77% DER on DIHARD-III Track 1, with larger decoder capacity $(\tt ns)$6 and 10 ms output resolution both improving performance, and embedding augmentation proving crucial: removing augmentation degraded VoxConverse DER from 4.55% to 10.10% under the $(\tt ns)$7 ms, $(\tt ns)$8 setting (Cheng et al., 2022). PET-TSVAD targeted the failure mode of erroneous first-pass profiles and reported 4.35% DER on VoxConverse and 25.88% DER on DIHARD-I evaluation without pseudo-speaker clustering, improving over transformer-based TS-VAD especially when clustering errors were severe (Wang et al., 2023).

Meeting diarization papers in the M2MeT/AliMeeting setting showed particularly large gains from TS-VAD. The DKU_DukeECE system reduced the best single-channel clustering result of 12.68% DER to 3.14% DER by fusing eight single-channel TS-VAD systems, and then to 2.26% DER with multi-channel cross-channel-attention TS-VAD on the AliMeeting evaluation set; the final submitted multi-channel system achieved 2.98% DER on the test set (Wang et al., 2022). The USTC-Ximalaya system emphasized data preparation and post-processing rather than architectural change: starting from 23.14% DER for clustering-based diarization on ALIMEETING Eval with 0 collar, threshold-only TS-VAD trained on progressively richer real and simulated data reached 16.50%, and the full pipeline of thresholding, short-gap merging, oracle-VAD fusion, DOVER-Lap, and iterative i-vector re-estimation reached 7.80% on Eval and 9.14% on Test (He et al., 2022).

Single-target formulations target different operating points. Personal VAD reported a lightweight embedding-conditioned model with about 0.13 million parameters and about 130 KB after 8-bit quantization; under weighted pairwise loss, the best lightweight ET configuration achieved mean AP 0.959 without multistyle training and 0.912 with multistyle training, while also performing almost identically to standard VAD on non-concatenated test utterances containing only the target speaker: speech AP 0.992 for standard VAD versus 0.991 for personal VAD without MTR, and 0.975 versus 0.979 with MTR (Ding et al., 2019). The noise-robust compact Conformer study reported that DN-APC pretraining improved mAP by $(\tt ns)$9 in clean conditions, $(\tt tss)$0 in seen noise, and $(\tt tss)$1 in unseen noise averaged across conditioning methods, with FiLM giving the best overall performance (Bovbjerg et al., 6 Jan 2025).

Audio-visual and extraction-oriented systems further broadened the application space. MIMO-TSVAD reported DERs of 4.18% on VoxConverse, 10.10% on DIHARD-III, and 8.15% on MISP 2022, while preserving audio-only, video-only, and mixed audio-visual operating modes in a unified sequence-to-sequence framework (Cheng et al., 2024). In continuous target speech extraction, A-TSVAD by itself outperformed off-the-shelf diarization baselines, and the best integration with personalized BSRNN in “Cascade approach 1” reached DER 26.5, JER 16.2, and INT 41.3, compared with DER 43.1 for the TSVAD baseline and 57.4/58.7 DER for Pyannote and VBx (Zhao et al., 2024).

6. Limitations, robustness strategies, and adjacent directions

Despite these gains, the literature repeatedly identifies three structural limitations. First, many TS-VAD systems depend on first-pass clustering-derived speaker profiles, and those profiles are often wrong through merging, splitting, missing speakers, or contaminated embeddings. PET-TSVAD directly targeted this weakness by adding five learnable 128-dimensional pseudo-speaker profiles, training with clustering-derived profiles rather than oracle profiles, and using PIT so that missing speakers could be assigned to pseudo-profile slots instead of being forced into a one-to-one mapping with erroneous first-pass clusters (Wang et al., 2023). Second, many systems still require a bounded speaker capacity. The 2021 unknown-speaker-count extension handled this by estimating speaker count with an external diarization system, padding with dummy i-vectors when $(\tt tss)$2, and selecting the $(\tt tss)$3 longest non-overlapping speakers when $(\tt tss)$4; with an 8-speaker model, DERs on LibriCSS were 12.4%, 11.3%, and 7.6% for 2-, 5-, and 8-speaker conditions, but applying a 5-speaker model to 8-speaker recordings caused catastrophic degradation to 42.8% DER (He et al., 2021).

Third, canonical TS-VAD is usually discriminative and profile-conditioned. Recent work has explored alternatives without fully discarding the framework. Flow-TSVAD introduced conditional flow matching into Seq2Seq-TSVAD, but showed that applying flow matching directly in the original binary label space failed badly, with 82.35% DER; mapping target-speaker activity sequences to a dense latent space before flow matching reduced CALLHOME DER to 11.21%, and a 3-sample DOVER-Lap ensemble further reduced it to 10.91% (Chen et al., 2024). This suggests that generative modeling may be viable for diarization refinement, but only after learning a continuous latent geometry for activity sequences.

Several adjacent directions also clarify what TS-VAD is not. EEND-SAA is explicitly presented as an enrollment-less, main-speaker VAD system rather than a TS-VAD system: it does not use enrollment speech and defines the target as the speaker who talks more steadily and clearly. Its dual-attractor model reduced main-speaker DER from 6.63% to 3.61% relative to an SA-EEND baseline under the paper’s task definition, but the target is role-defined rather than identity-defined, so it is not a direct substitute for standard TS-VAD when the desired speaker is known in advance (Wu et al., 15 Sep 2025). At the opposite end, the two-stage AVTSE system for edge devices uses a compact visual VAD front end that predicts visible-target activity from cropped lip video and then injects that hard activity cue into a lightweight extractor. Its VVAD stage reported 78.46% accuracy, 87.65% precision, and 83.96% recall, and the full system required only 1.89 GMacs and 1.36M parameters in the table, but this is a visual target-activity front end rather than a diarization-style profile-conditioned TS-VAD (Li et al., 28 May 2025).

Taken together, these variants indicate that TS-VAD has become a family of target-conditioned framewise activity models rather than a single architecture. The common core remains stable: explicit conditioning on target identity or target role, frame-level activity prediction, and overlap-aware output structure. What changes across papers is the scope of the target set, the way conditioning is obtained, and the degree to which the system depends on first-pass speaker inventories, modality availability, or deployment constraints.