---
title: Target-Speaker Voice Activity Detection
url: https://www.emergentmind.com/topics/target-speaker-voice-activity-detection-ts-vad
type: topic
---

# Target-Speaker Voice Activity Detection

Target-Speaker Voice Activity Detection (TS-VAD) is a speaker-conditioned, frame-level activity estimation paradigm in which an input recording and one or more target-speaker profiles are used to predict whether each target speaker is active at each frame. In speaker diarization, TS-VAD is typically used as a refinement stage after a first-pass clustering-based system supplies hypothesized speakers and speaker profiles, and its main technical advantage is that it can represent overlapping speech naturally because multiple target speakers can be marked active at the same time [2210.16127][2309.12521]. Across the literature, the term also covers close single-target relatives such as “personal VAD” and “speaker-dependent VAD,” which retain the same target-conditioned framewise detection idea but narrow the task to one enrolled speaker, often for streaming or on-device gating rather than full diarization [1908.04284][2009.09906].

## 1. Definition and task variants

In its diarization-oriented form, TS-VAD takes acoustic features from a recording together with a set of speaker profiles and predicts frame-level activity for each target speaker. The original CHiME-6 formulation “directly predicts the activity of each speaker on each time frame,” takes conventional speech features together with i-vectors for each speaker, and uses a set of binary classification output layers to produce activities of each speaker [2005.07272]. Transformer-based work later described TS-VAD as a speaker-conditioned multi-label VAD: for every frame and every target speaker, the model estimates whether that speaker is active, making it especially suitable as a resegmentation or refinement stage after a first-pass diarization system has already identified speakers and extracted profiles [2208.13085].

Single-target variants narrow this formulation. “Personal VAD” detects, at the frame level, whether the current frame is non-speech, target-speaker speech, or non-target-speaker speech, and was explicitly motivated by gating a streaming on-device ASR system so that it triggers only for the target user [1908.04284]. “Speaker-dependent VAD” defines the target as one designated speaker and treats silence, noise, and speech from non-target speakers as negative, which makes it an early precursor and special case of later target-speaker VAD formulations [2009.09906].

A useful way to distinguish these formulations is by their target definition and output structure.

| Formulation | Conditioning | Output structure |
|---|---|---|
| Canonical diarization TS-VAD | Set of speaker profiles | One activity stream per target speaker |
| Personal VAD | Single enrolled target embedding or score | \(\{\tt ns, tss, ntss\}\) multiclass output |
| Speaker-dependent VAD | Single target i-vector | Binary target-speech vs everything-else output |

This division also clarifies a common misconception. Not every target-speaker VAD system is a canonical diarization TS-VAD system. Personal VAD and speaker-dependent VAD are target-conditioned frame-level detectors, but they do not attempt simultaneous multi-speaker attribution for all speakers in a recording, and they are typically not designed for overlap-aware diarization in the full multi-speaker sense [1908.04284][2009.09906].

## 2. Output spaces, labels, and probabilistic formulations

The literature contains two main output geometries. The first is the single-target formulation, where the model predicts whether the enrolled speaker is active. In “Personal VAD,” for each frame \(t\), the system predicts one of three labels: non-speech \((\tt ns)\), target-speaker speech \((\tt tss)\), or non-target-speaker speech \((\tt ntss)\). Under embedding-conditioned training, the input is
\[
\hat{\mathbf{x}}_t = [\mathbf{x}_t, \mathbf{e}^{\mathrm{target}}],
\]
and the network produces
\[
\mathbf{z}_t = [z_t^{\tt tss}, z_t^{\tt ntss}, z_t^{\tt ns}].
\]
The same paper also defines a weighted pairwise loss to deemphasize confusion between \(\tt ns\) and \(\tt ntss\) while keeping target-speech confusions maximally important [1908.04284].

The second is the canonical multi-speaker formulation. Transformer-based TS-VAD writes the acoustic input as a sequence of log mel-filterbank features and the speaker profiles as
\[
P = [p_1,\ldots,p_S]^T \in \mathbb{R}^{S \times P},
\]
with one output stream per supplied profile. In this view, TS-VAD is a profile-conditioned multi-speaker activity detector whose output is a frame-by-speaker matrix of speech activity posteriors [2309.12521]. Sequence-to-sequence TS-VAD makes this explicitly multi-label:
\[
\hat{\mathbf{Y}} \in [0,1]^{N \times K},
\]
where each row corresponds to a target speaker and each column to an output time step; because the final projection uses sigmoid, several speakers can be simultaneously active at the same output time step, which is how overlapping speech is represented [2210.16127].

Earlier online and CHiME-6 systems implemented the same idea with per-speaker binary heads. The CHiME-6 TS-VAD used four output layers, one per speaker, implemented in Kaldi as four 2-class softmax outputs, yielding an 8-dimensional training target vector corresponding to four pairs of silence and speech probabilities [2005.07272]. The online AliMeeting system similarly estimated per-frame presence probabilities for all tracked speakers and used binary cross-entropy over the frame-by-speaker activity matrix [2207.05920].

This output design is central to what TS-VAD changes relative to clustering-based diarization. Clustering systems generally assume one speaker per segment. TS-VAD instead predicts parallel frame-level activity streams anchored to specific speaker profiles, so overlap becomes an ordinary multi-label prediction problem rather than an exceptional case [2210.16127][2309.12521].

## 3. Architectures and conditioning mechanisms

The original diarization-oriented TS-VAD architecture in the CHiME-6 dinner-party work processed each target speaker with the same speaker-detection subnetwork and then combined the per-speaker hidden representations. The decisive architectural choice was to process each speaker by the same 2-layer BLSTMP speaker-detection block and then combine outputs for all speakers by one more BLSTMP layer, with parameters of the per-speaker block shared across speakers [2005.07272].

Subsequent work focused on variable speaker count, profile order invariance, memory scaling, and online conditioning. Transformer-based TS-VAD retained the encoder–ISD–JSD decomposition but replaced speaker-axis concatenation with an explicit tensor of shape \(\mathbb{R}^{T \times S \times F}\), then applied transformer layers without positional encoding over the speaker axis. This made the model naturally handle a variable number of speakers and insensitive to the order of the supplied speaker profiles; the best reported configuration used BLSTM on the time axis and transformer on the speaker axis [2208.13085].

Sequence-to-sequence TS-VAD factorized acoustic-time modeling and speaker modeling even further. Its front end is a ResNet-34 operating on 80-dimensional log Mel-filterbank energies, followed by segmental statistical pooling, a Conformer encoder, and a speaker-wise decoder inspired by DAB-DETR. The decoder uses target-speaker embeddings as auxiliary queries, self-attention across target speakers, and a final projection that can emit voice activity at either 80 ms or 10 ms resolution. The paper’s central architectural claim is that this changes memory growth from \(\mathcal{O}(T\times N)\) to \(\mathcal{O}(T+N)\) when feature dimensions are treated as constants [2210.16127].

Online TS-VAD for AliMeeting made the conditioning state itself causal. A ResNet34 front end produced frame-level speaker embeddings for each incoming block, a Transformer encoder processed each speaker-conditioned sequence, and a BiLSTM predicted per-speaker presence. The target-speaker embeddings were not supplied by an offline clustering system; instead, a target speaker embedding buffer stored running means and counts and was updated block by block from high-confidence, non-overlap frame assignments [2207.05920].

Several papers extended TS-VAD beyond the single-channel audio-only setting. The M2MeT system from DKU_DukeECE used a ResNet speaker front end, per-speaker encoders, a BiLSTM for cross-speaker interaction, and then a multi-channel extension in which channel-wise speaker-conditioned representations were arranged as a \(T \times N \times C \times 2D\) tensor and fused by a 2-layer, 2-head cross-channel Transformer encoder before average pooling over channels [2202.02687]. MIMO-TSVAD generalized the same principle to audio-visual diarization with a ResNet-34 audio extractor, a modified ResNet18-3D lip-video extractor, a Conformer-based multimodal encoder, and three decoder branches for audio-based, video-based, and mixed audio-visual target-speaker activity estimation [2401.08052].

Conditioning itself has also diversified. Personal VAD compared score-conditioned training, embedding-conditioned training, and combined score-plus-embedding conditioning, with embedding-conditioned training offering the clearest “TS-VAD-style” formulation for a single target speaker [1908.04284]. A later noise-robust compact TS-VAD study compared concatenation, addition, multiplication, FiLM, and FiLM with embedding preprocessing in a small causal Conformer with two encoder layers and sub-150k parameter counts, and reported that FiLM provided the best overall performance [2501.03184].

## 4. Supervision, data construction, and optimization

A recurring difficulty is that public corpora rarely contain exactly the frame-level target-speaker labels and target enrollments that TS-VAD requires. Much of the literature therefore relies on synthetic conversations, overlap simulation, and label generation from alignments. Personal VAD built training examples from LibriSpeech by concatenating utterances from multiple speakers, sampling the number of concatenated utterances as
\[
n \sim \mathrm{Uniform}(a,b), \qquad a=1,\ b=3,
\]
choosing one speaker as the target, and relabeling frame-level speech as target-speaker or non-target-speaker speech according to the selected target. Forced alignment with a pretrained ASR model provided frame-level ground truth [1908.04284].

Large-scale diarization TS-VAD training follows the same pattern at greater scale. Seq2Seq-TSVAD generated synthetic diarization training data from VoxCeleb2 and LibriSpeech by removing non-speech with WebRTC VAD and then randomly mixing 1–4 speakers online, with MUSAN noise and RIR augmentation; training then proceeded in three stages, from frozen front end to mixed simulated-plus-real data to real-data fine-tuning, all with BCE supervision on per-speaker activity sequences [2210.16127]. PET-TSVAD pre-trained on 21k hours of simulated conversations with 1–10 speakers and overlap ratio up to \(30\%\), then replaced oracle speaker profiles by clustering-derived profiles from AHC and NME-SC to reduce the train/test mismatch caused by imperfect first-pass diarization. Because speaker-profile errors and pseudo-speaker slots make fixed speaker-to-output assignments inappropriate, PET-TSVAD used permutation invariant training, with the best permutation found by the Hungarian algorithm [2309.12521].

Objective functions depend on the formulation. Multi-speaker diarization systems typically use binary cross-entropy over time and speakers [2210.16127][2309.12521]. Personal VAD used standard multiclass cross-entropy
\[
L_{\mathrm{CE}}(y,\mathbf{z}) = -\log \frac{\exp(z^y)}{\sum_k \exp(z^k)},
\]
but also proposed the weighted pairwise loss
\[
L_{\mathrm{WPL}}(y,\mathbf{z}) = - \mathbb{E}_{k\neq y}\left[w_{<k,y>} \cdot \log \frac{\exp(z^y)}{\exp(z^y)+\exp(z^k)}\right],
\]
with \(w_{\tt <tss,ns>} = w_{\tt <tss,ntss>} = 1\) and \(w_{\tt <ns,ntss>}\) varied in \(\{0.01, 0.05, 0.1, 0.5, 1.0\}\), where the best performance occurred at \(w_{\tt <ns,ntss>}=0.1\) [1908.04284].

More recent work has introduced task-specific pretraining and multimodal masking. The noise-robust compact TS-VAD paper proposed causal DN-APC pretraining, in which noisy or reverberant input features are used to predict clean log-Mel features \(k=3\) frames ahead, thereby learning a causal denoising representation before full TS-VAD fine-tuning [2501.03184]. MIMO-TSVAD used four-stage training, profile shuffling, model-level attention masking, and data-level masking of speaker and lip profiles so that the system could function under audio-only, video-only, and mixed-modality conditions [2401.08052]. In continuous target speech extraction, A-TSVAD used BCE supervision and then converted posterior activity estimates into timestamps through thresholding and morphological smoothing,
\[
\hat{\rm Label} ={\rm Erosion}({\rm Dilation}(\delta(\hat{P}\ge \alpha ))),
\]
with \(\alpha = 0.025\) [2401.15993].

## 5. Applications and reported empirical performance

TS-VAD has been deployed across at least three major application regimes: overlap-aware diarization, streaming target-speaker gating, and target-conditioned extraction. In CHiME-6, the original multi-speaker TS-VAD reduced DER from 63.42 to 32.84 on the development set and from 68.20 to 36.02 on the evaluation set relative to the x-vector+AHC baseline, a reduction of more than 30 absolute DER points [2005.07272]. Transformer-based TS-VAD later reported 4.57% DER on VoxConverse and 11.18% DER for EDA-TS-VAD on CALLHOME, framing TS-VAD as both a strong diarization backend in its own right and a stronger matching module for end-to-end neural diarization with attractors [2208.13085].

Sequence-to-sequence TS-VAD then emphasized scalability in speaker count and temporal resolution. Its best system achieved 4.55% DER on VoxConverse and 10.77% DER on DIHARD-III Track 1, with larger decoder capacity \(L\) and 10 ms output resolution both improving performance, and embedding augmentation proving crucial: removing augmentation degraded VoxConverse DER from 4.55% to 10.10% under the \(R=10\) ms, \(L=30\) setting [2210.16127]. PET-TSVAD targeted the failure mode of erroneous first-pass profiles and reported 4.35% DER on VoxConverse and 25.88% DER on DIHARD-I evaluation without pseudo-speaker clustering, improving over transformer-based TS-VAD especially when clustering errors were severe [2309.12521].

Meeting diarization papers in the M2MeT/AliMeeting setting showed particularly large gains from TS-VAD. The DKU_DukeECE system reduced the best single-channel clustering result of 12.68% DER to 3.14% DER by fusing eight single-channel TS-VAD systems, and then to 2.26% DER with multi-channel cross-channel-attention TS-VAD on the AliMeeting evaluation set; the final submitted multi-channel system achieved 2.98% DER on the test set [2202.02687]. The USTC-Ximalaya system emphasized data preparation and post-processing rather than architectural change: starting from 23.14% DER for clustering-based diarization on ALIMEETING Eval with 0 collar, threshold-only TS-VAD trained on progressively richer real and simulated data reached 16.50%, and the full pipeline of thresholding, short-gap merging, oracle-VAD fusion, DOVER-Lap, and iterative i-vector re-estimation reached 7.80% on Eval and 9.14% on Test [2202.04855].

Single-target formulations target different operating points. Personal VAD reported a lightweight embedding-conditioned model with about 0.13 million parameters and about 130 KB after 8-bit quantization; under weighted pairwise loss, the best lightweight ET configuration achieved mean AP 0.959 without multistyle training and 0.912 with multistyle training, while also performing almost identically to standard VAD on non-concatenated test utterances containing only the target speaker: speech AP 0.992 for standard VAD versus 0.991 for personal VAD without MTR, and 0.975 versus 0.979 with MTR [1908.04284]. The noise-robust compact Conformer study reported that DN-APC pretraining improved mAP by \(+2.15\) in clean conditions, \(+2.18\) in seen noise, and \(+2.21\) in unseen noise averaged across conditioning methods, with FiLM giving the best overall performance [2501.03184].

Audio-visual and extraction-oriented systems further broadened the application space. MIMO-TSVAD reported DERs of 4.18% on VoxConverse, 10.10% on DIHARD-III, and 8.15% on MISP 2022, while preserving audio-only, video-only, and mixed audio-visual operating modes in a unified sequence-to-sequence framework [2401.08052]. In continuous target speech extraction, A-TSVAD by itself outperformed off-the-shelf diarization baselines, and the best integration with personalized BSRNN in “Cascade approach 1” reached DER 26.5, JER 16.2, and INT 41.3, compared with DER 43.1 for the TSVAD baseline and 57.4/58.7 DER for Pyannote and VBx [2401.15993].

## 6. Limitations, robustness strategies, and adjacent directions

Despite these gains, the literature repeatedly identifies three structural limitations. First, many TS-VAD systems depend on first-pass clustering-derived speaker profiles, and those profiles are often wrong through merging, splitting, missing speakers, or contaminated embeddings. PET-TSVAD directly targeted this weakness by adding five learnable 128-dimensional pseudo-speaker profiles, training with clustering-derived profiles rather than oracle profiles, and using PIT so that missing speakers could be assigned to pseudo-profile slots instead of being forced into a one-to-one mapping with erroneous first-pass clusters [2309.12521]. Second, many systems still require a bounded speaker capacity. The 2021 unknown-speaker-count extension handled this by estimating speaker count with an external diarization system, padding with dummy i-vectors when \(\hat N < N\), and selecting the \(N\) longest non-overlapping speakers when \(\hat N > N\); with an 8-speaker model, DERs on LibriCSS were 12.4%, 11.3%, and 7.6% for 2-, 5-, and 8-speaker conditions, but applying a 5-speaker model to 8-speaker recordings caused catastrophic degradation to 42.8% DER [2108.03342].

Third, canonical TS-VAD is usually discriminative and profile-conditioned. Recent work has explored alternatives without fully discarding the framework. Flow-TSVAD introduced conditional flow matching into Seq2Seq-TSVAD, but showed that applying flow matching directly in the original binary label space failed badly, with 82.35% DER; mapping target-speaker activity sequences to a dense latent space before flow matching reduced CALLHOME DER to 11.21%, and a 3-sample DOVER-Lap ensemble further reduced it to 10.91% [2409.04859]. This suggests that generative modeling may be viable for diarization refinement, but only after learning a continuous latent geometry for activity sequences.

Several adjacent directions also clarify what TS-VAD is not. EEND-SAA is explicitly presented as an enrollment-less, main-speaker VAD system rather than a TS-VAD system: it does not use enrollment speech and defines the target as the speaker who talks more steadily and clearly. Its dual-attractor model reduced main-speaker DER from 6.63% to 3.61% relative to an SA-EEND baseline under the paper’s task definition, but the target is role-defined rather than identity-defined, so it is not a direct substitute for standard TS-VAD when the desired speaker is known in advance [2509.11957]. At the opposite end, the two-stage AVTSE system for edge devices uses a compact visual VAD front end that predicts visible-target activity from cropped lip video and then injects that hard activity cue into a lightweight extractor. Its VVAD stage reported 78.46% accuracy, 87.65% precision, and 83.96% recall, and the full system required only 1.89 GMacs and 1.36M parameters in the table, but this is a visual target-activity front end rather than a diarization-style profile-conditioned TS-VAD [2505.22229].

Taken together, these variants indicate that TS-VAD has become a family of target-conditioned framewise activity models rather than a single architecture. The common core remains stable: explicit conditioning on target identity or target role, frame-level activity prediction, and overlap-aware output structure. What changes across papers is the scope of the target set, the way conditioning is obtained, and the degree to which the system depends on first-pass speaker inventories, modality availability, or deployment constraints.

Source: https://www.emergentmind.com/topics/target-speaker-voice-activity-detection-ts-vad