TST-Bench: Target Speaker Tagging Benchmark

• TST-Bench is a synthetic benchmark that defines target speaker tagging by integrating diarization, verification, and identification in long, open-set recordings.
• The benchmark is built from MLS recordings, creating controlled multi-session scenarios with global labels for enrolled and unknown speakers.
• Empirical findings show that over-clustering improves tagging performance and that a reference-anchored protocol effectively mitigates the impact of segmentation errors.

TST-Bench is a benchmark for target speaker tagging (TST), a task defined for long multi-speaker recordings with a set of pre-enrolled target speakers. In this setting, a system must detect and label speech segments belonging to known speakers while rejecting speech from unknown speakers. The benchmark was introduced to support evaluation of this integrated problem at scale, with a focus on long recordings, global speaker identities across sessions, open-set conditions, and end-to-end evaluation from diarization to target-speaker assignment (Lee et al., 12 Jun 2026).

1. Task definition and relation to adjacent speaker tasks

Target speaker tagging combines elements of speaker diarization, speaker verification, and speaker identification into a single workflow. The input is a long recording with multiple speakers, together with a subset of pre-enrolled speakers. The required output is a segmentation of the recording into single-speaker regions, assignment of globally meaningful speaker identities to regions spoken by enrolled speakers, and rejection of all other speech as non-target (Lee et al., 12 Jun 2026).

This formulation differs from conventional speaker diarization because diarization normally produces only anonymous, session-local labels such as local speaker clusters. TST instead requires global identity assignment across sessions for enrolled speakers, and it explicitly includes unknown-speaker rejection. It also differs from speaker verification and standard speaker identification because those tasks are usually formulated on pre-segmented utterances, whereas TST must first operate on long conversational recordings and then perform open-set assignment on the resulting segments.

A central point in the task definition is that TST is not merely diarization followed by identification. The paper emphasizes an asymmetry between clustering errors: under-clustering, where multiple true speakers are merged into one cluster, can irreversibly corrupt downstream speaker representation and severely harm identification, whereas over-clustering, where one true speaker is split across multiple clusters, is often less damaging because multiple clusters may still be assigned to the same enrolled speaker. This suggests that optimization for TST is not equivalent to optimization for diarization metrics alone.

2. Benchmark structure and dataset composition

TST-Bench is a large-scale synthetic benchmark built from real single-speaker source recordings and mixed into multi-speaker sessions with global annotations. Its stated purpose is to provide a benchmark that supports global speaker labeling across sessions, a large enrolled population, long-form multi-speaker recordings, and an evaluation protocol for both diarization and the full TST pipeline (Lee et al., 12 Jun 2026).

The benchmark statistics reported in the paper are as follows.

Aspect	Value
Source corpus	MLS (English)
Total speakers	350
Enrolled speakers	150
Unknown speakers	200
Sessions	300
Session duration	20–60 minutes
Speakers per session	8–30
Unknown speakers per session	0–10
Enrollment duration	20 s per enrolled speaker
Audio format	16 kHz mono
Annotation format	RTTM with global speaker labels
Total audio	approximately 200 hours

The benchmark uses global speaker labels, so the same person retains the same identifier across sessions. This is a defining property of the resource, since TST aims to recognize known speakers globally rather than assign only session-local anonymous labels. Unknown speakers are included explicitly: there are 200 unknown speakers in the global pool, each session includes 0 to 10 unknown speakers, and their speech must be tagged as non-target rather than matched to the enrolled gallery.

The paper also stresses the importance of per-session speaker count. TST-Bench supports 8–30 speakers per session, which is intended to create a more demanding open-set identification regime than typical meeting corpora with fewer participants. This suggests that TST-Bench is designed not only as a larger dataset, but as a benchmark whose session structure amplifies the interaction between diarization errors and open-set tagging.

3. Construction pipeline

The benchmark is synthesized from single-speaker MLS recordings. To ensure sufficient source material, the authors randomly select 350 English speakers, each with at least one hour of speech. Because MLS recordings are crowdsourced and may encode speaker-specific channel conditions, the source recordings are enhanced with Resemble Enhance denoising before synthesis (Lee et al., 12 Jun 2026).

Word-level time boundaries are obtained using the Montreal Forced Aligner (MFA). Source speech chunks are then formed by grouping consecutive voiced intervals, using silence gaps greater than 0.3 s as boundaries. The full speaker inventory is partitioned into 150 enrolled speakers and 200 unknown speakers. For each enrolled speaker, approximately 20 seconds of speech-only audio is selected as enrollment material and excluded from evaluation.

Session generation proceeds in four stages. In session planning, each session duration is sampled uniformly from 20 to 60 minutes, the number of speakers is sampled from 8 to 30, each session includes at least one enrolled speaker, each enrolled speaker appears in 10 to 30 sessions, and each session includes 0 to 10 unknown speakers. In turn layout, turns are arranged sequentially with inter-turn gaps of 0.15 to 2.5 s and turn durations of 0.5 to 15 s. Speaker participation proportions are sampled from a symmetric Dirichlet distribution with concentration parameter $\alpha = 3.0$.

In audio mixing, speech segments are drawn from the source corpus without replacement so that no speech content is reused across sessions. Each session is mixed onto a continuous ambient-noise track, with session-level SNR sampled uniformly from 0 to 10 dB. Ambient noises are collected from Freesound, restricted to candidate recordings of at least 3 minutes, filtered with an in-house VAD to keep recordings with less than 5% detected speech, and repeated with fade-in and fade-out when shorter than session duration. In annotation generation, each session is labeled with an RTTM file containing onset time, duration, and a global speaker label, with each label additionally marked as enrolled or unknown.

The paper is explicit that TST-Bench is synthetic rather than a real-conversation corpus. It also states that this synthesis does not fully simulate reverberation, crosstalk, far-field microphone effects, or realistic meeting-room acoustics. A plausible implication is that the benchmark prioritizes controlled scale, exact labeling, and reproducibility over full acoustic realism.

4. Evaluation protocol and metrics

TST-Bench defines two evaluation scenarios: speaker diarization and the full TST pipeline (Lee et al., 12 Jun 2026).

For the full pipeline, the paper defines Detection and Identification Rate (DIR) and False Alarm Rate (FAR). Let $\mathbf{D}$ denote the set of evaluated single-speaker segments, $\mathbf{D}_T$ the target segments from enrolled speakers, and $\mathbf{D}_N$ the non-target segments from unknown speakers. With threshold $\theta$, the thresholded tagging decision is

$$\text{id}_{\text{tag}(d, \theta)} = \begin{cases} \widetilde{\text{id}_{\text{tag}(d)}} & \text{if } \text{score}_{\max}(d) \ge \theta \ \text{non-target} & \text{otherwise.} \end{cases}$$

The benchmark then reports

$$\text{DIR}(\theta) = \frac{ \left| \left\{ d_t \mid \text{id}_{\text{tag}(d_t,\theta)} = \text{id}_{\text{ans}(d_t)},\ d_t \in \mathbf{D}_T \right\} \right| }{|\mathbf{D}_T|}$$

and

$$\text{FAR}(\theta) = \frac{ \left| \left\{ d_n \mid \text{score}_{\max}(d_n) \ge \theta,\ d_n \in \mathbf{D}_N \right\} \right| }{|\mathbf{D}_N|}.$$

The principal reported end-to-end metric is therefore DIR@FAR at several operating points.

A notable methodological contribution is the reference-anchored evaluation protocol. Because different diarization systems produce different segmentations, the benchmark first selects a fixed set of reference evaluation segments from the reference RTTM: these must be non-overlapping, single-speaker, and have duration at least

$\tau_{\min} = 1 \text{ s}.$

For each reference segment, the system-generated segment with the longest temporal overlap is identified, and its label is transferred to the reference segment for scoring. This design makes system comparisons segmentation-independent in a controlled way. It also naturally penalizes under-clustering: if one long system segment spans several reference regions, only those matching the assigned identity are correct. If a reference segment has no overlapping system output, it remains in the denominator and lowers DIR.

For diarization-only evaluation, the benchmark reports DER and also JER in the main diarization results.

5. Baseline system design

The paper evaluates TST-Bench using a pipeline that combines a diarization frontend with open-set speaker identification (Lee et al., 12 Jun 2026).

The diarization system is built on a High-Resolution Embedding Extractor (HEE) trained on VoxCeleb1 and VoxCeleb2, followed by dimensionality reduction, attention-based aggregation, and spectral clustering. Open-set speaker identification uses ECAPA-TDNN, also trained on VoxCeleb1 and VoxCeleb2, with a 4-second window, 1.5-second shift, and 256-dimensional speaker embeddings. If a segment or enrollment utterance yields multiple embeddings, score-level average pooling is used. Similarity scoring is performed with cosine similarity and adaptive symmetric normalization (AS-Norm), using a cohort of 2,000 speakers randomly selected from VoxBlink2 and adaptation size 20.

The paper characterizes this as a largely naive integration of diarization and open-set identification, and then studies several TST-specific modifications. These include clustering tendency adjustment with baseline, under-clustered, and over-clustered configurations; segment margin extension, where diarized segments are symmetrically expanded before embedding extraction; and short-utterance compensation, which aggregates evidence from other segments with the same diarization label using strategies such as Top-1, Top-2, Top-3, and label-based aggregation.

This suggests that the benchmark is not only a dataset release but also a framework for testing how speaker-tagging performance depends on choices that are largely invisible in traditional verification-style benchmarks.

6. Empirical findings

On TST-Bench diarization, the baseline system achieves DER = 8.70%, decomposed into Miss = 6.23%, False alarm = 0.35%, and Speaker confusion = 2.12%, with JER = 15.66% (Lee et al., 12 Jun 2026). The dominant error source is missed speech, which the paper attributes mainly to overlapping speech and the lack of explicit overlap handling in the baseline diarization system.

For the full TST pipeline on TST-Bench, the baseline results are:

Operating point	DIR
FAR = 0.5%	88.79%
FAR = 1%	93.00%
FAR = 5%	96.80%
FAR = 10%	97.61%

The clustering-tendency experiment yields one of the paper’s main conclusions. On TST-Bench, the baseline gives DIR@FAR=0.5% = 88.79%, the under-clustered configuration gives 86.75%, and the over-clustered configuration gives 89.46%. The paper therefore concludes that over-clustering gives the best TST result, whereas under-clustering hurts most, even though over-clustering increases speaker confusion relative to the baseline. This directly supports the claim that TST imposes design pressures not captured by conventional diarization objectives.

The segment-margin experiment shows only modest changes. On TST-Bench, 0.1 s margin gives 89.05% at FAR 0.5%, compared with 88.79% for no margin, while larger margins yield little additional gain. The interpretation given is that tightly cropped diarization segments can lose useful speaker information near boundaries.

For short-utterance compensation, Top-N aggregation improves results consistently on TST-Bench. The reported values at FAR 0.5% are 88.95% for Top-1, 88.94% for Top-2, and 89.03% for Top-3, compared with 88.79% without compensation. The label-based strategy performs poorly at strict operating points, dropping to 81.82% at FAR 0.5%, which the paper attributes to contamination from diarization errors, especially under-clustering.

The paper also evaluates on real ICSI meetings. The direction of the main findings remains consistent, but the effects are smaller. A plausible implication, explicitly suggested in the paper, is that TST-Bench’s larger speaker counts per session make the benchmark more sensitive to interactions between diarization and identification than smaller real-meeting corpora.

7. Scope, limitations, and significance

TST-Bench is presented as a response to the absence of suitable evaluation resources for target speaker tagging, but the paper is also explicit about its limitations (Lee et al., 12 Jun 2026). The source speech comes from MLS, which consists of read speech rather than spontaneous conversation. Session turn-taking is algorithmically generated and does not fully reproduce backchannels, interruptions, floor-holding behavior, or richer multi-party conversational dynamics. The acoustic mixing process also does not fully simulate reverberation, crosstalk, far-field capture, or realistic meeting-room acoustics. In addition, the paper does not specify an official train/dev/test split in the conventional sense; the benchmark is framed primarily as an evaluation resource.

Within those limits, the benchmark’s significance lies in two areas. First, it supplies a controlled large-scale resource with global speaker labels, 150 enrolled speakers, 200 unknown speakers, 300 sessions, and a reference-anchored protocol for evaluating both diarization and end-to-end TST. Second, it exposes properties of the task that conventional speaker benchmarks do not make salient, especially the asymmetry between under-clustering and over-clustering and the importance of handling short diarization-driven segments under open-set conditions.

In that sense, TST-Bench is best understood not simply as a new dataset, but as an attempt to define a distinct evaluation regime for multi-session, multi-speaker, open-set target speaker tagging with global speaker identities.