Who Spoke When in Multi-Conversation: Target Speaker Tagging Task and Benchmark

Abstract: We present target speaker tagging (TST), a task that integrates speaker diarization, verification, and identification into a unified workflow for multi-speaker conversations. Given long recordings and pre-enrolled speakers, TST detects and labels speech segments of known speakers while rejecting unknown ones. Despite its practical importance, research has been limited by the absence of suitable evaluation resources. To address this, we introduce TST-Bench, a large-scale synthetic benchmark with over 150 enrolled speakers, 300 sessions of 20-60 minutes, and reference annotations with global speaker labels. We define an evaluation protocol encompassing diarization and full-pipeline scenarios. Experiments on both real and synthetic data show that TST poses challenges not captured by conventional benchmarks, and that dedicated system design yields significant gains over naive integration of existing solutions. The benchmark dataset and evaluation protocols are publicly released.

• The paper presents Target Speaker Tagging (TST) as a unified framework combining diarization, identification, and rejection tasks.
• It introduces TST-Bench, a synthetic, large-scale benchmark with controlled session parameters and perfect global annotations.
• Empirical results demonstrate that over-clustering and short-utterance aggregation significantly improve tagging accuracy in diverse multi-session settings.

Target Speaker Tagging: Unified Speaker Recognition and its Benchmark

Background and Motivation

Speaker recognition research has historically segmented fundamental tasks such as speaker diarization, verification, and identification into independent problem domains. These areas, while leveraging intersecting technologies like speaker embeddings, address different operational scopes: diarization labels speakers within a session but does not assign external identities; verification confirms if two utterances originate from the same person; identification assigns fixed-known labels. However, real-world applications increasingly demand integrated solutions capable of dealing with large enrolled populations, multi-session recordings, and rejection of out-of-domain speakers, which creates evaluation scenarios not adequately covered by existing datasets or methodologies.

Current benchmarks suffer from several limitations, including session-local labels that prevent cross-session identification, small and often unrealistic speaker populations per session, and a lack of evaluation protocols unifying diarization with identity and rejection accuracy. The absence of a robust evaluation framework stymies progress in the engineering of end-to-end systems suitable for multi-speaker, multi-session scenarios.

Target Speaker Tagging Task Definition

Target Speaker Tagging (TST) is proposed as a comprehensive formulation for integrated speaker recognition in practical environments. In the TST task, given a long recording and a gallery of enrolled speakers:

• The system must segment audio into single-speaker regions (the diarization component).
• It must assign consistent, global speaker identities to those segments corresponding to enrolled (target) speakers.
• It should accurately reject all segments spoken by unknown (non-enrolled) speakers.

This formulation cannot be addressed by naively cascading existing diarization, identification, and verification modules. Specific error modes such as under-clustering in diarization (i.e., grouping different speakers together) are particularly detrimental, because once segment embeddings are contaminated, downstream modules cannot recover the partitioning. Conversely, over-clustering errors (where a single speaker is split into multiple clusters) can sometimes be mitigated by the identification component.

Figure 1: Overview of the target speaker tagging system, highlighting the relationship between diarization, enrollment, session specification, and speaker assignment.

TST-Bench: Large-Scale Synthetic Benchmark for TST

Recognizing the deficits in existing corpora, the authors introduce TST-Bench, a synthetic benchmark constructed from single-speaker audiobook readings (Multilingual LibriSpeech, MLS). It offers:

• Over 150 enrolled speakers, expandable to 350, and 300 sessions of 20–60 minutes each.
• Per-session speaker populations of 8–30, with both enrolled and unknown speakers.
• Systematic control over session length, background noise (ambient recordings, 0–10 dB SNR), and speaker turn layout (sampled via Dirichlet distributions for realistic participation skew).
• Perfect, global RTTM annotations and deterministic simulation of the enrollment process.

The synthetic design allows detailed empirical control and reproducibility, enabling ablation studies and performance scaling with respect to speaker density, session duration, and noise.

Figure 2: TST-Bench synthesis pipeline, illustrating source data preparation, session planning, and audio mixing with corresponding annotations.

Evaluation Methodology

The TST evaluation protocol critically addresses unified assessment across diarization, identification, and rejection metrics:

• DIR (Detection and Identification Rate): Proportion of target segments correctly detected and identified above a calibrated similarity threshold.
• FAR (False Alarm Rate): Proportion of non-target segments falsely assigned to an enrolled identity.
• DER (Diarization Error Rate): Classical segmentation accuracy, augmented with consideration for the unique impact of over-/under-clustering.

Evaluation is performed under two scenarios:

1. Isolated assessment of diarization (DER).
2. Full-pipeline analysis, where diarized segments are mapped to identification outcomes via a consistent, reference-aligned evaluation protocol that ensures comparability regardless of segmentation granularity.

This protocol naturally penalizes under-clustering more than over-clustering due to its irreversible effect on tagging accuracy.

Empirical Results and Analysis

Strong empirical findings include:

• Baseline system achieves DIR@FAR=0.5% of 88.79% on TST-Bench and 94.51% on ICSI, illustrating intrinsic difficulty due to scale and diversity.
• Over-clustering in diarization is shown to be preferable for TST, as it enables possible correction at the identification stage, whereas under-clustering contaminates embeddings irreparably.
• Short-utterance compensation—combining segment embeddings within the same label, especially using top-N similarity selection—leads to consistent DIR improvements, particularly for short segments and large speaker pools.

Numerical results unambiguously demonstrate that margin expansion around speech boundaries (+0.1–0.25s) offers small accuracy increases by stabilizing edge embeddings, but excessive expansion risks capturing neighboring speakers’ audio in high-density turn-taking, especially in real meeting corpora.

Comparison across ICSI (real) and TST-Bench (synthetic) shows consistency in the relative effects of system design decisions, supporting the transferability of benchmark insights to real environments, despite the latter’s read speech and artificial mixing.

Implications and Future Directions

The TST framework and TST-Bench dataset advance the field by enabling realistic, challenging, and reproducible evaluation aligned with deployment demands in call centers, transcription services, and group conversational analytics. The research demonstrates the criticality of integrating diarization and identification pipelines thoughtfully, rather than maximizing isolated component accuracy.

The work supports the bold claim that optimizing diarization alone is insufficient for practical speaker recognition, and that over-clustering and short-utterance aggregation should become standard tactics in system design. Given that synthetic data cannot fully replicate conversational spontaneity or environmental complexity, future work will likely need to blend TST-Bench with large-scale, naturally occurring meeting data augmented with global speaker labels and densely overlapping participant sets.

The availability of TST-Bench is expected to catalyze research into scalable, unified architectures that optimize for the joint diarization-identification-rejection objective, as well as facilitate robust meta-evaluation of self-supervised and data-efficient learning techniques in the complex multi-conversation setting.

Conclusion

This paper introduces the Target Speaker Tagging (TST) task and TST-Bench benchmark, providing a unified framework and reproducible testbed for evaluating integrated speaker recognition systems in large-scale, multi-conversation audio. The approach highlights the inadequacy of traditional, component-wise metrics and emphasizes the importance of coordinated system design. The benchmark and protocol enable granular, controlled investigation of errors and their propagation, and offer a foundation for methodological advances in speaker diarization and open-set identification for real-world deployments.