Target Speaker Tagging Advances

• Target Speaker Tagging is a unifying abstraction that assigns speaker-specific labels to temporal units using enrollment audio, identity codes, or text cues.
• It applies varied methodologies—enrollment-based embeddings, frame-level features, speaker attractors, and semantic cues—to condition extraction, diarization, and ASR tasks.
• Evaluation metrics like DER, SI-SDRi, and WER highlight its impact on improving target speech detection, open-set identification, and overall system performance.

Searching arXiv for papers on Target Speaker Tagging and related target-speaker conditioning to ground the article. Target Speaker Tagging (TST) denotes the assignment of speaker-specific labels to temporal frames, time–frequency units, speech segments, or decoded words so that only the activity of a designated target speaker is retained, time-stamped, or further processed. Across recent work, the term covers several closely related formulations: personalized detection and time-stamping of target activity in continuous recordings, frame- or T–F-level target-consistency tagging inside extraction networks, target-conditioned word emission in ASR, and open-set labeling of diarized segments with rejection of unknown speakers. In all cases, the central object is an auxiliary target cue—derived from enrollment audio, learned identity codes, self-updated speaker slots, speaker attractors, or unaligned text—that biases the downstream model toward one speaker and away from interferers (Zhao et al., 2024).

1. Definitions and task boundaries

TST is distinct from both generic diarization and non-personalized VAD. General diarization answers “who spoke when” across all speakers, often via clustering or multi-label prediction, whereas TST focuses only on a single designated target speaker using that speaker’s enrollment and is therefore personalized. Non-personalized VAD detects speech versus non-speech irrespective of identity; TST is target-conditioned and must suppress non-target activity even in overlap regions (Zhao et al., 2024).

The notion broadens when different application layers are considered. In target speech extraction, TST may refer to identifying, at the frame or T–F unit level, which portions of a mixture belong to a known target speaker, with the resulting tags conditioning a separator (Zeng et al., 2024). In online diarization, TST becomes the maintenance of a persistent set of target-speaker embeddings whose activity is predicted block by block and updated over time without clustering (Wang et al., 2023). In target-speaker ASR, the term is often not used explicitly, but the operational behavior is equivalent: the system emits only the target speaker’s words and, when timestamps are enabled, aligns them to the mixture timeline (Ma et al., 2023). In long multi-conversation recordings, TST integrates diarization, verification, and identification into a single open-set task that must detect enrolled speakers and reject unknown ones (Lee et al., 12 Jun 2026).

This suggests that TST is best understood as a unifying abstraction rather than a single architecture. The common requirement is not a specific model family but the production of target-conditioned temporal decisions with speaker attribution preserved.

2. Target cues and tagging representations

Recent work uses several distinct target representations. The most conventional is the enrollment-based speaker embedding extracted from pre-recorded target speech by a speaker encoder such as ResNet34, x-vector, ECAPA-TDNN, or SSL-based models. In continuous target speech extraction, for example, a ResNet34-based speaker encoder maps enrollment speech to a 256-dimensional target embedding that conditions both A-TSVAD and pBSRNN (Zhao et al., 2024). In cross-task target-speaker processing, enrollment embeddings are compared against ideal one-hot identity codes, and the paper reports that speaker verification performance is somewhat unrelated to TS task performance while one-hot codes outperform enrollment-based embeddings in the speaker-closed setting (Ashihara et al., 2024).

A second line replaces global embeddings with frame-level target features. USEF-TSE is explicitly embedding-free: mixture frames query enrollment features through cross multi-head attention, producing a target-speaker-consistent feature $H$ or $E_{\mathrm{spk}}$ aligned to the mixture length (Zeng et al., 2024). Rather than compressing enrollment speech into a fixed-dimensional vector, the model retains full temporal content from the reference. The paper states that attention weights can also be viewed as an explicit soft tag, although in the reported implementation the effect is achieved implicitly via $E_{\mathrm{spk}}$ and fusion.

A third representation is the speaker attractor. In SATTS, speaker attractors learned by a separation model tag T–F bins through cosine similarity and softmax assignment; the selected target attractor is then reused as a robust, mixture-aware conditioning vector for zero-shot TTS (Goswami et al., 2022). Here, tagging and representation learning are inseparable: the same attractor both identifies target-consistent bins and summarizes the target speaker.

A fourth representation is semantic rather than biometric. pTSE-T conditions extraction or stream selection on unaligned text cues derived from slides or posters. TPE uses a frozen CLAP text encoder to obtain a semantic embedding that modulates audio features through hierarchical FiLM, while TSR associates separated streams with text prompts by cross-attention and contrastive learning (Jiang et al., 2024). In this formulation, the “target” is specified by content similarity rather than enrollment audio.

Finally, some systems generate target tags online from the recording itself. OTS-VAD extracts frame-level embeddings for each block, predicts speaker activity conditioned on previously estimated slot embeddings, and then updates those embeddings using high-confidence frames from the current block (Wang et al., 2023). This removes the need for a priori clustering-based target embeddings and fixes speaker-slot identity over time.

3. Core architectures and operationalizations

A compact comparison of representative operationalizations is useful before examining them in detail.

Operationalization	Tagging unit	Representative work
A-TSVAD for continuous TSE	Frame-level target activity labels	(Zhao et al., 2024)
CMHA-based USEF-TSE	Mixture frames or T–F units aligned to enrollment	(Zeng et al., 2024)
Prompt-tuned target-speaker ASR	Target words and optional timestamps	(Ma et al., 2023)
SANET attractors in SATTS	Time–frequency bins	(Goswami et al., 2022)
OTS-VAD	Per-speaker frame activity over online blocks	(Wang et al., 2023)
pTSE-T	Target segments via extracted speech or selected stream	(Jiang et al., 2024)
TST-Bench full pipeline	Reference-anchored speech segments with open-set IDs	(Lee et al., 12 Jun 2026)

In Continuous TSE, A-TSVAD implements TST directly. The mixture is framed with window length 1.5 s and hop 0.25 s; each frame is encoded with the same speaker encoder used for enrollment, concatenated with the target embedding, and processed by convolutional blocks followed by two Transformer blocks with residual connections and layer normalization. The network outputs a posterior $\hat{p}$ of target presence, which is thresholded at $\alpha = 0.025$ and then morphologically smoothed by dilation and erosion to generate timestamps of target activity (Zhao et al., 2024). Those timestamps can gate pBSRNN in a cascaded extraction pipeline.

USEF-TSE shifts the tagging locus inside the separator. In the time-domain instantiation, a shared Conv1d encoder maps mixture and enrollment waveforms to latent sequences, cross-attention uses mixture features as queries and enrollment features as keys and values, and the resulting $E_{\mathrm{spk}}$ is fused with mixture encoding via FiLM before SepFormer predicts a mask. In the T–F instantiation, shared Conv2d encoding of stacked real-imaginary STFT features is followed by cross-attention based on TF-GridNet’s attention block, channel concatenation, TF-GridNet separation, TConv2d channel reduction, and iSTFT reconstruction (Zeng et al., 2024). The key architectural point is that the tag has the same temporal extent as the mixture and therefore functions as a one-to-one frame or T–F unit conditioner.

Prompt-tuned Whisper operationalizes TST in sequence-to-sequence ASR. A 512-dimensional x-vector extracted from target enrollment is projected to model width and concatenated with encoder-side soft prompts and the acoustic features, while decoder-side prompts are inserted into special-token embeddings. Deep prompting and residual prompt reparameterization extend the conditioning to intermediate layers, with all base Whisper parameters frozen (Ma et al., 2023). The model does not emit explicit binary tags, but the decoded target transcript and optional timestamps constitute target-conditioned tagging at the word and segment levels.

In SATTS, SANET learns T–F embeddings and speaker attractors from separation training. Tagging is performed through cosine similarity $s_{t,f,k} = \cos(e_{t,f}, a_k)$ and speaker-wise softmax masks $M_{t,f,k}$. The target attractor selected at inference by spherical K-means is injected as global conditioning throughout the VITS backbone: posterior encoder, flow, and decoder (Goswami et al., 2022). A plausible implication is that this architecture turns TST into a reusable speaker representation problem rather than only a detection problem.

OTS-VAD uses an explicitly online TST loop. A front-end produces frame-level embeddings at 0.08 s resolution; a back-end Conformer followed by BiLSTM predicts activity probabilities for a fixed set of speaker slots; and slot embeddings are updated by averaging or accumulating high-confidence frame embeddings across blocks. New speakers are detected when all speaker probabilities in the last $m$ frames fall below a lower threshold, after which the next unused slot is seeded (Wang et al., 2023). Because each slot persists through time, speaker permutation inconsistency is avoided.

pTSE-T introduces a non-enrollment cue. In TPE, waveform features are encoded by 1-D convolution, text prompts are embedded by frozen CLAP, repeated FiLM fusion generates text-conditioned features, and a DPRNN mask estimator produces a target waveform whose VAD-positive regions can be tagged as target. In TSR, a DPRNN first separates $I$ streams, then cross-attention between speech and text embeddings yields an association score $E_{\mathrm{spk}}$0 that identifies which stream corresponds to the prompt (Jiang et al., 2024).

4. Mathematical formulations and decision rules

Several mathematically distinct tagging rules appear across the literature. In attention-based TST, a standard scaled dot-product form is common. For USEF-TSE, if $E_{\mathrm{spk}}$1 and $E_{\mathrm{spk}}$2, then for head $E_{\mathrm{spk}}$3,

$E_{\mathrm{spk}}$4

with attention weights

$E_{\mathrm{spk}}$5

and tagged feature

$E_{\mathrm{spk}}$6

The multi-head output $E_{\mathrm{spk}}$7 has the same temporal length as the mixture and therefore constitutes a per-frame tag (Zeng et al., 2024). A-TSVAD uses the same core attention mechanism inside its Transformer blocks, but its explicit output is the per-frame posterior $E_{\mathrm{spk}}$8 of target activity (Zhao et al., 2024).

A binary target decision can be produced by thresholding. In A-TSVAD, the rule is

$E_{\mathrm{spk}}$9

with $E_{\mathrm{spk}}$0, followed by morphological smoothing (Zhao et al., 2024). In explicit TST adaptations of USEF-TSE, per-frame tag scores can be derived from attention maps by pooling across enrollment time, such as $E_{\mathrm{spk}}$1 or $E_{\mathrm{spk}}$2, though this is presented as an adaptation rather than the reported training objective (Zeng et al., 2024).

Attractor-based TST uses similarity to cluster prototypes. In SATTS, cosine similarities between each T–F embedding and each attractor are converted to masks by speaker-wise softmax, and target-only tagging uses the target attractor and its mask to score bins as target versus non-target (Goswami et al., 2022). This is a T–F assignment rule rather than a segment-level detection rule.

Open-set TST in long recordings is formulated as thresholded identification. Given segment embedding $E_{\mathrm{spk}}$3 and enrollment embeddings $E_{\mathrm{spk}}$4, the decision rule is

$E_{\mathrm{spk}}$5

where $E_{\mathrm{spk}}$6 is typically cosine similarity with AS-Norm (Lee et al., 12 Jun 2026). This combines verification-style rejection and identification over a gallery.

Target-speaker conditioning itself can also be parameterized differently. In the unified investigation of target-speaker processing, Hadamard product conditioning $E_{\mathrm{spk}}$7 consistently outperformed addition and concatenation across TS-ASR, TSE, and p-VAD (Ashihara et al., 2024). The same paper introduces inference-time gradient-based optimization of the tag $E_{\mathrm{spk}}$8, arguing that the optimal tag depends on the input mixture rather than only on the target identity.

5. Training objectives, metrics, and evaluation regimes

Training objectives reflect the granularity at which TST is used. When the output is a target activity posterior, binary cross-entropy over frames is standard, as in A-TSVAD and OTS-VAD (Zhao et al., 2024, Wang et al., 2023). When TST is embedded in extraction, the dominant objective is SI-SDR maximization or equivalently negative SI-SDR minimization, used in USEF-TSE, TPE in pTSE-T, and the separation component of SATTS (Zeng et al., 2024, Jiang et al., 2024, Goswami et al., 2022). When TST is embedded in ASR, the loss is sequence cross-entropy or CTC, as in prompt-tuned Whisper and the unified TS-ASR study (Ma et al., 2023, Ashihara et al., 2024).

Reported evaluation metrics vary accordingly. Continuous TSE uses diarization error rate (DER), Jaccard error rate (JER), interference leakage (INT), SI-SNR, and PESQ (Zhao et al., 2024). USEF-TSE reports SI-SDRi, SDRi, and PESQ on WSJ0-2mix, WHAM!, and WHAMR! (Zeng et al., 2024). Prompt-tuned Whisper reports only WER, relying on Whisper’s normalization pipeline (Ma et al., 2023). pTSE-T reports SI-SDRi, SDRi, PESQi, STOIi, and extraction accuracy (Jiang et al., 2024). OTS-VAD reports DER and JER with latency tied to block shift (Wang et al., 2023). TST-Bench defines two scenarios: Scenario 1 for diarization with DER and JER, and Scenario 2 for full TST using DIR@FAR at fixed false-alarm operating points (Lee et al., 12 Jun 2026).

The evaluation regime itself is a substantive part of the topic. TST-Bench formalizes a setting with long 20–60 minute sessions, 8–30 speakers per session, 150 enrolled and 200 unknown speakers overall, and reference-anchored non-overlapping single-speaker evaluation regions of duration at least 1 s for full-pipeline scoring (Lee et al., 12 Jun 2026). This differs from personalized extraction or ASR benchmarks because unknown rejection and global cross-session labeling become first-class requirements.

A plausible implication is that no single metric suffices across TST variants. DER and JER characterize target activity tracking; SI-SDR and related measures characterize downstream separability; WER characterizes target-conditioned lexical fidelity; and DIR@FAR captures open-set speaker tagging under controlled false alarms.

6. Empirical results, comparative findings, and limitations

The empirical record shows that TST quality depends strongly on the representation used for the target cue and on how tightly tagging is integrated with downstream processing. In Continuous TSE, the sequential cascade A-TSVAD $E_{\mathrm{spk}}$9 pBSRNN achieved the best combined diarization and extraction performance, with DER 26.5, JER 16.2, INT 41.3, SI-SNR 10.4 dB, and PESQ 2.91, outperforming both parallel integration and the reverse cascade (Zhao et al., 2024). The paper also reports that 1.5 s windows with Transformer blocks yield the best A-TSVAD behavior among the tested settings.

USEF-TSE reports state-of-the-art target speaker extraction on standard benchmarks. On WSJ0-2mix, USEF-SepFormer achieved SI-SDRi 19.9 dB and USEF-TFGridNet achieved SI-SDRi 23.3 dB, while on WHAM! and WHAMR! USEF-TFGridNet achieved 17.6 dB and 16.1 dB respectively, outperforming the cited baselines. The ablations indicate that deeper CMHA, matched attention heads between CMHA and separator, and FiLM fusion in the time domain improve results, while T–F backbones are more robust under reverberation and same-gender difficulty remains higher though reduced (Zeng et al., 2024).

Prompt-tuned Whisper demonstrates that TST can be implemented with a frozen foundation ASR model using only a small task-specific parameter budget. Deep prompting on Whisper-medium uses 1.31M parameters, compared with 762.85M for full fine-tuning, and achieved 13.89% WER on Libri2Mix test-clean and 29.81% on test-both; LoRA obtained lower WER in the reported comparisons, but prompting preserved Whisper’s inverse text normalization and timestamping behavior (Ma et al., 2023). The paper further shows that encoder prompts are crucial, while decoder-only prompting is clearly insufficient.

The unified cross-task study of speaker representations reaches a result that complicates common assumptions: speaker verification EER is a weak proxy for target-speaker utility. ECAPA-TDNN-c1024 yields the best ASV EER among the listed encoders, yet does not dominate TS-ASR, TSE, or p-VAD; one-hot speaker codes perform best in the speaker-closed condition across all three tasks, and gradient-based optimization of the tag at inference yields approximately 31–32% relative WER reduction on clean TS-ASR mixtures (Ashihara et al., 2024). This suggests that task-optimal tagging and verification-optimal embedding are not the same problem.

Online TST also shows that explicit slot persistence can replace clustering in some regimes. OTS-VAD achieves DER 13.31% on DIHARD III and 4.50% on multi-channel AliMeeting test at the best reported settings, with latency equal to block shift and without permutation instability (Wang et al., 2023). The paper attributes much of the performance to self-generated target embeddings and SV-pretrained front-ends.

Non-enrollment text cues are viable but cue quality matters. On MMSpeech-2mix, pTSE-T’s TPE achieved SI-SDRi 12.16 dB, SDRi 12.66 dB, PESQi 0.830, STOIi 0.150, and 96.46% extraction accuracy. Ablations show that removing FiLM or replacing CLAP-based text conditioning with weaker alternatives degrades performance, while mismatched prompts can be worse than no prompt in TSR (Jiang et al., 2024). The limitation is explicit in the paper: ambiguous or generic prompts and semantically similar competing speakers remain difficult.

At the benchmark level, TST-Bench shows that open-set TST introduces challenges not captured by diarization or verification alone. The baseline full pipeline achieved DIR@FAR values of 88.79% at 0.5%, 93.00% at 1%, 96.80% at 5%, and 97.61% at 10%. Over-clustering improved strict-operating-point performance relative to balanced clustering, small non-overlapping segment margins helped, and top-$\hat{p}$0 short-utterance compensation improved tagging, whereas aggressive label-based assignment was fragile under under-clustering on the synthetic benchmark (Lee et al., 12 Jun 2026). This directly contradicts the intuition that better diarization in the conventional sense always implies better TST.

Across these studies, the main limitations recur. Enrollment quality and availability remain important for enrollment-based systems; attention-based methods incur $\hat{p}$1 cost with long references; same-speaker or same-gender overlaps remain difficult; prompt- or text-conditioned systems depend on cue fidelity; and open-set TST requires careful threshold calibration and overlap-aware diarization (Zeng et al., 2024, Zhao et al., 2024, Ma et al., 2023, Jiang et al., 2024, Lee et al., 12 Jun 2026). A plausible implication is that future progress will come less from a single universal architecture than from better alignment between target cue, operating regime, and evaluation objective.

7. Historical positioning and research trajectory

The recent literature places TST at the intersection of several older threads: speaker-conditioned separation, TS-VAD and diarization refinement, speaker-attractor separation, zero-shot speaker adaptation, and personalized ASR. Earlier embedding-conditioned approaches such as SpeakerBeam-, VoiceFilter-, or ECAPA-conditioned systems rely on global speaker vectors and a separate speaker recognition model; recent embedding-free and task-specific approaches challenge that design by arguing that verification-optimized embeddings may underuse temporal and contextual detail needed for tagging or extraction (Zeng et al., 2024).

At the same time, TST is moving from component-level personalization toward full-pipeline evaluation. Continuous TSE addresses variable overlap and target absence rather than only always-active overlap cases (Zhao et al., 2024). Online diarization work removes offline clustering and speaker-order ambiguity (Wang et al., 2023). Foundation-model adaptation shows that target conditioning can be integrated into large frozen ASR systems while retaining timestamping and normalization behavior (Ma et al., 2023). Semantic conditioning extends the task to settings where identity cues are unavailable but topical cues are present (Jiang et al., 2024). TST-Bench then consolidates these tendencies into an explicit benchmark with global identities, unknown rejection, and long-session operating conditions (Lee et al., 12 Jun 2026).

This suggests that “Target Speaker Tagging” is becoming a common evaluative language for a family of personalized speech tasks that were previously studied separately. The most stable conclusion across the cited work is not that one conditioning mechanism is universally superior, but that target specification must be aligned with the downstream objective: frame-level activity tagging for continuous extraction, sequence-aware conditioning for separation, persistent slot embeddings for online diarization, task-tuned tags for ASR and p-VAD, semantic cues when enrollment is absent, and thresholded open-set scoring when global labeling and unknown rejection are required.