Speaker Registration Mechanism

• Speaker Registration Mechanism is a system that registers speaker identities by converting speech into persistent embeddings through defined enrollment and pooling processes.
• It employs methods like recursive attentive pooling, online action-space expansion, and multimodal token conditioning to adaptively update and manage speaker states.
• These mechanisms are crucial in applications such as diarization, speaker-attributed ASR, and beamforming, enhancing accuracy and operational efficiency in dynamic environments.

Searching arXiv for the cited papers and related speaker registration work. A speaker registration mechanism is the set of procedures by which a system establishes speaker identities, converts speaker evidence into reusable representations, and conditions later inference on those representations. In recent work, registration is not a single fixed operation: it appears as fixed-set enrollment with reference embeddings, recursive or attentive pooling that determines which parts of speech define the stored representation, online creation of new speaker identities during diarization, multimodal speaker tokens injected into LLMs, and visual-anchor-based discovery of speakers in open-world video. Across these formulations, the registered object may be a vector template, a dynamically created action or arm, a prompt-like enrollment segment, a multimodal token sequence, or a prototype tied to a visual cluster (Morrone et al., 2024, Lin et al., 2020, Yin et al., 8 Aug 2025, Huang et al., 17 Mar 2026).

1. Conceptual scope and system forms

The literature treats speaker registration as a mechanism that links incoming speech to persistent speaker-specific state. In configuration-driven speaker identification systems, that state is a fixed set of reference embeddings associated with known identities. In fully online diarization, it is an extendable action space in which new identities are created only when feedback confirms a previously unseen speaker. In multimodal LLM-based systems, registration becomes an optional modality consisting of speaker embeddings, names, and special delimiter tokens. In open-world audiovisual diarization, it begins from unsupervised visual anchor clusters and later expands to off-screen speakers through audio-text refinement (Morrone et al., 2024, Lin et al., 2023, Yin et al., 8 Aug 2025, Huang et al., 17 Mar 2026).

A useful organizing distinction is between mechanisms that assume prior enrollment and mechanisms that do not. The first class includes reference-set speaker identification, speaker-conditioned extraction, and target-speaker prompting. The second includes contextual-bandit diarization, reinforcement-learning diarization, recursive extraction of embeddings directly from mixtures, and visual-media registration from active-speaker face tracks. This suggests that “registration” in contemporary systems denotes not only offline enrollment, but more generally the creation and maintenance of speaker-resolved internal state.

Paradigm	Registered object	Representative systems
Fixed enrollment	Reference embedding set	SAM/Audioma/FlyScribe toolkit
Online creation	New arm or speaker label	MiniVox, online RL diarization
Prompt/token conditioning	Enrollment utterance, speaker tokens, names	MC-LExt, SpeakerLM, SpeakerLLM
Open-world discovery	Visual cluster and timbre prototype	CineSRD

2. Embedding-based enrollment and reference-set registration

The canonical enrollment mechanism is embedding-based. In the joint diarization and identification toolkit built around the Speaker Analysis Microservice, speaker identification is “entirely embedding-based, leveraging Wespeaker models trained on VoxCeleb2.” For an enrollment segment $x$, the extractor computes a fixed-dimensional vector

$\mathbf{e} = f(x) \in \mathbb{R}^d.$

If only one segment is available, the reference embedding for speaker $s$ is

$\mathbf{r}_s = f(x_s).$

If multiple segments are available, the reference can be averaged as

$$\mathbf{r}_s = \frac{1}{N_s}\sum_{i=1}^{N_s} f(x_{s,i}).$$

The resulting registered set is

$$\mathcal{R} = \{\mathbf{r}_1,\mathbf{r}_2,\ldots,\mathbf{r}_K\}.$$

Audioma configuration selects the SI model and the registered speakers’ set to load; FlyScribe exposes that selection to the end user; and SAM applies the selected model and reference set during inference. The same framework supports closed-set SI, in which the system always returns the identity with the highest score, and open-set SI, in which a similarity threshold $\tau$ permits an “unknown” decision with labels such as Speaker1, …, SpeakerN (Morrone et al., 2024).

This reference-set view reappears in speaker-conditioned extraction. In the multi-channel extraction system based on SincNet, an enrollment utterance is first converted to utterance-level speaker embeddings

$$\mathbf{u}_k = \frac{1}{T}\sum_{t=1}^T \mathbf{e}_{k,t},$$

and then to a more robust global speaker embedding

$$\mathbf{g}_k = \frac{1}{U_k}\sum_{u=1}^{U_k}\mathbf{u}_{k,u}.$$

At evaluation, the system averages three enrollment utterances per speaker. These registered embeddings are then injected into a dedicated speaker branch so that output $k$ is explicitly aligned with embedding $\mathbf{e} = f(x) \in \mathbb{R}^d.$0, eliminating label permutation ambiguity in multi-speaker extraction (Zhang et al., 2021).

SpeakerLM preserves the same logic while embedding registration directly into an end-to-end multimodal LLM. A frozen speaker verification model, by default ERes2NetV2, produces a speaker embedding for each registered speaker; embeddings from 2–10 s clips are averaged; a linear speaker projector maps the result into the LLM token space; and the resulting speaker token is paired with the speaker name inside registration blocks delimited by <start> and <end>. The formal registration setting is defined by the number of registered speakers $\mathbf{e} = f(x) \in \mathbb{R}^d.$1: $\mathbf{e} = f(x) \in \mathbb{R}^d.$2 for No-Regist, $\mathbf{e} = f(x) \in \mathbb{R}^d.$3 for Match-Regist, and $\mathbf{e} = f(x) \in \mathbb{R}^d.$4 for Over-Regist (Yin et al., 8 Aug 2025).

A recurrent property of these systems is model-compatibility between enrollment and inference. The registered template is not arbitrary metadata; it is a representation tied to the embedding network, projection space, and downstream decision rule actually used at test time.

3. Registration as pooling, weighting, and representation formation

Several works move the registration mechanism inside the representation extractor itself. In attentive deep speaker embedding, the attention module acts as a frame selector. For frame-level features $\mathbf{e} = f(x) \in \mathbb{R}^d.$5, it computes

$\mathbf{e} = f(x) \in \mathbb{R}^d.$6

and replaces uniform pooling by weighted statistics pooling. The weighted mean becomes

$\mathbf{e} = f(x) \in \mathbb{R}^d.$7

In this formulation, enrollment is no longer merely the storage of an utterance-level vector; it is the result of a learned utterance-specific weighting over frames. The study shows that attentive pooling improves x-vector performance on SRE’16 from EER 11.47% to 11.10% and $\mathbf{e} = f(x) \in \mathbb{R}^d.$8 from 0.873 to 0.853, and that transferring attention weights to i-vector extraction reduces EER from 13.04% to 12.18% and $\mathbf{e} = f(x) \in \mathbb{R}^d.$9 from 0.826 to 0.797; the abstract also reports a 9.0% EER reduction and 3.8% $s$0 reduction for i-vectors in another setting (Wang et al., 2018).

Recursive Attentive Pooling extends this idea from one speaker per recording to multiple speakers in a single recording without pre-registration. Let

$s$1

For speaker $s$2, the mechanism defines a coverage vector

$s$3

and computes recursive attention logits

$s$4

Speaker-specific pooled statistics then yield speaker embeddings $s$5. The same logits support speaker counting through

$s$6

The result is a single model that can produce one embedding per speaker from a mixture while also estimating the number of speakers. With ECAPA-TDNN, the reported EER in single-versus-mixture trials drops from 24.51% for the baseline to 7.71%, and mixture-versus-mixture EER drops from 35.26% to 14.13% (Horiguchi et al., 2024).

Feature fusion mechanisms can also be understood as registration operators because they determine which parts of the enrollment signal dominate the stored embedding. In the magnitude–phase co-attention framework, parallel Thin ResNet34 subnetworks process FBank and MODGD features, and a correlation matrix

$s$7

drives bidirectional channel re-weighting before self-attentive pooling. On VoxCeleb1, the co-attention system achieves Top-1 accuracy 97.20% and speaker-verification EER 2.04%, compared with 96.48% and 2.37% for FBank-only AAM-Softmax, and 97.04% and 2.26% for traditional feature-level fusion (Su et al., 17 Oct 2025).

These works show that registration quality depends not only on what is stored, but on how the enrollment signal is pooled, weighted, and fused before storage.

4. Online, interactive, and dynamically updated registration

A different line of work removes the offline enrollment stage entirely. In MiniVox, registration is part of a fully online contextual-bandit formulation in which actions are “New,” “No Speaker,” and “User $s$8.” Each arm $s$9 maintains

$\mathbf{r}_s = f(x_s).$0

and the selected action is $\mathbf{r}_s = f(x_s).$1. When the system predicts “New” and the user confirms a new speaker, a new arm is initialized. If the system mistakenly predicts an existing user for a truly new speaker, the new arm is initialized by copying the mistaken user arm’s parameters, explicitly transferring representations to mitigate cold start. Missing feedback is handled through semi-supervised or self-supervised updates with K-means, KNN, or GMM clustering (Lin et al., 2020).

The reinforcement-learning framework for online diarization generalizes this view by treating speaker registration as an action-space expansion problem in an MDP $\mathbf{r}_s = f(x_s).$2. The special “new speaker” action creates a new label online, and the desktop proof-of-concept uses standard Q-learning,

$\mathbf{r}_s = f(x_s).$3

to refine label assignment through user feedback rather than prior enrollment (Lin et al., 2023).

Interactive Speaker Recognition makes registration sequential and data-efficient rather than passive. Each speaker has a pre-computed voice print $\mathbf{r}_s = f(x_s).$4, obtained by averaging eight sentence-level X-vectors, and the system actively requests a small number of informative words $\mathbf{r}_s = f(x_s).$5 from a vocabulary $\mathbf{r}_s = f(x_s).$6. The state evolves as

$\mathbf{r}_s = f(x_s).$7

where $\mathbf{r}_s = f(x_s).$8 is the X-vector of the requested word. A PPO-trained enquirer selects the next word, and a guesser pools the collected word embeddings into a compact enrollment representation. For $\mathbf{r}_s = f(x_s).$9 guests and $$\mathbf{r}_s = \frac{1}{N_s}\sum_{i=1}^{N_s} f(x_{s,i}).$$0 requested words, the RL enquirer achieves approximately $$\mathbf{r}_s = \frac{1}{N_s}\sum_{i=1}^{N_s} f(x_{s,i}).$$1, compared with approximately 85.1% for a heuristic policy and approximately $$\mathbf{r}_s = \frac{1}{N_s}\sum_{i=1}^{N_s} f(x_{s,i}).$$2 for random selection (Seurin et al., 2020).

Consent-management systems add another dynamic layer: registration, removal, and re-registration under memory constraints. Contrastive Embedding Replay groups speakers into buckets, trains contrastive encoders per bucket, uses embeddings as replay buffers, and assigns new speakers to buckets by minimizing L2 distance to bucket prototypes. Dynamic registration uses only a fraction $$\mathbf{r}_s = \frac{1}{N_s}\sum_{i=1}^{N_s} f(x_{s,i}).$$3 of old utterances together with all new utterances. The reported experiments show that using $$\mathbf{r}_s = \frac{1}{N_s}\sum_{i=1}^{N_s} f(x_{s,i}).$$4 of old utterances generally preserves accuracy, and that after three rounds dynamic registration requires about 5.5 minutes in supervised mode, versus about 29 minutes for retraining all speakers from scratch (Shahmansoori et al., 2022).

Taken together, these systems show that registration can be interactive, incremental, and reversible, with identity state updated online rather than precomputed once.

5. Multimodal, prompt-based, and open-world registration

In multimodal and generative systems, registration often becomes a conditioning mechanism rather than a standalone preprocessing stage. MC-LExt realizes this literally by prepending a monaural enrollment utterance $$\mathbf{r}_s = \frac{1}{N_s}\sum_{i=1}^{N_s} f(x_{s,i}).$$5 to every channel of a multi-channel mixture: $$\mathbf{r}_s = \frac{1}{N_s}\sum_{i=1}^{N_s} f(x_{s,i}).$$6 The registration signal is therefore the earliest speech onset on every channel, and the network jointly learns speaker identity and spatial cues from the combined enrollment-plus-mixture spectrogram. Optional speaker-embedding fusion can later inject a conventional embedding $$\mathbf{r}_s = \frac{1}{N_s}\sum_{i=1}^{N_s} f(x_{s,i}).$$7. On WHAMR!, MC-LExt with TFGridNetV2 and SI-SDR loss reaches SDRi 18.5 dB and SI-SDRi 20.0 dB; on 4-channel MC-Libri2Mix it reaches SDRi 18.6 dB and SI-SDRi 16.3 dB; and on negative pairs its energy suppression ratio is 61.0 dB, compared with 45.7 dB for a Vanilla TSE baseline (Ling et al., 17 Oct 2025).

SpeakerLM integrates registration into a single multimodal LLM for “who spoke when and what.” Registered speaker embeddings are projected into the LLM embedding space, names are tokenized by the frozen Qwen2.5 tokenizer, and the resulting registration blocks are concatenated with audio tokens. This mechanism explicitly supports No-Regist, Match-Regist, and Over-Regist. On AliMeeting-Eval, Match-Regist yields CER 13.98%, saCER 15.57%, and $$\mathbf{r}_s = \frac{1}{N_s}\sum_{i=1}^{N_s} f(x_{s,i}).$$8; Over-Regist with $$\mathbf{r}_s = \frac{1}{N_s}\sum_{i=1}^{N_s} f(x_{s,i}).$$9 random in $$\mathcal{R} = \{\mathbf{r}_1,\mathbf{r}_2,\ldots,\mathbf{r}_K\}.$$0 yields CER 13.96%, saCER 15.71%, and $$\mathcal{R} = \{\mathbf{r}_1,\mathbf{r}_2,\ldots,\mathbf{r}_K\}.$$1, indicating only a small penalty for redundant registered speakers (Yin et al., 8 Aug 2025).

SpeakerLLM turns registration into a hierarchical tokenization and reasoning problem. A frozen speaker encoder produces an utterance-level embedding $$\mathcal{R} = \{\mathbf{r}_1,\mathbf{r}_2,\ldots,\mathbf{r}_K\}.$$2 and frame-level features $$\mathcal{R} = \{\mathbf{r}_1,\mathbf{r}_2,\ldots,\mathbf{r}_K\}.$$3. These become embedding-level tokens

$$\mathcal{R} = \{\mathbf{r}_1,\mathbf{r}_2,\ldots,\mathbf{r}_K\}.$$4

and sequence-level tokens

$$\mathcal{R} = \{\mathbf{r}_1,\mathbf{r}_2,\ldots,\mathbf{r}_K\}.$$5

combined as

$$\mathcal{R} = \{\mathbf{r}_1,\mathbf{r}_2,\ldots,\mathbf{r}_K\}.$$6

The verification-reasoning stage organizes outputs into the schema

$$\mathcal{R} = \{\mathbf{r}_1,\mathbf{r}_2,\ldots,\mathbf{r}_K\}.$$7

SpeakerLLM-Base reports 96.1% generated same/different accuracy on VoxCeleb1-O, while SpeakerLLM-VR produces 100% valid three-block outputs and 72.7% clause-level profile grounding (Nam et al., 14 May 2026).

CineSRD addresses open-world visual media by registering initial speakers through visual anchor clustering. For a visual cluster $$\mathcal{R} = \{\mathbf{r}_1,\mathbf{r}_2,\ldots,\mathbf{r}_K\}.$$8, the dominant co-occurring audio cluster is chosen by majority vote, and the timbre prototype is computed as

$$\mathcal{R} = \{\mathbf{r}_1,\mathbf{r}_2,\ldots,\mathbf{r}_K\}.$$9

The registered speaker set is

$\tau$0

Speaker turns are then refined by combining an audio LLM probability $\tau$1 with timbre similarity $\tau$2,

$\tau$3

with $\tau$4 in experiments. Off-screen speaker supplementation uses a group score $\tau$5, and if $\tau$6 with $\tau$7, the system registers a new speaker prototype for that group (Huang et al., 17 Mar 2026).

These multimodal formulations expand registration beyond speaker templates. Registration can be an onset prompt, a token block in a generative model, a structured verification trace, or a cross-modal prototype anchored in faces and subtitles.

6. Operational behavior, applications, and recurring misconceptions

The surveyed systems make clear that speaker registration is not synonymous with a closed-set assumption. The joint diarization and identification toolkit supports both closed-set assignment and open-set rejection through a similarity threshold, returning unknown labels when no registered speaker is sufficiently similar. The same speaker labels are then attached to ASR output, yielding speaker-attributed transcripts, with registered speakers shown by name and unknown speakers shown by generic labels (Morrone et al., 2024).

A second misconception is that registration must always precede use. Several systems derive effective speaker spaces without registering the actual test speaker in advance. A neural-network speaker classifier trained on 200 in-domain speakers maps out-of-domain conversational speech to 200-dimensional log-likelihood vectors, and speaker change detection is performed by comparing adjacent interval means with Euclidean distance. In the reported TIMIT-based experiments, the method captures close to 97% of the changes by comparing the current second of speech with the previous second, and the underlying classifier achieves 100% file-level accuracy on any testing file once speech duration reaches at least 0.97 seconds (Ge et al., 2017).

Registration mechanisms also determine target selection in enhancement and beamforming. In the Speaker Selection Mechanism for end-to-end beamforming, the target speaker during training is the one minimizing the absolute undershot angle,

$\tau$8

so that geometry defines which speaker is “registered” as the desired output for that utterance, even though inference uses audio only. At $\tau$9 dB SNR with two speakers, the NN+SSM system reports STOI 0.526, PESQ 1.366, and SI-SDR $$\mathbf{u}_k = \frac{1}{T}\sum_{t=1}^T \mathbf{e}_{k,t},$$0 dB, compared with STOI 0.447, PESQ 1.322, and SI-SDR $$\mathbf{u}_k = \frac{1}{T}\sum_{t=1}^T \mathbf{e}_{k,t},$$1 dB for MVDR, and STOI 0.346, PESQ 1.219, and SI-SDR $$\mathbf{u}_k = \frac{1}{T}\sum_{t=1}^T \mathbf{e}_{k,t},$$2 dB for the same neural architecture trained without SSM (Fiorio et al., 24 Mar 2025).

A plausible implication of these results is that speaker registration is best viewed as an interface between identity evidence and downstream decision-making. Depending on the application, the registered object may support speaker-attributed ASR, diarization under overlap, multi-target extraction, user authorization, or privacy-preserving consent management. The mechanism is therefore defined less by one particular storage format than by three invariant functions: acquiring speaker evidence, binding it to persistent speaker-specific state, and reusing that state under the task’s open-set, closed-set, online, or multimodal constraints.