---
title: 'SpeakerLM: End-to-End Speaker Diarization'
url: https://www.emergentmind.com/topics/speakerlm
type: topic
---

# SpeakerLM: End-to-End Speaker Diarization

SpeakerLM is a multimodal large language model for end-to-end speaker diarization and recognition, formulated around the requirement to predict “who spoke when and what” within multi-speaker audio [2508.06372]. In this usage, the term denotes a unified speaker-attributed transcription system rather than a generic speech language model. The model is positioned against cascaded speaker diarization and automatic speech recognition pipelines, and against LLM-based post-processing of such pipelines, by treating speaker diarization and recognition as a single multimodal generation problem in which audio content, speaker information, and text are jointly modeled [2508.06372]. Within the broader literature, adjacent systems use closely related ideas for speaker profiling, verification reasoning, multi-speaker spoken language understanding, and speaker-aware dialogue generation, but they target different operational outputs and evaluation regimes [2605.15044].

## 1. Problem formulation and scope

SpeakerLM is defined around Speaker Diarization and Recognition (SDR), which extends conventional speaker diarization by requiring attributed transcription rather than speaker turns alone [2508.06372]. The practical target is not only “who spoke when,” but “who spoke when and what,” a formulation motivated by meeting transcription and dialogue systems where speaker labels without lexical content are insufficient [2508.06372].

The paper identifies three limitations in traditional SDR cascades. First, diarization mistakes propagate into ASR. Second, overlapping speech remains difficult for many speaker diarization systems. Third, separate optimization of SD and ASR prevents joint exploitation of speaker identity and lexical content [2508.06372]. It also distinguishes SpeakerLM from LLM post-correction of cascaded outputs, arguing that post-processing cannot recover information lost in upstream SD or ASR and may hallucinate or alter transcript content [2508.06372].

This problem framing aligns with later benchmark evidence that current speech and multimodal LLMs are often better at understanding what was said than at identifying who said it. In spoken dialogue question answering, models such as WavLLM and Qwen-Audio show much lower accuracy on identity-critical questions than on context-based questions, suggesting limited speaker awareness from audio [2409.04927]. Likewise, M3-SLU reports that models can often capture content but frequently fail speaker attribution in long, natural conversations [2510.19358]. These results situate SpeakerLM as a direct response to a documented weakness in contemporary speech-language systems.

## 2. Core architecture

SpeakerLM uses an encoder–projector–LLM architecture in which a pretrained audio encoder processes multi-speaker audio, a projector maps acoustic features into the LLM embedding space, and a pretrained text LLM generates the final SDR output autoregressively [2508.06372]. The text backbone is Qwen2.5-7B-Instruct, the audio encoder is SenseVoice-large, and the speaker identity pathway uses the pretrained speaker embedding model ERes2NetV2 with a linear projector [2508.06372].

The model contains two modality-alignment pathways. In the audio path, acoustic embeddings from the frozen or pretrained audio encoder are passed through a two-layer Transformer projector followed by a CNN layer for dimensional alignment [2508.06372]. In the speaker registration path, registered speaker names are tokenized by a frozen text tokenizer, registered speaker speech samples are encoded by a frozen speaker embedding extractor, and the resulting embeddings are projected by a single linear layer into the LLM space [2508.06372]. The registered speaker information is then concatenated with audio and text tokens so that the LLM can condition transcription on known identities [2508.06372].

The autoregressive output is speaker-attributed text. In the no-registration setting, SpeakerLM emits anonymous speaker identifiers such as `spk 0`, `spk 1`, and so forth; in registration settings, it emits actual speaker names [2508.06372]. Conceptually, the generated sequence alternates speaker labels and recognized text, so segmentation, assignment, and transcription are all learned within a single generation process [2508.06372].

A later line of work suggests one limitation of this formulation. G-STAR characterizes SpeakerLM as strong in local diarization and recognition, but without an explicit mechanism for cross-chunk identity persistence in long-form meeting-scale inference [2603.10468]. This suggests that SpeakerLM is best understood as a foundational end-to-end SDR architecture whose initial formulation emphasizes unified attributed transcription rather than long-context global speaker tracking.

## 3. Speaker registration and operational settings

A major contribution of SpeakerLM is its flexible speaker registration mechanism, intended to support diverse real-world conditions rather than a single supervised regime [2508.06372]. The paper defines the number of ground-truth speakers as $N_{gt}$ and the number of registered speakers as $N_{rg}$, with three operational settings:
$$
N_{rg} =
\begin{cases}
0, & \text{if No-Regist} \\
N_{gt}, & \text{if Match-Regist} \\
N_{gt} + N_{ov}, & \text{if Over-Regist}
\end{cases}
$$
where $N_{ov} > 0$ is the number of extra registered speakers [2508.06372].

In **No-Regist**, no prior identity information is supplied and the system behaves as a conventional anonymous diarization-and-recognition model [2508.06372]. In **Match-Regist**, all true speakers are pre-registered, corresponding to personalized transcription in which active speakers should be mapped to known names [2508.06372]. In **Over-Regist**, more speakers are registered than actually appear, so the model must ignore irrelevant identities while using the relevant ones [2508.06372].

The registration order is randomized in Match-Regist and Over-Regist, and representative speaker vectors are formed by segmenting each registered speaker’s speech into 2–10 second clips, extracting embeddings, and averaging them [2508.06372]. During Over-Regist training, the number of extra speakers is sampled uniformly from 1 to 50 [2508.06372].

This registration mechanism is significant because it broadens the task definition from anonymous SDR to mixed identity-aware transcription. A plausible implication is that SpeakerLM was designed not only for benchmark SDR, but also for scenarios in which user authorization, personalization, or roster-constrained transcription matter. That broader operational rationale is made explicit in adjacent work such as SpeakerLLM, which argues that audio-first agents require understanding who is speaking, how the voice sounds, and how recording conditions affect speaker cues [2605.15044].

## 4. Multi-stage training and data regime

SpeakerLM is developed with a four-stage training strategy intended to stabilize optimization and exploit large-scale data [2508.06372]. The first stage is ASR pretraining, producing SpeakerLM-ASR, using 600,000 hours of public ASR data; speaker embedding modules are not yet included, and LoRA is used for parameter-efficient LLM adaptation [2508.06372]. The second stage trains the projectors on 5,000 hours of simulated SDR data while freezing the audio encoder and LLM [2508.06372]. These simulated mixtures are 50 seconds long, contain 2–4 speakers, and include real-world noises and reverberation with SNR uniformly sampled from 10–20 dB [2508.06372].

The third stage adapts the model to real SDR data while keeping the LLM frozen and fine-tuning the audio encoder and projectors [2508.06372]. The fourth stage performs full joint fine-tuning, again using LoRA on the LLM, to integrate acoustic and linguistic information more deeply [2508.06372]. The paper reports monotonic improvement across stages on AliMeeting-Eval and Simulation-Test, while also noting that Stage 2 can hurt CER on AISHELL4 and AISHELL5 because simulated data does not match those domains [2508.06372].

The real SDR training corpus includes AliMeeting-Train at 104.75 hours, AISHELL4-Train at 107.50 hours, In-House-Train at 7,426.70 hours, and In-House-Valid at 30.29 hours, for a total of 7,638.95 hours of real training data [2508.06372]. Test sets are AliMeeting-Eval at 10.00 hours, AISHELL4-Eval at 12.72 hours, and AISHELL5-Eval at 3.58 hours, with AISHELL5 described as acoustically challenging and out-of-domain because it contains in-car, far-field, noisy speech with wind, tire, music, and air-conditioning noise [2508.06372].

Training uses 16 kHz audio resampling, random clips of 40–50 seconds, AdamW, a learning rate warmed from $1\times10^{-5}$ to $5\times10^{-5}$ and then cosine decay, dynamic batching with a maximum of 6K tokens, 1M steps per stage, and validation every 10K steps on 4 NVIDIA A800 GPUs [2508.06372]. The prominence of real SDR data in the later stages is consistent with broader evidence that speaker-attributed reasoning is not solved by generic speech-language pretraining alone [2510.19358].

## 5. Evaluation methodology and empirical performance

SpeakerLM is evaluated with speaker-agnostic and speaker-attributed transcription metrics. The paper uses CER, cpCER, saCER, and the attribution deltas
$$
\Delta \textrm{cp} = \textrm{cpCER} - \textrm{CER}
$$
and
$$
\Delta \textrm{sa} = \textrm{saCER} - \textrm{CER}
$$
with cpCER and saCER treated as the main SDR metrics because they jointly reflect recognition and speaker assignment quality [2508.06372].

Under **No-Regist**, the strongest reported SpeakerLM results with 7,638.95 hours of SDR data are CER 13.97, cpCER 16.05, and $\Delta$cp 2.08 on AliMeeting-Eval; CER 17.17, cpCER 18.37, and $\Delta$cp 1.20 on AISHELL4-Eval; and CER 47.24, cpCER 47.81, and $\Delta$cp 0.57 on AISHELL5-Eval [2508.06372]. The paper reports absolute cpCER improvements over the strongest cascaded baseline of 6.60% on AliMeeting-Eval, 6.56% on AISHELL4-Eval, and 13.82% on AISHELL5-Eval [2508.06372]. Among cascaded SD+ASR systems, Diarizen-large + Paraformer is the strongest baseline, while zero-shot LLM correction is reported to be poor and to sometimes worsen CER substantially [2508.06372].

Under registration settings, SpeakerLM remains close in CER and cpCER between Match-Regist and Over-Regist, while $\Delta$sa is lower in Match-Regist, as expected [2508.06372]. On AliMeeting-Eval, Match-Regist yields CER 13.98, saCER 15.57, and $\Delta$sa 1.59, while Over-Regist yields CER 13.96, saCER 15.71, and $\Delta$sa 1.75 [2508.06372]. On AISHELL4-Eval, Match-Regist yields CER 17.13, saCER 19.73, and $\Delta$sa 2.60, while Over-Regist yields CER 17.15, saCER 20.16, and $\Delta$sa 3.01 [2508.06372]. On AISHELL5-Eval, Match-Regist yields CER 47.05, saCER 47.36, and $\Delta$sa 0.31, while Over-Regist yields CER 46.69, saCER 47.35, and $\Delta$sa 0.66 [2508.06372]. The model also significantly outperforms SA-Transformer on AliMeeting-Eval, with a reported 25.98% absolute improvement in saCER [2508.06372].

The ablation on the speaker embedding frontend compares ERes2NetV2 and CAM++. ERes2NetV2 performs better in all registration settings; on AliMeeting-Eval, for example, No-Regist changes from CER 14.63 and cpCER 16.74 with CAM++ to CER 13.97 and cpCER 16.05 with ERes2NetV2, and Match-Regist saCER changes from 17.23 to 15.57 [2508.06372].

| Setting | Dataset | Reported result |
|---|---|---|
| No-Regist | AliMeeting-Eval | CER 13.97, cpCER 16.05, $\Delta$cp 2.08 |
| No-Regist | AISHELL4-Eval | CER 17.17, cpCER 18.37, $\Delta$cp 1.20 |
| No-Regist | AISHELL5-Eval | CER 47.24, cpCER 47.81, $\Delta$cp 0.57 |
| Match-Regist | AliMeeting-Eval | CER 13.98, saCER 15.57, $\Delta$sa 1.59 |
| Over-Regist | AliMeeting-Eval | CER 13.96, saCER 15.71, $\Delta$sa 1.75 |

These results are important because they show both data scaling and robustness to redundant registration lists. They also support one of the paper’s clearest negative findings: zero-shot LLM correction of cascaded SD+ASR outputs is unreliable for SDR because speaker relabeling prompts can still trigger transcript hallucination [2508.06372].

## 6. Relation to adjacent speaker-aware models and benchmarks

SpeakerLM belongs to a rapidly expanding cluster of speaker-aware audio-language research, but its output structure is specific. It directly generates speaker-attributed transcripts, whereas other systems target profiling, verification reasoning, or descriptive generation. SpeakerLLM, for example, is a speaker-specialized audio-LLM that unifies single-utterance speaker profiling, recording-condition understanding, utterance-pair speaker comparison, and evidence-organized verification reasoning in a natural-language interface [2605.15044]. Its core innovations are a hierarchical speaker tokenizer over utterance-level and frame-level representations and a structured three-block verification trace, rather than end-to-end SDR [2605.15044].

CoLMbo reframes speaker recognition as prompt-conditioned descriptive profiling. It integrates a frozen speaker encoder with GPT-2 through a learned prefix mapper, and generates attributes such as gender, age, dialect, ethnicity, accent, and regional traits from audio plus prompt [2506.09375]. This is a different output regime from SpeakerLM: CoLMbo produces caption-like profiles, while SpeakerLM produces speaker-attributed transcripts [2506.09375].

GLSC-SDR, by contrast, remains within end-to-end SDR but improves speaker discriminability through joint training with a Global-Local Speaker Classification objective. It uses Qwen2.5-Omni-7B as backbone, ERes2Net for speaker representation extraction, and a joint objective
$$
L = \alpha L_{GLSC} + (1-\alpha) L_{SDR}
$$
to improve attribution without adding a separate speaker encoder or complex multi-encoder architecture [2603.25377]. This suggests an alternate design axis: SpeakerLM emphasizes multimodal generation with registration, while GLSC-SDR emphasizes explicit hierarchical speaker supervision during training [2603.25377].

Benchmark work has also sharpened the evaluation context for SpeakerLM. “Just ASR + LLM?” argues that strong spoken dialogue QA performance often overstates actual speaker awareness, because many benchmark questions are solvable from transcripts alone [2409.04927]. MSU-Bench extends evaluation to 16 speaker-centric tasks and 2,300 QA instances in a two-tier framework from speaker grounding to dialogue reasoning, reporting that all tested models still face challenges in complex speaker grounding and multi-speaker reasoning [2606.22868]. M3-SLU, built from CHiME-6, MELD, MultiDialog, and AMI, reports that even strong cascaded and end-to-end MLLMs still fail at identifying and reasoning about who spoke what in long, natural conversations [2510.19358]. Collectively, these benchmarks suggest that SpeakerLM addresses one central subproblem—speaker-attributed transcription—but does not exhaust the broader problem of speaker-centric conversational intelligence.

## 7. Limitations, misconceptions, and subsequent directions

A common misconception in this area is that an ASR system followed by an LLM is sufficient for speaker-aware speech understanding. Several results directly challenge that view. SpeakerLM reports that zero-shot LLM correction of cascaded SD+ASR outputs often worsens performance because of hallucination [2508.06372]. “Just ASR + LLM?” further shows that strong benchmark performance can mask limited speaker awareness when questions are solvable from transcripts alone [2409.04927]. M3-SLU similarly finds that even with good content understanding, models still struggle on speaker attribution [2510.19358].

SpeakerLM itself has identifiable limits. The paper evaluates on Mandarin SDR data, so cross-lingual generalization is not established [2508.06372]. The system depends on large-scale ASR pretraining and a large amount of real SDR data, suggesting that end-to-end speaker attribution is data-hungry [2508.06372]. It also relies on an external pretrained speaker embedding extractor, so registration performance depends on the quality of that module [2508.06372]. In registered settings, it is not a fully self-contained identity-discovery system, because speaker embeddings must already be available [2508.06372].

Later work clarifies the next technical frontier. G-STAR targets timestamped speaker-attributed ASR for long-form, multi-party speech with overlap, and explicitly positions prior Speech-LLM systems as tending to prioritize either local diarization or global labeling [2603.10468]. Its addition of a time-aware speaker-tracking module, an Arrival-Order Speaker Cache, interleaved temporal fusion, and hierarchical weighting of timestamp and speaker tokens suggests that meeting-level identity consistency and temporal grounding remain open problems beyond the original SpeakerLM formulation [2603.10468].

This suggests a broader interpretation of SpeakerLM’s significance. It marks a shift from pipelines and post-hoc correction toward direct multimodal generation for SDR, while later systems and benchmarks indicate that robust speaker grounding, global speaker tracking, verification reasoning, and multi-speaker dialogue understanding remain distinct and only partially solved research directions [2508.06372].

Source: https://www.emergentmind.com/topics/speakerlm