---
title: 'Sortformer: End-to-End Diarization & ASR'
url: https://www.emergentmind.com/topics/sortformer
type: topic
---

# Sortformer: End-to-End Diarization & ASR

Searching arXiv for recent papers on Sortformer and closely related speaker diarization work.
Sortformer is an end-to-end neural model for speaker diarization that addresses the speaker permutation problem by sorting output speakers according to arrival time, rather than relying exclusively on permutation-invariant matching during training. In its original formulation, it is also a mechanism for multispeaker ASR: speaker label estimation is embedded into the ASR encoder state so that frame-level diarization timestamps and speaker tokens in text are aligned under a common sorted convention [2409.06656]. Subsequent work treats Sortformer both as a diarization model in its own right and as a reference point for streaming diarization, synthetic-data training studies, corpus benchmarking, and downstream speech-analysis pipelines.

## 1. Problem setting and design objective

Speaker diarization assigns anonymous speaker labels to regions of speech, and the arbitrariness of those labels creates the **speaker permutation problem**: the mapping between predicted speaker indices and reference speaker indices is ambiguous. Prior end-to-end diarization systems are described as relying heavily on **permutation invariant loss (PIL)**, which searches over assignments and optimizes for the permutation that yields the lowest error [2409.06656].

In the formulation reported for Sortformer, if the ground-truth speaker activity matrix is $\mathbf{Y} \in \mathbb{R}^{K \times T}$, the predicted activity matrix is $\mathbf{P} \in \mathbb{R}^{K \times T}$, and $\Pi$ is the set of all $K!$ permutations, then PIL is written as
$$
\mathcal{L}_{\text{PIL}}(\mathbf{Y}, \mathbf{P}) = \min_{\pi \in \Pi} \frac{1}{TK} \sum_{k=1}^{K} \sum_{t=1}^{T} \mathcal{L}_{\text{BCE}}(y_{\pi(k),t}, p_{k,t}).
$$
The reported limitations are computational and architectural: PIL is described as $\mathcal{O}(K!)$ or $\mathcal{O}(K^3)$ with the Hungarian algorithm, and it complicates joint optimization with token-level ASR objectives [2409.06656].

Sortformer was proposed specifically to change that training geometry. The model is described as a novel neural model for speaker diarization, trained with unconventional objectives compared to existing end-to-end diarization models, and as an encoder-based diarization model originally designed to supervise speaker tagging in speech-to-text systems [2409.06656][2510.23320]. A common misconception is that Sortformer is simply a replacement for diarization clustering. The papers instead frame it more precisely: it is an end-to-end diarization model whose sorted supervision scheme is intended both to resolve speaker permutation and to make diarization outputs compatible with multispeaker ASR training [2409.06656].

## 2. Arrival-time ordering and Sort Loss

The defining mechanism is **arrival-time order (ATO)**. Let $t_k^{(0)}$ be the first frame where speaker $k$ is active, expressed as
$$
\Psi(\mathbf{y}_k) = \min \{ t' \mid y_{k,t'} \neq 0, \ t' \in [1, T] \}.
$$
A sorting permutation $\eta$ is then defined so that
$$
\Psi(\mathbf{y}_{\eta(1)}) \leq \Psi(\mathbf{y}_{\eta(2)}) \leq \ldots \leq \Psi(\mathbf{y}_{\eta(K)}).
$$
Ground-truth speaker activity is reordered by that arrival-time rule, and Sort Loss is given as
$$
\mathcal{L}_{\text{Sort}}(\mathbf{Y}, \mathbf{P}) = \frac{1}{K} \sum_{k=1}^{K} \mathcal{L}_{\text{BCE}}(\mathbf{y}_{\eta(k)}, \mathbf{q}_k).
$$
The stated effect is that the model is forced to align each output row with the speaker occupying the corresponding arrival position, so permutation is resolved by a deterministic, signal-derived convention rather than by post hoc search [2409.06656].

The original paper does not position Sort Loss as requiring the elimination of PIL. Instead, it explicitly states that Sortformer can autonomously resolve permutation **with or without PIL**, and it reports a hybrid objective
$$
\mathcal{L}_{\text{hybrid}} = \alpha \cdot \mathcal{L}_{\text{Sort}} + (1-\alpha) \cdot \mathcal{L}_{\text{PIL}},
$$
with $\alpha$ set empirically, for example $0.5$ [2409.06656]. That detail matters because the empirical record in the same paper shows three distinct variants—PIL-only, Sort-only, and Hybrid-Loss—rather than a single canonical training recipe.

The literature repeatedly emphasizes that the sorting is **active** rather than a passive relabeling of outputs. In the original exposition, the network is trained to internalize arrival-time ordering, and later descriptions summarize this as a Sort Loss that enforces a consistent ordering of speaker labels, jointly modeling speaker assignment and temporal continuity [2409.06656][2510.23320]. A plausible implication is that Sortformer is best understood as a permutation-resolution principle embedded into model learning, not merely an output-formatting convention.

## 3. Diarization architecture and the ASR bridge

The diarization backbone reported for the original model uses **NEST (NeMo Encoders for Speech Tasks)**, a Fast-Conformer-based architecture with efficient **8x subsampling** and **frame length 80 ms**, followed by a transformer stack with **18 layers**, **hidden size 192**, and **relative positional embedding** [2409.06656]. The model takes audio features $\mathbf{x}_{1:T}$ and outputs a posterior probability matrix
$$
\mathbf{P} = f_\Theta(\mathbf{X}), \qquad \mathbf{P} \in [0,1]^{K \times T},
$$
with sigmoid activations because speaker activities are treated as independent rather than mutually exclusive [2409.06656]. In later work, Sortformer is also described as producing up to **four speaker activity tracks** and being trained for up to **4 speakers per session** [2507.18446][2409.06656].

The second architectural contribution is the explicit bridge between diarization timestamps and ASR tokens. Sortformer inserts special speaker tokens such as $\{<\!\!|spk0|\!\!>, <\!\!|spk1|\!\!>, \ldots\}$ into transcripts, with those tokens also sorted by arrival-time order. For each word or token, a speaker token is inserted using word-level alignment, or approximated word-level timestamps derived from segment-level labels using a syllable-based heuristic when word timestamps are unavailable [2409.06656].

To inject diarization supervision into ASR, the original paper introduces a **sinusoidal kernel** over speaker indices. For speaker $k$ and feature dimension $z$,
$$
\kappa_{k,z} = \sin\left(\frac{2\pi k z}{M}\right),
$$
and the speaker kernel for speaker $k$ is
$$
\gamma_k = [\kappa_{k,1}, \ldots, \kappa_{k,M}].
$$
If $\Gamma \in \mathbb{R}^{K \times M}$ collects these kernels and $\mathbf{A}$ is the ASR encoder state, the enriched state is given as
$$
\tilde{\mathbf{A}} = \frac{\mathbf{A}}{\|\mathbf{A}\|_2} + \Gamma^T \cdot \mathbf{P}.
$$
This is the mechanism by which speaker label estimation is embedded within the ASR encoder state using a sinusoidal kernel function, thereby enabling standard token-level objectives for multispeaker ASR under a sorted speaker convention [2409.06656].

The multispeaker ASR variant is also reported to use **adapter techniques** for parameter-efficient transfer to large pretrained ASR models, with adapters inserted in every encoder and decoder layer and only adapters fine-tuned for multispeaker adaptation [2409.06656]. This supports the broader claim in the original title that Sortformer bridges timestamps and tokens.

## 4. Reported performance across representative evaluations

The empirical record presented across the cited papers is heterogeneous because Sortformer appears in several roles: as the primary model in its original paper, as a zero-shot diarization baseline on LibriConvo, and as the basis for a streaming extension. The following table summarizes representative reported results.

| Setting | Configuration | Representative result |
|---|---|---|
| Original diarization benchmarks | Sortformer-Hybrid-Loss | DIHARD3: 14.76; CALLHOME-part2: 5.87/8.46/12.59; CH109: 6.86 |
| LibriConvo zero-shot baseline | `diar_sortformer_4spk-v1` | Validation DER: 12.9; Test DER: 11.1 |
| Streaming diarization | Streaming Sortformer, 1.04s latency | CALLHOME 2spk: 7.35; DIHARD III $\leq 4$ spk: 14.57 |

In the original benchmark study, **Sortformer-Sort-Loss** and **Sortformer-PIL** are reported as near each other on several diarization conditions, while **Sortformer-Hybrid-Loss** is strongest overall: **14.76** on DIHARD3, **5.87/8.46/12.59** on CALLHOME-part2 for **2/3/4 speakers**, and **6.86** on CH109. The paper characterizes this as performance competitive with state-of-the-art end-to-end diarization models trained exclusively with PIL, and the detailed excerpt states that the Hybrid-Loss variant outperforms many previous EEND variants [2409.06656].

On **LibriConvo**, Sortformer is evaluated zero-shot, “as-is,” without additional fine-tuning on the new corpus. Using the `diar_sortformer_4spk-v1` model, the reported **DER** is **12.9** on Validation and **11.1** on Test, compared with **25.6** and **24.4** for the pyannote pipeline. The accompanying analysis states that Sortformer achieves less than half the DER of pyannote on both splits and exhibits a narrower per-recording DER distribution, particularly in conversations with overlap or rapid turn-taking [2510.23320].

These evaluations support two distinct conclusions that should not be conflated. First, the original paper argues that Sort Loss is a viable replacement or complement to PIL in offline diarization training. Second, later corpus studies show that a pretrained Sortformer can generalize strongly, even without corpus-specific adaptation, on realistic simulated conversational data [2409.06656][2510.23320].

## 5. Streaming Sortformer and the Arrival-Order Speaker Cache

The original Sortformer is described as **offline-only**, which makes it unsuitable for low-latency deployment. “Streaming Sortformer: Speaker Cache-Based Online Speaker Diarization with Arrival-Time Ordering” extends the framework by introducing an **Arrival-Order Speaker Cache (AOSC)** that stores frame-level acoustic embeddings of previously observed speakers, ordered by speaker index corresponding to their arrival-time order [2507.18446].

At each streaming step, predictions are formed from the concatenation of three elements: the **AOSC speaker cache**, a **FIFO queue of preceding audio chunks**, and the **current input buffer**. The rationale is that arrival-time ordering already resolves the chunk-to-chunk permutation ambiguity that complicates prior streaming EEND approaches, so cache organization can follow the same convention [2507.18446].

The cache update logic is explicit. Let $M$ be the maximum cache length and $K$ the minimum number of frames kept per speaker. Given frame-wise model outputs $P$, the speaker score is written as
$$
S_i = \log P_i + \sum_{j \neq i} \log(1 - P_j).
$$
Frames with all $P_k < 0.5$ are treated as silence for average silence embedding computation; frames where $P_i < 0.5$ are assigned $S_i = -\infty$; recent frames may be boosted by $\delta$; the top $K$ frames per speaker may be boosted by $\Delta$; and $A$ copies of the average silence embedding are appended after each speaker with $S = +\infty$. The final cache is obtained by selecting the top $M$ scored frames while preserving order [2507.18446].

The reported results indicate graceful degradation as latency decreases. For **CALLHOME 2spk**, **Offline Sortformer** reports **6.7%**, while **Streaming Sortformer** reports **6.8%** at **10s**, **7.35%** at **1.04s**, and **8.60%** at **0.32s** latency. For **DIHARD III $\leq 4$ spk**, the corresponding figures are **15.47%** for offline and **14.79%**, **14.57%**, and **14.63%** for the three streaming settings. The paper also states that the **Real-Time Factor** is always below **0.2**, even at the lowest latency [2507.18446].

A common misunderstanding is that streaming Sortformer is merely the original model run chunk-wise. The cited paper makes a narrower claim: the streaming extension depends on an explicit cache mechanism keyed to arrival-order speaker indices. In that sense, AOSC is not an optimization detail but the central device that operationalizes Sortformer’s ordering principle in online inference [2507.18446].

## 6. Synthetic-data studies, specialized deployments, and comparative position

Later work uses Sortformer as a test case for how diarization quality depends on synthetic conversation design. In “Mind the Gap: Impact of Synthetic Conversational Data on Multi-Talker ASR and Speaker Diarization,” Sortformer is described as an end-to-end neural diarization system designed to resolve speaker permutations via a Sort Loss using the order of speaker arrival times, built on a **NEST-FastConformer** backbone. The study reports that simulation choices are highly task-dependent: fitting turn-taking statistics to real corpora yields modest gains, but **overlap boosting degrades diarization**, moving macro DER from **26.1%** to **27.6%**. Acoustic augmentation is beneficial, with **None** at **26.1**, **+ noise** at **24.6**, **+ reverb** at **24.3**, and **+ noise + reverb** at **22.2**. For training strategies, **Synthetic only** gives **22.2**, **Real only** gives **17.4**, **Real + synthetic** gives **16.3**, and **Synthetic → real** gives **15.5** macro DER [2605.15442]. This suggests that Sortformer is unusually informative for simulation research because its end-to-end diarization behavior is sensitive to overlap, source diversity, and acoustic realism.

Sortformer has also been tested in a specialized clinical-adjacent setting. In **Korean toddler speech**, where young female caregivers often use **aegyo** and thereby become acoustically similar to the child, the reported result is that **NeMo SortFormer achieves 88.69% speaker count accuracy and 33.04% diarization error rate (DER)**. The paper attributes this to the arrival-time-sorted transformer architecture, which can use temporal order and conversational structure when voice characteristics alone are confounded [2606.10213]. A plausible implication is that the arrival-order prior is especially useful when role structure is stable but speaker timbre is not strongly separable.

Within joint ASR and diarization, Sortformer occupies an intermediate historical position. “Train Short, Infer Long: Speech-LLM Enables Zero-Shot Streamable Joint ASR and Diarization on Long Audio” describes Sortformer as a **strong recent baseline** for local, non-streaming joint ASR and diarization on audio up to **20s**. In that comparison, Sortformer reports **cpWER 26.71** on **AMI Test** and **21.45** on **CH109**, while the new Speech-LLM system is reported to surpass it in the local setting and to address long-form streamable inference more directly [2511.16046]. The key limitation identified there is not that Sortformer fails at speaker-attributed transcription, but that its original architecture is local and offline, whereas long-audio streaming requires a persistent speaker memory or a different global mechanism [2511.16046].

Taken together, the cited literature presents Sortformer as more than a single diarization model. It is a family of sorted-objective methods centered on arrival-time ordering: originally an offline diarization model with Sort Loss and an ASR bridge [2409.06656], later extended to low-latency diarization with AOSC [2507.18446], and subsequently used as a benchmark or component in studies of realistic simulated conversations [2510.23320], synthetic-data recipes [2605.15442], and domain-specific pipelines such as toddler pronunciation evaluation [2606.10213].

Source: https://www.emergentmind.com/topics/sortformer