Sortformer: End-to-End Diarization & ASR

• Sortformer is an end-to-end neural model for speaker diarization that resolves the permutation problem by ordering speakers by arrival time.
• It embeds speaker label estimation into the ASR encoder using a sinusoidal kernel, ensuring alignment of diarization timestamps with text tokens.
• Variants like Sort-only, PIL-only, and Hybrid-Loss demonstrate its adaptability, with streaming extensions leveraging an Arrival-Order Speaker Cache.

Searching arXiv for papers on Sortformer and closely related speaker diarization work. Sortformer is an end-to-end neural model for speaker diarization that addresses the speaker permutation problem by sorting output speakers according to arrival time, rather than relying exclusively on permutation-invariant matching during training. In its original formulation, it is also a mechanism for multispeaker ASR: speaker label estimation is embedded into the ASR encoder state so that frame-level diarization timestamps and speaker tokens in text are aligned under a common sorted convention (Park et al., 2024). Subsequent work treats Sortformer both as a diarization model in its own right and as a reference point for streaming diarization, synthetic-data training studies, corpus benchmarking, and downstream speech-analysis pipelines.

1. Problem setting and design objective

Speaker diarization assigns anonymous speaker labels to regions of speech, and the arbitrariness of those labels creates the speaker permutation problem: the mapping between predicted speaker indices and reference speaker indices is ambiguous. Prior end-to-end diarization systems are described as relying heavily on permutation invariant loss (PIL), which searches over assignments and optimizes for the permutation that yields the lowest error (Park et al., 2024).

In the formulation reported for Sortformer, if the ground-truth speaker activity matrix is $\mathbf{Y} \in \mathbb{R}^{K \times T}$, the predicted activity matrix is $\mathbf{P} \in \mathbb{R}^{K \times T}$, and $\Pi$ is the set of all $K!$ permutations, then PIL is written as

$$\mathcal{L}_{\text{PIL}}(\mathbf{Y}, \mathbf{P}) = \min_{\pi \in \Pi} \frac{1}{TK} \sum_{k=1}^{K} \sum_{t=1}^{T} \mathcal{L}_{\text{BCE}}(y_{\pi(k),t}, p_{k,t}).$$

The reported limitations are computational and architectural: PIL is described as $\mathcal{O}(K!)$ or $\mathcal{O}(K^3)$ with the Hungarian algorithm, and it complicates joint optimization with token-level ASR objectives (Park et al., 2024).

Sortformer was proposed specifically to change that training geometry. The model is described as a novel neural model for speaker diarization, trained with unconventional objectives compared to existing end-to-end diarization models, and as an encoder-based diarization model originally designed to supervise speaker tagging in speech-to-text systems (Park et al., 2024, Gedeon et al., 27 Oct 2025). A common misconception is that Sortformer is simply a replacement for diarization clustering. The papers instead frame it more precisely: it is an end-to-end diarization model whose sorted supervision scheme is intended both to resolve speaker permutation and to make diarization outputs compatible with multispeaker ASR training (Park et al., 2024).

2. Arrival-time ordering and Sort Loss

The defining mechanism is arrival-time order (ATO). Let $t_k^{(0)}$ be the first frame where speaker $k$ is active, expressed as

$$\Psi(\mathbf{y}_k) = \min \{ t' \mid y_{k,t'} \neq 0, \ t' \in [1, T] \}.$$

A sorting permutation $\mathbf{P} \in \mathbb{R}^{K \times T}$0 is then defined so that

$\mathbf{P} \in \mathbb{R}^{K \times T}$1

Ground-truth speaker activity is reordered by that arrival-time rule, and Sort Loss is given as

$\mathbf{P} \in \mathbb{R}^{K \times T}$2

The stated effect is that the model is forced to align each output row with the speaker occupying the corresponding arrival position, so permutation is resolved by a deterministic, signal-derived convention rather than by post hoc search (Park et al., 2024).

The original paper does not position Sort Loss as requiring the elimination of PIL. Instead, it explicitly states that Sortformer can autonomously resolve permutation with or without PIL, and it reports a hybrid objective

$\mathbf{P} \in \mathbb{R}^{K \times T}$3

with $\mathbf{P} \in \mathbb{R}^{K \times T}$4 set empirically, for example $\mathbf{P} \in \mathbb{R}^{K \times T}$5 (Park et al., 2024). That detail matters because the empirical record in the same paper shows three distinct variants—PIL-only, Sort-only, and Hybrid-Loss—rather than a single canonical training recipe.

The literature repeatedly emphasizes that the sorting is active rather than a passive relabeling of outputs. In the original exposition, the network is trained to internalize arrival-time ordering, and later descriptions summarize this as a Sort Loss that enforces a consistent ordering of speaker labels, jointly modeling speaker assignment and temporal continuity (Park et al., 2024, Gedeon et al., 27 Oct 2025). A plausible implication is that Sortformer is best understood as a permutation-resolution principle embedded into model learning, not merely an output-formatting convention.

3. Diarization architecture and the ASR bridge

The diarization backbone reported for the original model uses NEST (NeMo Encoders for Speech Tasks), a Fast-Conformer-based architecture with efficient 8x subsampling and frame length 80 ms, followed by a transformer stack with 18 layers, hidden size 192, and relative positional embedding (Park et al., 2024). The model takes audio features $\mathbf{P} \in \mathbb{R}^{K \times T}$6 and outputs a posterior probability matrix

$\mathbf{P} \in \mathbb{R}^{K \times T}$7

with sigmoid activations because speaker activities are treated as independent rather than mutually exclusive (Park et al., 2024). In later work, Sortformer is also described as producing up to four speaker activity tracks and being trained for up to 4 speakers per session (Medennikov et al., 24 Jul 2025, Park et al., 2024).

The second architectural contribution is the explicit bridge between diarization timestamps and ASR tokens. Sortformer inserts special speaker tokens such as $\mathbf{P} \in \mathbb{R}^{K \times T}$8 into transcripts, with those tokens also sorted by arrival-time order. For each word or token, a speaker token is inserted using word-level alignment, or approximated word-level timestamps derived from segment-level labels using a syllable-based heuristic when word timestamps are unavailable (Park et al., 2024).

To inject diarization supervision into ASR, the original paper introduces a sinusoidal kernel over speaker indices. For speaker $\mathbf{P} \in \mathbb{R}^{K \times T}$9 and feature dimension $\Pi$0,

$\Pi$1

and the speaker kernel for speaker $\Pi$2 is

$\Pi$3

If $\Pi$4 collects these kernels and $\Pi$5 is the ASR encoder state, the enriched state is given as

$\Pi$6

This is the mechanism by which speaker label estimation is embedded within the ASR encoder state using a sinusoidal kernel function, thereby enabling standard token-level objectives for multispeaker ASR under a sorted speaker convention (Park et al., 2024).

The multispeaker ASR variant is also reported to use adapter techniques for parameter-efficient transfer to large pretrained ASR models, with adapters inserted in every encoder and decoder layer and only adapters fine-tuned for multispeaker adaptation (Park et al., 2024). This supports the broader claim in the original title that Sortformer bridges timestamps and tokens.

4. Reported performance across representative evaluations

The empirical record presented across the cited papers is heterogeneous because Sortformer appears in several roles: as the primary model in its original paper, as a zero-shot diarization baseline on LibriConvo, and as the basis for a streaming extension. The following table summarizes representative reported results.

Setting	Configuration	Representative result
Original diarization benchmarks	Sortformer-Hybrid-Loss	DIHARD3: 14.76; CALLHOME-part2: 5.87/8.46/12.59; CH109: 6.86
LibriConvo zero-shot baseline	diar_sortformer_4spk-v1	Validation DER: 12.9; Test DER: 11.1
Streaming diarization	Streaming Sortformer, 1.04s latency	CALLHOME 2spk: 7.35; DIHARD III $\Pi$7 spk: 14.57

In the original benchmark study, Sortformer-Sort-Loss and Sortformer-PIL are reported as near each other on several diarization conditions, while Sortformer-Hybrid-Loss is strongest overall: 14.76 on DIHARD3, 5.87/8.46/12.59 on CALLHOME-part2 for 2/3/4 speakers, and 6.86 on CH109. The paper characterizes this as performance competitive with state-of-the-art end-to-end diarization models trained exclusively with PIL, and the detailed excerpt states that the Hybrid-Loss variant outperforms many previous EEND variants (Park et al., 2024).

On LibriConvo, Sortformer is evaluated zero-shot, “as-is,” without additional fine-tuning on the new corpus. Using the diar_sortformer_4spk-v1 model, the reported DER is 12.9 on Validation and 11.1 on Test, compared with 25.6 and 24.4 for the pyannote pipeline. The accompanying analysis states that Sortformer achieves less than half the DER of pyannote on both splits and exhibits a narrower per-recording DER distribution, particularly in conversations with overlap or rapid turn-taking (Gedeon et al., 27 Oct 2025).

These evaluations support two distinct conclusions that should not be conflated. First, the original paper argues that Sort Loss is a viable replacement or complement to PIL in offline diarization training. Second, later corpus studies show that a pretrained Sortformer can generalize strongly, even without corpus-specific adaptation, on realistic simulated conversational data (Park et al., 2024, Gedeon et al., 27 Oct 2025).

5. Streaming Sortformer and the Arrival-Order Speaker Cache

The original Sortformer is described as offline-only, which makes it unsuitable for low-latency deployment. “Streaming Sortformer: Speaker Cache-Based Online Speaker Diarization with Arrival-Time Ordering” extends the framework by introducing an Arrival-Order Speaker Cache (AOSC) that stores frame-level acoustic embeddings of previously observed speakers, ordered by speaker index corresponding to their arrival-time order (Medennikov et al., 24 Jul 2025).

At each streaming step, predictions are formed from the concatenation of three elements: the AOSC speaker cache, a FIFO queue of preceding audio chunks, and the current input buffer. The rationale is that arrival-time ordering already resolves the chunk-to-chunk permutation ambiguity that complicates prior streaming EEND approaches, so cache organization can follow the same convention (Medennikov et al., 24 Jul 2025).

The cache update logic is explicit. Let $\Pi$8 be the maximum cache length and $\Pi$9 the minimum number of frames kept per speaker. Given frame-wise model outputs $K!$0, the speaker score is written as

$K!$1

Frames with all $K!$2 are treated as silence for average silence embedding computation; frames where $K!$3 are assigned $K!$4; recent frames may be boosted by $K!$5; the top $K!$6 frames per speaker may be boosted by $K!$7; and $K!$8 copies of the average silence embedding are appended after each speaker with $K!$9. The final cache is obtained by selecting the top $$\mathcal{L}_{\text{PIL}}(\mathbf{Y}, \mathbf{P}) = \min_{\pi \in \Pi} \frac{1}{TK} \sum_{k=1}^{K} \sum_{t=1}^{T} \mathcal{L}_{\text{BCE}}(y_{\pi(k),t}, p_{k,t}).$$0 scored frames while preserving order (Medennikov et al., 24 Jul 2025).

The reported results indicate graceful degradation as latency decreases. For CALLHOME 2spk, Offline Sortformer reports 6.7%, while Streaming Sortformer reports 6.8% at 10s, 7.35% at 1.04s, and 8.60% at 0.32s latency. For DIHARD III $$\mathcal{L}_{\text{PIL}}(\mathbf{Y}, \mathbf{P}) = \min_{\pi \in \Pi} \frac{1}{TK} \sum_{k=1}^{K} \sum_{t=1}^{T} \mathcal{L}_{\text{BCE}}(y_{\pi(k),t}, p_{k,t}).$$1 spk, the corresponding figures are 15.47% for offline and 14.79%, 14.57%, and 14.63% for the three streaming settings. The paper also states that the Real-Time Factor is always below 0.2, even at the lowest latency (Medennikov et al., 24 Jul 2025).

A common misunderstanding is that streaming Sortformer is merely the original model run chunk-wise. The cited paper makes a narrower claim: the streaming extension depends on an explicit cache mechanism keyed to arrival-order speaker indices. In that sense, AOSC is not an optimization detail but the central device that operationalizes Sortformer’s ordering principle in online inference (Medennikov et al., 24 Jul 2025).

6. Synthetic-data studies, specialized deployments, and comparative position

Later work uses Sortformer as a test case for how diarization quality depends on synthetic conversation design. In “Mind the Gap: Impact of Synthetic Conversational Data on Multi-Talker ASR and Speaker Diarization,” Sortformer is described as an end-to-end neural diarization system designed to resolve speaker permutations via a Sort Loss using the order of speaker arrival times, built on a NEST-FastConformer backbone. The study reports that simulation choices are highly task-dependent: fitting turn-taking statistics to real corpora yields modest gains, but overlap boosting degrades diarization, moving macro DER from 26.1% to 27.6%. Acoustic augmentation is beneficial, with None at 26.1, + noise at 24.6, + reverb at 24.3, and + noise + reverb at 22.2. For training strategies, Synthetic only gives 22.2, Real only gives 17.4, Real + synthetic gives 16.3, and Synthetic → real gives 15.5 macro DER (Polok et al., 14 May 2026). This suggests that Sortformer is unusually informative for simulation research because its end-to-end diarization behavior is sensitive to overlap, source diversity, and acoustic realism.

Sortformer has also been tested in a specialized clinical-adjacent setting. In Korean toddler speech, where young female caregivers often use aegyo and thereby become acoustically similar to the child, the reported result is that NeMo SortFormer achieves 88.69% speaker count accuracy and 33.04% diarization error rate (DER). The paper attributes this to the arrival-time-sorted transformer architecture, which can use temporal order and conversational structure when voice characteristics alone are confounded (Woodbridge et al., 8 Jun 2026). A plausible implication is that the arrival-order prior is especially useful when role structure is stable but speaker timbre is not strongly separable.

Within joint ASR and diarization, Sortformer occupies an intermediate historical position. “Train Short, Infer Long: Speech-LLM Enables Zero-Shot Streamable Joint ASR and Diarization on Long Audio” describes Sortformer as a strong recent baseline for local, non-streaming joint ASR and diarization on audio up to 20s. In that comparison, Sortformer reports cpWER 26.71 on AMI Test and 21.45 on CH109, while the new Speech-LLM system is reported to surpass it in the local setting and to address long-form streamable inference more directly (Shi et al., 20 Nov 2025). The key limitation identified there is not that Sortformer fails at speaker-attributed transcription, but that its original architecture is local and offline, whereas long-audio streaming requires a persistent speaker memory or a different global mechanism (Shi et al., 20 Nov 2025).

Taken together, the cited literature presents Sortformer as more than a single diarization model. It is a family of sorted-objective methods centered on arrival-time ordering: originally an offline diarization model with Sort Loss and an ASR bridge (Park et al., 2024), later extended to low-latency diarization with AOSC (Medennikov et al., 24 Jul 2025), and subsequently used as a benchmark or component in studies of realistic simulated conversations (Gedeon et al., 27 Oct 2025), synthetic-data recipes (Polok et al., 14 May 2026), and domain-specific pipelines such as toddler pronunciation evaluation (Woodbridge et al., 8 Jun 2026).