Arrival-Order Speaker Cache in Streaming Sortformer

• AOSC is a speaker memory mechanism that retains frame-level acoustic embeddings in arrival order to enable real-time diarization via Streaming Sortformer.
• It replaces traditional speaker-tracing buffers by eliminating explicit permutation resolution, ensuring continuity and stable speaker indexing across audio chunks.
• Its dynamic score-based update compresses embeddings, integrates silence markers for transition detection, and supports robust low-latency streaming performance.

Searching arXiv for the primary paper and related Sortformer diarization work. Searching arXiv for "Sortformer speaker diarization". Arrival-Order Speaker Cache (AOSC) is a speaker memory mechanism introduced as part of Streaming Sortformer, a streaming extension of the Sortformer speaker diarization framework whose defining property is that output speakers are ordered by arrival time. AOSC stores frame-level acoustic embeddings extracted from the NEST encoder pre-encode module and organizes them by speaker index in arrival-time order. In Streaming Sortformer, this cache serves simultaneously as state maintenance and as a streaming alignment device: it carries speaker evidence across chunks while preserving the same ordering semantics used by Sortformer, thereby removing the need for an explicit inter-chunk permutation-resolution step (Medennikov et al., 24 Jul 2025).

1. Definition and core role

AOSC is a speaker cache designed for online diarization in which the first speaker to appear is stored as speaker 0, the second as speaker 1, and so on. Its function is to retain a compact history of speaker evidence from past chunks so that each new chunk can be diarized using both the accumulated speaker memory and the current input chunk together with right context (Medennikov et al., 24 Jul 2025).

The mechanism was introduced because the original Sortformer is an offline model that relies on full-length self-attention and therefore cannot directly operate in streaming settings; it also has a maximum input-length limitation. AOSC addresses this by making Sortformer streaming-compatible without changing its core arrival-order prediction principle. A second purpose is to avoid explicit permutation resolution between chunks, which is a standard difficulty in streaming diarization. The design is positioned as a replacement for a conventional speaker-tracing buffer and does not rely on self-attention attractors, local/global attractors, or permutation-resolution operations for speaker-tracing buffers (Medennikov et al., 24 Jul 2025).

The resulting interpretation of AOSC is precise: it is not merely a history buffer, but a speaker-aware, arrival-order-preserving cache whose internal indexing matches Sortformer’s output convention by construction. This suggests that AOSC is best understood as a memory structure specialized to the inductive bias imposed by Sortformer’s arrival-time ordering.

2. Streaming inference formulation

Streaming Sortformer with AOSC is summarized by the recurrence $\mathbf{P}_n$6 $\mathbf{P}_n$7 $\mathbf{P}_n$8 where $\mathbf{C}_n$ is the current chunk, $\mathbf{B}_n$ is the cache at step $n$, $\mathbf{P}_n$ are predictions for the current chunk, $\mathbf{P}_n^{buf}$ are predictions corresponding to the buffered frames, and the underscore denotes predictions for the cache region that are discarded after inference (Medennikov et al., 24 Jul 2025).

This formulation makes the division of labor explicit. AOSC updates the speaker memory from past buffered predictions and past audio, and Sortformer consumes the concatenation of cache and current chunk. The cache therefore mediates continuity across chunks while leaving the underlying diarization network unchanged at inference time except for the addition of streaming state.

The streaming system concatenates three forms of context before inference: the speaker cache $\mathbf{B}_n$, a FIFO queue of preceding audio chunks, and the input buffer containing the current chunk and right context. The FIFO queue exists because short-chunk inference harms accuracy due to insufficient context. It permits a larger effective context and allows the cache to be updated less frequently than every single chunk, improving robustness and efficiency. When frames in the FIFO queue are pushed out, they are processed by the AOSC update mechanism (Medennikov et al., 24 Jul 2025).

AOSC and the FIFO queue therefore play distinct roles. The former is the speaker-state memory; the latter is a temporal context buffer. A plausible implication is that the architecture separates identity persistence from local acoustic context rather than conflating the two in a single buffer.

3. Relation to conventional speaker-tracing buffers

The paper contrasts AOSC with the conventional speaker-tracing buffer (STB) used in earlier streaming diarization systems. In the STB formulation, $\mathbf{P}_n$9 $\mathbf{P}_n^{buf}$0 $\mathbf{P}_n^{buf}$1 $\mathbf{P}_n^{buf}$2 $\mathbf{P}_n^{buf}$3 where $S$ is the number of speakers, $\text{perm}(S)$ is the set of speaker permutations, $\text{CC}$ is the correlation coefficient, and $\psi$ is the permutation that maximizes similarity between buffered predictions and newly predicted buffered outputs (Medennikov et al., 24 Jul 2025).

The critical distinction is that STB is fundamentally a reordering or matching device. It stores previous predictions and previous chunk information, then performs an explicit matching step to resolve speaker permutations across chunks. AOSC removes that permutation search. Because Sortformer predicts speakers in arrival-time order and the cache stores embeddings in the same order, speaker identity across chunks is maintained naturally rather than through an explicit buffer permutation algorithm (Medennikov et al., 24 Jul 2025).

The comparison can be summarized succinctly:

Mechanism	Stored state	Cross-chunk alignment
STB	Previous predictions and previous chunk information	Explicit permutation search using $\mathbf{B}_n$0
AOSC	Frame-level acoustic embeddings organized by speaker arrival order	No permutation search; ordering is preserved by construction

This difference is architectural rather than cosmetic. AOSC is specialized to preserve Sortformer’s ordering semantics, so the streaming system does not need an explicit inter-chunk permutation-resolution step (Medennikov et al., 24 Jul 2025).

4. Cache update and compression mechanism

The AOSC update procedure is score-based and bounded by a maximum speaker cache length $\mathbf{B}_n$1. If the input sequence length is smaller than $\mathbf{B}_n$2, AOSC behaves as a no-op. If the sequence is longer than $\mathbf{B}_n$3, the cache is compressed back to length $\mathbf{B}_n$4 (Medennikov et al., 24 Jul 2025).

For each frame and speaker index $\mathbf{B}_n$5, the paper defines the score $\mathbf{P}_n^{buf}$4 where $\mathbf{B}_n$6 is the predicted activity probability for speaker $\mathbf{B}_n$7. A high score indicates that speaker $\mathbf{B}_n$8 is likely active while other speakers are inactive. This functions as a frame-level confidence score for speaker assignment (Medennikov et al., 24 Jul 2025).

The update mechanism then identifies silence frames, defined as frames where the model assigns low probability to all speakers, and computes the average silence embedding over those frames. This silence embedding is later used as a placeholder for silence-related cache entries. Non-speech scores are suppressed by the rule $\mathbf{P}_n^{buf}$5 Thus, frames that do not provide credible evidence for speaker $\mathbf{B}_n$9 are excluded from selection (Medennikov et al., 24 Jul 2025).

The scoring is further shaped in two ways. First, newly added embeddings receive a score increase by $n$0, which biases the cache toward newer information. Second, the top $n$1 scores for each speaker are boosted by a constant $n$2, ensuring that every represented speaker retains at least some frames in the cache. During training, this step was applied twice: strong boosting of $n$3 frames per speaker by $n$4, and weak boosting of $n$5 frames per speaker by $n$6 (Medennikov et al., 24 Jul 2025).

The method also appends $n$7 scores of $n$8 for each speaker, corresponding to the average silence embedding. The paper explicitly states that $n$9 silence embeddings are appended after each speaker’s embeddings to facilitate speaker change detection. After concatenating the scores for all speakers, the system selects the $\mathbf{P}_n$0 highest-scoring frames and returns the corresponding embeddings while preserving order. For positions associated with $\mathbf{P}_n$1 or $\mathbf{P}_n$2 scores, the average silence embedding is used (Medennikov et al., 24 Jul 2025).

Several design properties follow directly from this procedure. The cache is not a fixed top-$\mathbf{P}_n$3 memory per speaker; the number of stored embeddings per speaker is determined dynamically by the score-based update mechanism, though a minimum of $\mathbf{P}_n$4 frames per speaker is enforced whenever possible. It retains dominant speech evidence, representation for active speakers, and explicit silence markers for transition detection within a fixed memory budget (Medennikov et al., 24 Jul 2025).

5. Dependence on arrival-time ordering in Sortformer

AOSC depends on a specific property of Sortformer: the model is trained so that output speakers are ordered by arrival time. The paper attributes this to Sort Loss, described at a high level as binary cross-entropy over sorted target labels together with conventional permutation-invariant loss. Although the full Sort Loss equation is not reproduced in the provided material, the stated role of the loss is to teach the model arrival-time ordering (Medennikov et al., 24 Jul 2025).

That training property is what makes AOSC possible. Since the model already produces output in arrival-time order, the cache can be organized in the same order, and no permutation alignment is needed across chunks. The first observed speaker is indexed as 0, the second as 1, and so forth; this indexing remains meaningful because the model’s output semantics are stable under streaming execution (Medennikov et al., 24 Jul 2025).

The dependence is therefore structural. AOSC is not a generic cache for arbitrary diarization architectures. It is tailored to a model whose speaker indices have a deterministic interpretation tied to first appearance. A plausible implication is that transferring AOSC to models without arrival-time ordering would require reintroducing a separate alignment mechanism or a different training objective.

6. Empirical role in low-latency streaming diarization

AOSC is central to making Sortformer function in streaming and low-latency configurations. The paper evaluates latency settings of 10.0 s, 1.04 s, and 0.32 s, with corresponding chunk sizes, right context, FIFO queue length, update period, and speaker cache size. The reported real-time factors are 0.005 for 10.0 s latency, 0.093 for 1.04 s latency, and 0.180 for 0.32 s latency (Medennikov et al., 24 Jul 2025).

The experiments support several specific conclusions about the cache mechanism. An Offline Sortformer-AOSC variant shows that AOSC can be attached even without streaming fine-tuning, but diarization error rate increases significantly in that configuration, indicating that training the model with AOSC is important for best performance. The fully streaming, AOSC-trained system performs very well across DIHARD III, CALLHOME, and CH109 and is competitive with or better than prior streaming systems. Performance degradation at 0.32 s latency is reported as not severe, which the paper highlights as evidence of robustness and flexibility (Medennikov et al., 24 Jul 2025).

The method also performs especially well on DIHARD III and CALLHOME subsets, and it remains solid on 5+ speaker subsets despite being designed for up to 4 speakers, suggesting that it can track the four most dominant speakers effectively. In DIHARD III and CALLHOME 4+ speakers, the streaming version outperforms the offline model; the paper attributes this likely to mismatch with 90-second training samples in the offline model, whereas streaming uses a fixed inference window (Medennikov et al., 24 Jul 2025).

These findings define AOSC’s empirical significance. It is not presented only as a memory-saving device, but as the enabling component that allows a previously offline arrival-order diarization model to operate in real time while preserving stable speaker indexing across chunks.

7. Interpretation, scope, and related conceptual usage

Within diarization, AOSC denotes a concrete mechanism: a bounded, dynamically maintained cache of frame-level acoustic embeddings ordered by speaker arrival time and augmented with silence embeddings for transition detection (Medennikov et al., 24 Jul 2025). Its direct purpose is cross-chunk speaker tracking without explicit permutation resolution, and its operation is tightly coupled to Streaming Sortformer’s inference loop and training semantics.

A separate 2025 paper, “Online Stochastic Matching with Unknown Arrival Order: Beating $\mathbf{P}_n$5 against the Online Optimum” (Sun et al., 25 Mar 2025), uses “arrival order” in a different theoretical sense. That work studies unknown-arrival-order online stochastic matching and introduces a stage-based, per-offline-vertex state mechanism rather than a literal speaker cache. The paper itself notes that this is not a cache in the literal sense, but that it is stateful and order-sensitive and can be viewed as a conceptual cousin to an arrival-order-sensitive cache (Sun et al., 25 Mar 2025). This comparison is interpretive rather than terminological: the matching paper does not define AOSC, nor does it address diarization.

A common misconception would therefore be to treat AOSC as a generic label for any online method that tracks state under unknown arrival order. In the diarization literature represented here, the term has a narrower meaning. It refers specifically to the cache mechanism in Streaming Sortformer that preserves arrival-time speaker ordering, stores NEST pre-encode frame embeddings, applies score-based compression with dynamic per-speaker retention, and inserts silence placeholders to facilitate speaker transition detection (Medennikov et al., 24 Jul 2025).

In that specific sense, AOSC occupies a precise place in the evolution of streaming diarization: it replaces the conventional STB’s explicit permutation matching with a cache whose indexing is already aligned with the model’s arrival-time ordering. The result is a streaming, low-latency diarization system that maintains speaker continuity by construction rather than by post hoc permutation search (Medennikov et al., 24 Jul 2025).