Pyannote 3.1 Diarization Pipeline

• Pyannote 3.1-based pipeline is a flexible diarization framework that segments audio, extracts speaker embeddings, and clusters them to form a global speaker timeline.
• It supports diverse local segmentation methods and modifications, enhancing inference speed and domain adaptation without discarding the core pipeline structure.
• Accelerations such as larger segmentation strides and per-chunk embedding, along with ASR coupling, reveal practical trade-offs between speed and diarization error rates.

Searching arXiv for the specified papers and closely related Pyannote 3.1 pipeline work. A Pyannote 3.1-based pipeline is a speaker diarization framework built on pyannote.audio 3.1 in which audio is segmented into local speech regions, transformed into speaker representations, clustered into speaker identities, and reassembled into a global “who-spoke-when” timeline. In recent work, this framework appears both in its standard segmentation–embedding–clustering form and in modified forms that replace or fine-tune the local segmentation stage, alter the embedding budget and clustering rules for faster inference, or couple diarization with ASR alignment and external VAD systems (Yamaguchi, 7 Jun 2026, Chowdhury et al., 5 Mar 2026, Polok et al., 16 Jun 2025, Han et al., 2024, Plaquet et al., 2024).

1. Canonical pipeline structure

In a standard Pyannote 3.1 pipeline description, a sliding window with default 1 s length and 1 s hop is passed through the pyannote/segmentation-3.0 neural model. The model outputs a per-frame “speech vs. non-speech” score, and post-processing yields time-stamped homogeneous speech segments. Each segment is then passed to a speaker-embedding network, such as ResNet34 or CAM++, producing a $D$-dimensional vector; clustering is performed with agglomerative hierarchical clustering (AHC) using a cosine similarity matrix and centroid linkage, merging until the linkage distance exceeds a fixed threshold, after which clusters smaller than a minimum cluster size are pruned and their segments reassigned to surviving clusters (Yamaguchi, 7 Jun 2026).

Other Pyannote 3.1 implementations preserve this global decomposition while changing the local front end. In one WavLM-based formulation, segmentation inherits the Pyannote pattern of splitting a long waveform into overlapping chunks of length $T_s=8$ s, with hop $H_s=0.8$ s at inference and 6 s at training; after local diarization, speaker embeddings are extracted on pure-speech regions with ResNet34-LM and clustered with AHC across chunks, followed by stitching through overlapping-chunk probability averaging (Han et al., 2024). In DiariZen, the usual Pyannote segmentation and embedding modules are bypassed: chunking and data loading still rely on FileCrop and SlidingWindowFeature, but segmentation is handled by a local EEND model and embeddings by an external ResNet34-LM model, while Pyannote’s AHC implementation is reused unchanged (Polok et al., 16 Jun 2025).

Variant	Pyannote 3.1 elements retained	Main modifications
Fast on-device pipeline	pyannote/segmentation-3.0, embedding extraction, AHC	coarser stride, per-chunk embedding, relative minimum cluster size
WhisperAlign diarization	speaker-diarization-community-1, segmentation-3.0	segmentation fine-tuning, Silero VAD masking, WhisperX word anchoring
DiariZen	chunking, AHC, post-processing	local EEND replaces segmentation, external ResNet34-LM embeddings
WavLM+Conformer pipeline	embedding extraction, AHC, stitching	SincNet+4×Bi-LSTM replaced by WavLM(Base+)+4×Conformer
Mamba-based pipeline	downstream embedding and clustering	Bi-LSTM segmentation replaced by BiMamba blocks

This recurring structure is important because it distinguishes Pyannote 3.1 as a reusable diarization scaffold rather than a single fixed model. A plausible implication is that many reported gains arise not from abandoning the pipeline, but from altering one stage while preserving the surrounding interfaces.

2. Local segmentation formulations

The local segmentation stage is the main locus of architectural variation. In WhisperAlign, the segmentation model is Pyannote segmentation-3.0 with approximately 1.4 M parameters, initialized from the community-1 checkpoint and fine-tuned on 113 Bangla recordings with CSV annotations of (start_time, end_time, speaker_id). Training uses Pyannote’s default binary cross-entropy loss on frame-level voice and speaker labels, converging in approximately 20 epochs on a single T4 GPU (Chowdhury et al., 5 Mar 2026).

In the WavLM-based Pyannote 3.1 formulation, the original local EEND module SincNet + 4×Bi-LSTM is replaced by WavLM (Base+) + 4×Conformer, with no positional encoding. WavLM layer outputs are fused through learned weights,

$$w_\ell = \frac{\exp(\alpha_\ell)}{\sum_{j=0}^{L}\exp(\alpha_j)}, \qquad f_t = \sum_{\ell=0}^{L} w_\ell \cdot h_t^{(\ell)},$$

then projected to $d=256$ and passed through four Conformer blocks. The classifier predicts powerset states for up to 4 speakers plus silence, yielding $C=11$ classes and optimized with multi-class cross-entropy (Han et al., 2024).

DiariZen uses a different local diarization formulation. Each approximately 30 s chunk is fed into an EEND model with WavLM Large as backbone, weighted-sum pooling over the 24 transformer layers, a 12-layer Conformer encoder, and a linear classification head over the $2^2=4$ powerset states $\{$silence, speaker A only, speaker B only, overlap$\}$. The objective is powerset cross-entropy,

$$L = - \sum_{t=1}^{T}\sum_{c \in \{S,T,N,O\}} y_{t,c}\log p(c \mid X)_t.$$

This configuration is explicitly two-speaker (Polok et al., 16 Jun 2025).

A third modification replaces the local segmentation model with Bi-Directional Mamba blocks. In that system, frozen WavLM Base features are linearly projected from 768 to 256 dimensions, processed by 7 BiMamba layers, and mapped through 2× Linear(256→128) + Linear(128→C) to per-frame speaker-activity scores. The Mamba recurrence is written as

$T_s=8$0

with gating

$T_s=8$1

The reported motivation is that attention-based models have unsuitable memory requirements for long-form audio, while Mamba can use longer local windows (Plaquet et al., 2024).

Taken together, these formulations show that “Pyannote 3.1-based” does not imply a uniform local objective. Binary cross-entropy on segmentation labels, powerset cross-entropy, multilabel objectives with permutation-invariant training, and sequence models ranging from Bi-LSTM to Conformer and Mamba all appear within the same downstream pipeline pattern.

3. Embeddings, clustering, and global speaker assignment

The embedding and clustering stage remains comparatively stable across the literature. In the on-device pipeline, each speech segment is passed to a speaker-embedding network such as ResNet34 or CAM++, producing a $T_s=8$2-dimensional vector; the implementation reported uses wespeaker-voxceleb-campplus in float16 (Yamaguchi, 7 Jun 2026). WhisperAlign instead uses wespeaker-voxceleb-resnet34-LM with 6.6 M parameters as a frozen embedding model within speaker-diarization-community-1 (Chowdhury et al., 5 Mar 2026). DiariZen also uses ResNet34-LM trained with WeSpeaker on VoxCeleb2, extracting a 256-dimensional x-vector from each speech-only segment (Polok et al., 16 Jun 2025).

AHC is the dominant global clustering mechanism. In the fast on-device pipeline, clustering is performed with centroid linkage over a cosine similarity matrix, stopping when the linkage distance exceeds the Pyannote threshold; clusters smaller than a minimum cluster size are pruned and their members reassigned (Yamaguchi, 7 Jun 2026). In the WavLM+Conformer system, pairwise cosine similarity

$T_s=8$3

is followed by agglomerative clustering with threshold $T_s=8$4, restricting the number of clusters to $T_s=8$5 and using minimum cluster size 30, while enforcing that embeddings from the same chunk cannot merge (Han et al., 2024). In WhisperAlign, the clustering threshold is 0.58, min_duration_off = 0.05 s, and exclusive_speaker_diarization=True enforces a single speaker per frame by assigning overlaps to the earliest dominant speaker (Chowdhury et al., 5 Mar 2026). In the Mamba-based system, centroid-linkage AHC is retained, but threshold and minimum cluster size are tuned by Optuna over 300 validation trials, with threshold in $T_s=8$6 and min_cluster_size in $[3,30]` (Plaquet et al., 2024).

Global speaker assignment also requires cross-window alignment and overlap handling. The WavLM+Conformer pipeline averages per-speaker probabilities across overlapping predictions after AHC-based cluster assignment (Han et al., 2024). DiariZen maps local chunk speaker A/B labels to two global clusters, stitches adjacent chunks, and resolves overlaps so that only one speaker remains per frame in the final RTTM (Polok et al., 16 Jun 2025). The Mamba system performs permutation alignment on overlapping window regions, selecting the speaker permutation that maximizes agreement with neighbors before averaging overlapped frame scores (Plaquet et al., 2024).

One methodological tension is explicit in the reported results: better local segmentation metrics do not always imply a better full pipeline. In the Mamba study, powerset improves BiLSTM local DER but harms final pipeline DER because of increased speaker confusion downstream (Plaquet et al., 2024). This directly counters the common assumption that local-frame objectives and end-to-end diarization outcomes are monotonically aligned.

4. Acceleration and on-device operation

A distinct line of work studies how far a Pyannote 3.1-based pipeline can be accelerated on consumer hardware while preserving DER. The reported recipe makes two changes: increasing segmentation stride from 1 s to 2 s or 3 s, and switching from “per-frame” to “per-chunk” embedding so that exactly one embedding is produced per speech chunk of length equal to the window size. With stride = 3 s, embeddings per recording drop by approximately $T_s=8$7, embedding-extraction time decreases sharply, and on AMI test with headset mix on an Apple M4 laptop with MPS backend the baseline stride = 1 s, per-frame configuration has [RTF](https://www.emergentmind.com/topics/real-time-factor-rtf) ≈ 0.061, DERmicro = 0.081, while stride = 3 s + per-chunk yields RTF ≈ 0.006 (≈9.9× speedup), DERmicro = 0.082 (Yamaguchi, 7 Jun 2026).

The same work shows that this apparent DER neutrality on AMI does not transfer directly to in-the-wild data. On VoxConverse with an RTX 5070 Ti GPU and CUDA backend, the baseline stride1, mcs12 configuration reports DERmicro = 0.075, RTF = 0.004, while stride3 + per-chunk, mcs12 reports DERmicro = 0.113, RTF = 0.001. The degradation is traced to speaker under-counting caused by the interaction between a fixed minimum cluster size and a reduced number of embeddings per speaker (Yamaguchi, 7 Jun 2026).

The proposed correction is a relative minimum cluster size,

$T_s=8$8

where $T_s=8$9 and $H_s=0.8$0 is the total number of segment embeddings for the recording. For AMI, where $H_s=0.8$1 thousands and each speaker contributes approximately 200 embeddings, this leaves DER essentially unchanged; for VoxConverse, where $H_s=0.8$2 under stride = 3 s, it makes mcs ≈ 1, so almost no cluster is pruned. Applied on top of stride = 3 s + per-chunk, this yields DERmicro = 0.079 and RTF = 0.00083 on VoxConverse, while on AMI the final accelerated pipeline reaches RTF = 0.0050 (12.2×) with DER = 0.083 (Yamaguchi, 7 Jun 2026).

The reported speed figures are specific: on Apple M4 with MPS backend, AMI RTF drops from 0.061 to 0.005; on the same laptop restricted to CPU only, AMI RTF drops from 1.84 to 0.22; on RTX 5070 Ti with CUDA backend, VoxConverse RTF drops from 0.0040 to 0.00083; and across all three datasets on Apple M4, the final accelerated pipeline runs at RTF ≈ 0.005–0.007 (Yamaguchi, 7 Jun 2026). The paper further states that a 60 min meeting can be diarized in 30–50 s on an Apple M4, or under 10 s on a mid-range GPU.

These results make a narrower point than a generic “speed–accuracy trade-off” claim. The observed failure is not merely that coarser stride reduces quality; it is that coarser stride changes the embedding budget per recording and thereby invalidates a fixed cluster-pruning heuristic.

5. Domain adaptation, ASR coupling, and task-specific integration

Pyannote 3.1-based pipelines are frequently embedded in larger speech systems rather than used as isolated diarizers. WhisperAlign provides a tightly coupled Bengali long-form ASR and diarization pipeline in which both WhisperX and Pyannote use Silero VAD, creating identical voiced regions. The final speaker timeline from Pyannote is logically intersected with the same Silero VAD mask, and each transcribed word is assigned the speaker label active at its midpoint,

$H_s=0.8$3

The diarization subsystem uses speaker-diarization-community-1, a fine-tuned segmentation-3.0, adaptive post-processing that merges gaps in [0.15 s, 0.8 s] anchored at 0.4 s, and drops segments shorter than 0.15 s (Chowdhury et al., 5 Mar 2026).

The reported ablation for Bengali diarization is also explicit. Public/private DER changes from 0.422 / 0.477 for the Pyannote 3.1 base, to 0.364 / 0.413 after pre-processing, to 0.314 / 0.376 for community-1 out of the box, to 0.200 / 0.261 after fine-tuned segmentation, and finally to 0.194 / 0.260 with dual-VAD intersection. Against the benchmark Pyannote base DER of approximately 40%, the final pipeline reaches approximately 19.4% public DER (Chowdhury et al., 5 Mar 2026).

Domain adaptation also appears in broader far-field diarization studies. In the WavLM+Conformer work, the compound training set is the union of AMI, AISHELL-4, and AliMeeting training sets for a total of 288.3 h; WavLM-updated reduces DER from 21.1/13.7 to 15.4/9.8 on AMI, from 13.9/7.7 to 11.7/5.9 on AISHELL-4, and from 22.8/13.6 to 17.6/10.2 on AliMeeting, compared with the Pyannote 3.1 baseline (Han et al., 2024). In DiariZen, pre-training uses heterogeneous far-field corpora including AMI, AISHELL-4, AliMeeting, NOTSOFAR-1, MSDWild, DIHARD3, RAMC, and VoxConverse, followed by structured pruning and fine-tuning on MLC-SLM data (Polok et al., 16 Jun 2025).

These integrations show that the pipeline often serves as an alignment substrate across tasks. In WhisperAlign, the diarization timeline is deliberately forced into temporal agreement with ASR token times through shared VAD and midpoint assignment (Chowdhury et al., 5 Mar 2026). A plausible implication is that Pyannote 3.1’s downstream utility often depends as much on temporal interoperability with adjacent modules as on intrinsic DER.

6. Failure modes, limitations, and methodological disputes

Several recurrent limitations are reported. First, on very challenging or heavily overlapped recordings such as MSDWild, the embedding-budget mechanism is not the main error driver; DER remains high, and relative minimum cluster size yields little extra benefit. Under CUDA on many.val, the baseline stride1, mcs=12 reports DERmicro/macro = 0.384/0.472, stride3 + per-chunk, mcs=12 reports 0.393/0.481, and + relative mcs (f=0.01) reports 0.389/0.484 (Yamaguchi, 7 Jun 2026). The recommendation given there is that further segmentation refinement or overlap handling may be required.

Second, overlap handling is task-dependent and not conceptually settled. WhisperAlign sets exclusive_speaker_diarization=True, thereby assigning overlaps to the earliest dominant speaker and enforcing single-speaker output per frame (Chowdhury et al., 5 Mar 2026). DiariZen also removes overlaps in the final RTTM after fusing diarization and VAD probabilities, redistributing overlap frames by choosing the speaker with the highest posterior (Polok et al., 16 Jun 2025). These choices are operationally useful for downstream alignment, but they should not be confused with explicit overlap-preserving diarization.

Third, simulated data does not behave uniformly across architectures. In the WavLM+Conformer study, training on simulated data only degrades both Fbank and WavLM systems, and fine-tuning from simulated to real data never benefits WavLM. The paper therefore concludes that simulated data is not needed when leveraging WavLM in those experiments (Han et al., 2024). This directly contradicts a common expectation that more synthetic mixtures automatically improve neural diarization.

Fourth, increased model complexity is not uniformly beneficial. In the Mamba study, a direct swap to a self-attention Transformer of approximately 100 M parameters under-performed both LSTM and Mamba without extreme hyperparameter tuning, and increasing LSTM parameter count did not close the gap to Mamba (Plaquet et al., 2024). The reported contrast is therefore not simply “larger model versus smaller model,” but a difference in sequence inductive bias under long-window diarization constraints.

Finally, annotation quality and clustering heuristics remain limiting factors even when the neural front end improves. DiariZen explicitly identifies missing speech segments and mislabeled internal silences in the MLC-SLM train/dev annotations and proposes VAD fusion as a mitigation (Polok et al., 16 Jun 2025). The fast on-device study shows that a fixed mcs = 12 can silently produce speaker under-counting when the number of embeddings per speaker falls to approximately 10 under stride-accelerated inference (Yamaguchi, 7 Jun 2026). These observations suggest that, within Pyannote 3.1-based pipelines, downstream decision rules and data curation can dominate the realized diarization error as strongly as the local neural architecture.