Fast and Robust On-Device Speaker Diarization: Relative Minimum Cluster Size for Stride-Accelerated Pipelines

Abstract: Speech applications such as meeting transcription and voice agents would benefit from on-device speaker diarization, but practical adoption is limited by inference cost. We study how far a Pyannote 3.1-based pipeline can be accelerated on consumer hardware (an RTX 5070 Ti GPU and an Apple M4 laptop) while preserving diarization error rate (DER). A simple recipe: coarser segmentation stride and per-chunk embedding, yields multi-fold speedups and is DER-neutral on AMI, but degrades sharply on in-the-wild data: on VoxConverse, DER rises from 0.075 to 0.113. We trace the failure to speaker under-counting in the clustering stage, caused by a fixed minimum cluster size interacting with the reduced number of embeddings per speaker. We propose a relative minimum cluster size, mcs = round(f * n) with f = 0.01, which adapts to the embedding budget per recording. A single value of f recovers VoxConverse DER to 0.079 (about 89% of the lost accuracy) while keeping AMI flat, and the accelerated pipeline reaches up to 12.2x speedup on AMI (MPS) over our CAM++ baseline.

• The paper introduces an adaptive clustering criterion using a relative minimum cluster size to recover most of the lost accuracy in accelerated diarization pipelines.
• It demonstrates that adjusting minimum cluster size relative to total embeddings yields up to 12.2x speedup with minimal degradation in DER on datasets like AMI and VoxConverse.
• The results highlight that tailoring clustering hyperparameters to resource constraints is essential for robust on-device diarization, though challenges remain for more complex datasets.

Fast and Robust On-Device Speaker Diarization via Relative Minimum Cluster Size Adjustment

Problem Formulation and Motivation

On-device speaker diarization pipelines are increasingly relevant in privacy-sensitive and resource-constrained deployment settings, such as voice agents and meeting transcription systems. Several recent Pyannote-based pipelines offer robust diarization but typically incur substantial inference costs, limiting practicality on consumer hardware. While prior works demonstrated that accelerating pipelines by coarsening the segmentation stride is effective, they also reported accuracy degradation, especially in scenarios with many speakers. This paper analytically investigates the underlying cause of this degradation and introduces an adaptive clustering criterion that recovers most lost accuracy without sacrificing speedup.

Methodology

Baseline and Acceleration Heuristics

The baseline system combines the Pyannote 3.1 segmentation model with the CAM++ embedding extractor, evaluated primarily on the AMI, VoxConverse, and MSDWild datasets. Inference speed is measured via the real-time factor (RTF), with lower RTF indicating faster processing. The main acceleration heuristics are:

• Increasing segmentation stride: The segmentation window stride is increased from 1s to 3s.
• Per-chunk embedding computation: Instead of per-frame embedding, a single embedding is extracted per chunk.

Both changes dramatically reduce the number of embeddings computed and subsequently clustered, yielding up to $12.2\times$ speedup on AMI.

Failure Mode Diagnosis

Despite effective speedup, the coarse pipeline exhibits significant DER degradation on in-the-wild data (DER rises from 0.075 to 0.113 on VoxConverse). Diagnostic analysis localizes this to the clustering stage: a fixed minimum cluster size (mcs) in agglomerative hierarchical clustering leads to systematic under-counting of speakers when the number of embeddings per speaker drops below mcs, often absorbing minority speakers into larger clusters. Quantitative and visual evidence substantiate that over-merging is the principal problem.

Figure 1: Comparative visualization of clustering outcomes for the "aggyz" file in VoxConverse; the stride-3 condition merges distinct speakers and loses cluster structure, as evidenced by similarity matrix, dendrogram, and PCA plots.

Proposed Solution: Relative Minimum Cluster Size

The central proposal is to replace the fixed minimum cluster size with a relative criterion: $\mathrm{mcs} = \mathrm{round}(f \cdot n),$ where $n$ is the total number of embeddings for a file, and $f$ is a tunable fraction. A value of $f \approx 0.01$ is found to recover the vast majority of lost DER on VoxConverse (0.113 $\rightarrow$ 0.079; almost 89% of lost accuracy recovered), while keeping DER on AMI essentially flat.

Figure 2: DER as a function of minimum cluster fraction $f$ for AMI and VoxConverse; VoxConverse degrades as $f$ increases, confirming the necessity of a low value.

This simple re-parameterization provides robustness across diverse datasets, setting a single $f$ suffices for both AMI and VoxConverse.

Results and Empirical Analysis

Speed-Accuracy Tradeoffs

• AMI: Speedup reaches $12.2\times$ (RTF $\mathrm{mcs} = \mathrm{round}(f \cdot n),$0) at negligible DER cost.
• VoxConverse: Absolute DER degradation due to coarse stride and per-chunk embedding is almost fully mitigated by the relative mcs, with DER increase limited to $\mathrm{mcs} = \mathrm{round}(f \cdot n),$1 over baseline.
• MSDWild: The relative mcs scheme yields only marginal differences versus a fixed mcs, suggesting its utility is dataset-specific.

Diagnostic Visualizations

Cluster visualization on problematic VoxConverse files demonstrates that with a coarse stride, block structure in the similarity matrix vanishes, dendrograms show fewer branches, and PCA reveals fewer well-separated clusters. These artifacts are corrected by tuning the mcs according to embedding cardinality.

Limitations

• The beneficial effect of relative mcs is not universal; on the more challenging MSDWild dataset, DER remains high ($\mathrm{mcs} = \mathrm{round}(f \cdot n),$2), and the adjustment makes negligible impact.
• The chosen $\mathrm{mcs} = \mathrm{round}(f \cdot n),$3 is robust but not exhaustively tuned on a true held-out development split; this cautions against overfitting.
• Measurement conditions (evaluation protocol and hardware backend) are not fully standardized across datasets but are clearly reported.

Implications and Future Prospects

This work demonstrates that for resource-aware diarization, clustering hyperparameters must adapt to the embedding regime induced by pipeline accelerations. The solution—scaling cluster size constraints with embedding cardinality—offers a computationally trivial yet robust safeguard against under-clustering and should generalize to other diarization pipelines employing agglomerative clustering under varying embedding budgets. The findings indicate a path forward for deploying high-quality diarization directly on consumer hardware, with practical real-time factors even without a discrete GPU.

From a theoretical perspective, the results underscore a critical dependency between embedding granularity and clustering constraints. Future research directions include:

• Extending the adaptive mcs approach to overlapping speech and online diarization scenarios.
• Systematic exploration of embedding budget effects across alternative clustering algorithms, e.g., spectral or mean-shift.
• Investigation of more sophisticated, possibly data-driven, rules for other clustering hyperparameters under compressed pipelines.

Conclusion

By dissecting the failure mechanisms of stride-accelerated diarization and proposing a relative minimum cluster size for adaptive clustering, this paper delivers a method that achieves significant inference acceleration without substantially sacrificing diarization accuracy on diverse datasets. The approach is computationally lightweight, hardware-agnostic, and highlights the importance of tailoring unsupervised hyperparameters to system resource profiles for robust real-world deployment.