Highly Efficient Real-Time Streaming and Fully On-Device Speaker Diarization with Multi-Stage Clustering (2210.13690v4)

Abstract: While recent research advances in speaker diarization mostly focus on improving the quality of diarization results, there is also an increasing interest in improving the efficiency of diarization systems. In this paper, we demonstrate that a multi-stage clustering strategy that uses different clustering algorithms for input of different lengths can address multi-faceted challenges of on-device speaker diarization applications. Specifically, a fallback clusterer is used to handle short-form inputs; a main clusterer is used to handle medium-length inputs; and a pre-clusterer is used to compress long-form inputs before they are processed by the main clusterer. Both the main clusterer and the pre-clusterer can be configured with an upper bound of the computational complexity to adapt to devices with different resource constraints. This multi-stage clustering strategy is critical for streaming on-device speaker diarization systems, where the budgets of CPU, memory and battery are tight.

• The paper introduces a multi-stage clustering framework that adapts methods based on audio length, using AHC for short inputs and spectral clustering for medium inputs to improve diarization accuracy.
• It employs dynamic compression for long audio segments to cap computational complexity, ensuring efficient processing on mobile devices.
• Experimental results show reduced diarization error rates and lower floating point operations, confirming its practical viability for live transcription and voice-activated applications.

Highly Efficient Real-Time Streaming and Fully On-Device Speaker Diarization with Multi-Stage Clustering

This paper presents a novel approach to speaker diarization that prioritizes computational efficiency for on-device applications, particularly on mobile devices. The proposed multi-stage clustering framework employs various clustering techniques to address the challenges of processing audio inputs of differing lengths while remaining within the constraints of limited device resources such as CPU, memory, and battery.

Multi-Stage Clustering Strategy

The paper's central contribution is a multi-stage clustering strategy that consists of several components:

1. Fallback Clusterer: Agglomerative Hierarchical Clustering (AHC) is employed as a fallback for short audio inputs, effectively distinguishing between single and multiple speakers. This is crucial for initial processing and when spectral clustering is not suitable due to insufficient data points.
2. Main Clusterer: Spectral clustering is utilized for medium-length inputs, leveraging its strength in accurately estimating the number of speakers via the eigen-gap criterion.
3. Pre-Clusterer with Dynamic Compression: For longer inputs, AHC is used to pre-cluster and compress the data before spectral clustering is applied. This approach ensures an upper limit on computational complexity, making it feasible for resource-constrained devices.

Performance and Evaluation

The authors conducted experiments across various datasets, revealing the effectiveness of their method in both short-form and long-form scenarios. Key findings include:

• The AHC fallback clusterer significantly improves diarization error rates (DER) for short-form audio, addressing the spectral clustering's limitations with small data sets.
• The dynamic compression technique allows the system to handle much larger data without exceeding computational limits. This is particularly evident in the lower floating point operations (FLOPs) compared to unbounded clustering methods.

Practical Implications

The implementation of this system is particularly beneficial for real-time, on-device applications such as live transcription, meeting or lecture recording, and voice-activated personal assistants. By effectively managing the balance between computational efficiency and diarization quality, it facilitates practical deployment on devices with stringent resource constraints.

Theoretical Implications and Future Work

Theoretically, the framework exemplifies a structured approach to integrating multiple clustering algorithms to resolve domain-specific challenges. Future research might explore other clustering combinations or adaptation to additional device constraints. Furthermore, advancing the balance between computational efficiency and accuracy can enhance the robustness and applicability of on-device diarization systems.

This paper's contribution underscores a substantial step toward efficient, real-time speaker diarization, offering insights and frameworks applicable to broader AI tasks where device-specific constraints are critical considerations.