Target-Speaker Voice Activity Detection: a Novel Approach for Multi-Speaker Diarization in a Dinner Party Scenario (2005.07272v2)

Abstract: Speaker diarization for real-life scenarios is an extremely challenging problem. Widely used clustering-based diarization approaches perform rather poorly in such conditions, mainly due to the limited ability to handle overlapping speech. We propose a novel Target-Speaker Voice Activity Detection (TS-VAD) approach, which directly predicts an activity of each speaker on each time frame. TS-VAD model takes conventional speech features (e.g., MFCC) along with i-vectors for each speaker as inputs. A set of binary classification output layers produces activities of each speaker. I-vectors can be estimated iteratively, starting with a strong clustering-based diarization. We also extend the TS-VAD approach to the multi-microphone case using a simple attention mechanism on top of hidden representations extracted from the single-channel TS-VAD model. Moreover, post-processing strategies for the predicted speaker activity probabilities are investigated. Experiments on the CHiME-6 unsegmented data show that TS-VAD achieves state-of-the-art results outperforming the baseline x-vector-based system by more than 30% Diarization Error Rate (DER) abs.

• The paper introduces the TS-VAD method that directly predicts individual speaker activities, enhancing the diarization process in complex acoustic environments.
• It iteratively refines speaker i-vectors and employs an attention mechanism in multi-microphone setups to effectively segregate overlapping speech.
• Experimental results demonstrate a DER reduction of over 30% compared to the baseline x-vector system on the CHiME-6 dataset.

Target-Speaker Voice Activity Detection: A Novel Approach for Multi-Speaker Diarization

The paper introduces a Target-Speaker Voice Activity Detection (TS-VAD) approach as a solution for speaker diarization in complex environments like multi-talker social gatherings. Diarization, a process that involves identifying who spoke when, is critical for various applications, such as automatic speech recognition (ASR) systems. This research effectively addresses the limitations of traditional clustering-based methods, which struggle with overlapping speech often found in real-world acoustic settings.

Key Contributions

The primary contribution of this work is the development of the TS-VAD system, which is designed to enhance the diarization process by directly predicting individual speaker activities within each time frame. This approach diverges from traditional methods, which commonly rely on post-clustering after speaker features extraction like i-vectors. By utilizing conventional speech features along with i-vectors iteratively refined using TS-VAD outputs, the model effectively segregates overlapping speech, demonstrating prowess in scenarios with concurrent speakers.

Technical Implementation

The model's architecture incorporates binary classification output layers producing activity predictions for each speaker. This architecture is initially robust against a backdrop of conventional clustering-based diarization to estimate initial i-vectors. Furthermore, TS-VAD integrates a multi-microphone extension, leveraging a straightforward attention mechanism over hidden representations from single-channel versions. This adaptation enables the model to harness spatial information effectively, enhancing its capabilities in multi-microphone setups.

A crucial aspect of the TS-VAD model is the use of iterative refinement of speaker i-vectors, which significantly boosts the model's performance. The derived post-processing techniques, including probability smoothing and segmentation fusion, enhance the quality of the diarization results.

Experimental Results

The evaluation on the CHiME-6 unsegmented dataset reveals that TS-VAD significantly outperforms the baseline x-vector system, reducing the Diarization Error Rate (DER) by over 30% absolute. This reflects enhanced robustness in handling the acoustic complexity and overlapping speech in the evaluation data, underscoring the system's state-of-the-art performance level.

Implications and Future Directions

This research offers substantial implications for practical applications in diarization, supporting more accurate ASR systems in real-world scenarios characterized by overlapping speech. The theoretical contributions also extend to neural diarization approaches, potentially inspiring further development in this domain.

Future advancements could involve extending the TS-VAD framework to settings with dynamic and unknown speaker numbers, a step towards universal application in diarization tasks. Furthermore, enhancing the system's adaptability using alternative speaker embedding methods may offer a pathway for improving the scalability and versatility of TS-VAD across various acoustic settings.

In conclusion, the TS-VAD approach represents a significant step forward in speaker diarization technology, particularly in handling the ubiquitous challenge of overlapping speech. This work lays the foundation for future improvements and applications in both ASR and broader audio processing tasks.