Personal VAD: Speaker-Conditioned Voice Activity Detection (1908.04284v4)

Abstract: In this paper, we propose "personal VAD", a system to detect the voice activity of a target speaker at the frame level. This system is useful for gating the inputs to a streaming on-device speech recognition system, such that it only triggers for the target user, which helps reduce the computational cost and battery consumption, especially in scenarios where a keyword detector is unpreferable. We achieve this by training a VAD-alike neural network that is conditioned on the target speaker embedding or the speaker verification score. For each frame, personal VAD outputs the probabilities for three classes: non-speech, target speaker speech, and non-target speaker speech. Under our optimal setup, we are able to train a model with only 130K parameters that outperforms a baseline system where individually trained standard VAD and speaker recognition networks are combined to perform the same task.

• The paper presents a personal VAD system that conditions on speaker-specific features to precisely classify each audio frame as target speech, non-target speech, or non-speech.
• It evaluates four architectures integrating speaker embeddings and verification scores, with the embedding conditioned training method achieving high accuracy at low computational cost.
• The system minimizes latency, battery usage, and computational overhead in on-device speech recognition, marking a significant advance in real-time audio processing.

Personal VAD: Speaker-Conditioned Voice Activity Detection

The paper "Personal VAD: Speaker-Conditioned Voice Activity Detection" introduces an efficient system designed to detect voice activity of a specific, target speaker at the frame level, which significantly enhances the operation of streaming on-device speech recognition systems. The proposed solution is particularly useful in limiting computational overhead and battery usage in scenarios where keyphrase detection methods such as wake word recognition are suboptimal or not preferred by users.

System Overview and Methodologies

The core concept of personal VAD involves conditioning a voice activity detection (VAD) framework on speaker-specific characteristics, effectively utilizing speaker embeddings or speaker verification scores. The focus is to classify each audio frame into one of three categories: non-speech, target speaker speech, and non-target speaker speech. The paper presents the superior efficiency of the personal VAD approach by demonstrating that a model with merely 130,000 parameters can surpass the performance of a combined standard VAD with a speaker recognition paradigm.

The personal VAD system is vital in situations demanding low latency and high precision, such as mobile devices where computational resources are constrained. It features architecture enabling streamlined operations through the frame-level analysis, rather than segment or window-based evaluations typically seen in speaker recognition setups. The paper outlines four architectures for personal VAD:

1. Score Combination (SC): Utilizes separate scores from pre-trained VAD and speaker verification systems, serving as a foundational baseline. Despite simplicity, it can suffer latency issues due to inconsistent evaluation models.
2. Score Conditioned Training (ST): Integrates speaker verification scores with acoustic features to train a new model. This mechanism, though computationally heavier due to reliance on ongoing speaker verification during inference, showcases enhanced performance.
3. Embedding Conditioned Training (ET): Directly employs speaker embeddings with acoustic features for training. This method benefits from the small size and minimal runtime cost while maintaining robust accuracy.
4. Score and Embedding Conditioned Training (SET): Combines both verification scores and embeddings, achieving optimal performance but at increased computational expense during runtime.

Experimental Results and Implications

The paper explores a weighted pairwise loss function to optimize model training, emphasizing the differentiation between target and non-target speaker activities in complex auditory environments. Through rigorous testing on modified LibriSpeech datasets with multistyle training enhancements (MTR), the ET architecture emerged as a favorable solution balancing efficiency and accuracy, particularly when deploying on-device systems with stringent resource constraints.

Empirical evaluation also established that personal VAD can substitute standard VAD in conventional speech/non-speech tasks with negligible performance declines, broadening its applicability as a universal VAD component across diverse auditory applications.

Theoretical and Practical Implications

This research advances the functional depth of voice activity detection systems by integrating speaker-specific dynamics into the detection process. The findings suggest practical ramifications in device battery conservation and computational economization. Theoretically, the personal VAD mechanism illustrates promising developments in real-time sequential audio processing, offering valuable insights into the design of future AI-driven acoustic systems.

Although the current methodology leverages pre-trained models for speaker embeddings, further progress in unsupervised or adaptive learning paradigms could enhance personalization and reduce enroLLMent overhead, making speaker-conditioned VAD even more seamless in user-device interactions.

Conclusion

The personal VAD system exemplifies a cutting-edge approach in the field of voice activity detection, offering substantial benefits for streaming speech recognition frameworks. Bridging speaker recognition and VAD capabilities, it sets a precedent for efficient, real-time user-specific audio processing, fostering significant advancements in on-device AI applications. Future research can expand upon this foundation, investigating adaptive learning models and optimizing noisy environment handling to further refine personal VAD systems.