Early Attentive Sparsification Accelerates Neural Speech Transcription (2506.15912v1)

Abstract: Transformer-based neural speech processing has achieved state-of-the-art performance. Since speech audio signals are known to be highly compressible, here we seek to accelerate neural speech transcription by time-domain signal sparsification early in the neural encoding stage, taking advantage of the interpretability of the self-attention mechanism in transformer audio encoders. With the Whisper family of models, we perform a systematic architecture search over the joint space of sparsification stage (a certain encoder layer) and compression ratio (sparsity). We found that the best resulting solutions under 1% accuracy degradation choose to sparsify the hidden state to 40-60% sparsity at an early encoding stage, and thereby achieve up to 1.6x runtime acceleration in English speech transcription tasks on Nvidia GPUs without any fine-tuning.

Early Attentive Sparsification Accelerates Neural Speech Transcription

This paper addresses a significant challenge in the field of automatic speech recognition (ASR): optimizing the inference efficiency of transformer-based ASR models without compromising accuracy. By leveraging the newly introduced methodology of Early Attentive Sparsification (EAS), this paper explores the potential for accelerating neural speech transcription through sparsification of audio signals at an early encoder stage using the self-attention scores of transformers. The focal point of the investigation is the Whisper family of models, which are distinguished by their performance in delivering cutting-edge speech transcription results.

Core Contributions

The primary contribution of the paper lies in the innovative method of EAS for neural ASR models, particularly aimed at the Whisper model family. Key strategies employed involve time-domain signal sparsification based on self-attention mechanism insights. The architecture search focused on two main parameters: the stage of the encoder layer where sparsification is applied and the degree of compression, resulting in solutions with less than 1\% degradation in accuracy but up to 1.6x speedups.

Key Findings

1. Attention-driven Token Sparsification: EAS exploits audio token redundancy using accumulated attention scores across multiple heads, creating a method for token selection and sparsification to accelerate runtime without fine-tuning the model.
2. Runtime Efficiency: By implementing sparsification at an early encoding stage with 40-60% sparsity, Whisper models achieved significant runtime acceleration (up to 1.6x) on Nvidia GPUs while maintaining accuracy. This speedup is particularly pronounced in models already benefitting from post-training compression techniques.
3. Generalizability: The proposed method demonstrates robust generalization across different Whisper variants, which vary in model size and architecture complexities.

Implications and Future Directions

The development of the EAS technique opens several paths for future research and practical applications:

• Compatibility: EAS's compatibility with a range of optimizations, such as lower precision computation and advanced attention algorithms like FlashAttention-2, suggests it can effectively integrate with and enhance existing ASR systems in production environments.
• Token Redundancy: The paper underscores the presence of redundancy in audio sequence tokens within neural ASR models, an insight that could guide further exploration into efficient model architectures that balance complexity and operational costs.
• Broader Impact: Although focused on the Whisper family, the methodological framework of EAS could be adapted and tested in other sequence-to-sequence models beyond speech recognition, potentially driving innovations in fields like natural language processing and image captioning.

Conclusion

This paper presents a nuanced approach to enhancing the efficiency of neural ASR systems through strategic sparsification without incurring significant accuracy losses. EAS is not only a technical advancement in model acceleration but also invites a re-examination of underlying data redundancies in speech representation. In an age where computational optimization is pivotal, this research provides a solid foundation for further advancements in AI-driven speech systems.