Sub-word Level Lip Reading With Visual Attention

Abstract: The goal of this paper is to learn strong lip reading models that can recognise speech in silent videos. Most prior works deal with the open-set visual speech recognition problem by adapting existing automatic speech recognition techniques on top of trivially pooled visual features. Instead, in this paper we focus on the unique challenges encountered in lip reading and propose tailored solutions. To this end, we make the following contributions: (1) we propose an attention-based pooling mechanism to aggregate visual speech representations; (2) we use sub-word units for lip reading for the first time and show that this allows us to better model the ambiguities of the task; (3) we propose a model for Visual Speech Detection (VSD), trained on top of the lip reading network. Following the above, we obtain state-of-the-art results on the challenging LRS2 and LRS3 benchmarks when training on public datasets, and even surpass models trained on large-scale industrial datasets by using an order of magnitude less data. Our best model achieves 22.6% word error rate on the LRS2 dataset, a performance unprecedented for lip reading models, significantly reducing the performance gap between lip reading and automatic speech recognition. Moreover, on the AVA-ActiveSpeaker benchmark, our VSD model surpasses all visual-only baselines and even outperforms several recent audio-visual methods.

• The paper introduces a novel visual transformer pooling module and sub-word tokenization to efficiently decode silent video speech.
• The proposed method achieves a 22.6% word error rate on LRS2, outperforming models trained on larger datasets.
• The study extends applications to visual speech detection, offering significant improvements over existing visual-only baselines.

A Detailed Summary of "Sub-word Level Lip Reading With Visual Attention"

Introduction and Motivation

The paper "Sub-word Level Lip Reading With Visual Attention" (2110.07603) addresses the challenging task of lip reading—decoding speech from silent video inputs. Unlike audio-based automatic speech recognition (ASR), lip reading requires processing high-dimensional video inputs, which involve substantial temporal and spatial complexity. The authors highlight practical applications, such as enhancing speech recognition in noisy environments and aiding communication for speech-impaired individuals. The inherent difficulty of lip reading due to homophemes and the necessity of video-specific tailoring—rather than merely adapting ASR techniques—motivate the need for a more specialized approach.

Methodology

Visual Backbone and Attention-based Pooling

The authors introduce an innovative visual backbone designed to address the intricacies of lip reading by focusing on visual encoding and tokenization strategies. Their approach includes a novel attention-based pooling mechanism to efficiently aggregate visual speech representations. This module, termed Visual Transformer Pooling (VTP), dynamically learns to focus on relevant spatial areas in the video inputs, enhancing feature aggregation over traditional methods like Global Average Pooling (GAP).

Figure 1: Proposed lip reading architecture. {\em Left: The input video frames are processed through a spatio-temporal CNN and a Visual Transformer Pooling module for feature extraction.}

Sub-word Tokenization

For the first time in lip reading, the authors employ sub-word (word-piece) tokens instead of traditional character-level outputs. This choice leverages the semantic richness and efficiency of sub-words, reducing sequence length and encoding inherent language priors, thus decreasing the model's dependency on extensive language modeling.

Experimental Results

The authors conduct extensive experiments on public datasets such as LRS2 and LRS3, achieving state-of-the-art results with their proposed architecture. Notably, their best model attains a 22.6% word error rate (WER) on LRS2, surpassing models trained on significantly larger datasets. The introduction of sub-word tokenization and the VTP module demonstrates considerable improvements in performance, highlighting the model's superior data efficiency.

Applications and Implications

Visual Speech Detection

Beyond lip reading, the visual backbone is leveraged for Visual Speech Detection (VSD), a crucial step for inferring speech activity from video-only inputs. The VSD model, built on top of the lip reading encoder, significantly outperforms existing visual-only baselines, even challenging some audio-visual methods.

Figure 2: Visual Speech Detection pipeline.

Future Directions and Ethical Considerations

The research positions itself as a foundational advancement in visual speech technology, offering both theoretical and practical implications. Future work could explore further integration with contextual and multimodal information to enhance robustness. The authors transparently address potential ethical concerns, such as privacy implications of lip-reading technology, emphasizing the intention to release their models to encourage research transparency and democratization.

Conclusion

"Sub-word Level Lip Reading With Visual Attention" advances the frontier of lip reading by introducing tailored architectural innovations and leveraging sub-word tokenization. This research not only presents empirically validated improvements but also extends the applicability of visual speech technology, underscoring the continued convergence of visual and language processing in machine learning.