End-to-End Attention-based Large Vocabulary Speech Recognition (1508.04395v2)

Abstract: Many of the current state-of-the-art Large Vocabulary Continuous Speech Recognition Systems (LVCSR) are hybrids of neural networks and Hidden Markov Models (HMMs). Most of these systems contain separate components that deal with the acoustic modelling, LLMling and sequence decoding. We investigate a more direct approach in which the HMM is replaced with a Recurrent Neural Network (RNN) that performs sequence prediction directly at the character level. Alignment between the input features and the desired character sequence is learned automatically by an attention mechanism built into the RNN. For each predicted character, the attention mechanism scans the input sequence and chooses relevant frames. We propose two methods to speed up this operation: limiting the scan to a subset of most promising frames and pooling over time the information contained in neighboring frames, thereby reducing source sequence length. Integrating an n-gram LLM into the decoding process yields recognition accuracies similar to other HMM-free RNN-based approaches.

End-to-End Attention-Based Large Vocabulary Speech Recognition

The paper "End-to-End Attention-Based Large Vocabulary Speech Recognition" by Dzmitry Bahdanau et al. investigates the development of a speech recognition system by replacing traditional Hidden Markov Models (HMMs) with a Recurrent Neural Network (RNN) that utilizes an attention mechanism. This work addresses several bottlenecks in the current hybrid approaches that combine Deep Neural Networks (DNNs) and HMMs, particularly the complexity and inefficiency of alignment and decoding stages.

Problem Statement and Contributions

Traditional LVCSR systems often require multiple components, including acoustic models, LLMs, and sequence decoders, typically combined in a hybrid manner. The authors propose an end-to-end model using an Attention-based Recurrent Sequence Generator (ARSG), where alignment between input features and the character sequences is learned automatically through the attention mechanism. This significantly simplifies the training pipeline by removing the need for forced alignment stages typically done using GMM-HMM models.

The key contributions of the paper are:

1. Efficient Training: Implementing a windowing mechanism to constrain the attention scan to the most relevant frames, which decreases the training complexity from quadratic to linear.
2. Time Pooling in RNNs: Introducing a recurrent architecture that performs pooling over time, thus reducing the length of the encoded input sequence and facilitating training on long sequences.
3. Integration with LLMs: Combining a character-level ARSG with an n-gram LLM using the Weighted Finite State Transducers (WFST) framework for improved recognition accuracy.

Methodology

The proposed system encodes sequences of speech frames into character sequences using RNNs for both the encoder and decoder. The encoder converts the input speech signal into a sequence of feature representations, while the decoder utilizes an attention mechanism to align these encoded features with the corresponding character outputs.

Recurrent Neural Networks (RNNs)

RNNs are employed to handle variable-length sequences, which are advantageous for speech recognition tasks. The authors implement Gated Recurrent Units (GRUs) due to their efficiency in managing long-term dependencies. The architecture features stacked bidirectional RNNs (BiRNNs) to achieve better contextual understanding by incorporating information from both past and future frames.

Attention Mechanism

The attention mechanism scans the encoded input representation to focus on pertinent frames for predicting each character in the output sequence. The paper details an improved attention schema incorporating convolutional features for context-based focusing. Limiting the attention range using the windowing approach during both training and decoding stages addresses scalability concerns.

Integration with LLMs

To enhance predictive performance, the authors integrate an n-gram LLM at the character level using a WFST framework. This integration is crucial, as LLMing capabilities are limited by the size of the training corpus. The WFST framework effectively bridges word-level LLMs with the character-level outputs of the ARSG, thus enhancing the overall system performance.

Results and Implications

Extensive experiments on the Wall Street Journal (WSJ) corpus demonstrate that the proposed method achieves competitive recognition accuracy. The end-to-end model shows superior performance compared to Connectionist Temporal Classification (CTC)-based systems, particularly when no external LLM is applied.

However, the ARSG-based model exhibits less improvement when combined with an external LLM compared to CTC systems. The authors hypothesize that this may be due to overfitting of the implicit LLM trained on the WSJ transcripts, highlighting the necessity for larger datasets to achieve optimal performance.

Future Directions

The paper suggests several promising avenues for future research:

1. Joint Training with Larger Text Corpora: Training ARSGs with pre-trained RNN LLMs on extensive text datasets could mitigate overfitting and obviate the need for auxiliary n-gram models.
2. Trainable Integration with External LLMs: Exploring methods for incorporating n-gram LLMs into ARSGs more effectively, possibly through continued joint training.
3. Scalability and Real-Time Processing: Investigating methods to further improve the computational efficiency for real-time large-scale applications.

Conclusion

This paper presents a significant advancement in speech recognition technology by demonstrating the feasibility and advantages of end-to-end attention-based models over traditional hybrid systems. The proposed methods simplify the training process, reduce dependencies on obsolete models, and offer a promising direction for developing more efficient and accurate speech recognition systems. The integration with LLMs remains an area of active research, with strong implications for future advancements in LVCSR.