Language Modeling with Deep Transformers (1905.04226v2)

Abstract: We explore deep autoregressive Transformer models in LLMing for speech recognition. We focus on two aspects. First, we revisit Transformer model configurations specifically for LLMing. We show that well configured Transformer models outperform our baseline models based on the shallow stack of LSTM recurrent neural network layers. We carry out experiments on the open-source LibriSpeech 960hr task, for both 200K vocabulary word-level and 10K byte-pair encoding subword-level LLMing. We apply our word-level models to conventional hybrid speech recognition by lattice rescoring, and the subword-level models to attention based encoder-decoder models by shallow fusion. Second, we show that deep Transformer LLMs do not require positional encoding. The positional encoding is an essential augmentation for the self-attention mechanism which is invariant to sequence ordering. However, in autoregressive setup, as is the case for LLMing, the amount of information increases along the position dimension, which is a positional signal by its own. The analysis of attention weights shows that deep autoregressive self-attention models can automatically make use of such positional information. We find that removing the positional encoding even slightly improves the performance of these models.

Overview of LLMing with Deep Transformers for Speech Recognition

The paper "LLMing with Deep Transformers" by Irie et al. explores the application of deep autoregressive Transformer models within the domain of LLMing, specifically targeting improvements in speech recognition systems. This research makes a notable contribution by comparing the efficacy of Transformer models against conventional LSTM recurrent neural networks (RNNs) in speech recognition tasks, highlighting the superior performance of Transformers under appropriate configuration.

Performance Comparison: Transformer vs. LSTM

A key finding presented in the paper is the performance enhancement offered by Transformer models over LSTMs when configured for LLMing. On the LibriSpeech dataset, the well-configured Transformers demonstrate improvements in both perplexity and word error rate (WER). Specifically, Transformer models with configurations such as $(L=42, d_{ff}=2048, d_{res}=512, H=8)$, where $L$ denotes the number of layers, $d_{ff}$ and $d_{res}$ denote dimensions, and $H$ denotes the number of heads, outperform deep LSTM models in lattice rescoring tasks and end-to-end ASR tasks using shallow fusion. Such findings suggest increased depth and careful tuning of hyperparameters in Transformers contribute significantly to modeling improvements.

Positional Encoding in Transformers

One of the bold claims made in the paper is related to the positional encoding in Transformer models. Conventionally, positional encoding facilitates sequence order information for self-attention mechanisms. However, the authors argue that for autoregressive models, this explicit positional encoding is redundant. They demonstrate that the information carried by the autoregressive setup naturally provides implicit positional signals, suggesting that deep self-attention models can leverage this inherent information without additional explicit positional encoding mechanisms. Through their experiments, they find that removing positional encodings might even slightly enhance model performance, which challenges traditional assumptions about self-attention mechanisms in Transformers.

Practical Implications and Future Directions

The implications of this research are significant for the practical deployment of LLMs in speech recognition tasks. The demonstrated performance gains via Transformer models pave the way for more efficient and accurate speech recognition systems in real-world applications. By eliminating the need for explicit positional encoding, the Transformer models can potentially be simplified and optimized further.

Looking forward, the research raises interesting questions regarding the architecture of deep learning models in NLP and speech recognition. A fruitful area of exploration is further tuning and scaling of Transformer models, along with investigating layers' interactions and attention mechanism optimizations. Additionally, the authors expressed interest in extending Transformer characteristics to enhance LSTM models, possibly leading to hybrid models with benefits from both architectures.

In conclusion, this paper provides valuable insights into the integration of Transformers in LLMing for ASR, challenging existing norms in model design, and opening avenues for further innovation in deep learning architectures.