Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation (1609.08144v2)

Abstract: Neural Machine Translation (NMT) is an end-to-end learning approach for automated translation, with the potential to overcome many of the weaknesses of conventional phrase-based translation systems. Unfortunately, NMT systems are known to be computationally expensive both in training and in translation inference. Also, most NMT systems have difficulty with rare words. These issues have hindered NMT's use in practical deployments and services, where both accuracy and speed are essential. In this work, we present GNMT, Google's Neural Machine Translation system, which attempts to address many of these issues. Our model consists of a deep LSTM network with 8 encoder and 8 decoder layers using attention and residual connections. To improve parallelism and therefore decrease training time, our attention mechanism connects the bottom layer of the decoder to the top layer of the encoder. To accelerate the final translation speed, we employ low-precision arithmetic during inference computations. To improve handling of rare words, we divide words into a limited set of common sub-word units ("wordpieces") for both input and output. This method provides a good balance between the flexibility of "character"-delimited models and the efficiency of "word"-delimited models, naturally handles translation of rare words, and ultimately improves the overall accuracy of the system. Our beam search technique employs a length-normalization procedure and uses a coverage penalty, which encourages generation of an output sentence that is most likely to cover all the words in the source sentence. On the WMT'14 English-to-French and English-to-German benchmarks, GNMT achieves competitive results to state-of-the-art. Using a human side-by-side evaluation on a set of isolated simple sentences, it reduces translation errors by an average of 60% compared to Google's phrase-based production system.

Overview of Google's Neural Machine Translation System

The paper "Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation" presents Google's Neural Machine Translation (GNMT) system. GNMT is an end-to-end learning system, designed to enhance machine translation by addressing several known issues in traditional Neural Machine Translation (NMT) systems. This review provides a thorough overview of the system architecture, training procedures, modifications for efficiency, and results.

Introduction and Background

The GNMT system improves upon conventional NMT techniques by using an eight-layer LSTM network with residual connections in both the encoder and decoder, incorporating attention mechanisms for better parallelism, and employing a wordpiece model to handle rare words more effectively than traditional approaches. The attention mechanism links the bottom layer of the decoder to the top layer of the encoder, enhancing training speed and overall performance.

Key Innovations

1. Deep LSTM Network: GNMT employs an 8-layer Long Short-Term Memory (LSTM) network for both the encoder and decoder. Residual connections facilitate gradient flow through deep networks, helping overcome the vanishing gradient problem.
2. Bi-directional Encoder: The first layer of the encoder is bi-directional, which captures context from both directions, essential for correctly translating certain words.
3. Wordpiece Model: GNMT introduces the wordpiece model that segments words into sub-word units. This model optimizes the trade-off between handling rare words and maintaining efficiency, facilitating better vocabulary management and improved translation accuracy.
4. Parallelism: To enhance computation speed, GNMT employs model and data parallelism strategies. The layers are partitioned across multiple GPUs, significantly speeding up training and inference.
5. Low-precision Arithmetic: To accelerate inference, GNMT uses low-precision arithmetic. This method, combined with custom hardware (Google's TPU), provides substantial speed improvements without sacrificing model accuracy.

Training Techniques

GNMT training proceeds in two primary stages:

1. Maximum Likelihood Training: Initial model training uses the maximum likelihood objective, combining the Adam optimizer for initial steps and transitioning to simple SGD to refine learning.
2. Reinforcement Learning (RL) Fine-tuning: The model undergoes additional fine-tuning using RL techniques to optimize directly for BLEU score, which improves translation relevance and performance.

Results and Performance

GNMT was evaluated on public datasets, namely WMT'14 English-to-French and English-to-German benchmarks, where it achieved state-of-the-art results. Detailed evaluations include:

1. BLEU Scores: GNMT achieved 38.95 BLEU on WMT'14 English-to-French and 24.61 BLEU on WMT'14 English-to-German datasets using a single model. Model ensembles further pushed the BLEU scores to 41.16 and 26.30, respectively.
2. Efficiency: Compared to CPU and GPU implementations, the GNMT system running on TPUs showcased a significant speedup, making real-time application feasible.
3. Human Evaluations: Side-by-side human evaluations placed GNMT’s output quality close to average human translations, marking a substantial improvement over previous phrase-based systems.

Practical and Theoretical Implications

GNMT's advancements address several machine translation challenges, making it robust for large-scale, real-world applications. The integration of wordpiece models and low-precision inference could set new benchmarks for computational efficiency in NLP tasks. Future developments may leverage these innovations to push the boundaries of automated translation systems further, potentially achieving near-human performance on diverse and complex language pairs.

Conclusion

Google's NMT system represents a significant step forward in the domain of machine translation. By integrating deep learning techniques, attention mechanisms, and efficient model management, GNMT narrows the gap between machine and human translation quality. The practical implementations, combined with theoretical advancements, ensure that GNMT not only sets a high standard in translation benchmarks but also excels in real-world applications, making it a cornerstone in the future of machine translation technology.