The NiuTrans System for WNGT 2020 Efficiency Task (2109.08008v1)

Abstract: This paper describes the submissions of the NiuTrans Team to the WNGT 2020 Efficiency Shared Task. We focus on the efficient implementation of deep Transformer models \cite{wang-etal-2019-learning, li-etal-2019-niutrans} using NiuTensor (https://github.com/NiuTrans/NiuTensor), a flexible toolkit for NLP tasks. We explored the combination of deep encoder and shallow decoder in Transformer models via model compression and knowledge distillation. The neural machine translation decoding also benefits from FP16 inference, attention caching, dynamic batching, and batch pruning. Our systems achieve promising results in both translation quality and efficiency, e.g., our fastest system can translate more than 40,000 tokens per second with an RTX 2080 Ti while maintaining 42.9 BLEU on \textit{newstest2018}. The code, models, and docker images are available at NiuTrans.NMT (https://github.com/NiuTrans/NiuTrans.NMT).

• The paper demonstrates a deep encoder-shallow decoder approach achieving high translation quality with a 45.5 BLEU score using teacher models.
• The paper describes efficient student models compressed via knowledge distillation, maintaining 42.9 BLEU while translating over 40,000 tokens per second.
• The paper employs CPU and GPU optimizations like FP16 inference and multi-threading to significantly reduce decoding times in practical NMT deployments.

Overview of "The NiuTrans System for WNGT 2020 Efficiency Task"

This paper presents the NiuTrans team's approach to the WNGT 2020 Efficiency Shared Task, focusing on the efficient deployment of deep Transformer models for neural machine translation (NMT) using the NiuTensor toolkit. With an emphasis on both operational efficiency and translation quality, the authors delve into various methodologies including model compression, knowledge distillation, and execution optimizations for both CPU and GPU environments.

Methodological Framework

The paper's core revolves around leveraging a deep encoder-shallow decoder architecture in Transformer models. The authors experimented with varying numbers of encoder and decoder layers to attain a balance between computational speed and translation performance. Their deep encoder models achieved high translation quality by incorporating techniques like knowledge distillation, where a smaller "student" model learns from a larger "teacher" ensemble model.

Teacher Models

The paper details the construction of several deep Transformer teacher models, incorporating dynamic linear combination of layers and relative position representations. These configurations resulted in a BLEU score of up to 45.5 on newstest2018, showcasing their efficacy in achieving high-quality translations.

Student Models

A significant contribution of this work is the compression of teacher models into compact, efficient student models through knowledge distillation. Notably, a 9-1 encoder-decoder model demonstrated the potential to maintain translation quality (42.9 BLEU) while significantly boosting speed, executing more than 40,000 tokens per second on an RTX 2080 Ti.

Optimization Techniques

The authors implemented a variety of optimizations targeting both CPU and GPU platforms:

• FP16 Inference: Reduced precision operations enabled faster computations without substantial loss in translation quality.
• Attention Caching and Dynamic Batching: These methods were employed to accelerate beam search and improve GPU throughput.
• Multi-threading and MKL Utilization: For CPU tasks, the system harnessed multi-core processing and Intel's Math Kernel Library to enhance computation efficiency.

Experimental Results

Through various implementations and adaptations, NiuTrans was able to translate a set of 1 million English sentences within 2 hours on both CPU and GPU platforms. The GPU models particularly benefitted from the reduced decoding times, attributable to efficient implementations and architectural adjustments.

Implications and Future Directions

The implications of this research extend into both practical deployments and theoretical advancements in NMT. Practically, the work underscores the viability of deploying high-efficiency Transformer models in resource-constrained environments. Theoretically, it highlights the potential of deep encoder-shallow decoder frameworks in achieving efficient translation without sacrificing quality.

Future research may explore further optimizations such as kernel fusion and vocabulary subset restrictions. Additionally, advancements in type conversion efficiency for FP16 operations could lead to even greater speed gains.

In sum, the NiuTrans system embodies a robust effort to harmonize translation quality with computational efficiency, reflecting evolving needs in real-world NMT applications. The methodologies and results presented hold promise for further AI developments in the domain of machine translation.