The Microsoft Toolkit of Multi-Task Deep Neural Networks for Natural Language Understanding (2002.07972v2)

Abstract: We present MT-DNN, an open-source natural language understanding (NLU) toolkit that makes it easy for researchers and developers to train customized deep learning models. Built upon PyTorch and Transformers, MT-DNN is designed to facilitate rapid customization for a broad spectrum of NLU tasks, using a variety of objectives (classification, regression, structured prediction) and text encoders (e.g., RNNs, BERT, RoBERTa, UniLM). A unique feature of MT-DNN is its built-in support for robust and transferable learning using the adversarial multi-task learning paradigm. To enable efficient production deployment, MT-DNN supports multi-task knowledge distillation, which can substantially compress a deep neural model without significant performance drop. We demonstrate the effectiveness of MT-DNN on a wide range of NLU applications across general and biomedical domains. The software and pre-trained models will be publicly available at https://github.com/namisan/mt-dnn.

The Microsoft Toolkit of Multi-Task Deep Neural Networks for Natural Language Understanding

The paper presents the Microsoft Toolkit for Multi-Task Deep Neural Networks (MT-DNN), an open-source framework designed to simplify the training of customized models for Natural Language Understanding (NLU). The toolkit is built upon PyTorch and Transformers, enabling a wide range of NLU tasks to be addressed with varying objectives, such as classification and regression, using different text encoders including but not limited to RNNs, BERT, RoBERTa, and UniLM.

Key Features and Design

MT-DNN introduces several notable features that enhance its utility in developing robust NLU models:

1. Adversarial Multi-Task Learning: MT-DNN supports an adversarial multi-task learning paradigm that boosts model robustness and transferability across tasks. This feature allows models to gain resilience and generalizability, which is critical in practical deployments where variations in data can challenge model performance.
2. Knowledge Distillation: The toolkit offers multi-task knowledge distillation capabilities, allowing substantial compression of deep neural networks without a significant performance trade-off. This is essential for deploying models in environments with strict memory and speed constraints.
3. Modularity and Flexibility: MT-DNN’s modular architecture allows for easy customization. It supports a large inventory of pre-trained models and tasks while providing a straightforward interface for developers to introduce novel tasks or objectives.
4. Production Deployment Efficiency: The combination of multi-task learning, adversarial training, and knowledge distillation makes MT-DNN suitable for efficient production deployment. It facilitates the creation of robust models that are both performant and lightweight.

Workflow and Implementation

The workflow described in the paper includes neural LLM pre-training followed by several options for fine-tuning and distillation:

• Fine-tuning Configurations: MT-DNN provides flexibility by supporting single-task, multi-task, and multi-stage configurations. Additionally, adversarial training can be incorporated into any stage to further enhance model capability.
• Distillation Strategy: A sophisticated multi-task knowledge distillation process facilitates the compression of models to make them suitable for online deployment, offering significant reductions in computational overhead.
• Pre-training and Auxiliary Tasks: Users can perform pre-training using objectives like masked LLMing and integrate these as auxiliary tasks in the fine-tuning phase to improve downstream task performance.

Applications and Experimental Results

The toolkit demonstrates efficacy across varied domains, including general benchmarks like GLUE, SNLI, and SQuAD, and specific applications in the biomedical field, such as named entity recognition and question answering. The experiments suggest that MT-DNN excels in leveraging multi-task learning and adversarial training, achieving notable improvements over baseline models. For instance, the combination of adversarial and multi-task training provides substantial performance gains on the GLUE benchmark, highlighting the system's robustness.

The effectiveness of adversarial training is further exemplified on challenging datasets like ANLI, where MT-DNN outperforms existing strong baselines, indicating its potential in handling adversarially-selected samples.

Implications and Future Directions

MT-DNN offers a comprehensive solution for researchers and practitioners aiming to build efficient and robust NLU models. Its open-source nature and extensive documentation make it accessible and adaptable for diverse linguistic tasks. As advancements in natural language processing continue, the toolkit's design allows for seamless integration of new architectures and training paradigms.

Future developments may extend the toolkit's capabilities to include natural language generation tasks and additional pre-trained models like T5, thereby broadening the spectrum of applications and enhancing its versatility in handling language tasks on a global scale.