2000 character limit reached
Memory-efficient NLLB-200: Language-specific Expert Pruning of a Massively Multilingual Machine Translation Model
Published 19 Dec 2022 in cs.CL, cs.AI, and cs.LG | (2212.09811v3)
Abstract: The recently released NLLB-200 is a set of multilingual Neural Machine Translation models that cover 202 languages. The largest model is based on a Mixture of Experts architecture and achieves SoTA results across many language pairs. It contains 54.5B parameters and requires at least four 32GB GPUs just for inference. In this work, we propose a pruning method that enables the removal of up to 80% of experts without further finetuning and with a negligible loss in translation quality, which makes it feasible to run the model on a single 32GB GPU. Further analysis suggests that our pruning metrics can identify language-specific experts.
- Massively multilingual neural machine translation. arXiv preprint arXiv:1903.00089.
- ediffi: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint arXiv:2211.01324.
- Task-specific expert pruning for sparse mixture-of-experts. arXiv preprint arXiv:2206.00277.
- Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116.
- No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672.
- Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860.
- GLaM: Efficient scaling of language models with mixture-of-experts. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 5547–5569. PMLR.
- Beyond english-centric multilingual machine translation. J. Mach. Learn. Res., 22(107):1–48.
- A review of sparse expert models in deep learning. arXiv preprint arXiv:2209.01667.
- Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.
- Ernie-vilg 2.0: Improving text-to-image diffusion model with knowledge-enhanced mixture-of-denoising-experts. arXiv preprint arXiv:2210.15257.
- William Held and Diyi Yang. 2022. Shapley head pruning: Identifying and removing interference in multilingual transformers. arXiv preprint arXiv:2210.05709.
- Adaptive mixtures of local experts. Neural computation, 3(1):79–87.
- Jae-young Jo and Sung-Hyon Myaeng. 2020. Roles and utilization of attention heads in transformer-based neural language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3404–3417.
- Michael I Jordan and Robert A Jacobs. 1994. Hierarchical mixtures of experts and the em algorithm. Neural computation, 6(2):181–214.
- Scalable and efficient moe training for multitask multilingual models. arXiv preprint arXiv:2109.10465.
- Do multilingual neural machine translation models contain language pair specific attention heads? arXiv preprint arXiv:2105.14940.
- Beyond distillation: Task-level mixture-of-experts for efficient inference. arXiv preprint arXiv:2110.03742.
- Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668.
- Are sixteen heads really better than one? In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
- Small-100: Introducing shallow multilingual machine translation model for low-resource languages. ArXiv, abs/2210.11621.
- What do compressed multilingual machine translation models forget? ArXiv, abs/2205.10828.
- Multimodal contrastive learning with limoe: the language-image mixture of experts. arXiv preprint arXiv:2206.02770.
- Carbon emissions and large neural network training. arXiv preprint arXiv:2104.10350.
- Maja Popović. 2015. chrf: character n-gram f-score for automatic mt evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395.
- Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186–191, Brussels, Belgium. Association for Computational Linguistics.
- Scalable transfer learning with expert models. arXiv preprint arXiv:2009.13239.
- Scaling vision with sparse mixture of experts. Advances in Neural Information Processing Systems, 34:8583–8595.
- Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538.
- Energy and policy considerations for deep learning in nlp. arXiv preprint arXiv:1906.02243.
- Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818–2826.
- Multilingual translation with extensible multilingual pretraining and finetuning. arXiv preprint arXiv:2008.00401.
- Attention is all you need. Advances in neural information processing systems, 30.
- Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. arXiv preprint arXiv:1905.09418.
- Deepnet: Scaling transformers to 1,000 layers. arXiv preprint arXiv:2203.00555.
- Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32.
- Speechmoe: Scaling to large acoustic models with dynamic routing mixture of experts. arXiv preprint arXiv:2105.03036.
- Twenty years of mixture of experts. IEEE transactions on neural networks and learning systems, 23(8):1177–1193.
- Improving massively multilingual neural machine translation and zero-shot translation. arXiv preprint arXiv:2004.11867.
- St-moe: Designing stable and transferable sparse expert models, 2022. URL https://arxiv. org/abs/2202.08906.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.