2000 character limit reached
Eliciting the Translation Ability of Large Language Models via Multilingual Finetuning with Translation Instructions (2305.15083v4)
Published 24 May 2023 in cs.CL
Abstract: Large-scale Pretrained LLMs, such as ChatGPT and GPT4, have shown strong abilities in multilingual translations, without being explicitly trained on parallel corpora. It is interesting how the LLMs obtain their ability to carry out translation instructions for different languages. In this paper, we present a detailed analysis by finetuning a multilingual pretrained LLM, XGLM-7B, to perform multilingual translation following given instructions. Firstly, we show that multilingual LLMs have stronger translation abilities than previously demonstrated. For a certain language, the performance depends on its similarity to English and the amount of data used in the pretraining phase. Secondly, we find that LLMs' ability to carry out translation instructions relies on the understanding of translation instructions and the alignment among different languages. With multilingual finetuning, LLMs could learn to perform the translation task well even for those language pairs unseen during the instruction tuning phase.
- In-context examples selection for machine translation. arXiv, 2212.02437.
- Maruan Al-Shedivat and Ankur Parikh. 2019. Consistency by agreement in zero-shot neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1184–1197, Minneapolis, Minnesota. Association for Computational Linguistics.
- The missing ingredient in zero-shot neural machine translation. CoRR, 1903.07091.
- Language models are few-shot learners. abs/2005.14165.
- Broken neural scaling laws. CoRR, 2210.14891.
- Scaling instruction-finetuned language models. CoRR, 2210.11416.
- CCAligned: A massive collection of cross-lingual web-document pairs. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP 2020), pages 5960–5969, Online.
- The unreasonable effectiveness of few-shot learning for machine translation. CoRR, 2302.01398.
- The Flores-101 evaluation benchmark for low-resource and multilingual machine translation. Transactions of the Association for Computational Linguistics, 10:522–538.
- Improved zero-shot neural machine translation via ignoring spurious correlations. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1258–1268, Florence, Italy. Association for Computational Linguistics.
- How good are gpt models at machine translation? a comprehensive evaluation. arXiv, 2302.09210.
- Parrot: Translating during chat using large language models. CoRR, 2304.02426.
- Is chatgpt a good translator? yes with gpt-4 as the engine. 2301.08745.
- Scaling laws for neural language models. 2001.08361.
- Diederik P. Kingma and Jimmy Ba. 2017. Adam: A method for stochastic optimization. CoRR, 1412.6980.
- Few-shot learning with multilingual generative language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9019–9052, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
- URIEL and lang2vec: Representing languages as typological, geographical, and phylogenetic vectors. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 8–14, Valencia, Spain. Association for Computational Linguistics.
- Improving pretrained models for zero-shot multi-label text classification through reinforced label hierarchy reasoning. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1051–1062, Online. Association for Computational Linguistics.
- Crosslingual generalization through multitask finetuning.
- OpenAI. 2023. Chatgpt (mar 23 version) [large language model].
- Multilingual BERT post-pretraining alignment. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 210–219, Online. Association for Computational Linguistics.
- Wikimatrix: Mining 135m parallel sentences in 1620 language pairs from wikipedia. CoRR, abs/1907.05791.
- Stanford alpaca: An instruction-following llama model.
- Llama: Open and efficient foundation language models. CoRR, 2302.13971.
- Prompting palm for translation: Assessing strategies and performance. 2211.09102.
- Prompting palm for translation: Assessing strategies and performance. CoRR, abs/2211.09102.
- Finetuned language models are zero-shot learners. CoRR, 2109.01652.
- Multilingual agreement for multilingual neural machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 233–239, Online. Association for Computational Linguistics.
- Prompting large language model for machine translation: A case study. CoRR, 2301.07069.
- Improving massively multilingual neural machine translation and zero-shot translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1628–1639, Online. Association for Computational Linguistics.
- Multilingual machine translation with large language models: Empirical results and analysis. CoRR, 2304.04675.
- Jiahuan Li (10 papers)
- Hao Zhou (351 papers)
- Shujian Huang (106 papers)
- Shanbo Cheng (23 papers)
- Jiajun Chen (125 papers)