Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
60 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
8 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Enhancing Multilingual Capabilities of Large Language Models through Self-Distillation from Resource-Rich Languages (2402.12204v1)

Published 19 Feb 2024 in cs.CL

Abstract: While LLMs have been pre-trained on multilingual corpora, their performance still lags behind in most languages compared to a few resource-rich languages. One common approach to mitigate this issue is to translate training data from resource-rich languages into other languages and then continue training. However, using the data obtained solely relying on translation while ignoring the original capabilities of LLMs across languages is not always effective, which we show will limit the performance of cross-lingual knowledge transfer. In this work, we propose SDRRL, a method based on Self-Distillation from Resource-Rich Languages that effectively improve multilingual performance by leveraging the internal capabilities of LLMs on resource-rich languages. We evaluate on different LLMs (LLaMA-2 and SeaLLM) and source languages across various comprehension and generation tasks, experimental results demonstrate that SDRRL can significantly enhance multilingual capabilities while minimizing the impact on original performance in resource-rich languages.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (74)
  1. The belebele benchmark: a parallel reading comprehension dataset in 122 language variants.
  2. xcot: Cross-lingual instruction tuning for cross-lingual chain-of-thought reasoning.
  3. Breaking language barriers in multilingual mathematical reasoning: Insights and observations.
  4. Distilling knowledge learned in bert for text generation.
  5. Improving pretrained cross-lingual language models via self-labeled word alignment.
  6. Free dolly: Introducing the world’s first truly open instruction-tuned llm.
  7. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672.
  8. Do multilingual language models think better in english?
  9. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805.
  10. Mitchell A. Gordon and Kevin Duh. 2019. Explaining sequence-level knowledge distillation as data-augmentation for neural machine translation.
  11. Knowledge distillation: A survey. International Journal of Computer Vision, 129:1789–1819.
  12. The Flores-101 evaluation benchmark for low-resource and multilingual machine translation. Transactions of the Association for Computational Linguistics, 10:522–538.
  13. XL-sum: Large-scale multilingual abstractive summarization for 44 languages. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4693–4703, Online. Association for Computational Linguistics.
  14. Distilling the knowledge in a neural network.
  15. Hiyouga. 2023. Llama factory. https://github.com/hiyouga/LLaMA-Factory.
  16. Not all languages are created equal in llms: Improving multilingual capability by cross-lingual-thought prompting.
  17. Zero-shot cross-lingual transfer of prompt-based tuning with a unified multilingual prompt.
  18. Mistral 7b. arXiv preprint arXiv:2310.06825.
  19. Mixtral of experts.
  20. Turning english-centric llms into polyglots: How much multilinguality is needed?
  21. Yoon Kim and Alexander M. Rush. 2016. Sequence-level knowledge distillation.
  22. Guillaume Lample and Alexis Conneau. 2019. Cross-lingual language model pretraining.
  23. Align after pre-train: Improving multilingual generative models with cross-lingual alignment.
  24. Bactrian-x: Multilingual replicable instruction-following models with low-rank adaptation.
  25. Pretrained language models for text generation: A survey.
  26. Label supervised llama finetuning.
  27. Few-shot learning with multilingual language models.
  28. Pre-training multilingual neural machine translation by leveraging alignment information.
  29. Mkqa: A linguistically diverse benchmark for multilingual open domain question answering.
  30. Cpt: Cross-modal prefix-tuning for speech-to-text translation. In ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6217–6221.
  31. Zhuoyuan Mao and Yen Yu. 2024a. Tuning llms with contrastive alignment instructions for machine translation in unseen, low-resource languages.
  32. Zhuoyuan Mao and Yen Yu. 2024b. Tuning llms with contrastive alignment instructions for machine translation in unseen, low-resource languages.
  33. Seallms – large language models for southeast asia.
  34. Lextreme: A multi-lingual and multi-task benchmark for the legal domain. In Findings of the Association for Computational Linguistics: EMNLP 2023. Association for Computational Linguistics.
  35. Jeroen Ooms. 2024. cld3: Google’s Compact Language Detector 3. R package version 1.6.0.
  36. OpenAI. 2022. ChatGPT. https://openai.com/chatgpt.
  37. OpenAI. 2023. GPT-4 technical report.
  38. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744.
  39. Saurabh Pahune and Manoj Chandrasekharan. 2023. Several categories of large language models (llms): A short survey. International Journal for Research in Applied Science and Engineering Technology, 11(7):615–633.
  40. Revisiting self-distillation.
  41. Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186–191, Brussels, Belgium. Association for Computational Linguistics.
  42. Cross-lingual prompting: Improving zero-shot chain-of-thought reasoning across languages.
  43. Leonardo Ranaldi and Giulia Pucci. 2023. Does the english matter? elicit cross-lingual abilities of large language models. In Proceedings of the 3rd Workshop on Multi-lingual Representation Learning (MRL), pages 173–183.
  44. Leonardo Ranaldi and Fabio Massimo Zanzotto. 2023. Empowering multi-step reasoning across languages via tree-of-thoughts.
  45. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD ’20, page 3505–3506, New York, NY, USA. Association for Computing Machinery.
  46. COMET-22: Unbabel-IST 2022 submission for the metrics shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 578–585, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  47. CometKiwi: IST-unbabel 2022 submission for the quality estimation shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 634–645, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  48. Advfusion: Multilingual adapter-based knowledge transfer for code summarization.
  49. Cross-lingual alignment of contextual word embeddings, with applications to zero-shot dependency parsing.
  50. Multilingual instruction tuning with just a pinch of multilinguality.
  51. Knowledge distillation for multilingual unsupervised neural machine translation.
  52. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca.
  53. Llama: Open and efficient foundation language models.
  54. Llama 2: Open foundation and fine-tuned chat models.
  55. Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research, 9(11).
  56. Openchat: Advancing open-source language models with mixed-quality data.
  57. Openchat: Advancing open-source language models with mixed-quality data. arXiv preprint arXiv:2309.11235.
  58. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837.
  59. Andrea Wen-Yi and David Mimno. 2023. Hyperpolyglot llms: Cross-lingual interpretability in token embeddings. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
  60. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics.
  61. Bloom: A 176b-parameter open-access multilingual language model.
  62. mt5: A massively multilingual pre-trained text-to-text transformer.
  63. Snapshot distillation: Teacher-student optimization in one generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  64. Bigtranslate: Augmenting large language models with multilingual translation capability over 100 languages.
  65. Langbridge: Multilingual reasoning without multilingual supervision.
  66. WeChat neural machine translation systems for WMT21. In Proceedings of the Sixth Conference on Machine Translation, pages 243–254, Online. Association for Computational Linguistics.
  67. Improving massively multilingual neural machine translation and zero-shot translation.
  68. Self-distillation: Towards efficient and compact neural networks. IEEE Transactions on Pattern Analysis; Machine Intelligence, 44(08):4388–4403.
  69. Self-distillation: Towards efficient and compact neural networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(8):4388–4403.
  70. Be your own teacher: Improve the performance of convolutional neural networks via self distillation.
  71. Continual knowledge distillation for neural machine translation.
  72. Yuanchi Zhang and Yang Liu. 2021. Directquote: A dataset for direct quotation extraction and attribution in news articles. arXiv preprint arXiv:2110.07827.
  73. Llama beyond english: An empirical study on language capability transfer.
  74. Question translation training for better multilingual reasoning.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (8)
  1. Yuanchi Zhang (7 papers)
  2. Yile Wang (24 papers)
  3. Zijun Liu (17 papers)
  4. Shuo Wang (382 papers)
  5. Xiaolong Wang (243 papers)
  6. Peng Li (390 papers)
  7. Maosong Sun (337 papers)
  8. Yang Liu (2253 papers)
Citations (5)
View on Semantic Scholar