Papers
Topics
Authors
Recent
2000 character limit reached

Teaching Large Language Models an Unseen Language on the Fly

Published 29 Feb 2024 in cs.CL | (2402.19167v2)

Abstract: Existing LLMs struggle to support numerous low-resource languages, particularly the extremely low-resource ones, for which there is minimal training data available for effective parameter updating. We thus investigate whether LLMs can learn a new language on the fly solely through prompting. To study this question, we collect a research suite for Zhuang, a language supported by no LLMs currently. We introduce DiPMT++, a framework for adapting LLMs to unseen languages by in-context learning. Using a dictionary and 5K parallel sentences only, DiPMT++ significantly enhances the performance of GPT-4 from 0 to 16 BLEU for Chinese-to-Zhuang translation and achieves 32 BLEU for Zhuang-to-Chinese translation. We also validate the effectiveness of our framework on Kalamang, another unseen language. Furthermore, we demonstrate the practical utility of DiPMT++ in aiding humans in translating completely unseen languages, which could contribute to the preservation of linguistic diversity.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (36)
  1. A few thousand translations go a long way! leveraging pre-trained models for African news translation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3053–3070, Seattle, United States. Association for Computational Linguistics.
  2. In-context examples selection for machine translation. In Findings of the Association for Computational Linguistics: ACL 2023, pages 8857–8873, Toronto, Canada. Association for Computational Linguistics.
  3. MEGA: Multilingual evaluation of generative AI. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4232–4267, Singapore. Association for Computational Linguistics.
  4. Qwen technical report. arXiv preprint arXiv:2309.16609.
  5. Instructalign: High-and-low resource language alignment via continual crosslingual instruction tuning. In Proceedings of the First Workshop in South East Asian Language Processing, pages 55–78.
  6. Scale: Synergized collaboration of asymmetric language translation engines. arXiv preprint arXiv:2309.17061.
  7. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672.
  8. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  9. Micha Elsner and Jordan Needle. 2023. Translating a low-resource language using GPT-3 and a human-readable dictionary. In Proceedings of the 20th SIGMORPHON workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 1–13, Toronto, Canada. Association for Computational Linguistics.
  10. The unreasonable effectiveness of few-shot learning for machine translation. In International Conference on Machine Learning, pages 10867–10878. PMLR.
  11. Dictionary-based phrase-level prompting of large language models for machine translation. arXiv preprint arXiv:2302.07856.
  12. Exploring human-like translation strategy with large language models. arXiv preprint arXiv:2305.04118.
  13. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations.
  14. Is chatgpt a good translator? yes with gpt-4 as the engine. arXiv preprint arXiv:2301.08745.
  15. The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6282–6293, Online. Association for Computational Linguistics.
  16. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781, Online. Association for Computational Linguistics.
  17. Franz Josef Och and Hermann Ney. 2003. A systematic comparison of various statistical alignment models. Computational Linguistics, 29(1):19–51.
  18. OpenAI. 2023. Gpt-4 technical report.
  19. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  20. MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7654–7673, Online. Association for Computational Linguistics.
  21. UNKs everywhere: Adapting multilingual language models to new scripts. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10186–10203, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  22. Maja Popović. 2015. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon, Portugal. Association for Computational Linguistics.
  23. Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186–191, Brussels, Belgium. Association for Computational Linguistics.
  24. Romanization-based large-scale adaptation of multilingual language models. arXiv preprint arXiv:2304.08865.
  25. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends® in Information Retrieval, 3(4):333–389.
  26. ChatGPT MT: Competitive for high- (but not low-) resource languages. In Proceedings of the Eighth Conference on Machine Translation, pages 392–418, Singapore. Association for Computational Linguistics.
  27. Synonym-based query expansion and boosting-based re-ranking: A two-phase approach for genomic information retrieval. In the Fourteenth Text REtrieval Conference (TREC 2005), NIST, Gaithersburg, MD.(October 2005).
  28. A benchmark for learning to translate a new language from one grammar book. In The Twelfth International Conference on Learning Representations.
  29. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
  30. Prompting PaLM for translation: Assessing strategies and performance. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15406–15427, Toronto, Canada. Association for Computational Linguistics.
  31. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100.
  32. Bigtrans: Augmenting large language models with multilingual translation capability over 100 languages. arXiv preprint arXiv:2305.18098.
  33. Empowering llm-based machine translation with cultural awareness. arXiv preprint arXiv:2305.14328.
  34. BLOOM+1: Adding language support to BLOOM for zero-shot prompting. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11682–11703, Toronto, Canada. Association for Computational Linguistics.
  35. Prompting large language model for machine translation: A case study. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 41092–41110. PMLR.
  36. Plug: Leveraging pivot language in cross-lingual instruction tuning. arXiv preprint arXiv:2311.08711.
Citations (11)
View on Semantic Scholar

Summary

  • The paper introduces DiPMT++, a framework that enables LLMs to learn unseen languages with minimal resources and achieve significant BLEU improvements.
  • It presents ZhuangBench, a comprehensive benchmark including a dictionary, parallel corpus, and test set to evaluate low-resource translation and related NLP tasks.
  • Experimental results across models like GPT-4 validate DiPMT++’s superior performance and language-agnostic adaptability in challenging translation scenarios.

Adapting LLMs to Unseen Languages with Minimal Resources: Insights from DiPMT++

Introduction to DiPMT++

This paper introduces DiPMT++, an innovative framework designed to adapt LLMs to translate languages they have never seen before, using only a small amount of linguistic resources. Specifically focusing on the Zhuang language, which is significantly low-resource and unsupported by current LLMs, this study demonstrates how LLMs can leverage in-context learning (ICL) for on-the-fly language learning. Through DiPMT++, GPT-4’s translation performance on Zhuang significantly jumps from zero to notable BLEU scores for both Zhuang-to-Chinese and Chinese-to-Zhuang directions.

ZhuangBench: A Benchmark for Unseen Language Translation

The creation of ZhuangBench, comprising a dictionary, a parallel corpus, and a machine translation test set specifically for Zhuang, is one of the paper's key contributions. This suite not only enriches linguistic resources for Zhuang but also serves as a challenging benchmark to test LLMs' ability to tackle entirely new languages through prompting. Significantly, ZhuangBench encompasses tasks beyond translation, like word sense disambiguation and cross-lingual retrieval, making it a versatile tool for NLP research on low-resource languages.

DiPMT++ Methodology

Moving beyond the original DiPMT, DiPMT++ introduces strategic extensions to handle extremely low-resource languages effectively. These include improved lexical coverage through methods like fuzzy matching, bilingual lexicon induction, and synonym expansion, as well as syntactically-informed exemplar selection to aid models in learning basic grammar. These refinements enable DiPMT++ to substantially outperform baselines and even the original DiPMT framework in translation tasks, showcasing its adaptability and effectiveness for languages completely new to LLMs.

Experimental Insights

DiPMT++’s superiority is validated through a series of experiments and comparative analyses. The framework boasts impressively high BLEU scores when applied with various backbones models like GPT-4, LLaMA-2-chat, and Qwen-chat across different settings. Additionally, the study broadens its scope by evaluating DiPMT++ on MTOB, a benchmark for translating between English and another low-resource language, Kalamang, reinforcing DiPMT++’s language-agnostic capabilities. These outcomes underscore the framework's potential in significantly advancing LLMs' proficiency in untranslated languages with minimal resources.

Practical Implications and Future Horizons

Crucially, DiPMT++ demonstrates practical utility in aiding human translation efforts for unseen languages, indicating its potential in linguistic preservation and NLP education. This aspect was explored through a user study, where participants with no prior knowledge of Zhuang achieved better translation quality and efficiency when assisted by DiPMT++. Given the framework's promising results, the paper speculates on further research directions, including expanding the typology of studied languages and enhancing the methodologies for an in-depth syntactical understanding.

Conclusion

DiPMT++ pioneers a novel approach in adapting LLMs to unseen languages efficiently, using limited resources. The framework’s success in handling Zhuang, a considerably low-resource language, opens new avenues for the preservation of underrepresented languages and broadens the horizons for NLP research in low-resource linguistics. While acknowledging the need for larger evaluations and exploring more diverse languages, this work lays a solid foundation for future explorations in the domain of on-the-fly language learning for LLMs.

File Document Download Save Streamline Icon: https://streamlinehq.com Markdown

Paper to Video (Beta)

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up

Tweets

Sign up for free to view the 2 tweets with 9 likes about this paper.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up for Free