Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
102 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Do Not Worry if You Do Not Have Data: Building Pretrained Language Models Using Translationese (2403.13638v2)

Published 20 Mar 2024 in cs.CL

Abstract: In this paper, we explore the utility of Translationese as synthetic data created using machine translation for pre-training LLMs (LMs). Pre-training requires vast amounts of monolingual data, which is mostly unavailable for languages other than English. Recently, there has been a growing interest in using synthetic data to address this data scarcity. We take the case of English and Indic languages and translate web-crawled monolingual documents (clean) into the target language. Then, we train LLMs containing 28M and 85M parameters on this translationese data (synthetic). We show that their performance on downstream natural language understanding and generative tasks is only 3.56% poorer on NLU tasks and 1.51% on NLG tasks than LMs pre-trained on clean data. Further, we propose the use of lightweight TinyLMs pre-trained on clean data to filter synthetic data efficiently which significantly improves the performance of our models. We also find that LMs trained on synthetic data strongly benefit from extended pretraining on a tiny fraction (10%) of clean data. We release the data we collected and created as a part of this work, IndicMonoDoc, the largest collection of monolingual document-level corpora, which we hope will help bridge the gap between English and non-English performance for LLMs.

View on arXiv
Definition Search Book Streamline Icon: https://streamlinehq.com
References (63)
  1. Towards a Cleaner Document-Oriented Multilingual Crawled Corpus. arXiv e-prints, page arXiv:2201.06642.
  2. IndicXNLI: Evaluating multilingual inference for Indian languages. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10994–11006, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  3. The falcon series of open language models.
  4. Nikolay Bogoychev and Rico Sennrich. 2019. Domain, translationese and noise in synthetic data for neural machine translation. CoRR, abs/1911.03362.
  5. Hindencorp-hindi-english and hindi-only corpus for machine translation. In LREC, pages 3550–3555.
  6. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc.
  7. DialogSum: A real-life scenario dialogue summarization dataset. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 5062–5074, Online. Association for Computational Linguistics.
  8. A survey of multilingual neural machine translation. ACM Comput. Surv., 53(5).
  9. Daniel Deutsch and Dan Roth. 2020. SacreROUGE: An open-source library for using and developing summarization evaluation metrics. In Proceedings of Second Workshop for NLP Open Source Software (NLP-OSS), pages 120–125, Online. Association for Computational Linguistics.
  10. Towards leaving no indic language behind: Building monolingual corpora, benchmark and models for indic languages. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12402–12426.
  11. Understanding back-translation at scale. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 489–500, Brussels, Belgium. Association for Computational Linguistics.
  12. Ronen Eldan and Yuanzhi Li. 2023. Tinystories: How small can language models be and still speak coherent english?
  13. Indictrans2: Towards high-quality and accessible machine translation models for all 22 scheduled indian languages. Transactions on Machine Learning Research.
  14. Martin Gellerstam. 1986. Translationese in swedish novels translated from english. Translation studies in Scandinavia, 1:88–95.
  15. Building large monolingual dictionaries at the Leipzig corpora collection: From 100 to 200 languages. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), pages 759–765, Istanbul, Turkey. European Language Resources Association (ELRA).
  16. Native language identification with user generated content. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3591–3601, Brussels, Belgium. Association for Computational Linguistics.
  17. Translationese in machine translation evaluation. CoRR, abs/1906.09833.
  18. XL-sum: Large-scale multilingual abstractive summarization for 44 languages. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4693–4703, Online. Association for Computational Linguistics.
  19. Kenneth Heafield. 2011. KenLM: Faster and smaller language model queries. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 187–197, Edinburgh, Scotland. Association for Computational Linguistics.
  20. Measuring massive multitask language understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
  21. Training compute-optimal large language models.
  22. Indicnlpsuite: Monolingual corpora, evaluation benchmarks and pre-trained multilingual language models for indian languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4948–4961.
  23. Yoon Kim and Alexander M. Rush. 2016. Sequence-level knowledge distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1317–1327, Austin, Texas. Association for Computational Linguistics.
  24. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
  25. Madlad-400: A multilingual and document-level large audited dataset. arXiv preprint arXiv:2309.04662.
  26. IndicNLG benchmark: Multilingual datasets for diverse NLG tasks in Indic languages. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5363–5394, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  27. The bigscience roots corpus: A 1.6 tb composite multilingual dataset. Advances in Neural Information Processing Systems, 35:31809–31826.
  28. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  29. Few-shot learning with multilingual generative language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9019–9052, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  30. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
  31. Bhasha-abhijnaanam: Native-script and romanized language identification for 22 indic languages.
  32. Aksharantar: Open Indic-language transliteration datasets and models for the next billion users. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 40–57, Singapore. Association for Computational Linguistics.
  33. Tagged back-translation revisited: Why does it really work? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5990–5997, Online. Association for Computational Linguistics.
  34. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906–1919, Online. Association for Computational Linguistics.
  35. Emille: Building a corpus of south asian languages. In Proceedings of the International Conference on Machine Translation and Multilingual Applications in the new Millennium: MT 2000.
  36. Revisiting round-trip translation for quality estimation. In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, pages 91–104, Lisboa, Portugal. European Association for Machine Translation.
  37. Abstractive text summarization using sequence-to-sequence rnns and beyond. arXiv preprint arXiv:1602.06023.
  38. Original or translated? a causal analysis of the impact of translationese on machine translation performance. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5303–5320, Seattle, United States. Association for Computational Linguistics.
  39. Asynchronous pipelines for processing huge corpora on medium to low resource infrastructures. Proceedings of the Workshop on Challenges in the Management of Large Corpora (CMLC-7) 2019. Cardiff, 22nd July 2019, pages 9 – 16, Mannheim. Leibniz-Institut für Deutsche Sprache.
  40. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  41. Maja Popović. 2017. chrF++: words helping character n-grams. In Proceedings of the Second Conference on Machine Translation, pages 612–618, Copenhagen, Denmark. Association for Computational Linguistics.
  42. Neural machine translation for translating into Croatian and Serbian. In Proceedings of the 7th Workshop on NLP for Similar Languages, Varieties and Dialects, pages 102–113, Barcelona, Spain (Online). International Committee on Computational Linguistics (ICCL).
  43. T5Score: Discriminative fine-tuning of generative evaluation metrics. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 15185–15202, Singapore. Association for Computational Linguistics.
  44. Ella Rabinovich and Shuly Wintner. 2015. Unsupervised identification of translationese. Transactions of the Association for Computational Linguistics, 3:419–432.
  45. Scaling language models: Methods, analysis & insights from training gopher.
  46. Samanantar: The largest publicly available parallel corpora collection for 11 indic languages. Transactions of the Association for Computational Linguistics, 10:145–162.
  47. Are references really needed? unbabel-IST 2021 submission for the metrics shared task. In Proceedings of the Sixth Conference on Machine Translation, pages 1030–1040, Online. Association for Computational Linguistics.
  48. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics.
  49. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 86–96, Berlin, Germany. Association for Computational Linguistics.
  50. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguistics.
  51. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. CoRR, abs/2206.04615.
  52. Roformer: Enhanced transformer with rotary position embedding.
  53. No language left behind: Scaling human-centered machine translation.
  54. The fact extraction and VERification (FEVER) shared task. In Proceedings of the First Workshop on Fact Extraction and VERification (FEVER), pages 1–9, Brussels, Belgium. Association for Computational Linguistics.
  55. Attaining the unattainable? reassessing claims of human parity in neural machine translation. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 113–123, Brussels, Belgium. Association for Computational Linguistics.
  56. Attention is all you need. Advances in neural information processing systems, 30.
  57. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461.
  58. Findings of the BabyLM challenge: Sample-efficient pretraining on developmentally plausible corpora. In Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning, pages 1–34, Singapore. Association for Computational Linguistics.
  59. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625–641.
  60. Bloom: A 176b-parameter open-access multilingual language model.
  61. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 483–498, Online. Association for Computational Linguistics.
  62. Bartscore: Evaluating generated text as text generation. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pages 27263–27277.
  63. Mike Zhang and Antonio Toral. 2019. The effect of translationese in machine translation test sets. CoRR, abs/1906.08069.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (3)
  1. Meet Doshi (4 papers)
  2. Raj Dabre (65 papers)
  3. Pushpak Bhattacharyya (153 papers)

Summary

Leveraging Translationese for Pre-training LLMs in Low-Resource Languages

Introduction to Synthetic Data in LLM Training

The reliance of state-of-the-art LLMs (LMs) on vast amounts of monolingual data presents a significant challenge for low-resource languages. Traditionally, the performance of LMs in languages such as English, with abundant data resources, towers over that in languages with scarcer resources. This disparity raises the question of how to effectively train LMs for these underrepresented languages. A novel approach addresses this by employing synthetic data, specifically "translationese", created through machine translation from high-resource languages. This paper endeavors to assess the viability and performance implications of using translationese for pre-training LMs across a range of languages, focusing primarily on English and Indic languages.

Methodological Approach to Synthetic Data Utilization

The procedure for synthetic data utilization includes several stages:

  1. Data Generation: Starting with web-crawled monolingual documents in English, these documents are translated into target Indic languages using state-of-the-art machine translation models, generating what is referred to as translationese or synthetic data.
  2. LLMs Training: Two sets of LMs, encompassing model sizes of 28M and 85M parameters, are trained on this synthetic data.
  3. TinyLMs for Data Filtering: To address the quality variance in synthetic data, Tiny LLMs (TinyLMs), pre-trained on high-quality, clean data, are utilized to filter the synthetic data based on perplexity scores. This step aims to refine the quality of data used for training larger models.
  4. Performance Evaluation: The paper evaluates the performance of LMs trained on synthetic, filtered data against those trained on clean data, across various natural language understanding (NLU) and generative (NLG) tasks.

Key Findings

The paper unveils several crucial insights:

  • LMs trained on synthetic data, even before filtering, demonstrate competitive performance, lagging by only 3.56% on NLU tasks and 1.51% on NLG tasks compared to those trained on clean data.
  • The introduction of TinyLM-based filtered synthetic data significantly enhances performance, bridging the gap between synthetic and clean data-trained models.
  • An intriguing outcome is observed when these synthetic data-trained LMs undergo further pre-training on a small fraction (10%) of clean data, which substantially improves their performance across tasks.

Bridging the Data Gap with IndicMonoDoc

Contributing to the literature and practical resources, the authors introduce IndicMonoDoc, a massive collection of monolingual document-level corpora, amounting to a total of 39.5 billion tokens across 22 scheduled languages plus English. This dataset, significantly larger than its predecessors, aims to furnish researchers and practitioners with the data necessary for training more capable and inclusive LLMs.

Implications and Future Directions

This work underscores the potential of using synthetic data, particularly translationese, as a strategic method to circumvent the data scarcity issue plaguing low-resource LLM training. The findings suggest that, with effective filtering strategies and supplementary training on clean data, models trained on synthetic data can closely rival those trained on voluminous clean data. These insights open avenues for future research in synthetic data generation and filtering techniques, aiming to democratize language technology across a broader spectrum of languages.

Furthermore, the release of IndicMonoDoc offers a valuable resource that could catalyze advancements in LLMing for a wide array of languages, marking a significant step toward bridging the performance gap between high and low-resource languages. As the field progresses, exploring the scalability of these techniques to larger models and their applicability across more diverse language families will be crucial in achieving genuinely global language technology.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown
Youtube Logo Streamline Icon: https://streamlinehq.com

YouTube