Transformers for Low-Resource Languages: Is Féidir Linn! (2403.01985v1)
Abstract: The Transformer model is the state-of-the-art in Machine Translation. However, in general, neural translation models often under perform on language pairs with insufficient training data. As a consequence, relatively few experiments have been carried out using this architecture on low-resource language pairs. In this study, hyperparameter optimization of Transformer models in translating the low-resource English-Irish language pair is evaluated. We demonstrate that choosing appropriate parameters leads to considerable performance improvements. Most importantly, the correct choice of subword model is shown to be the biggest driver of translation performance. SentencePiece models using both unigram and BPE approaches were appraised. Variations on model architectures included modifying the number of layers, testing various regularisation techniques and evaluating the optimal number of heads for attention. A generic 55k DGT corpus and an in-domain 88k public admin corpus were used for evaluation. A Transformer optimized model demonstrated a BLEU score improvement of 7.8 points when compared with a baseline RNN model. Improvements were observed across a range of metrics, including TER, indicating a substantially reduced post editing effort for Transformer optimized models with 16k BPE subword models. Bench-marked against Google Translate, our translation engines demonstrated significant improvements. The question of whether or not Transformers can be used effectively in a low-resource setting of English-Irish translation has been addressed. Is f\'eidir linn - yes we can.
- The digital divide and social inclusion among refugee migrants. Information Technology & People.
- Optimizing transformer for low-resource neural machine translation. arXiv preprint arXiv:2011.02266.
- Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.
- On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 610–623.
- Random search for hyper-parameter optimization. Journal of machine learning research, 13(2).
- Bisong, E. (2019). Google colaboratory. In Building Machine Learning and Deep Learning Models on Google Cloud Platform, pages 59–64. Springer.
- Findings of the 2017 conference on machine translation (WMT17). In Proceedings of the Second Conference on Machine Translation, pages 169–214, Copenhagen, Denmark. Association for Computational Linguistics.
- Findings of the 2018 conference on machine translation (WMT18). In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 272–303, Belgium, Brussels. Association for Computational Linguistics.
- On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv:1409.1259.
- A call for prudent choice of subword merge operations in neural machine translation. arXiv preprint arXiv:1905.10453.
- Smt versus nmt: Preliminary comparisons for irish.
- Gage, P. (1994). A new algorithm for data compression. C Users Journal, 12(2):23–38.
- Deep reconstruction-classification networks for unsupervised domain adaptation. In European Conference on Computer Vision, pages 597–613. Springer.
- Finding the optimal vocabulary size for neural machine translation. arXiv preprint arXiv:2004.02334.
- Opennmt: Open-source toolkit for neural machine translation. arXiv preprint arXiv:1701.02810.
- Kudo, T. (2018). Subword regularization: Improving neural network translation models with multiple subword candidates. arXiv preprint arXiv:1804.10959.
- Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv:1808.06226.
- Quantifying the carbon emissions of machine learning. arXiv preprint arXiv:1910.09700.
- Pivot machine translation using chinese as pivot language. In China Workshop on Machine Translation, pages 74–85. Springer.
- Responses to language barriers in consultations with refugees and asylum seekers: a telephone survey of irish general practitioners. BMC Family Practice, 9(1):1–6.
- Montgomery, D. C. (2017). Design and analysis of experiments. John wiley & sons.
- Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318.
- Popović, M. (2015). chrf: character n-gram f-score for automatic mt evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395.
- Informing the use of hyperparameter optimization through metalearning. In 2017 IEEE International Conference on Data Mining (ICDM), pages 1051–1056. IEEE.
- SEAI (2020). Sustainable Energy in Ireland.
- Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909.
- A study of translation edit rate with targeted human annotation. In Proceedings of association for machine translation in the Americas, volume 200. Citeseer.
- Results of the wmt15 metrics shared task. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 256–273.
- Dgt-tm: A freely available translation memory in 22 languages. arXiv preprint arXiv:1309.5226.
- On optimal transformer depth for low-resource language translation. arXiv preprint arXiv:2004.04418.
- Attention is all you need. arXiv preprint arXiv:1706.03762.
- Way, A. (2020). MT Developments in the EU: Keynote AMTA 2020.
- Séamus Lankford (17 papers)
- Haithem Afli (13 papers)
- Andy Way (46 papers)