Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Mitigating the Linguistic Gap with Phonemic Representations for Robust Cross-lingual Transfer (2402.14279v3)

Published 22 Feb 2024 in cs.CL and cs.AI

Abstract: Approaches to improving multilingual language understanding often struggle with significant performance gaps between high-resource and low-resource languages. While there are efforts to align the languages in a single latent space to mitigate such gaps, how different input-level representations influence such gaps has not been investigated, particularly with phonemic inputs. We hypothesize that the performance gaps are affected by representation discrepancies between these languages, and revisit the use of phonemic representations as a means to mitigate these discrepancies. To demonstrate the effectiveness of phonemic representations, we present experiments on three representative cross-lingual tasks on 12 languages in total. The results show that phonemic representations exhibit higher similarities between languages compared to orthographic representations, and it consistently outperforms grapheme-based baseline model on languages that are relatively low-resourced. We present quantitative evidence from three cross-lingual tasks that demonstrate the effectiveness of phonemic representations, and it is further justified by a theoretical analysis of the cross-lingual performance gap.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (35)
  1. Nash equilibria and pitfalls of adversarial training in adversarial robustness games. In International Conference on Artificial Intelligence and Statistics, pages 9607–9636. PMLR.
  2. A theory of learning from different domains. Machine Learning, 79:151–175.
  3. Analysis of representations for domain adaptation. In Advances in Neural Information Processing Systems, volume 19. MIT Press.
  4. Phonologically aware neural model for named entity recognition in low resource transfer settings. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1462–1472.
  5. Adapting word embeddings to new languages with morphological and phonological subword representations. arXiv preprint arXiv:1808.09500.
  6. AdvPicker: Effectively Leveraging Unlabeled Data via Adversarial Discriminator for Cross-Lingual NER. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 743–753, Online. Association for Computational Linguistics.
  7. Adversarial deep averaging networks for cross-lingual sentiment classification. Transactions of the Association for Computational Linguistics, 6:557–570.
  8. InfoXLM: An information-theoretic framework for cross-lingual language model pre-training. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3576–3588, Online. Association for Computational Linguistics.
  9. Canine: Pre-training an efficient tokenization-free encoder for language representation. Transactions of the Association for Computational Linguistics, 10:73–91.
  10. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440–8451.
  11. Alexis Conneau and Guillaume Lample. 2019. Cross-lingual language model pretraining. Advances in neural information processing systems, 32.
  12. Xnli: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
  13. Ryan Cotterell and Georg Heigold. 2017. Cross-lingual character-level neural morphological tagging. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 748–759, Copenhagen, Denmark. Association for Computational Linguistics.
  14. Phoneme level language models for sequence based low resource asr. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6091–6095. IEEE.
  15. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
  16. Universal neural machine translation for extremely low resource languages. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 344–354, New Orleans, Louisiana. Association for Computational Linguistics.
  17. Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation. In International Conference on Machine Learning, pages 4411–4421. PMLR.
  18. Phoneme-based contextualization for cross-lingual speech recognition in end-to-end models. arXiv preprint arXiv:1906.09292.
  19. Detecting change in data streams. In Proceedings of the Thirtieth International Conference on Very Large Data Bases - Volume 30, VLDB ’04, page 180–191. VLDB Endowment.
  20. Word translation without parallel data. In International Conference on Learning Representations.
  21. Ilya Loshchilov and Frank Hutter. 2018. Decoupled weight decay regularization. In International Conference on Learning Representations.
  22. Mixed precision training. In International Conference on Learning Representations.
  23. Epitran: Precision G2P for many languages. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
  24. Xphonebert: A pre-trained multilingual model for phoneme representations for text-to-speech. arXiv preprint arXiv:2305.19709.
  25. Cross-lingual name tagging and linking for 282 languages. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1946–1958, Vancouver, Canada. Association for Computational Linguistics.
  26. Massively multilingual transfer for ner. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 151–164.
  27. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
  28. Polylm: An open source polyglot large language model. arXiv preprint arXiv:2307.06018.
  29. On learning universal representations across languages. In International Conference on Learning Representations.
  30. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100.
  31. Shijie Wu and Mark Dredze. 2020. Are all languages created equal in multilingual BERT? In Proceedings of the 5th Workshop on Representation Learning for NLP, pages 120–130, Online. Association for Computational Linguistics.
  32. Enhancing cross-lingual transfer by manifold mixup. arXiv preprint arXiv:2205.04182.
  33. Bridging the gap between native text and translated text through adversarial learning: A case study on cross-lingual event extraction. In Findings of the Association for Computational Linguistics: EACL 2023, pages 754–769, Dubrovnik, Croatia. Association for Computational Linguistics.
  34. Breaking physical and linguistic borders: Multilingual federated prompt tuning for low-resource languages. In International Workshop on Federated Learning in the Age of Foundation Models in Conjunction with NeurIPS 2023.
  35. Consistency regularization for cross-lingual fine-tuning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3403–3417, Online. Association for Computational Linguistics.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (7)
  1. Haeji Jung (4 papers)
  2. Changdae Oh (12 papers)
  3. Jooeon Kang (3 papers)
  4. Jimin Sohn (4 papers)
  5. Kyungwoo Song (38 papers)
  6. Jinkyu Kim (51 papers)
  7. David R. Mortensen (40 papers)
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now