Polyglot Contextual Representations Improve Crosslingual Transfer

Abstract: We introduce Rosita, a method to produce multilingual contextual word representations by training a single LLM on text from multiple languages. Our method combines the advantages of contextual word representations with those of multilingual representation learning. We produce LLMs from dissimilar language pairs (English/Arabic and English/Chinese) and use them in dependency parsing, semantic role labeling, and named entity recognition, with comparisons to monolingual and non-contextual variants. Our results provide further evidence for the benefits of polyglot learning, in which representations are shared across multiple languages.