Continuous multilinguality with language vectors

Abstract: Most existing models for multilingual NLP treat language as a discrete category, and make predictions for either one language or the other. In contrast, we propose using continuous vector representations of language. We show that these can be learned efficiently with a character-based neural LLM, and used to improve inference about language varieties not seen during training. In experiments with 1303 Bible translations into 990 different languages, we empirically explore the capacity of multilingual LLMs, and also show that the language vectors capture genetic relationships between languages.