Emergent Word Order Universals from Cognitively-Motivated Language Models (2402.12363v2)

Abstract: The world's languages exhibit certain so-called typological or implicational universals; for example, Subject-Object-Verb (SOV) languages typically use postpositions. Explaining the source of such biases is a key goal of linguistics. We study word-order universals through a computational simulation with LLMs (LMs). Our experiments show that typologically-typical word orders tend to have lower perplexity estimated by LMs with cognitively plausible biases: syntactic biases, specific parsing strategies, and memory limitations. This suggests that the interplay of cognitive biases and predictability (perplexity) can explain many aspects of word-order universals. It also showcases the advantage of cognitively-motivated LMs, typically employed in cognitive modeling, in the simulation of language universals.

Exploring the Roots of Linguistic Word Order Through the Lens of LLMs

Introduction

The linguistic community has long been fascinated with the universals of word order across human languages. A particular area of interest is the implicational universals, patterns that seem to govern the structure of languages worldwide, such as the prevalent Subject-Object-Verb (SOV) order often accompanying postpositions. Unraveling the origins and the cognitive underpinnings of these patterns is essential for both theoretical and applied linguistics. This summary explores a paper that harnesses computational simulations with LLMs (LMs) to shed light on these phenomena. Specifically, it examines how typologically common word orders align with lower perplexity estimates by LMs incorporating features mimicking human cognitive biases.

Exploring Word Order Bias in LMs

At the heart of this investigation is the concept that the predictability and cognitive load associated with processing different word orders can be quantified using perplexity measures generated by LMs. By training various LMs on artificial languages engineered to reflect different word order configurations, the paper demonstrates a correlation between the LMs' perplexity estimates and the frequency of these configurations in attested languages. Key findings highlight that LMs reflecting syntactic biases, parsing strategies, and memory limitations—factors grounded in human cognitive processes—better echo the typological distribution of word orders.

Contributions to Linguistic Theory

The significance of this paper lies in its multidisciplinary approach, bridging computational linguistics with cognitive modeling. It posits that cognitive biases in predictability, streamlined by specific parsing strategies and memory constraints, are instrumental in shaping the typological patterns of word order. This nexus between cognitive plausibility and language universals marks a leap in understanding the evolution of linguistic structures. Moreover, the research underlines the utility of cognitively-oriented LMs in simulating human language processing mechanisms, thereby opening avenues for probing into linguistic theories with computational tools.

Implications and Prospects

From a theoretical standpoint, these findings enrich our comprehension of language evolution, suggesting that innate cognitive biases may significantly influence linguistic universality. Furthermore, from an applied perspective, the insights gleaned could enhance natural language processing algorithms, offering a nuanced understanding of the cognitive aspects driving human language comprehension and generation.

Looking forward, the paper sets the stage for further explorations into the intricate interplay between cognitive constraints and language structure. It beckons a closer examination of how other linguistic features, beyond word order, might emerge from cognitive predispositions. Moreover, it encourages the development of more sophisticated computational models that can encapsulate the multifaceted nature of human language cognition. As we stride towards unraveling the complexities of linguistic phenomenology, this paper underscores the indispensable role of computational simulations, married with cognitive insights, in advancing our grasp of language universals.