Dissociating language and thought in large language models (2301.06627v3)

Abstract: LLMs have come closest among all models to date to mastering human language, yet opinions about their linguistic and cognitive capabilities remain split. Here, we evaluate LLMs using a distinction between formal linguistic competence -- knowledge of linguistic rules and patterns -- and functional linguistic competence -- understanding and using language in the world. We ground this distinction in human neuroscience, which has shown that formal and functional competence rely on different neural mechanisms. Although LLMs are surprisingly good at formal competence, their performance on functional competence tasks remains spotty and often requires specialized fine-tuning and/or coupling with external modules. We posit that models that use language in human-like ways would need to master both of these competence types, which, in turn, could require the emergence of mechanisms specialized for formal linguistic competence, distinct from functional competence.

An Expert Overview of "Dissociating Language and Thought in LLMs"

The paper entitled "Dissociating language and thought in LLMs" by Kyle Mahowald et al. presents a nuanced examination of the capabilities and limitations of contemporary LLMs in relation to human linguistic and cognitive functions. The paper delineates a pivotal conceptual division between "formal linguistic competence" and "functional linguistic competence." This distinction draws from cognitive neuroscience and provides a critical framework for assessing the adequacy of LLMs as models of human language and thought.

Key Thesis: Distinguishing Competence Types

Central to their argument is the differentiation between formal and functional linguistic competence. Formal competence involves the mastery of linguistic rules and patterns, akin to the capabilities demonstrated by LLMs in generating coherent and grammatically correct text. Functional linguistic competence, by contrast, entails the ability to use language pragmatically to achieve goals, requiring non-linguistic cognitive capacities such as reasoning and social understanding. The authors ground this distinction in neuroscience, emphasizing that distinct neural networks support these competencies in humans.

Evaluation of LLMs' Competence

The authors evaluate LLMs on the basis of this bifurcation:

1. Formal Linguistic Competence:
   • LLMs exhibit near-human levels of formal competence in English, demonstrating sophisticated understanding of syntactic structures and abstractions. The models have shown remarkable advances in generating text that adheres to complex grammatical conventions and exhibit hierarchical understanding. The ability of LLMs to encapsulate linguistic abstractions is highlighted, challenging prior assertions that statistical models could not achieve such linguistic sophistication.
2. Functional Linguistic Competence:
   • In domains requiring functional competence, however, LLMs fall short. The paper discusses limitations in areas such as formal reasoning, world knowledge, situation modeling, and social reasoning. These competencies require integration with other cognitive faculties that LLMs, as presently developed, do not replicate satisfactorily. Notably, the authors address models' struggles with logical reasoning and maintaining consistent world knowledge, underscoring that these are not innately linguistic capabilities.

Implications and Future Directions

The research poses significant implications for both artificial intelligence development and cognitive science. Practically, it suggests that LLMs would benefit from a modular approach, mirroring the brain's separation of language and other cognitive processes. Future LLM advancements may require distinct architectural modules for different competence types—formal vs. functional. The authors propose "Architectural Modularity," which involves explicitly incorporating separate modules, and "Emergent Modularity," which encourages the development of specialized subcomponents naturally through training.

From a theoretical perspective, exploring the limits of how much functional competence can be learned solely through linguistic input remains an open question. The authors speculate that aspects of functional competence may require grounding in experiential data beyond the language itself. This necessitates revisiting benchmarks used to assess LLMs and designing those that more rigorously distinguish between formal and functional capabilities.

Conclusion

The paper by Mahowald et al. offers a comprehensive and critical review of LLMs' capacities in relation to human-like language usage. By articulating the distinction between formal and functional competencies, the authors clarify the scope and limitations of LLMs, painting a clearer picture of how these models parallel—and diverge from—human neural processes. This distinction not only guides future AI model development towards more human-like capabilities but also provides an informative framework for understanding the cognitive structure of language and thought in humans. In sum, while LLMs excel in formal linguistic competence, achieving functional competence remains a significant challenge, necessitating innovative approaches and methodologies.