Language Models as Agent Models (2212.01681v1)

Abstract: LLMs (LMs) are trained on collections of documents, written by individual human agents to achieve specific goals in an outside world. During training, LMs have access only to text of these documents, with no direct evidence of the internal states of the agents that produced them -- a fact often used to argue that LMs are incapable of modeling goal-directed aspects of human language production and comprehension. Can LMs trained on text learn anything at all about the relationship between language and use? I argue that LMs are models of intentional communication in a specific, narrow sense. When performing next word prediction given a textual context, an LM can infer and represent properties of an agent likely to have produced that context. These representations can in turn influence subsequent LM generation in the same way that agents' communicative intentions influence their language. I survey findings from the recent literature showing that -- even in today's non-robust and error-prone models -- LMs infer and use representations of fine-grained communicative intentions and more abstract beliefs and goals. Despite the limited nature of their training data, they can thus serve as building blocks for systems that communicate and act intentionally.

Analyzing LLMs as Agent Models

The paper "LLMs as Agent Models" by Jacob Andreas scrutinizes the capability of LLMs (LMs) to model communicative intentions, beliefs, and desires of hypothetical agents solely based on textual data. The research primarily addresses the notion that LMs, although limited by their training on text without explicit information about agent states, can infer and represent aspects of agent-like behavior. This essay explores the paper's arguments, case studies, and implications for artificial intelligence research.

The core argument presented is that LMs can function as narrow models of agents by predicting relations between agent observations, internal states, and subsequent actions or utterances during the task of next-word prediction. Andreas posits two primary claims: the ability of LMs to infer partial representations of agent states, and the influence of these inferred representations on model predictions. This perspective shifts the view of LMs from simple statistical models to potential simulators of agent-like purposeful communication.

The paper offers several evidential insights through case studies, each evaluating different facets of agent modeling by LMs. In a controlled toy experiment, LLMs were trained on a dataset comprised of documents exhibiting varying agent beliefs. The paper demonstrated that LMs could infer author identity based on text segments and generate text consistent with specific belief types, underscoring the models' ability to mimic behavior consistent with agent states.

In the context of modeling communicative intentions, the well-known sentiment analysis experiment featuring an LSTM trained on product reviews highlights the representational capacity of LMs. Notably, a single neuron within the LSTM's architecture was revealed to encode sentiment, influencing the sentiment of generated text when manipulated—a striking example of how latent features related to agent intentions emerge in LMs.

Further, the research explores belief modeling using transformer-based models in dynamic domains requiring entity state tracking. LMs demonstrated accurate linear encodings of entities' properties and states, impacting text generation processes. This capability emphasizes the models' potential in understanding and simulating agents' belief states regarding their environments.

The concept of modeling desires is investigated through the lens of LM response to truthfulness-oriented prompts. The paper reveals that altering the context or prompt can significantly influence the LM's tendency to generate truthful or misleading content. The alignment of model behavior with the described communicative goals in prompts shows another dimension of agent-like simulation within LMs.

The implications of these findings are profound for both theoretical and practical developments in AI. The ability to simulate aspects of agent behavior suggests new approaches to building interactive systems that leverage these inferred agent states for more coherent and goal-driven natural language understanding and generation. However, the analysis also acknowledges the limitations inherent in current models—namely, issues of context length, architecture constraints, and incomplete representations of complex states—highlighting directions for future model enhancements and hybrid learning paradigms.

Practically, employing LMs as agent models could lead to more efficient training protocols and better simulation of communicative environments, bridging gaps between human language processing and machine understanding.

In conclusion, the paper provides a compelling examination of LMs' potential to model aspects of intentional communicative behavior. While recognizing current limitations, it paves the way for future exploration into integrating LLMs more deeply into agent-oriented applications, thereby enhancing AI systems' effectiveness and interaction capabilities. This work serves as an insightful contribution to the discussion on the evolving role of LMs in simulating and interacting with agent-like constructs within AI systems.