Relative Value Biases in Large Language Models

Abstract: Studies of reinforcement learning in humans and animals have demonstrated a preference for options that yielded relatively better outcomes in the past, even when those options are associated with lower absolute reward. The present study tested whether LLMs would exhibit a similar bias. We had gpt-4-1106-preview (GPT-4 Turbo) and Llama-2-70B make repeated choices between pairs of options with the goal of maximizing payoffs. A complete record of previous outcomes was included in each prompt. Both models exhibited relative value decision biases similar to those observed in humans and animals. Making relative comparisons among outcomes more explicit magnified the bias, whereas prompting the models to estimate expected outcomes caused the bias to disappear. These results have implications for the potential mechanisms that contribute to context-dependent choice in human agents.

• The paper demonstrates that LLMs such as GPT-4 Turbo and Llama-2-70B exhibit relative value decision biases in tasks akin to human learning.
• It employs a bandit task framework, revealing that explicit outcome comparisons sharpen biases while prompting value estimates mitigates them.
• The findings suggest that LLMs mirror human cognitive features, offering actionable insights into machine psychology and decision-making processes.

Introduction

Exploring the relative value biases in decision-making processes exemplifies the current trajectory of research into the cognitive abilities of LLMs. The intersection between reinforced learning patterns in humans and machines provides a fertile ground for understanding the nuances of context-dependent choice and value-based decision-making. Our analysis of a new study investigates this phenomenon in state-of-the-art LLMs.

Methodology Overview

The study employed a bandit task adaptation with gpt-4-1106-preview (GPT-4 Turbo) and Llama-2-70B models as test subjects. Methodologically, these models engaged in a learning phase that involved repeated choices between fixed option pairs to maximize payoffs. This setup replicated conditions similar to human experiments, where subsequent phases required extrapolating past learned values to make optimal decisions without additional feedback.

Findings

Results indicated that both GPT-4 Turbo and Llama-2-70B models displayed relative value decision biases. Explicit comparisons of outcomes sharpened this bias, while invoking the models to estimate expected outcomes mitigated it. GPT-4 Turbo, in particular, showed a pronounced bias influenced by relative value, hence paralleling human behavioral tendencies under similar task constraints.

Discussion

The implications of the observed biases in LLMs extend beyond a mere replication of human and animal decision-making patterns. They invite a larger conversation on the emergent properties of LLMs, considering the role of training datasets and model architecture. By demonstrating substantial parallels between human cognitive features and the behavior of LLMs, the research contributes to the broader discussion on machine psychology, emphasizing how the structured nature of language processing in AI agents can lead to complex behavioral outcomes.

In dissecting the mechanisms of these models, future inquiries might explore the internal representations of value within these models. Such introspections could unravel the computational emergence of biases and the potential for LLMs to mimic human-like decision-making processes even more closely.