In-Context Learning May Not Elicit Trustworthy Reasoning: A-Not-B Errors in Pretrained Language Models (2409.15454v1)

Abstract: Recent advancements in artificial intelligence have led to the creation of highly capable LLMs that can perform tasks in a human-like manner. However, LLMs exhibit only infant-level cognitive abilities in certain areas. One such area is the A-Not-B error, a phenomenon seen in infants where they repeat a previously rewarded behavior despite well-observed changed conditions. This highlights their lack of inhibitory control -- the ability to stop a habitual or impulsive response. In our work, we design a text-based multi-choice QA scenario similar to the A-Not-B experimental settings to systematically test the inhibitory control abilities of LLMs. We found that state-of-the-art LLMs (like Llama3-8b) perform consistently well with in-context learning (ICL) but make errors and show a significant drop of as many as 83.3% in reasoning tasks when the context changes trivially. This suggests that LLMs only have inhibitory control abilities on par with human infants in this regard, often failing to suppress the previously established response pattern during ICL.

In-Context Learning May Not Elicit Trustworthy Reasoning: A-Not-B Errors in Pretrained LLMs

The paper "In-Context Learning May Not Elicit Trustworthy Reasoning: A-Not-B Errors in Pretrained LLMs" explores an essential aspect of the cognitive abilities of LLMs. It takes inspiration from a well-documented psychological experiment, the A-Not-B task, which is used to investigate inhibitory control in human infants. The findings presented suggest substantial limitations in LLMs' reasoning capabilities, even those that are state-of-the-art.

Summary of Findings

The paper outlines systematic experiments designed to evaluate LLMs on a textual adaptation of the A-Not-B task, where models are tested for their inhibitory control -- the ability to avoid habitual responses when conditions change. This setup aims to see if LLMs can resist established response patterns when trivial changes occur in the context.

Key findings from the research include:

1. Inhibitory Control in LLMs:
   • The paper reveals that LLMs exhibit significantly limited inhibitory control abilities, akin to those observed in infants. For instance, models like Llama3-8b demonstrated a drop in accuracy by as much as 83.3% when presented with trivial changes, indicating their failure to inhibit previously learned patterns.
2. Model Size and Resilience:
   • Larger models, such as Qwen1.5-72b, showed better resilience to A-Not-B errors compared to smaller models like Qwen1.5-7b. Despite this, even the large models were not entirely immune, highlighting a general weakness across current LLM architectures.
3. Few-Shot Learning Impact:
   • The number of few-shot examples significantly affects model performance. Increased examples lead models to more strongly adhere to established trivial patterns, enhancing their susceptibility to A-Not-B errors.
4. Task Type Differences:
   • Different reasoning tasks showed varying levels of LLM susceptibility to A-Not-B errors. Arithmetic reasoning posed the highest challenge, whereas scientific reasoning exhibited minimal impact, some of which might be attributable to data contamination.

Experimental Approach

The experiments were categorized into original and A-Not-B prompting scenarios, with four representative reasoning tasks (arithmetic, commonsense, causal, and scientific reasoning) being tested. Models were provided several few-shot examples before being asked a critical question where the answer diverged from the established pattern. Performance was measured by the models' accuracy in answering these critical questions correctly.

Implications

Practical Implications

The observed limitations in LLMs' reasoning capabilities have notable implications for their deployment in real-world applications:

• Reliability: Models' tendency to adhere to trivial patterns can lead to significant reliability issues in critical applications such as medical diagnosis, legal reasoning, and autonomous systems where context changes dynamically.
• Trustworthiness: The drop in accuracy under minimal perturbations points to potential trustworthiness concerns. For instance, in applications such as customer service or educational tools, where LLMs need to adapt reliably to changing contexts, such limitations might undermine user trust.

Theoretical Implications

From a theoretical standpoint, these findings indicate a gap in current LLM architectures' ability to model human-like inhibitory control. The results call for enhancing internal mechanisms that allow models to differentiate and inhibit previously learned patterns when faced with new, albeit similar, contexts. This could involve more robust attention mechanisms or advancements in architectures that better mimic human cognitive processes.

Future Research Avenues

Several future research directions are suggested by the paper:

1. Enhanced Pretraining Strategies:
   • Improving the quality and diversity of pretraining data can endow LLMs with more robust cognitive abilities, reducing susceptibility to A-Not-B errors.
2. Incorporating Advanced Reasoning Techniques:
   • Techniques such as self-explanation have shown limited success. Future research could focus on integrating more sophisticated reasoning capabilities that allow models to better manage context shifts.
3. Exploring Cognitive Theories in AI:
   • Drawing deeper insights from cognitive science might offer new designs that can better align LLMs' reasoning with human cognitive processes.
4. Evaluating Other Cognitive Phenomena:
   • Beyond A-Not-B errors, exploring other cognitive phenomena could provide a holistic view of LLMs' capabilities and limitations.

Conclusion

The paper presents a compelling inquiry into the limitations of LLMs in contexts that require reliable and adaptive reasoning. The paper's findings highlight crucial gaps and suggest that while LLMs achieve impressive performance on many fronts, trustworthiness in reasoning under dynamic contexts remains a significant challenge. Addressing this could substantially enhance the applicability and reliability of these models across various domains. The research paves the way for future investigations that could bridge this gap, aligning AI reasoning more closely with human cognitive abilities.