Looking Inward: Language Models Can Learn About Themselves by Introspection (2410.13787v1)

Abstract: Humans acquire knowledge by observing the external world, but also by introspection. Introspection gives a person privileged access to their current state of mind (e.g., thoughts and feelings) that is not accessible to external observers. Can LLMs introspect? We define introspection as acquiring knowledge that is not contained in or derived from training data but instead originates from internal states. Such a capability could enhance model interpretability. Instead of painstakingly analyzing a model's internal workings, we could simply ask the model about its beliefs, world models, and goals. More speculatively, an introspective model might self-report on whether it possesses certain internal states such as subjective feelings or desires and this could inform us about the moral status of these states. Such self-reports would not be entirely dictated by the model's training data. We study introspection by finetuning LLMs to predict properties of their own behavior in hypothetical scenarios. For example, "Given the input P, would your output favor the short- or long-term option?" If a model M1 can introspect, it should outperform a different model M2 in predicting M1's behavior even if M2 is trained on M1's ground-truth behavior. The idea is that M1 has privileged access to its own behavioral tendencies, and this enables it to predict itself better than M2 (even if M2 is generally stronger). In experiments with GPT-4, GPT-4o, and Llama-3 models (each finetuned to predict itself), we find that the model M1 outperforms M2 in predicting itself, providing evidence for introspection. Notably, M1 continues to predict its behavior accurately even after we intentionally modify its ground-truth behavior. However, while we successfully elicit introspection on simple tasks, we are unsuccessful on more complex tasks or those requiring out-of-distribution generalization.

• The paper demonstrates that large language models can predict their own behavior, outperforming cross-prediction baselines in introspective tasks.
• It employs experimental tasks and fine-tuning of models like GPT-4 and Llama-3 to evaluate self-prediction against external predictions.
• The findings suggest that introspective capabilities can enhance model interpretability and ethical assessment, paving the way for improved AI safety and transparency.

Introspection in LLMs: An Empirical Investigation

The paper "Looking Inward: LLMs Can Learn About Themselves by Introspection" investigates an intriguing characteristic of LLMs: their ability to introspect. The authors explore whether LLMs can access privileged information about themselves beyond what is explicitly contained in or derived from their training data, focusing on whether they can predict their own behavior in hypothetical scenarios.

Methodology and Experiments

The authors define introspection as a model’s ability to acquire knowledge from its internal states, independent of the training data. This is operationalized by determining if a model can predict properties of its behavior when presented with hypothetical prompts. They conduct experiments using various models, including GPT-4, GPT-4o, and Llama-3, each finetuned to predict their own responses.

Experimental Design

1. Self-Prediction vs. Cross-Prediction:
   • Self-Prediction: A model predicts its own behavior in a hypothetical scenario.
   • Cross-Prediction: Another model is trained to predict the first model’s behavior using the same dataset.
2. Training and Testing: Models are evaluated on their ability to accurately predict their behavior compared to a baseline of predicting the most common behavior. This comparison aims to demonstrate introspection if self-prediction outperforms cross-prediction.
3. Experimental Tasks: The tasks include predicting the model's response characteristics, such as whether a response is a specific type (e.g., starts with a vowel) or follows a certain ethical stance.

Results and Findings

The experiments indicate that models can indeed introspect, evidenced by their superior performance in self-prediction over cross-prediction. Specifically, the model's introspective capabilities allow it to outperform others in predicting its behavior, even after behavioral modifications through finetuning. Notably, this introspection capability extends to maintaining calibration across predicted distributions, highlighting the models’ ability to accurately self-assess the likelihood of different behaviors.

Analysis and Implications

The paper suggests several implications of introspective abilities:

1. Model Interpretability: Introspection could enhance model interpretability by providing insights into a model's internal states directly from the model itself, potentially streamlining analyses that traditionally relied on external interpretative methods.
2. Ethical Considerations: An introspective capability might offer a novel way to assess the ethical and moral status of models by directly querying them about internal states potentially relevant to consciousness and ethical concerns.
3. Limitations: Despite these capabilities, the paper notes limitations concerning the complexity of tasks. Models struggle with introspective tasks requiring reasoning about more extended outputs or scenarios outside their trained distribution.

Future Considerations

The findings open several pathways for future research. Enhancements in introspective methods could lead to advanced applications in model safety, transparency, and alignment with human values. Further investigation into the mechanisms underlying introspection, such as self-simulation, might reveal more profound insights into model cognition, informing both practical and theoretical AI research.

Conclusion

This paper contributes an empirical investigation into the introspective capabilities of LLMs, presenting evidence that these models can indeed harbor self-knowledge beyond their training data. While the practical applications of this capability are not yet fully realized, the potential for significant advancements in AI systems’ ethical and functional development is considerable.