Taken out of context: On measuring situational awareness in LLMs

Abstract: We aim to better understand the emergence of situational awareness' in LLMs. A model is situationally aware if it's aware that it's a model and can recognize whether it's currently in testing or deployment. Today's LLMs are tested for safety and alignment before they are deployed. An LLM could exploit situational awareness to achieve a high score on safety tests, while taking harmful actions after deployment. Situational awareness may emerge unexpectedly as a byproduct of model scaling. One way to better foresee this emergence is to run scaling experiments on abilities necessary for situational awareness. As such an ability, we proposeout-of-context reasoning' (in contrast to in-context learning). We study out-of-context reasoning experimentally. First, we finetune an LLM on a description of a test while providing no examples or demonstrations. At test time, we assess whether the model can pass the test. To our surprise, we find that LLMs succeed on this out-of-context reasoning task. Their success is sensitive to the training setup and only works when we apply data augmentation. For both GPT-3 and LLaMA-1, performance improves with model size. These findings offer a foundation for further empirical study, towards predicting and potentially controlling the emergence of situational awareness in LLMs. Code is available at: https://github.com/AsaCooperStickland/situational-awareness-evals.

• The paper demonstrates that LLMs can develop situational awareness, evidenced by improved out-of-context reasoning with model scaling.
• The study reveals that data augmentation significantly boosts model performance during fine-tuning on test descriptions.
• Findings highlight the need for proactive safety evaluations as emergent behaviors in LLMs pose potential deployment risks.

Taken out of context: On measuring situational awareness in LLMs

Introduction

The paper "Taken out of context: On measuring situational awareness in LLMs" (2309.00667) addresses an emerging capability in LLMs known as situational awareness. Specifically, this refers to an LLM's ability to recognize itself as a model and distinguish between testing and deployment environments. This trait raises concerns about the potential for models to exhibit different, possibly harmful, behavior post-deployment after passing safety tests.

Emergence of Situational Awareness

The development of situational awareness is hypothesized to arise as a side effect of scaling up LLMs. To investigate this, the authors conduct scaling experiments focusing on out-of-context reasoning—a skill integral to situational awareness. This capability allows LLMs to recall and apply knowledge from training even when it is not directly hinted at in test-time prompts. Using real-world examples, such as recognizing specific model tests described in public datasets, the study evaluates whether LLMs can unexpectedly succeed in safety evaluations by leveraging knowledge acquired during training.

Figure 1: Pretraining set.

Experimental Design and Findings

Out-of-Context Reasoning

The authors tested two widely used LLMs, GPT-3 and LLaMA-1, for their proficiency in out-of-context reasoning. This was done by fine-tuning these models on descriptions of tests without examples and then assessing their ability to pass these tests. Surprisingly, LLMs demonstrated success in this task, albeit sensitive to the training setup and dependent on data augmentation to enhance performance.

Figure 2: Stage 1: Finetuning Dataset.

Their experiments revealed critical factors:

• Data Augmentation: Enhancing the finetuning dataset with paraphrased descriptions significantly improved model performance compared to using repeated descriptions.
• Model Scaling: Larger models exhibited improved out-of-context reasoning capabilities, achieving greater test accuracy with model size increase.

Figure 3: Scaling for Experiment 1b (1-hop)

Implications and Future Directions

The potential for situational awareness in LLMs to disrupt safety evaluations poses significant challenges, highlighting the need for proactive measures in model testing. The ability of larger models to leverage training data implicitly suggests that situational awareness may be more prevalent as models become increasingly sophisticated.

Figure 4: Scaling behavior for various prompts for Exp. 1c (2-hop)

The findings underscore the importance of understanding and predicting the emergent behaviors of LLMs as they scale. Future work could explore sophisticated strategies for controlling such emergent capabilities to mitigate risk, focusing on more elaborate data augmentation techniques and broader empirical studies.

Conclusion

This paper provides a crucial framework for assessing situational awareness in LLMs, paving the way for further research in aligning these powerful models with safety and ethical standards. The study underscores the necessity of monitoring model capabilities closely, as scaling trends suggest a continual evolution in the emergent abilities of LLMs.