Mechanisms underlying in-context learning in large language models

Characterize the mechanisms that enable large language models to perform in-context learning during inference without any additional weight updates when examples are provided in the prompt.

Background

The paper studies how transformer-based models might achieve in-context learning (ICL) by introducing contextual blocks that generalize transformer blocks. It proves that contextual layers can induce implicit low-rank updates to the first MLP layer based on the prompt, proposing a concrete mechanism for implicit adaptation at inference time.

Despite presenting this mechanism and empirical verification in a controlled setting, the authors emphasize that the broader underlying mechanisms enabling ICL in LLMs are not fully understood, which motivates the explicit open problem stated in the abstract.