In-Context Learning Overview

Updated 6 July 2025

In-context learning is a method where models adapt to tasks by conditioning on input–output examples (e.g., for translation or question answering) without altering their parameters.
It leverages both context-scaling and task-scaling, with performance improving as more relevant demonstrations are provided or as pretrained task diversity increases.
While ICL offers rapid task generalization and broad application potential, challenges in prompt construction and sample efficiency continue to drive further research.

In-context learning (ICL) is the capability of models—particularly LLMs and general autoregressive architectures—to rapidly generalize to new tasks by conditioning on example demonstrations provided in their input, rather than by updating model parameters. This paradigm has reshaped both the empirical landscape of natural language processing and the theoretical paper of adaptive machine learning, revealing rich phenomena distinct from classical learning frameworks.

1. Definition and Fundamental Mechanisms

In-context learning refers to the process by which a pretrained model is presented with a prompt comprising several input–output demonstration pairs (the “context”) alongside a new input query. Without updating its weights, the model predicts the output for the query, ideally leveraging the relationships among the demonstrations (2302.04931, 2303.07895).

Early work established that ICL emerges as a byproduct of large-scale pretraining: LLMs, for example, can be adapted to tasks such as classification, question answering, and translation simply by supplying relevant demonstrations as input (2303.07895). Importantly, ICL does not entail gradient updates to the model's parameters during task adaptation; all “learning” happens inside the forward pass over the prompt.

Mechanistically, two complementary theoretical perspectives have been articulated (2402.02212):

Skill Recognition: The model identifies, via Bayesian or probabilistic inference, which of the tasks implicitly embedded in its pretraining best matches the context examples. Practically, this can be described as selecting a pre-learned data generation function; the prompt sharpens the model's internal distribution toward the specific task at hand.
Skill Learning: The model dynamically fits a new data generation function to the provided demonstrations, mimicking an inner-loop learning algorithm (often analogous to a gradient descent step) within its forward pass. This enables adaptation to label mappings or functional relations not observed during pretraining.

Mathematically, one formalization defines the in-context prediction as

$\hat{y} = \arg\max_{y'} f_\theta(p \oplus x \oplus y'),$

with $f_\theta$ the frozen model, $p$ the prompt encoding the in-context examples, and $x$ the new input (2303.07895).

2. Scaling Behaviors: Context-Scaling and Task-Scaling

Two critical scaling dimensions in ICL have been delineated (2410.12783):

Context-scaling: Performance improves as the number of in-context demonstrations increases, holding the diversity of pretraining tasks fixed. Transformers (and even simplified variants with fixed attention) exhibit this property, with accuracy or regression error decreasing as more examples appear in the prompt.
Task-scaling: Performance increases as the diversity and number of pretraining tasks grow, even with fixed prompt length. Standard multi-layer perceptrons tend to benefit from task-scaling but not from context-scaling.

This distinction was made explicit through rigorous experiments and theoretical analysis. For example, standard MLPs trained on vectorized inputs did not improve when the prompt length was increased, while Transformers and hybrid feature-mapped MLPs did. The mathematical insight is that the key component for context-scaling is a data-dependent feature map (often realized in practice by a kernel smoother or as a single attention layer in a transformer); task-scaling relies on diverse task exposure during pretraining.

3. Empirical Advancements and Applications

ICL has enabled advances in a range of applications, from classical NLP problems to novel modalities. Examples include:

Many-shot instruction tuning: Efficient transformer architectures such as EvaLM scale context windows up to hundreds of thousands of tokens, enabling the processing of far more demonstrations per prompt than previously feasible (2302.04931). This led to an average accuracy gain of 4.1% over strong baselines on diverse downstream tasks, with optimal performance typically achieved using long (e.g., 12k-token) contexts.
Domain extension: By translating ICL concepts into domains such as graphs (2305.12600), molecules (2310.08863), images, or even EEG signals (2501.06256), models can generalize rapidly to new graph classification or property prediction tasks provided only a handful of in-context examples, without parameter updates.
Self-optimizing context retrieval: Recent frameworks enable LLMs to self-select and optimize their in-context demonstrations (retrieval, ranking, and ordering) via reinforcement learning, leading to robust and diverse prompt construction with direct performance gains over static or hand-crafted example selection (2408.07505).

Applications include dynamic adaptation in recommendation systems, biomedical property prediction, and rapid environmental modeling, among others (2305.12600, 2310.08863, 2406.13493).

4. Theoretical Frameworks and Limitations

The paper of ICL has prompted the development of new theoretical frameworks that extend classical learning theory. Notably, the Probably Approximately Correct (PAC) paradigm has been adapted to in-context settings, showing that—with a polynomial number of pretraining examples plus a limited number of in-context examples—ICL achieves finite sample complexity bounds comparable to Bayes optimal learners, under mild assumptions about the pretraining distribution (2303.07895).

The efficiency of ICL is also subject to inherent limitations. Recent work has shown that while ICL matches the sample complexity of the Bayes estimator in the few-shot regime, its efficiency deteriorates in long-context, many-shot scenarios; it requires up to 45% more demonstrations than the Bayes-optimal learner to achieve top-tier performance (2502.04580). This “technical debt” arises from information-theoretic constraints: the excess risk does not vanish as more demonstrations are added, and the mutual information gain from each additional demonstration diminishes.

Table 1: Relative Sample Complexity in ICL

Regime	ICL vs. Bayes Optimal Sample Complexity
Few-shot	+10% demonstrations
Many-shot	Up to +45% demonstrations

[Data from (2502.04580).]

Such analyses have motivated hybrid models that combine demonstration-based and adaptive update-based learning to overcome sample inefficiency in ultra-long contexts.

5. The Role of Pretraining Data and Prompt Construction

Recent studies have revealed that the emergence and quality of ICL are highly sensitive to the properties of pretraining data and the form of the prompt:

Instance-level curation: A small subset of “supportive” pretraining instances—selected through gradient alignment with ICL loss—can dramatically boost downstream ICL performance, despite lacking strong domain similarity to test tasks (2306.15091). These supportive examples are enriched with rare tokens and challenging long-range contexts.
Repetitions and sequence structure: The explicit inclusion of exact token or exemplar repetitions in the pretraining sequences facilitates stable, non-transient ICL, likely by enhancing the “look-up” mechanism (2501.06256).
Prompt selection: With the advent of long-context models (context windows up to 2M tokens), the importance of sophisticated selection strategies for prompt examples diminishes. Random sampling performs nearly as well as optimized selection in the many-shot regime, redirecting the challenge from curation to filling the context with sufficient data, e.g., via data augmentation (2412.16926).

Theoretical work underlines that prompts more similar to the target distribution—formally, with lower Maximum Mean Discrepancy (MMD)—yield less biased implicit knowledge distillation and thus better ICL performance (2506.11516). This has directly informed the design of retrieval- and RL-based demonstration selection systems.

6. Extensions, Variations, and Meta-Learning

ICL research has expanded beyond core language and vision domains:

In-context in-context learning: Models can be equipped to leverage not just the current dataset, but also multiple related datasets at inference, improving the accuracy of process-specific predictions. Architectures such as the ICICL-TNP integrate multiple datasets efficiently via pseudo-token-based attention (2406.13493).
Meta-in-context learning: Models exposed to sequences of tasks recursively adapt their in-context learning algorithm—modifying priors and downstream adaptation strategies through contextual exposure alone, achieving competitive results with classical online learning algorithms in regression and reinforcement learning settings (2305.12907).
Energy-based formulations: Recasting ICL as conditional energy function modeling broadens its applicability, decoupling the input and output space and accommodating more complex, unconstrained task scenarios (2406.12785).

7. Open Problems and Future Directions

Several open questions and research priorities have emerged:

Quantifying and improving efficiency: Understanding the persistent “excess risk” in ICL and developing on-the-fly adaptive mechanisms that combine demonstration-based and weight-update paradigms are viewed as crucial to eliminating inefficiency in long contexts (2502.04580).
Instruction and hypothesis class guidance: Embedding explicit instructions or hypothesis class descriptions within prompts (ICL-HCG) leads to significantly higher accuracy, especially in out-of-distribution generalization, and highlights the ability of models to generalize both across hypothesis classes and sequence lengths when provided richer context (2502.19787).
Robustness, emergent behavior, and model scaling: Studies continue to clarify under what conditions skill learning is robust to corrupted or adversarial prompts, how emergent abilities arise with model scale and pretraining diversity, and how architectural innovations (e.g., pseudo-token transformers, efficient long-range attention) might reconcile scaling properties with computation constraints (2402.02212, 2302.04931, 2406.13493).
Prompt engineering and automated context optimization: Understanding ICL as implicit knowledge distillation analytically motivates the minimization of prompt–target distribution MMD, guiding both prompt engineering strategies and the development of automated, reward-driven demonstration selection (2506.11516, 2408.07505).

Conclusion

In-context learning represents a paradigmatic shift in both the usage and theoretical understanding of adaptable machine learning systems. It intertwines pretraining distribution properties, prompt design, architectural features, and statistical learning theory. The confluence of efficient model design, prompt optimization strategies, and deeper understanding of generalization and scaling properties continues to drive research and application in ICL, with implications spanning language, vision, structured data, and beyond.