In-Context Learning Strategies Overview

• In-context learning is a paradigm where frozen models perform predictions based on provided demonstration examples without updating parameters.
• Demonstration selection strategies, such as similarity-based retrieval and conditional entropy minimization, are crucial for aligning context with model expectations.
• Advanced prompt engineering and curriculum-based training enhance the model’s reasoning and adaptability across diverse application domains.

In-context learning (ICL) is a paradigm in which a well-trained LLM or related system—without any parameter updates—is presented with a context containing a small number of input–output examples (demonstrations) and, optionally, an instruction or task description; the model then makes predictions for new queries by conditioning on this demonstration context. Formally, the ICL process is characterized by taking a frozen model $M$, a (possibly structured) context $C$ (composed of $k$ formatted demonstration pairs and instructions), and a new input $x$. The model then estimates the likelihood $P(y_j|x) = f_M(y_j, C, x)$ for candidate outputs $y_j$, producing the final output $\hat{y} = \arg\max_{y_j \in Y}P(y_j|x)$, without modifying $M$'s parameters. This approach contrasts with conventional few-shot learning, which adapts model weights via gradient-based updates. ICL has become central to evaluating and operationalizing LLMs and other foundation models across NLP, multimodal, and even structured data tasks (Dong et al., 2022).

1. Formal Frameworks and Theoretical Foundations

Recent research has formalized ICL within probabilistic and learning-theoretic frameworks. In the PAC formalism, ICL involves two phases: a pretraining phase, during which the model is trained on a mixture of latent tasks resulting in a fixed function $f_\theta$, and a test/inference phase where a prompt $p$ comprising concatenated input–output pairs is constructed (e.g., $$p := x_1 \oplus y_1 \oplus \cdots \oplus x_k \oplus y_k$$). For a new test input $x$, the model predicts by evaluating $f_\theta(p \oplus x \oplus y')$ for each $y'$, measuring performance with a loss such as

$$L_{\text{in-context}, \tilde{\mathcal{D}}} = \mathbb{E}_{(x,y)\sim\tilde{\mathcal{D}}}\left[\ell_{0-1}\left(\arg\max_{y'} f_\theta(p \oplus x \oplus y'), y\right)\right].$$

A principal result is that, under mild assumptions, the ICL process yields finite, polynomial sample complexity guarantees: exponential concentration in the likelihoods of the true latent task versus competing tasks ensures that with polynomially many context examples, the model approaches Bayes-optimality in the new task (Wies et al., 2023).

The theoretical insights indicate that downstream adaptation through ICL is primarily realized by task identification: the demonstration context serves to reveal the underlying latent task already encoded in the pretrained model, rather than instigating new learning in the conventional sense. This can be characterized as reweighting the prior over latent tasks and using the context to select the appropriate mapping (Wies et al., 2023).

2. Strategies for Demonstration Selection and Ordering

Demonstration selection is a critical determinant of ICL effectiveness and exhibits both data- and model-dependence. Multiple approaches exist:

• Similarity-based Retrieval: Candidate demonstrations are selected from the training set using (unsupervised) metrics such as embedding-based kNN or mutual information, or supervised similarity with a scoring model. Retrieval model and inference model alignment is critical; mismatches can yield suboptimal demonstration sets (Peng et al., 22 Jan 2024).
• Conditional Entropy Minimization (TopK + ConE): After TopK similarity retrieval, ConE reranks candidate demonstrations according to their contribution to reducing the model’s conditional uncertainty about the test example. The optimal group $c^*$ minimizes $H_\theta(x|c)$, yielding a demonstration set whose inclusion tightens the information gap between context and query (Peng et al., 22 Jan 2024).

Ordering is similarly influential. Curriculum-based approaches, such as In-Context Curriculum Learning (ICCL), progressively order demonstration examples from easy to hard (assessed via perplexity or human/LLM-derived difficulty metrics), guiding models along a complexity gradient (Liu et al., 16 Feb 2024). Knowledge-aware ordering strategies—such as sorting multi-answer sets by model confidence or perplexity—have been demonstrated to enhance the extraction of parametric knowledge while reducing hallucination, particularly for multi-label question answering (Lee et al., 2023).

Adaptive strategies that iteratively select demonstrations using model feedback—sequentially building a set by maximizing model uncertainty over remaining candidates—mitigate exemplar redundancy and improve coverage of diverse knowledge components, outstripping batch-selected or static non-adaptive baselines (Cai et al., 23 Dec 2024).

3. Prompt Engineering and Formatting Approaches

Prompt construction in ICL extends beyond selection and ordering to the inclusion of explicit task instructions, demonstration format, and intermediate reasoning steps:

• Instruction Augmentation: Addition of natural language instructions or task descriptions as part of the context (preceding examples) consistently improves accuracy, as explicit guidance reduces ambiguity in the modeled mapping (Lin et al., 27 Feb 2025).
• Demonstration Formatting: Reasoning-augmented prompting, such as chain-of-thought (CoT), Self-Ask, or Least-to-Most decomposition, introduces intermediate reasoning steps, enabling models to solve complex compositional or multi-step tasks with enhanced robustness (Dong et al., 2022). For knowledge-intensive domains, mixing demonstrations that are known (i.e., whose answers can be retrieved from parametric memory) with unknown ones can best harness the model’s stored knowledge and encourage educated guessing (Lee et al., 2023).

Formatting also impacts surface regularization: demonstrations standardize the label space and output format, which empirical studies show to be the dominant source of ICL improvements; demonstration content itself only marginally enhances discriminative capacity in many tasks (Long et al., 11 Apr 2024).

4. Advanced Training Strategies

Pretraining regimes and warmup strategies influence the emergence and robustness of ICL:

• Supervised Pretraining/MetaICL: Exposure to multitask demonstrations or diverse tasks during finetuning (e.g., instruction tuning, MetaICL) bridges the distributional gap between pretraining and in-context adaptation use (Dong et al., 2022).
• Self-Supervised Pretraining: Construction of pseudo-demonstrations from unlabeled corpora for masked modeling, as in PICL, imparts the mapping from context to answer in a self-supervised way (Dong et al., 2022).
• Curriculum-based Learning: Presenting subtasks in compositional curricula before composite tasks fundamentally alters internal computation, enabling zero-shot compositional generalization and robust intermediate representation learning (Lee et al., 16 Jun 2025).
• Component-wise Optimization: Analysis of weights- and context-dependent components in the transformer’s internal representations demonstrates that training plateaus can be overcome by boosting the weights component. Strategies include weights warm-up on unambiguous data, mixed training with easier/harder examples, or the addition of auxiliary losses targeting subnetwork representations (Fu et al., 2023).

Active and temporary forgetting during training can balance structural in-context learning (generalization over the structure of new or rare tokens) versus in-weights memorization (head-token performance), supporting dual process strategies (Anand et al., 28 May 2024).

5. Application Domains

ICL strategies are now deployed well beyond classic NLP settings:

• Standard NLP Tasks: Sentiment analysis, translation, information extraction, text-to-SQL, and NLI routinely leverage ICL, especially in low-resource or few-shot scenarios (Dong et al., 2022).
• Multimodal and Vision-Language (LVLMs): ICL principles have been extended to visual domains using demonstration selection frameworks that consider both image and text. Reinforcement-learning-based approaches, such as exploration–exploitation frameworks, jointly model demonstration interactions and foster diversity by treating selection as a combinatorial policy optimization problem, leading to improved generalization and VQA performance (Chen et al., 11 Jun 2025).
• Scientific Domains and Music Theory: Tailored prompt designs and worked exemplars (chain-of-thought) allow LLMs to be educated in complex categorical domains, with performance depending on the richness of in-context instructions and model familiarity with relevant formats (Pond et al., 28 Mar 2025).
• 3D Data (Point Clouds): Unified frameworks such as Point-in-Context (PIC) extend ICL to 3D data modalities, using joint sampling modules and dynamic labeling (for segmentation and registration) to enable in-context generalization without fixed label assignments (Liu et al., 18 Apr 2024).
• Imbalanced Regression and Data Engineering: In-context strategies offer a bias-reduction mechanism for regression with uneven label distributions, outperforming in-weight learning by emphasizing localized, contextually relevant exemplars retrieved via nearest-neighbor or density compensation (Nejjar et al., 28 May 2024). ICL is also used for cost-effective pseudo-labeling and automatic knowledge graph construction (Dong et al., 2022).

6. Empirical Insights and Limitations

Empirical syntheses across many tasks, models, and configurations reveal:

• Most observed ICL gains in standard benchmark tasks are due to demonstrations “regulating” the model’s label space and output format, rather than significant improvement in discriminative semantic understanding (Long et al., 11 Apr 2024).
• Retrieval-based demonstration selection (semantic similarity or embedding distance) enhances discriminative capability, but can undermine label diversity and force excessive class bias if not regularized (Long et al., 11 Apr 2024, Peng et al., 22 Jan 2024).
• ICL performance exhibits high variance both across data partitions and inference models, and can be highly sensitive to selection and ordering of demonstrations (Peng et al., 22 Jan 2024).
• Theoretical analysis demonstrates that ICL "learnability" is limited primarily by task identification; with a sufficient prompt, the model's in-context prediction converges exponentially quickly to Bayes optimal, but relies on the model’s structural competence from pretraining (Wies et al., 2023, Wurgaft et al., 21 Jun 2025).
• Complexity–loss trade-offs, as formalized in recent hierarchical Bayesian frameworks, explain the emergence and transience of different ICL strategies (generalizing versus memorizing), with the transition point scaling superlinearly with task diversity (Wurgaft et al., 21 Jun 2025).

7. Open Challenges and Future Directions

Despite rapid progress, several themes remain open:

• Robustness and Sensitivity: Reliable ICL depends on mitigating demonstration selection and ordering sensitivity. Performance can fluctuate dramatically due to prompt perturbations, and there is often a trade-off between robustness and peak accuracy (Dong et al., 2022).
• Scalability: The standard concatenation-based approach is fundamentally constrained by context window size and quadratic attention complexity; scaling to large demonstration sets remains nontrivial (Dong et al., 2022).
• Deeper Understanding of Mechanisms: While empirical and theoretical work suggests analogies with Bayesian inference, gradient descent emulation, or implicit induction heads, the precise neural mechanisms remain partially explained and are active domains of research (Dong et al., 2022, Wurgaft et al., 21 Jun 2025).
• Automated Context Generation: Progress in automatic demonstration/instruction synthesis via self-prompting and reinforcement learning—where the LLM itself selects, reranks, and optimizes context—shows promise for systematizing ICL and reducing manual overhead (Yang et al., 2023, Long et al., 14 Aug 2024).
• Generalization and Adaptation: Curriculum strategies suggest compositional generalization benefits, and meta-in-context frameworks demonstrate recursive adaptation across task sequences, but more theory and benchmarking are needed for complex and dynamic environments (Coda-Forno et al., 2023, Lee et al., 16 Jun 2025).
• Model Compression and Smaller Models: Techniques for distilling ICL abilities from large to smaller models, or enhancing parameter efficiency through optimized curriculum or ordering, are highlighted as key next steps (Dong et al., 2022, Liu et al., 16 Feb 2024).

In summary, the paper and practice of in-context learning strategies encompasses a broad taxonomy—from carefully engineered pretraining curricula, adaptive demonstration selection and ordering, prompt engineering for diverse modalities, to reinforcement learning-driven optimization—each of which leverages both model-internal capabilities and the explicit information and structure presented in the context. The field is moving toward increasingly unified, data- and model-adaptive approaches, grounded in theoretical understanding, to maximize the flexibility and generalization capacity of foundation models in few-shot settings.