Retrieval-Based In-Context Learning

• Retrieval-based in-context learning is a method that dynamically selects demonstrations from a pool to adapt large language models to task-specific requirements.
• It employs diverse scoring techniques—such as BM25, dense embeddings, and LM-aware methods—to optimize relevance, diversity, and structure in demonstration selection.
• Empirical studies show that these strategies enhance task accuracy and label discrimination across domains like semantic parsing, logical reasoning, and relation extraction.

Retrieval-based in-context learning is a class of methods in which the demonstrations shown to a LLM are selected dynamically from an example pool rather than fixed in advance. In the standard formulation, a model answers a new input $x$ from a prompt containing $k$ retrieved demonstrations and the query itself, $$\hat{y} \sim P_{\mathrm{LM}}(y \mid d_1,\dots,d_k,x)$$. The central problem is therefore not merely nearest-neighbor search, but the construction of a small, query-specific context that supplies the right label space, format, reasoning pattern, and task knowledge while avoiding redundancy and conflict with what the model already knows (Kweon et al., 15 Sep 2025, Long et al., 2024).

1. Formalization and scope

Retrieval-based ICL assumes a pool $D=\{(x_i,y_i)\}_{i=1}^N$ and a retriever that, for each query, selects a small subset of demonstrations from that pool. In Dr.ICL this appears as $Demo_{x_q}=Retriever(x_q,D_{\text{train}},n)$, after which the frozen model predicts $p_\theta(y_q \mid Demo_{x_q},x_q)$ (Luo et al., 2023). The demonstrations may be plain input–label pairs, but on reasoning tasks they may also include chain-of-thought traces, so that retrieval operates over question–reasoning–answer triples rather than over bare labels (Luo et al., 2023).

Most work in the area retrieves labeled task examples rather than free-form evidence passages. Long et al. explicitly distinguish this regime from classical retrieval-augmented generation: retrieval-based ICL uses retrieved input–output pairs as a learning signal for task adaptation, whereas classical RAG retrieves external documents to supply factual content (Long et al., 2024). Later work narrows that boundary rather than erasing it. RAVEN studies retrieval-augmented encoder–decoder models under in-context prompting, and recent theory on RAG interprets retrieved evidence as a source of in-context optimization rather than static context (Huang et al., 2023, Li et al., 25 May 2026).

The scope of retrieval-based ICL now extends well beyond standard text classification. The same basic pattern appears in semantic parsing, open-domain QA, relation extraction, math and logical reasoning, misinformation detection, unsupervised domain adaptation, tabular learning, and even the training of retrieval models that themselves consume in-context examples (Rubin et al., 2021, Han et al., 2024, Liu et al., 2024, Wen et al., 5 Feb 2025, Tejaswi et al., 2024).

2. Retrieval signals and scoring objectives

The earliest and still commonest family of methods ranks demonstrations by surface or embedding similarity. TopicK summarizes the standard options as lexical BM25, dense embedding $k$-NN, and diversity-oriented variants such as DPP-style selection, BERTScore-based coverage, or majority-vote heuristics (Kweon et al., 15 Sep 2025). Dr.ICL shows that even simple BM25 can outperform random demonstration selection, while dense retrievers such as GTR provide a stronger semantic baseline and can be specialized further with task-specific supervision (Luo et al., 2023).

A second family scores candidates with the LLM itself. In EPR, a candidate example $\bar e_l$ is evaluated by the probability it induces for the gold target, $$s(\bar e_l)=\operatorname{Prob}_{\hat g}(y \mid \bar e_l,x)$$, and those LM-defined scores are then used to train a dense prompt retriever (Rubin et al., 2021). Related lines include MDL, MDR, and ConE, which rank examples by entropy reduction, predicted error, or input likelihood, thereby making retrieval directly LLM-aware (Kweon et al., 15 Sep 2025). This often improves alignment with model behavior, but it is also expensive because candidate evaluation can require a separate model call per query–demo pair, and it is awkward when closed-source APIs do not expose log-probabilities (Kweon et al., 15 Sep 2025).

A third family treats retrieval as a multi-task ranking problem with explicit feedback calibration. TDR argues that, in mixed-task pools, naive dense retrieval can confuse tasks with similar inputs but incompatible outputs, and that directly fitting retriever score distributions to LLM score distributions is brittle. It instead optimizes correlation between retrieval probabilities and fine-grained LLM feedback while explicitly decoupling same-task refinement from cross-task suppression (Chen et al., 24 Jul 2025).

Representative families are summarized below.

Family	Representative methods	Primary signal
Similarity-based	BM25, TopK, GTR, SimCSE, CEIL, Set-BSR	Lexical overlap, dense similarity, DPP, token-level coverage
LLM-aware / probability-based	EPR, Demo-GTR, MDL, MDR, ConE, TDR	LM likelihood, entropy, error prediction, fine-grained feedback
Set-wise / combinatorial	TopicK, InSQuAD	Topic coverage, SMI-based quality and diversity
Structure-aware	AMR-RE, RGER	AMR graphs, reasoning graphs, graph kernels
Domain- or modality-specific	Adapt in Contexts, RAEmoLLM, TabRAG	Target-domain contexts, affective embeddings, tabular row retrieval
Architecture-level extensions	RAVEN, RARe	Retrieval-augmented encoder–decoders, in-context examples for retrievers

An important corrective to a common misconception comes from Long et al. Random demonstrations do not mainly improve the model’s discriminative knowledge; they mainly regulate label space and label format, making outputs fall into the expected verbalization regime. Retrieval helps chiefly on the discrimination component: semantically similar examples make the model better at choosing the right label among allowed alternatives, but they also introduce a trade-off with label diversity when all retrieved examples collapse onto one class (Long et al., 2024).

3. Set-wise coverage, diversity, and task awareness

A major development in retrieval-based ICL is the move from independent candidate ranking to set-wise optimization. TopicK makes this transition explicit by modeling three quantities: required topics for the input $t_x$, covered topics for each demonstration $k$0, and model topical knowledge $k$1. Its relevance score,

$k$2

prioritizes demonstrations that cover topics required by the query and under-mastered by the model (Kweon et al., 15 Sep 2025). TopicK then updates cumulative topic coverage after each selection step so that later demonstrations are rewarded for introducing newly covered required topics rather than repeating what earlier examples already supplied. This turns demonstration retrieval into a budgeted coverage problem over model-weak knowledge, not a sequence of isolated nearest-neighbor decisions.

InSQuAD pushes the same idea into a more explicitly combinatorial framework. It casts both exemplar annotation and exemplar retrieval as targeted selection under Submodular Mutual Information, so that the selected set is optimized jointly for relevance to the query and internal diversity (Nanda et al., 28 Aug 2025). Because the objective is submodular, greedy optimization inherits the usual diminishing-returns behavior: near-duplicates rapidly lose marginal value. InSQuAD further trains the retriever with a likelihood-style objective defined over positive and distractor sets rather than over isolated pairs, and supplements this with paraphrase-based supervision so that the embedding space learns both coverage and non-redundancy (Nanda et al., 28 Aug 2025).

Task awareness enters at the same level of granularity. TDR addresses multi-task example pools where textual similarity alone may retrieve examples with incompatible answer formats or label spaces. Its task-decoupled losses separate cross-task rejection from same-task ranking, while its feedback term uses LLM token log-likelihoods of the gold answer as a supervision signal for retrieval probabilities (Chen et al., 24 Jul 2025). Across these methods, the underlying shift is the same: exemplar retrieval is increasingly treated as a constrained selection problem over sets, with relevance, diversity, and task compatibility optimized jointly rather than sequentially.

4. Structure-aware and domain-aware retrieval

For some tasks, semantic proximity at the sentence level is an inadequate proxy for demonstration usefulness because the operative similarity lies in latent structure. Relation extraction is the clearest case. AMR-RE argues that standard ICL retrieval for RE over-emphasizes language similarity and overlooks entity relations, so it retrieves examples using Abstract Meaning Representation graphs rather than only sentence embeddings. Its graph representations focus on the shortest path between subject and object entities, with local and global variants that prove useful under different datasets and supervision regimes (Han et al., 2024).

Reasoning tasks motivate a parallel move from sentence embeddings to graph comparison. RGER first asks an LLM for an initial response, then converts the resulting intermediate reasoning into a graph and reranks candidate exemplars with graph kernels. The retrieved examples are therefore matched not just by semantic topic but by the structure of the solution process itself, which is especially relevant for multi-step math and logical reasoning (Lin et al., 2024).

Other work substitutes domain-specific latent signals for general semantic similarity. RAEmoLLM constructs an affective retrieval database using embeddings from an emotional LLM and retrieves source-domain misinformation examples whose emotion and sentiment profiles are close to the target input. The retrieved text–label pairs then become few-shot demonstrations for a downstream LLM, yielding a specifically affect-based form of cross-domain retrieval (Liu et al., 2024). Adapt in Contexts studies unsupervised domain adaptation under ICL and retrieves unlabeled target-domain examples as cross-domain contexts for source-labeled instances, jointly optimizing task supervision on the source and language modeling on the target contexts (Long et al., 2023). In tabular learning, scalable TabICL uses a customized retrieval module together with retrieval-guided instruction tuning, so that the LLM conditions on a bounded set of retrieved rows rather than on the full training table (Wen et al., 5 Feb 2025).

Taken together, these methods generalize the unit of retrieval. The retrieved object need not be “the most similar text example” in a generic embedding space. It may instead be the example with the right topic coverage, AMR path structure, reasoning graph, affective profile, domain context, or schema-local neighborhood.

5. Mechanistic interpretations and architectural extensions

Mechanistic work has increasingly recast retrieval-based ICL as an instance of memory access and implicit optimization rather than literal on-the-fly parameter learning. One influential line models the LLM as a large associative memory and treats in-context exemplars as clues that trigger Hopfield-style retrieval. In that account, attention implements content-addressable memory lookup, and exemplar selection can be analyzed through retrieval error terms that depend on how well context patterns match the target and how much interference they introduce (Zhao, 2023).

Head-level analyses provide a more local circuit picture. The Atlas of In-Context Learning identifies in-context heads, parametric heads, task heads, and retrieval heads in retrieval-augmented question answering. Task heads are associated with comprehension of the intensional frame of the question, retrieval heads with copying answer tokens from retrieved context, and parametric heads with internally stored knowledge. The same study shows that these head types can be ablated, patched with function vectors, and used to trace whether a generated answer came from context or from parametric memory (Kahardipraja et al., 21 May 2025).

Architectural work has also broadened what counts as retrieval-based ICL. RAVEN shows that retrieval-augmented encoder–decoder models can be made substantially stronger in ICL by aligning pretraining with prompting through retrieval-augmented masked language modeling followed by retrieval-augmented prefix language modeling, and then using Fusion-in-Context Learning to distribute larger numbers of in-context examples across Fusion-in-Decoder inputs without extra training (Huang et al., 2023). RARe asks the converse question: whether retrievers themselves can benefit from in-context examples. It finds that naively prepending query–document demonstrations to a retrieval query does not work out of the box, and proposes fine-tuning with in-context examples so that the retriever learns to use them, yielding stronger out-of-domain generalization (Tejaswi et al., 2024). A more recent theoretical extension studies RAG as in-context optimization, showing that one linear self-attention layer can implement one gradient-descent step on a unified linearized RAG objective and using that result to motivate forward-only, context-conditioned updates to a frozen generator-side evidence-use interface (Li et al., 25 May 2026).

These interpretations converge on a common point: retrieval-based ICL is not only about bringing external examples into the prompt. It is also about how those examples reconfigure the model’s internal evidence-use pathways.

6. Empirical profile, limitations, and current directions

Across the literature, several results have become representative benchmarks for the area.

Paper	Setting	Reported outcome
TopicK (Kweon et al., 15 Sep 2025)	6 datasets, open- and closed-source LLMs	Average relative improvement $k$3; up to $k$4 accuracy over ConE; ConE is $k$5 slower on QNLI
AMR-RE (Han et al., 2024)	Relation extraction	Unsupervised average 40.63 vs 37.67 best baseline; supervised average 75.63 vs 73.74 for GPT-RE_FT
Dr.ICL (Luo et al., 2023)	One-shot prompt retrieval	Demo-GTR improves over GTR by $k$6 on average
TDR (Chen et al., 24 Jul 2025)	30 NLP tasks	68.3 average vs 66.5 for LLM-R; cross-task retrieval rate drops from 6.67% to 2.23%
RAEmoLLM (Liu et al., 2024)	Cross-domain misinformation	Highest increases of 15.64%, 31.18%, and 15.73% on three benchmarks
RARe (Tejaswi et al., 2024)	BeIR and RAR-b retrieval	Up to $k$7 nDCG
InSQuAD (Nanda et al., 28 Aug 2025)	9 ICL benchmarks	Up to 21.6% on classification, 16.4% on multi-choice, and 7% on generation
RAVEN (Huang et al., 2023)	Retrieval-augmented encoder–decoder ICL	Raven 11B + FiCL $k$8 reaches 33.5 EM on NQ and 67.3 EM on TriviaQA

These results suggest several stable empirical regularities. First, retrieval almost always dominates random or fixed demonstration choice once tasks become nontrivial. Dr.ICL shows this even for BM25, and Long et al. show that semantically similar retrieval principally improves discrimination rather than merely label formatting (Luo et al., 2023, Long et al., 2024). Second, the largest gains tend to appear when surface similarity is a poor proxy for task needs: specialized knowledge domains, explicit relational structure, multi-step reasoning, or cross-domain shift. Third, a substantial part of current progress comes from changing the retrieval objective itself—coverage, diversity, task consistency, or structure—not from changing the backbone LLM.

The limitations are also becoming clearer. LM-aware scorers such as MDL, MDR, and ConE can require many per-candidate model calls and may be unusable when closed-source APIs do not expose reliable token probabilities; TopicK reduces that online cost but depends on topic mining quality and on a flat topic vocabulary (Kweon et al., 15 Sep 2025). Structure-aware retrieval inherits parser and graph-construction error: AMR-RE depends on AMR quality and remains sentence-level, while RGER notes that reasoning-graph acquisition still benefits from human intervention (Han et al., 2024, Lin et al., 2024). RARe improves retrieval quality but increases latency because in-context examples lengthen the query representation, and TDR notes that strict task decoupling can over-penalize cross-task examples that are in fact useful (Tejaswi et al., 2024, Chen et al., 24 Jul 2025). At the theoretical end, the exact equivalence between in-context retrieval and gradient descent is confined to linear regimes; under nonlinear architectures it becomes feature-distribution dependent (Li et al., 25 May 2026).

Current directions therefore cluster around a few themes already visible in the literature: hierarchical or graph-structured knowledge units instead of flat labels, dynamic or multi-step retrieval during generation, stronger retriever–generator co-design, broader modality coverage, and more faithful provenance analysis of whether a prediction came from demonstrations, retrieved evidence, or parametric memory (Kweon et al., 15 Sep 2025, Kahardipraja et al., 21 May 2025, Long et al., 2023). The field has moved well beyond “retrieve nearest neighbors and prepend them.” It increasingly treats retrieval as a structured control problem over what information the prompt should contain, how that information should be distributed across examples, and how the model should use it once it is there.