ICICLE: Expanding Retrieval with In-Context Documents

Abstract: Generative retrieval (GR) maps queries directly to document identifiers (docids) using parametric knowledge, However, this design makes corpus expansion costly: adding new documents requires updating model parameters to encode new document-docid associations incurs repeated training and catastrophic forgetting of previously indexed documents. In this work, we revisit incremental GR as an in-context retrieval problem, where newly added documents are supplied as inference-time document-docid evidence. We propose ICICLE, an in-context indexing framework that performs source-aware docid generation over both parametric memory and context-provided document-docid pairs. ICICLE combines a [COPY]-based routing mechanism, preference-based calibration, and large context adaptation to distinguish context-grounded retrieval from parametric retrieval. Experiments on MS MARCO and NQ320K show that ICICLE improves retrieval of newly introduced documents while preserving seen-document retention without corpus-specific retraining. Our analysis further shows that high-shot degradation is mainly caused by routing failure, highlighting source-selection calibration as a key bottleneck for scaling in-context generative retrieval.

• The paper introduces ICICLE, a novel method that employs in-context indexing to expand generative retrieval without retraining.
• The study leverages [COPY]-based routing, preference optimization, and LoRA adaptation to balance context retrieval and parametric recall.
• Experimental results show robust unseen document retrieval and stable seen corpus performance, highlighting the method's efficiency and scalability.

ICICLE: In-Context Indexing for Dynamic Generative Retrieval Expansion

Motivation and Problem Formulation

Traditional generative retrieval (GR) models encode query–docid associations into model parameters, eliminating auxiliary index storage and complex ANN search. However, this parametric paradigm incurs significant retraining costs and catastrophic forgetting when deployed in evolving corpora, since adding new documents traditionally requires parameter updates and re-optimization. The paper introduces ICICLE, reframing incremental GR as an in-context retrieval problem where newly added documents are supplied as prompt-side evidence at inference, forgoing corpus-specific retraining. The architecture enables context-aware constraint decoding, leveraging both parametric memory and dynamically supplied document–docid pairs.

Figure 1: ICICLE architecture, illustrating context-aware constraint decoding and dynamic trie construction for unseen document integration without retraining.

Methodology

ICICLE is founded on three key components:

1. In-Context Template Learning: GR models are fine-tuned on prompts including document–docid pairs, labeled for two modes: context-dependent (gold document supplied in the candidate set, model copies from context) and query-irrelevant (answer absent, model falls back to parametric memory retrieval). Hard negatives are introduced to avoid shallow prompt exploitation. Document compression via Qwen2.5-14B-Instruct is pre-applied, optimizing prompt density by truncating candidates to 256 tokens while retaining entity-rich semantic cues.
2. [COPY]-Based Routing and Calibration: Explicit routing supervision is provided via the [COPY] token, distinguishing between context retrieval and parametric recall. When the relevant document is present, the output format is [COPY] + docid; otherwise, only the docid is generated. Standard next-token cross-entropy supervises this behavior. Direct Preference Optimization (DPO) is subsequently applied to calibrate routing activation and beam ranking, leveraging preference pairs mined from routing and ranking failures, thus optimizing for robust context usage.
3. Long-Context Adaptation with LoRA: To bridge train-test mismatch between short-shot training and large candidate sets at inference time (up to $N=100$), a lightweight LoRA-based adaptation stage is performed with title-only contexts. Parametric backbone weights are frozen, and LoRA modules are trained for efficient context extension, leveraging only document titles to compactly represent extended candidate pools.

Experimental Results

ICICLE is evaluated against both non-parametric (BM25, DPR) and parametric (DSI, DSI++, DOME, New-Doc FT, From-Scratch) baselines on MS MARCO and NQ320K. The key findings are:

• Unseen Document Retrieval: ICICLE achieves Hits@1 of 0.607 (MS MARCO) and 0.649 (NQ320K) on newly added documents, outperforming DSI++ and DOME, and approaching oracle baselines trained directly with new-doc supervision.
• Retention on Seen Corpus: Competitive Hits@1 is maintained (0.519 on MS MARCO; strongest among GR methods on NQ320K), demonstrating no catastrophic forgetting.
• Update Cost: Unlike retraining or FT baselines ($\mathcal{O}(M)$ or $\mathcal{O}(N+M)$), ICICLE expands the retrievable corpus non-parametrically by updating only prompt-side candidates and trie constraints—zero document-specific re-optimization.

Ablation Studies

Component-wise ablation corroborates the necessity of each stage:

• The [COPY] token introduces a substantial gain in unseen document retrieval (increase from 0.102 to 0.526 Hits@1).
• DPO further balances context usage and retention, optimizing routing fidelity.
• Long-context adaptation is required for scaling candidate pools to $N=100$, with only modest degradation in seen corpus performance.

Scalability, Calibration, and Error Decomposition

ICICLE's retrieval accuracy diminishes as in-context candidate count grows, correlating strongly with calibration error as measured by expected calibration error (ECE):

Figure 2: Retrieval performance as a function of in-context candidate count, $N$, for seen and unseen documents.

Figure 3: Retrieval accuracy (Hit@1) versus routing mode calibration error (ECE), showing degradation under increased candidate pool size.

Detailed error decomposition reveals that performance bottlenecks are attributable to routing failures—i.e., inability to reliably activate [COPY] for correct context-based retrieval—rather than post-routing candidate ranking. DPO’s main contribution is in improving routing recall under large context windows, while conditional retrieval accuracy given correct routing remains stable ($P(\text{hit} | [COPY]) \approx 0.943$–$1.000$).

Figure 4: Routing recall and retrieval accuracy as a function of candidate pool size, isolating the impact of DPO versus baseline [COPY].

Contextual Copying Signal Analysis

Further analysis on [COPY] token routing probability shows that when the answer is present in context, ICICLE assigns near-unity probability, demonstrating robust preference for context evidence over parametric memory. However, the probability monotonically declines as $N$ increases, correlating with retrieval accuracy degradation. Spurious [COPY] emissions and missed routing decisions are quantified, with error rate sharply rising at high shot counts ($\sim$16.9% at $N=100$).

Figure 5: [COPY] token probability and retrieval accuracy under context-dependent and all-noise scenarios, highlighting the model’s shifting preference for context evidence.

Implications, Limitations, and Future Directions

ICICLE inaugurates a scalable, efficient approach for expanding generative retrievers using purely prompt-side document–docid evidence, removing the requirement for costly retraining or model editing. The architecture is practical for rapidly-growing corpora and supports real-time document ingestion. Theoretically, ICICLE extends in-context learning paradigms to retrieval, integrating dynamic index expansion with robust routing calibration.

Practical limitations include dependence on context window size—current experiments accommodate $\mathcal{O}(M)$0 candidates, but real-world ingestion may require thousands, suggesting a need for hierarchical or cascade retrieval pipelines. Context degradation is apparent at high shot counts, analogous to LLM prompt scaling phenomena, and may be mitigated with emerging techniques such as inference-time context optimization or adaptive KV-budget allocation. Computational overhead from extended prompt prefill is non-trivial, with prefill cost scaling linearly in prompt size; optimization via caching and adaptive resource allocation is an open avenue.

Future developments should address scalable two-stage selection, robust calibration for large context pools, and efficient prompt representation strategies. The underlying [COPY]-based routing paradigm could generalize to other domains where in-context evidence dynamically alters the retrieval space.

Conclusion

ICICLE advances generative retrieval in dynamic, evolving corpora by introducing in-context indexing, router-aware constrained decoding, and preference-based calibration. The framework achieves state-of-the-art retrieval of newly ingested documents without retraining, preserves seen corpus retention, and shifts the adaptation bottleneck to inference-time prompt conditioning. Routing calibration emerges as the dominant challenge for scaling to larger candidate pools, marking it as a critical research direction for robust, scalable in-context generative retrieval.