Prompt-Driven Candidate Retrieval

• Prompt-driven candidate retrieval is a family of methods that use prompts not only to condition LLM outputs but actively determine which candidates are generated, recalled, or reranked.
• It leverages techniques like automatic prompt selection, Bayesian optimization, and retrieval-aware evidence refinement to enhance efficiency and precision.
• Empirical evaluations show improved performance across tasks such as question answering, tool selection, and cross-modal retrieval by aligning prompts with decision boundaries.

Prompt-driven candidate retrieval is a family of retrieval and optimization methods in which prompts do not merely condition a downstream LLM, but actively determine which candidates are generated, recalled, filtered, reranked, transformed, or selected before the final decision is made. In recent work, this pattern appears in black-box prompt search over discrete prompt strings (Ballew et al., 5 Oct 2025), retrieval-augmented reranking for non-knowledge-intensive tasks (Guo et al., 2023), automatic selection from a finite prompt database (Do et al., 2024), retrieval-aware evidence refinement for generation (Rodrigues et al., 2024), and retrieval-to-prompt conversion for segmentation (Chae et al., 17 Oct 2025). Taken together, these systems treat prompts as a control surface for candidate retrieval, and in some cases treat retrieval itself as a mechanism for constructing prompts.

1. Scope and recurring formulations

The literature uses prompt-driven candidate retrieval in several distinct but related senses. In one formulation, the retrieved object is itself a prompt. Automatic Prompt Selection retrieves the best prompt from a fixed synthetic prompt database by scoring input–prompt compatibility with a learned evaluator, rather than querying the downstream LLM with every candidate (Do et al., 2024). BO-LLM instead treats prompt strings as the search space of a Bayesian optimization loop, where an LLM expands seed prompts and a Gaussian Process surrogate ranks the resulting candidates (Ballew et al., 5 Oct 2025). CRPO retrieves top-$k$ reference prompts from HelpSteer2 with BM25 and uses contrastive reasoning over quality tiers or quality dimensions to synthesize an improved prompt (Lee et al., 2 Sep 2025).

In a second formulation, prompts guide retrieval or reranking of external evidence. PGRA uses a task-agnostic first-stage retriever and a prompt-guided second-stage reranker so that task-specific relevance is injected through prompts rather than separate retriever training (Guo et al., 2023). PROPEX-RAG uses prompts to extract triples, gate facts, seed graph traversal, rerank passages, and constrain evidence-first answer synthesis in GraphRAG (Sarnaik et al., 3 Nov 2025). RAG-MCP retrieves tool schemas from an external MCP index so that only the selected tool description is placed in the final prompt (Gan et al., 6 May 2025).

In a third formulation, retrieval is used to create prompts or prompt-like control signals. Memory-SAM retrieves a nearest exemplar from a small memory bank and converts mask-constrained correspondences into foreground and background point prompts for SAM2 (Chae et al., 17 Oct 2025). ICICLE supplies document–docid pairs in context and trains a generative retriever to decide whether to copy a docid from prompt-side evidence or retrieve from parametric memory (Den et al., 26 May 2026). MulVul retrieves category-level and contrastive vulnerability evidence, while cross-model prompt evolution generates Router and Detector prompts for coarse-to-fine CWE detection (Wu et al., 26 Jan 2026).

Work	Candidate retrieved	Prompt role
BO-LLM	Prompt variants	Search target and acquisition object
PGRA	Evidence passages	Task-specific reranking signal
PROPEX-RAG	Fact triples and passages	Fact gating, seeding, reranking, synthesis
RAG-MCP	MCP tool schemas	Tool selection filter
ICICLE	Context docids vs memory docids	Source-aware retrieval routing
Memory-SAM	Retrieved exemplars	Automatic point-prompt generation

2. Retrieving prompts as optimization objects

A direct formulation of prompt-driven candidate retrieval treats prompt selection as a retrieval or search problem over a finite or dynamically generated prompt set. APS formalizes the objective as learning a prompt generator $\mathtt{G}^*$ for zero-shot question answering, but operationally reduces the problem to retrieving the best prompt from a finite database $\mathcal{P}$ using a learned prompt evaluator: $$p^*=\mathtt{G}(q',c') = \arg \max_{p\in \mathcal{P}} \mathtt{E}_\theta(q',c',p).$$ Its pipeline clusters training inputs with Sentence-BERT and $K$-Means, generates $n_p$ prompts per cluster, synthesizes good/bad prompt preferences with GPT-3.5-turbo-0613, and trains facebook/opt-125m as the evaluator (Do et al., 2024). With $c=10$, $n_p=3$, and top-$k=5$ voting, APS yields 30 prompts and 3000 synthetic training samples, and reaches 81.49 on GSM8K, 100 on MultiArith, and 64.57 on AQuA (Do et al., 2024).

BO-LLM frames prompt engineering as black-box optimization because the true objective,

$f(p)=\mathbb{E}_{(x,y)\sim \mathcal{D}}\big[\mathbbm{1}\{\mathrm{LLM}_p(x)=y\}\big],$

is expensive, noisy, and non-differentiable (Ballew et al., 5 Oct 2025). Each round fits an LLM-powered Gaussian Process on prior prompt evaluations, expands the current seed prompts with error-conditioned edits, and selects a single candidate with Upper Confidence Bound,

$\mathtt{G}^*$0

where prompts are represented not by text embeddings but by their prediction vectors on a fixed control minibatch $\mathtt{G}^*$1 and compared with an RBF kernel over those vectors (Ballew et al., 5 Oct 2025). On LIAR, BO-LLM peaks at 58.5% at round 7 versus 60.0% at round 9 for ProTeGi, but reports a 32% relative improvement over its initial prompt versus 27.7% for ProTeGi; on ETHOS, it peaks at 95.75% at round 7 versus 94.5% at round 2 for ProTeGi (Ballew et al., 5 Oct 2025).

CRPO uses retrieval over an annotated prompt corpus rather than over prompt mutations. Given a query $\mathtt{G}^*$2, it retrieves

$\mathtt{G}^*$3

with BM25 and $\mathtt{G}^*$4, then performs either tiered contrastive reasoning over high-, medium-, and low-quality prompts or multi-metric reasoning over the best prompt for helpfulness, correctness, coherence, complexity, and verbosity (Lee et al., 2 Sep 2025). On HelpSteer2, CRPO-Tiered improves the average normalized score of GPT-4o from 0.6003 for plain RAG to 0.6355, and improves LLaMA 3-8B from 0.5137 for plain RAG to 0.5654 (Lee et al., 2 Sep 2025).

Contrastive Reflection extends this idea from prompt retrieval to retrieval of behavioral slices for prompt repair. The slice selector

$\mathtt{G}^*$5

retrieves error-heavy regions with nearby successes, and a Teacher LLM proposes a targeted edit that is accepted only if validation improves (Koh et al., 29 Jun 2026). On a HotpotQA retrieval-augmented QA setup, one tree-selected contrastive repair improves held-out exact-match accuracy from 51.4% to 60.4%, whereas failure-only and random-evidence variants improve less and break more previously correct examples (Koh et al., 29 Jun 2026). This suggests that in prompt retrieval, the retrieved object need not be a prompt string; it can be a contrast set of behaviors that defines where a prompt edit should act.

3. Prompt-conditioned retrieval and reranking of external evidence

A large portion of the literature uses prompts to modulate relevance functions over external candidates. PGRA is a canonical two-stage design. The first stage retrieves top-$\mathtt{G}^*$6 evidence from Wiki1M with a task-agnostic retriever using

$\mathtt{G}^*$7

and the second stage reranks those candidates with prompt-conditioned OPT-13b hidden states: $\mathtt{G}^*$8 The prompt prefix is built from manually written task descriptions and 8-shot exemplars (Guo et al., 2023). With T5-base, PGRA obtains 82.3 average versus 81.8 for FiD and 81.6 for RAG; with T5-large, it obtains 83.8 average versus 83.5 for FiD and 82.9 for RAG (Guo et al., 2023). Its analysis further attributes gains to increased pseudo-label consistency between query and retrieved evidence (Guo et al., 2023).

PROPEX-RAG makes prompting part of graph retrieval itself. It first constructs a heterogeneous symbolic graph $\mathtt{G}^*$9 from LLM-extracted fact triples

$\mathcal{P}$0

then retrieves top-$\mathcal{P}$1 fact triples by semantic similarity with text-embedding-3-large, filters them with a keep–drop prompt into $\mathcal{P}$2, extracts seed entities

$\mathcal{P}$3

and runs Personalized PageRank: $\mathcal{P}$4 Passages are then retrieved by

$\mathcal{P}$5

and fed, together with seed entities and filtered facts, into an evidence-first answer-generation prompt (Sarnaik et al., 3 Nov 2025). The system reports F1 scores of 80.7 on HotpotQA and 78.9 on 2WikiMultiHopQA, with Recall@5 scores of 97.1 and 98.1 (Sarnaik et al., 3 Nov 2025).

Meta-prompting Optimized RAG does not change retriever parameters; it optimizes the instruction used to refine retrieved contents before generation. A meta-prompt stores prior instructions and their scores, generates new refinement instructions, evaluates them on six sampled examples, and retains the best-scoring instruction in history (Rodrigues et al., 2024). On StrategyQA, Llama-2-70b-chat with plain query+contents reaches 26.12%, while refined query+contents with iterative meta-prompting reaches 34.69%; a no-iteration brute-force variant reaches 25.92%, indicating that the gain comes from the iterative optimization loop rather than from trying many instructions indiscriminately (Rodrigues et al., 2024).

Behavioral evidence can also serve as the retrieved candidate set. ProRBP retrieves user-driven behavior neighbors from daily search logs by filtering to exposure PV $\mathcal{P}$6, keeping query–item pairs with $\mathcal{P}$7, ranking from the query and item perspectives, and retaining top-$\mathcal{P}$8 neighbors with $\mathcal{P}$9 (Chen et al., 2024). These neighbors and industrial knowledge-base attributes are incorporated into progressively richer prompts,

$$p^*=\mathtt{G}(q',c') = \arg \max_{p\in \mathcal{P}} \mathtt{E}_\theta(q',c',p).$$0

whose outputs are aggregated with an exponential kernel (Chen et al., 2024). On Alipay mini-app search data, GLM-10B + ProRBP reaches AUC 0.9143, F1 0.8801, and FNR 0.0973 (Chen et al., 2024).

Tool selection is a closely related case. RAG-MCP indexes MCP function schemas, usage examples, descriptions, and parameter information in an external vector index, semantically retrieves the top-$$p^*=\mathtt{G}(q',c') = \arg \max_{p\in \mathcal{P}} \mathtt{E}_\theta(q',c',p).$$1 MCPs for a query, optionally validates them, and injects only the selected schema into the final LLM prompt (Gan et al., 6 May 2025). On MCPBench web search tasks, RAG-MCP achieves 43.13% accuracy with 1084.00 average prompt tokens, compared with 18.20% and 1646.00 for Actual Match, and 13.62% and 2133.84 for Blank Conditioning (Gan et al., 6 May 2025).

4. Prompt-based retrieval representations and dynamic indexing

Another line of work uses prompts to define the retrieval representation itself. PromptReps prompts an instruction-tuned decoder LLM to summarize a text with one important lowercase word. The last-layer hidden state of the final input token is used as a dense representation, and the next-token logits are sparsified into a weighted lexical representation via stopword filtering, SPLADE-style postprocessing, top-128 pruning, and integer quantization (Zhuang et al., 2024). Dense and sparse scores are min-max normalized and linearly interpolated with equal weights. On BEIR, PromptReps with Llama3-70B-Instruct obtains 46.19 average nDCG@10 in sparse-only form and 50.07 with hybrid+BM25, compared with 44.61 for E5-PTlarge and 43.70 for BM25 (Zhuang et al., 2024).

Prompt tuning for neural retrievers modifies the encoder rather than the scoring objective. The 2022 study on parameter-efficient prompt tuning uses P-Tuning v2 prefix prompts, updating roughly 0.1% of model parameters while keeping the backbone frozen (Tam et al., 2022). For dense retrieval, the similarity remains

$$p^*=\mathtt{G}(q',c') = \arg \max_{p\in \mathcal{P}} \mathtt{E}_\theta(q',c',p).$$2

and for ColBERT-style late interaction it remains the sum over token-level maxima (Tam et al., 2022). The reported effect is improved out-of-domain generalization and calibration: on BEIR, P-Tuning v2 improves DPR by an absolute 5.2 points on average over the fine-tuned baseline, and improves ColBERT by 2.4 points on average; on OAG-QA it beats full fine-tuning on 20/22 topics for DPR and 18/22 topics for ColBERT (Tam et al., 2022).

PromptDSI adapts prompt-based continual learning to Differentiable Search Index. After pretraining on the initial corpus $$p^*=\mathtt{G}(q',c') = \arg \max_{p\in \mathcal{P}} \mathtt{E}_\theta(q',c',p).$$3, the encoder is frozen and new corpora are indexed by updating only prompt parameters and the expanded classification head (Huynh et al., 2024). To avoid the two-forward-pass cost of standard prompt selection, PromptDSI replaces the initial pass with an intermediate-layer average-token representation and thereby achieves about a 1.9$$p^*=\mathtt{G}(q',c') = \arg \max_{p\in \mathcal{P}} \mathtt{E}_\theta(q',c',p).$$4 speedup with negligible trade-off in performance (Huynh et al., 2024). Its topic-aware prompt pool uses BERTopic-derived neural topic embeddings as fixed keys rather than learnable keys, and on NQ320k PromptDSI$$p^*=\mathtt{G}(q',c') = \arg \max_{p\in \mathcal{P}} \mathtt{E}_\theta(q',c',p).$$5 reaches 91.2% new-corpus Hits@10, while on MS MARCO it reaches Hits@1 62.3% and Hits@10 90.2% in detailed SBERT results (Huynh et al., 2024).

ICICLE moves dynamic indexing into the prompt at inference time. The model receives a candidate set

$$p^*=\mathtt{G}(q',c') = \arg \max_{p\in \mathcal{P}} \mathtt{E}_\theta(q',c',p).$$6

of document–docid pairs, and uses a source-aware target

$$p^*=\mathtt{G}(q',c') = \arg \max_{p\in \mathcal{P}} \mathtt{E}_\theta(q',c',p).$$7

If decoding begins with [COPY], the rest of the docid is constrained to a context trie; otherwise decoding proceeds in the global trie (Den et al., 26 May 2026). DPO calibration is then used to correct ranking and routing failures, and a final large-context adaptation stage trains only LoRA adapters on title-only contexts with up to $$p^*=\mathtt{G}(q',c') = \arg \max_{p\in \mathcal{P}} \mathtt{E}_\theta(q',c',p).$$8 candidates (Den et al., 26 May 2026). On MS MARCO, ICICLE reaches 0.607 Hits@1 and 0.800 Hits@10 on $$p^*=\mathtt{G}(q',c') = \arg \max_{p\in \mathcal{P}} \mathtt{E}_\theta(q',c',p).$$9, and 0.519 and 0.772 on $K$0; on NQ320K, it reaches 0.649 and 0.772 on $K$1 (Den et al., 26 May 2026).

MVR-cache addresses semantic caching as a prompt-retrieval problem. It first retrieves Top-20 coarse candidates with an HNSW index, then segments prompts at learned punctuation-based split positions, embeds each segment, and reranks with symmetric MaxSim: $K$2

$K$3

The best candidate is then passed to vCache’s correctness-aware exploit/explore policy (Noshad et al., 24 May 2026). Across established benchmarks, MVR-cache reports cache-hit improvements of up to 37% while maintaining the same correctness guarantees (Noshad et al., 24 May 2026).

5. Cross-modal and application-specific formulations

Prompt-driven candidate retrieval is not confined to text-only search. In efficient text–video retrieval, ProCLIP conditions frame selection on the query prompt rather than on query-agnostic saliency. It computes word-aware and sentence-aware cross-attention between text features and frame features,

$K$4

then fuses them with a learned gate before selecting frames (Zhang et al., 21 Jul 2025). A first-stage lightweight extractor prunes videos to the top $K$5, and CLIP reranks only that subset; the paper chooses $K$6 as the best trade-off (Zhang et al., 21 Jul 2025). On MSR-VTT, ProCLIP reports text-to-video $K$7, $K$8, and 75.3% latency reduction versus baselines (Zhang et al., 21 Jul 2025).

In visual in-context learning, LaPR argues that prompt retrieval should be label-aware rather than image-only. Each prompt $K$9 is mapped to a joint representation $n_p$0, while the query image is routed through a mixture-of-experts model to produce a query-adaptive embedding $n_p$1 and query-conditioned prompt embedding $n_p$2 (Luo et al., 4 Apr 2026). Retrieval then uses cosine similarity

$n_p$3

Experts are trained with a VICL performance-guided contrastive loss, and the router with a label-guided contrastive loss plus load balancing (Luo et al., 4 Apr 2026). LaPR reports segmentation average mIoU 41.36, detection mIoU 32.01, and colorization MSE 0.60, and in cross-fold transfer surpasses SupPR by about 14.8% and Partial2Global by about 13.9% (Luo et al., 4 Apr 2026).

Memory-SAM is an explicit retrieval-to-prompt system. A query image is encoded with DINOv3 ViT-L/16, a top-1 exemplar is retrieved from a FAISS cosine-similarity memory bank,

$n_p$4

and the retrieved mask constrains foreground and background patch matching: $n_p$5 The system keeps one foreground point and all background points above threshold, packages them as SAM2 prompts, and selects the highest-confidence mask output (Chae et al., 17 Oct 2025). On a mixed test split of 600 expert-annotated tongue images, Memory-SAM reaches mIoU 0.9863, compared with 0.8188 for FCN and 0.1839 for a detector-to-box SAM baseline (Chae et al., 17 Oct 2025).

MulVul applies the same logic to vulnerability detection. A Router agent retrieves broad evidence from a vulnerability knowledge base and predicts the top-$n_p$6 coarse CWE categories, and category-specific Detectors retrieve positives, clean negatives, and hard negatives from contrastive pools before predicting exact vulnerability types (Wu et al., 26 Jan 2026). Prompt generation is automated by cross-model prompt evolution, in which Claude Opus 4.5 mutates prompts and GPT-4o evaluates them with Recall@$n_p$7 for Router prompts and F1 for Detector prompts (Wu et al., 26 Jan 2026). Evaluated on PrimeVul with 130 CWE types, MulVul reports 34.79% Macro-F1 at type level and 50.41% at category level (Wu et al., 26 Jan 2026).

6. Empirical patterns, limitations, and adversarial manipulation

Across these systems, efficiency gains are a central motivation. BO-LLM evaluates only one selected prompt per round on the expensive minibatch $n_p$8, giving $n_p$9 expensive calls instead of $c=10$0 for multi-candidate strategies, though the paper notes that single-candidate BO-LLM is more sample-efficient but less stable than ProTeGi (Ballew et al., 5 Oct 2025). PromptDSI’s single-pass prompt selection nearly halves training and inference time, and its prompt pool is about 315 KB, compared with the much larger caches required by IncDSI (Huynh et al., 2024). RAG-MCP reduces prompt tokens while improving tool selection accuracy, and MVR-cache improves cache hit rate while preserving vCache’s correctness constraints (Gan et al., 6 May 2025, Noshad et al., 24 May 2026).

A recurring empirical theme is that better candidate retrieval often comes from aligning retrieval with the downstream decision boundary rather than with raw similarity. PGRA’s gains track pseudo-label consistency between query and retrieved evidence (Guo et al., 2023). LaPR reports that higher label consistency between query and prompt correlates positively with VICL performance, and ProCLIP shows that query-agnostic salient-frame selection can bias retrieval when the visually dominant scene is not the query-relevant one (Luo et al., 4 Apr 2026, Zhang et al., 21 Jul 2025). ICICLE finds that the main bottleneck under high shot counts is routing failure rather than retrieval after [COPY], leading the paper to identify source-selection calibration as the key bottleneck for scaling in-context generative retrieval (Den et al., 26 May 2026).

The same studies also emphasize instability and failure modes. BO-LLM reports reversed-label prompts as a failure mode, especially on LIAR, and notes that identical prompts can vary by 5–10% accuracy across runs due to LLM nondeterminism; suggested mitigations include a variance floor $c=10$1, repeated evaluation, batch-UCB, and random restarts (Ballew et al., 5 Oct 2025). Meta-prompting Optimized RAG evaluates with gold passages, so retrieval noise is explicitly removed from its reported StrategyQA results (Rodrigues et al., 2024). ProRBP depends on high-confidence behavior logs and daily freshness of those logs, and its public evidence comes from an in-house industrial dataset rather than an open benchmark with historical behavior (Chen et al., 2024).

Security work shows that prompt-driven candidate retrieval can be manipulated as well as improved. DeRAG appends a short adversarial suffix $c=10$2 to a query $c=10$3 and optimizes a hinge-loss ranking objective

$c=10$4

with Differential Evolution so that a targeted incorrect document enters the retriever’s top-$c=10$5 (Wang et al., 20 Jul 2025). The attack succeeds against both dense and sparse retrievers, often with suffixes of $c=10$6 tokens, and the reported detector performance is near chance, with AUROC 0.2023 at strict target FPR $c=10$7 and AUPRC 0.4665 (Wang et al., 20 Jul 2025). This places prompt-driven candidate retrieval at a dual frontier: it is a mechanism for sample-efficient and task-specific retrieval control, but it is also a control surface that can be attacked.

The aggregate picture is that prompt-driven candidate retrieval replaces one-size-fits-all retrieval objectives with prompt-conditioned selection rules, prompt-side routing decisions, or retrieval-to-prompt conversions. This suggests a common design principle across otherwise dissimilar systems: retrieval quality improves when prompts are used not only to interpret retrieved candidates, but to determine which candidates should count as relevant evidence, useful instructions, valid tools, plausible docids, or correct demonstrative exemplars in the first place.