Instruction-Tool Retrieval (ITR)

• Instruction-Tool Retrieval (ITR) is a framework where retrieval relevance is conditioned on explicit natural language instructions tailored to user-specific requirements.
• It dynamically adapts the retrieved content—such as documents, reasoning scaffolds, or tool descriptions—based on the provided instruction alongside traditional queries.
• ITR employs advanced architectures and training signals with metrics like nDCG and MRR to improve instruction compliance, efficiency, and robustness in various domains.

Instruction-Tool Retrieval (ITR) denotes retrieval regimes in which relevance is conditioned on an explicit natural-language instruction rather than on a query alone. In its most canonical information-retrieval form, the objective is written as $p(d \mid q, I)$, where $q$ is a query, $I$ is an instance-wise instruction, and $d$ is a document or passage (Oh et al., 2024). Recent work extends the same conditioning principle beyond document ranking to inference-time retrieval of structured reasoning procedures for small LLMs and to dynamic exposure of tool subsets and system-prompt fragments for agentic LLMs (Alkiek et al., 15 Oct 2025, Franko, 1 Dec 2025). Across these settings, ITR replaces a fixed notion of relevance with a conditional one: the same query may require different retrieved objects when the instruction changes.

1. Conceptual scope and formal foundations

The formal core of ITR appears in instruction-following retrieval work that contrasts standard retrieval, which models $p(d \mid q)$, with instruction-conditioned retrieval, which models $p(d \mid q, I)$ (Oh et al., 2024). In the encoder-based formulation summarized for InstructIR, the input is the concatenation of query and instruction, $h=\mathrm{Encoder}_1([q;I])$, while documents are encoded as $u=\mathrm{Encoder}_2(d)$ and scored by cosine or dot-product similarity. This framing is intended to let a retriever adapt its ranking when the same query is paired with different user-specific constraints such as background, search goal, preferred sources, or reading level.

Earlier large-scale work on retrieval with instructions established the same basic idea in multitask form. TART defines a scoring function $s:(t,q,d)\mapsto\mathbb{R}$, with $t$ denoting the instruction, and trains a retriever on BERRI, a collection of approximately 40 retrieval datasets with instructions (Asai et al., 2022). In that formulation, relevance is no longer a property of the query-document pair alone; it is conditioned on the task description, domain, and retrieval unit encoded in the instruction.

In the subsequent literature, the term has been used for several closely related settings.

Setting	Retrieved object	Representative papers
Instruction-following IR	Documents or passages	(Asai et al., 2022, Oh et al., 2024)
Inference-time reasoning retrieval	Structured instructions	(Alkiek et al., 15 Oct 2025)
Agentic instruction/tool exposure	Prompt fragments and tools	(Franko, 1 Dec 2025, Xu et al., 2024)

This terminological spread suggests that ITR is best understood as a retrieval paradigm rather than a single architecture. What varies across instantiations is the external object being retrieved: a passage, a reasoning scaffold, a tool description, or a runtime subset of a larger control interface.

2. Retriever architectures and training signals

Instruction-conditioned retrievers have evolved from simple concatenation-based bi-encoders to architectures that isolate instruction processing, inject hard instruction negatives, or synthesize paired supervision that forces the model to attend to the instruction itself. TART-dual encodes $q$0 and $q$1 with a shared Transformer and scores them by inner product, while TART-full uses a joint encoder over $q$2 and reranks candidates produced by a fast bi-encoder (Asai et al., 2022). On BEIR, TART-full reaches an average nDCG@10 of 44.8, and instruction ablation reduces this to 36.2 when instructions are removed at both training and test time; on the X²-Retrieval setup, the average nDCG@10 falls from 59.1 to 49.9 or 48.3 under analogous ablations. The same work reports that removing hard and instruction-unfollowing negatives reduces BEIR average nDCG@10 from 44.8 to 41.0.

I3 generalizes instruction conditioning with a parameter-isolated “pluggable introspector” that reasons jointly over query and instruction while keeping the base dual-encoder frozen (Pan et al., 2023). Its scoring function is $q$3, and training combines a contrastive intent retrieval loss with an instruction-distinction loss, $q$4 with $q$5. On the 14 out-of-domain BEIR tasks plus MS MARCO, I3 reports an average nDCG@10 of 52.9, compared with 50.5 for the COCO-DR backbone and 51.1 for SGPT 5.8B. Removing the introspector reduces the average to 48.9, while the pruned introspector sustains 1,150 queries per second relative to 700 without pruning and 1,600 for the backbone.

Promptriever moves instruction following to the level of per-query prompting. It concatenates instruction and query as [Instruction]<|sep|>[Query], trains on approximately 0.5 million instruction-augmented MS MARCO instances, and introduces “instruction negatives” that are topically relevant but violate the instruction (Weller et al., 2024). The reported gains are concentrated on instruction sensitivity rather than on conventional web retrieval: p-MRR on FollowIR rises by +14.3 points, Robustness@10 on InstructIR increases from 50.2 to 63.1, and a small pool of generic prompts yields an average +1.4 nDCG@10 gain on BEIR without retraining.

More recent training corpora make the instruction-query interaction itself the object of supervision. InF-IR expands MS MARCO pairs into 38,759 positive $q$6 triplets and adds 77,518 hard negatives by poisoning either the instruction or the query, with validation by o3-mini (Zhuang et al., 27 May 2025). The associated InF-Embed model uses instruction-query attention and a multivariate contrastive objective; the paper reports that embedding-only backbones gain +6.9 to +9.4 p-MRR and that InF-Embed surpasses competitive baselines by 8.1% in p-MRR across five instruction-based retrieval benchmarks. Dual-View Training further sharpens instruction sensitivity by “polarity reversal”: given a query, an instruction, a positive document, and an instruction-violating hard negative, an LLM synthesizes a complementary instruction under which the original negative becomes positive (Zeng et al., 20 Apr 2026). On a 305M-parameter encoder, FollowIR p-MRR rises from 5.21 to 7.57 on instruct-only data and from 5.27 to 8.30 when all data are dual-view.

3. Inference-time retrieval of reasoning instructions

A distinct but closely related instantiation appears in “Big Reasoning with Small Models: Instruction Retrieval at Inference Time,” where a small LLM retrieves structured reasoning procedures instead of generating them from scratch (Alkiek et al., 15 Oct 2025). The method builds an Instruction Corpus by clustering similar training questions, generating cluster-level instructions with GPT-5, and retrieving the most relevant instructions at inference time without any additional fine-tuning. The retrieved artifact is not a document passage in the usual retrieval-augmented generation sense; it is a reusable reasoning scaffold consisting of “## Background Knowledge” and “## Reasoning Steps.”

Instruction corpus construction proceeds in three phases. First, every training question is embedded using text-embedding-3-large, and agglomerative clustering with average linkage is applied under cosine distance,

$q$7

The default setting chooses the finest partition, with thresholds $q$8 for MedQA, $q$9 for MMLU Professional Law, and $I$0 for MathQA, yielding average cluster sizes of 1–1.3 examples and maxima of about 12. Second, up to five representative questions per cluster are fed to GPT-5 via a standardized template, which produces the two-part instruction. Third, each instruction is scored by GPT-5 on five-point Likert scales for Knowledge Comprehensiveness, Knowledge Relevance, Reasoning Accuracy, Reasoning Relevance, and Clarity; mean scores exceed 4.7/5 on all axes.

At inference time, the same embedding model encodes the test question and each instruction. Relevance is computed by cosine similarity,

$I$1

or equivalently $I$2 after $I$3 normalization. The system retrieves the top-$I$4 instructions, with $I$5 by default, and backs off to smaller $I$6 if the concatenated prompt would exceed the model context window. The inference procedure is explicitly training-free: the retrieved instructions are simply prepended to the user prompt, and decoding uses temperature $I$7 and top-$I$8.

The reported results cover MedQA, MMLU Professional Law, and MathQA with models from 3B to 14B parameters. With High-School Concise instructions, instruction retrieval yields +9.4 percentage points on MedQA, +7.9 on MMLU Law, and +5.1 on MathQA (Alkiek et al., 15 Oct 2025). The paper provides concrete examples such as Gemma-2 9B IT on MedQA, which rises from 37% to 54%, and Mistral 7B Instruct on MMLU Law, which rises from 22% to 38%. On MedQA and MMLU Law, 14B SLMs with instruction retrieval surpass GPT-4o zero-shot accuracy.

The ablations identify several recurring regularities. Concise instructions outperform both baseline and verbose variants by approximately +5 percentage points on average, and the paper characterizes verbosity as an “irrelevance tax”: each doubling of instruction length costs about –8 percentage points in accuracy per log-token. Audience framing has a smaller and inconsistent effect, below $I$9 percentage point. On MedQA, knowledge-only instructions yield +6 percentage points over zero-shot, reasoning-only instructions yield +4, and the full two-part instruction yields +9, which the paper interprets as complementarity between background knowledge and explicit procedure. Varying cluster granularity from an average of approximately 1.2 up to 6.2 questions has negligible effect within $d$0 percentage point, and models below about 3B parameters often show mixed or negative effects.

4. Tool retrieval and dynamic tool exposure

In tool-augmented systems, ITR addresses a different bottleneck: selecting a small, relevant subset from a large and irregularly updated tool library. One line of work formulates tool retrieval as a dense retrieval problem over tool descriptions. “Enhancing Tool Retrieval with Iterative Feedback from LLMs” defines the task as retrieving the top-$d$1 tools for a user instruction $d$2 from a tool library $d$3 with scoring $d$4 and top-$d$5 selection $d$6 (Xu et al., 2024). Its distinctive feature is multi-round feedback from an LLM critic using comprehension, assessment, and refinement prompts. On TR-bench, the method reports in-domain NDCG@1/3/5 of 88.53 / 87.00 / 88.83 versus 75.73 / 83.19 / 83.06 for ToolRetriever, and out-of-domain gains on T-Eval from 82.10 / 72.03 / 74.15 to 84.45 / 73.31 / 74.45.

A more operational view appears in “Dynamic System Instructions and Tool Exposure for Efficient Agentic LLMs,” which also uses the label ITR for a RAG-style runtime system that retrieves only the relevant system-prompt fragments and the minimal necessary subset of tools at each step (Franko, 1 Dec 2025). System prompt fragments are pre-chunked into 200–600 token units, tool documents into 150–800 token units, and retrieval combines dense similarity, BM25, and cross-encoder reranking through

$d$7

with an analogous score for tools. Selection is budget-aware: under a per-step token budget $d$8, the system solves a knapsack-style objective using marginal-gain proxies and, in practice, a greedy score-per-token rule. The runtime prompt includes a “SAFETY_OVERLAY,” selected instruction fragments, selected tool schemas plus exemplars, and a routing note that limits the model to exposed tools unless it declares “tool discovery.” On the controlled benchmark, ITR with $d$9 and $p(d \mid q)$0 reduces tokens per step from 30,000 to 1,500, improves Tools-Correct from 62% to 82%, raises API-Success from 64% to 79% on the T2 setting, and reduces cost per episode from \$p(d \mid q)$10.86.

A separate problem is instruction vagueness. “Tool Retrieval Bridge” introduces VGToolBench, derived from ToolBench’s 16,464 APIs and about 200K instruction-API pairs, to simulate “human vague instructions” by stripping away API names and technical terms (Chen et al., 9 Apr 2026). The bridge model rewrites a vague instruction $p(d \mid q)$2 into a more specific pseudo-instruction $p(d \mid q)$3, first by supervised fine-tuning on $p(d \mid q)$4 pairs and then by Direct Preference Optimization using retrieval reward defined as the average of NDCG@5 and NDCG@10. On the I2 subset, BM25 drops from 18.70 on ToolBench to 9.27 on VGToolBench and rises to 19.59 with TRB, while ToolRetriever drops from 77.83 to 53.96 and rises to 65.23 with TRB. The same paper reports that vague instructions can cause performance drops of up to 50% relative.

Large-scale synthetic tool ecosystems extend these ideas to noisy, domain-specific environments. FinToolSyn builds a repository of 43,066 financial tools and synthesizes more than 148K dialogue instances with dynamic retrieval, using vector retrieval, graph-enhanced retrieval, and globally controlled dialogue planning (Huang et al., 25 Mar 2026). Its benchmark, FinToolBench, contains 843 human-verified instances, and fine-tuned models achieve up to a 21.06% relative improvement in overall CB-HWS. In a different domain, the RAIT framework for process-engineering calculations retrieves tool protocols and documentation chunks with BGE embeddings, reranks them, and injects the top-$p(d \mid q)$5 tool documents into a ReAct-style prompt; removing the external-knowledge retrieval pipeline reduces Recall@K by about 20–25 points and BLEU by about 30 points (Sakhinana et al., 2024).

5. Benchmarks and metrics for instruction following

A central finding of the ITR literature is that standard topical relevance metrics are often insufficient to measure whether instructions were actually followed. InstructIR addresses this by pairing 1,267 queries with multiple diverse instructions, yielding 9,906 instances over a 16,072-document pool, and by introducing Robustness@$p(d \mid q)$6, defined as the average worst-case nDCG@$p(d \mid q)$7 across all instructions attached to the same query (Oh et al., 2024). This metric targets instruction sensitivity directly: a retriever that succeeds only for some variants of a query’s instructions is penalized even if its average nDCG remains high.

InfoSearch extends the evaluation target beyond content relevance to six document-level attributes—Audience, Keyword, Format, Language, Length, and Source—and introduces Strict Instruction Compliance Ratio (SICR) and Weighted Instruction Sensitivity Evaluation (WISE) (Zhou et al., 2024). The benchmark contains 600 core queries, 1,598 instructed queries, 1,598 reversed queries, and 6,392 documents. The highest instruction compliance reported in that study is for GPT-4o, with WISE = 33.5 and SICR = 32.1%, while the top dense model, GritLM, still has negative WISE at –11.1 despite 6.9% SICR. The paper’s explicit conclusion is that most models still fall short of instruction compliance.

Expert-domain retrieval introduces additional difficulty because the instruction itself may encode specialized constraints. IFIR contains 2,426 examples spanning finance, law, healthcare, and science literature, with three levels of instruction complexity per domain (Song et al., 6 Mar 2025). Its LLM-based evaluation metric, INSTFOL@$p(d \mid q)$8, measures the gain in instruction alignment when explicit instructions are added to the query and correlates with expert human relevance judgments at Pearson 0.704 on 400 query-passage pairs. Reported averages include BM25 at nDCG = 0.34 and INSTFOL = +0.50, E5-mistral-7B-instruct at nDCG = 0.48 and INSTFOL = +3.05%, GritLM-7B at nDCG = 0.51 and INSTFOL = +3.06%, and Promptriever-7B at the strongest INSTFOL of +8.00% but lower nDCG of 0.32.

Multilingual instruction following is evaluated by mFollowIR, which builds on TREC NeuCLIR narratives in Russian, Chinese, and Persian and uses paired narrative edits to isolate instruction-following behavior (Weller et al., 31 Jan 2025). The main instruction metric is p-MRR, which measures whether documents that should become non-relevant under an altered instruction actually move downward in the ranking. Promptriever-Llama3.1 illustrates the gap between cross-lingual and multilingual settings: in English-to-non-English retrieval it reaches nDCG@20 = 0.521 and p-MRR = 10.4%, whereas in same-language multilingual retrieval it reaches nDCG@20 = 0.541 but p-MRR = 5.2%.

Tool-retrieval and agentic settings add yet another layer of evaluation. Tool retrieval papers frequently report NDCG@$p(d \mid q)$9, Recall@$p(d \mid q, I)$0, or MRR, while agentic tool-use benchmarks also track episode-level success, cost, latency, miss-rate, parameter consistency, or composite scores such as CB-HWS (Xu et al., 2024, Huang et al., 25 Mar 2026). This divergence in metrics reflects a practical distinction: in passage retrieval, the instruction usually modifies document relevance; in tool retrieval, it may also alter downstream execution success and runtime efficiency.

Benchmark	Focus	Signature metric(s)
InstructIR	Instance-wise instruction sensitivity	Robustness@10, nDCG@10
InfoSearch	Document-level attribute compliance	SICR, WISE
IFIR	Expert-domain instruction following	INSTFOL@20, nDCG@20
mFollowIR	Multilingual instruction following	p-MRR, nDCG@20
TR-bench / FinToolBench	Tool retrieval and tool use	NDCG@k, Recall@K, CB-HWS

6. Recurring findings, misconceptions, and open problems

A recurring misconception in the literature is that attaching a task description to a retriever is sufficient for robust instruction following. InstructIR directly challenges this view: INSTRUCTOR-base, INSTRUCTOR-large, and INSTRUCTOR-XL underperform their non-instruction-tuned GTR counterparts, and the paper attributes this to overfitting to relatively uniform task-description instructions (Oh et al., 2024). The same benchmark reports strong instruction-order sensitivity, with INSTRUCTOR gaining more than 30 points in nDCG if query and instruction are swapped, as well as substantial paraphrase sensitivity for lexical systems.

Another recurring misconception is that strong nDCG or MRR implies strong instruction compliance. InfoSearch shows that models can preserve topical relevance while ignoring user constraints, and even the highest-scoring model in that study leaves most instances outside strict compliance (Zhou et al., 2024). IFIR makes the same point in expert domains: all models degrade from Level 1 to Level 2 and Level 3 instructions, longer legal instructions can exceed 1,024 tokens, and highly customized biomedical constraints remain difficult even for frontier retrievers (Song et al., 6 Mar 2025).

Model scale and instruction style also interact in nontrivial ways. The instruction-retrieval framework for small LLMs reports that gains emerge consistently above roughly 3B parameters, while smaller models often cannot reliably exploit external scaffolds (Alkiek et al., 15 Oct 2025). The same work argues for concision over verbosity and recommends approximately 500–750 tokens for retrieved instructions, with versions longer than 1.3K tending to dilute relevance. In multilingual retrieval, mFollowIR reports that English-based instruction tuning transfers better in cross-lingual than in fully multilingual settings, suggesting that non-English instruction corpora remain a bottleneck (Weller et al., 31 Jan 2025).

The literature also identifies real-world vagueness and domain specialization as persistent failure modes. VGToolBench shows that vague instructions materially damage tool retrieval and that explicit rewriting can restore much of the lost performance (Chen et al., 9 Apr 2026). IFIR and FinToolSyn indicate that finance, law, healthcare, and large financial tool spaces require domain-specific constraints, richer synthetic data, or dynamic retrieval regimes rather than generic instruction tuning alone (Song et al., 6 Mar 2025, Huang et al., 25 Mar 2026).

Several forward directions recur across papers. These include larger and more domain-diverse instruction-aware corpora, RLHF or related preference-based training for retrieval, stronger hard-negative construction, hybrid lexical-semantic or dense-rerank architectures, multilingual instruction data, dynamic tool updates, and retrieval pipelines that expose only the minimal relevant tool subset or prompt fragment at each step (Oh et al., 2024, Zhuang et al., 27 May 2025, Franko, 1 Dec 2025). A plausible implication is that future ITR systems will increasingly be judged not only by topical relevance but by whether they can condition retrieval on precise, instance-wise instructions without sacrificing efficiency or robustness.