Tool Selection Engineering Fundamentals

• Tool Selection Engineering is the systematic design of representations, decision procedures, and evaluation protocols to optimize tool choice under dynamic task constraints and uncertainty.
• It aligns task requirements with tool capabilities through shared semantic spaces, structured contracts, and ranking mechanisms to enhance performance and safety.
• Advances include learning-based policies, causal filtering, and dynamic inventory management that improve execution efficiency and robustness across various domains.

Tool selection engineering denotes the design of representations, decision procedures, control loops, and evaluation protocols that determine which tool, equipment item, model, or external function should be used for a given task, and, in many systems, how that tool should subsequently be used. Across the literature, the problem is not treated as a single classification primitive. In domestic robotics it is framed as a coupled perception-and-control problem; in LLM agents it becomes retrieval, ranking, orchestration, or sequential policy optimization over large and changing tool inventories; in communications it is embedded in a broader lifecycle comprising tool creation, discovery, selection, learning, and benchmarking (Saito et al., 2021, Lumer et al., 2024, Liu et al., 13 Jan 2026).

1. Scope and conceptual boundaries

The surveyed work uses the term across several technically distinct but structurally related settings. What unifies them is that selection is made under task constraints, partial observability, or resource tradeoffs, and that the chosen artifact is expected to improve downstream execution rather than merely satisfy a semantic label.

Setting	Selection target	Representative mechanism
Domestic robot manipulation	ladle or turner, plus handling policy	active perception with multimodal latent dynamics (Saito et al., 2021)
Manufacturing ramp-up planning	robots, feeders, vision systems	guided state-machine copilot with RAG and structured catalogs (Werheid et al., 2024)
Large tool-equipped agents	API/function subset from large registries	tool knowledge bases, search-and-load, or inertia graphs (Lumer et al., 2024, Gaurav et al., 22 Sep 2025, Jia et al., 18 Nov 2025)
Software and model engineering	sentiment tools, feature-selection methods, pretrained models	dataset-profile mapping, benchmarking frameworks, and bandit search (Obaidi et al., 2 Jul 2025, Tabakhi et al., 2022, Kannan et al., 2023)

A recurrent shift in this literature is from static attachment to dynamic availability. In robotic manipulation, the relevant question is not only which tool exists, but whether the robot can infer a tool–object–action relation from active interaction and then generate the correct grasp strength, inclination, and transfer motion (Saito et al., 2021). In LLM-agent settings, the shift is from fixed inventories hard-coded into the prompt toward dynamic candidate retrieval from hundreds or thousands of tools, often followed by filtering, ranking, or staged loading (Lumer et al., 2024, Gaurav et al., 22 Sep 2025). In communications, the same shift is described more abstractly as the movement from tool creation and discovery to context-aware selection and tool-chain orchestration (Liu et al., 13 Jan 2026).

This suggests that tool selection engineering is best understood as a systems discipline rather than a single algorithmic task. The selected artifact may be a physical implement, a software component, a pretrained model, or a callable API, but the engineering questions recur: how to represent tool capabilities, how to represent task requirements, how to compare them under uncertainty, and how to evaluate the consequences of wrong selection.

2. Task and tool representations

Selection quality is tightly coupled to representation design. A major line of work represents both the task and the tool in a shared semantic space. In domestic robot tool use, object properties are divided into extrinsic properties—shape, color, size, posture, and position—and intrinsic properties—weight, friction, and hardness. The system then infers tool–object–action relations from multimodal sensorimotor traces rather than from explicit symbolic labels (Saito et al., 2021). In low-dimensional attribute alignment, ToolNet defines 115 common tools using 13 attributes spanning physical, functional, and psychological properties, and tool selection is performed by matching attributes required by the scenario against attributes predicted from tool images (Hao et al., 28 May 2025). In open-world multimodal selection, RaTA-Tool standardizes both tasks and tools into the JSON schema input / process / output, then retrieves the tool whose description best matches the generated task description in embedding space (Mattioli et al., 16 Apr 2026).

In agentic tool ecosystems, richer metadata becomes essential. AutoTool represents each tool with name, functionality, input-output schema, and usage constraints, embeds those descriptions into the model’s hidden space, and then performs repeated trajectory-level tool selection over the current candidate set (Zou et al., 15 Dec 2025). Toolshed enriches tool documents with tool name, description, argument schema, hypothetical questions, and key topics or intents before indexing them in a vector database, arguing that name-plus-description alone is too weak for large-scale selection (Lumer et al., 2024). ToolChoiceConfusion makes the representation even more operational by assigning each tool a lightweight contract

$t_i = (d_i, R_i, E_i, c_i, \rho_i),$

where $d_i$ is a description, $R_i$ required state variables, $E_i$ produced state variables, $c_i$ optional cost, and $\rho_i$ optional risk (Babu et al., 4 Jun 2026).

A representative retrieval formulation appears in RaTA-Tool: $$t^* = \arg\max_{t \in \mathcal{T}} E(\tilde d)\cdot E(d_t),$$ where $\tilde d$ is the generated task description and $d_t$ is the structured tool description (Mattioli et al., 16 Apr 2026). A representative attribute-alignment formulation appears in ToolNet: $$\mathbf{t}^* = \arg\max_{\mathbf{t}_i} s(f_l(\mathbf d), f_v(\mathbf t_i)),$$ where language and vision encoders map scenarios and tool images into a common 13-dimensional attribute space (Hao et al., 28 May 2025).

The common engineering lesson is that direct tool identifiers are usually too brittle. Selection systems increasingly depend on normalized capability representations, structured contracts, or latent relational spaces that make task-tool compatibility explicit.

3. Selection architectures and decision procedures

The simplest architectures use static binding or keyword matching. The communications survey describes this as the most basic form of tool selection: a hard-coded tool-task association or a tool selected from rudimentary keyword analysis of the query (Liu et al., 13 Jan 2026). The same survey contrasts this with dynamic context-aware selection, in which the agent uses user intent, interaction history, current state, tool descriptions, and operational metrics such as latency, robustness, and resource consumption, and with tool-chain orchestration for multi-step tasks (Liu et al., 13 Jan 2026).

Large tool ecosystems have driven retrieval-centric architectures. Toolshed’s Advanced RAG-Tool Fusion decomposes the pipeline into pre-retrieval document enhancement, intra-retrieval query rewriting/decomposition/expansion, and post-retrieval reranking and self-reflection. On ToolE single-tool, ToolE multi-tool, and Seal-Tools, it reports absolute Recall@5 improvements of 46%, 56%, and 47% over baseline retrievers (Lumer et al., 2024). Dynamic ReAct applies a related idea to MCP environments: rather than binding all tools, it keeps meta-tools such as search_tools and load_tools always available, decomposes the request into atomic queries, retrieves $d_i$0 candidates per query, caps results at $d_i$1 per application, and typically loads fewer than 5 tools (Gaurav et al., 22 Sep 2025).

A different architecture exploits sequential regularity. AutoTool models “tool usage inertia” with a directed graph built from historical agent trajectories. On ScienceWorld, the entropy of next-tool prediction drops from 3.50 bits at 0-order to 2.52 bits at first order and 1.93 bits at second order, indicating strong local dependence (Jia et al., 18 Nov 2025). Its scoring function combines frequency and contextual relevance: $d_i$2 and inertial invocation is used only when the score exceeds a threshold (Jia et al., 18 Nov 2025).

Another design axis is causal filtering rather than semantic relevance. CMTF searches for a minimal causal path from current symbolic state to goal and exposes only the next executable frontier. In the main benchmark with 102 tasks, 100 tools, four LLM backends, and 2448 task-method-model runs, it matches the strongest causal baseline at 0.99 success while reducing visible tools from 100.00 to 1.00 per step and average tokens from 24,569 to 2,405 (Babu et al., 4 Jun 2026).

Physical robotics presents a more tightly coupled variant. In the ingredient-transfer system, tool selection is not a separate classification head; after a fixed active-perception stirring phase, the inferred latent state conditions the entire motor trajectory, so reaching toward the ladle or turner emerges as part of a generated sensorimotor sequence (Saito et al., 2021). This suggests that in some systems the correct architecture is not “classify then plan,” but a single latent dynamical model whose state jointly determines recognition and execution.

4. Learning, adaptation, and optimization

Many recent systems treat tool selection as a learnable policy rather than as a purely hand-engineered rule. VisTA formalizes visual tool selection as policy learning over a sequence-valued action $d_i$3 and trains that policy with GRPO using outcome-only reward. Its reward is $d_i$4 when tools rescue an otherwise wrong answer, $d_i$5 when tools turn a correct answer into an incorrect one, $d_i$6 when both with-tools and without-tools fail, and $d_i$7 when tools preserve correctness (Huang et al., 26 May 2025). On ChartQA, ChartQA-OoD, and Geometry3K, this learning-based selector outperforms training-free baselines, with especially large gains on out-of-distribution chart examples (Huang et al., 26 May 2025).

AutoTool for dynamic agentic reasoning pushes the same idea into open-world, multi-domain LLM agents. It constructs a 200k-instance dataset with explicit tool-selection rationales over 1,346 tools and more than 100 task types, trains first for trajectory stabilization and then for KL-regularized Plackett–Luce ranking of complete tool-using trajectories, and evaluates on settings where 460 tools are seen in training and 886 are held out until inference (Zou et al., 15 Dec 2025). Across ten benchmarks, the paper reports average gains of 6.4% in math and science reasoning, 4.5% in search-based QA, 7.7% in code generation, and 6.9% in multimodal understanding over prior training paradigms (Zou et al., 15 Dec 2025).

Open-world multimodal selection has also been formulated as retrieval-grounded preference optimization. RaTA-Tool uses an MLLM to rewrite a multimodal query into a structured task description, then applies Direct Preference Optimization so that preferred descriptions are those that rank the ground-truth tool highest in retrieval. On a benchmark with 14,924 queries and 920 tools split at the tool level, the best configuration—RaTA-Tool-7B with Qwen3-Embedding and DPO—reaches weighted average accuracy $d_i$8 (Mattioli et al., 16 Apr 2026).

Not all learning-based systems optimize tool IDs directly. GreenRunnerGPT uses GPT-4 to infer weights over accuracy, size, and complexity from a use-case description, then frames model selection as a multi-armed bandit problem over a repository of pretrained models (Kannan et al., 2023). In the reported experiment, the repository contains 71 ImageNet-pretrained image classifiers, and Thompson Sampling is used to reduce the “evaluation tax” of brute-force model selection on ObjectNet (Kannan et al., 2023). This indicates that tool selection engineering also includes preference elicitation and budgeted evaluation, not only direct ranking.

The broader implication is that learning enters the problem at several levels: latent representation learning, retrieval-compatible task rewriting, reinforcement learning over tool subsets, and adaptive budget allocation over candidate models. The field is therefore less about a single “selector” than about where the optimization signal is attached.

5. Domain realizations

Embodied robotics provides the clearest examples of selection coupled with execution. In the ingredient-transfer task, a Nextage Open humanoid must transfer ingredients from a pot to a bowl using either a ladle or a turner, with the correct choice depending on both visible and hidden properties. The full image+force+tactile configuration reaches an overall success rate of 71.7%, more than double the ablated settings, and the robot succeeds in tool selection in all untrained-ingredient experiments while also achieving at least two successful pickup-and-transfer trials out of five for each untrained ingredient (Saito et al., 2021). A complementary line of work shifts the problem from choosing a permanently mounted end-effector to choosing among externally graspable passive tools: a mechanical tool architecture for 2-finger parallel grippers uses two symmetric parallelograms and four torsion springs, can be picked up like an ordinary object, supports varied tooltips, and is experimentally reported to lift an object of 8 kg at a tooltip–gravity angle of 15° (Hu et al., 2019). This turns tool selection into a library-design and grasp-planning problem.

Industrial planning systems instantiate selection in decision support rather than execution control. The manufacturing copilot built with GPT-4o, LangGraph, RAG, and an SQLite equipment database supports robots, feeders, and vision systems in ramp-up planning (Werheid et al., 2024). Across 22 equipment prompts, 19 reached the correct equipment selection level while considering most requirements, and 6 met all relevant requirements (Werheid et al., 2024). Here, selection is staged: requirement grouping, elementary-operation inference, subtype narrowing, structured candidate retrieval, and a reflection step that can request more information before final recommendation (Werheid et al., 2024).

Software engineering and data-centric tooling add another realization. In sentiment analysis for software engineering, 10 developer-communication datasets from five platforms are profiled using 13 linguistic features and 8 statistical features, then mapped to suitable sentiment tools via a questionnaire-based procedure. The results show that SetFit and RoBERTa are consistently strong, but that tool effectiveness remains context-dependent across App Reviews, Code Reviews, GitHub, Jira, and Stack Overflow (Obaidi et al., 2 Jul 2025). In feature selection, UniFeat is engineered as a dedicated Java framework rather than a small module in a general ML toolkit; it organizes methods into filter, wrapper, and embedded packages, reports accuracy and execution time, exports reduced datasets, and includes a Friedman test panel for statistical comparison (Tabakhi et al., 2022). In model repository selection, GreenRunnerGPT makes the tradeoff explicit by combining target-task accuracy with model size and complexity, rather than selecting by benchmark accuracy alone (Kannan et al., 2023).

A more cognitively structured realization appears in low-dimensional attribute alignment. The ToolNet framework uses 115 common tools and 13 attributes to align visual tool perception with linguistic task understanding, reaching 74% accuracy on tool selection, compared with 20% for direct tool matching and 21%–58% for smaller multimodal models (Hao et al., 28 May 2025). This suggests that when tool functionality is latent and substitution matters, an explicit affordance-like bottleneck can be more effective than direct label matching.

6. Evaluation, robustness, and unresolved problems

Evaluation protocols in tool selection engineering vary sharply by domain, but a common pattern is that end-task correctness is not enough. Robotic systems break success down into tool selection, tool grasping, picking up the ingredient, and pouring (Saito et al., 2021). Retrieval systems emphasize Recall@$d_i$9, especially Recall@5 in large tool libraries (Lumer et al., 2024). Causal filtering reports task success, wrong-tool count, premature-action count, tools exposed per step, and token cost (Babu et al., 4 Jun 2026). Sentiment-tool mapping uses micro-$R_i$0, macro-$R_i$1, and an overall mean score (Obaidi et al., 2 Jul 2025). UniFeat integrates repeated runs, visualization, and Friedman testing for multi-method comparison (Tabakhi et al., 2022). This diversity indicates that tool selection engineering is inseparable from application-specific failure models.

Security and robustness have become central. ToolCert, called CATS in the manuscript, models tool selection as an i.i.d. Bernoulli success process under adaptive adversarial tool injection and computes a 95% Clopper–Pearson lower bound on robust accuracy (Yeon et al., 5 Oct 2025). Under deceptive injection or retrieval saturation, the certified bound drops near zero, with an average performance drop of over 60% compared with non-adversarial settings; for adversarial tool injection, the certified bound falls from 0.92 at 0 rounds to 0.18 after one round and to 0.00 after ten rounds (Yeon et al., 5 Oct 2025). A complementary black-box metadata attack perturbs only the target tool’s textual information at word and character level and substantially increases its selection rate or ranking. On ToolBench I3, for example, BM25 Hit@1 rises from 9.93 to 70.51 and Ada Hit@1 from 12.89 to 59.11 under conditional attack; on ScienceQA, Vicuna-7b’s target-tool usage probability rises from 0.0519 to 0.9929 (Chen et al., 7 Apr 2025).

The literature also exposes unresolved structural problems. Tool discovery and tool selection remain distinct but tightly coupled in open environments (Liu et al., 13 Jan 2026). Contract quality and state tracking are hard dependencies for causal filtering (Babu et al., 4 Jun 2026). Retrieval quality remains sensitive to metadata design, as shown by Toolshed’s emphasis on enriched tool documents and by RaTA-Tool’s dependence on standardized model-card descriptions (Lumer et al., 2024, Mattioli et al., 16 Apr 2026). Dynamic inventories, unseen tools, permissions, versioning, and maintenance burdens remain persistent concerns in MCP-scale systems and communication-network settings (Gaurav et al., 22 Sep 2025, Liu et al., 13 Jan 2026).

A plausible implication is that future tool selection engineering will converge on hybrid designs: structured capability metadata, explicit state or contract models where available, retrieval and ranking layers for scale, learned policies for sequential use, and verifier or shield modules for safety and robustness. The surveyed work does not yet provide a universal architecture, but it does establish the main engineering principle: selecting a tool is rarely sufficient in isolation; the system must also reason about when the tool is needed, how it should be parameterized, whether it is feasible or safe to invoke, and how its use changes the rest of the control process.