RaTA-Tool: Open-World Multimodal Tool Selection

Updated 7 July 2026

RaTA-Tool is a framework that transforms multimodal inputs (text, image, audio) into machine-readable task descriptions to enable flexible tool selection.
It employs a two-step pipeline where an MLLM generates structured tasks that are semantically matched to standardized tool descriptions for open-world applicability.
Direct Preference Optimization (DPO) refines the generation process to improve retrieval accuracy, significantly outperforming closed-set baselines.

RaTA-Tool is a framework for open-world multimodal tool selection in Multimodal LLMs (MLLMs). It addresses a specific failure mode of prior tool-use systems: most existing methods assume purely textual inputs and learn direct mappings from queries to a fixed, closed set of tool identifiers, which prevents reliable handling of image- or audio-conditioned instructions and blocks selection of tools unseen during training. RaTA-Tool reformulates the problem as structured task description generation followed by retrieval over machine-readable tool descriptions, and further aligns generation with selection quality through Direct Preference Optimization (DPO) (Mattioli et al., 16 Apr 2026).

1. Problem setting and conceptual shift

Tool learning with foundation models aims to endow AI systems with the ability to invoke external resources such as APIs, computational utilities, and specialized models. Within that broader agenda, RaTA-Tool targets the setting in which the input query may contain text plus optional image or audio, and the available tool inventory is not fixed at training time (Mattioli et al., 16 Apr 2026).

The central motivation is the mismatch between contemporary tool ecosystems and prevailing tool-selection formulations. In the closed-world setup, a model is trained to predict one identifier from a predefined tool set. This design is compatible with supervised classification, but it fails when a tool is absent from the training label space. It also presumes that the query can be interpreted from text alone. RaTA-Tool replaces this direct query-to-tool mapping with a two-step pipeline: first, an MLLM converts the multimodal user query into a structured, machine-readable task description; second, the system retrieves the best matching tool by comparing that description to a library of similarly structured tool descriptions (Mattioli et al., 16 Apr 2026).

This retrieval-based formulation makes extensibility a first-class property. Because new tool descriptions can be added directly to the retrieval database, the MLLM and embedding model do not need retraining when novel tools arrive. A plausible implication is that RaTA-Tool treats tool selection less as label prediction and more as semantic matching over a dynamically maintainable tool index.

2. Structured task descriptions as the intermediate representation

The core intermediate object in RaTA-Tool is a JSON-style task description generated from the multimodal query. Given a query $q$ consisting of text plus optional image or audio, a pre-trained MLLM $G$ is fine-tuned in a supervised fashion to output a description $\tilde d = (z_1, z_2, \ldots, z_T)$ by minimizing

$L_{\mathrm{SFT}} = - \sum_{t=1}^T \log p_G(z_t \mid q, z_{<t}).$

Each generated description has exactly three fields:

$L_{\mathrm{SFT}} = - \sum_{t=1}^T \log p_G(z_t \mid q, z_{<t}).$ 0

An example given for this schema is:

$L_{\mathrm{SFT}} = - \sum_{t=1}^T \log p_G(z_t \mid q, z_{<t}).$ 1

This design compresses the query into a normalized task specification that is agnostic to the eventual tool identifier (Mattioli et al., 16 Apr 2026). The representation is deliberately operational: the input field captures modality and format, process captures the functional transformation, and output captures the expected return structure. This is important because open-world selection depends on semantic comparability between query-derived descriptions and tool-derived descriptions.

A common misconception is that retrieval alone solves open-world tool use. RaTA-Tool indicates otherwise: retrieval quality depends on the prior generation of a task description whose semantics align with the retrieval space. The framework therefore decomposes tool selection into multimodal instruction understanding and semantic retrieval rather than eliminating the generative component.

3. Retrieval mechanism and preference-based alignment

RaTA-Tool embeds both generated task descriptions and tool descriptions into a shared text space using a frozen embedding model $E(\cdot)$ , specifically Qwen3-Embedding-8B. If $\tilde d$ is the generated query description and $d_t$ is the description of tool $t$ , then

$E_q = E(\tilde d), \qquad E_d = E(d_t).$

Similarity is computed either as an inner product over $L_2$ -normalized embeddings or explicitly as cosine similarity:

$G$ 0

$G$ 1

The selected tool is

$G$ 2

This retrieval stage is paired with a preference-based optimization stage. After supervised fine-tuning, the generated descriptions may still be suboptimal for retrieval. RaTA-Tool therefore uses DPO with weak supervision derived from retrieval performance. For each query $G$ 3, the system samples $G$ 4 candidate descriptions $G$ 5 using diverse decoding strategies, ranks them by similarity to the ground-truth tool description, and defines a preferred description $G$ 6 and a less preferred one $G$ 7. With policy model $G$ 8 and frozen reference $G$ 9, the DPO loss is

$\tilde d = (z_1, z_2, \ldots, z_T)$ 0

Here $\tilde d = (z_1, z_2, \ldots, z_T)$ 1 is the logistic sigmoid and $\tilde d = (z_1, z_2, \ldots, z_T)$ 2 controls the strength of pull-toward-reference (Mattioli et al., 16 Apr 2026).

The significance of this stage is not a large absolute performance jump, but alignment pressure toward descriptions that retrieve the correct tool more consistently. This suggests that, in RaTA-Tool, description generation is optimized not only for linguistic plausibility but also for downstream geometric separability in embedding space.

4. Tool description corpus and benchmark design

RaTA-Tool introduces what is described as the first dataset for open-world multimodal tool use. Each tool in the dataset is a Hugging Face model. The model card of each tool is scraped, and an LLM, Qwen3-8B, is prompted to extract the same three fields—input, process, and output—into a standardized JSON schema. The result is a consistent machine-readable library $\tilde d = (z_1, z_2, \ldots, z_T)$ 3 of tool descriptions (Mattioli et al., 16 Apr 2026).

The benchmark is built on ToolMMBench and reprocesses 14,924 query instances and 920 Hugging Face tools. Tools are split 90/10 into train and test, so that 94 tools in the test set are never seen during supervised fine-tuning; queries inherit their tool’s split. The benchmark covers three modality configurations: pure text, text+image, and text+audio (Mattioli et al., 16 Apr 2026).

Two aggregate metrics are defined in addition to modality-wise accuracy. $\tilde d = (z_1, z_2, \ldots, z_T)$ 4 is the unweighted average across the three modalities, and $\tilde d = (z_1, z_2, \ldots, z_T)$ 5 is the weighted average by number of queries per modality. This distinction matters because multimodal tool benchmarks are often imbalanced across modalities.

The standardized tool-description library is a critical design choice. Tool descriptions derived from heterogeneous model cards become retrievable only after being projected into a consistent schema. This places RaTA-Tool in direct conversation with work arguing that tools are under-documented and that document enrichment materially affects retrieval quality; Tool-DE, for example, reports that incomplete and heterogeneous documentation creates a semantic gap for retrieval systems (Lu et al., 26 Oct 2025). RaTA-Tool’s standardization step can therefore be read as a schema-constrained alternative to broader document expansion.

5. Empirical results and relation to adjacent retrieval paradigms

Using Qwen2.5-Omni-7B as generator and Qwen3-Embedding as retriever, RaTA-Tool reports the following aggregate weighted results:

Setting	$\tilde d = (z_1, z_2, \ldots, z_T)$ 6
Zero-shot	$\tilde d = (z_1, z_2, \ldots, z_T)$ 7
After SFT	$\tilde d = (z_1, z_2, \ldots, z_T)$ 8
After DPO	$\tilde d = (z_1, z_2, \ldots, z_T)$ 9

All modalities see large boosts versus closed-set baselines, which were below 20% in zero-shot (Mattioli et al., 16 Apr 2026). The numerical pattern is instructive. The principal gain comes from supervised training of structured description generation, while DPO contributes a smaller improvement in consistency. This supports the interpretation that the main bottleneck is semantic normalization of multimodal queries, with preference optimization acting as a secondary calibration step.

RaTA-Tool belongs to a broader family of retrieval-centric tool-selection methods, but its design priorities differ from neighboring systems. MassTool emphasizes precise query understanding through a two-tower architecture combining tool-usage detection, query-centric graph convolution, search-based user intent modeling, and adaptive knowledge transfer (Lin et al., 1 Jul 2025). Tool2Vec and ToolRefiner instead focus on efficient small-model retrieval, usage-driven tool embeddings, and staged reranking over large tool pools (Moon et al., 2024). Toolshed uses a tool knowledge base plus pre-, intra-, and post-retrieval RAG-style interventions without model fine-tuning (Lumer et al., 2024). ToolNet organizes tools into a sparse directed graph and navigates local successor sets to scale to thousands of tools with moderate token growth (Liu et al., 2024).

Against that landscape, RaTA-Tool’s distinctive contribution is the coupling of multimodal query interpretation with open-world retrieval over standardized tool descriptions. It does not primarily optimize retrieval over massive static registries, nor does it move tool knowledge into parameter modules as in ParaTool, where unseen tools cannot be used unless an adapter is trained (Yu et al., 28 May 2026). Instead, it preserves externalized tool semantics and relies on structured semantic matching, which is precisely what allows new tools to be added without retraining (Mattioli et al., 16 Apr 2026).

6. Limitations, security implications, and future directions

RaTA-Tool’s extensibility is conditioned on documentation quality. The paper states that retrieval quality depends on the richness and correctness of tool descriptions, and that model cards vary in style and detail. Complex multi-step tool pipelines are not yet handled. Future work is identified in richer modalities such as video and 3D, dynamic tool chaining, and online updating of descriptions (Mattioli et al., 16 Apr 2026).

These constraints delimit what “open-world” means in the framework. It means that unseen tools can be incorporated into the retrieval index without retraining the generator or embedder; it does not mean that arbitrary undocumented tools can be selected reliably, nor that compositional multi-tool orchestration is already solved. This suggests that open-world single-step selection and long-horizon orchestration remain separate research problems. A broader review of tool use in LLM agents explicitly distinguishes isolated single-call selection from long-horizon multi-tool orchestration under state, feedback, safety, and cost constraints (Xu et al., 24 Mar 2026).

Open-world retrieval also raises security concerns. MalTool shows that malicious tool attacks can combine manipulative tool descriptions with malicious code implementations, and that existing detectors, including commercial malware detection approaches and methods tailored to LLM-agent settings, exhibit limited effectiveness (Hu et al., 12 Feb 2026). In a retrieval-based ecosystem that ingests external tool descriptions, this implies that semantic matching alone is insufficient as a trust mechanism. A plausible implication is that future open-world tool selectors will need code-description consistency checks and stronger runtime guardrails in addition to improved retrieval.

Taken together, RaTA-Tool marks a transition from fixed-label tool classification toward semantically indexed, multimodal, open-world tool selection. Its main technical claim is not merely that retrieval is useful, but that a structured intermediate task description can make multimodal intent and heterogeneous tool documentation commensurable within a shared embedding space. That design, combined with DPO-based alignment and a standardized tool-description corpus, establishes a concrete recipe for extending MLLM tool selection beyond text-only and closed-world assumptions (Mattioli et al., 16 Apr 2026).