SMART-ER: Dataset for Tool-Aware Language Models

• SMART-ER is a dataset of compositional, multi-domain queries designed to assess when language models should leverage internal knowledge versus external tools.
• It utilizes a semi-automated, model-assisted pipeline to generate and annotate multi-hop queries with human-readable rationales for each sub-goal.
• The dataset underpins the SMARTAgent benchmarks, demonstrating significant reductions in unnecessary tool calls and substantial accuracy improvements.

SMART-ER is a dataset introduced to support the development and evaluation of LLM agents that can dynamically balance parametric knowledge and external tool use. Designed to address Tool Overuse—where models unnecessarily rely on external resources even for problems solvable by their internal parametric knowledge—SMART-ER provides compositional, multi-domain queries that require alternating between internal reasoning and explicit tool invocation, with human-readable rationales at every step. The dataset is central to training and benchmarking the SMARTAgent family of models, which achieve substantial improvements in both tool-use efficiency and overall accuracy (Qian et al., 17 Feb 2025).

1. Domain Coverage and Dataset Scope

SMART-ER comprises over 3,000 compositional queries distributed equally across three distinct domains. Each domain is strategically selected to reveal differing capabilities and failure modes in contemporary LLMs:

• Math: Tasks adapted from the MATH benchmark feature a mix of straightforward arithmetic operations (additions, subtractions) and more complex symbolic manipulations requiring algebraic reasoning and multi-step proofs.
• Time: Based on FreshQA, this domain blends “slow-changing” factoid questions (e.g., historical dates) with “fast-changing” knowledge (e.g., current office holders, recent events) that necessitate retrieval capabilities.
• Intention: Leveraging the Intention-in-Interaction (IN3) dataset, this domain centers on instruction comprehension and response generation, where critical user preferences or intents are only accessible via clarifying questions.

This breadth ensures that SMART-ER systemically tests alternation between knowledge-based and tool-dependent reasoning, fostering agent self-awareness regarding when external resources are warranted (Qian et al., 17 Feb 2025).

2. Data Construction and Annotation Workflow

SMART-ER employs a semi-automated, model-assisted pipeline to generate compositional multi-hop questions and annotate their sub-goals:

• Compositional Query Creation:
  • Math: Selection is limited to problems that baseline GPT models answer incorrectly, guaranteeing at least one simple arithmetic component and a distinct, more complex deduction in each task.
  • Time: Fast-changing questions are filtered and augmented through self-instruction techniques and then fused with slow-changing sub-questions to mandate multi-hop reasoning.
  • Intention: Vague instructions, chosen such that missing user preference recovery is pivotal, are included only if solvable without tools.
• Automated Sub-goal Decomposition: Given a query $Q$, GPT-4o is used to decompose $Q$ into a sequence $S = \{s_1, ..., s_n\}$ of ordered sub-goals.
• Tool Necessity Annotation: For each sub-goal $s_i$, a binary annotation $A(s_i)$ is assigned:
  • $A(s_i)=0$: solvable with parametric knowledge alone
  • $A(s_i)=1$: requires an explicit tool call
• Tool Mapping: When $A(s_i)=1$, a mapping function $T(s_i)$ selects the required tool from $\{$Code, Search, AskUser$Q$0.
• Stepwise Reasoning and Rationale Generation: For parametric sub-goals, an auxiliary model generates the reasoning text $Q$1. For tool-dependent sub-goals, the model produces tool parameters $Q$2, executes $Q$3 to obtain the output $Q$4, and records the chain. Each sub-goal, regardless of type, is annotated with a concise natural-language justification $Q$5 detailing why parametric knowledge suffices or tool use is required.

The individual steps in every chain are formatted as structured JSON entries containing subgoal descriptions, annotations, tools, parameters, outputs, reasoning, and justifications (Qian et al., 17 Feb 2025).

3. Dataset Format, Partitioning, and Statistics

SMART-ER’s data structure is meticulously organized to support supervised and interactive learning protocols:

• Reasoning Chain Format: Each query comprises an average of 4–6 sequential sub-goals, with 1–2 requiring explicit tool invocation. Every sub-goal is accompanied by a rationale justification.
• Corpus Size and Annotation Mass: The dataset contains over 3,000 compositional queries, yielding approximately 15,000–18,000 justification annotations across all sub-goals.
• Splitting Ratios: Queries are split into 70% training ($Q$62,100 examples), 10% validation ($Q$7300), and 20% test ($Q$8600) subsets for robust model evaluation.
• Example Schema:

$S = \{s_1, ..., s_n\}$8

This structure supports granular supervision and model introspection at each decision point.

4. Metrics and Formal Evaluation Criteria

SMART-ER defines and deploys multiple quantitative measures to assess agent proficiency and efficiency in reasoning and tool use:

• Tool-Use Rate ($Q$9): The mean number of tool calls per query over the evaluation set.
• Accuracy (In-Domain): The fraction of responses matching ground-truth answers for Math and Time tasks. For Intention, two domain-specific metrics—Missing-Details-Recovery and Intention-Coverage—evaluate effectiveness in preference recovery and task coverage.
• Tool Overuse Rate ($S = \{s_1, ..., s_n\}$0): Given total queries $S = \{s_1, ..., s_n\}$1 and $S = \{s_1, ..., s_n\}$2 those answerable without tools:
  • $S = \{s_1, ..., s_n\}$3: Intrinsic reasoning capability
  • $S = \{s_1, ..., s_n\}$4 (proportion of $S = \{s_1, ..., s_n\}$5-questions on which the model needlessly uses tools)
  • $S = \{s_1, ..., s_n\}$6: Fraction of all questions that reflect unnecessary tool use
• Change in Tool Overuse:

$S = \{s_1, ..., s_n\}$7

Quantifies relative reduction in unnecessary tool calls.

• Performance Gain: Relative increase in accuracy compared to the best-performing baseline method.

These metrics capture both resource efficiency (minimizing redundant calls) and solution correctness, facilitating calibrated model comparison (Qian et al., 17 Feb 2025).

5. Evaluation Protocols and Model Benchmarks

SMART-ER’s protocols distinguish between in-domain and out-of-distribution (OOD) evaluation regimes:

• In-domain Evaluation: For “no-tool” baselines, models process queries solely via chain-of-thought. For tool-enabled baselines and SMARTAgent, an interactive inference loop is used: whenever a tool call is produced, it is executed and the output is returned to the model until a “Final Response” is generated. Key outcomes are Tool Used (average calls/query), Accuracy (%), and domain-specific metrics for the Intention tasks.
• Out-of-distribution Generalization: Benchmarked using GSM8K (math word problems) and MINTQA (multi-hop QA over novel/tailed knowledge). With Llama-3.1-8B as baseline, SMARTAgent reduces tool calls from 2.53 (GSM8K) and 4.03 (MINTQA) respectively to 0.76 and 1.06, while maintaining or slightly improving accuracy (e.g., GSM8K: 83.17% to 83.40%).
• Key Findings: SMARTAgent achieves a 24% macro-average reduction in tool-use and a 37% macro-average accuracy gain over traditional non-strategic baselines. On OOD benchmarks, the tool call count is reduced to approximately one-fifth, with performance parity or improvement, underscoring strong generalization in self-aware tool usage.

Evaluation Setting	Baseline Tool Calls	SMARTAgent Tool Calls	Accuracy (GSM8K)
In-domain Macro Avg.	—	—	+37% over best baseline
OOD: GSM8K	2.53	0.76	83.17% → 83.40%
OOD: MINTQA	4.03	1.06	—

This suggests that training with SMART-ER instills robust decision-making about when external augmentation is necessary, enabling smaller models to achieve large-model or even proprietary model parity through strategic tool use.

6. Applications and Broader Significance

SMART-ER is pivotal for research aimed at building agents that are not only proficient at leveraging tools but are also discerning in their deployment. Its richly annotated chains, spanning multi-hop and cross-domain reasoning, are essential for:

• Training agents (such as SMARTAgent) that minimize computational overhead while maximizing quality, thus bridging the gap between model scale and performance.
• Developing robust self-awareness protocols in LLM-based architectures for both general and specialized reasoning tasks.
• Providing a structured foundation for meta-reasoning and introspection studies in AI tool-use, including tool selection, invocation timing, and rationale transparency.

An implication is that benchmark datasets designed in the style of SMART-ER could facilitate further reductions in unnecessary resource calls and generalizable self-diagnostic capabilities across broader AI systems (Qian et al., 17 Feb 2025).