Model-Adaptive Tool Necessity Reveals the Knowing-Doing Gap in LLM Tool Use

Abstract: LLMs increasingly act as autonomous agents that must decide when to answer directly vs. when to invoke external tools. Prior work studying adaptive tool use has largely treated tool necessity as a model-agnostic property, annotated by human or LLM judge, and mostly cover cases where the answer is obvious (e.g., fetching the weather vs. paraphrasing text). However, tool necessity in the wild is more nuanced due to the divergence of capability boundaries across models: a problem solvable by a strong model on its own may still require tools for a weaker one. In this work, we introduce a model-adaptive definition of tool-necessity, grounded in each model's empirical performance. Following this definition, we compare the necessity against observed tool-call behavior across four models on arithmetic and factual QA dataset, and find substantial mismatches of 26.5-54.0% and 30.8-41.8%, respectively. To diagnose the failure, we decompose tool use into two stages: an internal cognition stage that reflects whether a model believes a tool is necessary, and an execution stage that determines whether the model actually makes a tool-call action. By probing the LLM hidden states, we find that both signals are often linearly decodable, yet their probe directions become nearly orthogonal in the late-layer, last-token regime that drives the next-token action. By tracing the trajectory of samples in the two-stage process, we further discover that the majority of mismatch is concentrated in the cognition-to-action transition, not in cognition itself. These results reveal a knowing-doing gap in LLM tool-use: improving tool-use reliability requires not only better recognition of when tools are needed, but also better translation of that recognition into action.

• The paper introduces a model-adaptive criterion for tool necessity based on repeated stochastic sampling of model outputs.
• The study quantifies a significant mismatch (26.5%–54.0%) between internal recognition of tool necessity and actual tool-call actions in LLMs.
• The analysis highlights that robust internal meta-cognition does not guarantee proper execution, calling for improved mapping of cognition to action.

Model-Adaptive Tool Necessity and the Knowing-Doing Gap in LLM Tool Use

Introduction and Motivation

The paper "Model-Adaptive Tool Necessity Reveals the Knowing-Doing Gap in LLM Tool Use" (2605.14038) addresses the mechanism underlying tool-use decisions in LLM agents. Moving beyond traditional static, model-agnostic notions of tool necessity—often based on human or strong LLM judgment—the authors propose a model-adaptive definition. This new criterion reflects each model's empirical performance: whether it can consistently answer queries correctly without external tools. The motivation arises from the observation that diverse LLMs exhibit markedly different capability boundaries, challenging prior assumptions that tool necessity is a universal query property. The central finding is the quantification and diagnosis of a substantial mismatch between models' empirical necessity to use tools and their actual tool-call actions, termed the "knowing-doing gap."

Figure 1: Overview of the two-stage cognition-execution modeling, showing necessity as model-adaptive, linear probing for cognition, and execution probes revealing near orthogonality and mismatch concentration at the action stage.

Model-Adaptive Tool Necessity: Formulation and Empirical Analysis

The model-adaptive definition of necessity is operationalized by running a model on a given query multiple times with stochastic sampling, marking the query as tool-necessary if at least one run yields an incorrect answer. This procedure robustly delineates the model's capability boundary and enables fine-grained, model-specific necessity annotations, in contrast to static approaches. Empirical analysis across four LLM variants (Qwen3-8B/4B and Llama-3.1/3.2) on arithmetic and factual QA tasks illuminates considerable disparity in boundaries: a query solvable reliably by a larger model may require tool assistance for a smaller counterpart.

Figure 2: Visualization of model-dependent tool-call necessity, revealing the heterogeneity of capability boundaries across models and tasks.

Quantifying Necessity-Action Mismatch

When models are allowed access to tools (calculator or search API), the distribution of tool-call behaviors on both tool-necessary and tool-unnecessary instances is scrutinized. The data reveals pronounced rates of necessity-action mismatch: between $26.5\%$ and $54.0\%$ for arithmetic, and $30.8\%$ to $41.8\%$ for TruthfulQA. These errors are not uniformly distributed; some models overuse tools when direct answers suffice, while others fail to call tools when outside their capability. The bias is context-, model-, and task-dependent, indicating non-trivial interplay between internal knowledge and execution policy.

Two-Stage Decomposition: Cognition vs. Execution

The authors decompose tool-use behavior into an internal cognition stage (whether the model believes a tool is necessary) and an execution stage (whether the model actually calls a tool). Probing hidden states at multiple layers and token positions with linear classifiers, they analyze both the separability of necessity and action signals, and the geometric relationship between their directions.

Figure 3: Necessity probe performance (MCC heatmaps), indicating task dependence and model-family similarity in linear separability of necessity.

Necessity is often linearly separable in hidden space, especially for arithmetic tasks and larger models; however, this signal does not guarantee aligned execution.

Figure 4: Action probe performance (MCC heatmaps) shows the action to call a tool is highly separable in late layers and end tokens.

Strikingly, necessity and action probe directions ($\mathbf{w}_c$ and $\mathbf{w}_a$) are only partially aligned in intermediate representations. In the final, generation-driving positions (late layers at last query token), their directions tend toward near orthogonality.

Figure 5: Cosine similarity between cognition and action probe directions, revealing collapse of alignment at late layers where action decisions are made.

Diagnosis of the Knowing-Doing Gap

To trace the root of necessity-action mismatch, per-sample trajectories are decomposed across factual necessity, internal cognition, and executed action. The majority of mismatches are concentrated in the cognition-to-action transition (stage two), not due to failures in internal meta-cognition. Models often encode correct necessity beliefs but fail to act accordingly. This is further corroborated by density plots of confidence vs. tool-calling probability: mismatches persist even at high meta-cognitive confidence.

Figure 6: Sankey diagram of two-stage decomposition, illustrating that end-to-end error is predominantly due to execution-stage failure (orange band).

Figure 7: Density plot of cognition confidence vs. tool call probability, showing mismatch occurs across the spectrum, unrelated to confidence calibration.

Implications and Future Directions

These results undermine the assumption that improving meta-cognitive representations alone will suffice for reliable tool-use behavior. The primary bottleneck is in the translation of internal necessity recognition into tool-call actions. Practically, improving agent reliability—especially in real-world deployment where task-oriented prompts predominate—will require mechanisms that guarantee downstream execution fidelity with internal beliefs. Theoretically, the pronounced orthogonality at late layers hints at architectural or training factors that decouple meta-cognition from decision policy, potentially implicating prompt engineering, loss landscapes, or misalignment in supervised/finetuning data.

This analysis points toward several future avenues:

• Fine-grained control over execution-stage mapping, possibly via auxiliary losses or architectural modifications to bridge cognition-action representations.
• Investigation of similar knowing-doing gaps in other agentic behaviors, and whether these misalignments generalize to more complex workflows.
• Assessments of close-source LLMs and exploration of robustness under variable $N$, $T$ sampling criteria for capability boundary estimation.

Conclusion

The paper provides a rigorous framework for evaluating tool-use decisions in LLMs, anchored in model-adaptive necessity and decoupled cognition–execution stages. Strong empirical evidence demonstrates that substantial necessity-action mismatches are primarily rooted in the failure to translate internal recognition into proper action—a knowing-doing gap. Future development of autonomous LLM agents must explicitly address this gap, ensuring that sophisticated self-awareness directly informs robust and calibrated decisions.