AgentSHAP: Interpreting LLM Agent Tool Importance with Monte Carlo Shapley Value Estimation (2512.12597v1)

Abstract: LLM agents that use external tools can solve complex tasks, but understanding which tools actually contributed to a response remains a blind spot. No existing XAI methods address tool-level explanations. We introduce AgentSHAP, the first framework for explaining tool importance in LLM agents. AgentSHAP is model-agnostic: it treats the agent as a black box and works with any LLM (GPT, Claude, Llama, etc.) without needing access to internal weights or gradients. Using Monte Carlo Shapley values, AgentSHAP tests how an agent responds with different tool subsets and computes fair importance scores based on game theory. Our contributions are: (1) the first explainability method for agent tool attribution, grounded in Shapley values from game theory; (2) Monte Carlo sampling that reduces cost from O(2n) to practical levels; and (3) comprehensive experiments on API-Bank showing that AgentSHAP produces consistent scores across runs, correctly identifies which tools matter, and distinguishes relevant from irrelevant tools. AgentSHAP joins TokenSHAP (for tokens) and PixelSHAP (for image regions) to complete a family of Shapley-based XAI tools for modern generative AI. Code: https://github.com/GenAISHAP/TokenSHAP.

• The paper presents a novel Monte Carlo Shapley-based method that quantifies individual tool contributions in LLM agents.
• It employs a combination of deterministic leave-one-out and randomized coalition sampling to achieve stable and faithful attribution with a mean cosine similarity of 0.945.
• Results validate practical benefits including toolset optimization, error diagnosis, and effective exclusion of irrelevant tools with high domain-specific accuracy.

AgentSHAP: Tool-Level Attribution via Monte Carlo Shapley Value Estimation for LLM Agents

Introduction

The proliferation of LLM-based agents capable of orchestrating external tools (e.g., calculators, API endpoints, search engines) has advanced the frontier of agentic AI systems. However, the tool selection mechanism is opaque; current practices log the invocation of tools but lack quantitative assessments of each tool’s actual contribution to the agent's output. Existing XAI methods focus on token- or feature-level attribution and neglect the unique problem of tool-level explainability in LLM agents.

AgentSHAP introduces a model-agnostic, black-box framework for tool importance attribution, leveraging Monte Carlo estimation of Shapley values. This approach provides agent designers and researchers with actionable metrics to optimize toolsets, diagnose agent failures, and understand agent-tool interactions in practical deployment scenarios (2512.12597).

Figure 1: AgentSHAP visualizes tool importance per prompt; for math queries, Calculator is dominant (0.72), while Weather is critical for weather queries (0.78).

Methodology

AgentSHAP operationalizes the concept of cooperative game theory for tool attribution in agentic systems. Each tool is modeled as a player; a coalition is any subset of tools presented to the agent. The value function $v(S)$ quantifies the semantic similarity (cosine similarity in embedding space) between the agent’s response with tool-set $S$ and the baseline response with all tools.

The Shapley value for each tool is computed as the expected marginal contribution across all possible tool orderings. Given the exponential cost $O(2^n)$ of exact evaluation for $n$ tools, AgentSHAP implements a Monte Carlo approximation, combining deterministic leave-one-out assessment and randomized coalition sampling parameterized by a sampling ratio $\rho$.

Figure 2: The AgentSHAP pipeline entails baseline response evaluation, randomized coalition sampling, and marginal contribution scoring via semantic similarity.

This method’s axiomatic fairness properties (efficiency, symmetry, null player) ensure robust and interpretable importance scores. The approach is compatible with any LLM or agent framework, requiring only I/O access and no introspection of internal weights or attention mechanisms.

Empirical Evaluation

AgentSHAP is benchmarked on the API-Bank suite, encompassing diverse toolsets and prompt domains.

Consistency

SHAP scores exhibit high stability across Monte Carlo runs, with a mean cosine similarity of 0.945 for SHAP vectors and top-1 accuracy at 100%. Dominant tools per domain (Calculator for math, QueryStock for finance) retain the highest scores across repeated evaluations.

Figure 3: SHAP attribution remains consistent across repeated runs for diverse queries and domains.

Faithfulness

Faithfulness is empirically validated by removing high-SHAP versus low-SHAP tools and quantifying the drop in output quality. Removing important tools incurs a mean semantic similarity drop of 0.67, compared to only 0.05 for low-SHAP tools—a 13-fold difference. This substantiates that AgentSHAP scores are meaningful and actionable for system debugging and optimization.

Figure 4: Removing high-SHAP tools induces severe degradation, whereas low-SHAP removals have minimal impact, verifying the faithfulness of attribution.

Irrelevant Tool Injection

AgentSHAP detects injected irrelevant tools by assigning them substantially lower SHAP scores (mean gap of 7x compared to the dominant tool). Top-1 accuracy for identifying the expected tool in this scenario is 86%.

Figure 5: AgentSHAP attributes negligible importance to injected irrelevant tools and preserves correct attribution for the target tool.

Cross-Domain Attribution

A heatmap of mean SHAP by domain-tool reveals correct assignment: Calculator (0.45) for Math, QueryStock (0.55) for Finance, Wiki (0.57) for Knowledge. Overall top-1 accuracy on domain attribution stands at 86%.

Figure 6: Cross-domain experiments confirm domain-relevant tools receive the highest mean SHAP values per prompt category.

Implications and Future Directions

Theoretical implications: AgentSHAP advances the integration of cooperative game-theoretic principles into XAI for agentic workflows. Model-agnostic, semantically grounded attribution enables generalization across LLM architectures and deployment settings.

Practical implications: AgentSHAP facilitates pruning of superfluous tools (cost/latency reduction), automated toolset configuration, agent behavior auditing, and tool-aware agent fine-tuning. Its independence from model internals supports applications to closed-source APIs and federated multi-agent frameworks.

Limitations exist regarding higher-order interactions and causal call-order effects. Extending AgentSHAP to multi-turn conversations, fine-grained call trace analysis, and multi-agent attribution will augment its applicability. Future work should explore tool set recommendation via SHAP patterns, dynamic tool curriculum learning, and scalable real-time estimation for interactive agents.

Conclusion

AgentSHAP presents a principled, Monte Carlo Shapley-based methodology for tool-level attribution in LLM agents, validated by experimental robustness, faithfulness, and low susceptibility to irrelevant tool injection. Its open-source implementation enables cost-efficient, black-box explainability for tool-augmented AI systems, providing researchers and practitioners with a vital diagnostic and optimization toolkit as agentic architectures proliferate.