---
title: 'SafeInstructTool: Pre-Execution Safety'
url: https://www.emergentmind.com/topics/safeinstructtool
type: topic
---

# SafeInstructTool: Pre-Execution Safety

SafeInstructTool is a risk assessment and mitigation framework for the *prospective* safety of tool utilization with large language models. It is introduced together with SafeToolBench to shift evaluation from retrospective inspection after tool execution to pre-execution risk analysis, with the explicit aim of preventing harms such as financial loss, privacy leakage, or physical injury. Its central design principle is that tool-use risk should be modeled simultaneously from the perspectives of the user instruction, the tool itself, and the interaction between the instruction and the tool, yielding a nine-dimension scoring framework for blocking or escalating risky execution plans before they are run [2509.07315].

## 1. Conceptual setting and benchmark context

SafeInstructTool was proposed in a setting where LLMs had already shown strong performance in autonomously calling tools in external environments, but where existing safety evaluations largely remained retrospective. The motivating claim is that retrospective evaluation is insufficient when tool execution can produce irreversible consequences. SafeToolBench was therefore introduced as a benchmark for *prospective* evaluation, and SafeInstructTool as the corresponding mitigation framework [2509.07315].

SafeToolBench contains 1200 adversarial user instructions spanning 16 real-world application domains and four overarching risk categories. Within this benchmark setting, SafeInstructTool is not merely a detector of malicious phrasing. It is presented as a framework for analyzing the full risk structure of tool use before execution, including risks embedded in tool APIs and risks that emerge only when a particular instruction is paired with a particular tool call [2509.07315].

This framing is significant because it treats tool-use safety as a planning problem rather than only a content-moderation problem. A plausible implication is that the framework targets failure modes missed by approaches that only inspect the user query or only audit outputs after the tool has already run.

## 2. Three-perspective architecture and the nine dimensions

SafeInstructTool organizes pre-execution risk into three perspectives and nine detailed dimensions. The three perspectives are the **User Instruction Perspective**, the **Tool Itself Perspective**, and the **Joint Instruction-Tool Perspective** [2509.07315].

| Perspective | Dimensions |
|---|---|
| User Instruction | Data Sensitivity; Harmfulness of the Instruction; Urgency of the Instruction; Frequency of Tool Utilization |
| Tool Itself | Key Sensitivity; Type of Operation; Impact Scope of the Operation |
| Joint Instruction-Tool | Alignment Between Instruction and Tool; Value Sensitivity |

The **User Instruction Perspective** asks what risks are present in the instruction itself. Its four dimensions capture whether the instruction references sensitive data, whether it expresses harmful or malicious intent, whether it pressures immediate execution, and whether the rate of requested tool use is suspiciously high. The inclusion of urgency and frequency means the framework does not reduce instruction risk to semantic harmfulness alone [2509.07315].

The **Tool Itself Perspective** captures risks inherent to the tool or API. Key Sensitivity examines whether parameters require or handle sensitive information. Type of Operation distinguishes intrinsically safer operations from irreversible ones such as delete or transfer money. Impact Scope of the Operation measures how broadly misuse or error could propagate [2509.07315].

The **Joint Instruction-Tool Perspective** is the interaction layer. Alignment Between Instruction and Tool asks whether the requested use is consistent with the tool’s intended function or instead exploits it to bypass safeguards. Value Sensitivity addresses cases in which the specific tool output or action, under the provided parameters, may violate societal values, ethics, or laws. The paper reports that the largest performance gap is seen in handling risks that arise from this joint perspective, and that most errors are due to failures in the Joint Instruction-Tool perspective, indicating that cross-perspective interaction is the hardest part of the task [2509.07315].

## 3. Scoring procedure and execution-plan risk

SafeInstructTool operationalizes its nine dimensions with a modular, compositional scoring process. The procedure begins by precomputing tool or API risk scores with an LLM-powered judge and storing them in an API Safety Database. For the \(m^{\text{th}}\) API in the \(i^{\text{th}}\) app, the three tool-perspective scores are defined as
$$
\{t^1_{im}, t^2_{im}, t^3_{im}\} = \mathcal{M}(a^m_i),
$$
and aggregated as
$$
\mathcal{T}_{im} = \sum_{n=1}^{3} t^n_{im}.
$$
These stored scores enable real-time lookup during risk assessment [2509.07315].

Given an input instruction \(q\), the framework computes four instruction-side scores:
$$
\{u^1, u^2, u^3, u^4\} = \mathcal{M}(q),
$$
with aggregate instruction risk
$$
\mathcal{U} = \sum_{n=1}^{4} u^n.
$$
This explicitly separates instruction analysis from tool analysis rather than collapsing them into a single prompt-level judgment [2509.07315].

For each anticipated tool call \(l^m_i\) in the planned sequence \(p\), the framework then computes the two joint scores:
$$
\{c^1_{im}, c^2_{im}\} = \mathcal{M}(l^m_i),
$$
and
$$
\mathcal{C}_{im} = \sum_{n=1}^{2} c^n_{im}.
$$
The overall execution-plan risk is finally defined as
$$
\mathcal{S} = \mathcal{U} + \max_{a^m_i \in p} (\mathcal{T}_{im} + \mathcal{C}_{im}).
$$
If \(\mathcal{S}\) exceeds a calibrated threshold \(\alpha\), experimentally set often to \(\alpha = 10\), the tool calls are flagged as high risk and are blocked or routed for further confirmation before execution [2509.07315].

This aggregation rule is notable because it combines a global instruction-level score with the maximum local tool-plus-joint risk in the planned sequence. A plausible implication is that one especially dangerous step can dominate the plan-level decision even if other steps appear benign.

## 4. Experimental protocol and reported gains

The reported evaluation compares SafeInstructTool with four alternative methods: **None**, **Simple Prompt**, **CoT**, and **Self-Consistency**. The metric is **Safety score** \((\mathcal{K})\), defined as the percentage of risky instructions correctly identified or blocked. The experiments reported results for Llama3.1-8B-Instruct, Qwen2.5-7B-Instruct, Qwen2.5-14B-Instruct, Qwen2.5-32B-Instruct, GPT-3.5, and GPT-4o [2509.07315].

| Model | Best Baseline | SafeInstructTool | Improvement |
|---|---:|---:|---:|
| Llama3.1-8B-Instruct | 29.2% | 44.0% | +14.8% |
| Qwen2.5-7B-Instruct | 48.5% | 54.3% | +5.8% |
| Qwen2.5-14B-Instruct | 51.4% | 60.2% | +8.8% |
| Qwen2.5-32B-Instruct | 56.5% | 71.2% | +14.7% |
| GPT-3.5 | 50.1% | 56.7% | +6.6% |
| GPT-4o | 71.7% | 80.7% | +9.0% |

The framework is reported to outperform all baselines across all listed models and across all four risk categories. The improvement is described as more pronounced for open-source models and for more complex, multi-app scenarios. The ablation findings further state that all three perspectives are crucial, and that dropping any one perspective leads to significant performance degradation, with especially strong effects from removing the User Instruction and Tool Itself perspectives [2509.07315].

The diagnostic analysis reports that 47% of errors are due to failures in the Joint Instruction-Tool perspective. Traditional high-risk domains such as finance and healthcare remain challenging, and some implicit risks, especially those requiring causal or context-based inference, remain difficult even for top LLMs [2509.07315].

## 5. Interpretation, strengths, and stated limitations

The paper characterizes SafeInstructTool as **holistic**, **preemptive**, **modular and explainable**, **automatable**, and **scalable across domains/tools**. Holistic refers to its coverage of all three sources of risk: the user, the tool, and their interaction. Preemptive refers to risk analysis before action execution. Modularity and explainability derive from the explicit fine-grained scores assigned to the nine dimensions. Automatable reflects the LLM-driven scoring and the API Safety Database. Scalability follows from precomputing tool risk and extending the database as new APIs appear [2509.07315].

The framework is also positioned as advancing tool-use safety by moving beyond approaches that inspect only surface cues in user instructions or that evaluate safety only after execution. In that sense, SafeInstructTool is a structured decision framework for execution planning rather than a purely reactive content filter [2509.07315].

The stated limitation is that the method does not currently model user-specific risks, described as personalization issues such as allergies. This matters because some high-stakes tool decisions depend not only on generic risk categories but also on individual constraints that are not encoded in the current nine dimensions [2509.07315].

## 6. Relation to adjacent research on safe tool use

SafeInstructTool belongs to a broader line of work that treats tool-use safety as a systems problem rather than only a response-alignment problem. Within that landscape, it is specifically a *prospective scoring framework* for pre-execution decision-making [2509.07315].

Other work addresses related goals with different operational abstractions. Task Shield enforces task alignment at test time by requiring each instruction and tool call to contribute to user-specified goals, reporting an attack success rate of 2.07% and utility under attack of 69.79% on GPT-4o in AgentDojo [2412.16682]. ToolSafe focuses on step-level tool invocation safety through TS-Bench, TS-Guard, and TS-Flow, and reports a 65 percent average reduction in harmful tool invocations together with approximately 10 percent improvement in benign task completion under prompt injection attacks [2601.10156]. AgentTrust places a runtime interception layer between agent and tool, returning verdicts of allow, warn, block, or review, and reports 95.0% verdict accuracy on a 300-scenario internal benchmark at low-millisecond latency [2605.04785].

A different branch of work emphasizes formal guarantees. The STPA- and MCP-based framework in “Towards Verifiably Safe Tool Use for LLM Agents” derives hazards, formalizes them as information-flow and temporal specifications, and argues that if labels and rules are correct and enforced at boundaries, violations are provably impossible within the scope of the model [2601.08012]. Multi-turn safety work such as ToolShield addresses longer-horizon attacks by self-exploration of tools and reports a 30% average reduction in attack success rate in multi-turn interactions [2602.13379].

Against this backdrop, SafeInstructTool is distinguished by its explicit aggregation of instruction risk, tool-intrinsic risk, and joint instruction-tool risk into a single execution-plan score \(\mathcal{S}\). That design places it between prompt-only safety assessment and full runtime or formally verified control layers: it is pre-execution and structured, but not presented as a formal verification system. This suggests that it is best understood as a prospective risk-analysis framework for deciding whether a planned tool-use sequence should proceed, be blocked, or require confirmation [2509.07315].

Source: https://www.emergentmind.com/topics/safeinstructtool