- The paper introduces agentic abstention, enabling LLM agents to deliberately refrain from risky or ambiguous actions.
- It presents an eight-category taxonomy and an automated paired task design (AbstainGen) to evaluate over- and under-abstention.
- Experiments show that improved task performance does not ensure calibrated restraint, underlining a crucial safety gap in LLMs.
"AgentAbstain: Do LLM Agents Know When Not to Act?" (2607.10059) introduces the first systematic approach for measuring the calibrated restraint of tool-using LLM agents. The paper addresses a critical yet neglected property in autonomous agents: the ability to abstain from action when a task is incorrectly specified, ambiguous, infeasible, or risky. Most existing benchmarks evaluate task success but ignore whether agents can recognize and refuse dangerous, ill-defined, or unjustified actions—an omission with significant implications for deploying LLM agents on consequential real-world tasks with side effects.
The central notion is agentic abstention: the capability of an LLM agent to make a deliberate and context-sensitive choice to not act—by explicit refusal or clarification request—when executing an action would be harmful, logically incoherent, or epistemically unsafe. The evaluation is grounded in concrete executable environments, with a formal separation of pre-execution (triggering abstention before execution) and runtime (recognizing signals only after partial environment interaction) triggers. The authors operationalize this within an original benchmark—AgentAbstain—which systematically measures both over- and under-abstention using paired control/counterfactual task design.
Benchmark Framework and Methodology
Abstention Taxonomy
The work presents an eight-category taxonomy of abstention situations, partitioned by when a sensible agent should refuse:
- Pre-execution triggers:
- S1: Missing Critical Parameter
- S2: Ambiguous Action Specification
- S3: Conflicting Constraints
- S4: High-Stakes Action
- S5: Insufficient Tool Capability
- Runtime triggers:
- S6: Critical Tool Failure
- S7: Conflicting Evidence
- S8: Emergent Risk Discovery
This stratification is necessary to differentiate failures in initial intent recognition from those requiring iterative inspection of world/tool state. Tasks are thus not limited to static QA but require sequential tool usage and environment state manipulation or inspection.
Paired Task Design
Each benchmark instance is a pair: a should-act task (T+) and a minimally perturbed should-abstain variant (T−) differing only in a single trigger dimension (instruction, environment, or toolset). Success requires agents to act when appropriate but to demonstrate restraint when the only change is a nontrivial abstention-relevant perturbation. By construction, always-act or always-refuse baselines are capped at 50% Paired Accuracy.
Automated Generation: AbstainGen
Manual generation of realistic, contamination-robust, paired scenarios is infeasible at scale. The authors contribute AbstainGen, a fully automated pipeline for synthesizing sandbox environments and generating paired tasks (validated with both deterministic replay and inter-LLM critics), enabling efficient creation and regeneration of fresh evaluation data.

Figure 1: Macro-level model comparison on Paired Accuracy, Act Accuracy, Abstain Accuracy, and CAR, averaged over the eight abstention categories.
Experimental Setup
Dataset Composition
AgentAbstain includes 263 paired tasks (526 unique tasks) across 42 LLM-generated sandbox environments. Task balance is enforced across abstention categories and between informational (read-only) and operational (environment-mutating) deliverables. Tool exposure, scenario mapping, DAG complexity, and environment diversity are systematically reported, underscoring robust coverage.
Model and Harness Coverage
The evaluation spans 17 frontier models from four major APIs (OpenAI, Anthropic, Google, OpenClaw), each exposed to identical Model Context Protocol (MCP) tool surfaces. For each task, models are executed once, and four primary metrics are reported: Act Accuracy, Abstain Accuracy, Paired Accuracy (primary), and Conditioned Abstention Rate (CAR; abstain accuracy conditioned on successful action ability).
Main Findings
Bounded Agentic Abstention
No model crosses 60% Paired Accuracy; the best (Gemini 3.1 Pro) achieves 59.5%, with most others in the 33–53% range. Abstention skill is not correlated with task-solving skill: models that demonstrate high Act Accuracy can perform poorly on abstention, with up to a 21-point gap between act and abstain performance. This fact is visible both in the macro statistics and in per-category, per-model scatter analyses.

Figure 2: Generational scaling within model families reveals that act and abstain improvements are semi-independent—improvements in task completion do not reliably translate to calibrated restraint.
Category-Differentiated Failure
Paired accuracy is far more sensitive to deficiency type than to execution phase. Informational gaps and logical contradictions (e.g., missing parameters, S1; conflicting constraints, S3) are substantially harder for models to recognize than capability limitations (S5, S6); runtime triggers are not uniquely challenging compared to pre-execution ones. This strongly motivates deficiency-aware supervision for calibrating agentic restraint.
Action-Type Agnosticism
Models are not more reliable at abstaining when tasks mutate state than when they merely return information; the failure rates (and the failure types) are statistically similar across operational and informational categories. This points to the absence of robust built-in risk aversion regarding irreversible state changes in today’s LLM agents.

Figure 3: Models tend to perform worse on operational tasks compared to informational tasks, as shown by the majority of points lying below the diagonal.
Independence of Task Competence and Calibration
The φ coefficient between act and abstain pass/fail outcomes on the same task pair is negative for the majority of models, often significantly (down to −0.24), demonstrating that improvements in core capability (scale, context length, etc.) are largely orthogonal to act/abstain calibration. This anti-correlation persists both within and across model families and over generations: progress in solving tasks does not transfer to knowing when not to act.
Failure Mode Taxonomy
Abstention failures are not monolithic:
- No Abstention: Agent acts and fails to recognize the need to stop (confidently wrong output).
- Implicit Abstention: Agent withholds action but does not provide a human-actionable reason (silent halt, no verbal signal).
- Post-hoc Abstention: Agent acts anyway and then claims to have abstained, combining irreversible side effects with false assurance.
- Partial Abstention: Only dangerous steps blocked, but benign steps executed.
Notably, approximately 2.6% of abstain runs execute a critical action and then issue verbal refusal—a high-severity deployment risk, as the user receives ex post rationalization rather than protection from harm.

Figure 4: Per-model breakdown of qualitative failure modes for abstain tasks, showing variance in failure type distribution.
Open-vs.-Closed Model Effects
Closed-source models show significantly higher abstention accuracy (9.4pp gap), especially for runtime-initiated triggers such as Critical Tool Failure, Emergent Risk Discovery, and High Stakes Action. Open-weight models have a higher rate of Implicit Abstention, indicating a lack of actionable signaling.
Implications and Future Directions
The results demonstrate that calibrated restraint is a distinct capability, not an automatic byproduct of increased action or planning proficiency. Agentic abstention must thus be treated as a separate training and evaluation target—distinct from end-task success or general safety refusal. Existing alignment and safety protocols, as well as RLHF, are insufficient for critical deployment scenarios where “knowing when not to act” is as essential as raw capability.
Practically, the lack of calibrated abstention undermines trustworthiness for high-stakes deployments. The taxonomy and benchmark also clarify that future systems will need scenario-sensitive act/abstain classifiers, finer-grained trigger detection, and explicit abstention policy supervision. Automated regeneration via AbstainGen further offers a path towards evaluation sets resistant to data contamination and superficial overfitting.
Conclusion
"AgentAbstain" represents a formal and comprehensive step toward measuring and developing calibrated restraint in tool-using LLM agents. The empirical results unambiguously indicate that agentic abstention lags far behind task completion, and that advancement in model core metrics does not guarantee progress on this critical safety axis. Thus, explicit, scenario-oriented abstention training and measurement is required for trustworthy deployment of autonomous agents with real-world side effects.
The release of the AgentAbstain benchmark and the AbstainGen pipeline establishes a foundation for further research, including abstention-aware fine-tuning, interpretability for refusal behavior, and investigation into the interaction between task definitions, environment realism, and agent calibration. Future progress will depend on recognizing and addressing the independence of action capability and calibrated restraint—an insight quantifiably demonstrated by this framework.