- The paper introduces RoboAbstention, an auditable benchmark of 6,069 imageโinstruction pairs across eight grounded abstention categories, generated from 1,250 real-world images through structured visual grounding, deterministic constraints, and 171 templates.
- The paper finds that frontier VLMs rarely refuse invalid tasksโthe best baseline abstention rate is 39.0%, while Gemini Robotics ER reaches only 16.5%โwith models often guessing ambiguous targets or fabricating grounded action plans.
- The paper shows that defensive prompting and in-context examples raise abstention to 93.6% for Gemini Robotics ER and 88.6% for GPT-5.4 Mini, but residual errors and unmeasured false-abstention rates leave open questions about safe deployment.
Motivation and problem statement
Vision-LLMs (VLMs) increasingly serve as high-level planners for embodied robots, converting natural-language instructions and visual observations into action plans. In this setting, the authors define abstention as withholding execution when an instruction is ill-posed, ungrounded, ambiguous, or physically infeasible relative to the observed scene and the agent's capabilities. This differs from text-only abstention: it requires grounding language in perception and reasoning over object properties, spatial relations, states, and physical constraints. Failure to abstain can cause incorrect task execution, compounding errors in multi-step plans, and degraded interaction.
Existing abstention benchmarksโsuch as AbstentionBench [2507.xxxx], CLAMBER, QuestBench, and "The Art of Saying No"โare largely text-only or target embodied question answering. The closest embodied prior work, VLN-NF (false-premise navigation) and AbstainEQA (abstention in 3D indoor EQA), has two limitations the authors identify: manual annotation pipelines that are difficult to scale, and a focus on room-scale verbal responses that omits action-plan-specific categories such as physical infeasibility and contradictory instructions. Embodied safety benchmarks (BadRobot, RoboPAIR, SafeAgentBench, VestaBench) address adversarial misuse rather than benign but non-executable instructions, which is the gap this paper targets.
Taxonomy of embodied abstention
The paper organizes abstention scenarios into three groups formalized as eight categories:
- Reference Grounding: Missing Referent (target absent from scene), Ambiguous Referent (multiple instances match without distinguishing attributes), Underspecified Intent (deictic/pronominal references without context), Subjective Intent (requires user preferences not inferable from vision).
- Execution Feasibility: Physical Infeasibility (violates spatial/physical constraints), Missing Capability (requires unavailable sensing or actuation; the paper assumes only camera vision and gripper manipulation), Contradictory Instructions (mutually unsatisfiable requirements).
- False Premise: instructions presupposing a world state contradicted by the scene (e.g., "turn off the monitor" when it is already off).
RoboAbstention generation pipeline
The framework generates abstention instructions from real images via a three-phase pipeline designed to separate perception from reasoning:
- Structured visual grounding: Claude Sonnet 4.6 extracts a JSON scene representationโobjects, attributes, states, sizes, manipulability, locations, absent-and-implausible object classesโusing controlled vocabularies. Crucially, learned models are confined to perceptual description; their output is treated as an explicit intermediate representation, not ground truth.
- Deterministic constraint derivation: entirely rule-based code derives category candidates from the structured representation (e.g., size-class comparisons yield physically infeasible objectโlocation pairs; stateful objects whose current state already satisfies an action's outcome yield false-premise candidates).
- Controlled instruction generation: 171 category-specific templates are instantiated only with valid phase-2 candidates, using fixed random seeds for reproducibility.
The resulting benchmark contains 1,250 images drawn equally from DROID, Robo2VLM, RoboVQA, BridgeV2, and EgoThink, yielding 6,069 (image, instruction) pairs. Category coverage is uneven: Missing Referent (1,250) and Underspecified Intent (1,246) dominate, while Physical Infeasibility contributes only 50 instancesโa coverage imbalance worth noting when interpreting per-category results.
The design trades linguistic diversity for auditability: template-based generation produces less varied phrasing than free-form LLM generation, but every example is programmatically verifiable against its extracted scene representation. The authors explicitly concede that correctness is conditional on phase-1 grounding accuracy; errors there can propagate to labels.
Evaluation methodology
Responses are classified as act/abstain by an LLM-as-a-judge validated against human annotation. Four annotators labeled 200 instructionโresponse pairs (Fleiss ฮบ=0.871); GPT-5.4 Mini was selected as judge with 97.5% accuracy and abstain-F1 of 0.953, showing 99% run-to-run agreement (ฮบ=0.927). Evaluated models include GPT-5.4 (and Mini/Nano), Claude Sonnet 4.6, Gemini 2.5 Flash, Gemini Robotics ER 1.6 Preview, Qwen 3.5 27B, and Llama 4 Maverick.
Main findings
Frontier VLMs rarely abstain. The best model, Gemini 2.5 Flash, abstains on only 39.0% of instructions; Qwen 3.5 27B is lowest at 9.2%. Notably, Gemini Robotics ER 1.6 Previewโan embodied robotics-specific plannerโabstains on just 16.5%, worse than several general-purpose models. Per-category results show ambiguous referents elicit almost no abstention (2.6% across models), whereas missing referents reach 41.3%. Manual inspection indicates models confidently guess under ambiguity but resist naming objects that clearly do not exist. A striking implication is that domain specialization for robotics does not confer calibrated refusal; if anything it appears to bias planners toward execution.
Scale and reasoning do not help. Within the GPT-5.4 family, abstention rates are broadly similar across sizes. More surprisingly, increasing reasoning effort reduces abstention monotonically: GPT-5.4 Mini falls from 14.8% (low reasoning) to 9.8% (high reasoning). The authors interpret this as deliberation enabling plausible workarounds rather than recognition that no valid execution existsโconsistent with prior findings that reasoning LLMs fail on unanswerable questions.
Failure modes center on fabricated grounding. Clustering GPT-5.4's non-abstaining errors via LLM-as-a-qualitative-judge yields twenty behavior clusters dominated by converting uncertainty into unwarranted actionability: Arbitrary Target Selection (n=964), Overconfident Grounding (n=702), Invented Target (n=430), and Search-and-Act (n=287). Other clusters include tactile workarounds for missing sensors (n=188), collapsing contradictory constraints into ordinary plans (n=180), and treating subjective preferences ("favorite," "ugliest") as objective visual properties (n=110).
Prompting mitigations are large but incomplete. On a 1,000-task subset, defensive prompting plus in-context learning raises abstention to 93.6% for Gemini Robotics ER 1.6 (from 18.2%) and 88.6% for GPT-5.4 Mini (from 15.4%). Defensive prompting alone reaches 90.8%/88.3%; ICL alone is more asymmetric (90.5% vs. 76.0%). These gains demonstrate that abstention failures are substantially prompt-addressable, yet residual failures persist even under combined mitigation, and such interventions risk over-triggering abstention on genuinely executable tasksโthe paper does not report false-abstention rates on valid instructions, which is an open question.
Limitations
The authors identify three principal limitations. First, label correctness depends on the phase-1 VLM scene extraction; missed objects or wrong attributes propagate into abstention conditions, so human validation remains necessary. Second, coverage is restricted to static imageโinstruction pairs with an assumed vision-plus-gripper embodiment; multi-turn clarification, long-horizon interaction, temporal scene change, and other sensor suites are out of scope. Third, template-based instruction generation limits linguistic diversity relative to unconstrained generation. Additionally, the LLM-as-a-judge, though validated against humans, may introduce classification noise, and because the dataset is public with fixed templates, future models may overfit its surface patterns.
Conclusion
This paper introduces RoboAbstention, a scalable, auditable benchmark of 6,069 visually grounded abstention instructions across eight categories, built by separating VLM-based perception from deterministic constraint derivation and templated generation. Its central empirical result is that frontier VLMsโincluding purpose-built robotic plannersโdefault to action under invalid or underspecified instructions, with best-case baseline abstention of 39.0%, and that higher reasoning effort actively worsens calibration. Simple prompting interventions recover most of the deficit but leave residual failure modes rooted in hallucinated grounding. The work positions abstention as a capability that should be measured explicitly in embodied AI evaluation, while leaving open how well these gains transfer to interactive settings and whether abstention can be improved without degrading execution of valid tasks.