RoboInspector: Diagnosing LLM Robotic Policies

• RoboInspector is a systematic pipeline that evaluates the reliability of LLM-generated robot policy code by analyzing the interplay between task complexity and instruction granularity.
• It categorizes failures into four classes—nonsense, disorder, infeasible, and badpose—providing actionable diagnostic insights for each issue.
• The feedback-guided refinement method improves manipulation success rates by up to 35%, demonstrating practical gains across diverse robotic frameworks.

RoboInspector is a systematic pipeline for diagnosing and improving the reliability of policy code generated by LLMs for robotic manipulation. The system focuses on uncovering failure modes in LLM-generated robot policy code by analyzing the interplay between the complexity of manipulation tasks and the granularity of user instructions. It provides a detailed taxonomy of unreliable behaviors, experimental quantification across a broad suite of tasks, instructions, and models, and demonstrates how structured feedback can refine LLM outputs for practical robotic manipulation (Ying et al., 29 Aug 2025).

1. Systematic Pipeline Overview

RoboInspector is designed to evaluate the reliability of LLM-generated policy code for controlling robotic manipulation tasks. Given the growing use of LLMs to generate controller code from natural language instructions, RoboInspector exposes how variations in task complexity and instruction granularity affect execution reliability.

• Task Decomposition: Robotic manipulation tasks are represented as compositions of primitive actions: Grasp, Move, and Rotate.
• Instruction Granularity: User instructions are categorized into levels: Iₐ (object/action only), Iₚ (object/action/purpose), and I𝒞 (object/action/purpose/condition).
• Frameworks Evaluated: The pipeline is validated across prominent manipulation frameworks, specifically VoxPoser and Code as Policies, and spans 168 combinations of tasks, instructions, and LLMs.
• Reliability Modeling: The impact of task complexity and instruction detail on reliability is quantified by the relation: $$R(f_\phi(T), f_\theta(I)) \propto \frac{f_\theta(I_i)}{f_\phi(T_i)}$$ where $R$ is reliability, $f_\theta(I_i)$ captures instruction granularity, and $f_\phi(T_i)$ quantifies task complexity.

This approach both benchmarks LLM system performance and reveals underlying causes of failure, enabling targeted intervention during robotic system design.

2. Failure Modes and Unreliable Behaviors

RoboInspector characterizes four primary classes of unreliable behavior encountered in LLM-generated policy code for manipulation:

Behavior	Description	Typical Triggers/Contributors
Nonsense	Generates non-conformant outputs (extraneous text, import statements, or extraneous descriptions)	Associated with models weak on instruction following
Disorder	Misorders substeps (e.g., incorrect sequencing in multi-action tasks)	Low-granularity instructions (Iₐ); weak context parsing
Infeasible	Action violates robot's workspace constraints (unreachable goals)	Perception modules with broad sensing, low granularity, task ambiguity
Badpose	Misaligned end-effector poses (e.g., grasp misalignment)	Oversimplified control models ignoring geometric details

• Nonsense arises when the model outputs text or code irrelevant to policy execution, exacerbated when LLMs ignore prompt structure or formatting.
• Disorder is especially prevalent for tasks with compositional or sequential requirements and coarse instructions lacking purpose or logical flow cues.
• Infeasible policies issue control commands outside workspace bounds; higher-granularity constraints in instructions (I𝒞) prompt the LLM to detect and avoid such cases.
• Badpose occurs when control logic treats both robot and objects as points, disregarding the proper end-effector orientation; this affects tasks needing precise spatial alignment.

These categories form the basis for both quantitative and qualitative failure analysis, guiding systematic improvement of LLM-driven policy generation.

3. Experimental Design and Evaluation Metrics

RoboInspector’s evaluation spans:

• Tasks: Eight manipulation tasks sampling combinations of primitive actions (Grasp, Move, Rotate), including complex multi-step manipulations such as ChangeClock, SlideBlockToTarget, PutRubbishInBin, and OpenWineBottle.
• LLM Models: Eight LLMs are tested, including the OpenAI GPT suite (3.5-turbo, 4, 4o, 4o-mini), Alibaba Cloud’s Qwen series (max, plus, turbo), and DeepSeek-V3 (open source).
• Instruction Granularity: Each task is prompted with instruction variants Iₐ, Iₚ, and I𝒞 to tease out the role of specificity in policy generation.
• Frameworks: VoxPoser and Code as Policies frameworks are used, implementing both simulated (RLBench, PyBullet) and real-world 6-DoF robotic arm controllers.
• Measurement: Success rates are computed as the proportion of fully executed tasks (e.g., Grasp with Iₐ, GPT-3.5-turbo in VoxPoser achieves 46% success). Each setting is evaluated over 168 configurations, with 50 trials per configuration.

This comprehensive coverage surfaces correlations between instruction detail, task difficulty, LLM type, and failure modes, with high granularity instructions consistently improving reliability metrics.

4. Feedback-Guided Refinement Method

A core contribution is the refinement approach driven by policy-failure feedback:

• Error Diagnosis: Execution failures are detected, classified according to the unreliable behaviors (nonsense, disorder, infeasible, badpose), and execution is halted with return to the initial robot pose.
• Feedback Prompt Construction: The failing policy code and a detailed error description are packaged into a prompt that is returned to the LLM.
• Iterative Code Regeneration: The LLM is reprompted using both task specification and failure feedback, encouraging it to avoid the previous source of unreliability.
• Empirical Gains: Applying this loop improves the manipulation success rate by up to 35%, as measured in both simulation and real-robot settings. This refinement is particularly effectual in reducing disorder and nonsense behaviors for lower instruction granularity.

This feedback mechanism provides a practical engineering tool to increase task-level reliability despite initial LLM policy generation flaws.

5. Analysis of Model and Instruction Sensitivities

The empirical analysis conducted by RoboInspector reveals:

• Model Selection: LLMs such as GPT-3.5-turbo and Qwen-turbo are more susceptible to generating “nonsense,” whereas models better trained in instruction adherence perform more reliably under equivalent conditions.
• Instruction Design: Elevating instruction granularity (from object/action to purpose/condition) markedly reduces disorder and infeasible outcomes, underscoring the need for detailed prompting or automated template completion in instruction pre-processing.
• Task Complexity: More complex manipulations (e.g., with numerous sequential or composite actions) expose LLMs to higher error rates, especially without sufficient instruction granularity or control system sophistication.
• Control Algorithm Limitations: Observed “badpose” failures highlight the need for control algorithms and perception modules that fully capture the spatial geometry of objects and grippers, beyond current point-based simplifications implemented in some frameworks.

This systematic breakdown enables practitioners to identify the limiting factor—be it the LLM, instruction structure, or control model—for a given class of manipulation task.

6. Implications and Future Research Directions

The RoboInspector pipeline motivates several avenues for further development:

• Deeper Physical Constraint Integration: Advancing control frameworks to better incorporate the true physical configuration and shape of robot and objects may mitigate “badpose” and related misalignment failures.
• Dynamic Prompting: Adaptive strategies that auto-adjust prompt granularity based on observed failure type or task complexity could further improve LLM policy robustness.
• Broader Model Benchmarking: Further studies are suggested quantifying resilience across diverse LLM architectures, especially under adversarial instruction variation.
• Real-World Transfer: Expanding trials beyond laboratory settings to variable and unstructured environments will provide greater assurance of system reliability in deployment.
• Multimodal Sensing/Inference: Combining LLM-generated policy with real-time VLM (vision-LLM) feedback could improve failure detection and course correction during manipulation execution.

These lines of research are essential to maturing LLM-enabled robotic manipulation into systems that combine the versatility of code generation with the reliability required in physical interaction domains.

7. Conclusion

RoboInspector provides a rigorous, experimentally validated framework for exposing, characterizing, and addressing unreliability in LLM-generated policy code for robotic manipulation (Ying et al., 29 Aug 2025). By identifying the interaction among user instruction granularity, task complexity, and model sensitivity, and by introducing a feedback-driven refinement method, the approach yields measurable gains in manipulation success. This systematic pipeline offers both diagnostic and remedial capabilities, leading toward more robust, adaptive, and reliable LLM-driven robotic systems in the evolving landscape of automation and embodied AI.