HumanCLAW: Can Vision-Language Models Act Through a Body?

Abstract: Evaluating whether a vision-LLM (VLM) can act through a physical body is challenging. The outcome of an action couples the VLM's decision with motor control. When a task fails, it is hard to tell whether the VLM made a bad choice or the motor controller simply failed to execute it, e.g., losing balance and falling. In this work, we introduce HumanCLAW, an evaluation framework that decouples action decision-making from low-level execution. At every step, a harnessed, off-the-shelf VLM issues an atomic skill command, and the command is translated into a sub-second chunk of continuous full-body motion with real physical consequences, including gravity and collisions. The body can therefore act freely in the physical world, while execution-side disturbances, balance and motor errors, are factored out. What remains measurable is the model's action intelligence: its moment-to-moment choice of what the body should execute next. Based on this framework, we build HumanCLAW-Bench: 1,218 long-horizon, egocentric find-navigate-interact episodes across 41 indoor scenes. We test nine state-of-the-art VLMs and find that none solves the benchmark; the best model reaches only a 16.8% success rate. Recognizing the target is not the bottleneck. What current VLMs lack is embodied self-awareness: they lose track of their own body, failing to tell where it is, whether it has reached the goal, or whether it has hit an obstacle.

• The paper reveals that even advanced vision-language models, such as Gemini-3.1, achieve strong visual recognition but struggle with embodied tasks due to poor self-localization.
• It decouples high-level decision making from low-level motor control, enabling clear attribution of failures to planning processes rather than motion execution.
• Empirical results on HumanCLAW-Bench highlight significant performance drops during navigation and interaction stages, indicating a need for improved embodied self-awareness and proprioceptive integration.

HumanCLAW: Evaluating Whether Vision-LLMs Can Act Through a Body

Introduction and Motivation

Recent advances in vision-LLMs (VLMs) have led to marked improvements in visual grounding and spatial reasoning, but the question of whether these models can instantiate plans as embodied actions—effectively “acting through a body”—remains open. HumanCLAW directly addresses this by decoupling high-level action decision-making from low-level motor control, providing a benchmark and framework to evaluate “action intelligence,” defined as the capacity for closed-loop, moment-to-moment bodily decision-making given egocentric perception and a specified task. HumanCLAW situates this evaluation in the context of long-horizon, physically grounded tasks where repercussions of decisions unfold through real-time interaction with the environment.

Framework: Decoupling Decision from Execution

The primary contribution of HumanCLAW is its harness architecture, which leverages off-the-shelf VLMs as frozen decision-makers that issue atomic skill commands at sub-second intervals. Each step consists of: (1) an egocentric RGB observation, (2) textual action history, and (3) a high-level language instruction. The VLM outputs an atomic skill (e.g., walk, turn, sit) and associated parameters, which are then instantiated as a continuous motion chunk by a skill-conditioned motion generator. The subsequent body-environment interactions are realized with a “half-physics” simulator—physical consequences such as collisions, gravity, and object displacement are modeled, but balance failures and low-level actuation noise are factored out.

Crucially, skills are parameterized and executed deterministically with no model retraining, allowing plug-and-play extensibility and attributability of failures to the VLM’s action logic rather than locomotion artifacts. This design ensures that action intelligence is probed in a controlled yet physically valid regime, without being obfuscated by brittle low-level controllers.

HumanCLAW-Bench: Task and Evaluation Protocol

HumanCLAW-Bench embodies the core test: agents complete 1,218 find-navigate-interact episodes across 41 diverse, multi-level indoor scenes. Each episode is initiated by a high-level instruction (e.g., “find a sofa and sit on it”)—leaving every execution detail, navigation choice, and motor coordination to the agent. Tasks are stratified in difficulty along geodesic path length, navigational branchiness, and environmental clutter.

Progress is scored in staged fashion:

• FindSR: Determines if the agent both sees and acknowledges the target object with sufficient semantic pixel coverage.
• NavSR: Assesses whether the agent brings its body within 20 cm of the target and explicitly ends navigation.
• InteractSR: Verifies if the agent executes a sit such that the pelvis achieves physical contact with the seat.

Additional metrics quantify bodily awareness (collision frequency, object disturbance) and action quality (motion jerk), ensuring both decision and execution are comprehensively assessed. The system also logs token consumption to facilitate cost analysis.

Technical Methodology: Structured Action Reasoning and Motion Generation

Action Reasoning Scaffold

The VLM decision harness is engineered for explicit contextual prompting and structured reasoning:

• Context Injection: The VLM is primed with agent identity, task goal, available skills, and recent history (actions, observations, reasoning traces), providing temporal coherence and spatial context.
• Ego-Perception Module: Before skill selection, the VLM must output a parsed representation of the egocentric scene, covering object layout, distances, target visibility, and environmental affordances.
• Hierarchical Reasoning: Decision-making proceeds from high-level intent (e.g., search vs. interact) to mid-level objectives (e.g., approach, align) and finally to atomic skill selection, with each transition checked for spatial plausibility.
• Skill-Specific Verifier: After a skill is proposed, a compact check verifies its spatial safety (e.g., not walking into an obstacle, sitting only when a seat exists), mitigating long-context hallucinations.

Skill-Conditioned Motion Synthesis

Motion generation employs a DiT backbone, trained on AMASS/BABEL with receding-horizon flow matching, and ControlNet-style adapters per skill for zero-shot parameter adherence. This architecture guarantees:

• Deterministic, Reliable Skill Execution: Ratios of achieved-to-commanded movement (step distance, sitting height, turning angle) are close to unity for all skills, with minimal variance.
• Natural Motion Transitions: Smooth blending of skill outputs, supporting complex bodily maneuvers while maintaining physical realism.
• Adapter Modularity: New skills are incorporated by attaching new adapters, preserving the frozen nature of the base and other skills.

Empirical Results: Quantitative and Qualitative Analysis

Key Findings

1. VLMs Fail at Embodied Tasks Beyond Perception

Despite strong performance in target recognition (FindSR up to 64.9%), all nine tested VLMs achieve low overall task success on HumanCLAW-Bench. The state-of-the-art, Gemini-3.1, completes only 16.8% of full find-navigate-interact episodes. Performance degradation is sharp at each transition: even when models recognize the target, the “landing rate” (NavSR/FindSR) is low (e.g., 0.65 for Gemini-3.1). The bottleneck is post-perception: VLMs are unable to accurately track their own body relative to the target and environment.

2. Embodied Self-Awareness is the Bottleneck

Failure analysis reveals that once perception succeeds, egocentric self-localization and body awareness account for the majority of subsequent failures (up to 81% at the interaction stage). Agents commonly stop short of the target, misjudge arrival, jam into obstacles, or attempt to interact while misaligned spatially—highlighting an inability to maintain and update an online belief over the embodied state.

3. High-Fidelity Motion Execution; Decision Failures are Attributable

The motion synthesis stack is robust: skill execution tracks parameterized commands closely (achievement ratios within 0.74–1.00, variance ≤ 0.12), outperforming text-conditioned baselines such as MoMask. Therefore, decision intelligence, not actuation fidelity, is the clear limiting factor.

4. Scaffolding and Verification are Essential

Ablation studies show that memory and mid-level objective scaffolding, plus per-step verification, are necessary for action intelligence. Removal of the verifier, for instance, drops navigation and interaction success drastically, even when perception remains stable. Excessive history or visual context can degrade performance, reflecting limitations in VLM memory and reasoning.

5. Body Awareness is Poor Across All Models

Collision analysis indicates frequent, unacknowledged contact by lower limbs and arms (28–45% collision frequency for feet/legs; 20–35% for arms/hands), signifying a lack of proprioceptive reasoning and awareness of one’s own body in space.

Table: VLM Performance Overview

Model	FindSR	NavSR	InteractSR	Motion Jerk	Coll.%	#Dtb/ep	Steps/ep
Gemini-3.1	64.9%	42.4%	16.8%	5.7	39.5	1.50	59.1
GPT-5.5	55.1%	13.9%	3.4%	4.2	43.4	1.53	82.4
Gemma-4-31B	58.1%	28.7%	11.1%	4.8	40.1	1.66	78.5
Other Models	≤51.0%	≤21.6%	≤3.5%	≥4.7	≥34.5	≥1.08	≥37.6

(Values are for the principal success metrics and show the gap between perception, navigation, and true embodied interaction.)

Theoretical and Practical Implications

HumanCLAW demonstrates that, under realistic physical assignment of bodily actions, contemporary VLMs exhibit a consistent deficit in embodied self-awareness—a capacity essential for generalist embodied intelligence. Critically, the performance gap is not one of observation or low-level actuation, but of maintaining persistent online self-localization, spatial belief over body-environment state, and ability to reason action-to-effect. This highlights an emerging research frontier: infusing VLMs with mechanisms akin to proprioception, persistent spatial memory, and consequence prediction.

From a practical perspective, HumanCLAW-Bench provides a clean, reproducible, and open venue to evaluate progress in embodied intelligence. It reveals that conventional scaling and context extension is insufficient; architectural and training advances targeting spatial self-modeling and cross-modal feedback integration are necessary.

Future Directions

The results suggest several promising research avenues:

• Explicit Proprioceptive Input Integration: Augmenting VLMs with body pose, joint state, or contact signals to induce self-modeling.
• Scene-Body Fusion Architectures: Developing models capable of jointly reasoning about scene layout and embodied state, possibly via memory-augmented networks or spatial transformers.
• Hierarchical and Long-Horizon Reasoning Enhancements: Improved scaffolding to sustain multi-step, multi-modal plan coherence and predict action outcomes in diverse, dynamic environments.
• Direct Embodiment Supervision: Targeted pretraining on egocentric datasets that include explicit bodily awareness or causal physical interaction labels.

Conclusion

HumanCLAW robustly benchmarks the embodied action capabilities of state-of-the-art VLMs, decoupling decision quality from actuation and exposing the clear bottleneck of embodied self-awareness. Despite strong perception, VLMs are unable to reliably locate, navigate to, and interact with objects using a body; the principal limitation is their inability to reason about their own spatial state and action consequences. The work highlights new evaluation standards and inspires a shift toward AI architectures that can truly inhabit and act through physical bodies, a prerequisite for scalable general intelligence in the real world (2607.27180).