- The paper introduces PhysCaP, a code-as-policy robot agent that uses proprioception-based mass and stiffness estimation plus Planner–Prioritizer exploration to resolve hidden physical properties.
- PhysCaP achieves 78% success in simulation and up to 90% on real-world tasks while reducing interactions and execution time compared with exhaustive exploration, including 300 seconds versus 516 seconds for avocado picking.
- The results show that separating exploration planning from candidate prioritization improves efficiency, although API latency, visual grounding errors, and limited task diversity restrict current generalization.
PhysCaP addresses a specific failure mode of current robot learning systems: tasks whose success depends on latent physical properties—mass, stiffness, occlusion state—that cannot be inferred from passive visual observation. The paper argues that vision-language-action (VLA) policies trained via imitation reproduce demonstrated behaviors but do not perform information-seeking interactions, and that reinforcement learning approaches to active perception suffer from sparse, delayed rewards and poor interpretability. The authors instead extend the code-as-policy (CaP) paradigm with an explicit exploration layer, producing an agent that decides when to interact, what to interact with, and when it has gathered sufficient evidence to act (2608.21031).
System architecture
PhysCaP builds on CaP-Agent0 as its code-generation backbone and adds three components. First, two training-free physical property extraction (PhysX) modules estimate latent quantities from proprioception alone, requiring no tactile or force sensors. The get_mass module executes a fixed 15 cm lift trajectory twice—once empty, once grasping the object—and computes mass from the differential joint torque Δτ projected through the vertical component of the end-effector Jacobian: m^=(Jz⋅Δτ)/(g∣Jz∣2), yielding a pose-robust estimate grounded in the principle of virtual work. The get_stiffness module performs a two-phase squeeze routine: a contact-verification phase that uses a backoff test to distinguish genuine elastic restoring force from gripper friction, followed by a closing phase that records displacement–effort pairs up to a normalized effort threshold of f∗=0.5, interpolating deformation Δd and mapping it to five discrete stiffness levels via calibrated thresholds. Each stiffness measurement is repeated five times per session with majority voting.
Second, a Planner agent evaluates whether visible information suffices for the task; if not, it emits a JSON list of exploration candidates specifying target objects, required properties, expected information, and estimated cost. Third, a Prioritizer agent filters implausible candidates using visual heuristics and ranks the remainder by priority score—for example, deprioritizing sealed cans in favor of open ones with straws. This dual-agent decomposition is itself an empirical finding: a merged single-agent variant (PhysCaP-joint), given both prompts jointly, degrades into unstructured exhaustive plans and underperforms the split design on all three real-world tasks.
Experimental results
Evaluation covers three real-world tabletop tasks on a 7-DoF AgileX PiPER arm (Find Blue Cube, Identify Empty Can, Pick Ripe Avocado) over 10 trials each, plus a simulated LIBERO replication of the empty-can task over 50 trials. All agents share Gemini 3.1 Pro as backbone; baselines ablate the system progressively: CaP (passive only), CaP+PhysX (exhaustive interaction), CaP+PhysX+Planner (stopping criterion, random ordering), PhysCaP-joint, and full PhysCaP.
The headline results show a clear efficiency advantage without sacrificing success rate:
| Method |
Task 2 SR / OI / Time |
Task 3 SR / OI / Time |
| CaP |
2/10 / 2.5 / 72 s |
1/10 / 1 / 26 s |
| CaP+PhysX |
7/10 / 3.9 / 268 s |
9/10 / 4 / 516 s |
| CaP+PhysX+Planner |
7/10 / 4 / 274 s |
8/10 / 4.125 / 563 s |
| PhysCaP |
8/10 / 2.5 / 239 s |
9/10 / 2 / 300 s |
On Task 1 (Find Blue Cube), PhysCaP reduces average interactions from 3 (exhaustive lifting) to 1.33 and execution time from roughly 77 s to 40 s by eliminating geometrically infeasible cups before touching them. The paper's central claim—that the Prioritizer is responsible for the efficiency gain rather than the Planner alone—is supported by the fact that adding a stopping criterion without prioritization actually increases interactions relative to exhaustive PhysX on Tasks 2 and 3.
An oracle test replacing hardware-derived masses with ground truth raises Identify Empty Can success from 8/10 to 10/10, isolating residual failure to mass-estimation noise rather than reasoning capacity. Module-level validation shows stable relative-mass discrimination across calibration weights spanning 13 g to 963 g (20 trials each), and stiffness measurements on a custom rubber-band-tensioned button mechanism align with avocado ripeness continua across ten trials per maturity category.
In simulation, VLA baselines fail almost entirely under partial observability: OpenVLA achieves 0%, π0.5 4%, and MolmoAct2 23% success, while CaP-based active methods reach 62–78%. PhysCaP attains the best simulated success rate (78%) with near-minimal interactions (OI 1.44). Because the VLAs were evaluated with their released LIBERO checkpoints on a task outside their training distribution's observability assumptions, this comparison primarily demonstrates the general point that feedforward visuomotor policies lack mechanisms for hypothesis-driven exploration; it does not establish that VLAs could not learn such behaviors with appropriate training data.
A backbone ablation on the empty-can task shows the framework is model-agnostic and that Claude Opus 4.8 (10/10 SR, OI 2.4, 217 s) outperforms the default Gemini 3.1 Pro (8/10, OI 2.5, 239 s), indicating reported main results may understate the architecture's ceiling.
Limitations
The authors identify three concrete limitations. Reliance on commercial VLM APIs introduces unpredictable latency and reasoning variability; locally hosted open-weight models would be needed for operational consistency. Object localization depends on Molmo2 2D pointing back-projected through a single ZED 2i depth camera, so 2D grounding errors propagate directly to 3D end-effector precision—a multi-view or 3D-native perception stack is left unimplemented. Finally, communication latency with the PiPER arm occasionally causes executed trajectories to diverge from generated code, an unresolved real-time control issue. Beyond these, the evaluation tasks use standardized commercial objects and pick-and-place primitives exclusively, so generalization to deformable objects beyond fruit, cluttered scenes, or contact-rich manipulation remains untested. Whether the heuristic-driven Prioritizer scales beyond small candidate sets, where learned value models might be required, is an open question the paper does not address.
Conclusion
PhysCaP demonstrates that augmenting code-as-policy agents with training-free proprioceptive property estimation and a structured Planner–Prioritizer exploration loop yields higher success rates with substantially fewer physical interactions than both passive and naive interactive alternatives, across real-world and simulated partially observable tasks. Its principal contribution is architectural: separating the decision of whether to explore from the decision of what to explore first measurably improves efficiency, and sensor-free physics modules can substitute for dedicated tactile hardware in constrained estimation regimes.
