VOFA: Visual Object Goal Pushing with Force-Adaptive Control for Humanoids

Abstract: The ability to push large objects in a goal-directed manner using onboard egocentric perception is an essential skill for humanoid robots to perform complex tasks such as material handling in warehouses. To robustly manipulate heavy objects to arbitrary goal configurations, the robot must cope with unknown object mass and ground friction, noisy onboard perception, and actuation errors; all in a real-time feedback loop. Existing solutions either rely on privileged object-state information without onboard perception or lack robustness to variations in goal configurations and object physical properties. In this work, we present VOFA, a visual goal-conditioned humanoid loco-manipulation system capable of pushing objects with unknown physical properties to arbitrary goal positions. VOFA consists of a two-level hierarchical architecture with a high-level visuomotor policy and a low-level force-adaptive whole-body controller. The high-level policy processes noisy onboard observations and generates goal-conditioned commands to operate in closed loop across diverse object-goal configurations, while the low-level whole-body controller provides robustness to variations in object physical properties. VOFA is extensively evaluated in both simulation and real-world experiments on the Booster T1 humanoid robot. Our results demonstrate strong performance, achieving over 90% success in simulation and over 80% success in real-world trials. Moreover, VOFA successfully pushes objects weighing up to 17kg, exceeding half of the Booster T1's body weight.

• The paper introduces a hierarchical framework that decouples vision-based planning from force-adaptive whole-body control.
• The paper demonstrates robust sim-to-real transfer with over 90% success rates in pushing tasks across varying object masses.
• The paper highlights the critical role of force adaptation in enabling recovery from disturbances and handling diverse object properties.

Visual Object Goal Pushing with Force-Adaptive Control for Humanoids

Introduction

The VOFA framework ("VOFA: Visual Object Goal Pushing with Force-Adaptive Control for Humanoids" (2605.01518)) addresses the task of autonomous closed-loop non-prehensile object pushing for humanoid robots utilizing visual goal conditioning and robust force-adaptive control. The system is designed for challenging scenarios where onboard egocentric perception, unknown object physical properties (mass, friction, center-of-mass configurations), and actuation uncertainties impede reliable task execution—conditions particularly germane to material handling in industrial environments. Unlike prior works, VOFA explicitly decouples high-level visual planning from low-level force-adaptive whole-body control, enabling robust policy learning and sim-to-real transfer without privileged object-state information.

System Architecture

VOFA employs a hierarchical control paradigm integrating a high-level visuomotor policy and a low-level force-adaptive whole-body controller (WBC), specifically leveraging FALCON for end-effector force adaptation. The high-level policy receives proprioceptive signals, depth images, and goal position data to generate commands, which are then mapped to joint-level actions via the low-level controller.

Figure 1: VOFA’s hierarchical structure couples a vision-based high-level policy with a force-adaptive controller for robust object pushing.

High-level policy training follows a teacher–student approach. The teacher is trained via PPO in simulation with privileged state information under an asymmetric actor–critic framework. DAgger distillation then transfers policy knowledge to a student operating solely on onboard sensor data and visual input with domain randomization to mitigate sim-to-real discrepancies.

Reinforcement Learning and Domain Randomization

Training is conducted in large-scale Isaac Gym environments utilizing extensive randomizations over object mass (1–8 kg), friction coefficients (0.1–1.0), initial placements, and camera parameters. Three principal vision augmentations are applied: far-plane depth perturbation, spatially correlated noise, and pixel dropout. These augmentations—tuned to mimic real-world sensor artifacts—were critical for robust policy transfer as evidenced by ablation. The reward function incorporates four main components: object-reaching, object-pushing towards the goal, head tracking, and object–goal alignment, with the latter enabling consistent repositioning before initiation of contact for long-horizon manipulation strategies.

Figure 2: Simulation depth input without noise augmentation.

Figure 3: Depth input with force-adaptive controller demonstrating robust perception adaptation.

Quantitative Performance and Ablations

VOFA achieves strong numerical results. In simulation, the teacher policy with force-adaptive control surpasses a 97% success rate across varying goal angles and mass configurations. The vision-based student policy maintains over 90% success rates, while ablated variants (without force adaptation) degrade substantially—falling below 60% for heavy object tasks. Real-world trials corroborate sim-to-real transferability, achieving over 80% success in pushing a 17 kg object (more than half the robot body weight), despite training only on lighter objects.

Figure 4: Object–goal alignment reward supports deliberate repositioning for robust pushing.

Figure 5: Real-world results across diverse goal positions show consistent policy performance.

Force-adaptive control proves indispensable for generalization to unseen object masses and configurations. Without it, behaviors degenerate to unstable, myopic strategies such as impulsive kicking, leading to frequent robot falls.

Closed-Loop Behaviors and Recovery

VOFA demonstrates closed-loop control with online feedback correction, disturbance rejection, and goal adaptation:

• External perturbation recovery: The robot reacts to object displacement, repositions, and realigns for continued pushing.

Figure 6: Policy recovers from external perturbations, illustrating robust closed-loop feedback.

• Sequential goal switching: Multiple goals trigger dynamic re-planning and repositioning without system reset.
• Off-centered mass: Compensation is achieved for geometric center-of-mass shifts, maintaining trajectory alignment.

These behaviors validate policy execution beyond open-loop motion planning, highlighting dynamic adaptability to environmental and task disturbances.

Implications and Future Directions

VOFA’s hierarchical architecture combined with visual goal-conditioning and force-adaptive control sets a new basemark for closed-loop humanoid loco-manipulation under real-world uncertainties. The results underscore critical design principles: explicit handling of unknown object properties, robust domain randomization in visual perception, and reward shaping from alignment incentives.

Practically, this enables deployment in industrial logistics, warehouse automation, and reactive material handling with humanoids. Theoretically, the decoupling of force-adaptive and visual policies opens avenues for modular, compositional skill training and scalable multi-task reinforcement learning.

Future development should extend VOFA to multi-object cluttered environments, joint perception-action integration for end-to-end goal recognition, and persistent operation under adversarial perturbations or highly diverse backgrounds. Fully autonomous goal specification from raw perception would remove dependence on external localization modules.

Conclusion

VOFA presents a rigorously evaluated solution for vision-based goal-conditioned humanoid object pushing with explicit force-adaptive control. Numerical, qualitative, and ablation results collectively demonstrate robust closed-loop, sim-to-real performance across task and environmental variations. The hierarchical framework and domain randomization methodology provide a principled pathway for advancing goal-directed loco-manipulation skills in complex, real-world settings using humanoid robots.