FetchMan: Sim-to-Real Humanoid Loco-Manipulation

• FetchMan is a two-stage visual learning system that combines behavior cloning with Flow-GRPO reinforcement learning to coordinate humanoid navigation, reaching, grasping, and lifting.
• The system trains on more than 150,000 procedurally generated scenes and improves simulated loco-manipulation success from 67% to 83% through sparse-reward reinforcement learning.
• A physical Unitree G1 achieves 73.3% ± 8.1% loco-manipulation success without real-world training or per-scene tuning, demonstrating the value of structured actions, domain randomization, and shared simulation-hardware controllers.

FetchMan is a two-stage sim-to-real system for learning visual humanoid loco-manipulation policies from simulated experience. It trains a Unitree G1 humanoid to visually navigate toward an object, reach for it, grasp it, and lift it, using behavior cloning for initialization and Flow-GRPO reinforcement learning for refinement. The system spans more than 150,000 procedurally generated scenes and is evaluated on FetchMan-Bench and a physical G1. Its single-object policy achieves a real-world loco-manipulation success rate of $73.3\% \pm 8.1\%$ without real-world training or per-scene tuning (Rayyan et al., 17 Aug 2026).

1. Problem formulation and system objectives

FetchMan addresses visual loco-manipulation, in which a humanoid must integrate perception, navigation, balance, torso positioning, arm reaching, grasping, and locomotion-to-manipulation coordination. The primary task is reach-and-pick: the robot begins away from a target, navigates toward it, reaches, grasps, and lifts it. A manipulation-only variant begins at grasp standoff and evaluates terminal manipulation independently of navigation.

The system uses a Unitree G1 humanoid with Dex1-1 grippers. The learned policy produces a 15-dimensional command at $10$ Hz:

$$a_t=\left(v_t,\bar h_t,\bar q_t^{\mathrm{waist}},\bar q_t^{\mathrm{arm}},\bar g_t\right).$$

Here, $v_t=(v_x,v_y,\omega)\in\mathbb{R}^3$ is the commanded planar base velocity and yaw rate, $\bar h_t$ is a target base height, $\bar q_t^{\mathrm{waist}}\in\mathbb{R}^3$ contains waist targets, $\bar q_t^{\mathrm{arm}}\in\mathbb{R}^7$ contains right-arm joint targets, and $\bar g_t$ is the right-gripper target. The observation is

$$o_t=\left(I_t^{\mathrm{head}},I_t^{\mathrm{wrist}},o_t^{\mathrm{prop}}\right),$$

where the head and wrist RGB images have resolution $224\times384\times3$. Proprioception contains base height, base roll, base pitch, upper-body joint positions, and gripper aperture.

FetchMan does not learn all joint torques directly. The pretrained SONIC lower-body controller tracks $10$0 at $10$1 Hz, while upper-body and gripper targets are tracked using joint-level PD control. Controllers ultimately produce torques at the $10$2-Hz physics rate. This reduces the learned action space from $10$3 joint torques to 15 semantically meaningful commands and provides a common controller stack in simulation and on the physical robot.

The learning problem is motivated by limitations of synthetic behavior cloning. The scripted demonstrator has access to privileged state, including object pose, grasp pose, occupancy maps, robot pose, and an internal manipulation phase. Its phase variable is

$10$4

The learned policy does not observe $10$5. It must infer phase transitions from a single visual-proprioceptive observation, even though the demonstrator’s actions can change discontinuously at navigation-to-reaching and reaching-to-manipulation boundaries. Behavior cloning also suffers from covariate shift: small action errors can move the policy into states absent from the demonstrations, and imitation loss is not identical to task-success optimization.

The central training hypothesis is therefore that synthetic demonstrations should initialize the policy but should not define its final behavior. FetchMan uses reinforcement learning with a sparse fetch-success reward to refine the cloned policy, particularly at the navigation-to-manipulation handoff.

2. Synthetic task generation and privileged demonstrations

Each simulated episode samples a task triple

$10$6

where $10$7 is an indoor scene, $10$8 is a pickable target object, and $10$9 is a robot start pose. Scenes are drawn from the ProcTHOR-10k, Holodeck, and ProcTHOR-Objaverse splits curated by MolmoSpaces.

For every scene, FetchMan precomputes a two-dimensional free-space occupancy grid $$a_t=\left(v_t,\bar h_t,\bar q_t^{\mathrm{waist}},\bar q_t^{\mathrm{arm}},\bar g_t\right).$$0. Static collision geometry is dilated by the G1 footprint radius, and the grid is used for start-pose rejection and navigation planning. Pickable objects are free-jointed assets. Each target has a grasp set $$a_t=\left(v_t,\bar h_t,\bar q_t^{\mathrm{waist}},\bar q_t^{\mathrm{arm}},\bar g_t\right).$$1, generated through antipodal sampling and post-filtering in MolmoSpaces. Candidate grasps are further filtered according to Dex1-1 gripper geometry and G1 inverse-kinematics feasibility.

Candidate robot bases are sampled from free cells in an annulus around the target:

$$a_t=\left(v_t,\bar h_t,\bar q_t^{\mathrm{waist}},\bar q_t^{\mathrm{arm}},\bar g_t\right).$$2

with orientation

$$a_t=\left(v_t,\bar h_t,\bar q_t^{\mathrm{waist}},\bar q_t^{\mathrm{arm}},\bar g_t\right).$$3

A candidate is accepted only if it is contact-free in the full simulator and the target is visible from the head fisheye camera. The final validity check is three-dimensional rather than based solely on a conservative flat footprint, allowing the G1 to stand near a table while its upper body reaches over it.

The privileged scripted demonstrator receives

$$a_t=\left(v_t,\bar h_t,\bar q_t^{\mathrm{waist}},\bar q_t^{\mathrm{arm}},\bar g_t\right).$$4

where $$a_t=\left(v_t,\bar h_t,\bar q_t^{\mathrm{waist}},\bar q_t^{\mathrm{arm}},\bar g_t\right).$$5 is the target pose, $$a_t=\left(v_t,\bar h_t,\bar q_t^{\mathrm{waist}},\bar q_t^{\mathrm{arm}},\bar g_t\right).$$6 is the selected grasp pose, $$a_t=\left(v_t,\bar h_t,\bar q_t^{\mathrm{waist}},\bar q_t^{\mathrm{arm}},\bar g_t\right).$$7 is the occupancy grid, $$a_t=\left(v_t,\bar h_t,\bar q_t^{\mathrm{waist}},\bar q_t^{\mathrm{arm}},\bar g_t\right).$$8 is the robot base pose, and $$a_t=\left(v_t,\bar h_t,\bar q_t^{\mathrm{waist}},\bar q_t^{\mathrm{arm}},\bar g_t\right).$$9 is the manipulation phase. During navigation, it plans an A* path through an extra-inflated occupancy grid, reduces the path to straight-line waypoints, and emits base velocity commands toward the next waypoint while maintaining target-facing orientation.

At grasp standoff, the demonstrator executes:

$v_t=(v_x,v_y,\omega)\in\mathbb{R}^3$0

The target grasp is selected by transforming candidates in $v_t=(v_x,v_y,\omega)\in\mathbb{R}^3$1 into world coordinates using $v_t=(v_x,v_y,\omega)\in\mathbb{R}^3$2 and choosing a reachable grasp with Mink inverse kinematics and self-collision avoidance. Base height is optimized jointly with arm joints, enabling torso adjustment for object reachability. Episodes with invalid paths, infeasible grasps, or phase timeouts are discarded.

The principal single-object dataset contains 150,000 bowl-pick demonstrations, approximately 150,000 distinct procedurally generated scenes, roughly 650 hours of simulated robot experience, generation at approximately 100 episodes per minute, and approximately 40 GPU-hours on one NVIDIA L40S. The demonstrations expose the learner to camera and proprioceptive observations while retaining the demonstrator’s privileged 15-dimensional commands as training targets.

The dataset uses 80% fetch episodes, consisting of navigation followed by reach-and-pick, and 20% pick-only episodes. Pick-only episodes randomize initial upper-body configurations and gripper aperture, preventing exclusive exposure to a default arms-forward and gripper-open configuration.

Domain randomization is applied independently per episode. It includes scene textures; light positions, colors, intensities, and shadow toggles; robot body color; camera intrinsics and extrinsics; supporting-surface height; poses of objects resting on moved surfaces; target planar pose; and zero-mean command noise before commands reach the low-level controllers. These variations target visual, geometric, sensing, and control discrepancies between simulation and hardware.

3. Visual policy architecture and hierarchical control

For the single-object policy, head and wrist images are encoded with a frozen DINOv3 ViT-B/16. The encoder produces per-patch visual tokens, while proprioception is mapped by an MLP to one state token. A DiT-style flow-matching transformer cross-attends from noised action tokens to the visual and state context and predicts an action chunk

$v_t=(v_x,v_y,\omega)\in\mathbb{R}^3$3

with chunk length $v_t=(v_x,v_y,\omega)\in\mathbb{R}^3$4. At deployment, the first eight actions are executed before a new chunk is generated from the latest observation.

Eleven dimensions corresponding to absolute pose or joint targets—base height, waist joints, and arm joints—are represented as deltas relative to an anchor obtained at the beginning of the chunk. Base velocity and gripper commands remain absolute. Conceptually,

$v_t=(v_x,v_y,\omega)\in\mathbb{R}^3$5

This parameterization reduces dependence on absolute robot configuration and controller calibration. The reported ablation compares it with absolute-action prediction.

The multi-object policy replaces the DINOv3 visual backbone with frozen dino.txt, whose image and text towers share a pretrained embedding space. The requested object name is encoded as a text token, fused with visual tokens, and used to ground the target object. The multi-object extension uses 350,000 demonstrations across pickable object categories.

The hierarchical control interface separates learned visual decision-making from low-level stabilization and actuation. SONIC internally dispatches between balance and walking behaviors using a hard threshold on commanded base velocity. Upper-body and gripper targets are directly tracked by PD control. The learned policy therefore determines locomotion commands, torso configuration, arm targets, and gripper behavior, while the lower-level stack handles the corresponding closed-loop dynamics.

This decomposition differs from approaches that learn directly in full torque space. FetchMan preserves a pretrained lower-body controller and learns a reduced, semantically structured command interface. A plausible implication is that this makes large-scale training and sim-to-real transfer more tractable, while restricting the learned policy’s ability to alter gait, stance, bracing, or balance strategies.

4. Behavior cloning and Flow-GRPO refinement

Behavior cloning

The first training stage clones the privileged demonstrations. A standard behavior-cloning formulation is

$v_t=(v_x,v_y,\omega)\in\mathbb{R}^3$6

The resulting checkpoint is called FetchMan-BC. Cloning provides a competent initialization but reaches an imitation ceiling. Increasing the single-object dataset from 5,000 to 50,000 demonstrations improves simulated loco-manipulation success from $v_t=(v_x,v_y,\omega)\in\mathbb{R}^3$7 to $v_t=(v_x,v_y,\omega)\in\mathbb{R}^3$8. Increasing it further to 150,000 demonstrations produces no measurable improvement, remaining at $v_t=(v_x,v_y,\omega)\in\mathbb{R}^3$9.

This plateau is attributed to the hidden phase structure of the demonstrator. More observations of the same privileged-state controller do not necessarily make phase transitions recoverable when the phase index is absent from the learner’s input.

Flow-matching action generation

The action head predicts a velocity field

$\bar h_t$0

where $\bar h_t$1 is a partially denoised action representation at flow time $\bar h_t$2. Deterministic Euler integration would use

$\bar h_t$3

For reinforcement learning, FetchMan uses a stochastic Flow-GRPO transition:

$\bar h_t$4

with

$\bar h_t$5

Each flow step is Gaussian, so its transition log-likelihood is tractable:

$\bar h_t$6

This permits importance-ratio computation between the rollout-time policy and the current policy during PPO-style optimization.

Sparse-reward reinforcement learning

The reinforcement-learning reward is deliberately minimal:

$\bar h_t$7

Success includes grasping and lifting the target. There is no per-step shaping reward.

Each update uses groups of identical task initializations. The same task is rolled out $\bar h_t$8 times, with variation arising from stochasticity in the flow policy. Group-relative advantages are computed as

$\bar h_t$9

where $\bar q_t^{\mathrm{waist}}\in\mathbb{R}^3$0 and $\bar q_t^{\mathrm{waist}}\in\mathbb{R}^3$1 are the group mean and standard deviation. No learned value function is used. Groups whose episodes are all successful or all unsuccessful are discarded because they provide no relative learning signal.

The single-object model uses $\bar q_t^{\mathrm{waist}}\in\mathbb{R}^3$2 and $\bar q_t^{\mathrm{waist}}\in\mathbb{R}^3$3, producing 256 episodes per update. The multi-object model uses $\bar q_t^{\mathrm{waist}}\in\mathbb{R}^3$4 and $\bar q_t^{\mathrm{waist}}\in\mathbb{R}^3$5, producing 512 episodes per update to compensate for its lower prior success rate and greater frequency of all-failure groups.

The actor is optimized with a PPO-style clipped objective over flow-step importance ratios. An optional KL penalty to the frozen behavior-cloning policy is used. The single-object model sets the KL coefficient to $\bar q_t^{\mathrm{waist}}\in\mathbb{R}^3$6, while the multi-object model uses $\bar q_t^{\mathrm{waist}}\in\mathbb{R}^3$7. Since successful episodes terminate at the lift and failures continue to the time limit, the actor loss is length-normalized by $\bar q_t^{\mathrm{waist}}\in\mathbb{R}^3$8, where $\bar q_t^{\mathrm{waist}}\in\mathbb{R}^3$9 is episode length and $\bar q_t^{\mathrm{arm}}\in\mathbb{R}^7$0 is the batch mean.

Starting from the saturated 150,000-demonstration checkpoint, Flow-GRPO increases simulated loco-manipulation success from $\bar q_t^{\mathrm{arm}}\in\mathbb{R}^7$1 to $\bar q_t^{\mathrm{arm}}\in\mathbb{R}^7$2. Applied to the weak 5,000-demonstration checkpoint, the same procedure oscillates around $\bar q_t^{\mathrm{arm}}\in\mathbb{R}^7$3 and peaks at only $$42\%`. The group-relative method therefore depends on a sufficiently competent prior to produce mixed-success groups.</p> <h2 class='paper-heading' id='fetchman-bench-and-experimental-results'>5. FetchMan-Bench and experimental results</h2> <p>FetchMan-Bench is a simulation benchmark with fixed held-out scenes and released scoring. It reports two metrics:</p> <ol> <li><strong>Manipulation success rate</strong>: the robot starts at grasp standoff and must reach, grasp, and lift the object.</li> <li><strong>Loco-manipulation success rate</strong>: the robot starts away from the object and must navigate to it before grasping and lifting.</li> </ol> <p>The benchmark uses 100 held-out initializations not used for training. Its task distribution varies scene layouts, object placement, support-surface height, camera parameters, textures, lighting, and robot initialization conditions.</p> <p>The principal results are:</p> <div class='overflow-x-auto max-w-full my-4'><table class='table border-collapse w-full' style='table-layout: fixed'><thead><tr> <th>Method</th> <th style="text-align: right">Sim. manipulation</th> <th style="text-align: right">Sim. loco-manipulation</th> <th style="text-align: right">Real manipulation</th> <th style="text-align: right">Real loco-manipulation</th> </tr> </thead><tbody><tr> <td>FetchMan-BC</td> <td style="text-align: right">$$\bar q_t^{\mathrm{arm}}\in\mathbb{R}^7$4 $\bar q_t^{\mathrm{arm}}\in\mathbb{R}^7$5 $\bar q_t^{\mathrm{arm}}\in\mathbb{R}^7$6 $\bar q_t^{\mathrm{arm}}\in\mathbb{R}^7$7 FetchMan (BC+RL) $\bar q_t^{\mathrm{arm}}\in\mathbb{R}^7$8 $\bar q_t^{\mathrm{arm}}\in\mathbb{R}^7$9 $\bar g_t$0 $\bar g_t$1

The largest improvement occurs in the full loco-manipulation task. Real manipulation-only success increases from $\bar g_t$2 to $\bar g_t$3, whereas real loco-manipulation success increases from $\bar g_t$4 to $\bar g_t$5. Qualitatively, the behavior-cloned policy sometimes initiates grasping when the object is nearby but still outside reach. The RL-refined policy more often continues walking until the object is within grasping distance.

The architecture ablations are:

Variant	Sim. manipulation	Sim. loco-manipulation	Real manipulation	Real loco-manipulation
DINOv3 + delta actions	$\bar g_t$6	$\bar g_t$7	$\bar g_t$8	$\bar g_t$9
SigLIP	$o_t=\left(I_t^{\mathrm{head}},I_t^{\mathrm{wrist}},o_t^{\mathrm{prop}}\right),$0	$o_t=\left(I_t^{\mathrm{head}},I_t^{\mathrm{wrist}},o_t^{\mathrm{prop}}\right),$1	$o_t=\left(I_t^{\mathrm{head}},I_t^{\mathrm{wrist}},o_t^{\mathrm{prop}}\right),$2	$o_t=\left(I_t^{\mathrm{head}},I_t^{\mathrm{wrist}},o_t^{\mathrm{prop}}\right),$3
Absolute actions	$o_t=\left(I_t^{\mathrm{head}},I_t^{\mathrm{wrist}},o_t^{\mathrm{prop}}\right),$4	$o_t=\left(I_t^{\mathrm{head}},I_t^{\mathrm{wrist}},o_t^{\mathrm{prop}}\right),$5	$o_t=\left(I_t^{\mathrm{head}},I_t^{\mathrm{wrist}},o_t^{\mathrm{prop}}\right),$6	$o_t=\left(I_t^{\mathrm{head}},I_t^{\mathrm{wrist}},o_t^{\mathrm{prop}}\right),$7

The results associate DINOv3 with stronger geometric transfer across the simulation-to-real appearance gap and delta actions with greater robustness to differences in robot configuration, calibration, and controller response.

The multi-object model is trained on 350,000 demonstrations with object names supplied as text conditioning. Its behavior-cloning success peaks near $o_t=\left(I_t^{\mathrm{head}},I_t^{\mathrm{wrist}},o_t^{\mathrm{prop}}\right),$8, while Flow-GRPO increases simulated loco-manipulation success to approximately $o_t=\left(I_t^{\mathrm{head}},I_t^{\mathrm{wrist}},o_t^{\mathrm{prop}}\right),$9. This remains below the $224\times384\times3$0 single-object result. The paper reports qualitative zero-shot real-world fetches across several object categories, but does not summarize multi-object real-world performance with a numerical success-rate table.

6. Sim-to-real transfer, limitations, and relation to adjacent systems

The final single-object policy is deployed on a physical Unitree G1 with Dex1-1 grippers without real-world training or per-scene tuning. It uses head-mounted fisheye RGB and wrist RGB sensing, proprioception matching the simulated observation, a policy command rate of $224\times384\times3$1 Hz, SONIC lower-body control at $224\times384\times3$2 Hz, upper-body and gripper PD tracking, and a shared 15-dimensional interface between simulation and hardware.

The reported physical results are $224\times384\times3$3 manipulation success over 22 trials and $224\times384\times3$4 loco-manipulation success over 30 trials. The deployment strategy combines a shared controller stack, matching observation modalities, camera-parameter randomization, command noise, frozen DINOv3 features, and delta-action parameterization.

FetchMan’s limitations include hidden phase inference, stateless observations, a fixed lower-body controller, limited task scope, sim-to-real discrepancies, and weaker multi-object grounding. The policy cannot modify SONIC’s gait or balance strategy, brace against heavy objects, widen its stance for reach, adapt gait to object weight, or recover from disturbances outside SONIC’s operating envelope. The primary task does not address articulated-object manipulation, placement, carrying, tool use, or long-horizon task composition.

The multi-object model uses a frozen and comparatively small dino.txt backbone. The paper identifies larger vision-language encoders and low-learning-rate backbone fine-tuning as possible improvements, but does not pursue them because of compute and hardware limitations. The physical gap can arise from visual differences, imperfect camera and robot calibration, unmodeled actuator and contact dynamics, gripper behavior, balance disturbances, and differences in object surfaces.

Relation to hierarchical language-conditioned control

The semantic-to-control decomposition in FetchMan is related to hierarchical approaches that separate high-level reasoning from low-level control. “Feudal Reinforcement Learning by Reading Manuals” uses a manager to generate grounded semantic sub-goals and a worker to execute primitive actions. Its backward multi-hop manager improves RTFM win rates to $224\times384\times3$5 on $224\times384\times3$6 environments and $224\times384\times3$7 on $224\times384\times3$8 environments (Wang et al., 2021). FetchMan differs in that its primary policy is a visual flow-matching controller producing continuous locomotion, torso, arm, and gripper targets rather than a manager-worker architecture with explicit semantic sub-goal sequences.

A plausible FetchMan extension would combine the two designs: a high-level planner could produce grounded object and manipulation sub-goals, while the FetchMan policy would execute navigation, reaching, grasping, and lifting through its learned 15-dimensional interface. Such an extension would require additional handling of partial observability, variable-length plans, object-instance grounding, and failure recovery.

Relation to open-world robot fetching

“Helpful DoggyBot: Open-World Object Fetching using Legged Robots and Vision-LLMs” separates open-vocabulary semantic perception from low-level whole-body control. It uses a quadruped, a front-mounted 1-DoF gripper, pretrained vision-LLMs, global and egocentric cameras, and a simulation-trained depth policy; it reports a 60% success rate for fetching a randomly placed stuffed toy after climbing over a queen-sized bed (Wu et al., 2024).

FetchMan shares the division between high-level perception and low-level locomotion-manipulation control but removes the requirement for a ceiling-mounted third-person camera in the reported system. Its visual policy uses head and wrist cameras directly and learns an integrated navigation-to-grasp behavior. Conversely, Helpful DoggyBot uses explicit open-vocabulary detection, segmentation, tracking, and reactive waypoint commands, whereas FetchMan’s single-object policy is trained end-to-end over visual observations and structured action chunks.

Relation to validated mission coordination

“GuideFetch: A Task Coordination Framework for Concurrent Navigation and Object Retrieval in Assistive Robot Dogs” distinguishes plan validity, execution progress, and mission completion. It uses capability-constrained action schemas, deterministic validation, dependency graphs, state-based execution, and explicit terminal predicates for visit, pickup, grasp, lift, retention, and delivery (Yin et al., 18 Aug 2026).

These mechanisms address issues not covered by FetchMan’s reach-and-pick objective. FetchMan verifies target fetching through task success, but its primary experiments do not establish long-horizon carrying, delivery, multi-robot concurrency, or explicit mission-ledger semantics. A broader FetchMan system could adopt GuideFetch’s distinctions by representing fetching as

$224\times384\times3$9

This would separate controller commands from physically verified object acquisition and delivery.

Broader significance

FetchMan’s principal methodological contribution is the treatment of synthetic behavior cloning as initialization rather than as the final objective. The results indicate that large-scale synthetic demonstrations can establish a competent policy, while sparse-reward Flow-GRPO can correct behaviorally important failures that arise from hidden demonstrator phases and covariate shift.

Its strongest demonstrated capability is zero-shot humanoid loco-manipulation across unseen simulated scenes and a physical Unitree G1. Its scope remains narrower than general-purpose fetching: the primary task is single-object reach-and-pick, the lower-body controller is fixed, manipulation is limited to the available grippers, and multi-object performance remains below the single-object setting. Within these constraints, FetchMan provides a sim-to-real recipe combining procedural scene generation, privileged scripted demonstrations, frozen visual representation learning, action-chunk flow matching, and sparse-reward reinforcement-learning refinement.