FetchMan: Learning Visual Humanoid Loco-Manipulation Policies from Simulated Experiences
Abstract: Visual loco-manipulation policies that can generalize to novel scenes and objects have long been a goal of robotics research. However, today's data-hungry algorithms make collecting sufficient demonstrations a struggle for tabletop manipulation, and even more so for humanoids that must also walk and balance. Learning from simulated data and transferring that behavior to the real world, as is commonly done in locomotion, sidesteps this struggle, so we replicate that recipe for loco-manipulation. In doing so, we find that cloning synthetic demonstrations results in a low performance ceiling no matter the amount of training data. Reinforcement learning breaks through it, and refining the cloned policy with Flow-GRPO on a single sparse reward yields performance that synthetic behavior cloning cannot match. Together, these stages form our end-to-end sim-to-real pipeline spanning more than 150,000 scenes, which we use to train FetchMan. We evaluate it on FetchMan-Bench, a simulation benchmark we release, and deploy it zero-shot on a real Unitree G1, where our single-object reach-and-pick policy walks to and grasps a target across unseen scenes at 73.3% success. Finally, we extend this recipe to multi-object training, a first step toward loco-manipulation generalist policies at this data scale.
Summary
- The paper introduces a two-stage pipeline that combines 150,000 synthetic demonstrations with Flow-GRPO reinforcement learning, raising simulated full-task success from 67% to 83%.
- The paper finds that reinforcement learning primarily improves the navigation-to-grasp transition, increasing zero-shot real-world loco-manipulation success from 56.7% to 73.3% on a Unitree G1 without real-world training.
- The paper shows that frozen DINOv3 visual features, delta-based action targets, and a fixed lower-body controller are critical for transfer, while the multi-object model reaches approximately 62% simulated success after refinement.
Problem formulation and central claim
“FetchMan: Learning Visual Humanoid Loco-Manipulation Policies from Simulated Experiences” (2608.17027) studies zero-shot sim-to-real transfer for visual humanoid loco-manipulation. The target task is deliberately narrow but structurally demanding: a Unitree G1 must navigate through an indoor scene, approach a designated object, reach with its upper body, grasp it, and lift it. The policy receives only head-fisheye and wrist RGB observations together with proprioception; it does not receive object poses, navigation maps, grasp annotations, or a phase variable. The paper’s central empirical claim is that large-scale synthetic behavior cloning is necessary but insufficient. Behavior cloning supplies a competent initialization, whereas Flow-GRPO reinforcement learning is required to overcome an imitation ceiling caused by privileged-state-dependent phase transitions in the scripted demonstrator.
This claim is evaluated in simulation through FetchMan-Bench and on a physical Unitree G1. The reported single-object policy is trained entirely in simulation and achieves real-world loco-manipulation success without real-world data, scene-specific tuning, or post-deployment adaptation. In simulation, reinforcement learning increases full-task success from for the behavior-cloned checkpoint to after refinement. The improvement is concentrated in navigation-to-manipulation coordination rather than terminal grasping.
Figure 1: FetchMan is trained entirely in simulation and deployed zero-shot to a Unitree G1 across unseen real-world scenes.
The paper frames loco-manipulation as a more difficult extension of established simulation-based locomotion pipelines. The difficulty is not simply the dimensionality of humanoid control. It arises from the combination of visual partial observability, contact-rich manipulation, scene diversity, balance constraints, and a mismatch between the information available to the scripted demonstrator and that available to the learned policy. The contribution is therefore a training recipe and an analysis of why its two stages are complementary, rather than merely a larger policy or a more diverse simulator.
Hierarchical control and policy interface
FetchMan uses the Unitree G1 with Dex1-1 grippers and exposes a 15-dimensional command interface at $10$ Hz. The learned policy predicts base velocity, base-height targets, waist and arm joint targets, and gripper aperture. The base command is executed by the pretrained SONIC lower-body controller, while upper-body and gripper targets are tracked through joint-level PD control. Torques are produced at the $200$ Hz physics rate.
This factorization is important for both optimization and transfer. Instead of learning directly in a 29-dimensional torque space, the policy operates over semantically structured whole-body commands. The lower-level controller provides closed-loop balance and gait regulation, reducing the policy’s burden to coordination between navigation, torso configuration, reaching, and grasping. Because the same control stack is used in simulation and on hardware, its response is treated as a fixed component rather than an additional sim-to-real discrepancy.
The decomposition also imposes a substantive limitation: FetchMan cannot modify gait or balance behavior in response to manipulation demands. It cannot, for example, widen its stance to increase reachability, brace against a heavy object, or recover from disturbances beyond the SONIC controller’s operating envelope. The reported results therefore characterize visual loco-manipulation under a fixed lower-body controller, not unconstrained whole-body control.
Synthetic task and demonstration generation
Training episodes are instantiated from MolmoSpaces scenes drawn from ProcTHOR-10k, Holodeck, and ProcTHOR-Objaverse splits. Each episode samples an indoor house, a pickable target object, a target pose and supporting-surface configuration, a feasible grasp, and a contact-free robot start pose. Candidate starts are selected from free-space occupancy grids and then validated against the full three-dimensional simulator. This validation permits upper-body occupancy over surfaces such as tables while rejecting physically invalid base configurations.
The scripted demonstrator has privileged access to the object pose, grasp transform, occupancy grid, robot base pose, and an internal phase index. It plans base motion with A* over an inflated occupancy grid, reduces the path to waypoints, and commands the robot toward the target while maintaining target-facing orientation. Once at the grasp standoff, it executes the discrete sequence “reach, descend, close, lift.” Grasp candidates are filtered using gripper geometry, inverse-kinematics feasibility, and self-collision avoidance; base height is optimized jointly with upper-body configuration.
Figure 2: Each synthetic episode samples a scene, target object, graspable goal, feasible start, navigation path, and independent domain-randomization variables.
The dataset contains an / mixture of fetch and pick-only episodes. Fetch episodes train the complete navigation-to-grasp behavior, while pick-only episodes increase exposure to terminal manipulation and randomize initial arm and gripper configurations. This mixture addresses a specific distributional problem: fetch demonstrations begin from a standardized arms-forward, gripper-open state, whereas deployment can encounter noncanonical upper-body configurations.
Domain randomization is applied independently per episode to textures, lighting, robot appearance, camera intrinsics and extrinsics, supporting-surface height, object pose, and action execution noise. The visual inputs consist of RGB images from head and wrist cameras, while proprioception includes base height, roll, pitch, upper-body joint positions, and gripper aperture.
The resulting dataset comprises approximately bowl-pick demonstrations, roughly $650$ hours of simulated robot experience, and approximately 0 distinct scenes. Generation runs at about 1 episodes per minute and requires approximately 2 GPU-hours on a single NVIDIA L40S. The scale is consequently substantial in scene count but modest in compute relative to large general-purpose foundation-model training.
Architecture and two-stage optimization
The policy uses a frozen DINOv3 ViT-B/16 visual encoder, which produces patch tokens independently for the head and wrist streams. A multilayer perceptron embeds proprioception as a state token. A DiT-style flow-matching action head cross-attends to this context and generates a chunk of 16 future actions. At deployment, only the first eight actions are executed before replanning from a new observation. Absolute joint and pose targets are represented as deltas relative to an observation-dependent chunk anchor; base velocities and gripper commands remain absolute.
Figure 3: Frozen visual patch tokens and a proprioceptive state token condition a flow-matching transformer that predicts structured whole-body action chunks.
Training proceeds in two stages. First, behavior cloning minimizes the discrepancy between policy action chunks and scripted trajectories. This stage is essential because the task is high-dimensional, partially observable, and sparse-reward RL from scratch is ineffective at the tested scale. Second, the cloned policy is refined online with Flow-GRPO using a binary success reward.
Figure 4: Behavior cloning provides an initialization, after which identical-task group rollouts support group-relative Flow-GRPO updates.
Flow-GRPO converts the deterministic flow integrator into a stochastic process by injecting Gaussian noise into each integration step. This produces tractable per-step transition likelihoods, enabling PPO-style clipped importance-ratio updates for a flow-matching policy. The rollout and optimization processes use the same noise schedule, preserving a well-defined likelihood ratio.
The RL batches use identical-state groups: each group contains eight rollouts from the same task and initial state, differing only in action-sampling noise. Returns are standardized within the group, and groups containing only successes or only failures are discarded because they provide no relative advantage signal. The single-object policy uses 3 groups, or 4 episodes, per update; the multi-object policy uses 5 groups, or 6 episodes. No learned value function is used. Since successful trajectories terminate earlier than failures, the policy loss is length-normalized so that episodes rather than transitions receive equal weight.
This design exploits a property specific to simulation: repeated rollouts from identical states are inexpensive and reproducible. The same baseline would be difficult to obtain on a physical humanoid, where repeated resets and matched initial conditions are costly and potentially unsafe.
Behavior cloning exhibits an imitation ceiling
The paper’s most important diagnostic result is that increasing the number of demonstrations does not indefinitely improve full-task performance. On a 100-scene validation set, loco-manipulation success increases from 7 with 8 demonstrations to 9 with 0, but remains at approximately 1 when the dataset is expanded to 2. Thus, the additional 3 demonstrations yield no measurable improvement in the reported setup.
Figure 5: Behavior cloning plateaus near 4 for the single-object task, while Flow-GRPO raises success to 5; in the multi-object setting, RL increases success from roughly 6 to 7.
The proposed explanation is structural rather than data-limited. The scripted controller transitions between navigation, reaching, descending, closing, and lifting according to a hidden phase index. The policy must infer this phase from a single observation, although the phase can be discontinuous at boundaries and is not always identifiable from the available visual and proprioceptive inputs. Near the locomotion-to-manipulation handoff, visually similar states can therefore be associated with different demonstrator actions depending on the inaccessible phase variable.
The result also reflects the conventional covariate-shift problem in behavior cloning. Small action errors place the learned policy in states absent from the demonstration distribution, while the behavior-cloning objective is not identical to task success. More demonstrations reduce estimation error but do not remove the information asymmetry or the mismatch between supervised action prediction and long-horizon success. The authors note that smoothing the demonstrator’s phase transitions could potentially raise the ceiling, but that would require task-specific controller engineering, weakening the scalability objective.
The empirical implication is direct: under this fixed demonstrator, data scaling alone is not a sufficient strategy for improving navigation-to-grasp success.
Flow-GRPO recovers performance beyond the demonstrator
Flow-GRPO raises simulated loco-manipulation success from 8 to 9 when initialized from the $10$0-demonstration checkpoint. This improvement is consistent with the interpretation that RL allows the policy to optimize the actual success objective and discover smoother handoff behavior instead of reproducing the demonstrator’s discrete phase logic.
The initialization dependence is equally important. Applying the same refinement schedule to the $10$1-demonstration checkpoint improves success only from $10$2 to a peak of approximately $10$3, with no clear upward trend. A weak policy produces mostly all-failure groups, which are discarded by the mixed-group filter, leaving insufficient gradient-carrying comparisons. Flow-GRPO therefore complements, rather than replaces, large-scale imitation pretraining.
The real-world results mirror the simulation diagnosis. Behavior cloning achieves $10$4 loco-manipulation success over 30 trials, whereas the BC+RL policy achieves $10$5. Terminal manipulation changes much less, from $10$6 to $10$7 over 22 trials. In simulation, manipulation success rises from $10$8 to $10$9, while full-task success rises from $200$0 to 200$1
The concentration of improvement in the full task supports the paper’s mechanistic account. The cloned policy can already perform the terminal grasp reasonably well; the principal defect lies in deciding when to stop walking and begin manipulation. Qualitative real-world rollouts show the BC policy attempting to grasp when the object is merely nearby or still too far away. After RL refinement, the policy continues walking until the object enters an effective reach envelope and then initiates grasping.
Figure 6: Flow-GRPO primarily repairs the locomotion-to-manipulation handoff, causing the robot to approach until the object is within reach before grasping.
Sim-to-real transfer and ablations
FetchMan transfers without real-world training or per-scene calibration. The single-object BC+RL policy reaches $200$9 loco-manipulation success on unseen physical scenes. This is a nontrivial transfer result because the policy depends on two camera streams, whole-body dynamics, contact interactions, and a fixed humanoid controller, yet it is evaluated directly on hardware after simulation training.
Two ablations identify particularly strong dependencies. Replacing frozen DINOv3 with SigLIP reduces simulated loco-manipulation success from $80\%$0 to $80\%$1 and produces $80\%$2 real-world loco-manipulation success; real-world manipulation success falls to $80\%$3. The authors attribute this gap to DINOv3’s stronger geometric representation and its ability to bridge appearance variation without finetuning. This attribution is plausible but not isolated completely: the encoders may differ in pretraining, tokenization, capacity, and inductive bias, so the experiment establishes a strong empirical dependency rather than a uniquely identified causal mechanism.
The action representation is similarly decisive. Replacing delta targets with absolute targets reduces simulation loco-manipulation success to $80\%$4 and produces $80\%$5 real-world loco-manipulation success; physical manipulation success is only $80\%$6. Delta parameterization keeps predicted corrections near the current configuration and appears to reduce sensitivity to simulator-to-hardware discrepancies in joint and pose targets. The magnitude of this ablation indicates that action coordinates are not a minor implementation detail in zero-shot humanoid transfer.
These results also qualify the headline transfer rate. The real-world evaluation contains 30 loco-manipulation trials for the reference model, yielding a binomial standard error of $80\%$7 percentage points. The estimate is therefore informative but statistically limited, and the paper does not establish robustness across robot units, payloads, disturbances, or substantially different camera placements.
Extension to multiple objects
The authors extend the pipeline to a text-conditioned model trained on $80\%$8 demonstrations spanning all pickable object categories. A frozen dino.txt vision-language backbone embeds the prompted object name into the visual context, allowing the policy to condition grasp behavior on the target category.
The multi-object BC policy reaches approximately $80\%$9 simulated loco-manipulation success, which Flow-GRPO increases to roughly $20\%$0. The result demonstrates that the two-stage recipe remains operational when the target distribution is broadened, but it also exposes a substantial performance gap relative to the $20\%$1 single-object result. Because the multi-object prior is weaker, more sampled groups are entirely unsuccessful and provide no group-relative learning signal; doubling the group count from $20\%$2 to $20\%$3 compensates partially for this filtering effect.
Figure 7: The text-conditioned policy performs fetches for multiple prompted object categories in simulation.
The paper explicitly does not claim parity between single-object and object-generalist policies. The lower multi-object performance is attributed partly to task difficulty and partly to the limited grounding capacity of the frozen dino.txt encoder. The authors suggest that a larger VLA encoder or low-learning-rate backbone finetuning could improve target-object grounding, but neither option is evaluated because of compute and hardware constraints. Qualitative physical rollouts succeed across several categories, though they are noticeably less robust than the single-object policy; no quantitative real-world multi-object success rate is reported.
Limitations and open questions
FetchMan is stateless: each action prediction uses one observation rather than a temporal window or recurrent memory. Consequently, the policy must infer the demonstrator’s hidden phase from a single frame. A short history could make phase estimation more identifiable, but the paper does not test this because frame stacking would increase visual-token and training costs.
The fixed SONIC lower-body controller restricts the learned behavior to its gait, balance, and disturbance-recovery envelope. Unfreezing lower-body control would substantially enlarge the action space and couple manipulation with balance learning, but the paper provides no evidence about whether such joint optimization would improve task success or simply destabilize training.
The task distribution is also limited to fetch-and-lift behavior. It excludes articulated-object interaction, object placement, bimanual manipulation, recovery behaviors, and longer-horizon task composition. In addition, the scripted controller’s feasibility filters may bias the training distribution toward objects and grasps that are already solvable under privileged planning. The evaluation uses fixed held-out initializations and relatively small physical trial counts, so generalization to arbitrary clutter, occlusion, object dynamics, payload variation, and hardware wear remains unresolved.
A specific open question is whether the reported RL gains arise primarily from correcting hidden phase transitions, from optimizing robustness to action noise, or from other effects of on-policy distribution shift. The paper’s ablations strongly support the phase-transition hypothesis, but it does not separately manipulate phase observability, demonstrator smoothness, temporal memory, and reward sparsity.
Conclusion
FetchMan establishes a two-stage sim-to-real method for visual humanoid loco-manipulation: large-scale synthetic behavior cloning followed by sparse-reward Flow-GRPO refinement. Its principal finding is that behavior cloning saturates at approximately 4 simulated full-task success despite scaling to 5 demonstrations, while RL refinement raises performance to 6 and improves zero-shot real-world success from 7 to 8. The evidence indicates that the limiting factor is not only dataset size but also the unobservable, discontinuous phase structure of the privileged scripted demonstrator. The method extends to text-conditioned multi-object training, although performance remains substantially lower at roughly 9 after refinement. The paper’s results support simulation as a viable source of diverse humanoid loco-manipulation experience, provided imitation pretraining is supplemented by an optimization stage aligned with task success.
Paper Prompts
Sign up for free to create and run prompts on this paper.
Top Community Prompts
Explain it Like I'm 14
1. What is this paper about?
The paper introduces FetchMan, a robot-learning system that teaches a humanoid robot to walk toward an object, reach for it, and pick it up.
The researchers train the robot almost completely inside a computer simulation. They then place the learned system on a real Unitree G1 humanoid robot without giving it extra real-world training. This is called zero-shot transfer: the robot tries the real task immediately, using what it learned in simulation.
The main challenge is that the robot must do two things at once:
- Locomotion: walking and keeping its balance.
- Manipulation: moving its arm and hand to grasp an object.
Together, these are called loco-manipulation.
2. What questions did the researchers investigate?
The paper focuses on four main questions:
- Is more simulated training data enough? The researchers wanted to know whether simply creating more robot demonstrations would keep improving performance.
- Can reinforcement learning improve the robot beyond the demonstrations? They tested whether the robot could learn from success and failure instead of only copying a scripted controller.
- Can a robot trained only in simulation work on a real humanoid? They examined whether the robot could handle new real rooms, lighting, objects, and camera views without being specially adjusted.
- Can the method work with many different objects? The first version mostly practiced picking up bowls. The researchers also tested a version that could respond to different object names and pick up several object types.
3. How was the research carried out?
Creating many simulated experiences
The researchers built thousands of different indoor scenes in simulation. These scenes contained rooms, tables, floors, lighting, and objects. In total, they used about:
- 150,000 scenes for the single-object version.
- About 350,000 demonstrations for the multi-object version.
A scripted computer controller acted as a teacher. It knew information that the robot’s learning system would not have in the real world, such as the exact location of the object and the best grasping position.
The scripted controller performed a sequence like this:
- Plan a safe path around obstacles.
- Walk toward the object.
- Move the arm into position.
- Lower the hand if needed.
- Close the gripper.
- Lift the object.
The researchers recorded what the robot’s cameras saw and which movements the scripted controller made.
Making the simulations varied
To help the robot work in the real world, the researchers changed many details from one simulated episode to another. This process is called domain randomization.
It is similar to practicing basketball with different balls, court colors, lighting conditions, and starting positions. The goal is to stop the robot from memorizing one exact environment.
The simulations changed:
- Wall, floor, table, and object textures.
- Light positions, colors, and brightness.
- Camera position and lens settings.
- Object positions and table heights.
- Small amounts of noise in the robot’s commands.
Stage 1: Behavior cloning
First, the robot used behavior cloning. This means learning by copying an expert, much like a student watching a teacher and trying to repeat the teacher’s actions.
The robot received information from:
- A camera on its head.
- A camera near its wrist.
- Body information, such as its height, body angle, arm positions, and gripper opening.
It then predicted the next movements, including walking speed, body height, arm movement, and gripper position.
The robot used a pre-trained walking controller for its legs. This lower-level system handled balance and walking, while the learned policy decided where to go and how to move the upper body. This made the learning problem simpler.
Stage 2: Reinforcement learning
Next, the researchers used reinforcement learning, where the robot learns through trial and error.
The robot received a very simple reward:
1if it successfully picked up the object.0if it failed.
The researchers used a method called Flow-GRPO. The technical details are complex, but the basic idea is straightforward:
- Reset several simulated robots to the same task.
- Let them try the task with slightly different random actions.
- Compare their results.
- Encourage the policy to repeat actions that led to success.
Because simulation can run many copies of the same situation at once, this comparison is cheap and fast.
This second stage helped the robot correct problems that came from copying the scripted teacher too closely.
4. What were the main findings?
More demonstrations eventually stopped helping
Increasing the number of demonstrations improved the robot at first. However, performance stopped improving after about 50,000 demonstrations.
For the full walking-and-picking task, behavior cloning reached about 67% success in simulation. Increasing the data to 150,000 demonstrations did not produce a meaningful improvement.
The researchers explain this using a hidden “phase” problem. The scripted controller internally knew whether it was currently walking, reaching, closing its hand, or lifting. The learning system could not directly see this hidden phase. It had to guess the phase from a single camera view and body reading.
This is like watching someone perform a task but not knowing which step of the instructions they are following. Even with many examples, some decisions remain difficult to copy exactly.
Reinforcement learning broke through the limit
After reinforcement learning was added, simulation success increased from:
| Training method | Full walking-and-picking success |
|---|---|
| Behavior cloning only | 67% |
| Behavior cloning plus reinforcement learning | 83% |
This shows that reinforcement learning helped the robot discover better timing and behavior than the original scripted demonstrations.
For example, the behavior-cloned robot sometimes tried to grasp an object before it was close enough. After reinforcement learning, it learned to walk closer first and grasp only when the object was within reach.
The method worked on a real robot
The final FetchMan system was tested on a real Unitree G1 robot. It had not received real-world training or special adjustments for each room.
In real-world tests, the robot achieved about 73.3% success on the full task of walking to and picking up an object in unfamiliar scenes.
The behavior-cloning-only version achieved about 56.7%, so reinforcement learning produced a large improvement.
The robot used only its head and wrist cameras, along with information about its own body. It did not receive the object’s exact position from an external tracking system.
Certain design choices were especially important
The researchers also tested different versions of the system.
A frozen DINOv3 vision model worked much better than another vision model called SigLIP. DINOv3 is a system that turns pictures into useful descriptions of shapes, positions, and visual patterns. “Frozen” means its main vision-processing abilities were not changed during robot training.
The researchers also found that predicting small changes to the robot’s current position worked better than predicting completely new absolute positions. For example, saying “move your hand 3 centimeters forward” was more reliable than saying “put your hand at exactly this location.”
When these design choices were removed, performance on the real robot dropped sharply.
The multi-object version showed promise
The researchers trained a larger version to pick up different types of objects based on an object name, such as a bowl or another household item.
Its simulated success was:
- About 40% after behavior cloning.
- About 62% after reinforcement learning.
This was worse than the single-object version’s 83%, but the task was more difficult because the robot had to recognize and grasp many different objects. The results still suggest that the method could eventually support more general-purpose household robots.
5. Why are these results important?
Training robots in the real world is expensive and risky. A humanoid robot can fall, damage itself, break objects, or need a person to supervise it. Collecting hundreds of thousands of real demonstrations would be extremely difficult.
Simulation provides a safer and cheaper alternative. Researchers can create many scenes and let robots practice repeatedly without damaging hardware.
The paper’s most important lesson is that simulation data alone may not be enough if the robot only copies a teacher. Behavior cloning reached a limit because the scripted teacher used hidden information and made abrupt changes between task stages. Adding reinforcement learning allowed the robot to improve its timing and behavior using the actual goal—successfully picking up the object.
This suggests a useful recipe for future robots:
- Use simulation to create a large and varied set of practice examples.
- Use imitation learning to give the robot a strong starting point.
- Use reinforcement learning to improve the robot beyond the examples.
- Transfer the result to real hardware using varied simulations.
Implications and limitations
FetchMan shows that a humanoid robot can learn a fairly complicated visual task almost entirely in simulation and then work in unfamiliar real-world scenes. This could make future robots cheaper and faster to train.
However, the system still has important limits:
- It mainly performs reach-and-pick tasks.
- It uses a fixed walking and balance controller, so it cannot completely change how it walks.
- It makes decisions from one current observation rather than remembering a sequence of previous views.
- It may struggle with heavy objects, unexpected pushes, or tasks requiring several steps.
- The multi-object version is still much less reliable than the single-object version.
Overall, the paper is an important step toward robots that can move around homes and interact with objects. It does not yet create a fully general household robot, but it shows how large-scale simulation, imitation learning, and reinforcement learning can work together to teach humanoid robots useful skills.
Knowledge Gaps
Knowledge Gaps, Limitations, and Open Questions
The paper leaves the following issues unresolved:
- Limited real-world evaluation: Zero-shot transfer is demonstrated on a single Unitree G1 embodiment, with only 22 manipulation trials and 30 loco-manipulation trials; broader evaluation across robots, hardware configurations, operators, environments, and substantially larger trial counts is needed.
- Unclear statistical reliability of real-world gains: The reported real-world improvements from 56.7% to 73.3% success have large standard errors, and the paper does not report hypothesis tests, confidence intervals beyond binomial standard errors, or repeated evaluations across independent scene sets.
- No systematic sim-to-real failure analysis: The study reports aggregate success rates but does not quantify whether failures arise from perception, navigation, balance, grasp selection, object dynamics, camera calibration, latency, or low-level control.
- Unverified attribution of RL improvements: The paper attributes the gain from Flow-GRPO primarily to correcting hidden phase-boundary artifacts, but does not compare against alternative explanations such as better exploration, recovery behavior, covariate-shift correction, or optimization effects.
- Insufficient ablation of the phase-boundary hypothesis: There is no controlled comparison with a demonstrator that uses smoothed transitions, an explicitly observable phase variable, recurrent history, phase labels, or a different task decomposition.
- No comparison with alternative reinforcement-learning methods: Flow-GRPO is not systematically compared with standard PPO, SAC, offline-to-online RL, behavior cloning followed by other policy-gradient methods, or flow-specific alternatives under matched compute and data budgets.
- Dependence on a strong imitation initialization: RL performs poorly from the 5k-demonstration checkpoint, but the minimum required pretraining quality, success rate, or dataset size for stable refinement is not established.
- Sensitivity to GRPO design choices is unexplored: The effects of group size, number of groups, mixed-group filtering, reward normalization, episode-length normalization, dual clipping, KL regularization, noise schedule, and rollout horizon are not reported.
- Potential selection bias from dropping homogeneous groups: Discarding all-success and all-failure groups may bias learning toward particular task distributions and may become ineffective as performance approaches either extreme; alternative estimators are not evaluated.
- Sparse-reward scalability remains uncertain: Although a binary fetch reward works for the reported tasks, it is unknown whether the same reward is sufficient for longer-horizon, multi-stage, articulated-object, or contact-intensive behaviors.
- Fixed lower-body controller limits behavior: Because SONIC is frozen, the policy cannot learn task-specific gait changes, stance widening, bracing, balance recovery, force adaptation, or manipulation strategies requiring coordinated lower-body motion.
- No evaluation under external disturbances: Robustness to pushes, uneven or slippery floors, moving obstacles, unexpected human contact, payload variation, falls, and recovery scenarios is not measured.
- Restricted manipulation capability: The robot uses only the right arm and gripper, leaving bimanual manipulation, left-arm use, hand-over-hand actions, whole-body contact, and object stabilization unexplored.
- Narrow task distribution: Training and evaluation focus primarily on reach-and-pick of a single target, with no demonstrated opening, placing, pushing, pulling, carrying, tool use, rearrangement, or multi-step household tasks.
- Limited object and scene coverage in reported results: Although the dataset contains many procedural scenes and object categories, the paper does not provide detailed breakdowns by object geometry, mass, material, size, occlusion, support surface, clutter level, or scene topology.
- Generalization outside the procedural asset distribution is uncertain: The extent to which performance depends on MolmoSpaces’ object models, textures, layouts, and grasp annotations is not tested against independently collected or substantially out-of-distribution environments.
- Object recognition and language grounding are weakly characterized: The multi-object policy is conditioned on object names, but the paper does not evaluate ambiguous names, synonyms, multiple instances of the same category, distractors, referring expressions, or language-grounding errors.
- No real-world quantitative evaluation of the multi-object model: Multi-object transfer is described only qualitatively, so its real-world success rate, category-wise performance, robustness, and comparison with the single-object model remain unknown.
- Unclear contribution of text conditioning: The paper does not isolate the benefits of text conditioning from those of increased data, object diversity, model changes, or training compute.
- Architecture comparisons are incomplete: Only SigLIP and absolute-action variants are tested, and the ablations are reported primarily for behavior cloning; the effects of recurrent policies, temporal attention, larger vision-LLMs, fine-tuned encoders, proprioceptive history, and different action horizons remain unresolved.
- Ablations are confounded by unequal real-world sample sizes: The reference and ablation variants use different numbers of real-world trials, making direct comparisons less precise and potentially confounding conclusions about the importance of the encoder and action parameterization.
- Frozen visual encoders are not fully investigated: The claim that DINOv3’s geometric features enable transfer is plausible but not causally established because feature quality, pretraining data, resolution, training schedule, and parameter count are not controlled across encoders.
- Temporal partial observability is not addressed experimentally: The policy is stateless, but no history-based, recurrent, or memory-augmented alternative is evaluated despite the demonstrator’s hidden phase and possible visual occlusions.
- Camera and sensing robustness is incomplete: Domain randomization perturbs camera parameters in simulation, but performance under real camera miscalibration, motion blur, exposure changes, image compression, occlusion, missing wrist views, and sensor failure is not quantified.
- Physics and contact-model fidelity are unvalidated: The paper does not measure sensitivity to friction, mass, compliance, grasp contacts, collision geometry, actuator dynamics, or discrepancies between simulated and real object behavior.
- Safety and failure recovery are not evaluated: The deployment results do not report falls, collisions, dropped objects, self-collisions, emergency stops, recovery time, damage risk, or operational safety constraints.
- Computational and energy costs at deployment are unspecified: Inference latency, hardware requirements, action-chunk timing, end-to-end control delay, power consumption, and real-time margins are not reported.
- Data efficiency is only partially characterized: The study varies the number of demonstrations but does not report the effect of scene diversity versus trajectory count, unique objects versus repeated objects, domain-randomization breadth, or simulation compute.
- Best-checkpoint selection may overestimate benchmark performance: The simulation results use the best validation checkpoint, but the paper does not describe checkpoint-selection criteria, the number of evaluations, or whether the held-out set was indirectly used during training decisions.
- Benchmark generality remains untested: FetchMan-Bench contains only 100 fixed initializations and primarily fetch-oriented tasks; its sensitivity to seed, task difficulty, scene novelty, and alternative embodiments or controllers is unknown.
- No reproducibility details for the full training pipeline: Important implementation information—such as model size, optimizer settings, training duration, simulator parameters, random seeds, exact dataset composition, and checkpoint-selection protocol—is insufficiently detailed to fully reproduce the reported results.
- Long-horizon composition is only proposed, not demonstrated: The claim that atomic fetch skills could support longer behaviors is not tested, and it remains unclear how errors, memory, task sequencing, and object state changes would be handled.
- Scaling beyond the reported dataset is unresolved: The paper uses up to 350,000 demonstrations and substantial simulation compute, but does not establish whether performance continues to improve with more scenes, more objects, larger models, or greater RL rollout budgets.
- Transfer of RL-learned behavior may depend on simulation-only exploitation: The study does not determine whether Flow-GRPO learns robust physical strategies or exploits simulator-specific regularities that happen not to appear in the limited real-world trials.
- The relationship between manipulation and locomotion performance is underexplored: Aggregate success rates do not separate navigation failures, approach-positioning failures, handoff timing failures, grasp failures, and lifting failures, limiting diagnosis of which subsystem constrains overall performance.
Practical Applications
Immediate Applications
- Simulation-first training pipeline for humanoid robotics (robotics industry and academia). Organizations can use the paper’s workflow—procedurally generated indoor scenes, privileged scripted demonstrations, behavior-cloning pretraining, and Flow-GRPO refinement—to train humanoids without collecting large volumes of risky real-world demonstrations. This is immediately actionable for research labs and robot manufacturers developing reach-and-pick, navigation, and whole-body control systems. Potential outputs: reusable training pipelines, synthetic trajectory generators, sim-to-real policy checkpoints, and internal evaluation suites. Dependencies: accurate robot and sensor models, a compatible low-level locomotion controller, sufficiently realistic object contacts, and access to GPU simulation infrastructure.
- Zero-shot fetch-and-pick for structured indoor environments (warehousing, logistics, and service robotics). A Unitree G1 equipped with head and wrist RGB cameras could be deployed for simple retrieval tasks such as approaching and picking up bowls, packages, containers, or tools from floors, tables, and counters. The reported real-world loco-manipulation success rate of 73.3% indicates feasibility for supervised pilot deployments rather than unattended production use. Potential workflows: item retrieval from staging areas, moving lightweight objects between workstations, and collecting objects for human operators. Dependencies: objects must be within the trained gripper and arm workspace; environments should be relatively safe and uncluttered; target identification and grasp annotations must be reliable; human oversight and emergency-stop mechanisms remain necessary.
- Robotic inspection and maintenance assistance (industrial facilities). The navigation-to-grasp behavior can support inspection workflows in which a humanoid walks to a known visual target and picks up a tool, sample, component, or removable cover. The learned handoff between locomotion and manipulation is particularly relevant to tasks where a robot must approach an object before interacting with it. Potential products: mobile tool-fetching assistants, component-retrieval modules, or visual pick-up capabilities integrated into industrial humanoid platforms. Dependencies: the current policy does not adapt its gait to heavy loads, disturbances, or unusual terrain, and it is limited to fetch-like interactions. Industrial deployment would require safety certification, force monitoring, and validation under task-specific loads.
- Benchmarking and regression testing for visual humanoid control (software and research infrastructure). FetchMan-Bench can be used to compare vision encoders, action parameterizations, imitation-learning methods, and reinforcement-learning algorithms under reproducible held-out scenes. The separation between manipulation success and full loco-manipulation success is useful for diagnosing whether failures arise from grasping or from navigation-to-manipulation coordination. Potential tools: continuous-integration tests for robot policies, standardized model cards, reproducible baselines, and performance dashboards. Dependencies: benchmark assets, simulator versions, physics parameters, and evaluation seeds must be standardized to ensure meaningful comparisons.
- Synthetic-data generation for robot learning (robotics software and simulation vendors). The task sampler and domain-randomization procedure provide a template for generating diverse training episodes by varying scene geometry, object poses, textures, lighting, camera calibration, and action noise. This can reduce the need for manually authored environments and support rapid iteration across robot embodiments. Potential outputs: procedural scene libraries, grasp-set generators, occupancy-grid navigation modules, and automated feasibility filters based on collision checking and inverse kinematics. Dependencies: synthetic assets must adequately represent real objects and contact dynamics; poor asset quality or insufficient randomization may produce brittle sim-to-real behavior.
- Improved post-training of flow-based action policies (machine learning research and robotics). Flow-GRPO can be applied to flow-matching or diffusion-like action policies after imitation pretraining, particularly when the action distribution lacks a simple likelihood and scripted demonstrations contain hidden phase transitions. The method’s sparse success reward aligns training with the actual task metric. Potential applications: refining manipulation, navigation, mobile-manipulation, and other action-chunking policies in simulation. Dependencies: the initial policy must already achieve enough successes for mixed groups to generate useful advantages. The method is therefore not a replacement for broad imitation pretraining, and sparse rewards may be ineffective for substantially harder tasks.
- Training and teaching platform for robotics education. Universities can use the open benchmark and simulation recipe to teach hierarchical control, behavior cloning, domain randomization, sim-to-real transfer, and policy-gradient refinement using a single coherent project. Students can reproduce the BC plateau and measure the effect of RL refinement. Dependencies: the full-scale experiments may exceed typical academic compute budgets, and hardware access is needed to reproduce real-world transfer results.
- Policy-development workflow for safer robotics experimentation. Teams can use simulation to explore failures, unsafe recovery states, and alternative policy behaviors before testing on a physical humanoid. This is especially valuable for tasks requiring balance restoration after failed grasps, where collecting equivalent real-world data is costly and hazardous. Dependencies: simulation cannot guarantee safety in unmodeled physical conditions. Real-hardware testing still requires staged deployment, workspace barriers, monitoring, and conservative motion limits.
Long-Term Applications
- General-purpose household humanoid assistance (daily life and home robotics). Extending the text-conditioned multi-object policy could enable a humanoid to locate and retrieve user-specified objects such as cups, toys, utensils, medicine containers, or laundry items. A natural-language object prompt could connect a voice assistant to a robot’s visual loco-manipulation policy. Potential products: “fetch this object” home assistants, object-retrieval services for older adults, and assistive robots for people with limited mobility. Dependencies: the current multi-object model reaches approximately 62% simulated success and is less robust in real-world qualitative tests. Deployment would require stronger vision-language grounding, temporal memory, clutter handling, user-intent disambiguation, reliable grasping of fragile objects, and rigorous safety validation.
- Multi-step household and workplace task composition. FetchMan’s reach-and-pick capability could become an atomic skill within longer-horizon behaviors such as “pick up the cup, place it on the counter, open the cabinet, and retrieve a plate.” The paper explicitly suggests composing atomic skills for articulated-object manipulation and longer tasks. Potential tools: hierarchical task planners, skill libraries, behavior trees, and language-conditioned robot controllers. Dependencies: reliable state estimation, memory of completed subtasks, recovery policies, collision-aware planning, and interfaces for resolving ambiguous or failed actions are not provided by the current stateless policy.
- Humanoid warehouse and fulfillment operations. At larger scale, a generalist loco-manipulation model could retrieve varied inventory from shelves, totes, carts, and floor locations, reducing the need for fixed-base automation. Text conditioning could allow operators to specify target categories or stock-keeping units. Dependencies: the present training distribution focuses on indoor fetch tasks and does not establish performance with heavy, deformable, occluded, or densely packed warehouse items. Commercial use would require high reliability, fleet-level monitoring, grasp-force control, multi-robot coordination, and adaptation to changing layouts.
- Healthcare and care-assistance robotics. A mature version could fetch lightweight supplies, mobility aids, or personal items in hospitals, rehabilitation centers, and homes. The ability to navigate to an object and manipulate it with no per-scene tuning is relevant to environments that cannot be fully structured in advance. Dependencies: medical settings impose stringent safety, hygiene, privacy, and reliability requirements. The current system has no explicit human-aware navigation, force limits for interaction with people, medical-device certification, or guarantees against dropping objects.
- Adaptive whole-body control with learned locomotion-manipulation coupling. Future systems could jointly learn lower-body balance, stance width, torso height, and arm motion rather than relying on the fixed SONIC controller. This would allow robots to brace against loads, reach farther, recover from disturbances, and operate on uneven surfaces. Potential outcomes: unified torque-level humanoid controllers, disturbance-recovery modules, and load-aware manipulation policies. Dependencies: expanding from the current 15-dimensional command interface to direct control of approximately 29 joint torques substantially increases training difficulty and safety risk. Large-scale simulation, better contact models, curriculum learning, and carefully designed rewards would be required.
- Robust operation in cluttered, dynamic, and previously unseen environments. Scaling procedural environments and improving domain randomization could support robots working around moving people, temporary obstacles, reflective surfaces, narrow passages, and changing furniture layouts. Short observation histories or recurrent policies could help infer hidden task phases that a stateless model cannot observe from one frame. Dependencies: current experiments use static indoor scenes and single-frame observations. Dynamic obstacle avoidance, temporal reasoning, uncertainty estimation, and safe fallback behaviors require new data and evaluation protocols.
- Simulation-based policy certification and public-sector deployment standards. FetchMan-Bench suggests a path toward standardized testing of humanoid systems before deployment in public spaces, workplaces, or care environments. Policy vendors and regulators could require evaluation across held-out scenes, object categories, perturbations, and failure-recovery conditions. Potential outputs: benchmark-based procurement requirements, safety cases, standardized sim-to-real reporting, and audit logs for robot-policy updates. Dependencies: simulation success rates alone are insufficient evidence of safety. Benchmarks would need validated correspondence with real-world risk, coverage of rare failures, hardware-in-the-loop tests, and transparent reporting of uncertainty.
- Object-generalist foundation models for humanoid robots. The text-conditioned extension could evolve into a generalist policy that maps visual observations and language instructions to whole-body actions across many objects and tasks. A larger vision-language backbone, joint backbone-policy fine-tuning, and substantially broader synthetic data could improve grounding and generalization. Dependencies: the paper identifies the frozen, relatively small dino.txt encoder as a limitation. Scaling model capacity and data would increase computational cost, while language grounding errors could cause the robot to grasp the wrong object. Real-world evaluation across diverse embodiments and object properties is still required.
- Self-improving robot fleets using simulation-generated failure distributions. A long-term workflow could collect failure statistics from deployed robots, reproduce representative conditions in simulation, and use group-relative RL to improve the policy without extensive physical experimentation. This would create a closed loop between fleet monitoring, synthetic replay, policy refinement, and controlled redeployment. Dependencies: failures must be accurately reconstructed in simulation, policy updates must avoid regressions, and deployment requires version control, safety gates, rollback mechanisms, and validation against distribution shifts. The identical-state group rollouts that make Flow-GRPO effective in simulation are not directly available on physical hardware.
- Transfer to other humanoid and mobile-manipulator platforms. The overall recipe could be adapted to different robots by replacing the low-level controller, robot geometry, camera configuration, grasp sets, and action interface. This could accelerate development across humanoids, quadruped-manipulators, and mobile arms. Dependencies: the reported transfer relies heavily on using the same lower-level control stack in simulation and reality, frozen DINOv3 features, and delta-action targets. New embodiments may require substantial retuning of dynamics, calibration, action scaling, camera placement, and domain randomization; zero-shot transfer should therefore not be assumed.
Glossary
- A* path: A graph-search algorithm that finds a low-cost path between locations, typically using heuristics to guide the search. “ plans an A* path on an extra-inflated copy of ”
- Antipodal sampling: A grasp-generation method that selects opposing contact points on an object to enable stable gripping. “precomputed in MolmoSpaces by antipodal sampling”
- Advantage: An estimate of how much better an action or trajectory performs relative to a reference return or baseline. “the advantage is the standardized group-relative return ”
- Behavior cloning: Imitation learning in which a policy is trained to reproduce actions from demonstrations. “In Stage~1~(\cref{sec:bc}), we behavior-clone the scripted controller”
- Binomial distribution: A probability distribution describing the number of successes in a fixed number of independent trials. “Standard errors assume a binomial distribution.”
- Closed-loop response: The behavior of a control system when its outputs are continually adjusted using feedback from its current state. “its closed-loop response is a fixed property of the system”
- Covariate shift: A distribution mismatch in which the input states encountered by a learned policy differ from those in its training data. “because covariate shift compounds small errors into out-of-distribution states”
- Cross-attention: A neural-network mechanism that allows one set of representations to selectively use information from another set. “a DiT-style transformer~ attends from noised action tokens to the observation context via cross-attention”
- Dexterous workspace: The region of positions and orientations that a robot manipulator can reach while retaining useful motion capability. “bring into the arm's dexterous workspace”
- DiT: A diffusion-transformer architecture that uses transformer layers to model denoising or flow-based generation. “The action head is a DiT-style transformer”
- Domain randomization: A sim-to-real technique that varies simulated appearance, physical parameters, or disturbances to improve real-world generalization. “On top of the task instance, every episode independently resamples the following axes”
- DINOv3: A self-supervised visual representation model used here as a frozen image encoder. “The vision encoder is a frozen DINOv3 ViT-B/16”
- Dual-clipping: An optimization technique that imposes an additional lower bound on policy-gradient losses for strongly negative-advantage samples. “with dual-clipping to bound the loss from below on large negative-advantage samples”
- Euler step: A numerical integration update that advances a differential equation using the derivative evaluated at the current point. “we replace the deterministic Euler step of the flow integrator”
- Extrinsics: Camera parameters describing its position and orientation relative to the world or robot. “cameras are perturbed in both intrinsics and extrinsics”
- Fisheye camera: A camera with an unusually wide field of view that introduces substantial radial image distortion. “a head-mounted fisheye RGB image”
- Flow-GRPO: An extension of group-relative policy optimization for policies represented by flow-matching models. “We therefore refine from the imitation checkpoint with Flow-GRPO”
- Flow integrator: A numerical procedure for integrating the vector field used by a flow-matching policy to generate actions. “the deterministic Euler step of the flow integrator”
- Flow matching: A generative-model training framework that learns a continuous vector field transporting noise into data samples. “a flow-matching policy is behavior-cloned on them”
- Gaussian density: The probability-density function of a Gaussian distribution, parameterized by a mean and covariance or variance. “the per-step log-likelihood is a closed-form Gaussian density”
- Group-relative policy optimization (GRPO): A reinforcement-learning method that estimates advantages by comparing returns from multiple rollouts of the same task. “a group-relative GRPO update refines in place”
- Grasp set: A collection of candidate gripper poses or contacts from which an object can potentially be grasped. “Each instance carries a grasp set ”
- Grasp standoff: A designated robot pose near an object from which the final reaching and grasping motions begin. “On arrival at the grasp standoff”
- Grounding: Linking a language description or token to the corresponding object or visual region in an image. “yielding grounded visual tokens that replace them in the context”
- Hard threshold: A fixed cutoff used to switch between discrete control modes or behaviors. “dispatched per step by a hard threshold on the commanded base velocity”
- Importance ratio: The ratio between the probability assigned to an action by the current policy and by the policy that generated the rollout. “the per-step importance ratio between the current and rollout-time policies”
- Imitation gap: The performance loss caused by differences between a demonstrator’s information or capabilities and those available to the learner. “scripted simulated data in particular is prone to the imitation gap”
- Inverse kinematics (IK): The computation of robot joint configurations that achieve a desired end-effector pose. “reachable under Mink~\citep{zakka10mink} inverse kinematics”
- Intrinsics: Camera parameters describing internal projection properties such as focal length and field of view. “cameras are perturbed in both intrinsics”
- Loco-manipulation: Robot behavior that combines locomotion with manipulation of objects or the environment. “visual humanoid loco-manipulation policies”
- Low-level controller: A controller that directly converts higher-level commands into joint-level actions such as torques. “The base command is tracked by a pre-trained SONIC lower-body controller”
- Marginal distribution: The probability distribution of a variable after integrating or averaging over other variables. “the marginal at remains the policy's data distribution”
- On-policy reinforcement learning: Reinforcement learning in which data is collected using the policy currently being optimized. “we fine-tune with large-scale on-policy RL”
- Occupancy grid: A spatial grid indicating which regions are free or occupied by obstacles. “we precompute a 2D free-space occupancy grid ”
- Out-of-distribution state: A state that is substantially unlike the states represented in the training data. “compounds small errors into out-of-distribution states”
- Partially observable control problem: A control problem in which the agent cannot directly observe the complete underlying environment state. “a usable prior for a high-dimensional, partially observable control problem”
- PD control: Proportional-derivative control that adjusts an actuator using present error and the rate of change of that error. “the upper-body and gripper targets are PD-tracked directly”
- Privileged information: State information available to an expert or simulator but withheld from the learned policy. “the demonstrator's access to privileged information results in imperfect imitation”
- Proprioceptive vector: A representation of the robot’s internal state, such as joint positions, body orientation, and gripper opening. “the proprioceptive vector ”
- Procedural generation: Automatic creation of environments, objects, or other data using parameterized rules. “an open ecosystem of procedurally generated indoor scenes”
- Rectified flow: A flow-matching formulation that uses a particular interpolation and velocity schedule to transport noise toward data. “matches the rectified-flow schedule”
- Reinforcement learning (RL): Learning through interaction by optimizing behavior according to rewards received from the environment. “Reinforcement learning breaks through it”
- Sim-to-real transfer: The transfer of a policy learned in simulation to physical hardware. “to close the sim-to-real gap”
- Sparse reward: A reward signal provided only for achieving a final task objective rather than for intermediate progress. “fine-tune with large-scale on-policy RL under a sparse task-success reward”
- Stochastic process: A system whose evolution includes random variables or random noise over time. “turning the deterministic ODE into a stochastic process”
- Success rate (SR): The proportion of evaluation trials that achieve the specified task objective. “Loco-manipulation SR climbs steeply from 5k to 50k”
- Synthetic data: Artificially generated training data produced by a simulator or computational process. “To overcome the data scarcity that robotics faces , synthetic data has been used”
- Token: A learned vector representation of an input element, such as an image patch or state. “the proprioceptive vector is projected into a single state token”
- Trajectory: A time-ordered sequence of states, observations, and actions produced during an episode. “The resulting trajectories pair the observations with 15-dimensional whole-body commands”
- Vision-LLM (VLA): A model that connects visual and language representations to predict or control actions. “a VLA-style one”
- ViT-B/16: A Vision Transformer configuration using a base-sized model and image patches of pixels. “a frozen DINOv3 ViT-B/16”
- Whole-body controller: A robot controller that coordinates multiple body parts, including locomotion and manipulation, as one system. “a scripted, privileged whole-body controller”
- Zero-shot deployment: Applying a trained model to new environments or hardware without additional task-specific training or tuning. “deploys it zero-shot to a real Unitree G1”
