FineVLA: Refined Humanoid Loco-Manipulation

• FineVLA is a system that integrates whole-body planning with locomotion and manipulation to overcome data bottlenecks in humanoid imitation learning.
• It employs a hybrid control abstraction that combines upper-body joint commands with RL-based lower-body locomotion to ensure dynamic stability and precise collision avoidance.
• The approach adapts contact-rich, object-centric skills via whole-body inverse kinematics, achieving significant performance gains on complex, long-horizon tasks.

HumanoidMimicGen is a demonstration-generation system for humanoid robot loco-manipulation that addresses the data bottleneck in imitation learning by adapting contact-rich whole-body skills from a small number of source demonstrations to new states and object poses, then interleaving those adapted skills with whole-body locomotion and manipulation planning to produce stable, collision-free demonstrations across diverse scenes and layouts (Lin et al., 26 May 2026). It is designed for settings in which a humanoid must both walk and manipulate with its arms, torso, and hands, and it is evaluated on a new simulated benchmark of nine loco-manipulation tasks. The central claim is that humanoid data generation cannot be reduced to naïve end-effector retargeting: locomotion, balance, stance selection, and contact-rich manipulation must be coordinated at the whole-body level.

1. Research context and motivation

HumanoidMimicGen is situated within a line of work that treats imitation learning as effective but fundamentally data hungry. Earlier systems showed that automated data generation can scale robot learning for manipulators by reusing a small seed set of demonstrations. MimicGen generated over 50K demonstrations across 18 tasks from about 200 human demonstrations by decomposing tasks into object-centric subtasks and retargeting those subtask segments to new contexts (Mandlekar et al., 2023). DexMimicGen extended this strategy to bimanual dexterous manipulation, generating 21K demonstrations from 60 source human demos while introducing parallel, coordination, and sequential subtasks for multi-arm settings (Jiang et al., 2024).

HumanoidMimicGen argues that these assumptions do not transfer directly to legged humanoids. Existing data-generation methods such as MimicGen and DexMimicGen adapt task-space end-effector motions around object-centric skills and assume that each limb can be controlled “independently enough” in task space. For humanoids, the lower body must actively maintain balance, manipulation often requires body repositioning before contact-rich arm motion is feasible, and pure task-space replay can yield collisions, instability, or unreachable postures (Lin et al., 26 May 2026). The system is therefore framed as a response to a specifically humanoid difficulty: the action space is high-dimensional and composite, and the legs are part of a balance-and-locomotion system rather than merely an additional pair of manipulators.

2. Formal model and hybrid control abstraction

The method models a humanoid with joint set

$${\cal J}=J_l\cup J_t\cup J_{a_l}\cup J_{h_l}\cup J_{a_r}\cup J_{h_r},$$

covering legs, torso, left and right arms, and hands, with robot state

$q \in \mathbb{R}^{|{\cal J}|}.$

Objects are represented by $SE(3)$ poses of coordinate frames $f \in {\cal F}$. The task is formulated as a POMDP with state space ${\cal S}$, observation space ${\cal O}$, action space ${\cal A}$, and initial-state distribution ${\cal S}_0$, with $\mathrm{supp}({\cal S}_0)\subseteq{\cal S}$ (Lin et al., 26 May 2026).

A state $s\in{\cal S}$ includes robot joints $q \in \mathbb{R}^{|{\cal J}|}.$0, end-effector poses $q \in \mathbb{R}^{|{\cal J}|}.$1, and object-frame poses $q \in \mathbb{R}^{|{\cal J}|}.$2. Observations contain proprioception and camera images. Actions are joint-space position targets $q \in \mathbb{R}^{|{\cal J}|}.$3, with induced end-effector pose $q \in \mathbb{R}^{|{\cal J}|}.$4. Behavior cloning policies are trained from a dataset of demonstrations

$q \in \mathbb{R}^{|{\cal J}|}.$5

The paper’s main conceptual intervention is a hybrid action space inspired by Homie. Upper-body behavior is represented through joint-position commands, while lower-body motion is requested through a locomotion command

$q \in \mathbb{R}^{|{\cal J}|}.$6

where $q \in \mathbb{R}^{|{\cal J}|}.$7 and $q \in \mathbb{R}^{|{\cal J}|}.$8 are planar base velocities, $q \in \mathbb{R}^{|{\cal J}|}.$9 is yaw rate, and $SE(3)$0 is desired torso height (Lin et al., 26 May 2026). A learned RL locomotion controller consumes these commands together with current and target arm and torso configurations and outputs dynamically feasible leg joint targets. This control abstraction is intended to let the generation algorithm request coarse base motion while delegating stable walking to a learned controller.

3. Skill representation, ordering, and trajectory adaptation

Input demonstrations are manually segmented into object-centric skills

$SE(3)$1

where $SE(3)$2 is the end-effector involved, $SE(3)$3 is the reference object frame, and $SE(3)$4 is a contiguous subsequence of the source demonstration (Lin et al., 26 May 2026). Each skill is annotated with precedence constraints $SE(3)$5 and optional concurrency constraints $SE(3)$6. A precedence pair $SE(3)$7 means that $SE(3)$8 must finish before $SE(3)$9 starts, whereas a concurrent pair $f \in {\cal F}$0 means that the two skills should start together.

Concurrency is reduced to precedence by transitively adding order constraints, yielding a DAG over skills, $f \in {\cal F}$1. Execution proceeds by repeatedly selecting the current frontier

$f \in {\cal F}$2

This produces batches of skills that are currently executable and allows the system to handle tasks in which some actions must occur in sequence and others may occur concurrently.

Adaptation is object-centric and spatially invariant. For a skill $f \in {\cal F}$3, the source end-effector pose is expressed relative to the source object frame and then re-expressed in the new object pose. The paper describes this as preserving the local geometry of the skill while changing the absolute object pose (Lin et al., 26 May 2026). During whole-body skill adaptation, source actions are replayed timestep by timestep in transformed form; adapted target poses are collected for all active skills in the batch, whole-body IK is solved, and the resulting configuration waypoints are appended to the adapted trajectory. Hand joint positions $f \in {\cal F}$4 are replayed without modification.

This representation is significant because it preserves the object-centric regularity exploited by earlier MimicGen systems, but embeds it within a whole-body adaptation procedure appropriate for contact-rich humanoid skills. A plausible implication is that the method retains the compositional advantages of subtask-based generation while avoiding the assumption that retargeted arm motion is feasible without stance adjustment.

4. Whole-body planning, locomotion interleaving, and collision handling

The generation pipeline is greedy and explicitly interleaves locomotion with manipulation (Lin et al., 26 May 2026). On each iteration, it selects currently executable skills from the precedence DAG, adapts their initial end-effector target poses to current object poses, solves whole-body inverse kinematics to obtain candidate full-body configurations, derives a “switch” configuration between locomotion and manipulation, plans a locomotion trajectory to reach that switch state, executes locomotion with the RL controller, replans and executes manipulation once stable at the switch state, replays the adapted skill trajectory with whole-body IK tracking, and then checks task success before continuing.

The switch-state construction is central. Given current state $f \in {\cal F}$5 and joint configuration $f \in {\cal F}$6, the algorithm computes target end-effector poses $f \in {\cal F}$7, calls

$f \in {\cal F}$8

and obtains candidate full-body solutions $f \in {\cal F}$9. For each candidate, it constructs a switch configuration ${\cal S}$0 by copying the current configuration and replacing locomotion joints with those from ${\cal S}$1:

${\cal S}$2

It then plans a locomotion motion primitive

${\cal S}$3

If locomotion succeeds, the achieved state becomes the new switch state and a manipulation motion is planned:

${\cal S}$4

The resulting organization is a walking phase followed by a static manipulation phase, repeated at the skill level.

Collision handling is implemented through a sphere-based geometry model. Robot and moving objects are represented as unions of spheres derived automatically from meshes by sampling surface points, building candidate tangent spheres, inflating radii by a small margin ${\cal S}$5 m, and greedily selecting a subset of ${\cal S}$6 spheres maximizing coverage (Lin et al., 26 May 2026). This representation supports GPU-accelerated collision checking and planning in cuRobo. Because sphere unions over-approximate geometry, the method shrinks spheres that are currently in collision before IK or motion-planning calls. For target end-effector poses, shrinking is applied only to links in the same rigid kinematic component as the end-effector.

The IK procedure also encodes a preference for minimal whole-body disturbance. The appendix describes an iterative strategy with free-joint sets

${\cal S}$7

corresponding to arm only, arm plus torso, arm plus leg, and full-body freedom, while minimizing the weighted ${\cal S}$8 distance

${\cal S}$9

This biases the planner toward solutions that change as few joints as possible and helps determine when locomotion is necessary rather than optional.

5. Benchmark, policy training, and empirical results

HumanoidMimicGen is evaluated on a simulated G1 humanoid benchmark in robosuite and MuJoCo comprising nine tasks: Box Lift Floor, Push Button, Box Lift, Push Shelf Forward, Drill Lift, Drill PnP, Box Table To Shelf, Pick Drill From Holder, and Drill Lift Obstacle (Lin et al., 26 May 2026). These tasks span different locomotion demands, single-arm versus bimanual interaction, vertical reaching, contact-rich object handling, and long-horizon sequences. Initial states are randomized over robot root pose and object poses, subject to feasibility constraints. Each task uses binary success.

For each task, the authors collect one human teleoperated source demonstration using a Pico VR controller, then generate 1,000 demonstrations in simulation. Policies are trained by fine-tuning a pretrained GR00T N1.6 VLA model with imitation learning using a single egocentric RGB camera at ${\cal O}$0 and proprioception. Training runs for 25k steps, with checkpoints every 5k steps; each checkpoint is evaluated on 100 episodes, and the best checkpoint is selected. The paper also compares against Diffusion Policy and a flow-matching transformer using the same generated dataset (Lin et al., 26 May 2026).

Method	Average PSR
1 Human Demo	0.26
100 Human Demos	0.48
DexMimicGen+	0.33
HumanoidMimicGen	0.89

These results indicate that HumanoidMimicGen-generated data substantially outperforms both limited real demonstration training and an extended DexMimicGen-style baseline without whole-body planning or collision-aware locomotion. The paper reports especially large gains on long-horizon and locomotion-heavy tasks such as Push Shelf Forward, Drill Lift, Box Table To Shelf, Pick Drill From Holder, and Drill Lift Obstacle (Lin et al., 26 May 2026).

The real-world evaluation uses flow-matching policies trained from scratch on four tasks: ThrowBottle, BoxToCart, PickCanister, and PickCanisterWithObstruction. Training mixes limited real demonstrations with synthetic demonstrations: 30 real + 500 simulated for ThrowBottle and BoxToCart, and 50 real + 1000 simulated for PickCanister and PickCanisterWithObstruction. Evaluation is on a real G1 humanoid with RGB observations from an OAK-D camera, with upper-body control at 25 Hz, lower-body inference at 50 Hz, and low-level position control at 200 Hz. Policies trained only on real data average 0.51, whereas policies co-trained with HumanoidMimicGen simulation data average 0.71, a +0.20 absolute gain. Task-wise gains are +0.15 on ThrowBottle, +0.25 on BoxToCart, +0.25 on PickCanister, and +0.15 on PickCanisterWithObstruction (Lin et al., 26 May 2026).

Ablations identify both policy-learning and data-generation sensitivities. Using the same 1,000 generated demonstrations, average PSR is 0.89 for the VLA, 0.86 for flow matching, and 0.51 for Diffusion Policy. Removing motion noise reduces average PSR from 0.89 to 0.49, and removing initialization noise reduces it to 0.51 (Lin et al., 26 May 2026). The paper therefore treats both trajectory diversity and initial-state diversity as critical to policy generalization.

6. Relation to adjacent research directions and limitations

HumanoidMimicGen extends the MimicGen family from stationary manipulation to legged whole-body loco-manipulation. MimicGen’s original formulation depends on object-centric segmentation, spatial transformation of end-effector trajectories, interpolation, open-loop execution, and success-based filtering to generate large-scale datasets for standard imitation learning (Mandlekar et al., 2023). DexMimicGen adds bimanual dexterous manipulation by introducing asynchronous execution, synchronization, and ordering constraints for dual-arm tasks (Jiang et al., 2024). HumanoidMimicGen inherits the emphasis on object-centric skills and successful rollout filtering, but replaces the assumption of independently retargetable end-effectors with a hybrid control abstraction, whole-body IK, explicit locomotion planning, and collision-aware switch-state reasoning (Lin et al., 26 May 2026).

It also differs from other humanoid imitation paradigms that are not primarily data-generation systems. GenMimic converts generated human videos into robot trajectories through 4D reconstruction, PHC retargeting, and a physics-aware RL tracking policy conditioned on 3D keypoints, with zero-shot deployment on a Unitree G1 (Ni et al., 4 Dec 2025). VisualMimic addresses visual humanoid loco-manipulation through a hierarchical stack comprising a task-agnostic low-level keypoint tracker and a task-specific high-level generator trained from egocentric depth and proprioception (Yin et al., 24 Sep 2025). A narrower but related line concerns socially interactive motion generation: imitation of human head motion for active-speaker detection on a Nao robot uses a VAE over head trajectories and an MLP mapping fixation targets to latent codes, with 174 recorded trajectories and real-time deployment in conversation (Ding et al., 2024). These systems share an interest in human-like humanoid motion, but their primary objects of study are motion tracking, visual control, or social gaze behavior rather than automated demonstration synthesis for whole-body loco-manipulation.

The limitations of HumanoidMimicGen are stated explicitly. It requires manual skill segmentation and manual precedence or coordination annotation; benchmark environments, initial-state distributions, and success conditions are manually designed; the method assumes a fixed set of object-centric skills and a fixed skill-sequence structure; and its rigid object-frame transformations do not handle large intra-category shape variation or ambiguous contact affordances well (Lin et al., 26 May 2026). The authors suggest future work on automating annotation and environment construction, potentially using foundation models, and on adopting methods such as CP-Gen for better constraint-preserving generalization. This suggests that the present system should be understood as a structured, planner-mediated data-generation framework rather than a fully autonomous discovery mechanism for humanoid behavior.