Papers
Topics
Authors
Recent
Search
2000 character limit reached

VLK: Learning Humanoid Loco-Manipulation from Synthetic Interactions in Reconstructed Scenes

Published 29 Jun 2026 in cs.RO, cs.AI, cs.GR, and eess.SY | (2606.30645v1)

Abstract: Perception-based humanoid loco-manipulation requires connecting egocentric observations and task instructions to whole-body motion. Learning this mapping requires synchronized egocentric images, language commands, and robot-compatible kinematic trajectories, yet no existing data source provides this complete tuple at scale. We address this bottleneck by generating vision-language-kinematics (VLK) supervision synthetically in reconstructed scenes. Our pipeline leverages 3D Gaussian Splatting to reconstruct metric-scale indoor environments, synthesizes navigation and object-interaction trajectories using privileged scene information, and renders paired egocentric observations after the fact. We produce 48,000 paired trajectories with no human intervention and train a VLK policy that predicts short-horizon whole-body kinematic trajectories. A whole-body tracker converts these predictions into actions on the physical humanoid. We evaluate on the physical Unitree G1 performing navigation and single-object transport, demonstrating that synthesized interactions in reconstructed scenes provide effective supervision for sim-to-real perception-based humanoid loco-manipulation. Project Website: https://vision-language-kinematics.github.io/

Summary

  • The paper introduces a fully synthetic VLK pipeline using 3D Gaussian Splatting and diffusion-based motion generation to synthesize paired vision, language, and kinematics data.
  • It achieves high sim-to-real transfer with over 90% success in navigation and manipulation tasks on the Unitree G1 through precise synthetic supervision.
  • The approach decouples perception and control via a tracking controller, enabling scalable generation of complex loco-manipulation behaviors.

Learning Perception-Conditioned Humanoid Loco-Manipulation from Synthetic Interactions in Reconstructed Scenes

Introduction

The policy learning bottleneck in humanoid loco-manipulation arises from a lack of large-scale, synchronized datasets containing egocentric RGB observations, linguistic instructions, and robot-compatible full-body kinematics. Addressing this, the "VLK: Learning Humanoid Loco-Manipulation from Synthetic Interactions in Reconstructed Scenes" (2606.30645) introduces a fully synthetic supervision pipeline that leverages high-fidelity scene reconstructions via 3D Gaussian Splatting (3DGS), scene-driven trajectory synthesis, and post hoc egocentric rendering to produce paired vision-language-kinematics (VLK) data. This enables the end-to-end training of policies mapping visual-linguistic inputs to short-horizon kinematic trajectories, closed by a tracking controller on hardware. The paper demonstrates sim-to-real generalization on the Unitree G1 across navigation and box-transport tasks, revealing high real-world efficacy with exclusively synthetic multi-modal data. Figure 1

Figure 1: Method overview. Synthetic scenes, scene annotations, and trajectory generation are used to synthesize paired VLK supervision, training a policy that maps egocentric observations and language to G1 trajectories. A tracker converts these for deployment.

Synthetic VLK Data Generation

The core contribution is a pipeline to efficiently generate high-quality paired (image, language, kinematics) tuples without manual demonstrations:

  • Scene reconstruction and annotation: 3DGS reconstructs metric-scale indoor scenes from RGBD scans (e.g., iPhone Polycam), preserving appearance and layout for realistic rendering. Scene semantics (objects, walkable regions) are manually annotated, enabling task grounding.
  • Waypoint and interaction synthesis: For each scene, task-specific object targets, feasible initial poses, and waypoints are sampled within semantic and geometric constraints. Navigation trajectories are synthesized using models trained on G1 motion datasets; object manipulation is synthesized by adapting SMPL-based human-object interactions (e.g., OMOMO, retargeted via OmniRetarget) to the G1 kinematic chain.
  • Conditional diffusion motion generation: Language- and scene-conditioned diffusion models synthesize G1 trajectories spanning navigation and object interaction tasks. For manipulation, object geometry, wrist contact, and relative poses are part of the conditioning, providing granular control over scene-object interactions.
  • Egocentric rendering: IsaacSim renders egocentric image sequences by replaying synthetic trajectories in reconstructed scenes, optionally applying domain randomization (lighting/camera/appearance) to bridge the sim-to-real gap.

This synthetic pipeline achieves a generational throughput of >48,000>48,000 unique trajectories per environment (12 modes × 4 layouts × 1000), supporting efficient exploration of data scale and task complexity. Figure 2

Figure 2: Examples of generated VLK supervision, showing the synchronization of egocentric image, language command, and G1 kinematic trajectory in each sequence.

Figure 3

Figure 3: Instances from the synthetic dataset, demonstrating diversity in visual input, instruction, and motion structure.

Vision-Language-Kinematics Policy Architecture

The VLK policy predicts a future window of G1 full-body states (positions, orientations, joint angles, wrist-object contact) conditioned on the current egocentric observation, linguistic prompt, and robot state. Key elements:

  • Input/Output: Inputs are egocentric RGB images, task instructions (templated), and current kinematics. Outputs are HH-step trajectory predictions (H=30H=30 for 1s at 30Hz), including wrist contact labels.
  • Initialization: The policy is initialized from a pretrained foundation model (π0.5\pi_{0.5} [intelligence2025pi_]), then fine-tuned with a flow-matching (diffusion) loss on the VLK dataset to match predicted and ground-truth future trajectories. Auxiliary objectives include foot-contact prediction, trajectory consistency, forward-kinematics end-effector accuracy, and foot-sliding penalties.
  • Training: The model is trained with synthetic data exclusively, demonstrating robust transfer to physical hardware without access to real-world teleoperation data or human motion capture paired with egocentric observations.

Whole-Body Tracking and Real-World Deployment

VLK trajectory outputs are executed on the G1 via a high-fidelity tracker (SceneBot [chen2026scenebot]), which:

  • Maps trajectory predictions to joint-level PD targets and controls wrist-object contact.
  • Is perception-agnostic: only receives kinematic references and proprioception.
  • Runs at 50 Hz50\,\text{Hz}, while VLK inference operates asynchronously at ∼1.8 Hz\sim 1.8\,\text{Hz}, communicating over shared memory and websockets.

System latency measurements show stable inference (total ∼\sim63ms per replan), leaving significant margin under real-time constraints. Figure 4

Figure 4: System deployment overview. Three asynchronous processes coordinate image ingestion, trajectory prediction (inference client), and control (tracker), with chunked trajectory merging enabling low-latency execution.

Empirical Results

Synthetic Data Volume and Diversity

Generation throughput enables rich data ablations. Increasing synthesized data volume notably improves policy performance, especially for high-contact manipulation tasks, confirming the hypothesis that complex loco-manipulation is data-hungry. Navigation tasks saturate with less data.

Sim-to-Real Evaluation

  • Navigation and floor-level box transport achieve high real-world success rates (e.g., >90%>90\%), both in closed-loop IsaacSim/MuJoCo and G1 deployments.
  • Surface-level interactions remain more challenging, as limited coverage in OMOMO constrains generalization to novel support heights and object classes. Figure 5

    Figure 5: Real-world deployment on the G1. The system performs multi-stage navigation and object manipulation in lab and apartment scenes, subject to varying layouts and visual conditions.

Effect of Visual Domain Randomization

Domain randomization during rendering (lighting, camera, appearance) proves essential for transfer robustness. Without randomization, success rates plummet (down to 41%); full domain randomization restores them to 90% in visual sim, indicating strong policy generalization to physical world visual idiosyncrasies.

Analysis and Implications

Practical Impact

The demonstrated pipeline presents an efficient, scalable alternative to traditional data-collection for humanoid loco-manipulation—sidestepping the prohibitive cost of teleoperation and motion-capture in favor of synthetic supervision. Success on the G1 validates that detailed scene reconstructions (3DGS) and task-driven synthesis produce actionable state-distribution coverage for generalization.

Theoretical Implications

This work provides strong empirical evidence that direct scene-to-action mapping can be learned purely from simulation, as long as the synthetic data is sufficiently scene-grounded and physically consistent. The decoupling of perception and low-level control (tracker abstraction) further enables modularity, where higher-capacity policies can leverage improved synthetic datasets or trajectory synthesizers.

Limitations and Future Work

  • Manipulation diversity: The current framework is constrained to bimanual transport of large (box-like) objects, as the interaction synthesis module is reliant on OMOMO's limited object taxonomy. Extension to small-object precise grasping will require new datasets and refined tracking/policy routines.
  • Closed-world grounding: Scene annotation remains semi-manual; scaling to open-world unstructured environments with in-the-wild semantics would demand automated annotation or self-supervised pipeline extensions.
  • Action granularity: The policy is trained on short-horizon trajectory windows. Long-horizon planning, multi-step compositionality, and more abstract task decomposition are open frontiers.

Conclusion

The paper establishes that large-scale synthetic VLK data, generated via photorealistic reactive scene reconstructions and task-conditioned motion synthesis, can close the supervision gap for humanoid perception-to-action learning. With strong real-world results on navigation and manipulation, this methodology paves the way for cost-effective and generalizable training frameworks for vision-language humanoid robotics, moving towards broad-scene sim-to-real deployments. Figure 1

Figure 1: Method overview. Synthetic scenes, trajectory generation, and VLK policy training for short-horizon kinematics prediction with tracker-based execution.

Figure 2

Figure 2: Supervision sequences pairing robot-view observation, language query, and trajectory.

Figure 5

Figure 5: Real-world executions on the G1 in varied scenes, tasks, and lighting.

Figure 3

Figure 3: Examples highlighting dataset visual and motion diversity.

Figure 4

Figure 4: System architecture for on-robot deployment of asynchronous perception, inference, and control.

References

  • "VLK: Learning Humanoid Loco-Manipulation from Synthetic Interactions in Reconstructed Scenes" (2606.30645)
  • SceneBot [chen2026scenebot]
  • π0.5\pi_{0.5} [intelligence2025pi_]
  • 3DGS [3dgs]
  • OMOMO [li2023omomo]
  • GaussGym [gaussgym]

End of Analysis

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Explain it Like I'm 14

Overview

This paper is about teaching a human‑shaped robot (a “humanoid”) to move around and handle objects using what it sees and what it’s told in plain language. The big idea is to create a huge set of practice examples without needing people to manually control the robot. The team builds “digital twins” of real rooms, makes the robot practice inside those digital rooms, and uses that practice to help the robot act correctly in the real world.

The Main Questions

To make this easy to follow, here are the main questions the researchers asked:

  • Can we create enough good training data that ties together what the robot sees (images), what it hears (a short instruction), and how it should move its whole body (a plan for its joints) without using expensive human demonstrations?
  • If we train only on synthetic (computer-generated) data from realistic 3D reconstructions of real rooms, will the robot’s skills transfer to the physical robot in the real world?
  • How much data is enough, and what tricks help the training work better (for example, randomizing lighting to mimic real-world changes)?

How They Did It

Think of their approach like building a super-realistic video game level from a real room, then letting the robot practice inside that level before trying it in real life.

Here’s the process, in everyday terms:

  1. Build 3D “digital twins” of real rooms
    • They scan real indoor spaces (like a lab and an apartment) using a phone app.
    • Using a method called “3D Gaussian Splatting” (you can think of it as making a 3D photo you can walk through), they get photorealistic, life-sized 3D scenes.
  2. Mark important stuff in the scene
    • They label walkable areas and place 3D boxes around objects (like tables or boxes) so the system knows where things are.
  3. Auto-generate robot motions
    • The robot they use is a Unitree G1 humanoid.
    • They automatically create short motion “stories” for two types of tasks:
      • Navigation: move to a target spot or face a certain direction.
      • Object interaction: pick up a box and put it down somewhere else (on the floor or on a surface).
    • “Kinematics” just means a description of where each part of the robot’s body is and how it moves, like a frame-by-frame animation for all the joints.
  4. Render the robot’s “eye view”
    • They replay the robot’s motion inside the digital scene and render the exact images a head‑mounted camera would see.
    • This gives them pairs of:
      • the robot’s camera image (what it “sees”),
      • a simple instruction (like “walk to the red chair”),
      • and the correct short-term full-body motion plan (where to move each joint next).
    • They also add “domain randomization,” which is like changing lights and camera settings so the robot doesn’t overfit to perfect, fake images.
  5. Train the decision-making model (the “VLK policy”)
    • “VLK” stands for Vision–Language–Kinematics.
    • Given one camera image, a short instruction, and the robot’s current body state, the model predicts the next 1 second of whole-body motion and simple contact signals (whether each wrist should be holding the object).
    • This is like the robot making a short, quick plan, then repeating that planning step over and over as it moves.
  6. Execute on the real robot with a tracker
    • A “whole-body tracker” turns the predicted motion plan into actual motor commands the real robot can follow, while respecting the contact signals for holding the object steady.
    • The whole system runs in real time (about 31 ms per planning step), which is fast enough for smooth movement.

In total, they automatically created 48,000 paired examples (image + instruction + motion plan) without human teleoperation.

What They Found

Here are the key results, explained simply:

  • The synthetic data worked in the real world
    • After training only on the synthetic data from the reconstructed scenes, the real robot could:
    • Walk to target places and turn to face certain directions with very high success.
    • Pick up and put down a box on the floor reliably.
    • Pick up and put down a box on tables/surfaces reasonably well (harder than floor tasks, but still often successful).
    • Example real-world results (20 trials per task in each of two rooms):
    • “Walk to” and “Turn around”: close to perfect.
    • “Pick (Floor)” and “Put (Floor)”: mostly successful.
    • “Pick (Surface)” and “Put (Surface)”: good but noticeably harder.
  • More data helps, especially for tricky manipulation
    • Navigation worked well even with less training data.
    • Handling objects, especially on different surface heights, improved a lot as they added more training examples.
  • Visual randomization is very important
    • Training without lighting/camera variation made the robot much worse when faces with changed lighting or viewpoint.
    • With full randomization, walking success under visual changes was about twice as high as without it.
  • Real-time and stable
    • The planning runs fast enough to be used on the robot continuously, with techniques to make motion chunks flow smoothly.

Why This Matters

This work shows a practical way to scale up training for humanoid robots without needing tons of expensive human demonstrations. By:

  • scanning real rooms once,
  • generating thousands of realistic practice runs in those scenes, and
  • training a model that predicts short, safe body motions,

the robot can learn to connect its “eyes and ears” (camera and instruction) to full-body actions. This approach could help future home or workplace robots get better, faster, and cheaper—especially for tasks that mix walking and handling objects in messy, human environments.

Limitations and What’s Next

  • Today, the system focuses on carrying a box-like object. It doesn’t yet handle small, precise grasps (like picking up a pen or a cup) that require finer control.
  • To reach that level, they’ll need richer training data for small-object interactions and a lower-level controller designed for precise fingered grasps.

In short, this paper’s method is a strong step toward teaching humanoid robots everyday skills by letting them “practice” in realistic digital versions of the real world and then bringing those skills back to real life.

View Paper Prompt  View All Prompts 

Knowledge Gaps

Knowledge gaps, limitations, and open questions

Below is a single, concrete list of what remains missing, uncertain, or unexplored in the paper, written to guide actionable future research.

  • Dependence on manual scene annotation: the pipeline requires semantic 3D bounding boxes and walkable-region labels; the extent of human annotation effort, its scalability, and ways to automate it (e.g., open-vocabulary 3D segmentation) are not addressed.
  • Generalization to unseen environments: training and evaluation focus on two scanned environments with layout variations; performance in entirely new, unscanned homes/offices and across diverse architectural styles is not evaluated.
  • Requirement for scene scans at train time: the approach assumes access to reconstructed scenes; feasibility when scans are unavailable or inaccurate, and alternatives (e.g., synthetic worlds only, or rapid on-site scanning) remain open.
  • Sensitivity to reconstruction errors: the impact of 3DGS artifacts (holes, floaters, scale drift, misalignment) on rendered observations and learned policies is not quantified.
  • Limited task diversity: tasks are restricted to navigation and single-box transport; extension to multi-object tasks, tool use, articulated objects (doors, drawers), and deformables is untested.
  • Small-object manipulation gap: the system (data and controller) is tailored to large, box-like objects and wrist–object “contact”; precise grasping of small objects with varied shapes and friction is unsupported and unevaluated.
  • Templated language only: instructions are simple templates; robustness to natural, diverse, compositional, or ambiguous language (and multi-lingual inputs) is unknown.
  • Policy input modality constraints: VLK uses a single RGB image (left camera) at each step; benefits of temporal visual context, stereo/depth, event cameras, or proprioception-only fallbacks are not explored.
  • Partial observability and memory: the policy lacks an explicit memory or belief state; handling occlusions, long-term dependencies, and persistent goals under POMDP settings is not studied.
  • Short-horizon prediction design: the choice of H=30 frames (~1 s) and its trade-offs (stability, responsiveness, long-horizon planning) are not ablated; effects of horizon length and overlap on performance remain open.
  • Contact supervision and execution mismatch: the tracker relies on predicted binary wrist-contact labels without sensing; how to detect, correct, or adapt when predicted contact disagrees with reality (slip, missed grasp) is unaddressed.
  • Absence of tactile/force feedback: no tactile sensing or force control is used; integrating tactile/impedance control for robust grasps and contact-rich manipulation is not explored.
  • Physics fidelity in synthesis: motion synthesis uses privileged scene information but does not enforce detailed physical constraints (e.g., friction, mass, compliance); the downstream sim-to-real impact is not quantified.
  • Surface-height and support variability: failures on surface-level tasks suggest inadequate coverage of support-surface heights and placements in training; systematic dataset expansion and synthesis for varied supports is needed.
  • Domain randomization limits: DR ablations are reported for walking under perturbations; effects on manipulation tasks (especially contact-rich ones) are not evaluated.
  • Dynamic scenes and moving agents: robustness to moving humans/objects, background motion, and non-stationary lighting is not tested.
  • Clutter and occlusion robustness: performance in heavily cluttered or visually ambiguous scenes (specularities, transparent objects, mirrors) is untested.
  • Safety and collision avoidance: beyond learned behaviors and annotated walkable regions, there is no explicit perception-driven collision avoidance or safety certification; guarantees under human proximity are absent.
  • Dependence on specific hardware: all real-world results are on the Unitree G1; portability to other humanoids with different kinematics/actuation and the retargeting/tracking adaptations required are not assessed.
  • Comparison to baselines: there is no head-to-head comparison against teleoperation-supervised VLAs, video-pretrained policies, or alternative sim-to-real pipelines; relative data efficiency and accuracy remain unclear.
  • Failure analysis depth: beyond success rates, detailed failure modes (perception vs. kinematics vs. tracking), recovery behaviors, and error attribution are not reported.
  • Scene diversity and scale: the dataset is generated from two environments; how performance scales with many more scenes, varied materials/textures, and broader object taxonomies remains unknown.
  • Object state estimation at deployment: no explicit object pose/identity tracking or state estimation loop is integrated; how the system recognizes grasp/placement completion or handles object pose drift is open.
  • Replanning frequency and latency: while 31 ms inference is reported, the sensitivity of performance to latency spikes, dropped frames, and control frequency on hardware is not studied.
  • Long-horizon chaining evaluation: chaining atomic prompts is shown qualitatively; quantitative evaluation of long-horizon, multi-stage tasks (success over extended durations, compounding error) is lacking.
  • Automatic waypoint and instruction generation: waypoints and instructions rely on annotated scene semantics; learning to infer task-relevant waypoints and generate diverse language automatically from scenes is not explored.
  • Robustness to camera calibration drift: DR includes camera perturbations, but real-world calibration changes over time (mount flex, impacts) and their effect on performance are not quantified.
  • Ethical and practical deployment considerations: the need to scan private indoor spaces, data privacy, and operational constraints in real homes/offices are not discussed.
View Paper Prompt  View All Prompts 

Practical Applications

Immediate Applications

Below are actionable use cases that can be deployed with modest engineering effort based on the paper’s demonstrated pipeline (3DGS scene reconstruction, synthetic VLK data generation, short-horizon kinematic prediction, contact-aware whole-body tracking, and real-time deployment on a Unitree G1).

  • Site-specific pretraining for warehouse and logistics robots (navigation + box transport)
    • Sectors: robotics, logistics, retail
    • What: Scan customer sites (aisles, stock rooms), generate synthetic egocentric data and G1-compatible kinematic trajectories for “walk to,” “pick/put box (floor/surface),” and “carry” tasks; fine-tune VLK and deploy with a whole-body tracker for day-one productivity.
    • Tools/workflows: Polycam or LiDAR-based scans → 3D Gaussian Splatting → waypoint-driven motion synthesis → Isaac Sim rendering with domain randomization → VLK fine-tuning → Triton real-time inference → contact-aware tracking
    • Assumptions/dependencies: Box-like objects; accurate scene scans and manual/semi-automatic annotations (walkable regions, object boxes); camera calibration match; sufficient on-site GPU or edge inference; Unitree G1 (or close morphology) and safety procedures; tasks remain within demonstrated capability (single-object transport, limited heights)
  • Facility services and lab operations (reconfiguration, light materials handling)
    • Sectors: robotics, facilities management, R&D labs
    • What: Use “scan-to-skill” for rearranging lab equipment boxes, moving supplies between rooms, or setting up event spaces using language-conditioned navigation and transport.
    • Tools/workflows: Same as above; chain atomic skills for longer sequences (e.g., pick → navigate → place)
    • Assumptions/dependencies: Stable layouts across shifts; predominately large objects; minimal dynamic crowds; basic contact stability without precision grasps
  • Fast QA and regression testing of humanoid policies in realistic digital twins
    • Sectors: software/robotics tools, QA/DevOps for autonomy
    • What: Build a repeatable test harness by scanning target environments and generating batteries of evaluation rollouts with lighting/camera perturbations for robustness checks.
    • Tools/workflows: 3DGS-based “SceneBench” of customer sites; scripted modes (walk/turn/pick/put/carry) with randomized lighting and calibration; automated pass/fail heuristics
    • Assumptions/dependencies: High-fidelity reconstructions; deterministic motion replay; defined success metrics; GPU render budget
  • Teleoperation data bootstrapping and augmentation
    • Sectors: robotics R&D, integrators
    • What: Pretrain VLK with synthetic VLK pairs to reduce teleop burden; add a small set of real demonstrations for task/domain-specific fine-tuning.
    • Tools/workflows: Synthetic pretraining → small real demo set → joint fine-tune → A/B validation in reconstructed scenes
    • Assumptions/dependencies: Teleop pipeline exists; data compatibility (kinematics conventions, camera viewpoints); compute budget for retraining
  • Education and academic research labs: turnkey sim-to-real coursework and benchmarks
    • Sectors: education, academia
    • What: Course modules for perception-to-kinematics learning using student-scanned rooms; reproducible assignments on navigation and object transport.
    • Tools/workflows: Student phone scans → 3DGS reconstructions → provided motion synthesis scripts → Isaac Sim → VLK training → demo on lab humanoid or sim-only
    • Assumptions/dependencies: Access to consumer scanners/phones; GPUs for training and rendering; safety review for on-robot execution
  • Pre-deployment safety and feasibility assessments
    • Sectors: policy/governance, robotics deployment
    • What: Validate line-of-sight, camera placement, walkable regions, and safe approach corridors via reconstructed scenes before on-site trials.
    • Tools/workflows: Annotate walkable regions and obstacle volumes; run scripted suites with contact-aware tracking; generate risk reports (e.g., near-collision counts)
    • Assumptions/dependencies: Permission to scan spaces; updated scans for layout changes; defined safety KPIs and acceptance thresholds
  • “Scan-to-skill” services for small businesses and hobbyists
    • Sectors: consumer robotics, maker communities
    • What: Offer a packaged pipeline to scan homes/garages and produce customized navigation + transport behaviors for simple chores (moving bins).
    • Tools/workflows: Mobile scan → cloud synthetic data generation → model fine-tune → on-device deployment with chunked real-time inference
    • Assumptions/dependencies: Hobbyist-grade hardware; safety training; limited to large, easy-to-contact objects

Long-Term Applications

The following are high-impact opportunities that require further research, scaling, or extensions (e.g., small-object manipulation, dynamic scenes, multi-object tasks, new embodiments).

  • Hospital logistics and in-ward assistance
    • Sectors: healthcare, robotics
    • What: Scan hospital units to train humanoids for delivery of meds, linen, and equipment; ultimately assist with small-object handling and precise placement.
    • Tools/workflows: HIPAA-compliant scanning → synthetic VLK generation for carts and totes → VLK+tracker with contact/grasp extensions → safety/sterility protocols
    • Assumptions/dependencies: Precision grasping and small-object perception not yet supported; strict privacy and infection control; robust obstacle avoidance and human-aware navigation
  • Retail restocking and backroom-to-shelf transfer
    • Sectors: retail, logistics
    • What: In-store digital twins for restocking routes and shelf placements; eventual fine-grained SKU manipulation and height-varying surfaces.
    • Tools/workflows: Nightly store scans → batch synthetic data updates → continuous VLK fine-tuning → integration with store inventory systems
    • Assumptions/dependencies: Generalization beyond box-like items; dynamic shoppers; compliance with store safety policies
  • Disaster response and building-scale search operations
    • Sectors: public safety, defense
    • What: Train in scanned buildings for path planning and heavy-object movement; simulate lighting loss and clutter; deploy for search and clear.
    • Tools/workflows: Pre-incident scans or rapid post-incident scans → extreme domain randomization → ruggedized controllers and state estimation
    • Assumptions/dependencies: Harsh conditions (dust, debris), unreliable GPS/lighting; robustness to non-rigid terrain; fall recovery; reliable comms
  • Elder care and home assistance
    • Sectors: healthcare, consumer robotics
    • What: Personalized home digital twins to teach robots to tidy, fetch, and set up items; voice instructions to chain long-horizon tasks.
    • Tools/workflows: Privacy-preserving home scans → user-in-the-loop task scripting → VLK with small-object dexterity and safe human-robot interaction
    • Assumptions/dependencies: Precision grasping and fine manipulation; safety and trust; evolving layouts; regulatory approvals for in-home care devices
  • Factory assembly and kitting
    • Sectors: manufacturing, robotics
    • What: Use CAD-derived or scanned cells to synthesize scene-grounded trajectories for pick/place at benches, palletizing, and ergonomic assists.
    • Tools/workflows: CAD/scan fusion → synthetic data for multi-height surfaces → VLK extended with grasp primitives and force control → MES integration
    • Assumptions/dependencies: Tight tolerances; force/impedance control for contact-rich tasks; certification and safety cages or advanced HRI safeguards
  • Cross-embodiment transfer and model marketplaces
    • Sectors: software, robotics ecosystem
    • What: Offer “VLK-as-a-service” for different humanoids via retargeting and tracker adapters; exchange scene reconstructions and synthetic datasets in a marketplace.
    • Tools/workflows: Retargeting pipelines (e.g., OmniRetarget) → embodiment-specific trackers → dataset licensing and validation suites
    • Assumptions/dependencies: IP/licensing for scans; standardization of kinematic schemas; QA across embodiments to avoid regressions
  • Standardized evaluation suites for humanoid loco-manipulation
    • Sectors: academia, standards bodies
    • What: Curate open 3DGS scene libraries and canonical task modes for benchmarking perception-conditioned kinematics and sim-to-real transfer.
    • Tools/workflows: Public scene zoo; task templates; unified metrics (success, foot sliding, contact stability) and leaderboards
    • Assumptions/dependencies: Community adoption; reproducible pipelines; clear metric definitions
  • Planning+VLK integration for long-horizon autonomy
    • Sectors: robotics software
    • What: Combine language planners/LLMs with VLK chunks to compose multi-step routines (e.g., “pick box A, deliver to room B, return to dock”).
    • Tools/workflows: High-level planner → VLK atomic skill library → chunk overlap smoothing → failure recovery policies
    • Assumptions/dependencies: Reliable state tracking and memory across steps; robust re-localization; fallback strategies for failure cases
  • Remote operations in energy facilities (plants, substations)
    • Sectors: energy, industrial robotics
    • What: Train in digital twins (scan or BIM) for navigation and material handling in constrained industrial spaces; eventually inspection and tool use.
    • Tools/workflows: Secure scanning/BIM ingestion → synthetic trajectory generation for corridors and staging areas → extended perception for gauges/tools
    • Assumptions/dependencies: Hazardous environments; strict safety and lockout/tagout compliance; tool-use capability not yet supported
  • Risk, compliance, and ROI modeling using digital twins
    • Sectors: finance/operations, insurance, policy
    • What: Simulate task success/failure rates, time-to-task, and near-miss statistics in reconstructed customer sites to support investment and safety decisions.
    • Tools/workflows: Batch evaluation in 3DGS twins with perturbations → reporting dashboards → scenario planning (layout, lighting, traffic)
    • Assumptions/dependencies: Valid correlation between sim metrics and real KPIs; access to updated site scans; acceptance of synthetic-eval in governance

Common Assumptions and Dependencies Across Applications

  • Scene assets: Metric-scale scans (e.g., Polycam + LiDAR) and at least semi-automatic annotations for walkable regions and 3D object boxes; periodic rescans for layout changes.
  • Task scope: Current methods excel at navigation and transporting large, box-like objects; small-object grasping and fine placement are limited.
  • Hardware: Unitree G1 or similar morphology; accurate camera mounting/calibration; contact-aware whole-body tracker; safety interlocks.
  • Compute: GPU resources for data generation (3DGS, rendering) and training; real-time inference (e.g., Triton on RTX-class GPUs).
  • Software stack: Isaac Sim for rendering, MuJoCo for tracking tests, motion synthesis models (navigation/interaction), retargeting pipelines, domain randomization, chunked inference.
  • Robustness: Domain randomization for lighting/camera helps sim-to-real transfer; dynamic obstacles, long-horizon error accumulation, and non-rigid environments need further handling.
  • Legal/ethical: Privacy and permissions for scanning; data governance for digital twins; compliance with workplace and healthcare safety regulations.
View Paper Prompt  View All Prompts 

Glossary

  • 3D Gaussian Splatting (3DGS): A neural scene representation and rendering technique that uses 3D Gaussian primitives to reconstruct and render photorealistic, metric-scale environments efficiently. "3D Gaussian Splatting (3DGS) \cite{3dgs} can efficiently reconstruct indoor environments with photorealistic detail and metric scale"
  • 6D joint rotations: A continuous rotation representation (often using two 3D vectors) used to avoid discontinuities and gimbal lock when encoding joint orientations. "6D joint rotations~\cite{zhou2019continuity}"
  • accumulated root-trajectory consistency: An auxiliary training loss that encourages the predicted base (root) motion to remain consistent over time, reducing drift. "auxiliary losses for foot-floor contact prediction, accumulated root-trajectory consistency, forward-kinematics accuracy, and foot-sliding regularization"
  • contact-aware whole-body tracker: A low-level controller that tracks full-body reference motions while explicitly handling contact events (e.g., wrist-object contacts) to stabilize interactions. "a contact-aware whole-body tracker~\cite{chen2026scenebot} uses the predicted kinematics and contact labels to produce robot actions on the physical humanoid."
  • conditional diffusion: A generative modeling approach where diffusion models are conditioned on inputs (e.g., waypoints, object states) to synthesize motions that satisfy specified constraints. "We build on the conditional diffusion formulation of prior human-object interaction synthesis methods~\cite{li2024chois,wu2025hihi}"
  • digital twins: High-fidelity virtual replicas of real environments or systems used for simulation, data generation, and testing. "Recent work uses digital twins and reconstructed scene representations, such as 3D Gaussian Splatting, to build realistic synthetic environments"
  • domain randomization: A sim-to-real technique that randomizes rendering parameters (e.g., lighting, camera intrinsics/extrinsics) during training to improve robustness to real-world visual variation. "we apply domain randomization during rendering, including lighting variation and perturbations to camera extrinsics and focal length."
  • egocentric observations: First-person visual inputs captured from the robot’s own viewpoint, used to ground perception and control. "Perception-based humanoid loco-manipulation requires connecting egocentric observations and task instructions to whole-body motion."
  • flow-matching formulation: A training paradigm related to continuous-time generative modeling that learns to map noisy samples to data via vector fields, here used to predict clean trajectories. "We train the policy with an x0x_0-prediction objective in a flow-matching formulation."
  • foot-floor contact prediction: Predicting discrete events when the feet are in contact with the ground, used to improve motion realism and stability. "auxiliary losses for foot-floor contact prediction, accumulated root-trajectory consistency, forward-kinematics accuracy, and foot-sliding regularization"
  • foot-sliding regularization: A regularizer that penalizes unrealistic sliding of feet during stance phases to ensure physically plausible motion. "auxiliary losses for foot-floor contact prediction, accumulated root-trajectory consistency, forward-kinematics accuracy, and foot-sliding regularization"
  • forward-kinematics accuracy: An auxiliary loss ensuring that predicted joint angles produce accurate end-effector and link positions when passed through forward kinematics. "auxiliary losses for foot-floor contact prediction, accumulated root-trajectory consistency, forward-kinematics accuracy, and foot-sliding regularization"
  • Gaussian noise: Random noise sampled from a normal distribution, injected into trajectories during training for denoising or diffusion-style objectives. "Given a ground-truth future trajectory xt+1:t+H\mathbf{x}_{t+1:t+H}, Gaussian noise ϵ\boldsymbol{\epsilon}, and interpolation coefficient α\alpha"
  • heading-normalized: A reference frame convention where positions and orientations are expressed relative to the robot’s current heading to reduce variability and simplify learning. "heading-normalized root planar displacement"
  • hindsight rendering: Rendering visual observations after synthesizing motion by replaying trajectories in a reconstructed scene, enabling flexible pairing of visuals and kinematics. "By exploiting targeted annotation, oracle-conditioned motion synthesis, and hindsight rendering, we can scale up data production"
  • interpolation coefficient: A scalar that blends ground-truth data with noise to create partially corrupted inputs for denoising objectives during training. "Given a ground-truth future trajectory xt+1:t+H\mathbf{x}_{t+1:t+H}, Gaussian noise ϵ\boldsymbol{\epsilon}, and interpolation coefficient α\alpha"
  • Isaac Sim: NVIDIA’s robotics simulation and rendering platform used here to render egocentric observations from reconstructed scenes. "We render egocentric observations by replaying the synthesized G1 trajectories in Isaac Sim."
  • kinematic trajectories: Time sequences of positions, orientations, and joint angles that describe motion without explicitly modeling forces or dynamics. "train a VLK policy that predicts short-horizon whole-body kinematic trajectories."
  • loco-manipulation: The integrated problem of locomotion (moving the robot) and manipulation (interacting with objects) in a coordinated whole-body manner. "Perception-based humanoid loco-manipulation requires connecting egocentric observations and task instructions to whole-body motion."
  • metric-scale: A property of scene reconstructions that preserve real-world units (meters), critical for accurate planning and control. "3D Gaussian Splatting to reconstruct metric-scale indoor environments"
  • MuJoCo: A physics engine for simulating articulated bodies and contact-rich interactions, used here for closed-loop control evaluation. "We test the full closed-loop system in MuJoCo simulation (with IsaacSim rendering)"
  • object-directed navigation: Navigation conditioned on target objects, requiring the robot to move toward specific, identified items in the scene. "two task families: object-directed navigation and object interaction."
  • object geometry features: Encoded descriptors of an object’s shape/geometry used to condition motion synthesis models for interaction. "object geometry features~\cite{prokudin2019efficient}"
  • oracle-conditioned motion synthesis: Generating motions conditioned on privileged information (e.g., exact object poses), available only in simulation, to simplify behavior generation. "By exploiting targeted annotation, oracle-conditioned motion synthesis, and hindsight rendering"
  • PD targets: Proportional-derivative controller setpoints for joint angles/velocities used by low-level control to track reference motions. "outputs joint-level PD targets for all actuated joints:"
  • privileged scene information: Simulator-only knowledge such as exact object poses, collision geometry, or walkable regions used to aid synthetic trajectory generation. "synthesizes navigation and object-interaction trajectories using privileged scene information"
  • proprioception: Internal sensing of the robot’s own state (e.g., joint angles, velocities), used by controllers for feedback. "it only receives converted reference targets and current robot proprioception"
  • Real2Sim: The process of bringing real-world assets (scenes, objects) into simulation while preserving realism and scale. "On the Real2Sim side, creating environments with sufficient realism, diversity, and geometric detail at scale has traditionally been prohibitive."
  • real-to-sim-to-real: A pipeline that starts from real data, builds or uses simulation for training, and then transfers back to real-world deployment. "can we connect these partial solutions into a real-to-sim-to-real pipeline for both scene assets and humanoid behaviors?"
  • retargeting: Mapping human motion data to a robot’s kinematic structure to produce feasible robot motions. "OMOMO~\cite{li2023omomo} sequences retargeted to G1 with OmniRetarget~\cite{yang2025omniretarget}"
  • scene-grounded waypoints: Waypoints tied to the scene’s geometry and semantics that guide motion synthesis and ensure feasibility. "sparse scene-grounded waypoints"
  • semantic 3D bounding boxes: 3D boxes with class labels that localize objects in the scene for planning and instruction templating. "we annotate semantic 3D bounding boxes for objects"
  • Sim2Real: The challenge and methodology of transferring policies trained in simulation to real-world hardware and conditions. "On the Sim2Real side, policies trained entirely on synthetic renderings must transfer back to physical hardware"
  • vision-language-action (VLA): Policies that take visual inputs and language instructions to produce robot actions end-to-end. "Vision-language-action (VLA) policies map visual observations and language instructions to robot actions"
  • vision-language-kinematics (VLK): A policy or dataset paradigm that maps visual inputs and language instructions to future kinematic trajectories instead of low-level actions. "train a VLK policy that predicts short-horizon whole-body kinematic trajectories."
  • whole-body motion tracking: Control methods that track full-body kinematic reference motions on humanoid hardware, often learned in simulation. "Whole-body motion tracking trained in simulation \cite{liao2025beyondmimic} can produce real-world robot policies that faithfully execute kinematic reference trajectories"
  • wrist-object contact labels: Binary indicators specifying whether each wrist should be in contact with the manipulated object, used to condition control and stabilize transport. "binary wrist-object contact labels"
  • x0-prediction objective: In diffusion/denoising-style training, an objective that predicts the clean target (x0) from a noisy input. "We train the policy with an x0x_0-prediction objective in a flow-matching formulation."
  • ZED 2i: A specific stereo RGB camera model used to capture egocentric views with known intrinsics/extrinsics for alignment with simulation. "a virtual ZED~2i camera mounted on the head"
View Paper Prompt  View All Prompts 

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 8 tweets with 126 likes about this paper.

Sign Up for Free