2000 character limit reached
OASIS: From Simulation Data Collection to Real-World Humanoid Loco-Manipulation
Published 7 Jun 2026 in cs.RO | (2606.08548v1)
Abstract: Recent progress in robot manipulation has been largely driven by learning from large-scale demonstrations. For humanoid robot loco-manipulation tasks, however, existing data sources force an unsatisfying tradeoff between trajectory quality and scalability. Real-world teleoperation provides the highest-quality trajectories but requires dedicated physical space and time-consuming scene resets. Simulation offers an alternative way out of this dilemma: it can produce clean, embodiment-aligned data at scale without any physical hardware. In this paper, we propose OASIS, a simulation-data-driven framework for humanoid loco-manipulation. OASIS automatically reconstructs realistic object assets from real-world images using a 3D generative model. Based on these assets, trajectories are first collected through teleoperation in simulation, and then augmented under diverse domain randomizations in a post-processing stage. With the resulting simulation data, we further design a hierarchical visuomotor policy for humanoid loco-manipulation. Extensive experiments on the real humanoid robot show that, under zero-shot deployment, the policy trained on our simulation data achieves higher success rates on most tasks than that trained on real-robot teleoperation data, owing largely to the broad lighting and environmental variations covered by our simulation rendering, which real-robot data fails to capture. The project page is available at https://oasis-humanoid.github.io/.
Summary
- The paper presents a novel simulation-driven pipeline that automates asset generation and VR teleoperation to collect high-quality demonstration data.
- It leverages hierarchical policy learning with a Transformer-based Flow Matching planner, achieving robust, long-horizon execution on complex tasks.
- Real-robot evaluations show that simulation-trained policies, augmented with visual randomization, yield 1.15รโ1.84ร faster data collection and competitive performance compared to real-teleoperation methods.
OASIS: Scalable Simulation-Driven Data Collection and Zero-Shot Transfer for Humanoid Loco-Manipulation
Introduction
OASIS proposes a simulation-first data pipeline to address the scalability bottlenecks inherent in collecting large-scale, high-quality demonstration data for humanoid loco-manipulation. The framework automates the generation of physics-aligned simulation assets from single-view real-world images and uses VR teleoperation to efficiently acquire rich, whole-body demonstration trajectories. These are paired with large-scale visual randomizations during offline post-processing to generate datasets with substantial environment diversity. Policies are trained in this simulation regime and subsequently deployed zero-shot on real hardware. The framework demonstrates that simulation data, augmented with appropriate randomization, can rival or even surpass policies trained on traditional real-robot teleoperation data under equivalent demonstration budgets.
Figure 1: OASIS collects whole-body demonstrations entirely in simulation and deploys the visuomotor policy zero-shot on the real Unitree G1 humanoid across diverse loco-manipulation tasks.
System Overview
OASIS follows a four-stage process designed to maximize data quality and transfer-ability:
- Simulation Asset Generation: Single-view images of real-world objects are converted to textured, scale-accurate 3D meshes using Hunyuan3D, with physical parameters such as scale and material estimated via Qwen3-VL. All key attributes (mass, density, friction) are computed, and domain randomization is applied to mitigate sim-to-real domain gaps.
- VR-based Teleoperation: Demonstrations are collected in simulation through VR interfaces (e.g., PICO 4U), where operators control the humanoid from a first-person perspective. Motions are retargeted using GMR and executed through Teleopit, an RL-based, whole-body controller, supporting real-time, high-yield collection.
- Offline Visual Augmentation: The recorded trajectory state sequences are high-fidelity rendered offline under significant texture, lighting, and camera-extrinsic randomization. Each base demonstration is thus expanded into ~20 visually diverse trajectories, multiplying data efficiency and robustness.
- Hierarchical Policy Learning: Training leverages a hierarchical design where a high-level, Transformer-based Flow Matching planner predicts reference motions from multimodal input, and the low-level controller executes these as joint angles in feedback. Rigorous curriculum-based rollout training stabilizes long-horizon prediction.
Figure 2: Overview of OASIS: asset generation, VR teleoperation, offline visual randomization, and hierarchical policy learning pipeline.
Real-Robot Evaluation
Deployment targets the Unitree G1 platform (29-DoF, three-fingered dexterous hands, multi-camera setup) with all policies executing on an NVIDIA RTX 4090 in real time. The main empirical findings include:
- Data Collection Efficiency: OASIS achieves 1.15รโ1.84ร speedup in demonstration throughput relative to real-robot teleoperation, with the margin growing with task complexity. This is attributed to fast, automatic scene resets and removal of manual intervention.
- Zero-Shot Transfer: Policies trained exclusively with OASIS simulation data consistently achieve success rates on par with or exceeding those trained on exclusively real-robot data when evaluated zero-shot on a spectrum of loco-manipulation tasks (tabletop, lifting, kneeling/wiping).
- Ablation Studies: Removing domain randomization, or ablating any single factor (texture, lighting, or camera), sharply degrades sim-to-real transfer. Lighting has the highest individual impact, but the combination of all factors is strictly optimal. Increasing the number of rendered variants per trajectory yields higher transfer success, saturating after ~20 variants.
Figure 3: Real-robot experiments on loco-manipulation tasks with increasing difficulty.
Data Augmentation and Robustness
Visual domain randomization is critical in bridging the sim-to-real gap. OASIS parameters span texture selection, roughness/metallic constants, lighting spectral properties and intensities, and fine-grained camera pose perturbations. Without such augmentation, zero-shot transfer fails reliably. Empirically, simulation-trained policies are more robust to deployment environment variations (background clutter, lighting changes, sensor noise) than real-data-trained counterparts, which are susceptible to overfitting scene specifics.
Methodological Contributions
- Automated, Scalable Asset Generation: Pairing state-of-the-art 3D generation with VLM-driven physical property estimation provides fast, realistic, and physically plausible simulation environments at scale, essential for humanoid systems with complex contact and balance constraints.
- Temporal Decoupling of Demonstration and Rendering: By disconnecting operator interaction from visual sample yield, OASIS achieves order-of-magnitude gains in dataset size and diversity without requiring proportionally more operator labor.
- Hierarchical Policy with Flow Matching: The policy stacks a Transformer-based flow-matching planner with a robust low-level controller, allowing for effective multimodal cue integration, future prediction, and smooth trajectory realization.
- Stable Long-Horizon Rollout: Curriculum-based rollout exposure during training is shown to be crucial for avoiding error accumulation in deployment; omitting it causes catastrophic failures on long-horizon tasks.
Implications and Future Directions
OASIS empirically validates that well-randomized, physically-grounded simulation is a viable substitute for extensive real-robot teleoperation for high-DoF humanoid loco-manipulation, significantly lowering the hardware and human costs for dataset acquisition. This has direct implications for scalable generalist robot training, enabling wider participation and broader task coverage as robot platforms proliferate.
Important open directions include:
- Physics-aware Trajectory Augmentation: While OASIS randomizes visuals, trajectory-level physics augmentation remains an open challenge due to the sensitivity of balance and contact in humanoids.
- Enhanced Asset Fidelity and Calibration: Transfer performance on contact-critical tasks is currently bounded by the realism of generated assets and estimated parameters. Incorporating tactile or multi-modal calibration will be essential for tasks requiring high-accuracy force control.
- Cross-Embodiment Generalization: As simulation pipelines become cheaper, studying transfer across humanoid morphologies via shared virtual experience will become viable, particularly leveraging actor-critic frameworks and large-scale reinforcement learning.
Conclusion
OASIS demonstrates that scalable, realistic, visually diverse simulation data collection can supplant or augment traditional real-robot teleoperation for humanoid loco-manipulation policy learning. The framework enables significant efficiency gains and robust zero-shot deployment, supporting a move toward simulation-first workflows in generalist humanoid research. Further advances in simulation fidelity, physics-aware augmentation, and scene diversity are likely to compound these benefits and accelerate progress in universal robotic control.
Reference: "OASIS: From Simulation Data Collection to Real-World Humanoid Loco-Manipulation" (2606.08548)
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Explain it Like I'm 14
What is this paper about?
This paper introduces OASIS, a new way to teach humanoid robots to walk around and use their hands (called โloco-manipulationโ) by collecting training data entirely in a computer simulation, then using it on a real robot right away. Think of it like practicing in a super-realistic video game and then performing the same actions in the real world on your first try.
What questions did the researchers ask?
They focused on three simple questions:
- Can collecting training data in simulation be faster than doing it with real robots?
- Which parts of their simulation โmakeoverโ (like changing lighting or camera angles) matter most for teaching a robot skills that work in real life?
- If you train only with simulated data, can the robot do as well as (or better than) training with real-world teleoperation data?
How did they do it?
The team built a four-step pipeline. Hereโs what happens, using everyday language:
- Build realistic 3D objects from photos
- They take pictures of real items (like a cup or a box) and use an AI 3D โcreatorโ to make detailed 3D models.
- Another AI guesses each objectโs size and material (for example, plastic or wood) so the physics in the simulator feels realโthings are the right weight and slide or bounce correctly.
- Collect human demonstrations in VRโbut in simulation
- A person wears a VR headset and controls a simulated humanoid robot from a first-person view, like playing an immersive game.
- The system records the robotโs and objectsโ positions and motions (not heavy video files yet) so it can replay everything perfectly later.
- Re-render those demonstrations many times with variety
- They replay each demonstration offline (not in real time) and create training images under many different conditionsโnew lighting, textures, and slightly shifted cameras.
- This โdomain randomizationโ is like practicing the same move in different rooms and lighting so the robot wonโt be confused by changes in the real world.
- Train a two-level robot controller
- High-level โplannerโ (the coach): looks at the camera views and the recent motion history, then decides what should happen next (like โstep forward, reach out, close handโ). This planner uses a technique called โFlow Matching,โ which you can think of as guiding a blurred idea into a clear, precise action sequence.
- Low-level โcontrollerโ (the athleteโs muscles): turns those high-level plans into exact joint movements for the robotโs body and fingers.
Once trained, they put this policy directly onto a real Unitree G1 humanoid robot. โZero-shotโ means they donโt fine-tune on the real robot firstโthey just try it, and it works.
What did they find, and why does it matter?
Here are the main results, explained simply:
- Faster data collection: Simulation was faster than real-robot teleoperation on every task, especially the tricky ones. Why? In the real world, if something falls or breaks, you must reset everything by hand. In simulation, you just click โrestart.โ
- Variety makes transfer happen: Changing visuals (lighting, textures, camera angles) during re-rendering was crucial. Without this, skills learned in simulation barely worked in real life. Lighting mattered the most, but the best results came from combining all types of randomization.
- How many re-renders per demo? Doing about 15โ20 different visual versions of each demonstration gave strong performance; beyond that, extra gains were small.
- Simulation data vs. real data: Training only on OASIS simulation data often matched or even beat training on real-robot data with the same number of demos. Mixing both was best of all, showing they complement each other: simulation adds rich variety, and real data adds exact real-world details.
- Real-world success, first try: The policy trained in simulation handled tasks like placing a cup in a box, wiping a monitor, lifting a basket and placing a cup, and kneeling to wipe under a tableโon the real robot, zero-shot.
Why is this important?
- Lower cost, safer, and faster: You donโt need tons of expensive robots or big labs to collect data. Itโs safer tooโno broken equipment if a move goes wrong in simulation.
- Better generalization: Practicing under many visual styles in sim helps the robot stay calm when real lighting or background changes.
- Practical training path: This suggests we can scale up robot learning massively by doing most of the practice in high-quality simulation, then deploy directly in the real world. When you can also sprinkle in some real data, you get the best of both worlds.
Limitations and whatโs next
- Motion variety: They mainly changed how things look, not the robotโs movements, because changing full-body motions can make the robot lose balance. Next step: safe, physics-aware ways to vary the motions too.
- Object accuracy: Automatically generated 3D objects may have small errors in shape or physical properties, which can matter for contact-heavy tasks. Better 3D reconstruction and physics calibration would help.
Overall, OASIS shows a practical path from โlearn in simโ to โwork in real life,โ making it faster and cheaper to teach humanoid robots useful, everyday skills.
Knowledge Gaps
Below is a concise list of concrete knowledge gaps, limitations, and open questions left unresolved by the paper. Each item is phrased to be directly actionable for future research.
- Quantify how errors in VLM-estimated physical dimensions and material labels propagate to sim dynamics and real-world performance (e.g., sensitivity analysis linking percent dimension error to success rates).
- Develop and evaluate automated calibration procedures that infer object mass, inertia, friction, and restitution from brief real interactions (system identification) to refine sim assets post-generation.
- Replace or complement single-view 3D asset synthesis with multi-view capture or depth scans; measure the gain in sim-to-real transfer for contact-rich tasks.
- Expand the material/density library beyond a handful of categories and support composite or hollow structures; verify with ground-truth mass/inertia measurements on diverse household objects.
- Introduce and evaluate physics-domain randomization at training time (e.g., contact damping, joint friction, actuator delays, compliance) and ablate which terms most improve transfer.
- Add sensor/renderer domain randomization beyond textures/lighting/camera extrinsics, including rolling shutter, auto-exposure, motion blur models, lens distortion, color calibration, noise for RGB and depth, and depth-to-RGB misalignment.
- Report and study the computational/energy cost, throughput, and scaling behavior of offline path-traced rendering; compare fidelityโcost trade-offs (path tracing vs rasterization + learned post-processing).
- Investigate physics-aware trajectory augmentation that preserves balance (e.g., short-horizon perturb-and-stabilize, contact-timing jitter, retargeted variants); quantify benefits vs. risk of introducing instability.
- Analyze the limits of using reference command history as proprioception: quantify drift under long rollouts and compare against using filtered real state or learned state estimators.
- Provide ablations on the high-level planner: Flow Matching vs diffusion vs supervised BC vs autoregressive transformers, including training stability, sample efficiency, and runtime latency.
- Evaluate the contribution of frozen encoders (CLIP/DINOv2) vs fine-tuned encoders on the augmented sim data; measure robustness to real-world distribution shifts.
- Clarify and study the role of language: how are instructions authored, what is the instruction vocabulary, and does the policy generalize to unseen paraphrases or new task-language compositions?
- Test generalization to unseen object instances and categories whose assets are generated only from a single reference image; assess failure modes when geometry/material priors are wrong.
- Evaluate robustness to larger camera pose changes than the small extrinsic jitters randomized at training, and to camera displacement/failure (e.g., head-only, wrist-only, or missing one view).
- Compare policies trained with head-only vs head+wrist cameras; quantify the marginal value of each viewpoint and the minimal camera set that maintains performance.
- Measure sim-to-real transfer for dynamics-critical tasks (e.g., heavy lifting, tool use with precise force control, peg-in-hole) to identify where current sim physics or asset fidelity breaks down.
- Examine external disturbance robustness (pushes, slipping surfaces, moving props) and recovery; determine whether additional dynamics randomization or disturbance training is needed.
- Benchmark across more diverse loco-manipulation scenarios (stairs, door opening, long-horizon multi-room tasks) to probe scalability beyond the four demonstrated tasks.
- Provide learning curves vs number of demonstrations and vs number of renderings per demo to derive data-scaling laws; verify that โ~20 renders/demoโ remains near-optimal across tasks and environments.
- Study inter-operator variability: collect from multiple operators and analyze how operator skill/style affects policy quality and required augmentation.
- Compare OASIS against other sim-data pipelines (e.g., RoboCasa, MimicGen, RL-generated corpora) under matched trajectory budgets and rendering randomizations.
- Quantify the individual impact of physics-parameter randomization (mentioned but not ablated) on transfer; specify ranges and conduct targeted ablations.
- Investigate replacing Teleopit with alternative low-level controllers (model-based WBC, torque-level MPC) and measure compatibility and transfer differences.
- Report the latency and closed-loop stability when executing 32-frame action chunks at 50 Hz; ablate chunk length and denoising steps for the speedโstability trade-off.
- Add fine-grained failure analysis on the real robot (balance losses, grasp slippage, perception mislocalization) to prioritize which components need improvement.
- Assess whether combining a small amount of real-world fine-tuning with large OASIS sim data yields better costโperformance than either alone; quantify the minimal real data needed.
- Explore automatic real-world reset/scene reconfiguration to further reduce real data collection overheads for mixed sim+real training pipelines.
- Validate cross-embodiment portability: train in sim for one humanoid morphology and deploy (zero-shot or with lightweight adaptation) to another platform.
- Provide reproducibility assets (object images, generated meshes, physics parameters, scripts) and define standardized benchmarks to enable fair comparisons and progress tracking.
Practical Applications
Immediate Applications
Below are specific, deployable applications that can be built with the paperโs methods as-is or with minimal engineering effort, grouped by sector and including assumptions and dependencies.
- Sim-first data factory for humanoid loco-manipulation (Robotics, Software)
- What: Stand up a scalable pipeline to collect demonstrations entirely in simulation, using VR teleoperation for trajectory capture and offline photorealistic re-rendering with domain randomization.
- Tools/products/workflows: โOASIS Data Factoryโ for Isaac Sim; VR teleop rig (e.g., PICO 4U) + GMR retargeting + Teleopit low-level controller; rendering harness with texture/lighting/camera randomization; dataset export for policy training.
- Assumptions/dependencies: Access to Isaac Sim (or equivalent), a recent GPU (e.g., RTX 4090) for training/inference, Hunyuan3D or similar 3D model, Qwen3-VL or equivalent VLM; licensing for models/assets; availability of a compatible humanoid (e.g., Unitree G1) or a comparable platform and controller.
- Zero-shot policy development and validation for service tasks (Robotics: facilities, cleaning, light logistics)
- What: Train hierarchical visuomotor policies (Flow Matching high-level + Teleopit low-level) in sim and deploy to real humanoids for tasks like wiping screens, placing items in containers, kneeling and cleaning under tables.
- Tools/products/workflows: โSim-to-Fieldโ workflow with automatic sim scene reset and batch augmentation; regression tests across lighting and camera configurations.
- Assumptions/dependencies: Camera placements/sensors on real robot roughly match simulated extrinsics; domain randomization covers deployment conditions; safe testing environments.
- Physics-ready 3D asset generation for robot training and testing (Software, Robotics, Media/Content)
- What: Convert single-view photos of household/industrial objects into textured, dimensioned, and material-parameterized assets for physics simulation.
- Tools/products/workflows: โReal-to-Sim Asset Builderโ (Hunyuan3D + VLM estimator + density/friction lookup + randomized physical parameters).
- Assumptions/dependencies: VLM estimates are close enough for contact-rich tasks; assets pass basic collision/mesh integrity checks; IP clearance for imagery.
- Automated visual robustness testing for robot perception (Software QA, Robotics)
- What: Use offline high-fidelity path-traced re-rendering with lighting/texture/camera randomization to stress-test perception and control pipelines.
- Tools/products/workflows: Continuous integration (CI) suite that replays trajectories under K visual seeds; โLighting Robustness Benchโ and โCamera Misalignment Check.โ
- Assumptions/dependencies: Reproducible replay of recorded states; coverage of relevant environmental variations; compute budget for path-traced renders.
- Rapid prototyping of task curricula and scenes (Academia, Robotics R&D)
- What: Assemble diverse scenes from generated assets and quickly iterate on task variants without physical setup, enabling faster experiment cycles and student projects.
- Tools/products/workflows: โCurriculum Composerโ for Isaac Sim; prebuilt scene packs (kitchen, office, lab bench); scripted resets and scene parameter sweeps.
- Assumptions/dependencies: Basic simulation expertise; asset libraries with metadata; access to VR teleop or scripted teleop.
- Synthetic data augmentation for robot perception models (Software, Academia)
- What: Use replayed, photorealistic, domain-randomized images to augment datasets for pose estimation, grasp detection, and scene understanding.
- Tools/products/workflows: Render-to-dataset pipelines for DINO/CLIP fine-tuning or contrastive pretraining; tagged camera parameters for calibration-aware training.
- Assumptions/dependencies: Sim images sufficiently representative of deployment; dataset labeling derived from ground-truth sim states.
- Operator training and process rehearsal in VR (Education, Industrial Training)
- What: Train human operators in VR with first-person views from simulated robots to practice teleoperation flows, task sequencing, and safety behaviors.
- Tools/products/workflows: VR modules mirroring robot views and dynamics; โReset-and-Replayโ for immediate failure recovery practice.
- Assumptions/dependencies: Motion retargeting latency kept low; curricula capture realistic force/motion cues (visual + optional haptics).
- Cost and safety sandboxing for pre-deployment (Policy, Safety Engineering)
- What: Use simulation-derived data and randomized test suites for hazard analysis and acceptance tests before field pilots.
- Tools/products/workflows: Test reports summarizing success, near-miss, and failure modes under visual/environmental perturbations; documentation for internal safety reviews.
- Assumptions/dependencies: Organizational acceptance of sim-based evidence; mapping between sim metrics and field KPIs.
- Data-sharing and reproducibility in academic benchmarks (Academia)
- What: Release simulation seeds, assets, and replay logs to enable exact replication of loco-manipulation experiments across labs.
- Tools/products/workflows: Dataset card templates; seed and randomization config bundles; standardized success metrics.
- Assumptions/dependencies: Community consensus on task specs; hosting/storage for large renders and assets.
- Pre-deployment evaluation for consumer humanoids (Consumer Robotics)
- What: Validate home tasks (e.g., fetching, placing, simple tidying) in sim using household object photos and zero-shot test on affordable humanoids.
- Tools/products/workflows: โHome Task Packโ asset sets; auto-generation from user-provided object images; on-device or edge inference.
- Assumptions/dependencies: Consumer robots with compatible degrees of freedom and sensing; careful safety constraints in home trials.
Long-Term Applications
These opportunities require additional research, scaling, or ecosystem development, but are natural extensions of the paperโs innovations.
- Personalized home-robot training from household photos (Consumer Robotics, Daily Life)
- What: Build a โcustom household twinโ by auto-reconstructing home objects and scenes from user images; train policies tailored to a specific home and deploy updates over-the-air.
- Tools/products/workflows: Home Twin Builder; background scanning; continual learning pipeline that retrains with new assets and domain randomization.
- Assumptions/dependencies: More accurate physical parameters for diverse objects; robust on-device safety and fail-safes; privacy-preserving data handling.
- Hospital and eldercare service robots trained in patient-specific sims (Healthcare)
- What: Construct ward/room twins and train robots for safe object delivery, light cleaning, and fetchingโstress-tested under lighting and clutter variations prior to clinical use.
- Tools/products/workflows: Healthcare Scene Packs (beds, medical carts, monitors); policy checkpoints with scenario-based validation; โno-contactโ safety layers.
- Assumptions/dependencies: Regulatory approval processes; high-fidelity contact and material modeling; infection control and privacy requirements.
- Retail and warehouse โSKU-to-skillโ training (Retail, Logistics)
- What: Auto-ingest product photos to generate physics-ready assets and continuously train/retrain humanoids for shelf stocking, picking, and restocking as SKU catalogs change.
- Tools/products/workflows: SKU Asset Ingestor; perpetual retraining with domain randomization; deployment monitoring and quick rollback.
- Assumptions/dependencies: Accurate dimensions/materials for product variance; integration with WMS/ERP; safety and throughput KPIs.
- Facility maintenance and inspection digital twins (Facilities, Smart Buildings)
- What: Create building-level simulation twins to train cleaning, inspection, and simple repair workflows; validate lighting-robust locomotion and manipulation.
- Tools/products/workflows: Building Twin Constructor using floor plans + photos; maintenance task libraries; scheduled retraining as building conditions change.
- Assumptions/dependencies: Scalable scene reconstruction; realistic physics for varied materials; cross-season lighting/weather models.
- Foundation models for humanoid loco-manipulation via synthetic corpora (Academia, Robotics, Software)
- What: Scale the OASIS pipeline into a โhumanoid data engineโ that pretrains generalist VLA or Flow-Matching action models on millions of diverse, randomized episodes.
- Tools/products/workflows: Data-engineering stack for simulation replay at scale; training clusters; evaluation suites for sim-to-real generalization.
- Assumptions/dependencies: Data curation to avoid overfitting to synthetic biases; cross-embodiment alignment techniques; compute budget.
- Certification frameworks incorporating sim evidence (Policy, Standards)
- What: Develop standards that formally accept structured simulation tests (with defined randomization ranges and seeds) as part of certification for service robots.
- Tools/products/workflows: Test protocols specifying lighting/camera perturbation sets; coverage metrics; pass/fail thresholds linked to deployment contexts.
- Assumptions/dependencies: Multi-stakeholder agreement on equivalence between sim and field tests; traceability and auditability of sim configurations.
- Physics-aware trajectory augmentation and failure-recovery training (Robotics R&D)
- What: Move beyond purely visual augmentation to perturb whole-body states in sim safely, training robust recovery policies for pushes, slips, and contact uncertainties.
- Tools/products/workflows: โTrajectory Perturberโ with stability constraints; curriculum rollout training with disturbance injections.
- Assumptions/dependencies: Advances in balance controllers and contact modeling; automated detection of unstable augmentations.
- Cross-embodiment transfer for non-humanoid platforms (Robotics)
- What: Apply the sim-first data pipeline to quadrupeds with arms, mobile manipulators, or exoskeletons through retargeting and policy adapters.
- Tools/products/workflows: Embodiment adapters; shared latent action spaces; replay of teleop traces onto multiple morphologies.
- Assumptions/dependencies: Reliable retargeting across kinematics; task abstractions that generalize; per-platform low-level controllers.
- Sim2Real-as-a-Service (Cloud Robotics)
- What: Offer cloud services that convert user-uploaded images/tasks into physics-ready sims, collect VR teleop demos, run offline augmentation, train policies, and deploy to fleets.
- Tools/products/workflows: Managed Isaac Sim clusters; asset services; MLOps for robot policies; fleet rollout and telemetry.
- Assumptions/dependencies: Cloud costs and latency; data sovereignty; secure deployment pipelines.
- Synthetic data marketplaces and shared task libraries (Software Ecosystem)
- What: Curated marketplaces for physics-ready assets, randomized renders, and replay logs for standard tasks (kitchen, office, retail), accelerating community progress.
- Tools/products/workflows: Metadata standards for materials/dimensions; quality scoring; licensing frameworks.
- Assumptions/dependencies: IP management; incentives for contributions; agreed benchmarks and evaluation methods.
Notes on Feasibility and Dependencies Across Applications
- Simulation fidelity: Success depends on accurate object geometry, material properties, and contact modeling; errors can degrade transfer, especially in contact-rich tasks.
- Domain randomization coverage: Visual and camera perturbations are crucial; insufficient coverage reduces robustness to deployment conditions.
- Low-level control: Requires a competent whole-body controller (e.g., Teleopit) and stable balance; porting to different robots may need tuning.
- Compute and tooling: Training Flow Matching policies and path-traced renders need substantial GPU resources; Isaac Sim or comparable platforms must be available.
- Safety and compliance: Real-world evaluation should enforce conservative safety interlocks; healthcare and public deployments will require formal approvals.
- IP and privacy: Using photos for asset generation must respect intellectual property and privacy policies; especially important for consumer and clinical settings.
Glossary
- 3D generative model: A model that synthesizes three-dimensional assets from data such as images or text. "OASIS automatically reconstructs realistic object assets from real-world images using a 3D generative model."
- Action-chunking policy: A control strategy that predicts or executes actions in fixed-length segments (chunks) rather than frame-by-frame. "our high-level planner is a Transformer-based, action-chunking policy that generates future motion sequences with Flow Matching"
- Autoregressive rollout: Executing a model by feeding its own previous predictions back as inputs over multiple steps. "this mechanism maintains stability under long-horizon autoregressive rollout at deployment."
- Camera extrinsics: Parameters describing a cameraโs position and orientation relative to a world or robot frame. "Textures, lighting, and camera extrinsics are randomized in the process,"
- CLIP: A vision-LLM that aligns text and images for representation learning. "encoded by a frozen CLIP"
- Curriculum-based rollout: A training strategy that gradually increases the difficulty by mixing model-predicted history into inputs over time. "We therefore adopt a curriculum-based rollout mechanism:"
- Denoiser: In generative modeling, a network that predicts velocity or noise to transform a random variable toward data. "The denoiser takes three inputs:"
- DINOv2: A self-supervised vision model used as a feature encoder for images. "encoded by a frozen DINOv2"
- DoF (degrees of freedom): Independent parameters that define a robotโs configuration, such as joint angles. "Together with the 14-DoF hand joints, the system outputs 43-DoF whole-body joint angles."
- Domain randomization: Varying visual and/or physical parameters during simulation to improve real-world transfer. "We apply domain randomization during offline rendering."
- Embodiment-aligned: Data or trajectories matched to a robotโs specific morphology and capabilities. "it can produce clean, embodiment-aligned data at scale without any physical hardware."
- Euler solver: A numerical integration method used to step differential equations in time. "At inference, we generate actions by integrating the learned velocity field with an Euler solver using $10$ denoising steps."
- Flow Matching: A generative modeling framework that learns a continuous velocity field to transform a prior into data. "We train the denoiser with the Flow Matching objective,"
- Gaussian prior: A normal distribution used as the starting point in generative sampling. "between a Gaussian prior and the target action chunk :"
- Hierarchical visuomotor policy: A control architecture with high-level visual decision-making and low-level motor control. "we further design a hierarchical visuomotor policy for humanoid loco-manipulation."
- Hunyuan3D: A large-scale system for generating high-resolution textured 3D assets. "we first leverage Hunyuan3D"
- IsaacSim: NVIDIAโs robotics simulation platform for high-fidelity rendering and physics. "Path-Tracing rendering mode in IsaacSim,"
- Kinematic state: The set of positions and velocities describing the configuration and motion of bodies without considering forces. "The first is the whole-body kinematic state of the robot,"
- Low-level controller: The component that translates high-level commands into actuator/joint-level targets. "the low-level controller converts these commands into target joint angles."
- Motion retargeting: Mapping human or source motions to a different embodiment while preserving intent. "The operator's motions are retargeted to the humanoid by GMR"
- Path-Tracing rendering mode: A photorealistic rendering technique that simulates light transport for high-fidelity images. "the offline setting enables Path-Tracing rendering mode in IsaacSim, which produces higher-fidelity images."
- Proprioception: A robotโs internal sensing of its own state (e.g., joint angles, velocities). "and robot proprioception over the most recent frames, encoded by an MLP."
- Real-Time rendering mode: A fast, lower-fidelity rendering mode optimized for low latency. "this stage employs the Real-Time rendering mode of IsaacSim,"
- Restitution coefficient: A parameter describing elasticity in collisions, affecting bounce. "the effective density, friction, and restitution coefficients,"
- Sim-to-real transfer: The process of deploying models trained in simulation to real-world systems. "How does each component of the OASIS data augmentation stage affect sim-to-real transfer?"
- Teleoperation: Controlling a robot remotely by a human operator. "Real-world teleoperation provides the highest-quality trajectories"
- Transformer-based: Utilizing Transformer architectures characterized by attention mechanisms for sequence modeling. "our high-level planner is a Transformer-based, action-chunking policy"
- Trajectory augmentation: Expanding a dataset by generating varied versions of existing trajectories. "Another scales data through trajectory augmentation,"
- Uniform-density assumption: Assuming constant density throughout an object to compute mass and inertia from volume. "under a uniform-density assumption,"
- Vision-LLM (VLM): A model that processes and relates visual inputs and text. "with a vision-LLM (VLM)."
- Vision-Language-Action (VLA) policies: Policies that integrate visual perception, language understanding, and action generation. "vision-language-action (VLA) policies"
- Visuomotor policy: A control policy that maps visual inputs to motor commands. "deploys the visuomotor policy zero-shot on the real Unitree G1 humanoid"
- Zero-shot deployment: Applying a trained model to new real-world tasks/environments without additional fine-tuning. "under zero-shot deployment, the policy trained on our simulation data achieves higher success rates"
Collections
Sign up for free to add this paper to one or more collections.
