Seeing Realism from Simulation: Efficient Video Transfer for Vision-Language-Action Data Augmentation

Abstract: Vision-language-action (VLA) models typically rely on large-scale real-world videos, whereas simulated data, despite being inexpensive and highly parallelizable to collect, often suffers from a substantial visual domain gap and limited environmental diversity, resulting in weak real-world generalization. We present an efficient video augmentation framework that converts simulated VLA videos into realistic training videos while preserving task semantics and action trajectories. Our pipeline extracts structured conditions from simulation via video semantic segmentation and video captioning, rewrites captions to diversify environments, and uses a conditional video transfer model to synthesize realistic videos. To make augmentation practical at scale, we introduce a diffusion feature-reuse mechanism that reuses video tokens across adjacent timesteps to accelerate generation, and a coreset sampling strategy that identifies a compact, non-redundant subset for augmentation under limited computation. Extensive experiments on Robotwin 2.0, LIBERO, LIBERO-Plus, and a real robotic platform demonstrate consistent improvements. For example, our method improves RDT-1B by 8% on Robotwin 2.0, and boosts $π_0$ by 5.1% on the more challenging LIBERO-Plus benchmark. Code is available at: https://github.com/nanfangxiansheng/Seeing-Realism-from-Simulation.

• The paper introduces a novel video augmentation pipeline that rewrites simulation captions, uses depth-based structural conditioning, and applies conditional diffusion to enhance VLA model performance.
• The methodology includes a three-stage velocity caching strategy that reduces runtime by over 60% and a coreset sampling method for efficient data selection.
• Experimental results show consistent performance gains (up to +15% success rate) in sim-to-real tasks across simulated benchmarks and real-world robotic scenarios.

Efficient Video Augmentation for Sim-to-Real Vision-Language-Action Model Generalization

Motivation and Problem Analysis

Vision-Language-Action (VLA) models have emerged as a dominant paradigm in robotic control, leveraging large-scale real-world trajectory datasets to learn generalizable policies. However, real-world data acquisition is costly and limited in environmental diversity, restricting generalization. Simulated data offer scalable collection but suffer from pronounced domain gaps—models trained with simulation often memorize spurious correlations and fail under disturbances such as lighting changes, texture variations, and layout perturbations. Quantitative analyses on the LIBERO-Plus benchmark show catastrophic drops from 95% to below 30% accuracy under minor distribution shifts, highlighting the brittleness of current VLA models in realistic conditions. The authors propose an efficient and scalable video augmentation framework to bridge this gap by converting simulated VLA videos into task-consistent, realistic training videos with enhanced environmental diversity, directly addressing the deficiencies of sterile simulation datasets.

Figure 1: Visual comparison in LIBERO-Plus showing failure of baseline VLA models under texture and lighting changes, while the augmented approach succeeds.

Pipeline: Structured Conditional Video Transfer

The framework synthesizes diverse, semantically aligned videos from simulation trajectories via four stages:

1. Video Captioning and Rewriting: Initial temporal video captions summarizing objects and spatial dynamics are extracted (e.g., via VideoChat2) and rewritten using a LLM (e.g., Qwen3-8B) to introduce environmental variations such as surface materials, background textures, and object colors, preserving semantic tasks while diversifying context.
2. Depth-Based Structural Conditioning: Depth maps are extracted as geometric constraints to enforce trajectory and spatial fidelity in the transferred videos, mitigating inconsistencies and preserving task actions robustly.
3. Conditional Diffusion Video Generation: A state-of-the-art conditional video diffusion model (Cosmos-Transfer 2.5) generates realistic video strips from the rewritten captions and depth inputs, producing visually varied and temporally consistent scenes aligned with the original action trajectories.

   Figure 2: Overall pipeline—simulation trajectories are sampled and augmented via conditional video transfer for robust VLA training.

Diffusion Feature Reuse: Velocity Caching for Scalable Generation

Scaling the generation of augmented videos is computationally intensive. The authors identify substantial temporal redundancy in the denoising procedure of diffusion models: adjacent velocity predictions exhibit prolonged stable phases, allowing for reuse via caching.

The three-stage velocity caching strategy:

• Initial phase: Compute velocity at every step due to rapid variation.
• Stable phase: Cache velocity every $\alpha$ steps and reuse it within a threshold.
• Adjustment phase: Resume full computation for final denoising refinements.

This mechanism achieves over 60% reduction in runtime with negligible quality degradation across 10 Robotwin 2.0 tasks, facilitating practical large-scale augmentation.

Figure 3: Stable phase in velocity prediction, visualized via Euclidean distance, enables effective cache-based acceleration.

Figure 4: Acceleration rates across Robotwin 2.0 tasks illustrate the substantial reduction (>60%) in video transfer runtime.

Coreset Sampling for Computational Efficiency and Dataset Diversity

Augmenting every simulated trajectory is computationally prohibitive and redundant. The coreset sampling strategy targets augmentation resources toward high-value samples by balancing policy difficulty (average action prediction loss from RDT-1B) and visual diversity (Cosmos-Embed1 embeddings):

• Builds a $k$-nearest neighbor graph over trajectory embeddings, weighting nodes by policy loss and visual proximity.
• Aggregates neighborhood difficulty via forward message passing to promote challenging and non-redundant samples.
• Redundancy suppression via reverse message passing penalizes selection of visually similar neighbors.
• Iterative greedy selection achieves compact, non-redundant coresets maximizing augmentation benefit.

This strategic selection improves both coverage of rare task configurations and inclusion of edge cases, substantially raising generalization with minimal added computation.

Figure 5: Schematic of the coreset sampling algorithm, balancing difficulty and diversity for efficient augmentation.

Figure 6: t-SNE visualization of coreset selection—coresets target high-loss, diverse regions of the training manifold.

Experimental Results: Sim-to-Real Enhancement and Robustness

Robotwin 2.0

Single-task and multi-task experiments under clean (“Easy”) and domain-randomized (“Hard”) settings reveal:

• Augmented data improves RDT-1B's performance by +10% in hard scenarios (average), demonstrating enhanced resilience to real-world complexities.
• Multi-task coreset-based augmentation (10% ratio) achieves +8% absolute gain, proving that even modest augmentation is highly effective when targeted for diversity and difficulty.

LIBERO and LIBERO-Plus

Testing $\pi_0$ and $\pi_{0.5}$ models on LIBERO-Plus’s spatial suite, the augmentation yields:

• $\pi_0$ gains an average of 5.1% across disturbance categories (light, layout, background, instructions).
• Gains are substantial where evaluation settings diverge from training distribution; marginal or negative in settings closely matching simulation, emphasizing the need for controlled augmentation.

  Figure 7: Performance improvement trends for different coreset sampling percentages in LIBERO-Plus, showing diminishing returns beyond 30%.

Real-World Robotic Generalization

Physical experiments on AgileX Piper using augmented VLA models ($\pi_0$, $\pi_{0.5}$) across three OOD scenarios (position shifts, background shifts):

• Consistent +13–15% average gains on success rate.
• Robustness to spatial and visual disturbances is enhanced without sacrificing in-distribution accuracy.

Video Quality Evaluation and Ablations

Comparative metrics with RoboTransfer baseline show:

• Substantial reductions in geometric error (~2–6$\times$ improvement in RMSE, Abs.Rel, Sq.Rel).
• Higher semantic alignment (VideoCLIP-XL similarity scores).
• Marginal runtime overhead compensated by velocity caching.

(Ablation on different hyperparameters for velocity caching and coreset sampling demonstrates stability and robustness across both simulated and real-world tasks.)

Figure 8: Augmented vs. original videos in Robotwin 2.0, visually validating fidelity and environmental richness.

Figure 9: LIBERO video augmentations illustrate diverse backgrounds and contexts via the proposed pipeline.

Figure 10: Real robot experiment videos—transferred data achieves sim-to-real visual consistency.

Practical and Theoretical Implications

The framework constitutes a robust solution for scaling VLA training without excessive real-world data collection, achieving notable sim-to-real generalization improvements via targeted data diversity and domain randomization. The velocity caching innovation allows practitioners to augment at scale, while coreset sampling provides a principled strategy for maximizing utility per computation budget. Theoretical implications include:

• Message-passing coreset designs balancing diversity and difficulty are adaptable to broader embodied-data tasks.
• Conditional diffusion models with structural control signals establish new standards for semantic-preserving simulation-to-real transfer.
• Controlled caption rewriting via LLMs demonstrates a practical path for scalable context diversification without semantic drift.

These methods promise to advance robotic policy learning in multi-modal domains, with future work likely to extend augmentation modalities (action, sensor noise, geometry), refine structural controls, and optimize coreset selection for continually evolving robotic benchmarks.

Conclusion

This work presents an efficient, scalable pipeline for sim-to-real video augmentation in VLA training, introducing structured caption rewriting, depth-based conditional diffusion, three-stage velocity caching, and message-passing coreset sampling. The strategy demonstrates robust gains in sim-to-real generalization, efficiency, and semantic fidelity across simulated, benchmark, and real-world tasks. These results establish benchmarks for systematic data-centric robotics research and highlight avenues for adaptive augmentation and advanced embodied world modeling in AI.

Figure 2: Overall framework of the proposed method: coreset sampling selects key simulation trajectories for augmentation via conditional video transfer, yielding realistic, diversified training data.