Demonstrating DVS: Dynamic Virtual-Real Simulation Platform for Mobile Robotic Tasks

Abstract: With the development of embodied artificial intelligence, robotic research has increasingly focused on complex tasks. Existing simulation platforms, however, are often limited to idealized environments, simple task scenarios and lack data interoperability. This restricts task decomposition and multi-task learning. Additionally, current simulation platforms face challenges in dynamic pedestrian modeling, scene editability, and synchronization between virtual and real assets. These limitations hinder real world robot deployment and feedback. To address these challenges, we propose DVS (Dynamic Virtual-Real Simulation Platform), a platform for dynamic virtual-real synchronization in mobile robotic tasks. DVS integrates a random pedestrian behavior modeling plugin and large-scale, customizable indoor scenes for generating annotated training datasets. It features an optical motion capture system, synchronizing object poses and coordinates between virtual and real world to support dynamic task benchmarking. Experimental validation shows that DVS supports tasks such as pedestrian trajectory prediction, robot path planning, and robotic arm grasping, with potential for both simulation and real world deployment. In this way, DVS represents more than just a versatile robotic platform; it paves the way for research in human intervention in robot execution tasks and real-time feedback algorithms in virtual-real fusion environments. More information about the simulation platform is available on https://immvlab.github.io/DVS/.

• The paper demonstrates that integrating virtual-real synchronization significantly enhances mobile robotic task execution and algorithm performance.
• It details an innovative architecture with real-time intervention and dynamic scene generation that improves sim-to-real generalization.
• Experimental validation shows increased grasping success rates and more accurate pedestrian trajectory predictions in complex scenarios.

Demonstrating DVS: Dynamic Virtual-Real Simulation Platform for Mobile Robotic Tasks

Introduction

The increasing complexity of robotic tasks in embodied AI necessitates advanced simulation platforms that can effectively support task learning, decomposition, and adaptive deployment. Traditional simulation platforms constrain robot capabilities, primarily due to idealized environments and limited data interoperability. These limitations create practical deployment challenges, especially for dynamic task scenarios and real-world feedback integration. The paper "Demonstrating DVS: Dynamic Virtual-Real Simulation Platform for Mobile Robotic Tasks" (2504.18944) introduces DVS, a novel simulation framework aimed at addressing these challenges by integrating dynamic virtual-real synchronization for mobile robots.

System Architecture

DVS distinguishes itself from existing platforms through its integration capabilities for virtual and real-world systems. This platform is designed to facilitate the synchronization of dynamic tasks between virtual simulations and physical operations. The core components of DVS include virtual-real fusion workflows, customizable large-scale indoor scenes, and dynamic pedestrian behavior models.

Key features of DVS are centered around three pillars:

1. Virtual-Real Fusion: Through high-precision optical motion capture and ROS-based communication, DVS ensures accurate synchronization between virtual simulation and physical robot states.
2. Dynamic Scene Generation: DVS supports advanced scene modeling with plugins for pedestrian behaviors and dynamic environmental interactions, enhancing the realism and variability of simulated scenarios.
3. Intervention-Enabled Workflow: The platform allows real-time scenario adjustments during task execution, providing improved adaptability for robots in dynamic environments.

   Figure 1: Virtual-Real Data Synchronization Framework. The central demonstrates the synchronization of object pose and robot motion through VRPN and ROS. The left and right parts depict the virtual simulation environment and physical real-world scenes.

Related Work

Simulation platforms have played a pivotal role in AI development, empowering research with controlled environments for robust algorithm training and validation. While platforms such as Habitat, iGibson, and Arena focus on increasing simulation realism and task-specific modeling, they often lack dynamic scene modeling capabilities and efficient integration with real-world feedback mechanisms. These gaps have contributed to notable challenges in sim-to-real generalization—a critical barrier for applying AI models trained in simulation to real-world tasks.

By leveraging DVS, researchers can bridge the discrepancy between simulated and real-world interactions, thus overcoming limitations of previous systems. The integration of dynamic elements within DVS sets a new standard for simulation fidelity, particularly by supporting pedestrian modeling and real-time interaction adjustments using HITL (Human-in-the-Loop) approaches.

Experimental Validation

The experimental validation of DVS showcases its versatility and applicability across a variety of robotic tasks. The platform supports data generation for augmented training datasets, specifically tailored to pedestrian trajectory prediction, robot path planning, and robotic arm grasping tasks.

DVS provides an enriched environment with adaptable human-agent interactions, detailed object pose synchronization, and complex scene configurations necessary for efficient robot training. Experiments show marked improvements in task execution success rates and algorithm performance, demonstrating both the theoretical and practical implications of DVS.

Figure 2: The interactive interface of the simulation platform: The left panel adjusts dynamic pedestrian parameters while the right selects perception data types.

Task-Specific Applications

Grasping Task Intervention

The platform optimizes robotic task execution through dynamic intervention capabilities during manipulation tasks. By intervening during real-world execution, DVS effectively enhances robotic grasping performance, notably increasing the success rate from near-zero to nearly 90% for repeated task execution without human intervention.

Figure 3: The robotic arm is interrupted while executing Prompt A and is requested to execute Prompt B. The first row shows the robotic arm in the virtual platform, and the second row shows the real robotic arm.

Dynamic Trajectory Prediction

DVS further supports pedestrian trajectory prediction and validation tasks in densely populated indoor environments. Algorithms such as STGAT and Trajectron++ are tested within synthetic indoor scenes, providing insight into prediction performance and adaptability within challenging environments.

Figure 4: Visualization of pedestrian trajectory prediction, where each color represents a different pedestrian. The accuracy of the prediction is higher when the predicted trajectory closely aligns with the ground truth.

Conclusion

DVS represents a significant advancement in robotic simulation platforms by facilitating dynamic, closed-loop task validation and optimization. With its features like virtual-real synchronization, real-time interaction capabilities, and dynamic scene modeling, DVS provides an integral environment for enhancing robotic task learning and sim-to-real generalization. The platform sets a precedent for future research in human-robot collaborative environments and adaptive robotic systems.

In summary, DVS paves the way for robust and practical deployment of embodied AI systems in complex real-world scenarios, effectively bridging the gap between virtual simulations and physical operations. Future research with DVS will continue along avenues of incorporating haptic feedback, AI-driven intervention strategies, and enhancing industrial robotic arm compatibility.