MagicSim: A Unified Infrastructure for Executable Embodied Interaction

Abstract: Robot learning and embodied agents now require simulation to serve as a shared execution substrate linking control, skills, and planning, not only as a renderer, controller testbed, or fixed task environment. Existing pipelines split these layers with "magic" actions, disconnected training environments, or forward-only renders that cannot reproduce, evaluate, and annotate the same episode. We present MagicSim, an embodied interaction infrastructure built around one deterministic batched runtime and a shared Markov decision process (MDP). From YAML-first specifications that decouple contents, placement, behavior, and agent exposure, MagicSim constructs diverse executable worlds spanning task families, interaction regimes, physics, layouts, sensors, avatars, and robot embodiments in one reset-and-step loop. A common execution interface grounds high-level commands through controllers, atomicskills, planner primitives, and asynchronous planning, realizing them as robot actions rather than simulator-side state edits. One task definition supports three capabilities: benchmark and RL evaluation, an autocollect interface that automatically turns commands into grounded trajectories, and agent/VLM-facing interaction. For automatic execution, commands flow through a Command->Skill->Planner->Robot->Record pipeline, while per-environment command, skill, planning, retry, annotation, and episode states advance independently above the shared physics tick. Successful rollouts are saved as structured multimodal trajectories aligning language supervision, action representations, visual/geometric representations, and task-level status with the executed episode. MagicSim thus unifies diverse world construction, embodied execution, task evaluation, automatic rollout generation, and interactive agent interfaces in one planner-in-the-loop runtime.

• The paper introduces a unified MDP-based infrastructure that consolidates simulation, control, planning, and annotation for reproducible embodied interactions.
• It features a parallel batched runtime supporting over 15 physics families and heterogeneous robot embodiments, enabling robust counterfactual testing with deterministic replay.
• The comprehensive sensor and annotation system facilitates high-throughput data generation and consistent evaluation across RL, scripted demos, and agent-driven tasks.

MagicSim: Unified Infrastructure for Executable Embodied Interaction

Introduction and Motivation

MagicSim introduces a comprehensive infrastructure designed to unify world construction, robot control, planner-in-the-loop interaction, annotation, and diverse research drivers within a deterministic, parallelized runtime. Traditional robot learning and embodied-AI simulators segment the embodied stack—control, skills, planning, agent-level reasoning—into disparate pipelines, frequently relying on non-physical "magic" actions or replayed fixed trajectories devoid of realistic closed-loop execution. MagicSim replaces these fragmented paradigms with a single Markov decision process (MDP) abstraction, ensuring every research driver (RL, demonstration collection, agent interaction) operates within one batched, replayable, and annotation-rich execution substrate Figure 1.

Figure 1: MagicSim at a glance—parallel deterministic runtime, Command-to-Skill-to-Planner-to-Robot-to-Record pipeline, three research drivers, and synchronized multimodal annotation.

System Architecture

Batched Deterministic Runtime and Parallelism

MagicSim's runtime supports large-scale parallel simulation, where many sub-environments, with heterogeneous content and layouts, share a single simulator clock but maintain independent semantic event ordering Figure 2. A manager-centric abstraction ensures local resets, domain randomization, deterministic replay via state snapshots, and per-module seeded streams. The runtime contract, implemented above NVIDIA Isaac Sim, encapsulates all simulation, control, and sensing within manager partitions.

Figure 2: Agent-level asynchrony—synchronized physics tick, independent semantic progress per environment.

Deterministic replay is guaranteed at all scales using a hierarchical seeded randomization scheme, lifecycle modes (P1/P2/P3) for asset management under hard/soft reset constraints, and snapshot/restore semantics. These design choices enable controlled counterfactuals and robust reproducibility for systematic evaluation.

Figure 3: Deterministic manager runtime—disjoint seeded streams and ordered object lifecycle management.

Heterogeneous Multi-Physics Engine

MagicSim is architected as a unification layer atop existing simulators. The engine natively supports co-execution and physical coupling across rigid bodies, articulations, soft-body FEM, cloth/garments, fluids, granular media, inflatables, ropes, avatars, effect fields, and more—representing 15 simulation families Figure 4, each parameterized by YAML-first configuration supporting extensive randomization of appearance, physics, and task affordances.

Figure 4: MagicSim family taxonomy and coupled multi-physics interactions.

World carriers (terrains, rooms) and layout managers drive environment instantiation, leveraging declarative, VLM-backed, and heuristic/constraint-based placements (Figure 5, 8). Domain randomization is managed per-axis, insulated by manager streams Figure 6, ensuring full decoupling of visual, physical, spatial, and sensory perturbations across parallel environments.

Figure 5: Randomized layouts at object and room scales with independent sampling per environment.

Figure 7: World carriers—planes, terrains, and navigation-ready rooms as carrier backbones.

Figure 6: Orthogonal domain randomization axes—spatial, visual, physical, sensory, and robot state.

Multi-Embodiment Control and Planning Stack

Robots (single/double-arm manipulators, dexterous hands, mobile bases, humanoids, quadrupeds) are all unified via a channel interface abstracting base, arm, and end-effector actions. Embodiments are dynamically composable per environment Figure 8, and actions are realized via a three-level stack: closed-loop control (including learned controllers and classical terms), mid-level planners (cuRobo, navigation planners), and skill/command decomposition.

Figure 8: Morphological span of robot embodiments (manipulators, mobile, dexterous, humanoid, quadrupeds).

cuRobo is extended with batched, per-environment, heterogeneous, multi-tool-frame, and asynchronous solving. Batched micro-inference is managed by a solve farm Figure 9, supporting high-throughput, collision-aware, motion and IK planning in closed loop with environment feedback. Collision geometry abstraction leverages per-link sphere models for efficient GPU collision distance checking Figure 10.

Figure 10: cuRobo collision-sphere abstractions for heterogeneous morphologies.

Asynchronous planner invocation ensures that no environment blocks the batch, enabling each episode to mix RL, skill-planned, or agent-driven actions, supporting long-horizon, hierarchical, and failure-recoverable task execution.

Sensor and Annotation Stack

MagicSim’s runtime integrates multiple observation channels: visual (RGB-D, normals, optical flow), tactile (force/pressure, visuotactile images, taxel-level probe contact), geometric (occupancy, navmesh, frame/affordance tracking), IMU, LiDAR (Figure 11, 17). Sensor data is manager-owned, ensuring reset consistency, batched processing, and efficient I/O.

Figure 11: Touch sensing stack—probe tactile, visuotactile, geometry-accurate raycasting.

Figure 12: Embodied sensor suite—LiDAR, IMU, navmesh, occupancy, and frame tracking layers.

Two-layer annotation system: asset-level (with programmatic grounding of keypoints, affordance priors, physics-validated candidate banks) and runtime (synchronized action, planner, skill, trajectory, and language streams). The annotation pipeline ensures that all rollouts are multimodally aligned and indexed Figure 13.

Figure 13: Runtime and asset-side annotation modalities—rendered and native fields, only persisted on success.

Task Benchmarking, Auto-Collection, and Serving

Tasks are defined as Gym-compatible MDPs, orthogonalized across embodiment and interaction regime families and supporting diverse manipulation, navigation, loco-manipulation, garment, dexterous, camera-embodiment, and HRI tasks Figure 14. Every task is accessible via three drivers (RL/training, scripted demo collection, agent/replay driver) with identical MDP boundary.

AtomicSkills form the physical interface between commands and execution, encoding the space of success-verifiable, planner-grounded actions Figure 15. Skills are realized using motion-generation, MPC, or learned policies, supporting recovery and retries with typed failure semantics, modularized per-command and per-skill annotations.

Figure 15: AtomicSkill repertoire across robot/task/physics families (grasp, manipulate, deform, HRI, navigation).

Episodes are only recorded upon validated success, with buffer flushing and stream alignment, producing high-quality, success-gated, semantically structured data corpora.

Batch- and microservice-level serving is supported through queueing, async client pools, and agent/VLM interaction APIs, enabling simulator-as-a-service and parallelized, multi-agent research at scale.

Empirical System Results and Capability Coverage

While MagicSim is primarily a systems paper, several strong system-level claims are supported:

• Breadth: Direct support for 15+ physics families, 33 registered robots, 8 cross-cut task families, and all relevant downstream research domains (robotics, physics, embodied VLM/VLA).
• Determinism: reset/replay contract with manager-local seeded streams, supporting reproducible, parameterized counterfactual experiment generation.
• Scalability: Parallel batched runtime, async planning, non-blocking execution—order-of-magnitude speedups for large-scale collection and RL.
• Physical Coupling: Demonstrated interaction across rigid, deformable, fluid, granular, and articulated families in one episode.
• Annotation Richness: Synchronized multi-modal data and structured language/narration for each episode, suitable for foundation model training and VLM/agent evaluation.
• Versatility: A single code/configuration base supports RL, scripted demonstration, and agent interaction (one MDP, three drivers paradigm).

Practical and Theoretical Implications

MagicSim moves the field toward a truly unified, reproducible, and extensible infrastructure layer for embodied AI research, closing the critical gap between RL, skill/data generation, planner-based reasoning, and agent/VLM evaluation. By committing to a deterministic, batched, manager-partitioned design with explicit skill/command boundary, the system supports consistent experimentation in long-horizon, contact-rich, cross-embodiment, and physically grounded settings—domains where modular benchmark/task-specific simulators frequently fail.

On the practical side, the infrastructure enables rapid benchmarking, generation of large-scale and diverse training corpora, and rigorous agent evaluation with aligned multimodal annotation. On the theoretical side, the ability to compare, replay, and systematically perturb complex human–robot–environment interactions paves the way for new research in counterfactual reasoning, embodied instruction following, causal probe design, generalist robot model evaluation, and dynamic foundation model alignment.

Future Directions

Open challenges identified include (1) further increasing simulation fidelity, particularly at solver boundaries and for fine-grained contact; (2) expanding and learning new AtomicSkill backends—including high-DOF dexterous control, deformable manipulation, force-aware tasks, and adaptive hierarchical planners; (3) supporting even longer, denser, and more compositional episodes spanning multi-agent (HRI, safety, autonomous lab) and multi-modal (active perception/exploration) domains.

Conclusion

MagicSim is an extensible, high-fidelity, open-ended infrastructure for embodied AI and robotics research, designed as a manager-based, deterministic, parallel, and annotation-rich execution substrate. With its one-MDP, three-driver paradigm, multi-physics reach, and semantic annotation pipeline, it enables systematic experimentation, scalable data generation, and robust agent evaluation across the full embodied stack, thereby supporting the next generation of embodied, interactive, physically grounded intelligence research.

(2606.17511)