A Heterogeneous Architecture for Robot RL Beyond GPU-Dominant Paradigms

Abstract: Simulation-based RL for contemporary robot control is increasingly organized around GPU-resident simulation: physics, rollout collection, and learning are placed on a single GPU-centric execution path. This paradigm has greatly improved training speed, but it has also encouraged a default assumption that efficient training requires physics to reside on the GPU. We revisit this assumption. Our view is that, in simulation-dominated robot control, the essential question is not which processor runs physics, but whether simulation throughput, policy learning, and runtime synchronization form an efficient end-to-end loop. We present UniLab, a heterogeneous CPU-simulation / GPU-learning architecture that decouples CPU-parallel simulation from GPU policy updates through a unified runtime for data movement, buffering, and synchronization. UniLab is implemented as a complete and extensible training system using MuJoCoUni and MotrixSim CPU-batched physics backends, supporting PPO, SAC, FlashSAC, TD3, and APPO. On representative simulation-based robot control tasks, UniLab improves end-to-end training efficiency by 3--10$\times$ under the same hardware configuration, while reducing dependence on the NVIDIA CUDA-based software stack and supporting cross-platform execution on the Apple macOS platform and the AMD ROCm and Intel XPU accelerator backends. These results show that GPU simulation is an effective path to efficient training, but not a necessary one, broadening the practical system choices available for robot RL training. Project page: https://github.com/unilabsim/UniLab.

• The paper introduces a heterogeneous framework that decouples CPU-based simulation from GPU learning to enhance training efficiency.
• It demonstrates 3–10× end-to-end performance gains over GPU-centric paradigms using standard RL algorithms like PPO, SAC, and TD3.
• Empirical results validate cross-platform robustness and reduced synchronization delays, thereby facilitating practical real-robot deployment.

UniLab: A Heterogeneous Architecture for Robot RL Beyond GPU-Dominant Paradigms

Context and Motivation

The field of robot RL has adopted GPU-resident simulation as the de facto training paradigm, leveraging GPU-centric pipelines for physics, rollout collection, and learning to maximize throughput. This approach, exemplified by systems such as Isaac Gym, Isaac Lab, Genesis, MuJoCo Playground, and ManiSkill3, has demonstrably accelerated training but has fostered the assumption that efficient robot RL mandates GPU-based environment execution. The paper re-examines this assumption, proposing that end-to-end training efficiency is determined by the orchestration and coordination of simulation throughput, learner utilization, synchronization, and data movement—factors beyond the mere hardware placement of physics.

UniLab Architecture

UniLab introduces a heterogeneous training framework that decouples CPU-based batched simulation from GPU policy learning, mediated by a unified runtime facilitating data movement, buffering, and synchronization. CPU backends (MuJoCoUni, MotrixSim) execute parallel rigid-body simulations, while the GPU performs dense learning updates. The runtime architecture ensures non-blocking GPU learning, maintaining high simulation throughput and minimal handoff overhead.

Figure 1: UniLab system architecture illustrating CPU-side simulation, unified runtime scheduler, and GPU learner with explicit data, synchronization, and scheduling paths.

This architecture diverges from GPU-dominant designs without introducing new policy optimization algorithms. Instead, it robustly supports PPO, SAC, FlashSAC, TD3, and APPO under unified interfaces, enforcing hardware role separation and runtime abstraction. UniLab’s backend contracts expose explicit task/physics semantics, decoupling simulator implementation from training throughput.

Empirical Analysis

Simulation Throughput

Contrary to entrenched expectations, batched CPU simulation provides competitive throughput across robot-control workloads, including contact-rich and dexterous manipulation domains. Throughput is sustained even under large environment batches, supporting high-frequency data generation on diverse hardware platforms.

Figure 2: CPU simulation throughput across representative robot control scenes, demonstrating simulator-side performance across tasks and backends.

End-to-End Training Efficiency

UniLab’s heterogenous split yields substantial wall-clock reductions under less synchronized, asynchronous training schedules. Empirical results demonstrate 3–10$\times$ end-to-end gains for SAC, APPO, and FlashSAC across locomotion, motion tracking, and manipulation tasks, compared to GPU-centric baselines.

Figure 3: End-to-end training efficiency improvements on representative robot control tasks with UniLab architecture.

Analysis via cycle placement ablation reveals that resource contention in GPU-resident simulators prolongs learner cycles. UniLab’s CPU simulation/GPU learning split ensures collector work completes ahead of updates, maximizing pipeline overlap.

Figure 4: Training-cycle placement ablation demonstrating the separation of simulation collection and learning during each learner cycle.

Cross-Platform Robustness and Real-World Deployment

UniLab’s unified interfaces enable execution on Apple macOS (M series), AMD ROCm, and Intel XPU backends, eliminating dependence on NVIDIA CUDA while retaining practical training efficiency. Real-robot deployment across six tasks is validated, showcasing portability and practical applicability.

Figure 5: Overview of sim-to-real experiments across six real-robot tasks with UniLab-enabled training and deployment.

Figure 6: Cross-platform training curves and final performance for representative devices, confirming platform-agnostic trainability.

System-Level Replay Path: SAC Case Study

The appendices detail the replay buffer path for SAC, contrasting baseline GPU-cache methods with CPU sample-before-transfer pipelines. UniLab's double-buffer path reduces learner cycle time from 211 ms to 136 ms (A100), eliminates collector stalls, and shrinks GPU memory requirements by removing persistent GPU replay caches. Replay sampling and H2D transfers are scheduled asynchronously, maximizing overlap with learner updates.

Figure 7: Baseline SAC and optimized SAC learner-cycle timelines showing drastic reduction in cycle time and elimination of collector stall.

Figure 8: System-attribution summary, batching efficiency, runtime overlap, and collector-side inference cost for optimized SAC trace.

Replay-path ablations confirm gains arise from structural pipeline changes (replay ownership and boundary timing), not algorithmic modifications.

Figure 9: C-to-baseline ablation for the SAC replay path, highlighting impact on wall-clock time, learner-cycle duration, boundary waiting, and GPU memory usage.

Analysis of buffer and communication overhead further quantifies data movement, weight-synchronization, and residual waiting, establishing that UniLab's pipeline maintains low coordination costs.

Figure 10: SAC buffer and communication overhead breakdown and auxiliary replay placement benchmarking.

Implications, Limitations, and Future Directions

UniLab establishes that the location of physics simulation (CPU, GPU, or mixed) is not the primary constraint for training efficiency in simulation-dominated robot RL. The results show efficient training is a systems-level coordination problem instead of a hardware-selection question. UniLab enables broader system design choices, reduces coupling to hardware-specific software stacks, and permits practical cross-platform robot RL.

Limitations are acknowledged: UniLab’s advantage is pronounced in simulation-dominated workloads where learning and simulation can be decoupled; strictly synchronized or vision-dominated tasks may not see comparable gains. Absolute throughput at extreme scale (multi-GPU) is not benchmarked, and coverage for deformable/soft-body simulation is absent. Additionally, the effect of backend physics semantics on sim-to-real transfer remains an open question.

Future development should focus on extending backend support for broader physics models, enabling distributed heterogeneous configurations, and formalizing sim-to-sim/sim-to-real robustness analysis under varying simulator fidelities.

Conclusion

UniLab demonstrates that efficient robot RL training is a closed-loop systems problem requiring high-throughput, well-coordinated simulation and learning. The results invalidate the assumption of necessity for GPU-resident physics and establish a design space for heterogeneous architectures that outperform GPU-centric paradigms in many settings. UniLab’s approach reduces platform dependency, enhances portability, and provides practical evidence for systems innovation beyond hardware specialization (2605.30313).