Actuator Reality Shaping for Zero-Shot Sim-to-Real Robot Learning

Abstract: Sim-to-real transfer in robot learning is often limited by discrepancies between the ideal actuator dynamics assumed during policy training and the nonlinear, hardware-dependent behavior of physical motors. While conventional approaches attempt to bridge this gap by increasing simulator fidelity through system identification, domain randomization, or learned actuator models, we introduce an alternative paradigm: actuator reality shaping. Instead of modifying the simulator to match the real world, our method shapes the closed-loop behavior of physical actuators to match the idealized second-order reference dynamics used in simulation. By equipping each joint with a two-degree-of-freedom feedforward--feedback controller, we decouple reference-response shaping from robust stabilization, thereby providing a standardized actuator interface for reinforcement learning policies. As a result, policies trained only with the prescribed reference model can be deployed zero-shot on real hardware without task-level fine-tuning or learned actuator models. We validate the approach on a single-joint high-gear-ratio servo under external loads and a 7-DOF robotic arm reaching task, where actuator reality shaping substantially reduces sim-to-real tracking error and improves zero-shot task performance compared with standard servo-control and representative real-to-sim-to-real baselines. We further demonstrate zero-shot transfer on a wheeled-legged robot driving over a slope and a humanoid robot walking, suggesting that actuator reality shaping can serve as a reusable interface for robot learning across diverse hardware platforms. Project page: https://syamamori.github.io/ActuatorRealityShaping.github.io/

• The paper introduces actuator reality shaping, standardizing each joint’s closed-loop response via a cascaded 2-DoF controller to match ideal simulation dynamics.
• The method dramatically reduces tracking errors, achieving up to 96.5% reduction in multi-DOF reaching tasks compared to traditional PID and PD controllers.
• It decouples policy learning from hardware-specific adaptations, eliminating the need for recurrent on-robot fine-tuning during sim-to-real transfer.

Actuator Reality Shaping for Zero-Shot Sim-to-Real Robot Learning

Introduction and Motivation

Sim-to-real transfer remains an unsolved challenge in robot learning due to actuator model mismatches between simulated environments—often built on idealized, second-order dynamics—and the complex, nonlinear behavior of real hardware actuators. Traditional approaches rely on narrowing this gap by making simulators more realistic through domain randomization, hardware identification, or learned actuator models. However, these methods suffer from limited policy portability and require hardware-specific adaptation and recurrent data collection, especially when actuator configurations change or degrade.

This paper introduces the paradigm of actuator reality shaping, a fundamental shift in perspective. Instead of adapting the simulator to match real hardware, the method shapes the physical actuator's closed-loop response at each joint, making it emulate the same reference dynamics used during policy training in simulation. The standardization occurs via a per-joint, two-degree-of-freedom (2-DoF) controller, combining feedforward and feedback pathways, optionally augmented with a disturbance observer (DOB). As a result, policies trained exclusively with idealized actuator assumptions can be deployed zero-shot on varied hardware platforms—eliminating the need for robot- or task-specific hardware fine-tuning.

Figure 1: Actuator reality shaping aligns the real actuator’s closed-loop response with the simulator’s ideal second-order reference dynamics, enabling zero-shot transfer.

Technical Method: Two-Degree-of-Freedom Control and Disturbance Rejection

The core of the actuator reality shaping architecture is a cascaded 2-DoF controller applied to each robot joint (Figure 2). This structure decouples reference trajectory tracking (handled by a feedforward path designed to match the simulator's reference model) from robust stabilization against disturbances (implemented via a feedback path, typically PID). Residual disturbances due to friction, load, or model error are further suppressed with a DOB.

Figure 2: Architecture of the actuator reality shaping driver: a cascaded 2-DoF controller (position outer loop, velocity inner loop) with a disturbance observer for robust, reference-model tracking.

Mathematically, the controller ensures that the closed-loop joint behavior closely matches the same second-order reference dynamics used during simulation, provided the hardware can track the desired model within its dynamic limits. The DOB increases robustness by estimating and rejecting actuator disturbances in real-time.

Standardized Hardware Interface

By enforcing a consistent reference dynamic at the actuator interface, the policy-level control is agnostic to hardware details like friction, backlash, or torque saturation. Only coarse motor parameters (inertia, torque constant, bandwidth) are required for initial driver design, and no real-world policy fine-tuning or learned actuator models are needed.

Experimental Validation and Numerical Results

Experiments are conducted on diverse platforms: single-joint high-gear-ratio servos, a 7-DOF robotic arm (under disturbance and trajectory following), a wheeled-legged robot, and a high-DOF humanoid robot. In all cases, policies are trained in Isaac Sim exclusively against the prescribed ideal reference model.

The 2-DoF+DOB driver layer yields drastic improvements in sim-to-real tracking error for both single-axis and multi-DOF reaching tasks, compared to PID cascade and simple PD baselines. For instance, in single-joint sinusoidal tracking, mean position deviation is reduced by 96.3% and mean velocity deviation by 99.1% relative to the factory PID cascade controller.

Figure 4: Phase portraits show that the 2-DoF+DOB controller maintains errors near the origin, tightly tracking the simulator’s reference model compared to large errors from baseline controllers.

On the 7-DOF arm lemniscate-reaching task, the proposed method achieves a 96.5% reduction in end-effector error versus the industrial cascade controller. The DOB yields an additional 50% reduction over the feedforward-only 2-DoF controller and a 72% reduction over a manually-tuned PD loop, highlighting the critical impact of disturbance compensation.

The approach is also compared to data-driven actuator modeling methods such as ASAP (Delta Action correction), which require substantial real-hardware rollouts for system identification and retraining. The actuator reality shaping method achieves superior zero-shot performance without any hardware-specific data collection, as shown in direct head-to-head experiments.

Figure 5: Comparison with ASAP shows that, despite ASAP’s real-data fine-tuning, the 2-DoF+DOB approach achieves lower sim-to-real error with zero hardware-specific adaptation overhead.

To demonstrate generality, the method is deployed on a wheeled-legged robot and a humanoid robot (each with different actuation technologies and loads). In both cases, policies trained on the simulator transfer zero-shot, enabling tasks such as climbing a slope and stable bipedal walking, despite unmodeled body and actuator dynamics.

Figure 3: Zero-shot deployment: (a) wheeled-legged robot climbing a slope and (b) humanoid robot walking, both using policies from the idealized simulation, demonstrating generality of the actuator reality shaping interface.

Implications for Robot Learning and Control

Actuator reality shaping repositions the sim-to-real transfer problem as a control-layer standardization issue, decoupling policy learning and hardware adaptation. This modularizes robot learning pipelines:

• Policy Portability: RL policies, once trained, can be deployed across platforms differing in actuator or mechanical details by adapting only the driver layer.
• Reduced System Identification Burden: Only coarse actuator parameters are needed, circumventing repeated, robot-specific system identification or domain randomization.
• Real-World Learning Efficiency: Policies can be deployed zero-shot—a critical requirement in data-scarce real-world robot applications—removing the cost of on-robot RL or fine-tuning.
• Generalization: By exposing the RL agent to an idealized, hardware-independent actuator interface, learned controllers become inherently more platform-agnostic.

However, the scope of actuator reality shaping assumes the hardware’s dynamics can be adequately matched to the reference model. For low-gear-ratio, highly nonlinear, or time-varying actuator systems, supplementary adaptation mechanisms may be required. Future work in adaptive reference model control and integration with more expressive system identification techniques could extend applicability to even broader hardware classes.

Conclusion

This work introduces actuator reality shaping—a per-joint, two-degree-of-freedom control architecture with optional disturbance observation—as a systematic interface for zero-shot sim-to-real transfer in robot learning. By relocating adaptation from simulation to the control layer, the method erases the sim-to-real gap induced by actuator dynamics, without requiring hardware-specific learning or data. This standardized dynamical abstraction enables efficient, modular, and generalizable robotic policy deployment in the real world.