RealityGrad: Online Sim-to-Real Transfer

• RealityGrad Algorithm is an online sim-to-real method that leverages differentiable physics to iteratively refine control policies and simulation models.
• The algorithm alternates between trajectory optimization in simulation and real-world rollouts, using gradient-based updates to minimize discrepancies in robot dynamics.
• Empirical results demonstrate significant error reductions (up to 63%) and efficient processing on commodity hardware, paving the way for more scalable robotic learning.

RealityGrad is an iterative, online sim-to-real transfer algorithm that harnesses live robot rollouts and differentiable physics to efficiently bridge the reality gap—the well-known discrepancy between simulated robot performance and real-world results. The algorithm alternates between trajectory optimization in a differentiable physics simulator and real robot rollouts, using gradient-based parameter refinement to improve both the control policy and the simulation model. RealityGrad achieves substantial reductions in task error within practical runtimes on commodity hardware and establishes a framework for scaling gradient-based sim-to-real transfer to more complex robotic environments (Collins et al., 2021).

1. Problem Statement and Motivation

Robotic control policies developed in simulation often exhibit degraded performance when deployed on physical hardware. This performance loss is due to the "reality gap," which emerges from imperfect simulation of dynamics, friction, sensor noise, and other model inaccuracies. Traditional sim-to-real approaches are categorized as:

• Zero-shot transfer: Techniques such as domain randomization and system identification generate robust policies by training controllers in diverse simulated conditions, then deploying them without further adaptation. While effective for robustness, these methods tend to produce conservative, suboptimal behaviors and require substantial computational resources.
• Online sim-to-real: In these methods, real-world data is used to iteratively update the model or policy. Classical implementations often rely on black-box surrogate models, evolutionary algorithms, or non-differentiable physics, resulting in slow convergence and heavy computational demands.

Differentiable physics simulators provide an opportunity to expose analytical or autodifferentiated gradients that enable direct, efficient optimization of both control trajectories (sim2real) and simulator parameters (real2sim), leading to improved sample efficiency and precision compared to finite-difference or sampling-based strategies (Collins et al., 2021).

2. Algorithmic Workflow

RealityGrad alternates between simulation-based and real-world components over multiple iterations. Each iteration consists of four key stages:

1. Trajectory Optimization (Sim2real): Generate $K$ optimal control trajectories in simulation using a differentiable simulator, formulating the problem as a finite-horizon Model Predictive Control (MPC) task.
2. Policy Regression: Supervised neural network regression fits a policy $\pi_\phi$ to the optimized trajectories, with training data $(x_t^k, x_T^{*,k}) \mapsto u_t^k$.
3. Real-World Rollout: Deploy the learned policy on the robot, collecting state and control sequences at high frequency over a short time horizon.
4. Model Parameter Refinement (Real2sim): Update simulator parameters $\theta$ to minimize the discrepancy between observed and simulated trajectories driven by the same controls, using gradient-based nonlinear least squares techniques aided by autodifferentiation.

This closed-loop process yields coordinated improvements in both controller robustness and simulator fidelity across iterations, with convergence observed after only 1–2 rounds in empirical trials (Collins et al., 2021).

3. Mathematical Formulation

The algorithm relies on differentiable parametrization of the robot's dynamics:

$x_{t+1} = f_\theta(x_t, u_t), \quad t=0,\dots,T-1,$

where $x_t\in\mathbb{R}^n$ is the state, $u_t\in\mathbb{R}^m$ the control input, and $\theta\in\mathbb{R}^p$ the vector of simulator parameters (including masses, inertias, damping, frictions, and gravity vector).

• Trajectory Optimization: For initial state $x_0$ and a sampled goal $x_T^*$, minimize the finite-horizon cost:

$\pi_\phi$0

subject to dynamics and torque limits, where $\pi_\phi$1 penalizes distance to $\pi_\phi$2.

• Policy Regression: Minimize:

$\pi_\phi$3

across all demonstration points.

• Model Parameter Update: Refine $\pi_\phi$4 via:

$\pi_\phi$5

with $\pi_\phi$6, and solve

$\pi_\phi$7

typically via reverse-mode autodifferentiation and nonlinear least squares (e.g., Ceres Solver).

4. Implementation Details

RealityGrad was demonstrated on a Kinova Mico$\pi_\phi$8 six-degree-of-freedom manipulator using the following infrastructure:

• Simulation: Tiny-Differentiable-Simulator (TDS), coupled with Adept for autodifferentiation and Ceres for nonlinear least squares.
• Robot State Sensing: Joint encoders (angles and velocities) acquired via ROS at 25 Hz. End-effector position obtained through forward kinematics.
• Control Policy: Feedforward neural network with two hidden layers (128 ReLU units, inputs: $\pi_\phi$9 goal pose$(x_t^k, x_T^{*,k}) \mapsto u_t^k$0, outputs: joint torques), trained on ADAM optimizer.
• Task: Random joint-space reaching from a standard "candle" home pose to a goal sampled within a 1-second trajectory, with no external contacts.

A single iteration entailed generation of 450 simulated trajectories, policy regression, a 6-second real-world rollout, and a 10-minute system identification procedure, totaling 21.3 minutes on a single AMD 3970 CPU with 128 GB RAM (Collins et al., 2021).

5. Empirical Performance

The algorithm's quantitative impact includes:

• Sim-to-real error (Euclidean end-effector path difference): Reduced from 129.8 m to 47.8 m (a 63% reduction) after the first iteration.
• Cumulative Error Improvement: The second iteration reduced simulation error to approximately 60 m, versus 150.7 m for the prior (non-updated) model—representing a further ∼60% improvement.
• Policy Reliability: Real-time policy rollout led to more consistent and rapid convergence (within 1 s) on goal positions after each iteration.
• Resource Efficiency: In contrast to domain randomization and SimOpt-style methods demanding extensive GPU resources and tuning, RealityGrad required only a single desktop-class CPU per iteration.

6. Scalability, Limitations, and Prospective Extensions

• Scalability: The primary computational bottlenecks are system-ID optimization and trajectory generation, both suitable for parallelization within simulation. The adoption of GPU-accelerated autodiff and more efficient integrators could, in principle, decrease wall-clock time per iteration below 10 minutes.
• Limitations: RealityGrad currently depends on differentiable simulators which encounter challenges handling hard contacts, discontinuities, and nontrivial collision models. Memory constraints restrict the real-to-sim identification window to short time horizons ($(x_t^k, x_T^{*,k}) \mapsto u_t^k$1s).
• Potential Extensions: Integration of differentiable contact models, larger scene graphs, longer real-ID horizons (through checkpointing or forward-mode autodiff), incorporation of differentiable rendering for vision-based control, and co-optimization of morphology, sensor noise models, or control-rate scheduling.

7. Conclusion and Significance

RealityGrad introduces an efficient, gradient-based iterative framework for online sim-to-real transfer. Its use of differentiable physics for both trajectory optimization and system identification enables rapid and scalable reduction of the reality gap—both for policy improvement and simulation fidelity—on commodity computing hardware. By serving as a template for leveraging precise gradient information in robotic learning loops, RealityGrad demonstrates efficacy in sample-limited regimes and offers a pathway for extending sim-to-real techniques to increasingly complex robotic domains (Collins et al., 2021).