---
title: Zero-Shot Sim-to-Real Transfer
url: https://www.emergentmind.com/topics/zero-shot-sim-to-real-strategy
type: topic
---

# Zero-Shot Sim-to-Real Transfer

Zero-shot sim-to-real strategy denotes a family of transfer procedures in which a model trained entirely, or predominantly, in simulation is deployed on physical hardware without task-level fine-tuning, additional real-world reinforcement learning, or deployment-time learned actuator models. The persistent obstacle is the “reality gap”: discrepancies between simulated and physical actuation, sensing, contact, appearance, and environment dynamics. In "Actuator Reality Shaping for Zero-Shot Sim-to-Real Robot Learning" [2607.02205], this problem is recast as actuator-interface design: instead of modifying the simulator to match the real world, the method shapes the closed-loop behavior of physical actuators to match the idealized second-order reference dynamics used in simulation. Across adjacent work, zero-shot transfer is also pursued through geometry-based latent representations, photorealistic or structure-guided visual adaptation, domain randomization, adversarial perturbation, residual correction, and hierarchical local control [2007.13960], [2503.02198], [2606.18953].

## 1. Problem setting and sources of the reality gap

A zero-shot sim-to-real pipeline is typically defined by a fixed deployment constraint: the policy, perception stack, or controller is trained in simulation and then used directly on real hardware. What differs across the literature is the mechanism chosen to make the simulation-trained artifact valid in reality. In actuator-dominated systems, the dominant gap can be the discrepancy between the ideal actuator dynamics assumed during policy training and the nonlinear, hardware-dependent behavior of physical motors. In contact-rich dexterous manipulation, the gap additionally includes tactile sensing, torque estimation, backlash, torque-speed saturation, and drive efficiency. In visual servoing and perception tasks, the gap can be dominated by rendering fidelity, illumination, camera calibration, sensor noise, and occlusion. Soft robots add high degrees of freedom and complex deformation behaviors; sonar and aerial systems add modality-specific artifacts such as terrain clutter, gate visibility loss, and motion blur [2607.02205], [2601.02778], [2303.04307], [2310.01667], [2503.02198].

This diversity makes “zero-shot sim-to-real strategy” a category rather than a single algorithm. Some strategies reduce the gap by modifying simulation; some standardize the real interface; some choose a representation that is intrinsically more invariant across domains; and some make the learned policy robust to a family of mismatches. A plausible implication is that the choice of zero-shot strategy is most effective when it targets the dominant mismatch channel rather than treating “sim-to-real” as a purely visual problem.

## 2. Actuator reality shaping as a control-theoretic strategy

The actuator reality shaping formulation begins from the actuator model used during policy training. Each revolute joint is modeled in simulation as a second-order under-damped spring–mass–damper,
$$
I_{\mathrm{sim}}\ddot{\theta}(t)=K_p(\bar{\theta}(t)-\theta(t))-D\dot{\theta}(t)+f_{\mathrm{ext}}(t),
$$
with reference transfer function
$$
G_{\mathrm{ref}}(s)\coloneqq \frac{\theta(s)}{\bar{\theta}(s)}
=\frac{K_p}{I_{\mathrm{sim}}s^2+Ds+K_p}
=\frac{\omega_n^2}{s^2+2\zeta\omega_n s+\omega_n^2},
$$
where $\omega_n=\sqrt{K_p/I_{\mathrm{sim}}}$ and $\zeta=D/(2\sqrt{I_{\mathrm{sim}}K_p})$. Policies are optimized assuming every joint behaves exactly like $G_{\mathrm{ref}}(s)$ [2607.02205].

On hardware, the motor and gearbox are modeled as
$$
P_{\mathrm{real}}(s)=\frac{1}{J_{\mathrm{real}}s^2+d_{\mathrm{real}}s},
$$
with Coulomb friction, saturation and cogging entering as a lumped disturbance $\tau_d$. The method defines the multiplicative plant mismatch
$$
\Delta P(s)=\frac{P_{\mathrm{real}}(s)}{P_{\mathrm{sim}}(s)}-1,
\qquad
P_{\mathrm{sim}}(s)=\frac{1}{I_{\mathrm{sim}}s^2+Ds},
$$
and aims to make $\theta_{\mathrm{real}}(s)\approx \theta_{\mathrm{ref}}(s)=G_{\mathrm{ref}}(s)\bar{\theta}(s)$ despite $\Delta P(s)$ and $\tau_d$.

The implementation is a two-degree-of-freedom feedforward–feedback controller. In compact form,
$$
u(s)=\frac{F(s)}{\hat{P}(s)}\bar{\theta}(s)+K(s)e(s),\qquad
e(s)=F(s)\bar{\theta}(s)-\theta(s).
$$
Standard 2-DoF analysis gives
$$
\theta(s)=\Bigl[1+\frac{\Delta P(s)}{1+P(s)K(s)}\Bigr]F(s)\bar{\theta}(s)
+\Bigl[\frac{P(s)}{1+P(s)K(s)}\Bigr]\tau_d(s).
$$
By choosing $K(s)$ sufficiently high-gain, the term $\Delta P/(1+PK)\to 0$ and disturbance sensitivity $P/(1+PK)$ is small, so $\theta\approx F\bar{\theta}$ [2607.02205].

The feedforward component is constructed from an inner velocity loop reference model
$$
F_2(s)=\frac{\alpha}{s+\alpha}
$$
and an outer position loop reference model
$$
F_1(s)=\frac{K_p}{I_{\mathrm{sim}}s^2+Ds+K_p}=G_{\mathrm{ref}}(s),
$$
so that
$$
F(s)=F_1(s)\cdot F_2(s).
$$
Each loop uses a PID,
$$
K(s)=k_p+\frac{k_i}{s}+k_ds,
$$
with $k_p$ damping transient error, $k_i$ enforcing zero steady-state error against constant $\tau_d$, and $k_d$ adding phase-lead and attenuating high-frequency noise. A disturbance observer with cutoff $\alpha_{\mathrm{dob}}$ further estimates
$$
\hat{\tau}_d(s)=\frac{\alpha_{\mathrm{dob}}}{s+\alpha_{\mathrm{dob}}}\,[\tau_{\mathrm{cmd}}(s)-(Js+d)\dot{\theta}(s)],
$$
which is then subtracted from the feedforward+feedback command.

The result is a standardized actuator interface for reinforcement learning. All tuning—picking $I_{\mathrm{sim}},D,K_p$, plus the 2-DoF PID gains and $\alpha,\alpha_{\mathrm{dob}}$—happens once, offline, from a one-shot identification of each motor’s $J$, $d$ and $K_{\tau i}$. No task-level rollouts are required for controller fitting. On a Dynamixel YM070 with a 99∶1 cycloidal gearbox, using a 20 deg-amplitude sine with 2.5 s period over 5 trials, 2-DoF + DOB reduced $|\Delta\theta|$ by 96–94% and $|\Delta\dot{\theta}|$ by 99% versus baselines. On a 7-DOF arm reaching task, 2-DoF + DOB achieved a 96.5% reduction in $\|\Delta p\|$ over Cascade, halved error versus simple PD, and 72% better than ASAP. The same drivers were then used, without any additional tuning, on a 31-DOF wheeled-legged robot climbing an unknown slope and a 22-DOF humanoid walking forward [2607.02205].

This strategy is distinct from approaches that model non-ideal actuation inside simulation. In dexterous grasping, actuator dynamics modeling is implemented by randomizing backlash, torque-speed saturation, and global drive efficiency [2601.02778]. In soft continuum visual servoing, transfer is obtained by decoupling kinematics from mechanical particulars through an RL kinematic controller and a local controller for actuation refinement [2504.16916]. The actuator reality shaping position is narrower and more explicit: it standardizes the physical joint so that the policy sees the same transfer function in sim and real.

## 3. Perception-side alignment and representation choice

A second major strategy class avoids direct pixel-level alignment and instead chooses a representation whose invariants are more stable across simulation and reality. KOVIS learns a keypoint representation from images using an autoencoder whose decoder reconstructs only geometric outputs—namely a depth buffer and a semantic mask—so that the latent must encode shape, not texture. The visual servoing network is then trained on stereo keypoint maps, with domain randomization and adversarial examples, and transfers zero-shot to grasping, peg-in-hole insertion with 4 mm clearance, and M13 screw insertion [2007.13960].

PencilNet pushes this logic further by applying a “pencil filter” that accentuates edges while normalizing away illumination. The network then predicts a unified pose tuple $\{x,y,d,\theta,c\}$ from filtered single-channel images. The reported rationale is that, by working in the “edge-only” domain, the method achieves invariance to shadows, highlights, over-/under-exposure, and motion blur; no real-world images or labels are used for training [2207.14131].

Object-centric residual RL adopts yet another representation: the residual policy observes only task-relevant object poses, robot proprioception, and the current VLA action,
$$
s_t=[\,s_t^{obj};\,s_t^{prop};\,a_t^{base}\,].
$$
This compact observation space is intended to avoid both the visual rendering gap and the privileged-state gap. The policy is trained in simulation with hierarchical uniform pose noise and dropout, then deployed zero-shot on a real Franka Research 3 robot, improving the success rate from 42% to 76% zero-shot across five manipulation tasks [2606.18953].

In soft robot proprioceptive sensing, the representation is a 1×256×256 binary image from an internal camera, decoded into a 3,174-point cloud. The model is trained on 6919 simulated frames and evaluated with real internal camera images. The reported zero-shot performance is a Chamfer distance of 8.85 mm and tip position error of 10.12 mm for Pattern 2, even with external perturbation, on a pneumatic soft robot with a length of 100.0 mm [2303.04307].

A broader perception-side trend is to alter the synthetic image distribution itself. ALDM-Grasping uses an adversarially supervised, layout-to-image diffusion model to generate photorealistic renderings from segmentation layouts and text prompts, yielding 75% grasp success under plain backgrounds and 65% in complex scenarios [2403.11459]. SICGAN translates virtual observations into “real-synthetic” images for hybrid-domain DRL training, after which the policy is deployed directly on real industrial manipulators [2601.16677]. FalconGym constructs a photorealistic NeRF environment from track-specific real imagery, then trains a Neural Pose Estimator plus Kalman-filter fusion and a self-attention-based multi-modal controller entirely in simulation [2503.02198]. Ontology-Guided Diffusion treats realism as a structured ontology of traits, using a GNN embedding and a symbolic planner to condition an instruction-guided diffusion model [2603.18719].

These examples suggest that representation choice is often as important as simulator fidelity. Geometry-only keypoints, edge-domain filtering, object poses, and semantically structured visual edits each attempt to remove nuisance variation before policy learning.

## 4. Robustification in simulation: randomization, adversaries, and pessimism

A third family of zero-shot strategies keeps the simulator imperfect but trains the policy to tolerate a family of imperfections. In fruit harvesting, each episode randomizes lighting, camera extrinsics, fruit positions, the number of distractors, and, optionally, surface friction coefficients, strawberry mass, and stem stiffness. The policy is trained in a custom MuJoCo environment with dormant ratio minimization and then deployed zero-shot on a Franka Panda with a 20 Hz policy loop and 1 kHz low-level impedance [2505.08458].

In dexterous force-based grasping, simulation randomizes object mass, size, and friction, restitution, damping, PD gains, encoder noise, and 5–10 mm Gaussian noise on the camera-estimated object height, while the actuator model includes randomized backlash, torque-speed saturation, and efficiency. The stated purpose is to force the policy to adapt to a broad spectrum of DC-motor behaviors so that no single model mismatch collapses transfer [2601.02778].

Adversarial perturbation is a stronger variant of this idea. KOVIS augments each mini-batch with one of FGSM, iterative FGSM or least-likely FGSM, with $\epsilon$ sampled uniformly in $[0.01,0.1]$ and 1–5 attack iterations, to encourage the encoder to ignore subtle pixel-level artifacts [2007.13960]. In autonomous driving, zero-shot transfer is framed as a zero-sum game in which an adversarial policy perturbs both state and action channels during training. The adversarially trained policy transfers zero-shot to a 1∶25 scale testbed and improves robustness relative to Gaussian noise injection [1903.05252].

SPiDR formalizes robustification in a constrained MDP by replacing the cost $c(s,a)$ with a penalized cost
$$
\tilde{c}(s,a)=c(s,a)+\kappa\cdot U(s,a),
$$
where $U(s,a)$ is an ensemble-disagreement proxy for worst-case model mismatch. The method then solves a penalized CMDP and provides an informal theorem stating that, under mild Lipschitz and finite-discrepancy assumptions, the resulting policy satisfies the real-world safety constraint $C_{p^\star}(\tilde{\pi})\le d$ [2509.18648].

For RGB deformable manipulation, SimWeaver grounds its photometric augmentation in measured camera behavior. On the D435i, it reports white-pixel bias $\Delta(R-G)\approx 39/255$, channel drift $\sigma\approx 12/255$ within 5 s, and residual brightness offset ~7–10 %. It then samples brightness, contrast, saturation, hue jitter, sharpness, and affine perturbations per frame. The ablation is unusually categorical: disabling photometric augmentation collapses real success to 0 % on all five tasks [2606.15338].

The common logic is not exact modeling but coverage. A plausible implication is that zero-shot robustness improves when the randomized variables correspond to actual deployment failure modes rather than generic stochasticity.

## 5. Deployment architectures and empirical operating regimes

Zero-shot strategies are frequently embedded in hierarchical deployment stacks rather than monolithic policies. In actuator reality shaping, the reinforcement learning policy outputs joint goal angles $\bar{\theta}$ and a transparent 2-DoF + DOB driver layer enforces the desired transfer function [2607.02205]. In soft continuum visual servoing, the RL policy issues kinematic commands in $(\kappa,\tau)$, while a local proportional controller refines pneumatic pressures until the commanded curvature and torsion are reached [2504.16916]. In object-centric residual RL, a frozen VLA produces the base action chunk and a TD3-trained residual policy supplies $\delta_t$, with final action $a_t=a_t^{base}\oplus\delta_t$ [2606.18953]. In garment reversal, the task is decomposed into Drag, Fling, and Insert & Pull, each parameterized by a small number of 2D keypoints in the overhead depth image [2509.15953].

Representative real-world outcomes illustrate the breadth of operating regimes rather than a single benchmark tradition.

| Domain | Strategy | Reported real-world result |
|---|---|---|
| 7-DOF arm reaching | Actuator reality shaping | 96.5% reduction in $\|\Delta p\|$ over Cascade |
| Dexterous in-hand rotation | Tactile + torque + actuation modeling | 25.1 consecutive 90° rotations before failure |
| Soft continuum arm visual servoing | RL kinematic controller + local controller | 67% success zero-shot |
| Quadrotor gate crossing | FalconGym + NPE + Kalman + multi-modal controller | 95.8% success rate, average error of just 10 cm |
| Deformable manipulation | SimWeaver | 91.30 average real-world success |
| VLA enhancement on FR3 | Object-centric residual RL | 42% to 76% zero-shot |

The diversity of these results matters. Some methods target low-level tracking error, some target task success, some target robustness under clutter or distribution shift, and some target safety constraints. This makes direct cross-paper comparison difficult. Nevertheless, the empirical record shows that zero-shot transfer is no longer confined to rigid, low-contact settings: the cited literature includes grasp force tracking, in-hand rotation, soft-arm servoing, fruit harvesting from dense clusters, wheeled-legged driving over a slope, humanoid walking, quadrotor gate crossing, and deformable-object manipulation [2601.02778], [2505.08458], [2607.02205], [2503.02198], [2606.15338].

## 6. Limitations, misconceptions, and extensions

A recurrent misconception is that “zero-shot” implies the absence of all real-world preparation. The cited literature does not support that interpretation. Actuator reality shaping requires a one-shot identification of each motor’s $J$, $d$ and $K_{\tau i}$ [2607.02205]. Dexterous grasping performs a one-time calibration per joint to map motor current to normalized torque [2601.02778]. Pneumatic soft robot proprioception uses a single straight-pose calibration image and one-time CMA-ES optimization of 32 renderer parameters [2303.04307]. FalconGym is built from real camera traversals and Vicon® poses for each track [2503.02198]. SICGAN is trained from unpaired RGB images collected from the real robot workspace without the target [2601.16677]. Zero-shot, in these examples, means no task-level fine-tuning on the target policy after simulation training, not the absence of offline calibration or real-world data collection.

Method-specific limitations are equally explicit. Actuator reality shaping assumes constant joint inertia $I_{\mathrm{sim}}$, which is stated to be valid for high-gear-ratio actuators but less so for quasi direct-drive where link-dependent inertia matters; the paper suggests model-reference adaptive control or configuration-dependent shaping as extensions [2607.02205]. The soft robot proprioception pipeline retains a residual sim-to-real gap in external-perturbation directions not explicitly modeled [2303.04307]. In FalconGym, failures in the U-turn track were traced to minor visual artifacts in that particular NeRF scene, with additional contributions from aerodynamic mismatch and offboard communication delays of ≈39 ms [2503.02198]. In SICGAN, the blue cube sometimes fails due to no blue-channel activation during training [2601.16677]. In soft continuum visual servoing, remaining failure modes arise from near-limit actuation, unmodeled payload dynamics, and lack of depth cues [2504.16916].

A broader controversy concerns where alignment should occur. One line of work increases simulator realism through NeRF rendering, diffusion-based image translation, Gaussian Splatting, or ontology-guided edits [2503.02198], [2403.11459], [2510.10637], [2603.18719]. Another line chooses invariant representations, such as keypoints, edge maps, or object poses [2007.13960], [2207.14131], [2606.18953]. Actuator reality shaping locates alignment in the real actuator interface itself [2607.02205]. This suggests that zero-shot sim-to-real is best understood not as a contest between “better simulation” and “better robustness,” but as a design decision about which interface—dynamics, observation, action, or task decomposition—can be most effectively standardized.

Source: https://www.emergentmind.com/topics/zero-shot-sim-to-real-strategy