Real-to-Sim-to-Real Shared Autonomy

• Real-to-Sim-to-Real shared autonomy is a framework where real-world data seeds simulation to bridge the gap between assistive control training and physical deployment.
• It leverages techniques like behavior cloning, residual reinforcement learning, and diffusion-based methods to blend human inputs with robotic actions.
• Empirical results demonstrate significant improvements in task success, enhanced generalization, and reduced human intervention across diverse robotic applications.

Real-to-Sim-to-Real shared autonomy denotes a class of human-in-the-loop robotic systems in which real interaction data are first used to construct simulator-aligned representations of tasks, users, or environments; simulation is then used to scale data, train assistive policies, or evaluate arbitration mechanisms; and the resulting controller is redeployed on physical hardware while retaining a nontrivial human role in task specification, motion generation, or supervision. Across recent work, this paradigm appears in manipulation, assistive arms, driving, prosthetics, and surgery, but the common structure is stable: real data narrow the action and observation gap, simulation supplies scalability and controllability, and the final deployed policy operates as an assistive layer rather than as a fully decoupled autonomous stack (Fang et al., 15 Mar 2025, Sha et al., 17 Mar 2026, Abou-Chakra et al., 4 Apr 2025, Fan et al., 15 May 2025).

1. Conceptual scope and problem setting

Within shared autonomy, human and robot jointly determine executed motion. Several formulations use an arbitration coefficient that blends a human command with an assistive command, as in

$$a^t = \beta^t a_{\mathcal{R}}^t + (1-\beta^t)a_{\mathcal{H}}^t,$$

or, in task-space telesurgery,

$$\Delta P_t^s = \tau \big[\alpha(t)\Delta P_t^h + (1-\alpha(t))\Delta P_t^r\big].$$

These formulations encode a spectrum between full teleoperation and full autonomy, with the robot intervening more strongly when confidence, task familiarity, or safety criteria support intervention (Jonnavittula et al., 2021, Jonnavittula et al., 2022, Zhang et al., 2022).

The Real-to-Sim-to-Real qualifier changes where the assistive policy comes from. Instead of training assistance solely from simulation and then attempting direct transfer, or relying only on expensive real-world data, recent systems use real trajectories, real sensor calibration, real scene structure, or real user behavior to seed the simulator. ReBot begins from real robot episodes and replays them in Isaac Sim to generate object-diverse but action-consistent robotic videos for finetuning VLA policies; Efficient and Reliable Teleoperation through Real-to-Sim-to-Real Shared Autonomy fits a human surrogate from less than five minutes of real teleoperation data and trains a residual copilot in simulation; Real-is-Sim continuously corrects a dynamic digital twin from real camera observations and lets policies act only in simulation while the real robot follows the simulated joints (Fang et al., 15 Mar 2025, Sha et al., 17 Mar 2026, Abou-Chakra et al., 4 Apr 2025).

A persistent motivation is the failure of purely offline generalist systems at last-mile deployment. ReBot explicitly describes this as the “last-mile deployment challenge in robot manipulation,” and SAPS targets the brittleness of frozen VLA policies under out-of-distribution spatial and semantic perturbations (Fang et al., 15 Mar 2025, Zhou et al., 14 Jun 2026). In assistive settings, SARI and its precursor argue that repeated interactions, rather than a predefined goal set, should be the substrate for learning assistance from scratch (Jonnavittula et al., 2022, Jonnavittula et al., 2021). This suggests that Real-to-Sim-to-Real shared autonomy is not a single algorithmic family but an architectural principle: use reality to define the manifold of legitimate tasks and interactions, use simulation to enlarge and stress-test that manifold, and return to reality with an assistive controller whose intervention is conditioned on human intent and uncertainty.

2. Real-to-Sim representations: trajectories, digital twins, and semantic abstractions

A central design question is what exactly gets transferred from reality into simulation. One approach is trajectory-grounded replay. ReBot formalizes a real robot dataset as

$$\mathcal{D} = \{\boldsymbol{\tau}_i\}_{i=1}^M,\qquad \boldsymbol{\tau}_i = \{(\mathbf{o}_t,\mathbf{a}_t,\mathcal{L})\}_{t=1}^T,$$

and constructs synthetic episodes

$$\boldsymbol{\tau}'_j = \{(\mathbf{o}'_t,\mathbf{a}_t,\mathcal{L}')\}_{t=1}^T$$

by reusing the original action sequence unchanged. Real-to-Sim is implemented by pose-aligning digital twins of the robot and cameras, estimating table height from depth with GroundingDINO, replaying the action sequence to infer grasp and release times, inserting new objects at the simulated grasp site, and filtering invalid replays through the Cartesian object–gripper distance during the manipulation window (Fang et al., 15 Mar 2025). In shared-autonomy terms, a small set of human demonstrations becomes a combinatorial library of semantically varied but kinematically consistent assistive exemplars.

A second approach is robot-isomorphic capture of real interaction. ExoGS uses a self-designed passive exoskeleton, AirExo-3, whose kinematic parameters, joint limits, workspace, and gripper range match the target robot. Joint trajectories are therefore embodiment-consistent by construction, while synchronized multi-view RGB-D is used to reconstruct robot, objects, and environment as editable 3D Gaussian Splatting assets. Object poses are tracked over time with FoundationPose and fused into a global pose sequence, while robot link poses are recovered from the URDF and forward kinematics. The result is a 4D scene representation in which robot and object assets are replayed by time-indexed rigid transformations rather than by framewise video synthesis (Wang et al., 26 Jan 2026). This suggests a particularly strong Real-to-Sim substrate for shared autonomy because the human demonstration already lives in robot coordinates.

A third approach is continuous digital-twin synchronization. Real-is-Sim maintains a dynamic digital twin based on the Embodied Gaussian simulator, running a 60 Hz physics loop and a 30 Hz visual correction loop. Policies are trained on simulator-derived representations and always act on the simulated robot; the real robot tracks the simulated joint positions, while RGB observations from multiple real cameras produce a photometric loss whose gradient is interpreted as fictitious visual forces that pull the simulated state back toward reality (Abou-Chakra et al., 4 Apr 2025). For shared autonomy, this arrangement is structurally attractive because both human and policy can be interpreted as acting on the same simulation substrate, with sim-to-real transfer delegated to synchronization rather than to the policy.

A fourth variant appears in deformable surgery. Real-to-Sim registration of soft tissue with position-based dynamics reconstructs a surface point cloud from stereo endoscopy, builds an initial volumetric tissue model, and then minimizes an SDF-based registration objective by injecting a registration gradient as a soft PBD constraint. The simulator thereby remains aligned with real deforming tissue even in partially occluded regions (Liu et al., 2020). Although the paper stops short of a full shared-autonomy controller, it establishes the data-to-model loop needed for predictive safety overlays during human operation.

At a more abstract level, TIAMAT argues that Real-to-Sim need not mean only photorealistic digital twinning. It emphasizes refinement of semantic anchors, scene graphs, knowledge graphs, logic, and low(er)-fidelity simulators from real-world experience, and then iterating abstract-to-real transfer through a feedback loop (Noorani et al., 14 Mar 2025). This broadens the ontology of Real-to-Sim representations: the transferred object may be a calibrated geometry, a user trajectory distribution, or an abstract task semantics.

3. Shared-control formulations and arbitration mechanisms

The shared-autonomy layer determines how human and robot contributions are combined after the simulator has been used to shape an assistive model. Repeated-interaction methods such as SARI represent the current task by a latent variable

$z \sim \mathcal{E}(\cdot \mid \tau^i),$

and decode an assistive action

$a_{\mathcal{R}} = \pi_{\mathcal{R}}(s,z).$

A discriminator $\mathcal{C}$ estimates whether the current partial behavior resembles previously seen tasks, and its confidence is mapped to the arbitration parameter $\beta$. The robot intervenes when confidence is high and returns control when the trajectory is off-distribution (Jonnavittula et al., 2021, Jonnavittula et al., 2022). This formulation is especially compatible with Real-to-Sim-to-Real because simulation can be used to generate “seen-like” and “unseen-like” trajectories for calibrating $\mathcal{C}$, while real repeated interactions define the latent task space.

Action-level blending appears in several deployment-oriented systems. SAPS blends continuous VLA actions and teleoperation commands in the first six action dimensions via

$$\mathbf{a}^{(1:6)}_{\text{blended}} = \alpha\,\mathbf{a}^{(1:6)}_{\text{VLA}} + (1-\alpha)\,\mathbf{a}^{(1:6)}_{\text{expert}},$$

while using

$$\Delta P_t^s = \tau \big[\alpha(t)\Delta P_t^h + (1-\alpha(t))\Delta P_t^r\big].$$0

for the gripper. Its most distinctive arbitration rule uses the cosine similarity between human and policy actions, mapped through a logistic with $$\Delta P_t^s = \tau \big[\alpha(t)\Delta P_t^h + (1-\alpha(t))\Delta P_t^r\big].$$1, to define $$\Delta P_t^s = \tau \big[\alpha(t)\Delta P_t^h + (1-\alpha(t))\Delta P_t^r\big].$$2 continuously from disagreement to agreement (Zhou et al., 14 Jun 2026). This is a model-agnostic, training-free shared-autonomy mechanism whose sim-to-real consistency derives from a stable action interface rather than from retraining.

Diffusion-based shared autonomy replaces convex blending with distributional editing. “To the Noise and Back” defines a partial forward diffusion ratio

$$\Delta P_t^s = \tau \big[\alpha(t)\Delta P_t^h + (1-\alpha(t))\Delta P_t^r\big].$$3

forwards the user action toward noise for $$\Delta P_t^s = \tau \big[\alpha(t)\Delta P_t^h + (1-\alpha(t))\Delta P_t^r\big].$$4 steps, and then reverse-diffuses under a state-conditioned expert action model $$\Delta P_t^s = \tau \big[\alpha(t)\Delta P_t^h + (1-\alpha(t))\Delta P_t^r\big].$$5. Small $$\Delta P_t^s = \tau \big[\alpha(t)\Delta P_t^h + (1-\alpha(t))\Delta P_t^r\big].$$6 preserves user authority; large $$\Delta P_t^s = \tau \big[\alpha(t)\Delta P_t^h + (1-\alpha(t))\Delta P_t^r\big].$$7 moves behavior toward expert-like actions (Yoneda et al., 2023). Diffusion-SAFE uses the same intuition for driving handover: a diffusion evaluator predicts human behavior, a diffusion copilot generates safe expert trajectories, and a gradual handover begins once an NLL-based risk metric exceeds a threshold; the paper selects $$\Delta P_t^s = \tau \big[\alpha(t)\Delta P_t^h + (1-\alpha(t))\Delta P_t^r\big].$$8 as a trade-off between unsafe rate and velocity variance during handover (Fan et al., 15 May 2025).

Context-conditioned arbitration is explicit in surgical shared control. A CNN predicts whether the current phase is “Move to next target,” “Bimanual operation,” or “Local operation,” and a thresholded probability-change rule determines whether the robot should be in high-autonomy mode, manual mode, or a soft transition between them (Zhang et al., 2022). This is a distinct arbitration regime from latent task confidence or action agreement, but it solves the same problem: assistance should depend on where the human is within a task, not only on state error.

A recurrent misconception is that shared autonomy necessarily presupposes a discrete goal set known a priori. The latent-space repeated-interaction literature, the diffusion formulations, and action-level VLA steering all violate that assumption in different ways (Jonnavittula et al., 2022, Yoneda et al., 2023, Zhou et al., 14 Jun 2026). Another misconception is that assistance must be realized as hard takeovers; the reviewed systems instead span continuous blending, confidence gating, partial diffusion, and phase-specific role adaptation.

4. Learning in simulation: imitation, reinforcement learning, and generative policy shaping

The simulator in Real-to-Sim-to-Real shared autonomy is not merely a testbed; it is the main site where assistive behavior is shaped. A basic route is behavior cloning on combined real and synthetic data. ReBot finetunes Octo and OpenVLA on $$\Delta P_t^s = \tau \big[\alpha(t)\Delta P_t^h + (1-\alpha(t))\Delta P_t^r\big].$$9 using the standard BC objective

$$\mathcal{D} = \{\boldsymbol{\tau}_i\}_{i=1}^M,\qquad \boldsymbol{\tau}_i = \{(\mathbf{o}_t,\mathbf{a}_t,\mathcal{L})\}_{t=1}^T,$$0

with synthetic episodes preserving the original action timing and replacing only objects, backgrounds, and instructions (Fang et al., 15 Mar 2025). Although the paper is framed as VLA adaptation rather than shared autonomy, the trained VLA functions as an assistive low-level controller conditioned on human language instructions.

Simulation-driven imitation learning for prosthetics pushes the same logic into a different modality. A wrist-mounted camera, proprioception, and procedurally generated reach–grasp–lift sequences form a large synthetic demonstration set, and sequence-model imitation learners such as ACT and VTM-VAE are trained by pure behavior cloning. The learned sim-to-real policy reaches over $$\mathcal{D} = \{\boldsymbol{\tau}_i\}_{i=1}^M,\qquad \boldsymbol{\tau}_i = \{(\mathbf{o}_t,\mathbf{a}_t,\mathcal{L})\}_{t=1}^T,$$1 grasp success in three realistic settings, with stronger generalization than a real-data baseline trained in a single white-table scene (Shi et al., 5 Jun 2026). For shared autonomy, the human supplies wrist motion and lift timing, while the policy handles finger closure and maintenance of grasp; the simulation stage therefore learns the low-level completion policy that turns coarse human intent into successful manipulation.

Residual reinforcement learning offers a different way to exploit simulation. Efficient and Reliable Teleoperation through Real-to-Sim-to-Real Shared Autonomy builds a k-nearest-neighbor human surrogate from less than five minutes of real teleoperation data and trains a residual copilot in Isaac Lab/Factory with PPO. The runtime command is

$$\mathcal{D} = \{\boldsymbol{\tau}_i\}_{i=1}^M,\qquad \boldsymbol{\tau}_i = \{(\mathbf{o}_t,\mathbf{a}_t,\mathcal{L})\}_{t=1}^T,$$2

where the residual corrects the human command in task space. Because the surrogate is non-parametric and remains close to the demonstration manifold, RL training is more stable than with a behavioral-cloning pilot under the same small-data regime (Sha et al., 17 Mar 2026). This is an explicit Real-to-Sim-to-Real shared-autonomy instantiation: the simulator learns not only the robot-side policy but also a user model derived from reality.

Model-free deep RL for shared autonomy predates many of these systems but already exhibits the same structural decomposition. The DQN-based copilot consumes the concatenated observation $$\mathcal{D} = \{\boldsymbol{\tau}_i\}_{i=1}^M,\qquad \boldsymbol{\tau}_i = \{(\mathbf{o}_t,\mathbf{a}_t,\mathcal{L})\}_{t=1}^T,$$3, where $$\mathcal{D} = \{\boldsymbol{\tau}_i\}_{i=1}^M,\qquad \boldsymbol{\tau}_i = \{(\mathbf{o}_t,\mathbf{a}_t,\mathcal{L})\}_{t=1}^T,$$4 is either raw user action or an inferred goal estimate, and then chooses the feasible action closest to the human suggestion according to a Q-threshold parameterized by $$\mathcal{D} = \{\boldsymbol{\tau}_i\}_{i=1}^M,\qquad \boldsymbol{\tau}_i = \{(\mathbf{o}_t,\mathbf{a}_t,\mathcal{L})\}_{t=1}^T,$$5 (Reddy et al., 2018). This formulation is especially relevant when simulation can provide a reward but not a trustworthy goal model. It also illustrates an early Real-to-Sim-to-Real logic: pretrain the copilot in a simulator, then fine-tune with real humans.

Generative behavior modeling makes the simulator responsible for learning distributions rather than point policies. The diffusion shared-autonomy formulation learns $$\mathcal{D} = \{\boldsymbol{\tau}_i\}_{i=1}^M,\qquad \boldsymbol{\tau}_i = \{(\mathbf{o}_t,\mathbf{a}_t,\mathcal{L})\}_{t=1}^T,$$6 from successful demonstrations and uses partial diffusion at test time to project human actions back toward the expert manifold, all without rewards, a user model, or a known goal set (Yoneda et al., 2023). Diffusion-SAFE extends this to closed-loop driving by pairing a human-behavior evaluator with a safe copilot trajectory generator (Fan et al., 15 May 2025). These methods are attractive in Real-to-Sim-to-Real settings because the heavy learning can happen on synthetic or demonstration-rich data in simulation, while the deployed controller remains a lightweight test-time inference procedure.

5. Empirical evidence across domains

Manipulation benchmarks provide some of the clearest quantitative evidence. ReBot reports that, in SimplerEnv with the WidowX robot, it improved the in-domain performance of Octo by $$\mathcal{D} = \{\boldsymbol{\tau}_i\}_{i=1}^M,\qquad \boldsymbol{\tau}_i = \{(\mathbf{o}_t,\mathbf{a}_t,\mathcal{L})\}_{t=1}^T,$$7 and OpenVLA by $$\mathcal{D} = \{\boldsymbol{\tau}_i\}_{i=1}^M,\qquad \boldsymbol{\tau}_i = \{(\mathbf{o}_t,\mathbf{a}_t,\mathcal{L})\}_{t=1}^T,$$8, and out-of-domain generalization by $$\mathcal{D} = \{\boldsymbol{\tau}_i\}_{i=1}^M,\qquad \boldsymbol{\tau}_i = \{(\mathbf{o}_t,\mathbf{a}_t,\mathcal{L})\}_{t=1}^T,$$9 and $$\boldsymbol{\tau}'_j = \{(\mathbf{o}'_t,\mathbf{a}_t,\mathcal{L}')\}_{t=1}^T$$0, respectively. In real-world Franka evaluation, it increased the success rates of Octo by $$\boldsymbol{\tau}'_j = \{(\mathbf{o}'_t,\mathbf{a}_t,\mathcal{L}')\}_{t=1}^T$$1 and OpenVLA by $$\boldsymbol{\tau}'_j = \{(\mathbf{o}'_t,\mathbf{a}_t,\mathcal{L}')\}_{t=1}^T$$2 (Fang et al., 15 Mar 2025). Since these gains come from real-to-sim replay plus sim-to-real video synthesis rather than from architectural changes, they show that last-mile shared-control competence can be altered substantially through the data pathway alone.

Action-level policy steering yields similarly strong deployment gains. SAPS improves task success over autonomous execution by up to $$\boldsymbol{\tau}'_j = \{(\mathbf{o}'_t,\mathbf{a}_t,\mathcal{L}')\}_{t=1}^T$$3 in both simulation and the real world, while drastically reducing human intervention relative to pure teleoperation and achieving faster task completion times than both autonomous execution and pure teleoperation (Zhou et al., 14 Jun 2026). On real Franka tasks, $$\boldsymbol{\tau}'_j = \{(\mathbf{o}'_t,\mathbf{a}_t,\mathcal{L}')\}_{t=1}^T$$4 reaches $$\boldsymbol{\tau}'_j = \{(\mathbf{o}'_t,\mathbf{a}_t,\mathcal{L}')\}_{t=1}^T$$5, $$\boldsymbol{\tau}'_j = \{(\mathbf{o}'_t,\mathbf{a}_t,\mathcal{L}')\}_{t=1}^T$$6, and $$\boldsymbol{\tau}'_j = \{(\mathbf{o}'_t,\mathbf{a}_t,\mathcal{L}')\}_{t=1}^T$$7 success on marker placement, drawer closing, and cabinet opening, whereas Cosine SAPS reaches $$\boldsymbol{\tau}'_j = \{(\mathbf{o}'_t,\mathbf{a}_t,\mathcal{L}')\}_{t=1}^T$$8, $$\boldsymbol{\tau}'_j = \{(\mathbf{o}'_t,\mathbf{a}_t,\mathcal{L}')\}_{t=1}^T$$9, and $z \sim \mathcal{E}(\cdot \mid \tau^i),$0, respectively (Zhou et al., 14 Jun 2026). The significance for Real-to-Sim-to-Real shared autonomy is not merely improved success; it is the fact that the same arbitration logic transfers from LIBERO/CALVIN-style simulators to real hardware without retraining.

Driving handover provides a different empirical profile. Diffusion-SAFE reports a $z \sim \mathcal{E}(\cdot \mid \tau^i),$1 successful handover rate while progressively correcting human actions and continuously sampling optimal robot actions (Fan et al., 15 May 2025). Its real system uses a ROS-based race car with a top-down pseudo-camera derived from a global track image and motion capture, closely mirroring the simulator’s state representation. This is a concrete case where real-to-sim alignment of observation structure is at least as important as dynamics fidelity.

Assistive and teleoperated manipulation studies emphasize human performance, not only policy performance. The residual copilot trained with a kNN human surrogate improves task success for novice operators and execution efficiency for experienced operators on nut threading, gear meshing, and peg insertion, and copilot-assisted teleoperation also produces higher-quality demonstrations for downstream imitation learning (Sha et al., 17 Mar 2026). ExoGS shows that robot-isomorphic exoskeleton collection is both more efficient and, for a hard contact-rich task, markedly higher quality than teleoperation: demonstration success is $z \sim \mathcal{E}(\cdot \mid \tau^i),$2 versus $z \sim \mathcal{E}(\cdot \mid \tau^i),$3 for Pick and Place, $z \sim \mathcal{E}(\cdot \mid \tau^i),$4 versus $z \sim \mathcal{E}(\cdot \mid \tau^i),$5 for Pick Place Close, and $z \sim \mathcal{E}(\cdot \mid \tau^i),$6 versus $z \sim \mathcal{E}(\cdot \mid \tau^i),$7 for Unscrew Bottle Cap (Wang et al., 26 Jan 2026). This suggests that the quality of the real side of the loop can dominate the eventual sim-trained assistive behavior.

In surgery, sim-to-real shared control on the dVRK reduces the path length of the remote controller by roughly $z \sim \mathcal{E}(\cdot \mid \tau^i),$8, reduces the total clutching number by about $z \sim \mathcal{E}(\cdot \mid \tau^i),$9, and decreases task completion time from about $a_{\mathcal{R}} = \pi_{\mathcal{R}}(s,z).$0 s to about $a_{\mathcal{R}} = \pi_{\mathcal{R}}(s,z).$1 s on peg transfer (Zhang et al., 2022). In prosthetics, the simulation-driven policy achieves over $a_{\mathcal{R}} = \pi_{\mathcal{R}}(s,z).$2 grasp success in three realistic settings (Shi et al., 5 Jun 2026). In repeated-interaction assistive arms, SARI matches classic Bayesian shared autonomy on known tasks and outperforms imitation-learning baselines on new tasks, while the 2021 repeated-interaction formulation reports superior performance on new tasks relative to imitation-learning baselines in both simulations and a user study (Jonnavittula et al., 2022, Jonnavittula et al., 2021). Across these heterogeneous domains, the pattern is consistent: a real-seeded simulation stage improves either success, intervention efficiency, or robustness to novelty, and often all three.

6. Limitations, debates, and future directions

A recurring limitation is calibration fragility. ReBot requires depth for table-height estimation, stable calibrated cameras, and a static environment; moving furniture or people would break alignment (Fang et al., 15 Mar 2025). Real-is-Sim depends on a dynamic digital twin whose correctness is bounded by the fidelity of the Embodied Gaussian simulator and by the ability of photometric corrections to pull the simulation back toward reality; tasks that diverge strongly from rigid-body assumptions remain problematic (Abou-Chakra et al., 4 Apr 2025). ExoGS is presently strongest for rigid objects and rigid interactions; deformable-object manipulation is explicitly identified as a limitation (Wang et al., 26 Jan 2026).

A second limitation concerns distribution shift between expert or simulated states and user-induced real states. Diffusion for shared autonomy explicitly notes that it does not address mismatch between the expert state distribution $a_{\mathcal{R}} = \pi_{\mathcal{R}}(s,z).$3 and the pilot-induced state distribution $a_{\mathcal{R}} = \pi_{\mathcal{R}}(s,z).$4, and it also lacks explicit goal conditioning, so high $a_{\mathcal{R}} = \pi_{\mathcal{R}}(s,z).$5 can wash out user intent when multiple goals share similar states (Yoneda et al., 2023). The repeated-interaction literature assumes relative stationarity in user behavior; co-adaptation is acknowledged but not modeled (Jonnavittula et al., 2021, Jonnavittula et al., 2022). Real-to-Sim-to-Real shared autonomy therefore inherits a two-sided shift problem: the world changes, and the human changes with the robot.

A third debate concerns simulator fidelity. A common view is that Real-to-Sim-to-Real should strive for ever more faithful digital twins. TIAMAT argues against that as a universal principle and instead emphasizes breadth of low(er)-fidelity simulations linked by shared semantics, plus refinement from real deployment (Noorani et al., 14 Mar 2025). This does not negate digital twins; rather, it reframes the question. For some problems, such as exoskeleton-grounded manipulation or synchronized joint-space twins, geometric fidelity is decisive. For others, especially mission-level or semantic shared autonomy, the transferable object may be a scene graph, an automaton, or a knowledge graph rather than a photorealistic scene.

A plausible implication is that future Real-to-Sim-to-Real shared autonomy systems will become hybrid across levels of abstraction. Real demonstrations may seed kinematic twins or 4D scene assets; simulation may train residual or generative assistive controllers; semantic layers may define goals, task phases, and safety constraints; and deployment-time arbitration may mix confidence, action agreement, uncertainty, and human preference rather than using a single scalar. The cited literature already points in this direction through adaptive role allocation in surgery, learned latent task spaces in repeated interaction, diffusion-based control authority knobs, residual copilot learning from tiny real datasets, and semantic refinement loops from TIAMAT (Zhang et al., 2022, Jonnavittula et al., 2022, Fan et al., 15 May 2025, Sha et al., 17 Mar 2026, Noorani et al., 14 Mar 2025).

The most consequential open problem is not simply transfer, but iterative transfer under human co-use. Several of the reviewed systems already contain the ingredients for closing the loop: SARI adds every completed interaction back into its dataset; ReBot can rerun its real-to-sim-to-real pipeline after new interventions; Real-is-Sim treats the simulator as the execution substrate; ExoGS can turn new real episodes into new 4D assets; and SAPS generates logged human corrections that could supervise learned arbitration (Jonnavittula et al., 2022, Fang et al., 15 Mar 2025, Abou-Chakra et al., 4 Apr 2025, Wang et al., 26 Jan 2026, Zhou et al., 14 Jun 2026). Real-to-Sim-to-Real shared autonomy is therefore best understood not as one transfer event but as a repeated estimation, synthesis, and redeployment cycle in which the human remains both operator and source of new structure.