Real2Sem2Real (R2Sem2R) Methods Overview

• Real2Sem2Real (R2Sem2R) methods are a family of robotics techniques that convert real-world observations into structured semantic, geometric, or physical representations, generating data or optimizing policies for real-robot deployment.
• These methods support scene reconstruction, object-centric representation transfer, and digital twin synchronization, among other applications, using techniques ranging from semantic field transfer to function-preserving geometric augmentation.
• R2Sem2R methods leverage intermediate semantic representations to bridge the sim-to-real gap, enabling robots to learn from human demonstrations and translate those demonstrations into actionable policies.

Real2Sem2Real (R2Sem2R) is a family of robotics methods that converts real-world observations, demonstrations, or scenes into structured semantic, geometric, physical, or simulation representations, uses those representations to generate data or optimize policies, and transfers the resulting behavior back to physical robots. The term is used most explicitly by “Real2Sem2Real: 3D Semantic Augmentations for Robot Learning from a Single Human RGB Demonstration” (Strong et al., 20 Oct 2026), where a single monocular human video, object scans, and photographs of additional object instances are transformed into thousands of robot demonstrations through semantic and functional alignment. More broadly, R2Sem2R encompasses reconstruction-grounded simulation, object-centric representation transfer, differentiable system identification, semantic scene generation, digital twins, function-preserving geometric augmentation, and distributional physical inference.

1. Conceptual scope and terminology

R2Sem2R extends the conventional real-to-sim-to-real paradigm by treating the intermediate representation as more than a simulator asset or a collection of pixels. The intermediate “Sem” stage may encode object identity, functional regions, spatial relations, physical parameters, collision geometry, camera-aligned scene structure, or uncertainty over plausible physical descriptions. It can therefore mediate both data generation and policy adaptation.

A generic R2Sem2R process can be represented as:

$$\text{real observations or demonstrations} \rightarrow \text{semantic/structured representation} \rightarrow \text{simulation, augmentation, or policy learning} \rightarrow \text{real-robot deployment}.$$

The semantic representation is not necessarily linguistic or symbolic. In “A Distributional Treatment of Real2Sim2Real for Object-Centric Agent Adaptation in Vision-Driven Deformable Linear Object Manipulation” (Kamaras et al., 25 Feb 2025), semantics are represented by a posterior distribution over interpretable physical properties such as DLO length and Young’s modulus. In “Bridging Simulation and Reality: Cross-Domain Transfer with Semantic 2D Gaussian Splatting” (Tang et al., 4 Dec 2025), semantics are object-centric spatial features derived from CLIP, SAM, and Gaussian splatting. In “Real2Sem2Real: 3D Semantic Augmentations for Robot Learning from a Single Human RGB Demonstration” (Strong et al., 20 Oct 2026), semantics include object categories, object-relative trajectories, functional regions, and relations such as a mug handle or coffee-machine tray.

The term therefore covers multiple intermediate-model regimes:

Regime	Intermediate representation	Primary downstream use
Scene reconstruction	Reconstructed geometry, appearance, poses, and scene relations	Simulation construction
Physical identification	Posterior over simulator parameters	Object-specific domain randomization
Semantic feature transfer	Object-centric spatial features	Sim-to-real policy invariance
Functional augmentation	Object meshes, task frames, and collision proxies	Synthetic contact-rich demonstrations
Digital-twin synchronization	Corrected simulator state	Simulation-native policy execution
Human-video transfer	Semantic object alignment and robot-relative trajectories	Robot demonstration generation

R2Sem2R should be distinguished from ordinary domain randomization. Domain randomization samples predefined simulator parameters, whereas R2Sem2R methods use real observations or semantic correspondences to construct, condition, or adapt the simulation distribution. It should also be distinguished from purely visual domain adaptation: the intermediate representation may explicitly preserve collision geometry, contact relations, functional regions, or object-specific physical uncertainty.

2. Real-to-structured-representation methods

The first stage of R2Sem2R converts physical observations into representations suitable for simulation, generation, or policy learning. The representation may be reconstructed from RGB, RGB-D, video, multi-view images, or dynamic manipulation trajectories.

Scene-specific reconstruction

“Re$^3$Sim: Generating High-Fidelity Simulation Data via 3D-Photorealistic Real-to-Sim for Robotic Manipulation” (Han et al., 12 Feb 2025) reconstructs a real tabletop workspace into a physics-enabled simulator. Its pipeline contains mesh recovery for collision and physical interaction, hybrid visual rendering, real-world alignment, privileged simulation data generation, and imitation learning. COLMAP estimates camera poses and a sparse point cloud, OpenMVS produces dense geometry and a detailed mesh, and simplified collision bodies are generated for stable physics. ArUco markers establish a reference frame, while ICP refines alignment between depth-camera point clouds and reconstructed meshes.

The method separates foreground and background representations. Foreground objects are reconstructed independently, principally with ARCode, so they can be segmented and inserted into different simulated placements. The background is represented with 3D Gaussian Splatting (3DGS), while foreground objects use reconstructed meshes rendered with ray tracing. Depth-based Z-buffer compositing combines the two. This division assigns appearance fidelity primarily to 3DGS and physical interaction to explicit meshes.

The resulting simulator uses NVIDIA Isaac Sim with PhysX. It includes reconstructed collision geometry, synchronized wrist and third-person camera views, simulated robot dynamics, motion planning, and privileged expert trajectories. The study uses a Franka Research 3 and two Intel RealSense D435i cameras. In three zero-shot tasks—bottle into basket, vegetable onto board, and stack cubes—the reported RE³SIM+IL success rates are 0.75, 0.25, and 0.75, respectively, with an average of approximately 58% (Han et al., 12 Feb 2025).

Compositional semantic reconstruction

“SimRecon: SimReady Compositional Scene Reconstruction from Real Videos” (Xia et al., 2 Mar 2026) addresses the conversion of cluttered RGB videos into semantically labeled, object-centric, simulation-ready scenes. Its Perception–Generation–Simulation pipeline first reconstructs scene-level objects, then completes individual assets, and finally assembles them using a scene graph.

Each object is represented through spatial attributes, appearance attributes, physical attributes, and relational attributes. Spatial information includes scale, rotation, translation, and a six-degree-of-freedom pose. Appearance includes a complete geometric mesh and physically based rendering textures. Physical attributes include semantic label, material, center of mass, and mass. Relations are represented by a graph whose edges include support and attachment relationships.

SimRecon uses 2D Gaussian Splatting for reconstruction, SceneSplat for semantic and instance decomposition, Rodin for single-object completion, and Qwen2.5-VL for intrinsic-attribute and scene-graph inference. Active Viewpoint Optimization searches for rendered viewpoints with high accumulated opacity and adds depth regularization to avoid degenerate views that approach the object surface. The Scene Graph Synthesizer uses DBSCAN-based spatial regions, region-specific viewpoints, VLM relation inference, graph merging, conflict adjudication, and hierarchical physics-based assembly.

On 20 ScanNet scenes using raw RGB video without supplied depth, normals, or semantic annotations, SimRecon reports a Chamfer Distance of 4.34, F-Score of 62.65, Normal Consistency of 87.37, PSNR of 24.43, SSIM of 0.924, LPIPS of 0.153, and MUSIQ of 73.56, with an average processing time of 21 minutes (Xia et al., 2 Mar 2026). The method does not itself train or deploy a robot policy; it supplies a front-end for a larger R2Sem2R system.

Object-centric semantic fields

S2GS constructs multi-view 2D semantic fields and projects them into a unified 3D representation through feature-level Gaussian splatting (Tang et al., 4 Dec 2025). CLIP supplies global, object-level, and language-linked features, while SAM supplies region masks. Each 2D Gaussian primitive receives a learnable semantic vector, and rendered features are decoded into semantic representations. CLIP similarity retrieves target objects from natural-language descriptions, while DBSCAN clustering and convex-hull completion filter false positives and fill incomplete surfaces.

The resulting policy representation is object-centric and spatial rather than RGB-based. Simulation and real images are both mapped into the S2GS feature space before policy training or deployment. In ManiSkill2 experiments, DP-S2GS achieves success rates of 1.00 on PickCube, 1.00 on PushCube, and 0.97 on StackCube, matching the state-based upper bound in simulation. Real deployment transfers from a 7-DOF Franka Panda in simulation to a 6-DOF UR5, with unseen household objects and no policy fine-tuning. Reported real-world averages are 86.7% for picking, 86.7% for pushing, and 80.0% for stacking (Tang et al., 4 Dec 2025).

3. Physical and functional semantics

R2Sem2R representations become particularly consequential when object identity alone is insufficient. Contact-rich manipulation depends on functional regions, material parameters, grasp geometry, support relations, and uncertainty over physical behavior.

Distributional physical identity

“A Distributional Treatment of Real2Sim2Real for Object-Centric Agent Adaptation in Vision-Driven Deformable Linear Object Manipulation” (Kamaras et al., 25 Feb 2025) infers a posterior distribution over DLO parameters from dynamic visual and proprioceptive trajectories. The parameter vector is:

$\boldsymbol{\theta}=\langle l,E\rangle,$

where $l$ is DLO length and $E$ is Young’s modulus. BayesSim with an RKHS distributional embedding estimates:

$\hat p(\boldsymbol{\theta}\mid \mathbf{x}^r).$

The posterior is represented by a four-component mixture of Gaussians. Its mixture structure captures alternative physical hypotheses, while covariance captures uncertainty within each hypothesis. The posterior is then used as the domain-randomization distribution for PPO training in Isaac Gym with the FleX physics engine and a corotational finite-element DLO model.

The system distinguishes stiffness more effectively than length. The two soft Shore 00-20 DLOs, with lengths of 200 mm and 290 mm, produce more overlapping posteriors than objects whose stiffness differs more strongly. The resulting policies display object-dependent trajectories, including different end-effector heights, reaching patterns, and motion structures. The paper evaluates six policies on four real DLOs over 96 deployments, without real-world policy fine-tuning (Kamaras et al., 25 Feb 2025).

This formulation interprets semantics as a probabilistic physical identity:

$$\mathcal{S}(\mathbf{x}^r)=\hat p(l,E\mid\mathbf{x}^r).$$

The representation is object-specific, physically interpretable, distributional, and directly usable for simulation and control. It is not a language-grounded semantic representation and is not demonstrated as a general representation across unrelated tasks.

Function-preserving geometry

“Function-Preserving Data Generation for Zero-Shot Real-to-Sim-to-Real Manipulation” (Xiang et al., 16 Sep 2026) defines semantics through task-critical regions and object functionality. A reconstructed mesh is partitioned into critical regions $\mathcal C$, movable handles $\mathcal H\subseteq\mathcal C$, and nonfunctional regions that may be reshaped. Constraint-guided deformation preserves topology, vertex correspondence, annotated interfaces, and manipulation relationships.

For stretching, the method uses slippage-preserving reshaping. For bending, it uses as-rigid-as-possible deformation. Each generated asset is a coupled bundle:

$(M_d,G_d,C_d),$

where $^3$0 is the deformed visual mesh, $^3$1 is the transferred task pose, and $^3$2 is the transferred collision proxy. Task poses are transferred through local patch matching and Kabsch–Umeyama rotation estimation. Collision proxies are transferred through barycentric correspondence on convex decompositions generated with CoACD.

The method generates demonstrations in PyBullet using task-specific scripted finite-state controllers. It does not require teleoperated source trajectories. Five tasks include object pickup, wrench-based screw fastening, assembly, single-gear extraction, and sequential extraction of three gears. Across unseen 3D-printed objects with preserved task interfaces, reported real-world success is 82% under nominal conditions, 74% with dynamic distractors, and 68% with dynamic lighting (Xiang et al., 16 Sep 2026).

Human-video semantic transfer

The eponymous R2Sem2R method, “Real2Sem2Real: 3D Semantic Augmentations for Robot Learning from a Single Human RGB Demonstration” (Strong et al., 20 Oct 2026), transfers both whole-object identity and functional-region correspondence. A mug can be replaced by other mugs, and a coffee machine can be replaced by other coffee machines while preserving the handle as a grasp region and the tray as a placement region.

The method estimates metric $^3$3 object poses from monocular RGB video using DINOv3-distilled Gaussian splats. It recovers a feasible parallel-jaw grasp from the human hand, generates digital cousins from one or a few photographs, and aligns source and cousin objects with bidirectional geometric ICP, semantic ICP, and task-weighted ICP. Functional regions receive 20 times the normal alignment weight. Demonstrated trajectories and object-relative grasps are then transferred to the aligned cousins.

The method supports tasks including mug placement on a coffee machine, pouring from a mug into a bowl, hanging a mug on a mug tree, and drill reorientation. It uses IsaacSim for photorealistic rendering and kinematic replay, and MuJoCo for contact dynamics and settling checks. A single human video produces approximately 2,000 retained robot episodes in about two hours on one RTX 5090, after approximately 25 minutes of preprocessing and planning (Strong et al., 20 Oct 2026).

4. Simulation, alignment, and policy-learning architectures

R2Sem2R systems differ in how they use the intermediate representation: some build a fixed simulator, some refine simulator parameters iteratively, some maintain a continuously synchronized digital twin, and others use the representation only as a policy input.

Real-to-sim-to-real scene reconstruction

RE³SIM uses reconstructed geometry and appearance to create simulation demonstrations. Privileged state information supplies scripted experts, while the learned policy receives only real-world-accessible observations: two camera views and proprioception. RRTConnect plans joint-space trajectories between desired 6D gripper poses, and failed trajectories are removed through rejection sampling. The policy is trained through an ACT-structured imitation-learning policy with DINOv2-Small visual encoding (Han et al., 12 Feb 2025).

Differentiable simulator refinement

“An Real-Sim-Real (RSR) Loop Framework for Generalizable Robotic Policy Transfer with Differentiable Simulation” (Shi et al., 13 Mar 2025) implements a closed-loop real-to-simulator-identification process. The simulator transition is written as:

$^3$4

where $^3$5 includes physical parameters such as friction, mass, and elasticity. Real trajectories are compared with simulator trajectories, and $^3$6 is updated by gradient descent through MuJoCo MJX as described by the framework. The policy is retrained in the updated simulator and redeployed.

The method adds an adaptive InfoGap loss to encourage real-world actions that are useful for simulator refinement. It combines a KL divergence between real and simulated data distributions with a Wasserstein distance measuring the distributional change induced by a candidate transition. The framework is evaluated on cube pushing and T-shaped block pushing using an AIRBOT Play robot and an Intel RealSense D435i. Reported trajectory-distribution KL values decrease from 16.3509 to 0.9739 in $^3$7 and from 36.3168 to 0.8384 in $^3$8 between the first PPO stage and the fourth RSR stage (Shi et al., 13 Mar 2025).

The paper describes its intermediate representation primarily as a differentiable physical simulator rather than a symbolic semantic model. It therefore represents a physical form of R2Sem2R, with active real-world data collection.

Continuously synchronized digital twins

“Real-is-Sim: Bridging the Sim-to-Real Gap with a Dynamic Digital Twin” (Abou-Chakra et al., 4 Apr 2025) keeps the policy entirely within simulation. A dynamic digital twin built with Embodied Gaussians is corrected continuously by real RGB observations. The policy acts on the simulated robot, and the physical robot tracks the simulated joint states. Real images are compared against Gaussian-splatting renders, and the discrepancy generates fictitious visual forces that correct simulated bodies.

The adapted system runs its online physics and correction loop at 60 Hz, while three RealSense D455 cameras acquire images at 90 fps. The policy may use simulated object pose and robot state, a static virtual-camera image, or a gripper-mounted virtual-camera image. On PushT, the reported success rates are 76% for simulator object pose plus robot state, 73% for static virtual-camera images, 82% for gripper-mounted virtual-camera images, and 63% for a real static-camera image baseline (Abou-Chakra et al., 4 Apr 2025).

This architecture shifts the sim-to-real problem from the policy to the digital twin. The policy never directly observes the physical domain; real observations continuously correct its simulator domain.

Physics and appearance alignment

“EmbodieDreamer: Advancing Real2Sim2Real Transfer for Policy Training via Embodied World Modeling” (Wang et al., 7 Jul 2025) separates physics alignment from appearance alignment. PhysAligner estimates friction and proportional and derivative control parameters through a differentiable surrogate model. VisAligner uses conditional video diffusion to transform low-fidelity simulation renderings into photorealistic videos conditioned on robot, foreground-object, and background information.

PhysAligner reduces the reported physical parameter estimation error by 3.74% relative to simulated annealing and improves optimization speed by 89.91%. The trajectory error changes from 0.2245 for SimplerEnv to 0.2161 for PhysAligner, while total computation time changes from 12,888.43 seconds to 1,299.92 seconds. VisAligner reduces FVD from 422.73 to 176.27 when object segmentation is added, while increasing PSNR from 23.067 to 24.518 and SSIM from 0.7939 to 0.8210 (Wang et al., 7 Jul 2025).

In downstream real-world tasks, reinforcement learning with EmbodieDreamer achieves an average success of 0.77, compared with 0.60 for three-view supervised fine-tuning and 0.49 for one-view supervised fine-tuning. The system therefore addresses both transition fidelity and observation fidelity, although the paper does not provide a full factorial ablation isolating the causal contribution of each module.

5. Data generation and augmentation

A central purpose of R2Sem2R is to expand limited real data without losing the task-relevant structure that makes demonstrations executable.

Photorealistic simulation data

RE³SIM generates 100 simulation trajectories per task and reports data-generation times of 12.35 minutes for bottle into basket, 13.78 minutes for vegetable onto board, and 6.45 minutes for stack blocks using eight RTX 4090 GPUs (Han et al., 12 Feb 2025). Its large-scale table-clearing experiment shows that very small synthetic datasets yield nearly negligible real-world performance, while increasing the number of synthetic trajectories improves performance before eventual convergence.

Semantic feature data

S2GS uses semantic spatial features rather than RGB observations as the policy input. This reduces dependence on texture, illumination, background appearance, and camera-specific pixel patterns. The policy receives a compact object-centric spatial representation and current end-effector pose. The approach does not generate new physical object instances; it instead aims to make the same representation available in simulation and reality (Tang et al., 4 Dec 2025).

Real-to-real 3D editing with video completion

“R2RDreamer: 3D-aware Data Augmentation for Spatially-generalized 2D Manipulation Policies” (Xu et al., 15 Jun 2026) uses incomplete point clouds and action trajectories for 3D editing, then generates RGB training videos through dense-control image-to-video completion. For object group $^3$9, the method applies the same transform to point clouds and end-effector poses:

$\boldsymbol{\theta}=\langle l,E\rangle,$0

The method explicitly models self-occlusion and external occlusion, blacking out unreliable projected regions before video completion. It supports Diffusion Policy and $\boldsymbol{\theta}=\langle l,E\rangle,$1 training. On four spatially shifted tasks, one source demonstration plus R2RDreamer augmentation improves success from 3.13% to 21.9–46.9%, depending on task and policy family (Xu et al., 15 Jun 2026).

R2RDreamer is a real-to-real augmentation method rather than a complete simulator-based R2Sem2R system. Its relevance is the separation of metric 3D editing for action consistency from 2D video generation for RGB-policy compatibility.

Function-preserving simulation data

The function-preserving framework generates 3,000 synthetic episodes per task from reconstructed objects and deformed asset bundles (Xiang et al., 16 Sep 2026). Visual and calibration randomization includes object placement, camera intrinsics and extrinsics, scene registration, lighting, textures, image-response effects, robot initialization, and distractors. The method uses these variations to train Diffusion Policies entirely in simulation and deploy them without real-world fine-tuning.

Its key distinction is that geometric diversity is constrained by functional semantics. Axis-aligned scaling can distort bores, mating surfaces, insertion clearances, and engagement geometry. Function-preserving deformation instead retains these interfaces and transfers both task poses and collision proxies.

Single-human-video generation

The eponymous R2Sem2R method generates robot demonstrations from one human RGB demonstration, source-object scans, and photographs of digital cousins. It uses IsaacSim for fast photorealistic data generation and MuJoCo for contact validation. The policy is a LoRA-fine-tuned $\boldsymbol{\theta}=\langle l,E\rangle,$2 vision-language-action model that receives a third-person RGB image and an 8-dimensional proprioceptive state and predicts 10 actions at 10 Hz (Strong et al., 20 Oct 2026).

Across four tasks and two robots, R2Sem2R exceeds same-object spatial augmentation by 26–54 percentage points. Policies trained on cousin-diverse data match or exceed the reported teleoperation baseline in the evaluated settings. On xArm7 coffee-machine placement, R2Sem2R reaches the best teleoperation performance at approximately 1,320 generated episodes and surpasses it at 2,000 episodes (Strong et al., 20 Oct 2026).

6. Results, limitations, and research directions

Empirical findings

Across the available studies, several recurring findings emerge.

Scene-specific geometry improves manipulation transfer. RE³SIM reports an average zero-shot success rate of approximately 58% across three rigid tabletop tasks, and its simulation-real success correlation on the bottle task is $\boldsymbol{\theta}=\langle l,E\rangle,$3 (Han et al., 12 Feb 2025).

Uncertainty can be useful when it is retained. BayesSim-based DLO adaptation does not collapse object identity to a single parameter estimate. Its posterior becomes the training distribution, allowing policies to train across plausible physical hypotheses (Kamaras et al., 25 Feb 2025).

Physical and semantic alignment complement visual realism. Function-preserving mesh deformation improves contact-rich tasks where simple scaling distorts critical interfaces (Xiang et al., 16 Sep 2026). S2GS improves transfer by suppressing irrelevant RGB variation and retaining object-centric spatial features (Tang et al., 4 Dec 2025). EmbodieDreamer separately aligns dynamics and visual appearance (Wang et al., 7 Jul 2025).

Object diversity can matter more than repeated trajectories. The single-human-video R2Sem2R results show that cousin-diverse episodes outperform large numbers of episodes involving only one demonstrated object pair (Strong et al., 20 Oct 2026).

Persistent simulators can simplify deployment. Real-is-sim keeps policy execution in a continuously corrected simulator, rather than requiring the policy itself to operate across a changing observation and dynamics domain (Abou-Chakra et al., 4 Apr 2025).

Common limitations

R2Sem2R methods remain dependent on the quality of their intermediate representation. Reconstruction errors, camera calibration errors, segmentation failures, VLM relation errors, and imperfect object completion can propagate into simulation and policy learning.

Many methods assume rigid objects or static workspaces. RE³SIM is restricted to rigid tabletop manipulation; S2GS uses rigid $\boldsymbol{\theta}=\langle l,E\rangle,$4 updates; SimRecon focuses on static cluttered scenes; function-preserving deformation assumes preserved mesh topology; and the single-human-video R2Sem2R method does not currently handle articulated-object trajectories such as doors, drawers, hinges, or screw mechanisms.

Physical parameters are often incompletely known. RE³SIM uses simulator default mass and friction values because static images do not identify them. The DLO method infers only length and Young’s modulus, excluding camera, workspace, and controller parameters from its principal inference vector. The differentiable RSR framework concentrates on explicitly represented parameters and does not adequately model implicit effects such as wind, turbulence, or unobserved environmental variables.

Photorealism does not guarantee physical correctness. Diffusion-generated videos may produce plausible but physically inaccurate appearance, while accurate collision meshes may still be rendered with unrealistic lighting or sensor artifacts. Conversely, physically accurate simulators may produce images that are unsuitable for RGB policies. The IsaacSim–MuJoCo comparison in (Strong et al., 20 Oct 2026) illustrates this tension: IsaacSim-generated data achieve 90% on the reported xArm7 placement condition, whereas MuJoCo-generated data achieve 62%, although MuJoCo supplies more appropriate contact dynamics.

Evaluation is also uneven across methods. Some studies report detailed success rates and ablations, while others provide qualitative evidence, trajectory plots, rendering metrics, or incomplete per-task results. Several papers do not report confidence intervals, statistical significance tests, complete per-task tables, or controlled ablations for every pipeline component.

Research directions

The supplied research suggests several directions for R2Sem2R development:

Unified semantic-physical world models: combine object identity, functional regions, scene relations, collision geometry, and distributions over physical parameters rather than treating them as separate modules.

Closed-loop real-data refinement: integrate S2GS- or scene-graph-based representations with differentiable simulator tuning, active data collection, or continuously synchronized digital twins.

Contact-aware semantic transfer: extend functional alignment beyond position and appearance to surface normals, friction, compliance, force closure, insertion clearance, and release dynamics.

Articulated and deformable objects: augment rigid $\boldsymbol{\theta}=\langle l,E\rangle,$5 representations with articulated joint models, deformable-body parameters, and task-specific dynamic inference.

Uncertainty calibration: quantify uncertainty in object tracking, semantic retrieval, functional-region correspondence, generated geometry, and simulator parameters using calibrated posterior metrics and deployment-oriented risk measures.

Scalable semantic diversity: generate large numbers of function-preserving object variants and cousin instances while preserving task interfaces, relational structure, and executable robot trajectories.

Policy-independent evaluation: establish common benchmarks that separately measure reconstruction fidelity, semantic correspondence, physical validity, image fidelity, action validity, simulation-real correlation, and real-robot success.

R2Sem2R is therefore best understood not as a single algorithm but as an architectural principle: real observations should be converted into structured representations that preserve the semantics needed for interaction, those representations should support controlled simulation or augmentation, and the resulting policies or trajectories should remain executable when returned to physical robots. The most complete systems combine geometric reconstruction, semantic correspondence, physical consistency, uncertainty handling, visual synthesis, and closed-loop validation rather than relying on any one of these components alone.