---
title: 'Gen2Real: Generated-to-Real Transfer'
url: https://www.emergentmind.com/topics/gen2real
type: topic
---

# Gen2Real: Generated-to-Real Transfer

Searching arXiv for papers that explicitly use the term "Gen2Real" and closely related usages.
arxiv_search query: "Gen2Real"
arxiv_search query: "\"Gen2Real\" games photorealism enhancement"
arxiv_search query: "\"General Robot Dynamics Learning and Gen2Real\""
arxiv_search query: "\"Real2Gen\" imitation learning"
arxiv_search query: "\"CG2Real\" synthetic-to-real translation"
Gen2Real denotes a research orientation concerned with transferring capability, structure, or appearance from generated, simulated, or otherwise non-real sources into real-world execution or real-looking outputs. In the literature considered here, the term appears both as an explicit method name and as a broader descriptor. "General Robot Dynamics Learning and Gen2Real" defines Gen2Real as transferring a general dynamics model learned in simulation to a specific physical robot [2104.02402]. "\textsc{Gen2Real}: Towards Demo-Free Dexterous Manipulation by Harnessing Generated Video" uses generated human-manipulation video as the sole demonstration source for dexterous policy learning [2509.14178]. Other works employ closely related language around Gen2Real-style photorealism enhancement in games and photorealistic text-to-image generation, even when "Gen2Real" is not the formal framework name [2508.17061] [2512.00473]. This suggests that Gen2Real is best understood as a family of generated-to-real transfer strategies rather than a single canonical algorithm.

## 1. Terminological scope and neighboring formulations

Within the cited corpus, Gen2Real spans at least three technical regimes: transfer of simulated dynamics to a real robot, transfer of generated video into robot demonstrations, and transfer of rendered or generated imagery toward real visual statistics. The common pattern is that a generated intermediate is not treated as an endpoint; it is used as a structured source of supervision, pretraining, or optimization signal.

| Work | Domain | Core generated-to-real bridge |
|---|---|---|
| "General Robot Dynamics Learning and Gen2Real" [2104.02402] | Robot dynamics | General simulated dynamics model adapted to a real robot |
| "\textsc{Gen2Real}: Towards Demo-Free Dexterous Manipulation by Harnessing Generated Video" [2509.14178] | Dexterous manipulation | Generated video parsed, optimized, retargeted, and stabilized for robot execution |
| "REGEN: Real-Time Photorealism Enhancement in Games via a Dual-Stage Generative Network Framework" [2508.17061] | Game photorealism | Offline robust teacher creates photorealistic targets for a fast runtime student |
| "RealGen: Photorealistic Text-to-Image Generation via Detector-Guided Rewards" [2512.00473] | Text-to-image generation | Detector-guided RL optimizes images toward realism rather than preference |

Several adjacent formulations sharpen the boundary of the term. Syn2Real is a benchmark for synthetic-to-real unsupervised domain adaptation across closed-set classification, open-set classification, and detection [1806.09755]. Deep CG2Real addresses synthetic-to-real image translation through image disentanglement into shading and albedo [2003.12649]. Real2Gen works in the reverse direction: a single real human demonstration is converted into a simulator and then amplified into robot-training data [2602.12734]. A plausible implication is that Gen2Real sits inside a broader ecosystem of synthetic-to-real, generated-to-real, and real-to-generated pipelines rather than replacing those formulations.

## 2. Gen2Real in robot dynamics learning

In [2104.02402], Gen2Real is the transfer stage of a larger general robot dynamics program. The paper starts from the rigid-body equation
$$
M(q)\ddot{q} + C(q,\dot{q})\dot{q} + G(q) = \tau + f
$$
with state
$$
s = [q^{\mathrm{T}}, \dot{q}^{\mathrm{T}}]^{\mathrm{T}},
$$
and formulates learning as next-state prediction from current state and torque:
$$
(s_t, \tau_t) \mapsto s_{t+1}.
$$
The central claim is that robot dynamics vary in three ways: dynamics parameters, topology configurations, and model dimensions. The paper therefore randomizes all three through
$$
P = \{P_p, P_c, P_d\},
$$
where \(P_p\) contains mass, center of mass, link length, inertia, and friction; \(P_c\) contains joint-axis configurations; and \(P_d\) contains the number of links.

The dataset construction is correspondingly large-scale. The paper generates 20 million trajectories, each with 50 state points, by sampling randomized robot models, sampling initial states, applying random torques, and integrating with fourth-order Runge–Kutta at \(\Delta t = 0.1s\) [2104.02402]. The architecture is a GPT-modified sequential model in which embeddings are replaced by a fully connected layer, softmax is removed because the outputs are continuous, and positional encoding is removed. Reported model sizes are about 141M parameters for the 2-link general robot dynamics model and about 200M parameters for the 3-link model.

Gen2Real is then defined as direct transfer from the general simulated model to a real target robot using a relatively small amount of experimental data. The paper contrasts this with a conventional Sim2Real workflow in which a new robot typically requires nominal parameter inspection, simulation-data generation for that specific robot, retraining, and only then transfer to reality [2104.02402]. Because the pretrained model is large, only a few layers are tuned during transfer; in the UR5e experiment, only the last layer is adjusted. For real dynamics transfer to UR5e, the reported mean RMSE is 0.018 with standard deviation 0.004 for Gen2Real, compared with 0.036 and 0.010 for Spe2Real. For inverse dynamics transfer, Gen2Real is competitive but not always best: the paper reports 0.83 \(N\cdot m\) mean torque RMSE with 0.21 standard deviation, versus 0.72 \(N\cdot m\) and 0.10 for both LSTM and Spe2Real [2104.02402].

The significance of this formulation is methodological rather than merely empirical. Gen2Real is presented not as dynamics randomization for a single platform, but as transfer from a pretraining distribution that spans robot property, connection, and dimensional variability. The paper explicitly frames this as lowering the threshold of dynamics learning by making general models reusable across robots.

## 3. Gen2Real as demo-free dexterous manipulation from generated video

In [2509.14178], Gen2Real refers to a demo-free dexterous manipulation framework that replaces human demonstrations with one generated human-manipulation video. The input at time 0 is an RGB image \(I_0 \in \mathbb{R}^{3\times H\times W}\), a depth image \(D_0 \in \mathbb{R}^{H\times W}\), and a natural-language instruction \(\mathbb{T}\); the output is an executable robot action sequence
$$
\tilde\tau^{\mathcal R} = \{\tilde a_1,\dots,\tilde a_T\}.
$$
The pipeline has three modules: demonstration generation, Physics-aware Interaction Optimization Model (PIOM), and demonstration learning.

The demonstration-generation stage uses Kling AI to synthesize a video from a single image and a text instruction. DAV predicts dense depth for each frame, HaMeR parses wrist pose and MANO hand joint angles, and FoundationPose estimates object pose. The parsed trajectory is
$$
\bar \tau = \{(\bar w_t,\bar j_t,\bar p_t)\}_{t=1}^T.
$$
The paper emphasizes that generated videos are noisy and often physically implausible, with jitter, hand-object misalignment, and penetration [2509.14178].

PIOM is introduced to repair that trajectory. Its inputs are hand and object point clouds and the parsed pose state. Two TokenPointNet backbones encode hand and object point clouds, a cross-attention block models hand-object interaction structure, Pose-Geometry Correspondence Attention projects pose features into the same latent space, and a temporal transformer outputs the optimized hand and object trajectories
$$
(\hat\tau^{\mathcal H},\hat\tau^{\mathcal O}) = \mathcal T_\theta(\mathbf z_1,\dots,\mathbf z_T).
$$
PIOM is trained with reconstruction, smoothness, and penetration losses. The final objective is
$$
\mathcal L_{\mathrm{total}} = \lambda_{\mathrm{rec}}\mathcal L_{\mathrm{rec}} + \lambda_{\mathrm{smooth}}\mathcal L_{\mathrm{smooth}} + \lambda_{\mathrm{pene}}\mathcal L_{\mathrm{pene}}.
$$

The demonstration-learning stage first retargets the optimized human trajectory to the robot hand by minimizing fingertip alignment, wrist-orientation alignment, and joint-angle similarity:
$$
J(w,j;\hat\tau^{\mathcal H}) = \|q-\hat q\|^2 + \lambda_o \left\| \log\!\left(R(w)R(\hat w)^\top\right)\right\|^2 + \lambda_a \|j-\hat j\|^2.
$$
Retargeting alone is not sufficient, so the paper trains an anchor-based residual PPO policy. The state includes robot joint positions and velocities, wrist pose and velocity, object pose and velocity, fingertip contact forces, the retargeted robot anchor trajectory, and the optimized object trajectory. The reward is
$$
r_t = \lambda_{object} r_t^{\mathcal O} + \lambda_{wrist} r_t^{w} + \lambda_{finger} r_t^{q} + \lambda_{contact} r_t^{C}.
$$

The reported simulation results are on six YCB objects in IsaacGym with 1000 trials per object. Per-object success rates are 72.5% for Tomato Soup Can, 70.7% for Power Drill, 88.3% for Bleach Cleanser, 86.8% for Cracker Box, 69.2% for Banana, and 76.0% for Mug, yielding a mean of 77.3% [2509.14178]. The paper also reports coherent executions on a real robot consisting of an Inspire Hand mounted on a UR5 arm, though no numeric real-world success rate is provided in the excerpt. The ablation results are especially strong: removing PIOM collapses performance to 0%, removing the residual policy yields 7.5%, removing retargeting yields 36.1%, and the full system yields 77.3%.

In this usage, Gen2Real is not only a transfer label but a complete pipeline for converting generated visual evidence into physically executable dexterous behavior. The generated video supplies task intent, PIOM imposes physical and temporal consistency, and residual RL closes the remaining embodiment and dynamics gaps.

## 4. Gen2Real-style photorealism enhancement and photorealistic generation

The term also appears in a broader visual sense. REGEN is described as a practical way to bring the quality of Gen2Real-style game-to-real photorealism enhancement closer to real-time deployment in an actual game engine [2508.17061]. Its setting is rendered GTA V frames translated toward the appearance of real-world Cityscapes scenes. The paper argues that robust unpaired image-to-image translation methods are strong but slow, while lightweight methods are faster but more artifact-prone. REGEN resolves this by splitting the problem into two stages.

The first stage uses the EPE framework from Richter et al. as a robust unpaired teacher. EPE consumes game frames, auxiliary engine information including G-Buffers, and unpaired real-world images. It uses patch matching via VGG-16 features and FAISS search, a multi-stream G-Buffer encoder guided by semantic classes defined by the stencil, and semantic guidance from MSEG [2508.17061]. The output is a semantically consistent photorealistic target for each game frame. The second stage then trains Pix2PixHD as a lightweight paired translator on the aligned pairs formed by the original GTA V frame and the EPE-enhanced frame. Because the first stage has already solved the hard unpaired realism problem, the runtime model only needs to learn a supervised mapping with pixel-level losses such as L1, perceptual loss, and adversarial loss.

Only the lightweight paired model runs during gameplay, and the final model requires only the rendered frame rather than low-level engine buffers, which the paper states makes it compatible with inference frameworks such as ONNX Runtime and integration as a post-process filter in UE5 or Unity [2508.17061]. Quantitatively, the paper reports that EPE runs at 0.9 FPS with 1110 ms/iter and 9.5 GB VRAM, whereas REGEN runs at 29.83 FPS with 33.53 ms/iter and 11.5 GB VRAM, corresponding to a 32.14× inference-speed improvement. On KID \(\times 100\) and FID, CUT reports 4.35 / 43.93, EPE 3.20 / 38.30, and REGEN 3.38 / 39.62. The stated conclusion is that REGEN is near EPE in visual quality while clearly outperforming the lightweight unpaired baseline CUT in realism and artifacts.

A related but distinct visual formulation appears in RealGen, which is presented in the source as advancing Gen2Real-style photorealistic text-to-image generation [2512.00473]. RealGen addresses the claim that modern text-to-image systems can follow prompts well yet still produce "fake" images with artifacts such as overly smooth skin and oily facial sheen. Its pipeline combines an LLM prompt optimizer with a diffusion model based on FLUX.1-dev with LoRA adaptation. The central technical contribution is a detector-guided reward with three components: semantic-level reward from Forensic-Chat, feature-level reward from OmniAID, and text-alignment reward from Long-CLIP. Both the LLM and diffusion model are optimized with GRPO in a two-stage post-training setup.

RealGen also introduces RealBench, a photorealism benchmark with 1000 high-quality real-world images, captions, 7 categories, and a strong emphasis on portraits [2512.00473]. Evaluation uses Detector-Scoring with Forensic-Chat, OmniAID, Effort, and a held-out GPT-5 detector prompt-based judgment, plus Arena-Scoring in which GPT-5 acts as a judge in at least 3000 random pairings. The paper argues that detector-guided optimization is better aligned with photorealism than human preference rewards such as PickScore or HPSv2.1, because realism is not the same as preference.

Taken together, these visual papers use Gen2Real not to mean robot deployment, but to denote progression from visibly synthetic renderings or generated images toward outputs that approach real-image appearance under practical runtime or training constraints.

## 5. Relation to synthetic-to-real benchmarks and adjacent pipelines

The broader generated-to-real landscape is clarified by Syn2Real, Deep CG2Real, and Real2Gen. Syn2Real provides a large-scale benchmark for synthetic-to-real unsupervised domain adaptation with one synthetic domain and two real-image domains, and defines three tasks: closed-set object classification, open-set object classification, and object detection [1806.09755]. Its central result is a performance gap across tasks. Closed-set classification can approach oracle performance; on the test target, Self-Ensembling reaches 91.8% while the oracle reaches 92.1% or 93.4% depending on backbone. Open-set adaptation is harder, and object detection is hardest: source-only target mAP is 0.06, while MMD, CORAL, and SE produce 0.03, 0.04, and 0.02, respectively. The benchmark therefore establishes that synthetic-to-real success in classification does not imply success in detection or open-set recognition.

Deep CG2Real addresses visual transfer by arguing that direct unpaired translation in image space is severely under-constrained and tends to hallucinate structure or distort textures [2003.12649]. Its two-stage pipeline first predicts PBR shading from OpenGL renderings using paired supervision and then refines the result to the real domain with a disentangled CycleGAN-like model operating separately on albedo and shading:
$$
I \approx A \odot S.
$$
On SUNCG, the reported OpenGL-to-Real FID is 53.48 for the full method, compared with 59.42 for CycleGAN and 65.33 for \(T^2\)Net; Amazon Mechanical Turk preference reaches 78.4% for the full method. The downstream claim is that models trained on the generated "real" images predict more accurate depth and normals than domain adaptation approaches, implying that realism enhancement can itself be a transfer mechanism.

Real2Gen is structurally adjacent but directionally reversed. It begins from a single real human demonstration, extracts object-centric task information, generates task-relevant 3D assets with Point-E plus marching cubes, aligns them with Zero-Shot-Pose, and then uses SAPIEN to generate arbitrarily many robot demonstrations for policy learning with PointFlowMatch [2602.12734]. The paper reports an average increase in success rate of 26.6 percentage points over the baseline and zero-shot real-world deployment after purely simulation-based training. In this case, the pipeline is real-to-generated-to-real: real observation seeds simulation, simulation amplifies data, and the learned policy transfers back to a real robot.

These neighboring works show that Gen2Real does not exhaust the space of generated/real transfer. Rather, it coexists with benchmark-oriented synthetic-to-real adaptation, disentangled realism translation, and real-to-generated data amplification.

## 6. Recurrent themes, misconceptions, and open technical issues

A recurrent misconception is that Gen2Real is interchangeable with standard Sim2Real. The robot-dynamics formulation explicitly distinguishes the two: conventional workflows often rebuild simulation data and retrain for each robot, whereas Gen2Real pretrains a general model and adapts it to a specific physical robot with limited real data [2104.02402]. The distinction matters because the source of generality is not only simulation, but randomized coverage of parameters, topology, and dimension.

A second misconception is that generated supervision can be used directly. The dexterous-manipulation framework shows the opposite: generated videos are noisy and often physically implausible, so the parsed hand-object trajectory must be corrected by PIOM and then further stabilized by retargeting and residual PPO [2509.14178]. Likewise, REGEN does not ask a lightweight unpaired model to solve the hardest realism problem directly; it first uses a robust offline teacher to create semantically consistent targets [2508.17061].

A third misconception is that realism can be reduced to preference. RealGen argues that human preference or aesthetic reward is not the same as photorealism and shows detector-guided reward outperforming PickScore and HPSv2.1 on held-out realism metrics [2512.00473]. Deep CG2Real makes a related point from the translation side: improving the visual realism of synthetic images can be more effective than imposing task-specific losses for downstream transfer [2003.12649].

Finally, benchmark evidence indicates that not all generated-to-real problems are equally tractable. Syn2Real shows that closed-set adaptation can be near-oracle while open-set classification and detection remain substantially harder [1806.09755]. A plausible implication is that any broad Gen2Real claim should be indexed to task type, representation, and deployment regime rather than treated as uniformly transferable across recognition, control, and image synthesis.

Across these formulations, the stable core of Gen2Real is the use of a generated intermediate that is subsequently constrained, distilled, aligned, or adapted until it becomes useful in a real domain. The specific mechanism varies—pretraining transfer, physics-aware trajectory optimization, teacher-student photorealism enhancement, detector-guided RL, or simulator-based data amplification—but the research program is consistent: generated structure is leveraged not as a substitute for reality, but as a computational scaffold for reaching it.

Source: https://www.emergentmind.com/topics/gen2real