RealMaster: Sim-to-Real Video Translation Framework

• RealMaster is a framework for sim-to-real video translation that preserves scene geometry, camera pose, and object identity while achieving photorealistic style.
• It employs a two-stage process with anchor-based keyframe enhancement and edge-conditioned propagation to ensure realistic material and lighting transformations.
• The IC-LoRA architecture uses low-rank adapters and in-context token conditioning for efficient, near-real-time inference on high-resolution videos.

RealMaster is a framework for sim-to-real video translation that lifts synthetic sequences generated by 3D engines into photorealistic video, preserving exact scene geometry, camera pose, object identity, and per-frame microstructure while effecting a holistic transformation of materials, lighting, and textures to resemble real-world footage. It leverages diffusion models in conjunction with explicit geometric conditioning and a two-stage anchor-based data generation pipeline, followed by parameter-efficient distillation into a single in-context finetuned video diffusion model. RealMaster’s approach explicitly addresses key technical challenges in bridging the sim-to-real gap, namely structural precision and global semantic transformation, in a way that balances granular 3D control with high-fidelity visual realism (Cohen-Bar et al., 24 Mar 2026).

1. Sim-to-Real Video Translation: Objectives and Challenges

RealMaster addresses the task of converting rendered (“synthetic”) video output by a 3D engine into a photorealistic video while enforcing strict structural fidelity with respect to the original scene. The two principal requirements are:

• Structural Precision: The transformation must preserve all geometry, dynamics, camera motion, and high-frequency scene details. No drifting, warping, or object hallucination is tolerated, and fine identity cues such as facial features must remain intact.
• Global Semantic Transformation: A comprehensive change of appearance is needed—materials, lighting, and textures must all be remapped to match real-world footage, thereby surmounting the “uncanny valley” common in 3D renders.

The solution decouples control and appearance: the 3D engine acts as a first-stage renderer responsible for geometry, layout, and scene structure, while a latent video diffusion model is learned as a “second-stage renderer” to achieve photorealistic style.

2. Anchor-Based Propagation: Paired Data Generation

Since high-quality paired (rendered, real) video is not available a priori, RealMaster employs a sparse-to-dense data synthesis pipeline:

• Keyframe Enhancement: For each synthetic clip, the first and last frames are selected and enhanced using a generic image editing model (Qwen-Image-Edit) with a prompt to induce photorealism. These serve as anchors defining the real-world target style.
• Edge-Conditioned Propagation: Simple edge maps (Canny or learned edge detectors) are extracted from each frame of the original rendered sequence. A VACE video diffusion model, conditioned on the two photo-edited anchors and per-frame edge maps, generates intermediate frames. Edge maps provide strong geometric priors, anchoring object boundaries, facial outlines, and motion, ensuring that style transfer respects both layout and dynamics.
• Ablation and Conditioning Choices: Alternatives such as depth maps, surface normals, or more than two edited keyframes were evaluated. Depth provided incomplete shape alignment and failed to maintain facial identity, while multiple anchors introduced temporal artifacts. Edge maps yielded optimal preservation of fidelity, stability, and detail.

Through this process, a dataset of aligned (rendered, photorealistic) video pairs is established for downstream learning.

3. IC-LoRA Model Architecture and Training Procedure

RealMaster distills the above-lengthy pipeline into a single efficient model via an In-Context Low-Rank Adapter (IC-LoRA):

• Base Backbone: The architecture builds upon a pre-trained text-to-video diffusion model (Wan2.2 T2V-A14B).
• LoRA Parameterization: Each weight matrix $W_0$ in the transformer is augmented as $W = W_0 + AB$, with $A, B$ learnable and low-rank (rank = 32), while $W_0$ is fixed.
• In-Context Conditioning: At each denoising step, the network receives both (i) noised tokens of the current frame and (ii) “clean context” tokens encoding the original rendered frames (at $t = 0$). Shared positional encodings preserve token correspondence, ensuring full alignment of output structure with input render.
• Loss Functions and Supervision: A standard diffusion reconstruction loss is used.

$$\mathcal{L}_{\rm diff} = \mathbb{E}_{t, \mathbf{x}_0, \epsilon} \left[ \| \epsilon - \epsilon_\theta(\mathbf{x}_t,\,\text{cond}) \|^2 \right]$$

No adversarial or perceptual loss is applied due to the strength of supervised signals and the prior.

• Distillation and Inference: The model is fine-tuned on (rendered, VACE-realized) paired sequences. After training, inference requires only the rendered sequence and minimal geometric cues (e.g., edges); explicit anchor editing and propagation are internalized.

4. Dataset Creation and Training Regimen

• Source Data: The SAIL-VOS dataset (GTA-V synthetic footage) is sampled at 8 fps, upsampled to 16 fps, and segmented into 81-frame, $800 \times 1200$ clips.
• Anchor Enhancement and Propagation: Initial keyframe edits plus VACE propagation produce approximately 3,050 raw pairs.
• Face-Identity Filtering: ArcFace cosine similarity between faces in the input/output is computed; only sequences with mean similarity $> 0.4$ are retained, yielding 1,216 high-quality pairs.
• Training Setup: Single H200 GPU, batch size of 8, LoRA rank 32, 1e-4 learning rate, AdamW optimizer, for 1,200 steps.

5. Inference Workflow, Efficiency, and Deployment

Inference on RealMaster consists of:

1. Direct ingestion of a rendered video and corresponding edge maps (length agnostic).
2. Tokenization of rendered frames as context at $t = 0$; at $t > 0$ the diffusion model predicts denoised, photorealistic frames.
3. No requirement for explicit anchor frames or separate propagation steps.

The model runs on a single GPU (H100/H200) with near-real-time performance for $800 \times 1200$ resolution. The LoRA adaptation adds approximately 10 MB to the backbone footprint.

6. Empirical Evaluation and Comparative Analysis

Evaluation on complex GTA-V (SAIL-VOS) benchmarks involves both quantitative metrics and user studies:

• Automatic Metrics:
  • Identity: ArcFace cosine similarity (higher is better).
  • Structure: Average $\ell_2$ DINO feature distance (lower is better).
  • Photorealism: GPT-4o score, both with and without reference (scale 1–10).
  • Temporal Consistency: Flicker score (stability) and motion smoothness.
• Results Summary:

Method	GPT-no-ref ↑	GPT-ref ↑	ArcFace ↑	DINO ↓	Flicker ↑	Smooth. ↑
Editto	5.10	3.84	0.204	41.79	0.972	0.972
Runway-Aleph	4.98	5.33	0.300	38.04	0.976	0.972
LucyEdit	3.48	4.20	0.375	36.68	0.976	0.986
RealMaster	5.30	7.33	0.473	30.28	0.976	0.973

• User Study:
  • Realism preferred in 73% of trials, faithfulness in 89%, overall visual quality in 80%.
  • Qualitative strengths: precise maintenance of object positions, lighting, facial identity; context-adaptive enhancements such as skin microstructure and dynamic weather (rain, snow) via prompt modification.
  • Failure modes include conservative treatment of small/distant objects and occasional temporal jitter with rapid motion.

7. Limitations and Future Directions

Several limitations are inherent to the RealMaster pipeline:

• Anchor Quality Dependence: The photorealism attainable is upper-bounded by the initial off-the-shelf image editing used for anchor enhancement.
• Synthetic Motion Inheritance: Implausible or non-realistic dynamics from the source render engine persist since motion is not directly corrected.
• Causality/Latency: The current model is non-causal; a streaming, low-latency version is a prospective development for real-time applications.
• Conditioning Modalities: Although edge maps perform well, research into leveraging richer geometric buffers (normals, albedo, semantic segmentation) or learning more advanced motion priors may yield improvements in fidelity and stability.

The overall significance of RealMaster is in demonstrating that anchor-based, geometry-aware paired data synthesis coupled with parameter-efficient in-context finetuning enables an effective sim-to-real lifting of rendered scenes—distinctly improving photorealism and structural consistency relative to prior methods (Cohen-Bar et al., 24 Mar 2026).