GeRM: A Generative Rendering Model From Physically Realistic to Photorealistic

Abstract: For decades, Physically-Based Rendering (PBR) is the fundation of synthesizing photorealisitic images, and therefore sometimes roughly referred as Photorealistic Rendering (PRR). While PBR is indeed a mathematical simulation of light transport that guarantees physical reality, photorealism has additional reliance on the realistic digital model of geometry and appearance of the real world, leaving a barely explored gap from PBR to PRR (P2P). Consequently, the path toward photorealism faces a critical dilemma: the explicit simulation of PRR encumbered by unreachable realistic digital models for real-world existence, while implicit generation models sacrifice controllability and geometric consistency. Based on this insight, this paper presents the problem, data, and approach of mitigating P2P gap, followed by the first multi-modal generative rendering model, dubbed GeRM, to unify PBR and PRR. GeRM integrates physical attributes like G-buffers with text prompts, and progressive incremental injection to generate controllable photorealistic images, allowing users to fluidly navigate the continuum between strict physical fidelity and perceptual photorealism. Technically, we model the transition between PBR and PRR images as a distribution transfer and aim to learn a distribution transfer vector field (DTV Field) to guide this process. To define the learning objective, we first leverage a multi-agent VLM framework to construct an expert-guided pairwise P2P transfer dataset, named P2P-50K, where each paired sample in the dataset corresponds to a transfer vector in the DTV Field. Subsequently, we propose a multi-condition ControlNet to learn the DTV Field, which synthesizes PBR images and progressively transitions them into PRR images, guided by G-buffers, text prompts, and cues for enhanced regions.

• The paper outlines a generative rendering model (GeRM) that incrementally transfers physically based renders to photorealistic images using distribution transfer vector fields.
• It employs a multi-condition ControlNet with cross-modal alignment and attention masks to maintain structural consistency during the transition.
• The framework achieves superior image quality metrics, outperforming state-of-the-art models in PBR synthesis, photorealistic editing, and stylization.

Bridging Physically-Based Rendering and Photorealistic Rendering: The GeRM Framework

Motivation and Problem Statement

Physically-Based Rendering (PBR) systems, grounded in the rendering equation, prioritize strict adherence to the laws of light transport and material interaction. However, the distribution of PBR outputs, which remain constrained by digital scene asset quality, often differs substantially from that of real photographs, leading to an underexplored gap between physically realistic images ($I_\phi$) and photorealistic images ($I_\rho$). The root cause is the inherent limitation of digital existence ($E_\phi$)—even mathematically perfect light simulation yields results that fail to capture the complexity and organic detail of real-world imagery ($E_\rho$). This renders standard PBR fundamentally insufficient for achieving true photorealism in practical synthesis and editing applications.

Figure 1: A P2P Quad characterizing the correlation between physical realism and photorealism. $I_\phi$ denotes physically realistic image, $I_\rho$ photorealistic image, $E_\phi$ digital existence, and $E_\rho$ real-world existence.

Prior attempts to bridge this Photorealistic Pathway (P2P) gap, such as GAN-based approaches and recent diffusion model pipelines (e.g., RGB$\leftrightarrow$X), either lack geometric consistency due to absence of explicit physical conditioning or fail to generate the nuanced detail that defines real photos. Moreover, the absence of large-scale, high-quality, paired datasets aligning PBR and PRR domains has made direct supervised approaches infeasible.

GeRM: Generative Rendering Model Framework

This work introduces GeRM, a multi-modal generative rendering model targeting unified synthesis across the PBR–PRR continuum. GeRM formulates the P2P task as a conditional distribution transfer problem, leveraging G-buffers and textual description as control signals, and aims to progressively inject photorealistic detail while strictly preserving physical plausibility and user control.

Figure 2: Overview of the GeRM framework. Multi-condition G-buffer and spatial constraints enable progressive generation; the multi-condition ControlNet predicts a DTV field guiding transition from physical to photorealistic appearance.

Distribution Transfer Vector Field Formalism

GeRM models the directional transition from $\mathbf{x}_t$ (source: physically-based rendering) to $I_\rho$0 (target: more photorealistic rendering) as iterative application of Distribution Transfer Vectors (DTVs):

$I_\rho$1

where $I_\rho$2 encodes concatenated G-buffer conditions and spatial semantic masks. Instead of attempting a single-step domain mapping, GeRM decomposes the journey into a sequence of small, semantically-incremental states, supervised by dataset pairs representing progressively increasing photorealism.

Large-Scale Expert-Guided P2P Dataset

A core challenge is the absence of explicit realism metrics or suitable PBR–PRR data. The authors address this by constructing a progressive, expert-guided pairwise dataset (P2P-50K) using a multi-agent vision-LLM (VLM) system. Three specialized agents (Global Auditor, Contextual Enricher, Local Refiner) collaboratively generate refinement prompts based on iterative critique of scene realism, feeding these into CLIPSeg for accurate spatial targeting.

Figure 3: Progressive P2P dataset construction pipeline. Multi-agent VLMs produce stepwise refinement prompts, CLIPSeg provides spatial masks, and a generative backbone evolves renders to higher photorealism.

Sample visualization confirms that this approach produces structurally aligned, photorealistically evolving scene pairs suitable for direct supervision:

Figure 4: Visualization of the constructed P2P dataset, demonstrating progressive enrichment of scenes from physically-based renders to photorealism.

Multi-Condition ControlNet with Cross-Modal Alignment

The core generative model is a multi-condition ControlNet that directly consumes G-buffers, spatial attention masks (from CLIPSeg or user input), and textual prompts. The network is trained to predict the DTV field, incrementally optimizing for photorealistic appearance without structural drift. The injection of attention masks via a cross-modal “transition perception boost” mechanism ensures semantic edits are localized and consistent with both the prompt and scene geometry.

Adaptive Stopping and Robustness

To automatically terminate progressive editing when convergence to a photorealistic state is achieved, GeRM employs semantic residual monitoring—the process halts when the masked residual between consecutive iteration outputs falls below a preset threshold, preventing over-editing and collapse.

Figure 5: Convergence analysis of progressive and single prompt editing, showing refinement of semantic targets and stabilization of differences by step 6.

Experimental Evaluation

GeRM is evaluated on PBR synthesis, PRR generation, semantic-aware editing, relighting, and stylization tasks.

PBR and PRR Image Synthesis

GeRM achieves PSNR of 30.17, SSIM of 0.9520, and LPIPS of 0.0469 for PBR reconstruction, significantly outperforming RGB$I_\rho$3X on both global and masked evaluation. In PRR generation, it delivers higher aesthetic and realism (as measured by KID, CLIP, and human preference studies) than IntrinsicControlNet, FLUX.1-Kontext, Nano Banana, and all industry-standard graphics engines tested.

Figure 6: Comparative results of PBR and PRR generation, demonstrating superior physical accuracy and photorealistic detail across scenes compared to robust baselines.

Semantic Control and Editing

GeRM excels on both user-specified and automatic semantic injection, with explicit spatial control avoiding artifacts common in existing SOTA text-guided editors. The pipeline achieves superior semantic alignment, structural preservation, and natural blending especially with fine-grained mask guidance.

Figure 7: Comparison of semantic editing. GeRM outperforms user-guided and prompt-driven baselines in structure fidelity and detail in both interactive and automated modes.

Generalization, Stylization, and Fidelity

GeRM enables photorealistic enhancement of raw renders, even for untextured 3D assets, producing images that approach the visual properties of real photographs.

Figure 8: PRR results of GeRM (top) compared to engine renders (middle) and real-world photos (bottom) show closing of the P2P distribution gap.

GeRM also generalizes to photorealistic stylization—e.g., “Winter” or “Cyberpunk” variants—by extending its vector field formulation:

Figure 9: PRR synthesis and stylization on two scenes; all variants maintain physical plausibility under G-buffer constraints.

Ablations and Convergence

Ablation studies establish the criticality of the P2P dataset and transition perception boost: removing either significantly lowers the aesthetic and realism scores, and increases undesired drift or incomplete edits. Automatic semantic stopping reliably converges generation in 6–9 steps for most scenarios.

Implications and Limitations

This work introduces a practical, architecturally unified approach to jointly controllable PBR-PRR generation and editing. The demonstrated ability to blend physically-anchored rendering with learned photorealistic priors offers new flexibility for graphics pipelines, content creation, and simulation-to-reality transfer tasks, making it highly relevant for digital artists, robotics perception, and adversarial domain adaptation.

The limitation to 2D image space (lacking 3D structural priors or volumetric coherence), and some sensitivity to stochastic editing hallucinatory artifacts, highlight important future research directions. Explicit 3D structural conditioning, temporal consistency for video, and principled treatment of advanced optical effects such as subsurface scattering and caustics remain open challenges.

Conclusion

GeRM establishes a new protocol for bridging the enduring gap between strictly mathematically accurate computer graphics and true, high-fidelity photorealistic image synthesis. By constructing a large-scale, progressively annotated P2P dataset, formulating generation as learnable distribution vector field flow, and introducing robust multi-condition cross-modal alignment, GeRM delivers controllable, structurally correct photorealistic images starting from PBR constraints. This work provides a strong foundation for future research in unified neural rendering, distributional editing, and photorealistic content generation (2604.09304).