GO-Renderer: Generative Object Rendering with 3D-aware Controllable Video Diffusion Models

Abstract: Reconstructing a renderable 3D model from images is a useful but challenging task. Recent feedforward 3D reconstruction methods have demonstrated remarkable success in efficiently recovering geometry, but still cannot accurately model the complex appearances of these 3D reconstructed models. Recent diffusion-based generative models can synthesize realistic images or videos of an object using reference images without explicitly modeling its appearance, which provides a promising direction for object rendering, but lacks accurate control over the viewpoints. In this paper, we propose GO-Renderer, a unified framework integrating the reconstructed 3D proxies to guide the video generative models to achieve high-quality object rendering on arbitrary viewpoints under arbitrary lighting conditions. Our method not only enjoys the accurate viewpoint control using the reconstructed 3D proxy but also enables high-quality rendering in different lighting environments using diffusion generative models without explicitly modeling complex materials and lighting. Extensive experiments demonstrate that GO-Renderer achieves state-of-the-art performance across the object rendering tasks, including synthesizing images on new viewpoints, rendering the objects in a novel lighting environment, and inserting an object into an existing video.

• The paper introduces GO-Renderer that fuses explicit 3D geometric proxies with video diffusion models to enable high-fidelity object rendering.
• It leverages pose-conditioned coordinate maps for robust multi-view consistency, precise viewpoint control, and flexible relighting in synthesized videos.
• GO-Renderer outperforms previous methods in PSNR, SSIM, and perceptual scores, proving its effectiveness for real-world applications and downstream integration.

GO-Renderer: Generative Object Rendering with 3D-aware Controllable Video Diffusion Models

Problem Context and Motivation

Object rendering from sparse imagery under arbitrary viewpoints and novel lighting presents a fundamental challenge for computer graphics, vision, and generative AI. Traditional pipelines separate 3D reconstruction from rendering, but geometric scaffolds derived from multi-view methods or generative 3D models often fail to capture material and lighting complexity required for photorealistic synthesis. Modern neural rendering techniques, such as NeRF and 3D Gaussian Splatting (3DGS), yield plausible novel view synthesis but are rigid in relighting due to baked-in illumination and suffer from multi-stage optimizations for physically accurate rendering. Reference-based video diffusion models offer direct synthesis of realistic videos conditioned on imagery, but lack multi-view consistency and precise camera/viewpoint control.

Methodological Contributions

GO-Renderer introduces a unified framework that fuses explicit 3D geometric proxies with video diffusion generative models to enable fully controllable, high-fidelity object video rendering in novel environments and viewpoints. The technical innovation is the use of pose-conditioned coordinate maps derived from coarse 3D reconstructions as strong geometric guidance to the video diffusion process.

Explicitly, sparse reference images are processed with feed-forward 3D reconstruction (e.g., ReconViaGen (Chang et al., 27 Oct 2025), VGGT [2503.]) to estimate reference camera poses and a point cloud or mesh proxy. From these, dense coordinate maps are rendered, with RGB channels encoding normalized 3D coordinates in the object’s local frame for both reference and target views. These coordinate maps are channel-wise concatenated with latent diffusion noise and appearance-guiding inputs, ensuring dense pixel-wise alignment between references and the target sequence. The resulting conditions are input to a Video Diffusion Transformer (DiT) architecture, with negative RoPE shift in temporal embeddings to avoid transition artifacts and appropriately isolate conditioning signals.

Notably, this method circumvents the need for explicit physical modeling of materials and illumination, exploiting the visual priors learned by foundation models and enabling joint optimization of multi-view consistency, accurate viewpoint control, and relighting flexibility. Moreover, the framework accepts optional appearance guidance (text, images, video) for environmental context specification.

Dataset and Training Procedure

GO-Renderer’s architecture requires a scale and diversity of training data not present in existing datasets. The authors construct a large, mixed-domain dataset—synthesizing 57,000+ high-quality object-centric video clips—using Blender-based synthetic rendering (with diverse 3D models and HDRI lighting), real-world object extraction (e.g., from CO3D (Reizenstein et al., 2021), OpenVidHD), and large-scale AI-generated object videos (Wan2.2 (Wan et al., 26 Mar 2025)). The data ensures balanced coverage of lighting, background, and trajectory variation, with accurate 3D proxy geometry and pose annotations.

Training is conducted with batch augmentation strategies to avoid overfitting to reference view order or spatial configurations, and the model is fine-tuned on a pre-trained Wan2.2Fun 5B Ref Control model (Wan et al., 26 Mar 2025).

Experimental Results and Analysis

Rendering, Relighting, and View Synthesis

Quantitative and qualitative evaluations demonstrate that GO-Renderer sets a new performance standard in controllable rendering tasks, outperforming 3DGS-based pipelines (AnySplat (Jiang et al., 29 May 2025)), generative 3D pipelines (ReconViaGen (Chang et al., 27 Oct 2025)), and two-stage compositing pipelines (UniLumos (Liu et al., 3 Nov 2025)) on both synthetic and real-world benchmarks. GO-Renderer achieves:

• Significant improvements in PSNR (18.26) and SSIM (0.684), as well as strong perceptual, temporal, and illumination scores (VBench++), compared to previous approaches.
• Superior consistency under drastic relighting—a challenge for both compositing pipelines (even those using ground-truth geometry) and reference-driven 2D diffusion, which both show color shifts and inconsistent lighting artifacts.

Multi-view Consistency and Downstream Utility

GO-Renderer demonstrates strict multi-view consistency and high-fidelity rendering when tested on canonical backgrounds to disentangle appearance preservation from relighting effects. CLIP and DINO metrics indicate superior semantic and structural alignment over prior methods. Unlike reference-based video diffusion approaches, the explicit 3D proxy ensures spatial determinism and eliminates geometric hallucination. The model further excels in downstream applications like object insertion, blending objects into real video with geometric and illumination coherence.

Ablation Studies

Ablation on the temporal offset for RoPE in DiT reveals that explicit separation (gap $g=3$) between reference and target frames is essential—without it, transition artifacts and severe performance degradation occur. Robustness experiments show that the model tolerates moderate spatial inaccuracy in the proxy, gracefully falling back to 2D visual priors when geometry is unreliable.

Practical and Theoretical Implications

By decoupling the bottleneck of physically-based material recovery from the synthesis pipeline and leveraging explicit geometric conditioning, GO-Renderer substantially advances the state of generative object rendering. This approach enables flexible, high-fidelity deployment of digital assets in film, advertising, AR/VR, and immersive content production without reliance on laborious multi-stage geometry/material optimization. Theoretically, the work suggests that explicit geometry-guided diffusion is a promising paradigm for bridging the gap between physically-motivated graphics and data-driven generation.

Potential future directions include refining geometric proxy extraction for even higher fidelity, dynamic object rendering, or incorporating generative modeling of proxy geometry itself for full end-to-end 3D-aware video generation.

Conclusion

GO-Renderer presents an explicit and effective solution to controllable high-fidelity 3D-aware object video rendering under arbitrary viewpoint and lighting, unifying explicit 3D geometric proxies with powerful video diffusion generative models. Through dense coordinate map conditioning, large-scale multi-modal data, and robust training, it establishes a new standard in generative rendering, enabling precise spatial and appearance control unattainable with prior methods (2603.23246). This framework significantly expands the practical toolkit for realistic content generation in digital media and raises new research possibilities for geometry-aware generative modeling.