Ref-NPR: Reference-Based Non-Photorealistic Radiance Fields for Controllable Scene Stylization (2212.02766v2)

Abstract: Current 3D scene stylization methods transfer textures and colors as styles using arbitrary style references, lacking meaningful semantic correspondences. We introduce Reference-Based Non-Photorealistic Radiance Fields (Ref-NPR) to address this limitation. This controllable method stylizes a 3D scene using radiance fields with a single stylized 2D view as a reference. We propose a ray registration process based on the stylized reference view to obtain pseudo-ray supervision in novel views. Then we exploit semantic correspondences in content images to fill occluded regions with perceptually similar styles, resulting in non-photorealistic and continuous novel view sequences. Our experimental results demonstrate that Ref-NPR outperforms existing scene and video stylization methods regarding visual quality and semantic correspondence. The code and data are publicly available on the project page at https://ref-npr.github.io.

• The paper presents Ref-NPR, a novel approach that uses a single stylized 2D reference to achieve controllable 3D scene stylization with semantic consistency.
• It leverages innovative ray registration to map 2D style cues into 3D space, ensuring geometric accuracy and cross-view coherence.
• Experimental results demonstrate that Ref-NPR outperforms current methods by producing visually superior and semantically faithful stylized scenes.

Ref-NPR: Reference-Based Non-Photorealistic Radiance Fields for Controllable Scene Stylization

The paper presents a novel approach named Reference-Based Non-Photorealistic Radiance Fields (Ref-NPR), addressing the limitations of current 3D scene stylization methods which often lack semantic correspondences in transferring styles. The proposed method introduces a controllable mechanism to stylize 3D radiance fields by utilizing a single stylized 2D view as a style reference.

Method Overview

Ref-NPR centers around two innovative processes: ray registration and template-based feature matching. These processes resolve the issue of style transfer that aligns with semantic elements across the scene. The ray registration framework maps the style reference into 3D space, facilitating pseudo-ray supervision, while the template-based feature matching leverages semantic correspondences to maintain style consistency, even in occluded regions of the scene.

Key Contributions

1. New Paradigm for Stylization:
   • The method introduces a new paradigm that provides significant control over style transfer in 3D scenes. This paradigm ensures that stylized results remain perceptually faithful to the style reference.
2. Reference-Based Ray Registration:
   • Ref-NPR incorporates a ray registration process that efficiently maps stylized reference views into the 3D domain of the radiance field. This enables reliable geometric consistency across different views.
3. Template-Based Semantic Correspondence:
   • Employing high-level semantic features, the template-based feature matching scheme fills occluded regions with contextually coherent styles, ensuring semantic consistency throughout the novel views.

Numerical Results

The experimental results showcase that Ref-NPR produces visually superior scene stylizations in comparison with state-of-the-art methods like ARF and SNeRF. Notably, Ref-NPR not only enhances visual quality but also substantially improves semantic correspondence, as quantified by robust results. The perceptual quality and cross-view consistency are evidenced by competitive scores in stylization metrics.

Discussion and Implications

Practically, Ref-NPR offers an effective tool for professionals in fields like game design and digital art, where stylized 3D content is desired. Theoretically, this work reinforces the potential of radiance fields in facilitating style transfers consistent with human perception.

Ref-NPR's compatibility with single-view reference stylization expands the toolset available for artistic rendering in virtual environments. This approach could be enriched by deploying advanced semantic feature extractors or integrating with emerging generative models to further elevate its adaptability.

Future Directions

Future research should explore enhancing the robustness of Ref-NPR by integrating multi-reference style input, particularly for complex scenes, which would enhance its adaptability and efficiency in large-scale applications. Additionally, exploring real-time adaptations and optimizations for computational efficiency remains an enticing direction. Aligning the capabilities of Ref-NPR with advancements in neural rendering and exploring synergies with other AI-driven stylization techniques could offer substantial advancements in robust and controlled scene stylization.

In conclusion, Ref-NPR presents a significant step toward achieving controlled, semantically consistent stylization in 3D environments, thereby contributing greatly to the field of computer-generated visual arts.