RTR-GS: Hybrid Neural-Graphics Model

• Hybrid Neural-Graphics Models are frameworks that combine explicit Gaussian splatting with neural rendering to produce photorealistic 3D reconstructions.
• RTR-GS employs diffuse, reflection, and transmittance Gaussian primitives alongside differentiable ray tracing to merge traditional light transport with neural functions.
• The model achieves state-of-the-art photometric accuracy and interactive performance by accurately modeling complex material properties and multi-bounce light effects.

Hybrid Neural-Graphics Models (RTR-GS)

Hybrid neural-graphics models integrate traditional graphics principles with neural scene representations to achieve photorealistic 3D reconstruction, relighting, and novel view synthesis. Specifically, in the context of real-time neural rendering, "Hybrid Neural-Graphics Models" often refers to architectures like RTR-GS—Ray Traced Gaussian Splatting—which unifies explicit 2D or 3D Gaussian splatting with physically-based light transport, BRDF modeling, and differentiable ray tracing. This class of models is designed to jointly capture diffuse and complex specular phenomena (reflection, transmission, and view-dependent effects) that challenge purely neural or purely graphics-based solutions. The resulting architectures leverage both explicit geometric primitives (e.g., Gaussian splats) and learned neural functions for superior scene modeling, photometric accuracy, and interactive frame rates, especially for challenging material properties and transparent or specular interfaces (Zeng et al., 1 Apr 2026).

1. Architectural Organization and Dataflow

RTR-GS employs three interleaved sets of 2D Gaussian primitives:

• Diffuse Gaussians: Model surface geometry and diffuse color via direct rasterization/splatting.
• Reflection Gaussians: Encode environment radiance for simulating view-dependent specular bounces.
• Transmittance Gaussians: Capture the radiance and structure behind semi-transparent surfaces, necessary for realistic modeling of glass and similar materials.

The dataflow proceeds in multiple coordinated passes:

1. Diffuse pass: Splat diffuse Gaussians to obtain per-pixel depth, normal, roughness ($\alpha$), base reflectance ($f_0$), specular weight ($k_s$), and diffuse color ($C_d$).
2. Reflection ray tracing: For each pixel, a reflection ray $r_r$ is constructed via $r_r = \mathrm{reflect}(\omega_o, n)$, which is traced through the set of reflection Gaussians using a differentiable ray tracer, yielding the specular color $C_r$.
3. Specular transmission: A transmission ray $r_t \approx \omega_o$ is traced through transmittance Gaussians for the “interior” color $C_{t,\rm in}$, and then from a mesh-guided "back surface" (using extracted mesh from the diffuse Gaussians) to account for “exterior-through” color $C_{t,\rm out}$.
4. Final blending: The rendering equation

$f_0$0

combines all light transport contributions.

Parameters are optimized by backpropagating photometric, material, depth, and perceptual losses through the entire hybrid pipeline (Zeng et al., 1 Apr 2026).

2. Mathematical Foundations

2.1 Gaussian Splatting

Each 2D Gaussian primitive $f_0$1 is parameterized by world-space mean $f_0$2, local tangent-plane covariance $f_0$3, opacity $f_0$4, and optionally a view-dependent color model $f_0$5. The per-pixel weight is

$f_0$6

with standard front-to-back $f_0$7 blending:

$f_0$8

2.2 Microfacet-Based Reflection and Transmission

Specular interaction is modeled via a Torrance–Sparrow microfacet BRDF with GGX distribution $f_0$9, Smith’s geometric term $k_s$0, and Schlick-style Fresnel $k_s$1. The BRDF for outgoing direction $k_s$2 and incident $k_s$3 reads:

$k_s$4

with $k_s$5.

Transmission adopts a delta-function approximation for thin surfaces:

$k_s$6

2.3 Differentiable Ray Tracing

Specular reflection and transmission contributions are computed by stochastically tracing rays through Gaussian fields:

$k_s$7

where $k_s$8 denotes Gaussian density, and $k_s$9 the per-Gaussian radiance.

3. Implementation and Optimization

RTR-GS leverages a fully differentiable hybrid renderer, combining explicit 2D Gaussian rasterization for geometry/diffuse appearance with custom CUDA-based (or comparable) differentiable ray-tracing kernels for specular and transmittance paths. The entire scene representation—including Gaussian parameters and per-Gaussian physical attributes ($C_d$0, $C_d$1, $C_d$2)—is optimized using a composite loss:

$C_d$3

Core losses include L1/D-SSIM photometric loss, specular mask regularization, first-hit depth regularization, normal-depth consistency, monocular normal supervision, and a VGG-based perceptual term. Transparent masks are used to supervise specular/transmittance weights, and mesh extraction from the diffuse Gaussians (e.g., via TSDF fusion) guides second-bounce trajectory for transmittance rays (Zeng et al., 1 Apr 2026).

4. Performance Benchmarks and Qualitative Behavior

Empirical evaluation on datasets with prominent transparent and specular features (Ref-Real, NU-NeRF) demonstrates that RTR-GS achieves:

• State-of-the-art PSNR and SSIM on 5 of 7 scenes, and lowest LPIPS on 6 of 7.
• Sharp near-field reflections (e.g., direct-light highlights on car surfaces).
• Faithful reconstruction of occluded or "behind-glass" objects (e.g., statues visible through transparent shells).
• Robustness in environments with complex interconnected light transport, outperforming purely rasterized or purely neural models in these regimes (Zeng et al., 1 Apr 2026).

5. Distinctions from Related Hybrid Approaches

Hybrid models for neural-graphics rendering exist along a spectrum. Compared to:

• HyRF: Which blends explicit per-Gaussian parameters (for high-frequency geometry/appearance) with neural fields for residual low-frequency and background properties, RTR-GS is focused on explicit physical processes—namely, accurate BRDF/BTF modeling, full handling of specular/transmissive interreflections, and mesh-guided multi-bounce light transport (Wang et al., 21 Sep 2025).
• HybridGS: Which disambiguates statics (multi-view consistent 3D Gaussians) and transients (per-image 2D Gaussians for scene elements appearing in only one view), RTR-GS decomposes physically-based light interaction channels, not solely static/transient content (Lin et al., 2024).

A key distinction for RTR-GS is the integration of both microfacet-based and differentiable-ray-tracing-based pipelines to model high-frequency specular and transmittance effects that are intractable for traditional "surface+diffuse" rasterization.

6. Limitations and Prospective Advances

Current limitations include:

• The assumption of infinitely thin transparent shells limits the method’s ability to represent volumetric refraction and internal caustics in thick dielectrics.
• Method sensitivity to mesh extraction in the diffuse channel; failures in mesh-guided second-bounce tracing can propagate artifacts.
• Omission of anisotropic microfacets and wavelength-dependent scattering.

Future directions include extending the pipeline to volumetric scattering/refraction, anisotropic and spectral microfacet models, dynamic or non-static scenes, global illumination, and automated Gaussian layout strategies, potentially reducing reliance on externally-guided mesh extraction (Zeng et al., 1 Apr 2026).

7. Broader Impact and Generalization

The hybrid neural-graphics paradigm underlying RTR-GS demonstrates that combining explicit, physics-based graphics components with data-driven neural representations delivers real-time, physically plausible appearance for complex materials and lighting conditions unattainable by either pipeline alone. This class of models enables significant advances in photorealistic view synthesis, material editing, mixed-reality compositing, and simulation environments demanding both high fidelity and interactive performance. The architectural principles—component factorization, differentiable light transport, and explicit/implicit primitive blending—inform a foundational strategy for next-generation neural rendering pipelines at scale.