RTR-GS: 3D Gaussian Splatting Inverse Rendering

• RTR-GS is a 3D Gaussian Splatting-based inverse rendering framework that recovers geometry, BRDF, and lighting from multi-view images.
• The method uses a hybrid rendering branch to separate low-frequency radiance from high-frequency reflection and stabilize geometry via radiance transfer.
• A physically based deferred rendering branch refines material decomposition and relighting, achieving efficient training and real-time-capable rendering.

Searching arXiv for the cited papers and closely related Gaussian Splatting work to ground the article. arXiv search query: "(Zhou et al., 10 Jul 2025) RTR-GS 3D Gaussian Splatting for Inverse Rendering with Radiance Transfer and Reflection" arXiv search query: "(Zeng et al., 1 Apr 2026) RT-GS Gaussian Splatting with Reflection and Transmittance Primitives" RTR-GS denotes “3D Gaussian Splatting for Inverse Rendering with Radiance Transfer and Reflection,” a 3D Gaussian Splatting–based inverse rendering framework designed to recover geometry, material (BRDF), and lighting from multi-view images while remaining fast enough for practical training and real-time-capable rendering (Zhou et al., 10 Jul 2025). Its central design choice is to separate low-frequency radiance from high-frequency reflection through a hybrid rendering model, then refine BRDF and illumination decomposition with an additional physically based deferred rendering branch. In that formulation, RTR-GS is not only a novel view synthesis system but also an inverse rendering pipeline oriented toward reflective objects, decomposition, and relighting.

1. Problem setting and motivation

Inverse rendering seeks to infer geometry, material, and lighting from images. RTR-GS treats this as a coupled estimation problem involving surface shape and normals, albedo, metallic, roughness, BRDF, environment illumination, and indirect illumination. The difficulty is especially acute for specular and reflective surfaces, where the observed image contains strong view-dependent appearance and where geometry, visibility, and reflectance are tightly entangled (Zhou et al., 10 Jul 2025).

The method is motivated by limitations in both NeRF-style and standard 3DGS-style formulations. NeRF-style methods can model appearance but are often slow and use MLPs that are expensive to optimize and render. Standard 3D Gaussian Splatting is fast and high-quality for novel view synthesis, but spherical harmonics are not ideal for sharp, high-frequency specular effects. In inverse rendering, SH overfitting can produce floating artifacts: tiny incorrect Gaussians or geometry distortions that detach from real surfaces. The reported consequences include degraded normal estimation, occlusion and visibility estimation, BRDF decomposition, and relighting quality (Zhou et al., 10 Jul 2025).

RTR-GS addresses these issues by isolating reflections into a dedicated reflection map or deferred-rendered component, replacing SH-based radiance modeling with radiance transfer, and introducing a physically based deferred rendering branch to refine material and lighting decomposition after geometry is stabilized. This suggests that the framework treats geometry recovery as the primary stabilizing scaffold and places high-frequency appearance in a representation less likely to perturb the reconstructed surface.

2. Architectural decomposition and pipeline

RTR-GS extends standard 3DGS in two stages. In the first stage, a hybrid rendering branch is used for geometry and appearance. The scene is initialized from sparse points, random or COLMAP, and the optimization covers Gaussian position, scale, rotation, opacity, normals, radiance-transfer attributes, and reflection map or reflection attributes. Rendering combines forward rendering for the low-frequency radiance part and deferred rendering for high-frequency reflection (Zhou et al., 10 Jul 2025).

In the second stage, a physically based deferred rendering branch is introduced for decomposition. Once geometry and visibility are stabilized, visibility is baked into a voxel grid, and the additional branch optimizes BRDF parameters, environment map, and indirect lighting parameters. This branch is used jointly with the hybrid branch to refine albedo, metallic, roughness, lighting, and relighting behavior (Zhou et al., 10 Jul 2025).

Stage	Core mechanism	Primary role
Stage A	Hybrid rendering with forward radiance transfer and deferred reflection	Recover stable geometry and separate high- and low-frequency appearance
Stage B	Physically based deferred rendering after visibility baking	Refine albedo, metallic, roughness, lighting, and relighting

A central screen-space blend is

$$I_{rgb} = C_r \cdot (1.0 - R_i) + C_{ref} \cdot R_i,$$

where $R_i$ is reflection intensity, $C_r$ is the radiance color, and $C_{ref}$ is the reflection color. In this design, weakly reflective pixels are dominated by radiance transfer, while strongly specular pixels are driven by the reflection branch. The stated purpose is to prevent high-frequency reflections from being forced into the same appearance representation as diffuse or global illumination terms (Zhou et al., 10 Jul 2025).

The reported geometric benefit follows directly from that separation. Low-frequency radiance is modeled by radiance transfer under shared global lighting constraints, while high-frequency reflection is handled by deferred rendering with a reflection map and local reflection attributes. The paper argues that this division reduces the tendency of Gaussians to drift away from surfaces in order to explain sharp highlights.

3. Mathematical formulation

RTR-GS retains the standard 3DGS alpha-blending form for screen-space accumulation:

$I_f = \sum_{i=0}^N f_i \alpha_i T_i,$

where $f_i$ is a Gaussian’s attribute, $\alpha_i$ is opacity, and $T_i = \prod_{j=1}^{i-1}(1-\alpha_j)$ is transmittance (Zhou et al., 10 Jul 2025). Each Gaussian’s normal is defined as its shortest axis, oriented toward the viewing direction, and later regularized by depth-derived pseudo-normals.

The distinctive change is the replacement of SH-only forward shading by radiance transfer. The view-independent diffuse term is written as

$$C_d \approx \boldsymbol{\rho_d} \sum_{j=0}^{n^2} c_j c^t_j,$$

and the view-dependent specular term as

$$C_s(\mathbf{o}) \approx \boldsymbol{\rho_s}\sum_{j=0}^{n^2} c_j c^t_j(\mathbf{o}), \quad \text{with} \quad c^t_j(\mathbf{o}) = G(f_t, \mathbf{o}),$$

where $R_i$0 is diffuse base color, $R_i$1 is specular base color, $R_i$2 are spherical harmonic lighting coefficients, $R_i$3 are transfer coefficients, $R_i$4 is a learned transfer feature, $R_i$5 is a lightweight 3-layer MLP with 64 hidden units, and $R_i$6 is reflection direction. The total outgoing radiance is

$R_i$7

Because all Gaussians share the same global lighting coefficients and MLP decoder structure while maintaining local transfer features, radiance transfer is presented as giving stronger structural constraints than independent SH per Gaussian (Zhou et al., 10 Jul 2025).

The reflection branch is explicitly parameterized as

$R_i$8

with $R_i$9 reflection tint, $C_r$0 reflection roughness, $C_r$1 reflection map, $C_r$2 normal, $C_r$3 view direction, and $C_r$4 a split-sum approximation. In the PBR branch, the general rendering equation is

$C_r$5

The diffuse term is approximated as

$C_r$6

where $C_r$7 is visibility from the baked voxel grid, $C_r$8 is direct environmental lighting, and $C_r$9 is indirect diffuse lighting. The specular term uses split-sum approximation, with the first term precomputed into a BRDF LUT and the second into a filtered cubemap (Zhou et al., 10 Jul 2025).

The appendix-level BRDF specification is a standard physically based Disney-style microfacet decomposition:

$C_{ref}$0

with

$C_{ref}$1

$C_{ref}$2

$C_{ref}$3

$C_{ref}$4

with $C_{ref}$5, $C_{ref}$6, and $C_{ref}$7 (Zhou et al., 10 Jul 2025).

4. Optimization, regularization, and implementation

Training is divided into two phases. Phase 1 uses only the hybrid rendering branch for 30,000 iterations. Radiance transfer is initialized at the start, view-dependent components are activated after 3,000 iterations, reflection intensity is initialized to 0.01 for all Gaussians, and radiance transfer order is set to 3. Phase 2 begins after visibility baking and jointly optimizes the hybrid and PBR branches for another 10,000 iterations while fine-tuning geometry for compatibility with the PBR branch (Zhou et al., 10 Jul 2025).

The main reconstruction loss for both hybrid and PBR rendering is

$C_{ref}$8

The final objective is

$C_{ref}$9

where $I_f = \sum_{i=0}^N f_i \alpha_i T_i,$0 is hybrid rendering loss, $I_f = \sum_{i=0}^N f_i \alpha_i T_i,$1 is physically based rendering loss, $I_f = \sum_{i=0}^N f_i \alpha_i T_i,$2 is light regularization, $I_f = \sum_{i=0}^N f_i \alpha_i T_i,$3 is a metallic or reflection prior, $I_f = \sum_{i=0}^N f_i \alpha_i T_i,$4 is normal consistency loss, $I_f = \sum_{i=0}^N f_i \alpha_i T_i,$5 is bilateral smoothness, and $I_f = \sum_{i=0}^N f_i \alpha_i T_i,$6 is an object mask constraint. The reported weights are $I_f = \sum_{i=0}^N f_i \alpha_i T_i,$7, $I_f = \sum_{i=0}^N f_i \alpha_i T_i,$8, $I_f = \sum_{i=0}^N f_i \alpha_i T_i,$9, and $f_i$0 (Zhou et al., 10 Jul 2025).

The regularizers are tied to specific decomposition goals. Normal loss is

$f_i$1

matching Gaussian normals to pseudo-normals from depth. Light regularization assumes natural white incident light:

$f_i$2

The metallic or reflection prior enforces consistency between PBR metallic and hybrid reflection intensity:

$f_i$3

For screen-space attributes such as normal, reflection intensity, roughness, and metallic, bilateral smoothness is

$f_i$4

and

$f_i$5

The object mask constraint is

$f_i$6

where $f_i$7 is the object mask and $f_i$8 (Zhou et al., 10 Jul 2025).

Implementation details reported in the paper are specific: optimization uses Adam; experiments were conducted on an NVIDIA RTX 4090; the MLP $f_i$9 has 3 layers and 64 hidden units; reflection map and environment map are $\alpha_i$0 cubemaps; and the visibility voxel grid is $\alpha_i$1 (Zhou et al., 10 Jul 2025).

5. Empirical evaluation and reported outcomes

RTR-GS is evaluated on TensoIR, Shiny Blender, and Stanford ORB, and also reports novel view synthesis on Ref-Real and Mip-NeRF 360. The metrics are PSNR, SSIM, LPIPS, MAE for normals, training time, and FPS (Zhou et al., 10 Jul 2025). The paper frames its evaluation around reflective scenes, relighting, normal estimation, and decomposition quality.

For novel view synthesis, the reported examples are: on TensoIR, Ours achieves 39.17 PSNR and Ours(HR) 41.39 PSNR; on Shiny Blender, Ours achieves 33.99 PSNR and Ours(HR) 35.24 PSNR; on Stanford ORB, Ours achieves 39.81 PSNR and Ours(HR) 40.49 PSNR (Zhou et al., 10 Jul 2025). These values are presented together with qualitative claims of better detail preservation, cleaner specular regions, reduced floating artifacts, and better stability in geometry.

For relighting, the reported PSNR values are 30.10 on TensoIR, 26.16 on Shiny Blender, and 28.93 on Stanford ORB. The qualitative observations accompanying those numbers are more realistic shadows, more accurate highlights, less aliasing in albedo and lighting, and better preservation of reflective details (Zhou et al., 10 Jul 2025).

Normal estimation is also emphasized. On Shiny Blender, the reported MAE for Ours is 3.091, described as better than GS-IR, R3DG, and GShader, and competitive with or better than 3DGS-DR depending on setting. The explanation given in the paper is that smoother and more plausible normals arise because reduced floating artifacts yield more stable geometry on specular objects (Zhou et al., 10 Jul 2025).

Ablation findings are central to the paper’s argument. Replacing radiance transfer with SH hurts performance, disabling propagation hurts slightly, removing metallic prior hurts relighting, freezing geometry hurts significantly, and disabling hybrid rendering also hurts significantly. This suggests that the method’s performance depends on the joint effect of radiance transfer, reflection separation, and geometry refinement rather than on any single component in isolation.

Efficiency claims are explicit. The hybrid rendering branch runs at about 96.4 FPS, the PBR branch at about 130.9 FPS, and the main table reports around 133 FPS for one branch setting and 96 FPS for the hybrid branch variant, depending on which output is measured. The paper presents these numbers as evidence that RTR-GS remains significantly faster than NeRF-based inverse rendering while improving rendering quality and decomposition (Zhou et al., 10 Jul 2025).

6. Scope, limitations, and relation to similarly named methods

RTR-GS is described as an inverse rendering system for reflective and diffuse scenes under environment lighting, with explicit recovery of geometry, BRDF parameters, and lighting. Its scope is strongest where the principal challenge is reflective appearance and where a separation between low-frequency radiance and high-frequency reflection can stabilize geometry. A plausible implication is that the method is particularly suited to object-centric inverse rendering setups, since its evaluation and decomposition objectives are organized around that regime.

Two limitations are stated explicitly. First, lighting is assumed to come from an infinite distance, which is less accurate for large-scale scenes or local lighting setups. Second, the method does not model more complex inter-reflections or higher-order indirect light beyond the simplified indirect terms used (Zhou et al., 10 Jul 2025). These are limitations of physical completeness rather than of rendering efficiency.

The nomenclature is easy to confuse with RT-GS, “Gaussian Splatting with Reflection and Transmittance Primitives,” which is a distinct framework oriented toward jointly modeling specular reflection and transmittance in Gaussian Splatting through separate Gaussian primitives and differentiable ray tracing (Zeng et al., 1 Apr 2026). RT-GS introduces diffuse Gaussians, reflection Gaussians, and transmittance Gaussians; uses a microfacet material model together with a specular BTDF; and targets realistic novel view synthesis in scenes containing both glossy reflections and transparent objects (Zeng et al., 1 Apr 2026). By contrast, RTR-GS centers on inverse rendering, radiance transfer, BRDF and lighting decomposition, and relighting (Zhou et al., 10 Jul 2025). This suggests that the similarity in acronyms masks a substantive difference in problem formulation: RT-GS is principally a reflection-and-transmittance rendering framework, whereas RTR-GS is principally an inverse rendering and decomposition framework.

Within 3D Gaussian Splatting research more broadly, RTR-GS is therefore best understood as a system that rebalances the representation of appearance so that radiance transfer carries low-frequency structure, deferred reflection carries high-frequency specular content, and a deferred PBR branch refines the physical decomposition after geometry has been stabilized. Its reported contribution is not merely sharper reflections, but a pipeline in which reflective appearance, geometry recovery, normal estimation, and relighting are optimized as mutually dependent components (Zhou et al., 10 Jul 2025).