Spherical Mipmap-Based Directional Encoding

• The paper introduces a roughness-aware spherical mipmap that decouples surface orientation from viewing direction, enhancing specular and refractive rendering.
• It employs deferred shading to blend 2D Gaussian surfels into per-pixel buffers and queries a learnable, multilevel spherical grid via trilinear interpolation.
• The RGS-DR variant utilizes the encoding as a residual correction module, capturing complex view-dependent effects and achieving improved reconstruction metrics.

Spherical mipmap-based directional encoding is a roughness-aware directional representation in which a reflected direction is projected onto a spherical parameterization and queried against a learnable multi-level feature hierarchy whose third interpolation axis is roughness. In the literature provided here, it appears first as the core directional encoding mechanism of Ref-GS, where deferred shading on a per-pixel surface estimate is used to reduce the ambiguity between surface orientation and viewing direction in Gaussian splatting (Zhang et al., 2024). In RGS-DR, the same general encoding principle is adapted as a residual appearance module layered on top of an explicit deferred inverse-rendering pipeline, where it refines reconstruction by modeling inter-reflections and other complex view-dependent effects while remaining excluded from relighting because it is illumination-specific (Kouros et al., 25 Apr 2025).

1. Definition and representational scope

In Ref-GS, spherical mipmap-based directional encoding is introduced to address a specific weakness of Gaussian splatting for specular and refractive effects: standard per-primitive view-direction queries are too ambiguous to reliably separate surface orientation from viewing direction, and they do not naturally represent appearance variation with roughness (Zhang et al., 2024). The encoding therefore moves directional reasoning out of primitive-level forward rendering and into deferred shading on a per-pixel surface estimate.

The directional feature is obtained from the reflected direction $\omega_r$, surface roughness $\rho$, and a learnable multi-level spherical feature grid $\mathcal{M}$: $$\mathbf{s} = \operatorname{Sph\text{-}Mip}(\omega_r, \rho, \mathcal{M}).$$ Here, the representation is not merely directional. It is explicitly roughness-aware, because roughness is treated as an interpolation coordinate in the spherical mip hierarchy rather than as an auxiliary scalar passed independently to the shader (Zhang et al., 2024).

RGS-DR adopts the same encoding principle in a narrower role. There, the spherical mipmap-based directional encoding is not the primary lighting model, but a learned residual correction mechanism that improves reconstruction quality by capturing effects not explained by the explicit lighting branch (Kouros et al., 25 Apr 2025). This establishes an important distinction: in Ref-GS the encoding is part of the main rendering factorization, whereas in RGS-DR it is a refinement module.

2. Deferred rendering as the precondition for directional encoding

The encoding is tied closely to deferred rendering in both systems. In Ref-GS, the method uses a three-stage pipeline. First, 2D Gaussian primitives are alpha-blended into per-pixel buffers containing diffuse color, learned feature $\mathbf{K}$, roughness map $\mathbf{M}$, and normal map $\mathbf{N}$. Second, the reflected direction is encoded through the spherical Mip-grid. Third, a lightweight MLP shader predicts the view-dependent component from the directional feature and its interaction with the learned spatial feature (Zhang et al., 2024).

The deferred shading formulation is written as

$$\mathbf{I} = \mathbf{I}_d + f_\Theta(\mathbf{S}, \mathbf{K} \otimes \mathbf{S}),$$

and in the supplementary as

$$\mathbf{I} = \gamma(\mathbf{I}_d + f_\Theta(\mathbf{S}, \mathbf{K} \otimes \mathbf{S})).$$

This postponement of directional evaluation is the mechanism by which Ref-GS “effectively reduces the ambiguity between orientation and viewing angle” (Zhang et al., 2024).

The motivation is an ambiguity analysis specific to Gaussian splatting. In vanilla 3DGS and 2DGS, each primitive carries its own SH-based view-dependent color and is composited directly along the ray. Under that formulation, high-frequency reflections can be “explained away” by changing primitive orientation or SH coefficients, especially because primitives are treated independently (Zhang et al., 2024). By first blending Gaussian attributes into a surface-like G-buffer and only then evaluating view-dependent color, Ref-GS makes the reflection direction a property of the inferred surface point rather than of an individual Gaussian disk.

RGS-DR applies the same deferred principle to 2D Gaussian surfels. Its G-buffer contains diffuse map $\mathbf{I}_d$, roughness map $\rho$0, specular tint map $\rho$1, feature map $\rho$2, normal map $\rho$3, and alpha map $\rho$4 (Kouros et al., 25 Apr 2025). The paper emphasizes that this deferred scheme computes lighting only on visible parts, uses smoother normals from 2D surfels, and avoids alpha-blending accumulation artifacts. In that setting, the residual directional encoding operates on per-pixel material and geometric state rather than on per-Gaussian outputs before blending.

3. Spherical Mip-grid construction and query mechanics

In Ref-GS, the spherical Mip-grid is a learnable, multi-resolution spherical feature grid for far-field lighting and roughness-aware reflection. It is laid out in longitude-latitude coordinates and unfolded into a 2D texture-like grid for efficient indexing (Zhang et al., 2024). The reflected direction $\rho$5 is mapped to spherical angles

$\rho$6

The grid is effectively three-dimensional: $\rho$7 are the directional axes and the third axis is roughness $\rho$8.

The base level is

$\rho$9

and higher levels $\mathcal{M}$0 are downsampled by $\mathcal{M}$1 in both height and width. The implementation parameters reported for Ref-GS are:

• $\mathcal{M}$2,
• $\mathcal{M}$3,
• $\mathcal{M}$4,
• number of levels $\mathcal{M}$5.

At query time, all mip levels are resized to a common resolution, and feature lookup is performed with trilinear interpolation over $\mathcal{M}$6 in the fragment shader (Zhang et al., 2024). The paper states that this makes the directional code efficient and differentiable. The implementation is in PyTorch, with Nvdiffrast used for efficient mipmap querying.

RGS-DR describes the same query pattern in the residual pass. For a screen-space point $\mathcal{M}$7, it uses viewing direction $\mathcal{M}$8, normal $\mathcal{M}$9, roughness $$\mathbf{s} = \operatorname{Sph\text{-}Mip}(\omega_r, \rho, \mathcal{M}).$$0, and learned feature $$\mathbf{s} = \operatorname{Sph\text{-}Mip}(\omega_r, \rho, \mathcal{M}).$$1. The reflection direction is

$$\mathbf{s} = \operatorname{Sph\text{-}Mip}(\omega_r, \rho, \mathcal{M}).$$2

which is converted to spherical coordinates $$\mathbf{s} = \operatorname{Sph\text{-}Mip}(\omega_r, \rho, \mathcal{M}).$$3, and then queried as

$$\mathbf{s} = \operatorname{Sph\text{-}Mip}(\omega_r, \rho, \mathcal{M}).$$4

by trilinear interpolation over the spherical mip hierarchy $$\mathbf{s} = \operatorname{Sph\text{-}Mip}(\omega_r, \rho, \mathcal{M}).$$5 (Kouros et al., 25 Apr 2025).

A concise comparison of the two uses is helpful:

System	Role of spherical mipmap-based directional encoding	Relation to lighting
Ref-GS	Core directional encoding in deferred Gaussian shading	Part of the main view-dependent rendering factorization
RGS-DR	Residual appearance correction in a deferred inverse-rendering pipeline	Complements explicit lighting; omitted during relighting

This comparison suggests that the same representational primitive supports two distinct design choices: a primary appearance model in one system and a residual correction layer in the other.

4. Roughness awareness and geometry-lighting factorization

A central property of the encoding is that it is roughness-aware by construction. In Ref-GS, roughness $$\mathbf{s} = \operatorname{Sph\text{-}Mip}(\omega_r, \rho, \mathcal{M}).$$6 is treated as a per-Gaussian attribute, splatted into a roughness map $$\mathbf{s} = \operatorname{Sph\text{-}Mip}(\omega_r, \rho, \mathcal{M}).$$7, and used as the third coordinate in the spherical Mip-grid (Zhang et al., 2024). The paper explicitly states the intended behavior:

• low roughness samples finer, sharper directional features,
• high roughness samples coarser features from lower mip levels, producing broader, blurrier specular responses.

This gives the representation a built-in mechanism for varying material blur. The supplementary figure on editing further notes that adjusting $$\mathbf{s} = \operatorname{Sph\text{-}Mip}(\omega_r, \rho, \mathcal{M}).$$8 changes the directional feature obtained from $$\mathbf{s} = \operatorname{Sph\text{-}Mip}(\omega_r, \rho, \mathcal{M}).$$9 interpolation, which confirms support for material editing and roughness variation (Zhang et al., 2024).

Ref-GS combines the directional feature with a learned spatial feature using an outer product: $\mathbf{K}$0 and, in the directional factorization description, a low-rank model

$\mathbf{K}$1

Here, $\mathbf{K}$2 is the spatial feature and $\mathbf{K}$3 is the directional feature. The outer product creates a block matrix that is flattened and fed to a lightweight MLP decoder (Zhang et al., 2024). The stated motivation is to separate geometry and material cues from lighting cues while still allowing interaction, thereby reducing feature-channel overhead relative to a monolithic high-dimensional vector. The paper further states that this helps capture global lighting, shadows, and self-occlusion, which is especially useful for near-field inter-reflections.

RGS-DR uses the same multiplicative interaction structure in its residual branch: $\mathbf{K}$4 and at image level,

$\mathbf{K}$5

The residual term appears in the full image decomposition as

$\mathbf{K}$6

with

$\mathbf{K}$7

In that formulation, the spherical code supplies a direction- and roughness-aware feature vector, and the outer product provides a richer modulation of direction-dependent effects with scene-specific learned features (Kouros et al., 25 Apr 2025).

5. Relation to prior directional encodings and lighting models

Ref-GS positions its spherical Mip-grid against several existing families of appearance models (Zhang et al., 2024). SH-based view-dependent color in 3DGS and 2DGS is described as fast but weak for complex reflections. Ref-NeRF and IDE-style reflection direction encodings are described as more effective in NeRF, but less effective in Gaussian splatting because primitive-wise independence and orientation ambiguity remain unresolved. The paper also contrasts its design with Gaussian Directional Encoding, Neural Directional Encoding, and other view-dependent Gaussian methods that rely on stronger MLP-based models or different lighting assumptions.

The encoding is also differentiated from global environment map approaches and other far-field lighting methods. Those methods assume distant illumination and therefore struggle with near-field effects, whereas the Ref-GS factorization is stated to support both near-field and far-field view-dependent effects (Zhang et al., 2024). The “Mip” aspect is explicitly related to Mip-NeRF-style multiscale representations, but the multiscale idea is applied in spherical directional space rather than along rays in 3D spatial space.

The paper identifies the novelty of its design as the combination of four elements: spherical directional indexing, roughness as a mip dimension, deferred surface-based query, and outer-product geometry-lighting factorization (Zhang et al., 2024). A common misconception would be to view the method as only a substitute for SH coefficients. The provided material instead indicates that the encoding is inseparable from the deferred, surface-based evaluation strategy that turns directional appearance into a property of the blended surface proxy.

RGS-DR adopts the directional encoding from Ref-GS but places it in a different architectural position (Kouros et al., 25 Apr 2025). The paper explicitly notes that Ref-GS has a similar directional factorization idea, but that its environment illumination is baked into the mipmap and is therefore not relightable. RGS-DR differs by using mipmap-based directional encoding only in the residual branch while keeping the lighting model explicit and editable. This separation preserves relightability.

6. Empirical behavior, limitations, and interpretive boundaries

The empirical findings reported for Ref-GS attribute several benefits to spherical mipmap-based directional encoding: better handling of high-frequency specular reflections, improved modeling of rough and smooth surfaces through mip levels, improved geometry recovery because shading is deferred to a surface proxy instead of being baked into primitive radiance, support for near-field and far-field view-dependent effects, and relatively efficient lookup via texture-style interpolation (Zhang et al., 2024). The ablations reported in the paper support the design: replacing $\mathbf{K}$8 with direct G-buffer and view-direction input degrades performance; removing the mipmap especially harms rough surfaces and introduces artifacts; removing deferred shading hurts reflection accuracy; and removing $\mathbf{K}$9 weakens near-field inter-reflection modeling.

Quantitatively, Ref-GS is reported to achieve the best or near-best results on reflective and glossy benchmarks, including strong improvements on Shiny Blender and Shiny Real and competitive results on NeRF Synthetic and Glossy Synthetic (Zhang et al., 2024). The paper highlights sharp reflections such as tree branches and buildings reflected in spheres, as well as better normal accuracy than competing Gaussian methods. A limitation noted in the supplementary is that the neural decoder makes rendering slower than plain 2DGS and harder to integrate directly into standard graphics engines, although the method remains real-time or near real-time.

For RGS-DR, the ablation table provides explicit metrics. A version “w/o mipmap” reports PSNR $\mathbf{M}$0, SSIM $\mathbf{M}$1, and LPIPS $\mathbf{M}$2, while the full model reports PSNR $\mathbf{M}$3, SSIM $\mathbf{M}$4, and LPIPS $\mathbf{M}$5 (Kouros et al., 25 Apr 2025). Removing the residual pass gives PSNR $\mathbf{M}$6, SSIM $\mathbf{M}$7, and LPIPS $\mathbf{M}$8, compared with the same full-model values. The paper frames the residual branch as improving photorealistic reconstruction, especially visible qualitatively on shiny details. It also reports sharper reflections and less blurring or distortion, particularly on scenes such as car, helmet, and the Glossy Synthetic set.

An important interpretive boundary is stated explicitly in RGS-DR: the residual term is illumination-specific and is not included when performing relighting with new environment maps (Kouros et al., 25 Apr 2025). Thus, the spherical mipmap-based directional encoding in that system improves reconstruction and novel view synthesis, but is deliberately excluded from relighting to avoid baking in scene-specific lighting artifacts. This suggests a broader methodological split within deferred Gaussian rendering: one can either place spherical mipmap-based directional encoding at the center of the appearance model, as in Ref-GS, or confine it to a residual refinement role, as in RGS-DR, depending on whether the priority is direct photorealistic reconstruction or explicit relightability.