Differentiable Rendering Techniques

• Differentiable rendering techniques are frameworks that compute gradients of rendering outputs with respect to scene parameters, enabling inverse graphics and efficient 3D reconstruction.
• Recent advances leverage neural implicit representations such as Signed Directional Distance Functions to model complex geometries with analytical guarantees and learn from partial depth data.
• Practical applications include shape inference and category-level modeling using depth or Lidar inputs, though challenges remain for complex topologies and infinite-distance predictions.

Differentiable rendering techniques are computational frameworks that enable the calculation of gradients of rendering outputs with respect to scene parameters, facilitating inverse graphics tasks such as 3D reconstruction, shape inference, and optimization of object representations using only indirect supervision such as depth or occupancy maps. Recent advances include the development of neural, implicit representations capable of synthesizing novel views, predicting distance measurements, and reconstructing object surfaces in a continuous and differentiable manner. One significant approach deploys deep networks to represent Signed Directional Distance Functions (SDDFs), enabling geometry-aware modeling, efficient learning from partial information, and precise analytical guarantees (Zobeidi et al., 2021).

1. Mathematical Foundations

Differentiable rendering utilizes mathematical models that support the computation of derivatives through the rendering process. Central to high-fidelity 3D object representation is the Signed Distance Function (SDF), mapping a point in $\mathbb{R}^3$ to the closest distance (with sign) to a surface. The SDDF generalizes the SDF by associating with each spatial point $p\in\mathbb{R}^3$ and unit direction $d\in S^2$ a signed distance along $d$ to the boundary $\partial O$ of a closed object $O$:

$h(p, d) := d_{(d)}(p, \partial O)$

where $$d_{(d)}(p, \partial O) = \min\{ t \in \mathbb{R} \mid p + t\,d \in \partial O \}$$. For the special case $d = e_3 = (0,0,1)$, $h(p,e_3)$ represents the signed distance along the $p\in\mathbb{R}^3$0 axis and defines the Z-monotonic SDF $p\in\mathbb{R}^3$1 (Zobeidi et al., 2021).

2. The Directional Eikonal Constraint

A critical geometric constraint in SDDF models is the directional Eikonal constraint, which enforces monotonicity along the direction of interest. For valid SDDFs, this states:

$p\in\mathbb{R}^3$2

for all $p\in\mathbb{R}^3$3 such that the ray $p\in\mathbb{R}^3$4 hits the same surface point. In differential form,

$p\in\mathbb{R}^3$5

For the Z-monotonic case, this reduces to the constraint $p\in\mathbb{R}^3$6, ensuring linear decrease of the SDDF along $p\in\mathbb{R}^3$7.

3. Neural Network Architecture and Encoding

Network architectures for learning SDDFs must, by construction, enforce the directional Eikonal constraint. The approach defines $p\in\mathbb{R}^3$8 such that $p\in\mathbb{R}^3$9. Generally, $d\in S^2$0 is rotated so $d\in S^2$1 aligns with the canonical axis, and the last coordinate of $d\in S^2$2 is ignored, forming $d\in S^2$3 with learnable $d\in S^2$4. In the Z-monotonic case, $d\in S^2$5 is the identity, $d\in S^2$6 projects $d\in S^2$7, and input to the MLP network comprises $d\in S^2$8 for optional latent code $d\in S^2$9. The network outputs $d$0, which is related to the SDDF through a strictly-monotonic squashing function $d$1 (e.g. $d$2):

$d$3

This network enforces the desired structure by design, rather than relying solely on data-driven learning (Zobeidi et al., 2021).

4. Training and Loss Function

SDDF models are trained using distance measurements from depth or Lidar sensors. Data is collected as triplets $d$4 and split into finite (F) and infinite (I) distance sets. The chosen loss (Eq. 10 in (Zobeidi et al., 2021)) for parameters $d$5 is: \begin{align*} \ell(\Theta; F, I) =\ &\alpha |F|^{-1} \sum_{(p,d,d_\text{true}) \in F} |\varphi(d_\text{true} + p^\top d) - q_\Theta(p,d)|^p \ &+ \beta |I|^{-1} \sum_{(p,d,\infty)\in I} r\big(\varphi(\infty) - q_\Theta(p,d)\big)^p \ &+ \gamma |\Theta|^p \end{align*} where $d$6 is $d$7 or softplus. For category-level learning, an additional regularizer $d$8 is applied to the latent codes. Only depth supervision is required; no RGB or mesh ground truth is needed.

5. Analytical Guarantees

The model yields analytical guarantees via its structural constraints:

• The directional Eikonal property is satisfied exactly by construction (Lemma 1, Eq. 2 in (Zobeidi et al., 2021)).
• Squashing with any strictly-monotonic function $d$9 preserves the required property (Lemma 4).
• Proposition 1 affirms that the reconstructed $\partial O$0 is a valid SDDF, ensuring linear decrease along direction $\partial O$1 with constant gradient.
• Prediction error is independent of distance to the surface, making dense sampling near the surface unnecessary and affording confidence in distant predictions.

6. Implementation: Training and Inference

The Z-monotonic SDDF can be implemented with the following algorithmic sketch.

Algorithm 1: Training

• Input: Training sets $\partial O$2, $\partial O$3
• Initialization: Network parameters $\partial O$4 (and latent codes $\partial O$5 for category-level)
• Repeat until convergence:
  • Sample minibatch $\partial O$6, $\partial O$7
  • Compute $\partial O$8 for $\partial O$9
  • Compute $O$0
  • Compute $O$1
  • Regularizer $O$2 (plus $O$3 when needed)
  • Loss $O$4
  • Update $O$5 ($O$6 for category-level)

Algorithm 2: Inference and Surface Extraction

• Given trained $O$7, and test $O$8,
  • To query SDF at $O$9:
  • $h(p, d) := d_{(d)}(p, \partial O)$0
  • $h(p, d) := d_{(d)}(p, \partial O)$1
  • For mesh extraction:
  • Evaluate $h(p, d) := d_{(d)}(p, \partial O)$2 on a 3D grid
  • Run Marching Cubes on grid of $h(p, d) := d_{(d)}(p, \partial O)$3 values to extract the surface mesh

7. Applications, Limitations, and Considerations

Differentiable rendering techniques based on SDDFs efficiently learn from partial, unstructured measurements, offering a direct connection to sensor modalities such as depth cameras or Lidar. These paradigms enable representation and generalization of entire shape categories, surface interpolation from incomplete data, and obviate mesh-based supervision or explicit geometry at training time. However, the Z-monotonic SDF only captures distances along the $h(p, d) := d_{(d)}(p, \partial O)$4 axis; for objects with complex topology (e.g., overhangs), some ray queries may yield infinite distance. Valid training requires rays with both finite and infinite intersections, and only depth data is supported (no RGB cues). Extraction with Marching Cubes inherits resolution and smoothness tradeoffs. The use of a scalar squashing $h(p, d) := d_{(d)}(p, \partial O)$5 and infinite-distance caps introduces slight bias near $h(p, d) := d_{(d)}(p, \partial O)$6, necessitating careful selection (e.g., $h(p, d) := d_{(d)}(p, \partial O)$7 or $h(p, d) := d_{(d)}(p, \partial O)$8). For category-level models, code optimization at test time requires suitable initialization.