Soft Rasterizer: Differentiable 3D Rendering

• Soft Rasterizer (SoftRas) is a fully differentiable rendering framework that converts 3D triangle meshes into 2D images using continuous, probabilistic formulations.
• It replaces discrete rasterization steps with smooth gradients, enabling end-to-end gradient-based optimization for tasks like single-view mesh reconstruction and image-based shape fitting.
• Empirical evaluations on ShapeNet show superior performance in 3D IoU and reconstruction quality compared to previous methods, advancing unsupervised 3D reasoning.

Soft Rasterizer (SoftRas) is a fully differentiable rendering framework for 2D image synthesis from 3D triangle meshes, specifically constructed to enable end-to-end gradient-based optimization in computer vision and graphics tasks such as single-view mesh reconstruction and image-based shape fitting. Standard rasterization in graphics pipelines is inherently non-differentiable due to discrete coverage and depth selection operations. SoftRas replaces these steps with continuous, probabilistic formulations, ensuring consistent, analytic gradients in both forward and backward passes. This design enables backpropagation of pixel-level losses directly to mesh vertex positions and attributes, even for occluded geometry, thus advancing both unsupervised and supervised 3D reasoning from images (Liu et al., 2019, &&&1&&&).

1. Motivation and Problem Setting

Classical rasterization is a two-stage process involving a hard inside/outside test for each pixel-triangle pair and a z-buffer-based winner-take-all for selecting the visible triangle at each pixel. Both steps are discrete, yielding zero gradient with respect to underlying mesh vertex positions or per-vertex attributes almost everywhere; thus, direct optimization of 3D meshes from 2D images using gradient-based methods is precluded (Liu et al., 2019, Liu et al., 2019).

Prior differentiable renderers such as OpenDR and Neural Mesh Renderer (NMR) circumvent this limitation by combining a non-differentiable forward pass with handcrafted, approximate backward gradients, resulting in inconsistencies and poor optimization dynamics, especially regarding occluded or far-range geometry (Liu et al., 2019). SoftRas addresses this by making both the forward and backward passes intrinsically and exactly differentiable via a probabilistic aggregation formulation, allowing for robust end-to-end learning and optimization.

2. Mathematical Formulation

The core differentiable operator in SoftRas replaces binary triangle coverage with a per-pixel per-triangle occupancy probability: $$D_i^t = \sigma\left(\delta_{t,i} \frac{d(i,t)^2}{\tau}\right),\quad \sigma(x) = \frac{1}{1+\exp(-x)}$$ where $\delta_{t,i}=+1$ if pixel $i$ is inside triangle $t$ and $-1$ otherwise, $d(i,t)$ is the minimum Euclidean distance from $i$ to the boundary of $t$, and $\tau$ is a sharpness hyperparameter (also denoted $\sigma$ in some works).

Silhouette probability at each pixel is computed by a differentiable relaxation of logical-OR across all triangles: $S_i = 1 - \prod_{t=1}^T (1 - D_i^t)$ This soft aggregation is extended for color/shading by introducing a depth-based softmax weight: $$w_i^t = \frac{D_i^t\,e^{z_t/\gamma}}{\sum_k D_i^k\,e^{z_k/\gamma} + \tilde w_i^b}$$ with $z_t$ as the normalized (inverse) depth and $\gamma$ the depth softness temperature. The rendered pixel color is then a weighted sum over all triangles' contributions plus (optionally) a background term. As $\tau\to 0$ and $\gamma\to 0$, SoftRas recovers the standard hard binary rasterization and z-buffer selection, respectively (Liu et al., 2019, Liu et al., 2019).

3. Differentiability and Gradient Flow

All steps in SoftRas's forward rendering pipeline are analytic functions of mesh vertex positions, enabling exact backpropagation. For silhouette-based learning, the chain rule gives: $$\frac{\partial S^i}{\partial v} = \sum_j \frac{\partial S^i}{\partial p_{ij}} \cdot \frac{\partial p_{ij}}{\partial d^2(i,j)} \cdot \frac{\partial d^2(i,j)}{\partial v}$$ Closed-form gradients are derived for every subexpression, including per-pixel per-triangle coverage, per-vertex positions, and the chain derivatives through the sigmoid and distance measures. Importantly, since $D_i^t > 0$ for all $(i, t)$ even for occluded or distant triangles, every mesh vertex receives nonzero gradients, and z-coordinates are explicitly handled. Control parameters $\tau$ (sharpness) and $\gamma$ (soft z-buffer) allow for annealing between soft and hard regimes, aiding convergence and avoiding local minima (Liu et al., 2019, Liu et al., 2019).

4. Integration into Deep Learning Frameworks

SoftRas is implemented as a modular layer in deep neural architectures, typically serving as a differentiable renderer downstream of a mesh generator network. For unsupervised 3D mesh reconstruction, the standard pipeline is as follows:

• Input: Single-view RGB image, optionally with ground-truth silhouette.
• Mesh Generator $\mathcal{G}_\theta$: Encoder-decoder network outputs per-vertex 3D displacements $\Delta V$ over a fixed topology template (e.g., sphere), resulting in a mesh $M$.
• Soft Rasterizer $\mathcal{R}$: Projects $M$ given camera parameters, computes coverage probabilities and aggregates output as a soft silhouette (or color image).
• Loss $\mathcal{L}$: Silhouette loss (e.g., intersection-over-union), laplacian mesh regularity, and flattening for face coplanarity.

Total loss is expressed as: $L = L_{IoU} + \lambda L_{lap} + \mu L_{fl}$ where $L_{IoU}$ is the intersection-over-union loss for silhouettes, $L_{lap}$ is the Laplacian smoothing loss, and $L_{fl}$ is the flattening regularization on adjacent faces (Liu et al., 2019). $\lambda$ and $\mu$ are determined via cross-validation.

SoftRas can be extended for color learning by attaching an $\ell_2$ photometric loss between rendered and reference RGB images, associating mesh vertices with color or UV attributes (Liu et al., 2019).

5. Empirical Performance and Applications

Extensive evaluation on ShapeNet across 13 object categories (using 64×64 renders and standard split) demonstrates SoftRas's superiority over previous unsupervised methods in 3D IoU, with mean scores of 0.623 vs 0.602 for N3MR and 0.574 for the Perspective Transformer (Liu et al., 2019, Liu et al., 2019). Qualitative improvements include smoother surface reconstructions, faithful recovery of thin structures, and fewer mesh self-intersections, despite training only on silhouettes. When compared to the supervised Pixel2Mesh (which uses 3D ground-truth), SoftRas is competitive and often outperforms on real-world data.

SoftRas's fully differentiable formulation also offers advantages in tasks requiring shape fitting to images. In rigid pose estimation, SoftRas reduces mean angular error compared to prior renderers; in non-rigid human body fitting under heavy occlusion, it can recover correct plausible poses by delivering gradients to all relevant geometry, unlike methods where occluded parts receive no supervision (Liu et al., 2019).

6. Limitations and Extensions

Silhouette-only supervision can create geometric ambiguities, such as the inability to reconstruct non-genus-0 surfaces or ambiguous planar regions. This can be partially alleviated by multi-view training (Liu et al., 2019). The framework does not natively encode shading or texture, missing rich geometric cues in color images; extending SoftRas with differentiable color rendering and photometric losses offers a plausible solution (Liu et al., 2019).

Other soft aggregation strategies, such as differentiable z-buffering for full RGBA rendering, represent natural directions. Handling arbitrary mesh topology or dynamic deforming meshes remains an open challenge. SoftRas enables gradients to occluded and background geometry, which enhances robustness and avoids the zero-gradient issues of prior techniques (Liu et al., 2019).

7. Context and Impact within Differentiable Rendering

Soft Rasterizer marks a foundational shift from approximate-gradient differentiable rendering to analytic, forward-backward consistent operators. Its probabilistic triangle coverage and soft depth selection yield pixel-wise gradients that inform all geometric and appearance parameters of the mesh, supporting a diverse range of learning, optimization, and fitting tasks in both supervised and unsupervised settings. By unifying color, shading, and silhouette rendering under a continuous framework, SoftRas has influenced subsequent work in neural rendering, mesh optimization, and general-purpose 3D-to-2D differentiable computation (Liu et al., 2019, Liu et al., 2019).