Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning (1904.01786v1)

Abstract: Rendering bridges the gap between 2D vision and 3D scenes by simulating the physical process of image formation. By inverting such renderer, one can think of a learning approach to infer 3D information from 2D images. However, standard graphics renderers involve a fundamental discretization step called rasterization, which prevents the rendering process to be differentiable, hence able to be learned. Unlike the state-of-the-art differentiable renderers, which only approximate the rendering gradient in the back propagation, we propose a truly differentiable rendering framework that is able to (1) directly render colorized mesh using differentiable functions and (2) back-propagate efficient supervision signals to mesh vertices and their attributes from various forms of image representations, including silhouette, shading and color images. The key to our framework is a novel formulation that views rendering as an aggregation function that fuses the probabilistic contributions of all mesh triangles with respect to the rendered pixels. Such formulation enables our framework to flow gradients to the occluded and far-range vertices, which cannot be achieved by the previous state-of-the-arts. We show that by using the proposed renderer, one can achieve significant improvement in 3D unsupervised single-view reconstruction both qualitatively and quantitatively. Experiments also demonstrate that our approach is able to handle the challenging tasks in image-based shape fitting, which remain nontrivial to existing differentiable renderers.

• The paper introduces Soft Rasterizer, a differentiable rendering framework that models rasterization as a soft probabilistic process to enable gradient flow.
• It employs a novel aggregation function that blends per-triangle color and depth information, facilitating backpropagation even to occluded vertices.
• Experiments demonstrate enhanced 3D reconstruction and robust shape fitting, outperforming prior methods in recovering fine details and handling occlusions.

Soft Rasterizer: A Differentiable Renderer for Image-Based 3D Reasoning

The paper "Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning" presents a novel framework that addresses the limitations of conventional non-differentiable rendering processes by introducing a truly differentiable rendering approach. The core of this work revolves around overcoming the discrete nature of standard rasterization and enabling end-to-end gradient flow for tasks involving 3D reasoning from 2D images.

Key Contributions

The primary contribution of this paper is the Soft Rasterizer (SoftRas), a differentiable rendering framework that models the rendering process as a probabilistic aggregation of contributions from all mesh triangles. This approach allows for efficient gradient back-propagation to mesh vertices and their attributes, encompassing silhouette, shading, and color images.

1. Differentiable Rendering:
   • The traditional rasterization process is non-differentiable due to discrete operations. SoftRas circumvents this by modeling rendering as a soft probabilistic process.
   • It computes a probability map for each triangle, capturing the likelihood of each triangle contributing to a pixel.
2. Novel Aggregation Function:
   • The proposed framework uses a differentiable aggregation function that combines per-triangle color maps based on the probability maps and relative depths.
   • This allows soft blending of regions covered by different triangles, enabling gradient flow even to occluded mesh vertices.
3. End-to-End 3D Reasoning:
   • SoftRas can be seamlessly integrated into neural networks, facilitating end-to-end training for tasks such as single-view 3D reconstruction and shape fitting without requiring 3D supervision.

Evaluation and Results

The paper demonstrates the effectiveness of SoftRas through experiments on 3D unsupervised single-view reconstruction and image-based shape fitting:

• Single-View Reconstruction:
  • SoftRas significantly improves performance on standard reconstruction metrics such as the 3D Intersection over Union (IoU).
  • It achieves superior qualitative and quantitative results compared to existing methods like Neural Mesh Renderer (NMR) and Pixel2Mesh, especially in recovering fine details and handling complex shapes.
• Image-Based Shape Fitting:
  • The framework successfully tackles the challenges of occlusion and local minima, showing its robustness in pose estimation tasks.
  • SoftRas facilitates smoother energy landscapes, contributing to a more stable optimization process.

Implications and Future Directions

This work has several implications for the field of computer vision and 3D graphics:

• Broader Application Scope:
  • The differentiability provided by SoftRas opens avenues for various applications beyond reconstruction, such as real-time graphics, shape analysis, and virtual/augmented reality.
• End-to-End Learning Capabilities:
  • By enabling differentiable rendering, SoftRas paves the way for developing more sophisticated models that can be trained end-to-end with pixel-level supervision.
• Future Developments:
  • There is potential to explore other forms of distance and aggregation functions that may refine rendering accuracy.
  • The integration of SoftRas with more diverse neural architectures could enhance its adaptability to different 3D reasoning tasks.

In summary, the introduction of Soft Rasterizer stands as a significant step towards more flexible and efficient frameworks for image-based 3D reasoning, offering robust solutions to previously intractable problems in differentiable rendering.