Differentiable Poisson Surface Reconstruction (DPSR)

• DPSR is a framework that reformulates classical Poisson surface reconstruction into an end-to-end differentiable pipeline combining neural networks with numerical solvers.
• It integrates techniques such as FFT-based solvers, MLP-parameterized implicit representations, and Fourier Neural Operators to enhance 3D reconstruction fidelity.
• The method enables geometry optimization via backpropagation with robust loss functions, improving noise resilience and allowing real-time surface updates.

Differentiable Poisson Surface Reconstruction (DPSR) refers to a class of methods that cast classical Poisson surface reconstruction—traditionally used to reconstruct 3D surfaces from point clouds—into a form amenable to end-to-end gradient-based learning. These frameworks replace or augment the standard linear Poisson system with either neural approximations, differentiable numerical solvers, or hybrid operator-parametric formulations, allowing geometry to be optimized with respect to arbitrary loss functions by means of backpropagation. DPSR underpins current research in integrating explicit surface reconstruction with neural networks, including implicit neural representations (INRs), Fourier Neural Operators (FNOs), and differentiable rendering-based pipelines.

1. Formalism: Poisson Surface Reconstruction and Its Differentiable Extensions

Classical Poisson surface reconstruction seeks a scalar field $\phi : \Omega \to \mathbb{R}$ whose zero level set approximates the target surface, typically given as an oriented point cloud $P = \{ (\mathbf{x}_i, \mathbf{n}_i) \}$. The classical PDE is given by

$$\Delta \phi(\mathbf{x}) = \nabla \cdot V(\mathbf{x}),$$

where $V$ is a vector field constructed from the point normals. The reconstructed surface is extracted as the isosurface $\phi = 0$ or another chosen threshold.

DPSR frameworks reformulate each component of this pipeline, prioritizing differentiability throughout. Key strategies include:

• Embedding the linear Poisson system as a differentiable layer in a deep network (Chen et al., 2023, Lin et al., 2022, Peng et al., 2021).
• Representing $V$ and $\phi$ either on regular grids for FFT-based solvers (Lin et al., 2022, Peng et al., 2021) or using operator-learning architectures such as neural networks (Andrade-Loarca et al., 2023, Park et al., 2023).
• Introducing generalizations to nonlinear variants like the $p$-Laplace (“$p$-Poisson”) PDE for enhanced control of reconstructed SDF regularity (Park et al., 2023).
• Employing variable-splitting, curl-free constraints, and auxiliary potentials to stabilize optimization and enforce physical PDE properties (Park et al., 2023).

The differentiability is ensured by restricting all operations—including sparse solver steps, FFTs, marching cubes, and vector field rasterization—to autodiff-compatible primitives.

2. Computational Schemes and Architectures

DPSR implementations span several discretization and operator-learning modalities, as summarized below.

Discretization Methods

Framework/Paper	$\phi$ Representation	Core Solver	Vector Field $P = \{ (\mathbf{x}_i, \mathbf{n}_i) \}$0
Shape As Points (Peng et al., 2021)	Regular 3D grid	FFT-based Poisson	Trilinear splat of normals
Diff. Rendering (Lin et al., 2022)	Regular 3D grid (coarse/fine)	FFT spectral, periodic	Gaussian-splat oriented pts
GradientSurf (Chen et al., 2023)	Regular 3D voxels	Sparse linear system, multigrid	On-the-fly from SLAM model
nPSR (Andrade-Loarca et al., 2023)	3D grid, arbitrary res; FNO outputs	Neural FNO, spectral	Rasterized or smoothed
PINC (Park et al., 2023)	MLP-parameterized (INR)	Hard constraint, algebraic	Implicit via SDF gradient/potentials

Network-Integrated Features

• PINC uses an MLP with shared encoding for the SDF $P = \{ (\mathbf{x}_i, \mathbf{n}_i) \}$1, vector potential $P = \{ (\mathbf{x}_i, \mathbf{n}_i) \}$2, and curl-free auxiliary field, directly enforcing the nonlinear $P = \{ (\mathbf{x}_i, \mathbf{n}_i) \}$3-Poisson PDE and curl constraints (Park et al., 2023).
• nPSR deploys a Fourier Neural Operator to achieve “resolution-agnostic” shape reconstruction, enabling super-resolution (Andrade-Loarca et al., 2023).
• GradientSurf and Shape As Points employ differentiable linear solvers (multigrid/FFT) wrapped in autodiff frameworks, supporting efficient backpropagation from downstream mesh or rendering losses (Chen et al., 2023, Peng et al., 2021).
• Differentiable Marching Cubes is used for mesh extraction, with custom backward passes to propagate gradients to grid representations of $P = \{ (\mathbf{x}_i, \mathbf{n}_i) \}$4 (Lin et al., 2022).

3. Loss Functions and Supervision Strategies

Losses in DPSR frameworks are directly tied to the PDE residuals, imposed priors, and downstream geometric criteria.

• PDE Residuals: Supervision of the Laplacian residual (classical Poisson) or its generalization ($P = \{ (\mathbf{x}_i, \mathbf{n}_i) \}$5-Laplace) over sampled or collocated domain points (Park et al., 2023, Peng et al., 2021, Chen et al., 2023).
• Boundary/Isosurface Constraints: $P = \{ (\mathbf{x}_i, \mathbf{n}_i) \}$6 forces the SDF or indicator field to vanish on the sampled surface (Park et al., 2023, Chen et al., 2023).
• Gradient/Normal Matching: Penalty on the distance between computed gradients of $P = \{ (\mathbf{x}_i, \mathbf{n}_i) \}$7 and known or estimated normals (Chen et al., 2023).
• Curl-Free Regularization: Losses to penalize non-conservative auxiliary gradient fields, crucial in variable-splitting approaches (Park et al., 2023).
• Screening Terms: Zeroth-order fidelity on sampled points to localize the surface (Chen et al., 2023).
• Minimal Area: Regularization handling topology holes via fill-in criteria (Park et al., 2023).
• Downstream Losses: For differentiable rendering, depth, silhouette, and photometric consistency all contribute gradients through the DPSR module (Lin et al., 2022).

Hyperparameters (e.g., loss weights, solver tolerances, grid size) are empirically selected for task and architecture stability.

4. Differentiability and Backpropagation Through the Solver

Full differentiability is achieved in all frameworks by careful design of the Poisson solver and associated operations. Key principles are:

• Linear System Differentiation: Implicit or iterative differentiation for solvers $P = \{ (\mathbf{x}_i, \mathbf{n}_i) \}$8, using $P = \{ (\mathbf{x}_i, \mathbf{n}_i) \}$9 (Peng et al., 2021, Chen et al., 2023).
• Operator-Learned Solvers: In nPSR, every block of the Fourier Neural Operator—FFT, complex multiplication, pointwise nonlinearity—is natively compatible with autodiff (Andrade-Loarca et al., 2023).
• Mesh Extraction: Gradients from loss functions on mesh vertices propagate to the volumetric field via differentiable marching cubes, which is commonly approximated using local surface normals (Peng et al., 2021, Lin et al., 2022).
• No Black-Box Solvers: PINC enforces all PDE conditions as algebraic or vector-field constraints within the MLP, avoiding unstable differentiation through high-order PDEs (Park et al., 2023).

The entire pipeline, including mesh extraction and any downstream geometric or photometric error, remains differentiable, enabling end-to-end learning and shape optimization.

5. Empirical Results and Performance Benchmarks

DPSR models are evaluated across surface reconstruction, multi-view geometry, and implicit shape learning tasks:

• Reconstruction Metrics: Two-sided Chamfer distance $$\Delta \phi(\mathbf{x}) = \nabla \cdot V(\mathbf{x}),$$0, Hausdorff $$\Delta \phi(\mathbf{x}) = \nabla \cdot V(\mathbf{x}),$$1, F-score, and normal consistency. PINC demonstrates state-of-the-art or on-par results with or without normals (Park et al., 2023).
• Generalization and Robustness: PINC achieves high-fidelity reconstruction, retaining detail (e.g., wing tips, bolt teeth) while remaining robust to noise and partial observation via algebraic and curl constraints (Park et al., 2023). nPSR attains order-of-magnitude improvements in low-data regimes and preserves geometric detail at higher resolutions without retraining (Andrade-Loarca et al., 2023).
• Efficiency: Shape As Points achieves %%%%21$P = \{ (\mathbf{x}_i, \mathbf{n}_i) \}$122%%%% speed-ups compared to neural implicit approaches (e.g., ConvONet) with acceleration from FFTs and optimized CUDA implementations (Peng et al., 2021). Real-time incremental surface updates are feasible on modern GPUs (Chen et al., 2023).
• Resolution Agnosticism: nPSR demonstrates “one-shot” super-resolution, training at 64$$\Delta \phi(\mathbf{x}) = \nabla \cdot V(\mathbf{x}),$$4 but evaluating at 128$$\Delta \phi(\mathbf{x}) = \nabla \cdot V(\mathbf{x}),$$5 with negligible loss in fidelity (Andrade-Loarca et al., 2023). Shape As Points and differentiable rendering approaches employ coarse-to-fine pipelines for improved coverage (Lin et al., 2022).

6. Limitations, Controversies, and Future Directions

Primary limitations and open challenges in DPSR research are:

• Scalability: Grid-based approaches scale cubically with resolution, constraining application to large scenes. Efficient domain decomposition (octrees) and adaptive methods remain active research targets (Peng et al., 2021).
• Topological Flexibility: Current DPSR frameworks, especially PINC, focus on closed surfaces; extension to open-surface or scene-level reconstructions is unresolved (Park et al., 2023).
• Autograd Overheads: Large $$\Delta \phi(\mathbf{x}) = \nabla \cdot V(\mathbf{x}),$$6-Laplace computations can be unstable; algebraic reformulation and variable splitting mitigate but do not eliminate this for increasingly high $$\Delta \phi(\mathbf{x}) = \nabla \cdot V(\mathbf{x}),$$7 or curl-based autodiff (Park et al., 2023).
• Data Requirements: Some methods still require oriented point samples or accurate normal fields; progress has been made on reconstruction without normals (e.g., PINC, Shape As Points) (Peng et al., 2021, Park et al., 2023).
• Rendering Coupling: Integration with photometric and silhouette-based differentiable rendering is promising but computationally intensive (Lin et al., 2022).
• Generalization: Applying learned operators over families of shapes (meta-SDF, multi-shape training) and across scale-space hierarchies remains an open direction (Park et al., 2023).