Deep Differentiable Simplex Layer (DDSL)

• Deep Differentiable Simplex Layer (DDSL) is a neural network module that enables differentiable rasterization of geometric signals defined on simplicial complexes into regular grids.
• It leverages a closed-form Non-Uniform Fourier Transform, spectral filtering, and inverse FFT to efficiently compute analytic gradients with respect to mesh parameters.
• DDSL supports arbitrary simplex orders and dimensions, facilitating advanced applications in mesh editing, shape optimization, and differentiable supervision for polygon generators.

The Deep Differentiable Simplex Layer (DDSL) is a neural network module enabling differentiable, efficient, and anti-aliased rasterization of geometric signals defined on simplicial complexes—such as point clouds, wires, triangle meshes, and tetrahedral meshes—into regular grids (pixels or voxels) suitable for deep learning. Distinct from prior differentiable rendering pipelines limited to projective 3D-to-2D settings or visibility/shading, DDSL generalizes to arbitrary simplex orders and ambient dimensions. Its mathematical core is the closed-form Non-Uniform Fourier Transform (NUFT) of piecewise-constant densities over simplices, spectral filtering to control aliasing, and an inverse FFT to obtain the raster. DDSL admits efficient, analytic gradient computation with respect to mesh vertex positions and simplex weights, thus facilitating end-to-end geometric learning, shape optimization, and differentiable supervision for polygon or mesh generators (Jiang et al., 2019).

1. Mathematical Framework and Formulation

DDSL operates on a homogeneous simplicial complex $\mathcal{S} = \{\Omega_n^j\}_{n=1}^N$ of $j$-simplices in $\mathbb{R}^d$, with each simplex $\Omega_n^j$ defined by vertices $\{\mathbf{x}_1,\dots,\mathbf{x}_{j+1}\}$. The piecewise-constant geometric signal is: $$f(\mathbf{x}) = \sum_{n=1}^N \rho_n\,\mathbf{1}_{\Omega_n^j}(\mathbf{x}),$$ where $\rho_n$ is the density on simplex $n$. The forward model applies the continuous NUFT: $$\hat{F}(\boldsymbol{\omega}) = \int_{\mathbb{R}^d} f(\mathbf{x})\,e^{-i\,\boldsymbol{\omega}\cdot\mathbf{x}}\,d\mathbf{x} = \sum_{n=1}^N \rho_n \int_{\Omega_n^j} e^{-i\,\boldsymbol{\omega}\cdot\mathbf{x}}\,d\mathbf{x}$$ and, leveraging barycentric parametrizations, the simplex integral admits the closed form: $$F_n^j(\boldsymbol{\omega}) = \rho_n\,i^{\,j}\,\gamma_n^j \sum_{t=1}^{j+1} \frac{e^{-i\,\boldsymbol{\omega}\cdot\mathbf{x}_t}}{\prod_{\ell\neq t}(\boldsymbol{\omega}\cdot\mathbf{x}_t - \boldsymbol{\omega}\cdot\mathbf{x}_\ell)},$$ where the content-distortion factor $\gamma_n^j$ is proportional to the simplex's volume and the metric induced by the Cayley–Menger determinant.

The overall spectral signal is summed over all simplices: $$\hat{F}(\boldsymbol{\omega}) = \sum_{n=1}^N F_n^j(\boldsymbol{\omega}).$$

2. Differentiable Rasterization and Spectral Filtering

After obtaining the spectral representation, rasterization onto a $d$-dimensional grid is performed via an inverse Fourier transform, discretized over $M$ frequencies: $$I(\mathbf{p}) \approx \frac{1}{(2\pi)^d} \sum_{m=1}^M \hat{F}(\boldsymbol{\omega}_m) G(\boldsymbol{\omega}_m) e^{i\,\boldsymbol{\omega}_m\cdot\mathbf{p}},$$ with $G(\boldsymbol{\omega})$ a Gaussian filter suppressing high-frequency aliases. The process is implemented efficiently via (GPU-based) Fast Fourier Transform (FFT), with frequency grid size and filter bandwidth set to balance spatial detail and aliasing.

Because each pipeline component—NUFT accumulation, spectral filtering, and inverse FFT—is analytic and differentiable with respect to the input mesh parameters, the forward rasterization is fully compatible with automatic differentiation and stochastic gradient descent.

3. Backpropagation and Computational Complexity

Backpropagating through DDSL requires computing gradients of the output raster $I(\mathbf{p})$ with respect to each mesh vertex $\mathbf{x}_{n,t}$: $$\frac{\partial I(\mathbf{p})}{\partial \mathbf{x}_{n,t}} = \frac{1}{(2\pi)^d} \sum_{m=1}^M \frac{\partial \hat{F}(\boldsymbol{\omega}_m)}{\partial \mathbf{x}_{n,t}} G(\boldsymbol{\omega}_m) e^{i\,\boldsymbol{\omega}_m\cdot\mathbf{p}}.$$ The analytic derivatives of $F_n^j(\boldsymbol{\omega})$ with respect to vertex positions leverage chain rules for the simplectic parameterization and linear algebra operators. All needed quantities (e.g., content factor, adjugate of Cayley–Menger) are computed per simplex per frequency in $O((j+1) N_{\text{simplices}}\,M)$ time. The dominant cost lies in the $M$-point inverse FFT ($O(M\log M)$); this workflow is efficient for meshes with thousands of elements and grids of millions of points on modern GPUs.

Compared to finite-difference schemes (which require $O(N_\text{verts})$ forward passes per gradient step), DDSL’s analytic backward is orders of magnitude faster and exact up to numerical precision.

4. Generalization to Arbitrary Simplex Orders and Dimensions

A defining property of DDSL is its agnosticism to both simplex order $j$ and ambient dimension $d$. The same machinery supports:

• 0‐simplices (point clouds)
• 1‐simplices (wires, polylines)
• 2‐simplices (triangle meshes)
• 3‐simplices (tetrahedral meshes)

Parameterizations differ only in the calculation of $\gamma_n^j$ and the Cayley–Menger determinant. The implementation supports both 2D rasterization (image generation) and 3D rasterization (voxels) from arbitrary mesh complexes.

5. Applications: Shape Optimization, Differentiable Polygon Loss, and Performance

DDSL unlocks end-to-end differentiable processing of mesh-based inputs in geometric learning settings. Two principal applications are described:

(a) Mesh Editing and Shape Optimization

DDSL enables gradient-based optimization of a mesh’s geometry, using neural network outputs as objectives. Typically, the mesh is rasterized with DDSL and passed through a pretrained CNN predicting a desired property (class label, aerodynamic coefficient, etc.). The resultant loss gradient is automatically propagated back to the mesh through the DDSL layer, facilitating real-valued shape editing.

For example, an MNIST polygon representing the digit “1” is iteratively deformed so its DDSL raster induces a classifier prediction for “3”; likewise, airfoil meshes are optimized to achieve target lift-drag ratios using CNN surrogates and gradient descent. In both, monotonic loss decrease demonstrates stable optimization.

(b) Differentiable Rasterization Loss for Polygon Generators

DDSL enables direct supervision of polygon- or mesh-generating neural networks with raster-based losses, as vertex positions can be adjusted by loss gradients defined on the raster images.

A multi-resolution rasterization loss is defined as: $$\mathcal{L}_{\mathrm{mres}} = \sum_{i=0}^3 \sum_{r \in \{224, 112, 56, 28\}} \| D_r(G_\theta^{(i)}(x)) - D_r(y) \|_1,$$ where $D_r(\cdot)$ is DDSL rasterization at resolution $r \times r$, $G_\theta^{(i)}(x)$ is the polygon at refinement level $i$, and $y$ the ground truth. Additional smoothness regularization on angles further stabilizes training. Trained on Cityscapes polygons, this pipeline yields state-of-the-art accuracy (mIoU 72.50% vs. Polygon-RNN++ 71.38%) and 100-fold speedup (0.029 s vs. 2.32 s per batch on Titan X), despite significantly smaller model size (24M vs. 100M parameters).

6. Implementation, Optimization, and Practical Considerations

The full forward and backward workflow is as follows:

Forward (V, E, D):

• Construct regular frequency grid $\{\omega_m\}_{m=1}^M$.
• For each simplex:
  • Compute content factor $\gamma_n^j$.
  • For each frequency:
  • Compute dot products $\sigma_{n,t} = \omega_m \cdot x_{n,t}$.
  • Accumulate spectral sum $S_n^j(\omega_m)$ via exponential numerators and denominator products.
• Multiply spectral coefficients by Gaussian filter $G$.
• Inverse FFT yields raster $I$.

Backward ($\partial L / \partial I$):

• Compute spectral gradient via FFT of $\partial L / \partial I$.
• For each simplex and vertex, compute analytic derivatives wrt position coordinates.
• Accumulate contributions, yielding $\partial L / \partial x_{n,t}$ for gradient descent.

Practical settings:

• Frequency grid size $M$: match or exceed output resolution, favoring powers of 2 for FFT.
• Spectral filter bandwidth: set for desired alias-detail tradeoff ($\sigma=1$–2 grid units).
• GPUs with single-precision are sufficient; no reliance on non-elementary special functions.
• Large meshes or high resolutions can be handled via tiling or multi-scale spectral blocking.

The pipeline maintains computational efficiency (forward and backward $O(N_{\text{simplices}} M + M\log M)$) and avoids numerical instability by analytic differentiation. For million-point grids and thousand-simplex meshes, real-time operation is achieved.

7. Significance and Impact

DDSL provides a theoretically grounded, dimension/order-agnostic, and efficiently differentiable layer for rasterizing geometric signals, establishing a bridge between mesh-based and grid-based deep learning workflows. Its ability to propagate gradients from raster losses down to mesh vertex coordinates opens previously inaccessible domains—such as direct shape optimization, neural mesh manipulation, and high-precision supervision of polygon generators—facilitating new methodologies in geometric machine learning and computer graphics (Jiang et al., 2019).