---
title: Differentiable Shadow Mapping
url: https://www.emergentmind.com/topics/differentiable-shadow-mapping
type: topic
---

# Differentiable Shadow Mapping

Searching arXiv for the cited differentiable shadow mapping papers and closely related work.
arXiv search query: "Differentiable Shadow Mapping inverse graphics visibility shadows 2308.10896 2104.00359 1812.04857 2206.08990 2303.15101"
Differentiable shadow mapping denotes a family of rendering formulations in which illumination visibility—or, equivalently, cast-shadow formation—is incorporated into an end-to-end differentiable inverse problem. Across recent work, the central technical objective is to retain the geometric information carried by shadows while avoiding the optimization dead ends caused by hard binary visibility tests. The resulting methods span point-light inverse rendering with Blinn–Phong shading, spherical-harmonic soft-shadow models, rasterization-based differentiable shadow maps for triangle meshes, occupancy-based shadow renderers for 3D reconstruction from shadow masks, and soft cast-shadow modules for uncalibrated photometric stereo [1812.04857] [2104.00359] [2206.08990] [2308.10896] [2303.15101].

## 1. Visibility as a rendering variable

In the cited literature, the mathematical role of shadows is expressed through an explicit visibility term. For diffuse inverse rendering under environment illumination, one formulation writes
$$
B(\mathbf{x}) = a(\mathbf{x}) \int_{\Omega} L(\omega)\, V(\omega,\mathbf{x})\, H(\omega,\mathbf{x})\, d\omega,
$$
where \(a(\mathbf{x})\) is diffuse albedo, \(L(\omega)\) is environment illumination, \(V(\omega,\mathbf{x})\) is visibility, and \(H(\omega,\mathbf{x})=\max((\omega\cdot \mathbf{n}(\mathbf{x})),0)\) is the Lambertian cosine term [2104.00359]. In this form, shadows are not a post hoc image effect; they are the visibility factor that determines whether incoming radiance contributes at all.

For point-light inverse rendering, a closely related formulation uses Blinn–Phong shading with ambient, diffuse, and specular terms:
$$
{\bf I}={\bf I}_a+{\bf I}_d+{\bf I}_s,
$$
with the full image model
$$
{\bf I} = {\bf k}_d\cdot{\bf L}_a + \sum_i {\bf S}_i \cdot {\bf I}_{L,i} \left( {\bf k}_d \cdot ({\bf L}_i-{\bf X})^t{\bf N} + {\bf k}_s \cdot ({\bf H}^t{\bf N})^\alpha \right).
$$
Here the binary shadow term \({\bf S}_i\) multiplicatively modulates both diffuse and specular shading, so cast shadows directly affect the photometric objective used for light estimation [1812.04857].

The same distinction appears in uncalibrated photometric stereo. DANI-Net factorizes visibility into a local attached-shadow component and a global cast-shadow component:
$$
m_{ij} = e_j\, s_{ij}\, \rho_{ij}\, \max(\mathbf{n}_i^\top \mathbf{l}_j, 0)
= e_j\, s_{ij}\, (\rho^d_{ij} + \rho^s_{ij})\, \max(\mathbf{n}_i^\top \mathbf{l}_j, 0),
$$
where \(s_{ij}\) is the cast-shadow term and \(\max(\mathbf{n}_i^\top \mathbf{l}_j,0)\) is the attached-shadow term [2303.15101]. This decomposition is important because cast shadows depend on global geometry and lighting, not merely on local surface orientation.

## 2. Why classical shadow mapping is not differentiable

The principal obstacle is that standard shadow mapping relies on a binary depth comparison. In the classical pipeline, one renders a shadow map from the light’s viewpoint, compares the stored depth \(z\) with the point-to-light distance \(d\), and then decides shadowed or not shadowed. That binary test blocks gradients [1812.04857]. In triangle-mesh form, the ideal shadow map is
$$
f(u)=\min\{d(x): L(x)=u,\; x\in\mathcal{X}\},
$$
and visibility is
$$
V(x)=\mathbf{1}[d(x)\le f(L(x))].
$$
As written, this is a discontinuous indicator function [2308.10896].

The broader inverse-rendering literature identifies two unsatisfactory extremes. Rasterization-based differentiable renderers usually handle the visible surface from the camera, but not visibility of lights, so they miss cast shadows and self-shadows. Ray-tracing differentiable renderers do handle visibility, but are typically too expensive for iterative optimization or for training deep networks over many iterations [2104.00359]. This computational gap is not merely an efficiency issue. When shadows are ignored during optimization, textures can get baked with shadows, geometry can deform incorrectly, and lighting estimates can become biased [2104.00359].

A further difficulty is that post hoc smoothing is insufficient if the underlying visibility construction is still discrete. For differentiable shadow maps based on rasterization, it was shown that a discrete, unsmoothed shadow map can cause optimization to fail even if the shadow-map moments are prefiltered afterward. The missing ingredient is differentiability of the shadow map itself, so that geometry changes influence shadow-boundary motion in the backward pass [2308.10896]. This suggests that useful gradients near shadow boundaries depend on how visibility is represented before filtering, not only on how the final shadow image is blurred.

## 3. Main formulation families

The recent literature contains several distinct strategies for making shadow computation differentiable.

| Paper | Visibility model | Differentiability mechanism |
|---|---|---|
| [1812.04857] | Binary shadow mapping for point lights | Finite-difference approximation of \(\partial {\bf S}/\partial {\bf L}\) |
| [2104.00359] | Soft visibility in spherical harmonics with sphere blockers | Fully differentiable SH products and blocker approximation |
| [2206.08990] | Binary shadow mask from occupancy field | Max pooling of occupancy values along light rays |
| [2308.10896] | Pre-filtered shadow maps for triangle meshes | Differentiable rasterization from the light viewpoint plus VSM |
| [2303.15101] | Soft cast shadow in UPS | Sigmoid of minimum sampled depth difference |

In point-light inverse rendering, one strategy is to keep ordinary shadow mapping but approximate the Jacobian of the shadow term numerically. The renderer remains differentiable almost everywhere except for the shadow test itself, and the shadow gradient with respect to light position is approximated by finite differences. Specifically, six shadow maps are rendered around the current light position to numerically approximate \(\partial {\bf S}/\partial {\bf L}\), while the rest of the graph is implemented in PyTorch automatic differentiation [1812.04857].

A second strategy replaces explicit ray-based visibility with a low-frequency approximation in spherical-harmonics space. In this approach, both illumination and visibility are represented in SH, and the occluding surface is approximated with spheres. Global visibility is expressed as a product of per-sphere blocker functions in SH, and outgoing radiance becomes
$$
B(\mathbf{x}) = a(\mathbf{x}) \Big(\boldsymbol{L} \cdot (\boldsymbol{V}(\mathbf{x}) * \boldsymbol{H}(\mathbf{x})) \Big).
$$
To avoid the \(\log(0)\) problem of binary blockers, each blocker is approximated by a softened function \(V'_i\) with \(\epsilon=3\), so that blocked directions take value \(e^{-3}\approx 0.05\) rather than zero [2104.00359].

A third family treats shadow computation as a differentiable rasterization problem from the light’s viewpoint. Standard pre-filtered shadow mapping is combined with an existing differentiable rasterizer, implemented on top of Nvdiffrast, so that shadow-map construction and filtering both participate in reverse-mode differentiation. The method uses variance shadow maps. After filtering the first and second depth moments,
$$
m_1 = G\,\tilde f, \qquad m_2 = G\,\widetilde{f^2},
$$
one obtains \(\sigma^2=m_2-m_1\otimes m_1\) and the visibility approximation
$$
\hat v(x)=
\begin{cases}
\frac{\sigma_x^2}{\sigma_x^2 + (d(x)-\mu_x)^2}, & d(x)>\mu_x,\\
1, & \text{else}.
\end{cases}
$$
This yields a visibility function that is continuously differentiable across \(d=\mu\) [2308.10896].

Two further formulations depart from rasterized shadow maps entirely. For reconstruction from a binary shadow image, one differentiable renderer samples an occupancy network along each light ray and uses a max over the sampled occupancies as a soft visibility test for opaque objects. The shadow at ground-plane coordinate \(x\) is
$$
\pi(\ell,\Omega)[x] = \max_{d \in \mathbb{R}} f_\Omega(\ell + d\,u_\theta),
$$
subject to the ray–ground-plane intersection constraint stated in the original formulation [2206.08990]. In DANI-Net, cast shadow is instead modeled by a sigmoid over the minimum sampled depth difference along the light segment:
$$
s_{ij} = \text{Sigmoid}\left(\alpha\left(\min\{w^k_{ij}-\hat{w}^k_{ij}\mid 1 \leq k \leq N_p\}\right) + \beta\right),
$$
with \(N_p=64\) and learnable \(\alpha,\beta\), producing a continuous relaxation of hard shadowing [2303.15101].

## 4. Optimization regimes and implementation patterns

Despite their different representations, these methods share an analysis-by-synthesis structure: render a shadow-aware image or shadow mask from current scene parameters, compare it with the observation, and backpropagate to geometry, light, pose, or appearance.

For point-light estimation, illumination retrieval is posed as
$$
\hat{\bf L}=\arg\min_{\bf L} \; \|{\bf I}({\bf G},{\bf M},{\bf L})-{\bf I}^*\|^2,
$$
where the scene is parameterized by geometry \({\bf G}=({\bf X},{\bf N},{\bf C})\), materials \({\bf M}=({\bf k}_d,{\bf k}_s,\alpha)\), and illumination \({\bf L}=({\bf L}_i,{\bf I}_{L,i})_i\). The optimization is done with gradient descent, and the distinct feature is continuous optimization of light position rather than search over a discrete set of light candidates [1812.04857].

For SH-based soft shadows, inverse problems are solved with an \(\ell_2\) image loss,
$$
\mathcal{L}(\theta, \boldsymbol{L}, a_{0,\dots,n}) = \| I_R(\theta, \boldsymbol{L}, a_{0,\dots,n}) - I \|_2^2,
$$
where \(\theta\) includes pose and embedded deformation parameters. The surface is approximated by spheres attached to a template mesh, and the final rendered image is produced by rasterization after shadow-aware radiance has been computed analytically in SH space [2104.00359].

For shadow-only 3D reconstruction, the optimization variables are latent shape code \(z\), light position \(\ell\), and pose \(\phi\):
$$
\min_{z,\ell,\phi} \; \mathcal{L}\!\left(s,\pi(\ell,\Omega)\right)
\quad \text{where} \quad
\Omega = \mathcal{T}_\phi(G(z)).
$$
The loss is binary cross-entropy, the latent dimension is 128, and the renderer samples 128 points along each light ray. Optimization uses spherical gradient descent for up to 300 steps, with step size \(1.0\) when pose and light are known and \(0.01\) in the unknown-light/unknown-pose case. Multiple random restarts are used because the objective is non-convex [2206.08990].

For differentiable shadow maps on meshes, the pipeline is modular: render the scene from the light’s viewpoint using a differentiable rasterizer, produce smoothed depth images \(\tilde f\) and \(\widetilde{f^2}\), filter them with box or Gaussian convolutions, and query the resulting moment maps during deferred shading. The implementation uses PyTorch and Nvdiffrast, typically with kernel size \(k=3\) or \(k=5\) and default shadow-map resolution \(256\times 256\), while supporting directional lights, spot lights, and multiple lights through multiple shadow-map passes [2308.10896].

For uncalibrated photometric stereo, DANI-Net integrates differentiable shadow handling with anisotropic reflectance modeling. The framework includes DepthMLP, MaterialMLP, learnable light directions \(\mathbf{l}_j\), light intensities \(e_j\), and learnable shadow softness parameters \(\alpha,\beta\). Grid bilinear interpolation is used to obtain sampled depth values from the depth map \(W\), and training proceeds in three stages with Adam for 2000 epochs total, using cosine annealing learning-rate decay [2303.15101].

## 5. Empirical role of shadows in inverse problems

A recurring empirical conclusion is that shadows provide constraints that other image terms do not. In point-light estimation, experiments deliberately perturb \({\bf k}_d\) and \({\bf k}_s\), including incorrect values and fractal noise, and the reported finding is that methods without shadows may fit the image well when reflectance is ideal, but incorporating shadows makes light recovery more robust under reflectance mismatch [1812.04857]. The stated explanation is that the shadow constraint provides additional geometric evidence that compensates for reflectance errors.

For SH-based differentiable visibility, the practical benefit is that texture recovery does not bake shadows into albedo, pose recovery must explain shadow placement rather than only object appearance, and deformation recovery can be constrained even from the shadow image alone [2104.00359]. For shadow-only reconstruction from a binary mask, the same logic is pushed further: under known light and pose, the method reaches about \(0.553\) average IoU versus \(0.467\) for regression and \(0.322\) for nearest neighbor; under unknown light and pose, it reaches about \(0.390\) average IoU versus \(0.303\) for regression [2206.08990]. The method also reports robustness on real-world images when ground-truth shadow mask is unknown.

For rasterized differentiable shadow maps, quantitative comparisons emphasize the trade-off between accuracy and runtime. In monocular pose estimation at \(512\times 512\) resolution, representative numbers are: Bunny—Mitsuba \(0.23^\circ\), translation error \(0.10\), runtime \(325.29\) s; the shadow-map method \(0.33^\circ\), \(0.22\), runtime \(3.76\) s; rasterizer only \(0.31^\circ\), \(0.26\), runtime \(1.58\) s. For Spot—Mitsuba \(0.03^\circ\), \(0.02\), runtime \(381.80\) s; the shadow-map method \(0.05^\circ\), \(0.05\), runtime \(3.76\) s; rasterizer only \(5.80^\circ\), \(23.15\), runtime \(1.77\) s. In face reconstruction from profile shadows, one shadow gives mean \(D_\text{mean}=0.013\), \(D_\text{Hausdorff}=0.105\), runtime \(18.83\) s, while three shadows give \(D_\text{mean}=0.008\), \(D_\text{Hausdorff}=0.068\), runtime \(27.32\) s [2308.10896].

In uncalibrated photometric stereo, shadow handling primarily improves geometry recovery. On DiLiGenT, average normal MAE is reported as \(6.54^\circ\) for DANI-Net, \(6.87^\circ\) for DANI-Net without differentiable shadow handling, and \(7.06^\circ\) when using the alternative shadow strategy drawn from LL22 [2303.15101]. The paper specifically highlights strong gains on shadow-heavy objects such as Reading and Harvest.

## 6. Assumptions, approximations, and unresolved issues

The current formulations remain approximate, and their assumptions differ substantially. The point-light Blinn–Phong model assumes known geometry, known or estimated camera pose, known material parameters \(({\bf k}_d,{\bf k}_s,\alpha)\), point light sources rather than environment maps or directional infinite lights, and a point-cloud representation for shadow computation; the latter leads to incomplete or aliased shadows because only visible depth information is used and rasterization is coarse [1812.04857].

The SH-based method assumes diffuse surfaces only, no inter-reflection, low-frequency lighting, and a sphere-based approximation of geometry. High-frequency lighting, such as a sharp directional light, is not well represented by the low-dimensional SH basis, and non-diffuse materials are outside the model [2104.00359]. The occupancy-based shadow renderer assumes a point light source above a ground plane and uses max pooling along the ray rather than physical volumetric integration, which is presented as a good approximation for binary shadow masks and solid opaque objects [2206.08990].

The rasterized shadow-map approach is limited to direct shadows, not indirect illumination, caustics, or full global light transport. Its main implementation is presented for triangle meshes, supports directional and spot lights directly, and uses variance shadow maps, which are known to suffer from light bleeding in regions of high depth variance [2308.10896]. DANI-Net, for its part, adopts a sampled soft-shadow approximation along the light segment and depends on stable depth and normal estimation during optimization; the cited work presents this as a means to make shadow cues usable rather than as an exact visibility solver [2303.15101].

A broader point of interpretation follows from the collection as a whole. These papers do not converge on a single canonical definition of differentiable shadow mapping. Instead, they establish a spectrum of methods that trade physical fidelity, representation choice, and computational cost against gradient quality. One end preserves hard-shadow structure and differentiates approximate rasterized visibility; another replaces visibility with soft or low-frequency surrogates that are easier to optimize; a third couples shadow likelihoods to strong priors, such as latent shape generators. A plausible implication is that “differentiable shadow mapping” is best understood not as one algorithmic primitive but as a design space for making visibility informative to gradient-based inverse graphics.

Source: https://www.emergentmind.com/topics/differentiable-shadow-mapping