Differentiable Ray Tracing Overview

• Differentiable ray tracing is a computational pattern that exposes scene gradients for inverse rendering and optimization in fields like graphics, radio, and ultrasound.
• It integrates a forward transport operator with a task loss, allowing parameter refinement through gradients for material, geometric, and physical properties.
• Techniques such as smoothing discontinuities and optimizing scene representations help stabilize gradients despite inherent visibility and approximation challenges.

Differentiable ray tracing denotes ray-tracing formulations in which outputs of interest—rendered pixels, channel impulse responses, coverage maps, SAR images, radiation fields, or ultrasound B-mode images—are treated as differentiable functions of scene, material, geometric, or acquisition parameters, so that gradients can be propagated through the transport computation and used for inverse rendering, calibration, localization, freeform design, and coupled physics optimization (Hoydis et al., 2023, Hoydis et al., 2023, Spencer et al., 16 Apr 2026). In contrast to classic ray tracing, which is typically used as a forward simulator, differentiable ray tracing exposes derivatives with respect to quantities such as material properties, antenna patterns, array geometries, positions and orientations, face geometry and reflectance, lens surfaces, or acoustic parameters, thereby recasting ray-based simulation as an optimization primitive rather than only an overview engine (Hoydis et al., 2023, Dib et al., 2019, Koning et al., 2023).

1. Core formulation and transport interpretation

The common structure across the literature is a forward transport operator followed by a task loss. In radio propagation, Sionna RT treats functions of generated field components, such as channel impulse responses and coverage maps, as differentiable with respect to material properties, antenna patterns, array geometries, and transmitter or receiver orientations and positions (Hoydis et al., 2023). In radio-environment calibration, the channel frequency response is written as a sum over propagation paths, $H(f)=\sum_{i=1}^{M} a_i e^{-j2\pi f \tau_i}$, and the resulting CIR becomes the object through which gradients are propagated to material, scattering, and antenna parameters (Hoydis et al., 2023).

In inverse rendering for faces, the rendered pixel is expressed as a differentiable Monte Carlo estimator, $$\mathbf{p}_i^T = \mathcal{F}(f_r, \mathbf{l}_j, \alpha, \beta, \mathbf{R}, \mathbf{t})$$, and optimization proceeds against photometric error while accounting for visibility, self-shadowing, and specular transport (Dib et al., 2019). In ultrasound, the image is decomposed as $$\mathbf{I}(\theta)=\mathcal{B}(\hat{\mathbf{P}}(\theta))$$, so gradients from a B-mode image loss are propagated through acoustic transport, beamforming, and post-processing back to impedance, roughness, transducer pitch, center frequency, steering, and related parameters (Spencer et al., 16 Apr 2026).

The same pattern also appears in scientific transport. Ray-trax solves the time-dependent emission–absorption equation

$$\frac{1}{c}\frac{\partial I}{\partial t} + \hat{\mathbf{n}}\cdot\nabla I = -\,\kappa(\mathbf{x})\,I + j(\mathbf{x}),$$

using a JAX implementation whose full ray-tracing map is differentiable with respect to emissivity, opacity, and derived parameters, which allows the radiative transfer operator to be embedded in differentiable hydrodynamic pipelines (Branca et al., 12 Nov 2025). This suggests that differentiable ray tracing is less a single algorithm than a computational pattern in which path generation, interaction physics, accumulation, and the application-level loss are composed into one differentiable graph.

2. Visibility discontinuities, plateaus, and gradient stabilization

A central technical difficulty is that ray tracing contains hard validity tests: visibility, hit or miss events, obstruction, convergence of path solvers, and path appearance or disappearance under infinitesimal geometric changes. In wireless optimization, these effects create abrupt objective variations and entire zero-gradient regions, which can cause gradient descent to stall (Eertmans et al., 2024). One response is explicit discontinuity smoothing: the step function

$$\theta(x)= \begin{cases} 1, & x>0\ 0, & \text{otherwise} \end{cases}$$

is replaced by a smooth approximation $s(x;\alpha)$, with sigmoid and hard-sigmoid examples, and the smoothness parameter $\alpha$ is increased from $1$ to $100$ over $100$ iterations as a continuation strategy (Eertmans et al., 2024).

Related stabilization devices appear in several domains. RayLoc addresses sparse gradients and local minima in wireless localization by convolving the localization loss with a 2D Gaussian kernel over target position and annealing the smoothing scale during optimization, so that the method begins with broad exploration and gradually moves toward local refinement (Han et al., 21 Jan 2025). In freeform optics, ordinary detector binning is replaced by a smooth Gaussian reconstruction filter, making irradiance differentiable with respect to lens parameters because ray contributions vary continuously as the surface changes (Koning et al., 2023). In practical face reconstruction, edge sampling is used to better estimate gradients near geometric edges, and Monte Carlo variance is reduced with importance sampling, $$\mathbf{p}_i^T = \mathcal{F}(f_r, \mathbf{l}_j, \alpha, \beta, \mathbf{R}, \mathbf{t})$$0 samples per pixel, and Gaussian smoothing using a $$\mathbf{p}_i^T = \mathcal{F}(f_r, \mathbf{l}_j, \alpha, \beta, \mathbf{R}, \mathbf{t})$$1 kernel with $$\mathbf{p}_i^T = \mathcal{F}(f_r, \mathbf{l}_j, \alpha, \beta, \mathbf{R}, \mathbf{t})$$2 (Dib et al., 2021).

Monte Carlo differentiability is also handled explicitly. In the ultrasound framework, gradients are not taken through the randomness itself; instead, sampled ray realizations are fixed during the backward pass, yielding a deterministic computation graph for those samples (Spencer et al., 16 Apr 2026). A recurrent distinction in the literature is therefore between automatic differentiation of a nominal ray tracer and the additional modifications required to obtain gradients that are continuous enough, dense enough, or stable enough for optimization.

3. Scene representations and acceleration structures

Differentiable ray tracing is tightly coupled to the scene representation, because differentiability and efficiency depend on how intersections, neighborhoods, and transport coefficients are encoded. One large family uses conventional geometric surfaces. Freeform optics papers represent surfaces with tensor-product B-splines, optimize control-point heights, and differentiate through refraction, surface normals, and detector irradiance (Wang et al., 2023, Koning et al., 2023). Hybrid metalens design embeds a phase-discontinuity interface into a differentiable ray tracer via generalized Snell’s law and solves implicit surface intersections with Newton’s method (Zhang et al., 2024).

A second family works with point-based or learned explicit geometry. A differentiable radio-propagation simulator operating directly on point clouds voxelizes the cloud into discretized point sets wrapped in AABBs, uses ray reception points for visibility construction, and replaces earlier SDF-based surfaces with a ray–disk intersection model in which each point is a planar disk with centroid, normal, and radius (Vaara et al., 5 Jul 2025). In unified RF simulation and view synthesis, scenes are represented as optimized 2D Gaussian primitives, approximated by lightweight polygonal proxies for hardware ray tracing; the same Gaussian scene then supports both alpha-composited rendering and specular multi-bounce RF path tracing (Vaara et al., 8 May 2026).

A third family seeks new primitives optimized for ray tracing itself. Radiant Foam represents space as a dense Voronoi tessellation whose cells store piecewise-constant density and view-dependent color; traversal proceeds cell-by-cell without a BVH, and the representation remains continuous with respect to site coordinates because topology changes are “hidden” in zero-volume regions (Govindarajan et al., 3 Feb 2025). The abstract of Power Foam describes a successor direction in which Voronoi foams are generalized to bounded power diagrams with controllable cell extents, paired with an oriented surface formulation and differentiable textures on those surfaces, with the explicit goal of unifying ray tracing and rasterization in one real-time differentiable representation (Govindarajan et al., 27 Apr 2026).

Triangles remain a particularly important primitive because they are native to hardware ray tracing and standard rasterization. UTrice replaces Gaussian proxy geometry with triangles as the unified primitive, builds the BVH directly from vertex and index buffers in OptiX, and introduces a differentiable triangle response function so the same triangle parameterization can be used both by rasterization-based optimization and by ray-traced rendering (Liu et al., 4 Dec 2025). The representation question is therefore not peripheral: it determines whether differentiability is attached to geometry itself, to a proxy of geometry, or only to downstream field computations after paths have been found.

4. Optimization patterns in inverse problems

Most differentiable ray-tracing systems are designed around inverse problems rather than forward-only rendering. In face reconstruction, the optimization is deliberately split into stages. A first stage estimates pose, geometry, statistical diffuse and specular albedo, and virtual light-stage illumination under strong priors; a second stage refines personalized diffuse albedo, specular albedo, and roughness, using the first-stage shadow estimate to prevent self-shadows from being baked into albedo (Dib et al., 2021). A related face model separates identity, expression, pose, diffuse albedo, specular reflectance, and area-light intensities, and minimizes a photometric data term together with priors and a light regularizer (Dib et al., 2019).

Inverse optical design shows a similar pattern. In B-spline freeform surface tailoring, optimization proceeds in two steps: first, an optimal-transport map induces desired target points and the loss is the squared distance between simulated and prescribed ray intersections; second, the initialized surface is refined with a direct irradiance discrepancy defined over a pixel grid (Wang et al., 2023). In achromatic hybrid metalens design, the optimizable variables include polynomial metalens coefficients, aspheric curvature, conic coefficient, higher-order aspheric coefficients, and the separation distance between metalens and refractive lens; optimization is performed in PyTorch with ADAM over $$\mathbf{p}_i^T = \mathcal{F}(f_r, \mathbf{l}_j, \alpha, \beta, \mathbf{R}, \mathbf{t})$$3 wavelengths from $$\mathbf{p}_i^T = \mathcal{F}(f_r, \mathbf{l}_j, \alpha, \beta, \mathbf{R}, \mathbf{t})$$4 to $$\mathbf{p}_i^T = \mathcal{F}(f_r, \mathbf{l}_j, \alpha, \beta, \mathbf{R}, \mathbf{t})$$5 nm and $$\mathbf{p}_i^T = \mathcal{F}(f_r, \mathbf{l}_j, \alpha, \beta, \mathbf{R}, \mathbf{t})$$6 rays per wavelength (Zhang et al., 2024).

Wireless inverse problems emphasize calibration and measurement design. RayLoc first performs background calibration to infer scene geometry and materials from CSI, then localizes the target by optimizing only the target position within the calibrated scene (Han et al., 21 Jan 2025). A later RF material-estimation framework uses a vision-LLM to infer material categories from scene images, map them to ITU-R conductivity priors, and choose informative transmitter or receiver placements before differentiable RT refinement with Adam (Kang et al., 26 Jan 2026). These workflows indicate that in practice, differentiable ray tracing is rarely deployed as naive end-to-end gradient descent from random initialization; priors, continuation, staged schedules, and task-specific parameterizations are typically required.

5. Application domains and empirical profiles

The application range of differentiable ray tracing is unusually broad. In computer graphics and novel-view synthesis, Radiant Foam reports rendering speeds around $$\mathbf{p}_i^T = \mathcal{F}(f_r, \mathbf{l}_j, \alpha, \beta, \mathbf{R}, \mathbf{t})$$7 FPS on Mip-NeRF 360 and $$\mathbf{p}_i^T = \mathcal{F}(f_r, \mathbf{l}_j, \alpha, \beta, \mathbf{R}, \mathbf{t})$$8 FPS on Deep Blending on an RTX 4090 while preserving ray-based effects such as reflection, refraction, transparency, distorted cameras, rolling shutter, and motion blur (Govindarajan et al., 3 Feb 2025). UTrice reports better LPIPS than 3DGRT by about $$\mathbf{p}_i^T = \mathcal{F}(f_r, \mathbf{l}_j, \alpha, \beta, \mathbf{R}, \mathbf{t})$$9, direct compatibility with triangles optimized by Triangle Splatting, and real-time rendering performance while avoiding proxy geometry (Liu et al., 4 Dec 2025).

In wireless and RF modeling, Sionna RT demonstrates learning material parameters from channel responses and optimizing transmitter orientation by gradient ascent on average received power (Hoydis et al., 2023). A point-cloud radio simulator reports multi-bounce propagation with up to five interactions, specular reflections and diffuse scattering in two indoor scenarios, each completing in less than $$\mathbf{I}(\theta)=\mathcal{B}(\hat{\mathbf{P}}(\theta))$$0 ms, and shows how electromagnetic properties can be learned from segmentation labels by backpropagating a normalized CIR loss (Vaara et al., 5 Jul 2025). DiffSBR casts mmWave reconstruction as optimization of a virtual 3D model against sparse radar point clouds, and the framework is explicitly described as capable of fine-grained reconstruction for novel objects unseen by the radar previously (Chen et al., 2023).

In computational astrophysics, Ray-trax runs directly on turbulent gas fields produced by hydrodynamic simulations, supports arbitrarily many frequency bins without architectural changes, and exposes end-to-end gradients; on a $$\mathbf{I}(\theta)=\mathcal{B}(\hat{\mathbf{P}}(\theta))$$1 grid, tracing $$\mathbf{I}(\theta)=\mathcal{B}(\hat{\mathbf{P}}(\theta))$$2 rays from $$\mathbf{I}(\theta)=\mathcal{B}(\hat{\mathbf{P}}(\theta))$$3 sources across the full domain took $$\mathbf{I}(\theta)=\mathcal{B}(\hat{\mathbf{P}}(\theta))$$4 seconds on a single NVIDIA H200 GPU, and the average relative error against the analytic uniform-medium solution is reported as $$\mathbf{I}(\theta)=\mathcal{B}(\hat{\mathbf{P}}(\theta))$$5 (Branca et al., 12 Nov 2025). In ultrasound, differentiable full-path Monte Carlo ray tracing is coupled to a differentiable bridge through beamforming and post-processing; the method recovers known parameters in a simulated-reference setting and effective parameters in a simulation-to-real setting, with finite-difference checks supporting the computed gradients (Spencer et al., 16 Apr 2026). In SAR, a differentiable ray-tracing engine based on CSVBSDF scattering learns spatially varying $$\mathbf{I}(\theta)=\mathcal{B}(\hat{\mathbf{P}}(\theta))$$6 from SAR images, and the KA+SPM double-scale model is reported to yield significantly lower RMSE than SPM alone (Wei et al., 2024).

These results do not imply a single performance regime. Some systems target real-time differentiable rendering, some prioritize physically interpretable calibration, and others accept higher per-iteration cost in exchange for end-to-end gradients through a full imaging or transport pipeline.

6. Limitations, misconceptions, and current directions

A recurring misconception is that “differentiable” always means the entire simulation is smoothly differentiable with respect to all scene parameters. The literature is more heterogeneous. In the point-cloud radio simulator, differentiability is used mainly for parameter learning after paths are found, rather than for end-to-end differentiable geometric optimization (Vaara et al., 5 Jul 2025). By contrast, fully differentiable wireless frameworks explicitly smooth visibility, validity, and convergence logic so that the loss is continuous everywhere with respect to scene parameters (Eertmans et al., 2024). The distinction matters because path search, occlusion, and topology changes are often the hardest components to differentiate.

Another persistent limitation is approximation. Ray-trax is built around the emission–absorption approximation rather than full scattering transport, uses discrete angular quadrature, and is first-order in the spatial marching step (Branca et al., 12 Nov 2025). The Gaussian-based RF digital-twin framework does not model diffraction, scattering, penetration, or temporal effects, and its ray tracing uses approximated proxy meshes rather than exact Gaussian surfaces (Vaara et al., 8 May 2026). The ultrasound system is explicitly a geometric ray approximation rather than a full-wave solver, and in the simulation-to-real cylinder experiment the recovered quantities are characterized as effective parameters rather than uniquely identified ground truth because of model mismatch (Spencer et al., 16 Apr 2026). Hybrid metalens design is performed in a ray-optics framework with a polynomial phase profile and discrete wavelength sampling, not full electromagnetic co-design (Zhang et al., 2024).

Initialization sensitivity and measurement design remain central practical issues. The VLM-guided RF material-estimation framework reports $$\mathbf{I}(\theta)=\mathcal{B}(\hat{\mathbf{P}}(\theta))$$7–$$\mathbf{I}(\theta)=\mathcal{B}(\hat{\mathbf{P}}(\theta))$$8 faster convergence and $$\mathbf{I}(\theta)=\mathcal{B}(\hat{\mathbf{P}}(\theta))$$9–$$\frac{1}{c}\frac{\partial I}{\partial t} + \hat{\mathbf{n}}\cdot\nabla I = -\,\kappa(\mathbf{x})\,I + j(\mathbf{x}),$$0 lower final parameter error than uniform or random initialization and random placement baselines, indicating that inverse RT can be strongly conditioned by semantic priors and placement strategy (Kang et al., 26 Jan 2026). This suggests that future systems may increasingly combine differentiable physics with learned priors, rather than attempting unconstrained optimization over large ray-traced parameter spaces.

Current research directions emphasize unification of rendering modalities and primitives. UTrice proposes triangles as a shared primitive for differentiable ray tracing and rasterization in particle-based 3D scenes (Liu et al., 4 Dec 2025). The abstract of Power Foam proposes bounded power diagrams, oriented surfaces, and differentiable texture embeddings to preserve state-of-the-art ray tracing efficiency while achieving rasterization performance competitive with current generation 3DGS, explicitly presenting a practical path toward unified real-time differentiable rendering (Govindarajan et al., 27 Apr 2026). Taken together, these developments indicate that the present frontier is not merely making ray tracing differentiable, but making it differentiable, efficient, and representation-compatible with the rasterization-dominated systems that currently define real-time 3D optimization.