Rapid Geometric 3D Reconstruction

• Rapid geometric 3D reconstruction is an approach that uses online Bayesian filtering and deep learning to infer depth from sensor data with minimal latency.
• It integrates per-pulse updates, assumed density filtering with Gaussian projections, and spatial mixture-of-predictors to maintain accuracy in dynamic and unconstrained environments.
• The method achieves high throughput and reduced error rates compared to traditional histogram-based techniques, enabling real-time recovery in applications like robotics and AR/VR.

Rapid geometric 3D reconstruction describes the class of algorithms and computational pipelines designed for near-real-time inference of 3D scene structure, geometry, and/or motion from sensor data, with a particular emphasis on low-latency per-frame operation, high robustness to dynamic or unconstrained inputs, and minimal reliance on batch post-processing or time-intensive optimization. In this paradigm, systems integrate Bayesian filtering, deep-learning-based geometry regression, feed-forward scene decoding, and GPU-parallel optimization to achieve interactive or real-time 3D reconstruction, often in settings such as single-photon detection, robotics, AR/VR, and physically dynamic environments. The following sections outline the key principles, probabilistic models, computational architectures, experimental results, and algorithmic considerations that underlie advances in rapid geometric 3D reconstruction, exemplified by the Bayesian online method for single-photon Lidar (Altmann et al., 2019).

1. Problem Statement and Classical Limitations

Traditional single-photon Lidar 3D reconstruction relies on the accumulation of time-of-arrival (ToA) detection histograms at each pixel by aggregating hundreds to thousands of laser pulses. The subsequent depth estimation, typically conducted via histogram peak-finding or cross-correlation, is effective in static or high-photon-flux regimes. However, such methods introduce several bottlenecks for rapid reconstruction in dynamic scenes:

• Integration times required for histogram formation induce latency, leading to motion blur and split peaks when scene elements move faster than the histogram window.
• Computational and memory requirements scale with the number of frames and pulses per pixel, making large-scale or long-duration deployments prohibitive in terms of system resources.
• Batch methods produce delayed updates, limiting their applicability in scenarios requiring immediate scene feedback or adaptive response.

The stated requirement is therefore an online, per-pulse (per-frame) depth-update approach that (i) processes each detection event individually, (ii) eliminates histogram construction altogether, (iii) operates at sub-frame latencies, and (iv) models spatiotemporal structure to maintain geometric fidelity during fast scene motion.

2. Bayesian Modeling for Event-Based Depth Estimation

The foundational model is a full Bayesian framework operating on a pixel-frame detection grid, notated by indices $p=1,..,P$ for pixels and $n=1,..,N$ for frames. The core random variables per pixel and frame are:

• $d_{p,n}$: True surface range at pixel $p$, frame $n$.
• $z_{p,n}$: Binary indicator of any photon detection (1: detected, 0: not detected).
• $w_{p,n}$: Probability that an event is a signal (laser return), not ambient.
• $y_{p,n}$: Observed time-of-arrival (if $z_{p,n}=1$).

The detection likelihood in the low-flux regime is given by

$$\Pr(z_{p,n}=1 | \pi_{p,n}) = \pi_{p,n}, \quad \pi_{p,n} \approx r_{p,n}S + B_{p,n} \ll 1$$

and the observation likelihood: $$f(y_{p,n} | z_{p,n}=1, w_{p,n}, d_{p,n}) = w_{p,n} \, f_s(y_{p,n} - 2d_{p,n}/c) + (1-w_{p,n}) \, \mathcal{U}_{[0, T_r)}(y_{p,n})$$ where $f_s(\cdot)$ is the known impulse response of the system.

The prior over range is modeled as a Gaussian mixture per pixel: $$d_{p,n} \sim \sum_{m=1}^M u_{p,n}^{(m)} \mathcal{N}(d; \mu_{p,n}^{(m)}, \sigma_{p,n}^{2(m)})$$

The joint posterior per frame is: $$p(d_n | y_n, z_n, w_n) \propto \prod_{p=1}^P f(y_{p,n} | z_{p,n}, w_{p,n}, d_{p,n}) \cdot p(d_{p,n})$$

3. Assumed Density Filtering (ADF) and Gaussian Projection

The ADF approach maintains computational tractability by propagating a single Gaussian posterior per pixel per frame. When a detection occurs ($z_{p,n}=1$), the exact posterior becomes a mixture over both signal/background and prior Gaussian components (total $2M$); this is projected back onto a single Gaussian: $$q_{p,n}(d) = \mathcal{N}(d; \mu_{p,n}, \sigma^2_{p,n})$$ with mean and variance determined by moment matching (minimizing Kullback–Leibler divergence): $$\mu_{p,n} = \mathbb{E}[d_{p,n} | y_{p,n}], \quad \sigma^2_{p,n} = \mathrm{Var}[d_{p,n} | y_{p,n}]$$ Prediction and assimilation steps at each time propagate priors and incorporate new observations using analytically tractable closed-form operations (Gaussian mixture integrals), maintaining fixed computational complexity per update.

4. Spatiotemporal Mixture-of-Predictors Model

To integrate spatial and temporal coherence, the prior for each pixel at the next frame is formulated as a mixture over local neighborhoods, explicitly modeling range dynamics: $$p(d_{p,n+1}) \propto \sum_{p' \in V_p} \nu_{p'} [q_{p',n} * \mathcal{N}(0, \gamma^2)](d_{p,n+1})$$ with $V_p$ the set of $M$ spatial neighbors, mixture weights $\nu_{p'}$, and a per-frame random-walk variance $\gamma^2$ modeling range change. This design allows local propagation of sharp, scene-dependent changes, supporting instantaneous tracking of emerging, moving, or occluded objects.

5. Computational Performance and Parallelization

The per-pixel, per-frame computational burden is dictated by:

• Prediction (prior fusion): $O(M)$
• Assimilation (posterior update): closed-form, $O(M)$
• Overall cost per frame: $O(P \cdot M)$ (P: number of pixels/cells, M: neighborhood size) All updates are parallelizable, with each pixel operating independently except prior calculation (local stencil), rendering the method amenable to vectorized and GPU-based execution.

Measured throughputs:

• 2.9 GHz 8-core CPU Matlab prototype: ~4 ms/frame (≈250 fps)
• Tailored GPU implementation: <1 ms/frame (>1000 fps)

No persistent memory of past frames beyond last posterior mean and variance is required.

6. Experimental Evaluation and Comparative Accuracy

Quantitative assessments on static scenes (129×95 px, 5000 frames) report the following RMSE convergence curves:

• Without spatiotemporal modeling: RMSE ≈ 50 (100 frames), ≈15 (500 frames)
• With spatiotemporal modeling: RMSE ≈15 (100 frames), ≈5 (500), ≈3 (5000) Increasing per-pixel detection probability ($\pi_{p,n}$) by a factor of 10 doubles speed and halves asymptotic RMSE.

Batch cross-correlation methods require aggregation of 50–100 frames for equivalent RMSE, introducing latency proportional to batch size and integration time, whereas the described method (O3DSP) updates instantaneously.

Dynamic scene experiments (100×100 px, 2400 frames, moving/occluding rectangles) demonstrate accurate surface tracking, real-time recovery of re-occluded objects, and uncertainty coherently reflecting scene dynamics.

End-to-end speed is sustained at 250 fps on CPU and >1000 fps on GPU.

7. Key Algorithmic Insights and Outlook

By updating depth estimates online at each detection event and eliminating histogram construction, the method affords sharp, immediate scene reconstruction even for rapidly moving and dynamically structured scenes. The ADF framework ensures fixed per-pixel complexity and analyticity—every update is Gaussian moment-matched rather than numerically integrated over data batches. Local neighborhood priors allow for the preservation of geometric detail and tracking of edge dynamics.

Potential extensions include:

• Enriched motion priors (adding velocity or non-Gaussian process models)
• Handling pixels with multiple returns (unknown mixture complexity)
• Deeper filtering under extremely low flux/high background (ADF bias mitigation)
• Explicit occlusion reasoning in cluttered scenes

This approach represents a canonical example of rapid geometric 3D reconstruction for photon-counting modalities. The analytic, online, per-frame inference loop, combined with spatial mixture-of-predictors priors, sets the standard for latency-minimized, resource-efficient geometric scene recovery, notably outperforming histogram-based and batch-optimized baselines in throughput, convergence, and dynamic fidelity (Altmann et al., 2019).