DENALI: A Dataset Enabling Non-Line-of-Sight Spatial Reasoning with Low-Cost LiDARs

Abstract: Consumer LiDARs in mobile devices and robots typically output a single depth value per pixel. Yet internally, they record full time-resolved histograms containing direct and multi-bounce light returns; these multi-bounce returns encode rich non-line-of-sight (NLOS) cues that can enable perception of hidden objects in a scene. However, severe hardware limitations of consumer LiDARs make NLOS reconstruction with conventional methods difficult. In this work, we motivate a complementary direction: enabling NLOS perception with low-cost LiDARs through data-driven inference. We present DENALI, the first large-scale real-world dataset of space-time histograms from low-cost LiDARs capturing hidden objects. We capture time-resolved LiDAR histograms for 72,000 hidden-object scenes across diverse object shapes, positions, lighting conditions, and spatial resolutions. Using our dataset, we show that consumer LiDARs can enable accurate, data-driven NLOS perception. We further identify key scene and modeling factors that limit performance, as well as simulation-fidelity gaps that hinder current sim-to-real transfer, motivating future work toward scalable NLOS vision with consumer LiDARs.

• The paper introduces DENALI, a dataset that captures three-bounce, non-line-of-sight LiDAR returns using consumer-grade sensors to enable spatial reasoning.
• It demonstrates the use of data-driven architectures, achieving localization RMSE as low as 0.0456 m and robust shape classification under varied conditions.
• Digital twins and sim-to-real experiments validate the approach, highlighting the impact of simulation fidelity and real-data integration on NLOS inference.

DENALI: A Large-Scale Resource for Data-Driven Non-Line-of-Sight Spatial Reasoning with Low-Cost LiDARs

Introduction

The increasing ubiquity of direct time-of-flight (dToF) LiDAR in consumer electronics introduces new opportunities for spatial perception, particularly through the exploitation of time-resolved histograms that record not only single-bounce (direct) returns but also multi-bounce (indirect) photon returns. Recent developments in non-line-of-sight (NLOS) imaging leveraging laboratory-grade LiDAR systems are well documented, but the translation of these approaches to consumer hardware remains unresolved due to severe limitations in temporal/spatial resolution, increased noise, and complex multipath phenomena [faccio2020non, callenberg2021low]. The paper "DENALI: A Dataset Enabling Non-Line-of-Sight Spatial Reasoning with Low-Cost LiDARs" (2604.16201) addresses these challenges by introducing DENALI, a large-scale, real-world dataset capturing three-bounce NLOS returns with low-cost, mobile-grade LiDAR systems. This resource systematically characterizes the NLOS information content in affordable LiDAR sensors and provides experimental benchmarks for data-driven NLOS perception.

Physical Basis and Dataset Acquisition

Conventional dToF LiDAR typically computes depth from the dominant peak in a per-pixel photon arrival histogram, representing photons reflected directly from visible surfaces. However, later time bins in the histogram correspond to photons undergoing multiple bounces, including paths such as “relay wall → hidden object → relay wall,” thereby containing encoded information about occluded, non-line-of-sight scene features (Figure 1).

Figure 1: Conventional dToF LiDAR reports primary single-bounce reflections, but also records multi-bounce returns informative of hidden objects via three-bounce paths.

DENALI systematically elicits and records these three-bounce signals via a controlled capture setup. The system employs ams TMF8828 consumer-grade, flood-illumination LiDAR modules (operating at 940 nm) co-located with RGB-D cameras and equipped with synchronized overhead trackers. Hidden objects are mounted on a motorized gantry, outside the line of sight, and positioned to ensure that only indirect, three-bounce photon returns contribute to the observed histograms. Retroreflective tape enhances photon yield from hidden targets, and extensive marker-based tracking enables accurate 6-DoF localization of all scene elements throughout the acquisition (Figure 2).

Figure 2: DENALI’s capture rig records synchronized LiDAR, RGB-D, and scene-tracking data to enable precise spatial correspondence for every histogram measurement.

The object set is diverse: 30 unique CAD models (letters, numbers, shapes) are fabricated and sampled at two scales (4-inch, 8-inch), yielding 60 distinct objects. For each object, 100 (x, y) planar positions are sampled within the out-of-view region, resulting in 72,000 scenes, each captured under two lighting conditions and at both $3 \times 3$ and $8 \times 8$ sensor resolutions (Figure 3).

Figure 3: Dataset objects span 60 unique shapes and two sizes; all have known CAD geometry for downstream ground-truth analysis and simulation.

Each captured scene includes detailed digital twins rendered using Mitsuba 3, offering a high-fidelity basis for simulation, modeling, and sim-to-real evaluation workflows.

Dataset Analysis and Benchmark Tasks

The DENALI dataset is engineered to support rigorous benchmarking of data-driven NLOS perception tasks—specifically, hidden object localization, shape classification, and size discrimination—using directly captured multibounce LiDAR histograms. The photon count tensor for each sample encodes both spatial and temporal structure ($n \times n \times 128$), and several architectures are evaluated:

• Baseline MLP: Structure-agnostic; flattens input tensor.
• 1D CNN (Time-only Convolution): Temporal structure; no explicit spatial modeling.
• 3D CNN (Spatiotemporal): Joint learning over spatial and temporal domains.
• Transformer (Time-token Encoder): Long-range dependencies via temporal self-attention.

Strong numerical results are obtained for NLOS localization and size estimation. The 1D CNN achieves RMSE of 0.0456 m for hidden object localization using $3 \times 3$ pixel histograms, with larger objects yielding lower error and broader spatial reach (Figure 4 and Figure 5).

Figure 4: Histogram examples show distinct three-bounce photon arrival patterns as a function of object size, position, LiDAR resolution, and lighting.

Figure 5: Spatial NLOS localization RMSE heatmaps. Localization accuracy degrades with smaller targets and with increasing distance from the relay wall; illumination conditions modulate error patterns non-uniformly.

Classification accuracy is similarly affected by these scene factors—top-1 object classification accuracy peaks for larger objects, and macro-F1 scores are highest for 8-inch targets (Figure 6).

Figure 6: NLOS object classification accuracy by shape and size. Larger, more photon-rich objects are classified substantially more reliably.

The results expose two principal limitations: (1) NLOS inference robustness is sensitive to object size and geometry, with performance degrading as signal strength decreases; (2) modeling entanglement exists between object, lighting, and geometry factors, evidenced by non-uniform error distributions under different experimental conditions.

Simulation Validation and Sim-to-Real Transfer

A distinctive contribution of DENALI is the co-captured digital twin for every physical scene, enabling direct comparison and calibration workflows for transient simulations (Figure 7). These digital twins are rendered under physically-accurate models (MiTransient), allowing researchers to isolate the effects of pulse shapes, temporal jitter, background noise, and scaling.

Figure 7: Scene-level digital twins allow precise evaluation and diagnosis of physical-vs-simulated LiDAR signal and the sim-to-real gap.

Sim-to-real transfer is quantified by measuring localization error as additional simulation fidelity is introduced or real samples are incorporated into the training regime (Figure 8). Adding pulse width, noise modeling, and intensity calibration incrementally reduces transfer error, but the largest performance gains are achieved by supplementing simulation with even modest amounts of real data.

Figure 8: Localization RMSE as a function of simulation fidelity and the number of real samples used for training; simulation refinements yield diminishing returns compared to adding real samples.

Task-specific sensor requirements—such as tolerance to temporal jitter—are also assessed, informing future hardware tradeoffs and motivating design of sensors tailored for NLOS performance, rather than solely direct depth estimation.

Implications and Future Directions

DENALI establishes the first large-scale, real-world benchmark for data-driven NLOS perception with commodity LiDAR, closing the methodological gap between laboratory demonstrations and the practical constraints of real device hardware. The dataset and analysis demonstrate that low-cost dToF sensors possess underexploited NLOS information content sufficient for spatial reasoning tasks such as localization and classification. However, this performance is modulated by target size, photon budget, scene layout, and the quality of both physical modeling and data-driven architectures.

This work suggests several future trajectories:

• Model Improvements: Architectures capable of explicitly disentangling object, geometric, and illumination factors are required to close NLOS generalization gaps and improve robustness.
• Simulation-to-Real Transfer: Advances in transient simulation fidelity, augmented with real-world calibration, will be critical for pre-training and evaluating scalable NLOS workflows across diverse conditions.
• Sensor Co-Design: Joint optimization of sensor hardware, scene embedding, and inference algorithms can yield devices purpose-built for robust NLOS perception.
• Autonomous Robotics and Consumer Applications: Deployment of data-driven NLOS spatial reasoning can enhance obstacle avoidance, safety, and perception robustness in robots and mobile devices operating in real-world, unstructured environments.

Conclusion

The DENALI dataset (2604.16201) delivers a foundational resource and experimental basis for studying non-line-of-sight spatial reasoning with low-cost, time-resolved LiDAR hardware. Experimentally, strong localization and classification results are achievable, but are currently limited by photon SNR, scene configuration, and model expressivity. The inclusion of digital twins uniquely enables end-to-end evaluation of the sim-to-real transfer for NLOS tasks. Overall, this work provides both a dataset and a methodological framework that will support the development of scalable, deployable NLOS sensing and inference pipelines for broad real-world impact.