InfraLiDARs Benchmark Overview
- InfraLiDARs Benchmark is a recurring evaluation framework that standardizes LiDAR data acquisition, calibration, and quantitative metrics across diverse applications.
- It emphasizes extrinsic calibration, common-frame fusion, and task-specific design to enable robust assessments in automotive, roadside, and specialized LiDAR domains.
- Quantitative results highlight that sensor fusion, optimized beam distribution, and tailored evaluation metrics significantly enhance perception performance.
The available literature suggests that “InfraLiDARs’ Benchmark” functions less as a single immutable benchmark release than as a recurring benchmark label for LiDAR-centered evaluation programs spanning automotive perception, roadside infrastructure sensing, scanning-pattern analysis, single-photon sensing, cross-domain urban perception, ecological inventory, industrial terrestrial laser scanning, and hybrid indoor robot perception. Across these variants, the shared core is a calibrated acquisition pipeline, an explicit task definition, and standardized quantitative evaluation, with recurrent metrics including RMSE, MOTA, MOTP, ID-switches, mAP, IoU, mIoU, Chamfer Distance, Occupied Cells Ratio, Map Score, Pearson’s correlation coefficient, Mean Map Entropy, and related task-specific measures (Schalling et al., 2020, Kloeker et al., 2020, Dong et al., 16 Jun 2026, Sanchez et al., 2023).
1. Terminological scope and benchmark families
The benchmark family spans at least three broad regimes. First, some works use the benchmark concept to study sensor limitations and evaluation methodology, as in automotive LiDAR benchmarking based on Pseudo Ground Truth (PGT), occupancy-grid comparison, and mapping/localization characterization with Normal Distributions Transform (NDT) (Schalling et al., 2020, Carballo et al., 2020, Carballo et al., 2020). Second, several works use it for infrastructure and roadside perception, including fused multi-LiDAR traffic monitoring, roadside 3D object detection, controlled multi-resolution diagnostics, and repetitive-versus-non-repetitive scanning comparisons (Kloeker et al., 2020, Zimmer et al., 2023, Cui et al., 23 May 2026, Zhao et al., 19 Sep 2025, Qi et al., 28 Oct 2025). Third, the same benchmark logic appears in specialized LiDAR domains, such as single-photon perception, domain generalization, forest inventory, industrial TLS semantics, indoor hybrid sim-to-real perception, and adverse-condition multimodal SLAM (Dong et al., 16 Jun 2026, Sanchez et al., 2023, Chang et al., 16 Apr 2026, Yin et al., 30 Mar 2026, Li et al., 13 Dec 2025, Gong et al., 2024).
| Benchmark setting | Representative papers | Core emphasis |
|---|---|---|
| Automotive and sensor characterization | (Schalling et al., 2020, Carballo et al., 2020, Carballo et al., 2020) | PGT generation, sensor limitations, NDT mapping/localization |
| Roadside and infrastructure perception | (Kloeker et al., 2020, Zimmer et al., 2023, Cui et al., 23 May 2026, Zhao et al., 19 Sep 2025, Qi et al., 28 Oct 2025) | Fusion, placement, scan patterns, detection, tracking, localization |
| Specialized LiDAR domains | (Dong et al., 16 Jun 2026, Sanchez et al., 2023, Chang et al., 16 Apr 2026, Yin et al., 30 Mar 2026, Li et al., 13 Dec 2025, Gong et al., 2024) | SPL, domain shift, ecology, industrial TLS, hybrid indoor, adverse conditions |
This breadth is important because the phrase can otherwise be mistaken for a single dataset with a fixed train/validation/test split. The published record instead shows a family of benchmark constructions that share methodology but differ substantially in sensing physics, annotation strategy, and downstream task.
2. Acquisition geometry, calibration, and ground-truth construction
A defining trait of these benchmarks is the emphasis on extrinsic calibration and common-frame fusion. In fused infrastructure traffic recording, each sensor-frame point is mapped into a global frame by
with duplicate points removed within a 1 cm voxel; noisy-fusion experiments additionally inject Gaussian perturbations in translation and rotation to simulate calibration uncertainty (Kloeker et al., 2020). In SP-TransientBench, the auxiliary Livox Avia LiDAR and IMU generate drift-reduced trajectories , and checkerboard-based rigid alignment yields
which supplies per-view SPL camera poses (Dong et al., 16 Jun 2026).
The physical deployments vary widely. Infrastructure traffic sensing includes eight Ouster OS1-64 LiDARs in simulation at 6 m height, with two sensors per corner tilted downward by 0.1 rad and 0.3 rad, and a real four-sensor intersection setup at 2 m height with each sensor about 28 m from the center (Kloeker et al., 2020). Roadside detection on the A9 test stretch uses two Ouster OS1-64 LiDARs and two Basler RGB cameras mounted 20 m above ground on a gantry bridge (Zimmer et al., 2023). MR-LiDAR fixes four LiDARs of 16, 32, 80, and 128 beams sequentially on the same 1.5 m bracket to isolate beam-count and beam-distribution effects under identical geometry (Cui et al., 23 May 2026). Bench-RNR compares one 128-line Hesai OT128 with two Livox Avia units in an open-air parking lot (Zhao et al., 19 Sep 2025).
Ground truth is likewise benchmark-specific. Automotive PGT benchmarking proposes automated annotation from LiDAR data itself through denoising, temporal accumulation, and occupancy-grid generation, then compares PGT to a “true” GT grid from simulation or a ground-truth sensor suite (Schalling et al., 2020). Infrastructure traffic recording exports simulator trajectories directly for scenarios A–D and derives real-world reference trajectories from a DJI Phantom 4 Pro drone at 100 m altitude, stabilized and geo-registered in UTM (Kloeker et al., 2020). SP-TransientBench aggregates auxiliary Livox point clouds into dense reference depth maps (Dong et al., 16 Jun 2026). InfraDet3D uses synchronized infrastructure sensors with target-less LiDAR–LiDAR and LiDAR–camera calibration based on FPFH, ICP, and MLE (Zimmer et al., 2023).
These designs indicate that calibration is not merely a preprocessing detail; it is part of the benchmark definition itself.
3. Task structure and evaluation methodology
The task taxonomy ranges from occupancy-map evaluation to 3D detection, tracking, localization, reconstruction, semantics, and biomass estimation. In the automotive PGT formulation, the pipeline is explicitly divided into five discrete steps: sensor limitation analysis, data acquisition, automated annotation, world representation generation, and evaluation (Schalling et al., 2020). The generated point cloud is rasterized into an occupancy grid, and the benchmark uses map-based KPIs including Occupied Cells Ratio, Map Score, and Pearson’s correlation coefficient:
The same paper defines Pearson’s between PGT and GT occupancy maps over cellwise occupancy probabilities (Schalling et al., 2020).
Infrastructure tracking benchmarks center on trajectory quality. The fused-LiDAR traffic benchmark evaluates RMSE, MOTA, MOTP, and IDSW, with RMSE defined over all trajectories and frames as
DIDLM, which targets SLAM under snowy, rainy, nighttime, bumpy, and rough-road conditions, instead emphasizes Absolute Trajectory Error (ATE), Relative Pose Error (RPE), Drift per Meter, and Robustness Score (Kloeker et al., 2020, Gong et al., 2024).
Semantic and detection benchmarks typically adopt IoU-derived metrics. ParisLuco3D defines
while object detection uses AP sampled at 50 recall levels and class-specific IoU thresholds of 0.7 for four-wheel vehicles, 0.5 for two-wheel objects, and 0.3 for pedestrians (Sanchez et al., 2023). Industrial3D adds head mIoU, tail mIoU, and H-IoU to quantify the long-tailed regime under a 215:1 head–tail imbalance (Yin et al., 30 Mar 2026). SP-TransientBench combines Chamfer Distance, Recall@ for 0 temporal-bin tolerances, IoU, SSIM, LPIPS, 1 depth error, PSNR on rendered histograms, mIoU, and Overall Accuracy across its depth, multi-view reconstruction, and 3D semantic-understanding tasks (Dong et al., 16 Jun 2026). Roadside scanning-pattern studies additionally use distance-segmented AP and High-Quality Detection Area, defining a cell as high quality when AP 2 (Qi et al., 28 Oct 2025).
A recurrent methodological feature is that benchmark metrics are coupled tightly to the representation under test: occupancy probabilities for map comparison, boxes for detection and tracking, transient histograms for single-photon sensing, and stratified point distributions for forestry and industrial TLS.
4. Quantitative findings in infrastructure and roadside perception
The strongest consistent result in infrastructure sensing is the benefit of fusion over single-sensor operation. In simulated urban scenarios A–D, fused LiDAR improves Car AP from 0.42 to 0.91, Car MOTA from 0.63 to 0.85, Ped AP from 0.19 to 0.67, and Bike MOTA from 0.40 to 0.81; position RMSE falls from 0.38 m to 0.18 m. In the real four-sensor intersection scenario, Car MOTA rises from 0.22 to 0.31, Ped MOTA from 0.01 to 0.67, Bike MOTA from 0.06 to 0.21, and positional RMSE decreases from 0.77 m to 0.47 m. The same study reports that single-sensor object-box returns drop below 30 beyond 30 m, whereas fused sensing remains above 100 up to 60 m (Kloeker et al., 2020).
MR-LiDAR refines this picture by isolating beam count and beam distribution. It reports an effective perception range at 3 points of about 20 m for 16-beam, 30 m for 32-beam, and more than 80 m for 80-beam and 128-beam LiDARs on cars; for VRUs, the effective range is about 15 m, 20 m, 60 m, and more than 60 m, respectively. For cars at 20–40 m, the reported results are stark: 16-beam Recall 4, 32-beam Recall 5, and 80/128-beam Recall 6, with ATE 7 m for the two higher-resolution sensors. Crucially, the paper states that an 80-beam LiDAR with optimized beam distribution can match or even outperform a 128-beam LiDAR with uniform beam distribution (Cui et al., 23 May 2026).
The repetitive-versus-non-repetitive scanning studies complicate any simple “more beams is better” narrative. In the CARLA-based roadside benchmark, highway BEV AP for PointPillars is 55.0 for 16-line repetitive, 83.9 for 64-line repetitive, 92.4 for 128-line repetitive, and 91.4 for non-repetitive Livox Avia; for PV-RCNN, the corresponding values are 57.6, 87.4, 93.3, and 92.6. In the crossroad scenario, PointPillars yields 84.0 AP for non-repetitive sensing versus 78.5 for 128-line repetitive, while the non-repetitive device is listed at \$\mathbf{p}_i^G = R_s \mathbf{p}_i^S + \mathbf{t}_s,$824,000 (Qi et al., 28 Oct 2025). Bench-RNR reaches a similar conclusion from real-world localization: Register-Loc reports 6.84 cm / 3.43 cm / 16.59 cm center-error statistics with 0.88° / 0.61° / 2.50° heading error on 128-line repetitive data, and 6.87 cm / 3.34 cm / 17.00 cm with 1.03° / 0.82° / 3.65° on 2×Livox Avia, indicating comparable localization accuracy when template registration is available (Zhao et al., 19 Sep 2025).
Roadside multimodal detection reinforces the value of combining geometry and appearance. InfraDet3D reports 68.48 mAP on the A9-I south1 test subset at mAP$\mathbf{p}_i^G = R_s \mathbf{p}_i^S + \mathbf{t}_s,$9. The same study states that fusing two LiDARs alone yields +1.32 pp mAP over single-LiDAR PointPillars, and adding cameras contributes another +1.90 pp (Zimmer et al., 2023).
Outside traffic sensing, Industrial3D quantifies a different failure mode: severe long-tail semantics and geometric ambiguity. Its best fully supervised method, Boundary-CB, attains 55.74% mIoU, whereas zero-shot Point-SAM reaches 15.79%, a 39.95 percentage-point gap attributed to both statistical rarity and geometric ambiguity (Yin et al., 30 Mar 2026).
5. Specialized benchmark variants beyond conventional roadside LiDAR
Single-photon sensing is represented by SP-TransientBench (STB), described as the first publicly released, large-scale, real-captured single-photon LiDAR benchmark for depth estimation, multi-view 3D reconstruction, and 3D semantic understanding. STB contains 10 diverse scenes and 10,297 views captured at $X_{\text{livox}}(i)\in\mathbb{R}^3$0 resolution, with $X_{\text{livox}}(i)\in\mathbb{R}^3$1 temporal bins and $X_{\text{livox}}(i)\in\mathbb{R}^3$2 ps per bin. Each view stores a full $X_{\text{livox}}(i)\in\mathbb{R}^3$3 transient histogram, and the benchmark includes 13-class 3D semantic annotations. Its forward model treats photon detections per bin as Poisson:
$X_{\text{livox}}(i)\in\mathbb{R}^3$4
with ToF-to-depth conversion
$X_{\text{livox}}(i)\in\mathbb{R}^3$5
The benchmark explicitly targets the multi-return and photon-starved regimes that simulated datasets often fail to reproduce (Dong et al., 16 Jun 2026).
For cross-domain urban perception, ParisLuco3D acts as a held-out target domain for zero-shot evaluation. It contains 7,501 full 360° scans over a 2.1 km loop in central Paris, collected with a Velodyne HDL-32E mounted vertically at about 3.70 m above ground. It provides 45 fine-grained semantic classes, 11 road-user categories for detection, and tracking across annotated frames, with participants required not to fine-tune on ParisLuco3D labels (Sanchez et al., 2023).
In forestry, the benchmark of long-term ecological monitoring sites integrates ULS, TLS, and MLS in a marker-free registration hierarchy. For the 35 m radius control plot CP2, the final products include 6,851,016 ULS points, 203,657,957 TLS points, and 32,859,532 MLS points, all in UTM Zone 32N (EPSG:32632). Evaluation is organized around registration accuracy, scanning efficiency, segmentation/QSM, and allometric biomass estimation, with explicit formulas for cylinder volume and tree volume (Chang et al., 16 Apr 2026).
INDOOR-LiDAR extends the benchmark logic to robot-centric 360-degree indoor perception by coupling a simulated subset of about 5,000 m² across classroom, laboratory, restaurant, and hospital scenes with a real subset of about 800 m² in offices, corridors, dining halls, lobbies, and laboratories. It supports 3D detection, BEV detection, SLAM, semantic segmentation, and domain adaptation, and reports that DLIO and LIO-SAM achieve sub-decimeter ATE (about 0.04–0.06 m RMSE) on real trajectories (Li et al., 13 Dec 2025).
Automotive benchmark lineages remain foundational. LIBRE evaluates 10 different 3D LiDAR sensors under static targets, adverse weather, and dynamic traffic (Carballo et al., 2020). Its NDT-based characterization study defines Mean Map Entropy and Mean Plane Variance as mapping metrics and reports that all evaluated LiDARs achieve sub-meter alignment against the MMS ground-truth map, with differing map entropy, plane variance, and convergence behavior (Carballo et al., 2020).
6. Limitations, misconceptions, and benchmark implications
A central misconception is that every InfraLiDAR benchmark is a ready-made leaderboard with complete numerical baselines. The automotive PGT paper explicitly does not present a “plug-and-play” benchmark with off-the-shelf numerical results; it contributes a methodology for generating PGT and evaluating LiDAR perception, but no completed case studies or numeric benchmark tables appear in the paper (Schalling et al., 2020). This distinguishes framework papers from mature benchmark releases.
Several specialized benchmarks carry explicit caveats. SP-TransientBench notes that cross-sensor alignment relies on auxiliary LiDAR, so residual SLAM drift may bias absolute metrics, and all scenes were captured with a single SPAD device/configuration (Dong et al., 16 Jun 2026). DIDLM provides extensive adverse-condition data but no official train/val/test split, leaving route-level or condition-level partitioning to the experimenter (Gong et al., 2024). Industrial3D shows that class-frequency re-weighting alone cannot resolve the industrial TLS regime because the bottleneck is not only imbalance but also cylindrical primitive sharing between head and tail classes (Yin et al., 30 Mar 2026).
Roadside studies also challenge overly simple hardware assumptions. The MR-LiDAR results dispute the claim that higher beam counts always yield better perception, and the repetitive-versus-non-repetitive benchmark shows that non-repetitive scanning LiDAR and the 128-line repetitive LiDAR can exhibit comparable detection performance across various scenarios, despite the non-repetitive device’s narrower forward field and limited perception range (Cui et al., 23 May 2026, Qi et al., 28 Oct 2025). A plausible implication is that benchmark design must account for beam distribution, field-of-view geometry, occlusion topology, and task-specific deployment range, not merely nominal channel count.
Taken together, these works define InfraLiDAR benchmarking as a methodological program rather than a single canonical dataset: rigorous calibration, explicit sensing assumptions, domain-appropriate ground truth, and metrics that are tightly matched to the physical and algorithmic structure of the problem.