---
title: InfraLiDARs Benchmark Overview
url: https://www.emergentmind.com/topics/infralidars-benchmark
type: topic
---

# InfraLiDARs Benchmark Overview

The available literature suggests that **“InfraLiDARs’ Benchmark”** functions less as a single immutable benchmark release than as a recurring benchmark label for LiDAR-centered evaluation programs spanning automotive perception, roadside infrastructure sensing, scanning-pattern analysis, single-photon sensing, cross-domain urban perception, ecological inventory, industrial terrestrial laser scanning, and hybrid indoor robot perception. Across these variants, the shared core is a calibrated acquisition pipeline, an explicit task definition, and standardized quantitative evaluation, with recurrent metrics including RMSE, MOTA, MOTP, ID-switches, mAP, IoU, mIoU, Chamfer Distance, Occupied Cells Ratio, Map Score, Pearson’s correlation coefficient, Mean Map Entropy, and related task-specific measures [2004.13433] [2006.12140] [2606.18952] [2310.16542].

## 1. Terminological scope and benchmark families

The benchmark family spans at least three broad regimes. First, some works use the benchmark concept to study **sensor limitations and evaluation methodology**, as in automotive LiDAR benchmarking based on Pseudo Ground Truth (PGT), occupancy-grid comparison, and mapping/localization characterization with Normal Distributions Transform (NDT) [2004.13433] [2004.01374] [2003.06129]. Second, several works use it for **infrastructure and roadside perception**, including fused multi-LiDAR traffic monitoring, roadside 3D object detection, controlled multi-resolution diagnostics, and repetitive-versus-non-repetitive scanning comparisons [2006.12140] [2305.00314] [2605.24777] [2509.15583] [2511.00060]. Third, the same benchmark logic appears in **specialized LiDAR domains**, such as single-photon perception, domain generalization, forest inventory, industrial TLS semantics, indoor hybrid sim-to-real perception, and adverse-condition multimodal SLAM [2606.18952] [2310.16542] [2604.14635] [2603.28660] [2512.12377] [2404.09622].

| Benchmark setting | Representative papers | Core emphasis |
|---|---|---|
| Automotive and sensor characterization | [2004.13433], [2004.01374], [2003.06129] | PGT generation, sensor limitations, NDT mapping/localization |
| Roadside and infrastructure perception | [2006.12140], [2305.00314], [2605.24777], [2509.15583], [2511.00060] | Fusion, placement, scan patterns, detection, tracking, localization |
| Specialized LiDAR domains | [2606.18952], [2310.16542], [2604.14635], [2603.28660], [2512.12377], [2404.09622] | SPL, domain shift, ecology, industrial TLS, hybrid indoor, adverse conditions |

This breadth is important because the phrase can otherwise be mistaken for a single dataset with a fixed train/validation/test split. The published record instead shows a family of benchmark constructions that share methodology but differ substantially in sensing physics, annotation strategy, and downstream task.

## 2. Acquisition geometry, calibration, and ground-truth construction

A defining trait of these benchmarks is the emphasis on **extrinsic calibration and common-frame fusion**. In fused infrastructure traffic recording, each sensor-frame point $\mathbf{p}_i^S$ is mapped into a global frame by
$$
\mathbf{p}_i^G = R_s \mathbf{p}_i^S + \mathbf{t}_s,
$$
with duplicate points removed within a 1 cm voxel; noisy-fusion experiments additionally inject Gaussian perturbations in translation and rotation to simulate calibration uncertainty [2006.12140]. In SP-TransientBench, the auxiliary Livox Avia LiDAR and IMU generate drift-reduced trajectories $X_{\text{livox}}(i)\in\mathbb{R}^3$, and checkerboard-based rigid alignment yields
$$
X_{\text{spl}} = R\cdot X_{\text{livox}} + t,
$$
which supplies per-view SPL camera poses $\{R_i,t_i\}$ [2606.18952].

The physical deployments vary widely. Infrastructure traffic sensing includes **eight Ouster OS1-64 LiDARs** in simulation at **6 m height**, with two sensors per corner tilted downward by **0.1 rad** and **0.3 rad**, and a real four-sensor intersection setup at **2 m height** with each sensor about **28 m** from the center [2006.12140]. Roadside detection on the A9 test stretch uses **two Ouster OS1-64 LiDARs** and **two Basler RGB cameras** mounted **20 m above ground** on a gantry bridge [2305.00314]. MR-LiDAR fixes four LiDARs of **16, 32, 80, and 128 beams** sequentially on the same **1.5 m bracket** to isolate beam-count and beam-distribution effects under identical geometry [2605.24777]. Bench-RNR compares one **128-line Hesai OT128** with **two Livox Avia** units in an open-air parking lot [2509.15583].

Ground truth is likewise benchmark-specific. Automotive PGT benchmarking proposes automated annotation from LiDAR data itself through denoising, temporal accumulation, and occupancy-grid generation, then compares PGT to a “true” GT grid from simulation or a ground-truth sensor suite [2004.13433]. Infrastructure traffic recording exports simulator trajectories directly for scenarios A–D and derives real-world reference trajectories from a **DJI Phantom 4 Pro drone** at **100 m altitude**, stabilized and geo-registered in UTM [2006.12140]. SP-TransientBench aggregates auxiliary Livox point clouds into dense reference depth maps [2606.18952]. InfraDet3D uses synchronized infrastructure sensors with target-less LiDAR–LiDAR and LiDAR–camera calibration based on FPFH, ICP, and MLE [2305.00314].

These designs indicate that calibration is not merely a preprocessing detail; it is part of the benchmark definition itself.

## 3. Task structure and evaluation methodology

The task taxonomy ranges from occupancy-map evaluation to 3D detection, tracking, localization, reconstruction, semantics, and biomass estimation. In the automotive PGT formulation, the pipeline is explicitly divided into **five discrete steps**: sensor limitation analysis, data acquisition, automated annotation, world representation generation, and evaluation [2004.13433]. The generated point cloud is rasterized into an occupancy grid, and the benchmark uses map-based KPIs including **Occupied Cells Ratio**, **Map Score**, and **Pearson’s correlation coefficient**:
$$
OC_R = \frac{\text{number of cells marked occupied in PGT}}{\text{number of cells occupied in GT}}.
$$
The same paper defines Pearson’s $r$ between PGT and GT occupancy maps over cellwise occupancy probabilities [2004.13433].

Infrastructure tracking benchmarks center on trajectory quality. The fused-LiDAR traffic benchmark evaluates **RMSE**, **MOTA**, **MOTP**, and **IDSW**, with RMSE defined over all trajectories and frames as
$$
\mathrm{RMSE}=\sqrt{\frac{1}{\sum_{n=1}^{N}T_n}\sum_{n=1}^{N}\sum_{t=1}^{T_n}\left\|\hat{\mathbf{x}}_t^{(n)}-\mathbf{x}_t^{(n)}\right\|_2^2}.
$$
DIDLM, which targets SLAM under snowy, rainy, nighttime, bumpy, and rough-road conditions, instead emphasizes **Absolute Trajectory Error (ATE)**, **Relative Pose Error (RPE)**, **Drift per Meter**, and **Robustness Score** [2006.12140] [2404.09622].

Semantic and detection benchmarks typically adopt IoU-derived metrics. ParisLuco3D defines
$$
\mathrm{IoU}_c=\frac{TP_c}{TP_c+FP_c+FN_c}, \qquad
\mathrm{mIoU}=\frac{1}{C}\sum_{c=1}^{C}\mathrm{IoU}_c,
$$
while object detection uses AP sampled at **50 recall levels** and class-specific IoU thresholds of **0.7** for four-wheel vehicles, **0.5** for two-wheel objects, and **0.3** for pedestrians [2310.16542]. Industrial3D adds **head mIoU**, **tail mIoU**, and **H-IoU** to quantify the long-tailed regime under a **215:1** head–tail imbalance [2603.28660]. SP-TransientBench combines **Chamfer Distance**, **Recall@$\delta$** for **$\delta=1,3,5$ temporal-bin tolerances**, **IoU**, **SSIM**, **LPIPS**, **$L_1$ depth error**, **PSNR on rendered histograms**, **mIoU**, and **Overall Accuracy** across its depth, multi-view reconstruction, and 3D semantic-understanding tasks [2606.18952]. Roadside scanning-pattern studies additionally use **distance-segmented AP** and **High-Quality Detection Area**, defining a cell as high quality when **AP $\ge 0.85$** [2511.00060].

A recurrent methodological feature is that benchmark metrics are coupled tightly to the representation under test: occupancy probabilities for map comparison, boxes for detection and tracking, transient histograms for single-photon sensing, and stratified point distributions for forestry and industrial TLS.

## 4. Quantitative findings in infrastructure and roadside perception

The strongest consistent result in infrastructure sensing is the benefit of **fusion over single-sensor operation**. In simulated urban scenarios A–D, fused LiDAR improves **Car AP from 0.42 to 0.91**, **Car MOTA from 0.63 to 0.85**, **Ped AP from 0.19 to 0.67**, and **Bike MOTA from 0.40 to 0.81**; position RMSE falls from **0.38 m** to **0.18 m**. In the real four-sensor intersection scenario, **Car MOTA rises from 0.22 to 0.31**, **Ped MOTA from 0.01 to 0.67**, **Bike MOTA from 0.06 to 0.21**, and positional RMSE decreases from **0.77 m** to **0.47 m**. The same study reports that single-sensor object-box returns drop below **30** beyond **30 m**, whereas fused sensing remains above **100** up to **60 m** [2006.12140].

MR-LiDAR refines this picture by isolating beam count and beam distribution. It reports an **effective perception range at $\ge 40$ points** of about **20 m** for **16-beam**, **30 m** for **32-beam**, and **more than 80 m** for **80-beam** and **128-beam** LiDARs on cars; for VRUs, the effective range is about **15 m**, **20 m**, **60 m**, and **more than 60 m**, respectively. For cars at **20–40 m**, the reported results are stark: **16-beam Recall $\le 5\%$**, **32-beam Recall $\sim 42\%$**, and **80/128-beam Recall $\ge 89\%$**, with **ATE $\le 0.15$ m** for the two higher-resolution sensors. Crucially, the paper states that an **80-beam LiDAR with optimized beam distribution can match or even outperform a 128-beam LiDAR with uniform beam distribution** [2605.24777].

The repetitive-versus-non-repetitive scanning studies complicate any simple “more beams is better” narrative. In the CARLA-based roadside benchmark, highway BEV AP for **PointPillars** is **55.0** for **16-line repetitive**, **83.9** for **64-line repetitive**, **92.4** for **128-line repetitive**, and **91.4** for **non-repetitive Livox Avia**; for **PV-RCNN**, the corresponding values are **57.6**, **87.4**, **93.3**, and **92.6**. In the crossroad scenario, **PointPillars** yields **84.0** AP for non-repetitive sensing versus **78.5** for 128-line repetitive, while the non-repetitive device is listed at **\$2,079** and the 128-line device at **\$24,000** [2511.00060]. Bench-RNR reaches a similar conclusion from real-world localization: **Register-Loc** reports **6.84 cm / 3.43 cm / 16.59 cm** center-error statistics with **0.88° / 0.61° / 2.50°** heading error on **128-line repetitive** data, and **6.87 cm / 3.34 cm / 17.00 cm** with **1.03° / 0.82° / 3.65°** on **2×Livox Avia**, indicating comparable localization accuracy when template registration is available [2509.15583].

Roadside multimodal detection reinforces the value of combining geometry and appearance. InfraDet3D reports **68.48 mAP** on the A9-I south1 test subset at **mAP$_{3D}@\mathrm{IoU}=0.1$**. The same study states that **fusing two LiDARs alone yields +1.32 pp mAP over single-LiDAR PointPillars**, and **adding cameras contributes another +1.90 pp** [2305.00314].

Outside traffic sensing, Industrial3D quantifies a different failure mode: severe long-tail semantics and geometric ambiguity. Its best fully supervised method, **Boundary-CB**, attains **55.74% mIoU**, whereas zero-shot **Point-SAM** reaches **15.79%**, a **39.95 percentage-point gap** attributed to both **statistical rarity** and **geometric ambiguity** [2603.28660].

## 5. Specialized benchmark variants beyond conventional roadside LiDAR

Single-photon sensing is represented by **SP-TransientBench (STB)**, described as the **first publicly released, large-scale, real-captured single-photon LiDAR benchmark** for **depth estimation**, **multi-view 3D reconstruction**, and **3D semantic understanding**. STB contains **10 diverse scenes** and **10,297 views** captured at **$256\times192$** resolution, with **$K=672$ temporal bins** and **$\Delta t = 750$ ps** per bin. Each view stores a full **$H\times W\times K$** transient histogram, and the benchmark includes **13-class 3D semantic annotations**. Its forward model treats photon detections per bin as Poisson:
$$
P(k;\lambda)=\frac{\lambda^k e^{-\lambda}}{k!},
$$
with ToF-to-depth conversion
$$
d=\frac{c\cdot \Delta t \cdot k}{2}.
$$
The benchmark explicitly targets the multi-return and photon-starved regimes that simulated datasets often fail to reproduce [2606.18952].

For cross-domain urban perception, **ParisLuco3D** acts as a held-out target domain for zero-shot evaluation. It contains **7,501 full 360° scans** over a **2.1 km loop** in central Paris, collected with a **Velodyne HDL-32E** mounted vertically at about **3.70 m** above ground. It provides **45 fine-grained semantic classes**, **11 road-user categories** for detection, and tracking across annotated frames, with participants required **not to fine-tune on ParisLuco3D labels** [2310.16542].

In forestry, the benchmark of long-term ecological monitoring sites integrates **ULS**, **TLS**, and **MLS** in a marker-free registration hierarchy. For the **35 m radius** control plot CP2, the final products include **6,851,016 ULS points**, **203,657,957 TLS points**, and **32,859,532 MLS points**, all in **UTM Zone 32N (EPSG:32632)**. Evaluation is organized around registration accuracy, scanning efficiency, segmentation/QSM, and allometric biomass estimation, with explicit formulas for cylinder volume and tree volume [2604.14635].

**INDOOR-LiDAR** extends the benchmark logic to robot-centric 360-degree indoor perception by coupling a simulated subset of about **5,000 m²** across **classroom, laboratory, restaurant, and hospital** scenes with a real subset of about **800 m²** in offices, corridors, dining halls, lobbies, and laboratories. It supports **3D detection**, **BEV detection**, **SLAM**, **semantic segmentation**, and **domain adaptation**, and reports that **DLIO** and **LIO-SAM** achieve **sub-decimeter ATE (about 0.04–0.06 m RMSE)** on real trajectories [2512.12377].

Automotive benchmark lineages remain foundational. **LIBRE** evaluates **10 different 3D LiDAR sensors** under static targets, adverse weather, and dynamic traffic [2003.06129]. Its NDT-based characterization study defines **Mean Map Entropy** and **Mean Plane Variance** as mapping metrics and reports that all evaluated LiDARs achieve **sub-meter alignment** against the MMS ground-truth map, with differing map entropy, plane variance, and convergence behavior [2004.01374].

## 6. Limitations, misconceptions, and benchmark implications

A central misconception is that every InfraLiDAR benchmark is a ready-made leaderboard with complete numerical baselines. The automotive PGT paper explicitly does **not** present a “plug-and-play” benchmark with off-the-shelf numerical results; it contributes a **methodology** for generating PGT and evaluating LiDAR perception, but **no completed case studies or numeric benchmark tables** appear in the paper [2004.13433]. This distinguishes framework papers from mature benchmark releases.

Several specialized benchmarks carry explicit caveats. SP-TransientBench notes that **cross-sensor alignment relies on auxiliary LiDAR**, so **residual SLAM drift may bias absolute metrics**, and all scenes were captured with **a single SPAD device/configuration** [2606.18952]. DIDLM provides extensive adverse-condition data but **no official train/val/test split**, leaving route-level or condition-level partitioning to the experimenter [2404.09622]. Industrial3D shows that class-frequency re-weighting alone cannot resolve the industrial TLS regime because the bottleneck is not only imbalance but also **cylindrical primitive sharing** between head and tail classes [2603.28660].

Roadside studies also challenge overly simple hardware assumptions. The MR-LiDAR results dispute the claim that **higher beam counts always yield better perception**, and the repetitive-versus-non-repetitive benchmark shows that **non-repetitive scanning LiDAR and the 128-line repetitive LiDAR** can exhibit **comparable detection performance across various scenarios**, despite the non-repetitive device’s narrower forward field and limited perception range [2605.24777] [2511.00060]. A plausible implication is that benchmark design must account for **beam distribution**, **field-of-view geometry**, **occlusion topology**, and **task-specific deployment range**, not merely nominal channel count.

Taken together, these works define InfraLiDAR benchmarking as a methodological program rather than a single canonical dataset: rigorous calibration, explicit sensing assumptions, domain-appropriate ground truth, and metrics that are tightly matched to the physical and algorithmic structure of the problem.

Source: https://www.emergentmind.com/topics/infralidars-benchmark