RESOLVE: A Multi-Resolution and Multi-Modal Dataset for Roadside Cooperative Perception

Abstract: LiDAR has increasingly been integrated into traffic cameras to expand coverage and mitigate occlusion in roadside cooperative perception. However, how unimodal and camera-LiDAR fusion architectures behave under variations in LiDAR point sparsity induced by sensor configurations and scene-dependent sensing conditions remains underexplored. We introduce RESOLVE, a large-scale real-world benchmark dataset featuring multi-resolution roadside LiDAR and synchronized camera-LiDAR sensing for systematic evaluation of unimodal and fusion-based architectures in roadside 3D detection and tracking. RESOLVE contains over 100k images and 26k point cloud frames with 220k manually annotated bounding boxes, captured at a real-world urban intersection across diverse lighting and weather conditions and spanning 10 classes of traffic participants. In particular, RESOLVE enables controlled evaluation across three LiDAR resolution levels while keeping all other sensing and environmental factors fixed. This allows fair cross-architecture comparisons under point cloud distribution shifts resulting from resolution variations, sensing distance, and training-inference resolution mismatches. Results from extensive benchmark experiments reveal insights into how multimodal fusion can compensate for LiDAR point sparsity, offering clues for designing cost-efficient roadside multimodal perception. The dataset and benchmark codes are available at https://github.com/ASU-Suo-Lab/RESOLVE.

• The paper presents RESOLVE, a novel dataset evaluating how varying LiDAR resolutions and sensor modalities impact 3D object detection and tracking in roadside settings.
• It details rigorous sensor calibration, synchronization, and controlled resolution regimes to assess cost-performance trade-offs in realistic intelligent transportation scenarios.
• Empirical findings underscore that multimodal fusion can mitigate low LiDAR resolution effects, while resolution mismatches drastically reduce detection performance, urging advanced domain adaptation.

RESOLVE: A Multi-Resolution and Multi-Modal Dataset for Roadside Cooperative Perception

Motivation and Context

Addressing the expanding role of infrastructure-based sensing in Intelligent Transportation Systems (ITS), RESOLVE (2606.31895) introduces a real-world benchmark dataset that systematically evaluates how LiDAR resolution—i.e., point sparsity—and sensor modality (LiDAR/camera) jointly impact 3D object detection and tracking performance in complex urban roadside perception settings. Unlike prior datasets where LiDAR configuration is static, RESOLVE enables controlled assessment of perception approaches under sensor deployment heterogeneity, replicating practical scenarios driven by budgetary, hardware, or installation constraints. The dataset is shaped to illuminate the performance trade-offs between unimodal LiDAR, unimodal camera, and LiDAR-camera fusion models across low, medium, and high LiDAR resolutions, as well as under conditions of domain shift induced by mismatched sensor resolutions between training and inference.

Dataset and Sensor Configuration

RESOLVE consists of over 100,000 images and 26,000 point cloud frames, annotated with 220,000 3D bounding boxes across 10 traffic participant classes, covering complex intersection topologies under diverse weather and illumination (including rare traffic classes like golf carts). Its sensor layout comprises four high-resolution traffic cameras and six LiDAR scanners (16-, 64-, and 128-beam variants) in two symmetric roadside clusters at diagonally opposed intersection corners.

Figure 1: A scene-level visualization of RESOLVE’s multi-resolution LiDAR point clouds, annotated sensors, and urban simulation map for reproducibility and precise ground-truth alignment.

Sensor synchronization is achieved via a PTP-disciplined edge compute node governing all LiDARs and NTP-coordinated cameras. Calibration between all sensors is carefully realized through checkerboard-based camera intrinsics, 2D-3D correspondences for camera-LiDAR extrinsics, and ICP refinement for inter-LiDAR alignment in the shared coordinate frame. This design ensures temporally and spatially consistent multi-modality at frame-level across reproducible conditions.

Figure 2: Visualization of LiDAR-LiDAR cross-calibration, showing dense registration at different resolutions to ensure geometric alignment for fair comparative evaluation.

The dataset is partitioned into three resolution regimes determined by the LiDAR beam count, with all other sensing factors held fixed. This design provides a unique axis for controlled ablation of point cloud density effects—critical for studying cost-performance trade-offs in practical deployments.

Benchmark Tasks and Statistical Properties

The benchmark spans:

• 3D Object Detection using both unimodal (LiDAR-only) and multimodal (LiDAR+camera) detectors, covering backbones based on SparseConv, Transformers, and State-Space Models (e.g., Mamba);
• 3D Multi-Object Tracking using tracking-by-detection paradigms, reported with AMOTA/AMOTP under the nuScenes metrics;
• Roadside Cooperative Perception, leveraging agent-level data fusion for distributed infrastructure setups.

The annotation pipeline enforces a minimum point-count threshold for 3D box validity per resolution (without excluding low-supervision boxes from training/evaluation), thus preserving statistical comparability. The analysis confirms that point density and geometric fidelity increase nonlinearly with resolution, benefiting large objects disproportionately at longer ranges, while small- and VRU-class targets (bicycles, motorcycles, pedestrians) suffer most severely under low resolution.

Figure 4: Analysis of LiDAR point density as a function of both resolution and object distance, with close-in targets most sensitive to sensor sparsity and resolution differences diminishing at increased range.

Strong Empirical Results and Contradictory Findings

3D Detection Gains from Resolution and Fusion

Comprehensive benchmarking shows:

• Increasing LiDAR resolution from low to medium yields a mean mAP boost of 5–9% across architectures, with diminishing returns (∼1% gain) at high resolution—underscoring that mid-resolution configurations suffice for capturing most geometric discriminability for common vehicle classes.
• SparseConv models benefit most from increased intra-voxel density, whereas Transformers and Mamba-based models demonstrate robustness to sparsity, but gain more from the medium resolution’s increased local structure.
• Multimodal (LiDAR+camera) methods demonstrate the capacity to compensate resolution loss: unimodal high-res mAP can be matched (or occasionally exceeded) by fusion models at lower LiDAR resolution.

Tracking and Cooperative Perception

Tracking performance, as expected, is highly detector-dependent; resolution increases reduce AMOTP by over 14% moving from mid to high, but provide non-monotonic AMOTA changes due to interaction with association strategies and track fragmentation on small classes. Intermediate fusion methods (feature-level exchange) in cooperative perception maximize the benefit at medium resolution, but cannot compensate for severe sparsity at the lowest beam counts.

Figure 6: Agent-level performance for cooperative perception under varying LiDAR resolutions, showing that intermediate fusion dominates early/late fusion in the medium regime, with all fusion paradigms limited by low-res input.

Effects of Resolution Mismatch and Point-Cloud Downsampling

A pronounced claim is established regarding the dangers of training-inference LiDAR resolution mismatch: mAP drops up to 40% when a model trained at low resolution is evaluated on high-res data (and vice versa for some model types), revealing a severe domain gap. Notably, common practice of simulating low-resolution LiDAR by random beam downsampling of high-res data fails to reproduce the statistical and detection profile of real low-beam hardware, due to unmodeled effects of beam angle layout and physical occlusion.

This contradicts assumptions in prior simulation-driven domain adaptation research that naive subsampling suffices for benchmarking or data augmentation in real-world sensing diversity.

Multimodal Feature Learning: Interpretability and Processing Dynamics

t-SNE analysis of backbone feature spaces shows that higher resolution LiDAR increases both inter-class variance and intra-class compactness, improving categorical separability. However, standard training of multimodal fusion models with LiDAR branch gradients reduced early-layer discriminability in LiDAR features, while boosting final performance and fusion-layer separability. This indicates a cross-modal regularization effect: fusion models learn LiDAR descriptors aligned to camera semantics, not just point-density cues. Truncated-gradient ablations further confirm that multi-modal loss changes early LiDAR representation distribution, supporting the conclusion that multimodal training encourages domain-agnostic, cross-modal features—enabling robust performance with lower-cost, sparse LiDAR.

Figure 3: Effect of multimodal fusion on feature representation: standard fusion training improves fusion-layer discriminability despite suppressed early-layer variance (top: mAP=93.1 vs bottom: mAP=89.4 with frozen LiDAR branch).

Practical and Theoretical Implications

RESOLVE enables rigorous, controlled evaluation of the cost-performance trade space in cooperative perception system design—permitting quantification of performance degradation vs. sensor sparsity, robustness of model architectures under domain shift, and the empirical value of camera fusion under adverse observation conditions. This informs practical deployment decisions regarding infrastructure sensor investment, redundancy, and cross-agent data sharing protocols.

The strong, often non-intuitive findings regarding resolution mismatch and downsampling highlight urgent needs for:

• Resolution-robust representation learning;
• Domain adaptation and model calibration methods that explicitly address hardware-level sensor heterogeneity;
• Physics-aware simulation for data augmentation beyond simple subsampling.

Future system designs can leverage these insights to build cost-effective, resilient perception stacks, balancing high downstream accuracy with budgetary and deployment constraints.

Limitations and Future Directions

While resolving many prior limitations, RESOLVE is currently restricted to data from a single urban intersection. Extending to multi-intersection or corridor-level deployments and investigating generalization/scalability of perception and cooperation protocols remain open. The dataset and code are provided for public benchmarking and extension.

Conclusion

RESOLVE (2606.31895) delivers a pivotal resource for empirical, architecture-agnostic assessment of roadside perception systems under multi-resolution, multi-modal infrastructure sensing. By isolating the effect of LiDAR resolution in real world, supplementing unimodal and multimodal analysis, and exposing critical domain mismatch phenomena, it supports both fundamental research in robust representation/learning and practical standards development for cost-efficient intelligent transportation infrastructure.