JRDB-PanoTrack: Robotics Panoptic Benchmark

• The paper introduces a robust benchmark that integrates synchronized panoramic 2D imagery and 3D LiDAR data for automated panoptic segmentation and tracking in crowded human environments.
• It employs fine-grained annotations and unified protocols for both closed- and open-world evaluation, addressing challenges like long-tail class diversity and extreme object density.
• The benchmark drives innovations in multi-modal robotic perception, paving the way for advanced social robotics, autonomous navigation, and human trajectory forecasting.

JRDB-PanoTrack is a large-scale, open-world panoptic segmentation and tracking benchmark specifically designed for robotics and autonomous agents operating in crowded human environments. As a successor and extension of the original JRDB dataset, JRDB-PanoTrack provides synchronized multi-modal (2D and 3D) sensory streams, comprehensive panoptic and tracking annotations, and protocols that stress both closed- and open-world robotic perception scenarios. Its combination of panoramic coverage, fine-grained annotation, and long-tail class diversity establishes it as a uniquely demanding resource for advancing robotic environmental perception and understanding in the wild (Le et al., 2024).

1. Dataset Structure and Modalities

JRDB-PanoTrack is constructed from 54 high-resolution video sequences captured by a mobile social-manipulator robot in both indoor and outdoor, human-crowded environments. The dataset comprises 20,000 frames (sampled at 1 Hz from the original 30 Hz streams), with an even split of 27 sequences (9,365 frames) for training/validation and 27 sequences (9,280 frames) for testing.

Sensor Modalities and Calibration

• Visual: Five global-shutter RGB cameras provide stitched, hardware-synchronized 360° panoramic imagery at 752×480 pixels, producing frame-aligned perspectives at 1 Hz for annotation.
• Range: A Velodyne HDL-32 3D LiDAR sensor delivers approximately 100,000 points per sweep at 10 Hz, subsampled at 1 Hz for synchronized annotation (yielding 4,000 annotated point clouds).
• All sensing modalities are spatially calibrated to a unified robot-centric coordinate frame, enabling joint 2D and 3D reasoning.

Scene Richness

JRDB-PanoTrack sequences capture varying crowd densities (including scenes with >1,000 simultaneously tracked objects in panoramic views), diverse environments (indoor/outdoor), and a wide range of object types and spatial arrangements.

2. Annotation Scheme and Label Projection

2D Panoptic Segmentation

Each image is annotated with both “thing” instance masks (e.g., pedestrian, bicycle, laptop) and “stuff” semantic regions (e.g., wall, floor, sky) using the panoptic segmentation paradigm [Kirillov et al., 2019]. The label taxonomy encompasses:

• 61 thing classes
• 11 stuff classes This results in 72 unique semantic labels. The dataset provides 428,000 panoptic masks and 7.3 billion labeled pixels.

Temporal Instance Tracking

For “thing” classes, each visible object instance is assigned a unique persistent track ID within its sequence, permitting long-term trajectory continuity (average tracklet length: 16 s; maximum: 117 s; with dense scenes exceeding 1,000 concurrent trajectories).

3D Label Projections

2D masks are projected into the LiDAR’s 3D space via known extrinsic and intrinsic calibration, assigning panoptic labels to corresponding LiDAR points. This “pseudo-ground-truth” 3D panoptic segmentation allows for direct training and evaluation of spatial and multi-modal scene understanding methods.

3. Recognition Tasks and Benchmark Protocols

JRDB-PanoTrack explicitly separates closed- and open-world perception:

• Closed-World: 43 “known” object classes (each with >10,000 mask instances) are used for supervised segmentation/tracking.
• Open-World: The full label set (72 classes), with 29 “unknown” classes forming a long-tail evaluation challenge; 28 of these are held out from training.

Benchmark Task Splits

1. Closed-World Panoptic Segmentation: Models are trained and evaluated only on the 43 known classes.
2. Open-World Panoptic Segmentation: Training on known, evaluation on all 72 classes, testing out-of-distribution and rare object recognition.
3. Closed-World Panoptic Tracking: Trajectory and instance evaluation for known classes.
4. Open-World Panoptic Tracking: Tracking across all 72 classes, including previously unseen objects.

Both 2D and 3D segmented data are available for each protocol, and the dataset is aligned with the structure of contemporary multi-modal robotics benchmarks (cf. (Martín-Martín et al., 2019)).

4. Evaluation Metrics

Panoptic Quality (PQ)

PQ measures the spatial and categorical accuracy of segmentation using the formula:

$$\mathrm{PQ} =\frac{\sum_{(p,g)\in TP}\mathrm{IoU}(p,g)} {|TP|+\tfrac12\,|FP|+\tfrac12\,|FN|}$$

where TP denotes matched predicted and ground-truth regions (IoU > 0.5), FP and FN are unmatched regions.

OSPA-Based Metrics for Open-World Evaluation

The Optimal Subpattern Assignment (OSPA) framework rigorously quantifies set-level differences, accommodating unmatched (unknown) object sets with a cardinality penalty:

$$\mathrm{OSPA}_{p,c}(X,Y) =\left(\frac{1}{\max\{m,n\}}\Bigl( \min_{\pi\in\Pi_m}\sum_{i=1}^m (1-\mathrm{IoU}(x_i, y_{\pi(i)}))^p + c^p(n-m)\Bigr)\right)^{1/p}$$

OSPA is computed per class and averaged over all classes for panoptic segmentation and over tracklets for panoptic tracking. This enables unified closed- and open-world evaluation by penalizing both misclassification and extra/missing predictions.

5. Baseline Performance and Observed Challenges

Baseline experiments establish the substantial difficulty of JRDB-PanoTrack:

Task	Model (ResNet-50, COCO pre-trained)	PQ (%)	Comments
Seg. (closed)	Mask DINO	36.6	Best overall; 33.1 thing, 46.7 stuff
Seg. (open)	ODISE (label-supervised)	10.6	Severe drop on unknown classes
Tracking (closed)	Mask DINO + BoT-SORT	STQ ≈14.5, OSPA ≈0.64	IDF1 ≈0.82
Tracking (open)	Mask DINO + BoT-SORT	STQ ≈9, OSPA worsens	IDF1 ≈0.91 (known), much lower (unknown)

Key failure modes include extreme instance density (up to 245 masks/view, >1,000 panoramic), dramatic object scale variation (floor markers, full-size humans), multi-layered/transparent occlusions (e.g., glass), and a steep long-tail class distribution in open-world protocols—driving pronounced drops in segmentation and tracking quality (Le et al., 2024).

6. Related Methodological Advances and Comparative Benchmarks

Research spawned from JRDB and JRDB-PanoTrack has produced a series of multimodal and panoramic tracking frameworks:

• MMPAT (MultiModality Panoramic multi-object Tracking): Integrates split-detect-merge panoramic detection (Cascade R-CNN with deformable convolutions), 2D-3D LiDAR association, affinity-based data association (appearance, motion, 3D proximity), and constant-velocity Kalman filter smoothing. On JRDB, MMPAT achieves AP 67.88% (object detection) and MOTA 31.7% (multi-object tracking), considerably surpassing previous methods such as Faster R-CNN and DeepSORT (He et al., 2021).
• OmniTrack++: Employs a feedback-driven, end-to-end transformer pipeline designed to address geometric distortion and ID consistency in panoramic scenes via DynamicSSM (feature stabilization), FlexiTrack Instances (trajectory-token feedback), ExpertTrack Memory (MoE-based appearance memory), and adaptive Tracklet Management. On JRDB, OmniTrack++ achieves 25.50 HOTA (E2E), 27.03 HOTA (TBD), and outperforms prior models in IDF1 and OSPA metrics (Luo et al., 1 Nov 2025).
• Human forecasting extensions: Trajectory prediction methods leveraging JRDB-PanoTrack data reveal the dominance of lower-body skeletal cues—achieving a 13% reduction in average displacement error compared to full-body or upper-body features. These findings hold across both highly-distorted equirectangular panoramas and 3D full-body label volumes (Le et al., 9 Feb 2026).

7. Significance and Applications

JRDB-PanoTrack constitutes a reference problem for the development of open-world, multi-modal, panoramic perception and tracking approaches suitable for social robotics, autonomous navigation, and environmental interaction in human-dense settings. The dataset drives methodological advances by combining:

• Dense, synchronized panoramic and 3D data,
• Extremely challenging tracking densities,
• Explicit open-world splits for evaluating rare/unknown class generalization,
• Unified protocols enabling fair, comprehensive benchmarking of segmentation, tracking, and human trajectory forecasting from a robot-centric viewpoint.

As such, JRDB-PanoTrack stands as a central testbed and challenge for next-generation robotic perception systems (Le et al., 2024).