ROAD-Waymo: Action-Aware AV Dataset

• ROAD-Waymo is a large-scale, action-aware autonomous driving dataset featuring 198K video frames and 12.7M multi-label annotations across diverse U.S. urban scenarios.
• It employs a rigorous annotation pipeline with SAT-solver checks and a structured label ontology to ensure semantic integrity and minimize manual error.
• The dataset benchmarks advanced perception tasks—including agent, action, and event detection—and facilitates domain adaptation through the ROAD++ composite.

ROAD-Waymo refers to a large-scale, requirement-compliant, action-aware dataset overlaid on the Waymo Open Dataset (WOD) for developing and benchmarking advanced perception systems in autonomous driving, together with a set of methodologies, annotation tools, and benchmarks for multi-label agent, action, location, and event understanding in complex U.S. urban driving scenarios. By providing an order-of-magnitude expansion over the original ROAD (Oxford, UK) and integrating SAT-solver–checked annotation integrity, ROAD-Waymo establishes the foundation for the cross-domain ROAD++ benchmark and enables rigorous evaluation of holistic scene understanding, domain adaptation, and neuro-symbolic reasoning in AV contexts (Khan et al., 2024).

1. Dataset Composition and Annotation Pipeline

ROAD-Waymo comprises 198,000 front-camera video frames, 54,000 agent tubes, approximately 3.9 million bounding boxes, and 12.7 million agent, action, and location multi-label annotations, drawn from diverse traffic, weather, and urban topologies spanning Phoenix, San Francisco, and additional U.S. cities (Khan et al., 2024). This scale is approximately 1.6× greater in duration, and 7–8× larger in tubes and labels versus the original Oxford-based ROAD, yielding substantially higher geometric, modal, and semantic complexity (mean 22 agents per frame in ROAD-Waymo).

The annotation pipeline guarantees logical consistency using 251 domain-specific requirements expressed as propositional logic constraints, e.g. “a traffic light cannot be red and green simultaneously” and mutual exclusivity between MoveAway and MoveToward actions. Each frame's label set is automatically checked using a SAT-solver (MiniSAT); this permits automated detection and revision of annotation violations. Label propagation ensures agent-type consistency within tubes and updates state labels only when they change, minimizing manual effort. Peer review, automated checks, and iterative correction cycles converge to zero violation remaining across all annotations.

Inheritance of agent tracks and bounding boxes leverages the official Waymo Open Dataset; traffic-light tracks are bootstrapped using YOLOv6 and manually curated.

2. Multidimensional Label Ontology

ROAD-Waymo employs a structured schema with three disjoint label axes:

• Agent types: Pedestrian, Car, Cyclist, Motorbike, Small/Medium/Large Vehicle, Bus, Emergency Vehicle, Traffic Light.
• Actions: Movement and signal states (Move, MoveAway, MoveToward, Braking, Stop, TurnLeft/Right, IndicateLeft/Right, HazardLights, Overtaking, Waiting, CrossingFromLeft/Right, PushObject, Wait2X); full enumeration of traffic-light phase.
• Locations: Lane-partitioned (AVLane, IncomingLane, OutgoingLane, CycleLane, BusLane), pavements (Left/Right), junctions, crossings, bus stops, and parking.

Events are defined as ordered triples (Agent, Action, Location) corresponding to “road events” for fine-grained scene comprehension. The strict logical requirement set enforces semantic integrity across these axes.

3. Benchmarking Tasks and Metrics

ROAD-Waymo closely follows the evaluation paradigm of the original ROAD but augments scale and modality diversity. The principal benchmark tasks include agent (object) detection, action detection, location detection, duplex detection (Agent+Action), event detection (Agent+Action+Location), AV-action segmentation (ego vehicle maneuvers), and cross-domain adaptation (Khan et al., 2024).

Detection proposals and ground-truth boxes are matched using IoU thresholds (δ=0.5 for frame-level and δ=0.2/0.5 for video-tube overlap). Average Precision (AP) is aggregated as mean Average Precision (mAP) over all queries q: $$\mathrm{mAP} = \frac{1}{|\mathcal{Q}|} \sum_{q \in \mathcal{Q}} \mathrm{AP}(q)$$

Baseline detection results (3D-RetinaNet, YOLOv8):

• Agents (YOLOv8): f-mAP ≈ 38.1% (val) / 31.6% (test)
• Event detection (3D-RetinaNet SlowFast-8): f-mAP ≈ 15.3% (agents), ≈14.0% (actions), ≈12.4% (locations), ≈5.3% (events)

Video tube detection at [email protected] is significantly lower; agents achieve ~6.5% (best), events ~4.3% on the test set, revealing the high granularity and difficulty.

Neuro-symbolic regularization by incorporating the logical requirements as differentiable t-norm loss improves test mAP by 3–6 percentage points across classes at zero extra annotation cost, highlighting the value of symbolic compliance.

4. Domain Adaptation and ROAD++ Benchmark

ROAD-Waymo is schema-compatible with the original UK ROAD, enabling the construction of the ROAD++ benchmark by concatenating U.S. and UK data (320,000 frames, ≈17,000 tubes) under a unified ontology. This joint dataset exposes significant cross-geographical domain shifts:

• Agent f-mAP drops from 24.1% (train on Oxford, test on Oxford) to 5.2% (train on Oxford, test on ROAD-Waymo).
• Corresponding drops in the reverse direction (ROAD-Waymo→Oxford) are milder (~15.4%→7.1%).

These large gaps result from structural traffic differences, lane geometry, traffic-light layout, and class prevalence. Application of reverse-gradient unsupervised domain adaptation produces partial improvements but the benchmarks remain closer to cross-domain lower bounds, indicating fundamental challenges in real-to-real adaptation for event and action detection.

5. Integration with Perception and Forecasting Pipelines

ROAD-Waymo provides tightly aligned agent tracks, action/event tubes, and contextual semantic labels inherited and cross-validated from the Waymo Open Dataset (Ettinger et al., 2021). This structure enables:

• Advanced object detection architectures (e.g., Cascade R-CNN, CenterNet, HorizonDet) previously benchmarked on WOD to be fine-tuned with ROAD-Waymo's richer and more challenging label set, incorporating action and location heads (Chen et al., 2020).
• Direct compatibility with joint/recurrent agent prediction pipelines, including LSTM, Transformer, and quantum-classical hybrid predictors (Oerlemans et al., 2024, Singh et al., 21 Nov 2025), for evaluating holistic agent behavior understanding and interactive event forecasting.

The high density and action granularity facilitate rare or high-stakes multi-agent situation mining (cf. risk-based filtering (Puphal et al., 30 Jun 2025)) and calibration of rule-conformant/violating predictions using empirical distributions derived from real U.S. driving (Bogdoll et al., 2022).

6. Limitations and Future Research Opportunities

ROAD-Waymo's annotation quality and semantic consistency surpass previous multi-label AV datasets, but notable limitations remain:

• Event detection mAP remains low; the task complexity associated with dense, occluded, and concurrent action labeling prompts the need for robust multi-frame aggregation and context modeling.
• Tube-level temporal consistency is challenged by abrupt behavior switches and crossing occluded tracks; label-propagation heuristics may under-annotate rapid events.
• Domain adaptation baselines show substantial degradation, reinforcing the research challenge for robust cross-domain, cross-modal models.

Open areas include integration of additional sensory inputs (vectorized LiDAR, multi-camera), expansion to higher-order structured interactions, and incorporating learned risk distributions (e.g., pairwise/second-order risk (Puphal et al., 30 Jun 2025)) into scenario prioritization. Coupling with real-time risk estimation frameworks (e.g., hierarchical Bayesian EVT on TTC metrics (Anis et al., 2024)) and 3D geometric reasoning (e.g., planar parallax (Yuan et al., 2021)) remains a powerful avenue for multimodal, action-aware perception advances.

7. Significance and Impact

ROAD-Waymo represents the most extensive, requirement-compliant, agent-and-action-aware annotation of U.S. real-world AV perception data. By unifying detection, action, and context understanding at scale, and enabling logically verified annotations, it provides an unprecedented testbed for holistic autonomous scene interpretation, neuro-symbolic learning, and domain adaptation. Coupled with the cross-country ROAD++ composite, it constitutes a fundamental benchmark for future research on cross-geographical, cross-modal, and cross-sensor generalization in autonomous driving (Khan et al., 2024).