--- title: Waymo End-to-End Driving Dataset url: https://www.emergentmind.com/topics/waymo-end-to-end-driving-dataset type: topic --- # Waymo End-to-End Driving Dataset The Waymo End-to-End Driving Dataset (WOD-E2E) is a curated benchmark targeting vision-based end-to-end (E2E) autonomous driving systems, with particular emphasis on rare, safety-critical long-tail scenarios. Comprising 4,021 segments totaling approximately 12 hours, each selected from a multi-million-mile corpus, WOD-E2E stresses the generalization and robustness capabilities of E2E agents by focusing on events with observed frequencies below 0.03% in daily driving. Each segment includes synchronized 360-degree imagery from eight cameras, high-level routing commands, and detailed ego state histories. A novel open-loop evaluation metric—Rater Feedback Score (RFS)—is introduced to more effectively assess multi-modal future trajectory prediction, leveraging expert rater preferences for validation. ## 1. Dataset Composition and Scenario Categorization WOD-E2E consists of 4,021 driving segments, each of 20 s duration, amounting to approximately 12 hours of challenging driving data. Segments are stratified into 11 “long-tail clusters,” which encapsulate infrequent but operationally critical scenarios: | Cluster | Example Subtypes | |------------------|---------------------------------------------| | Construction | Road closures, surfacing, flaggers | | Intersection | Unprotected turns, traffic violations | | Pedestrians | Occlusions, erratic movement | | Cyclists | Loss of control, group interactions | | Multi-Lane Maneuvers | Freeway merges, overtakes | | Single-Lane Maneuvers | Narrow-road overtakes, open-doors | | Cut-ins | Aggressive adjacent/oncoming lane entries | | Foreign Object Debris | Animals, furniture, hazardous roads | | Special Vehicles | Emergency pull-over/blocking | | Spotlight | MLLM-driven mining for novel objects | | Others | Unclassified critical events | The train/val/test splits are as follows: - Training: 2,037 segments (full 5 s future trajectories visible) - Validation: 479 segments (full 5 s futures + rater labels) - Test (held out): 1,505 segments (12 s input, 8 s future hidden, rater labels withheld for challenge) [2510.26125]. ## 2. Sensor Modalities and Data Structure WOD-E2E provides synchronized multi-modal data for each segment: - **360° Camera Array:** Eight JPEG cameras (front, front-left, front-right, side-left, side-right, rear, rear-left, rear-right) with a frame rate of 10 Hz. Each image is stored at manufacturer resolution (~1920×1280), typically down-sampled to 768×768 for modeling. Horizontal field-of-view per camera spans 70°–90°, covering the complete 360° perimeter. - **Calibration:** Intrinsics and extrinsics are supplied for accurate 3D projection. - **Ego State History:** 4 s of past trajectory at 4 Hz (16 points of x, y in vehicle-centered coordinates), with aligned velocity and acceleration profiles. - **Routing Commands:** Enumerated as {GO_STRAIGHT, GO_LEFT, GO_RIGHT}, derived from future log headings. - **Data Organization:** Segments are provided in a standardized directory layout, supporting efficient loading per modality. Train/val inlude ground-truth futures; validation includes rater preference labels; test omits future and rater labels [2510.26125]. ## 3. Rater Feedback Score (RFS) Metric The RFS metric supersedes conventional average (ADE) and final displacement error (FDE) metrics, specifically optimizing for the multi-modal and preference-weighted nature of rare driving events. ### Formalism Let $i \in \{1,2,3\}$ index rater-tagged trajectories, each assigned a score $s_i \in [0,10]$. At horizons $t \in \{3, 5\}$ seconds: - $\Delta_{\text{lat}i}(t)$, $\Delta_{\text{lng}i}(t)$: model-prediction lateral/longitudinal errors to rater $i$'s trajectory. - $\tau_{\text{lat}}(t)$, $\tau_{\text{lng}}(t)$: trust-region thresholds scaled by initial speed $v$. Per-trajectory score: $$\text{score}_i(t) = s_i \cdot 0.1^{\max\Bigl(\max\{\tfrac{\Delta_{\text{lng}i}(t)}{\tau_{\text{lng}}(t)},\, \tfrac{\Delta_{\text{lat}i}(t)}{\tau_{\text{lat}}(t)}\} -1,\,0\Bigr)}$$ Aggregate best rater score: $$\text{Score}(t) = \max_{i=1..3} \text{score}_i(t)$$ Overall RFS (floored at 4 for large errors): $$ \displaystyle \text{RFS} = \max\Bigl(4,\;\Bigl\lfloor\frac{\text{Score}(3)+\text{Score}(5)}{2}\Bigr\rfloor\Bigr) $$ Trust-region thresholds scale by $v$ as: $$ \text{scale}(v) = \begin{cases} 0.5, & v < 1.4 \\ 0.5 + 0.5\frac{v-1.4}{11-1.4}, & 1.4 \leq v < 11 \\ 1, & v \geq 11 \end{cases} $$ Base thresholds: - At $t=3$s: $(\bar{\tau}_{\text{lat}}=1.0 \text{m},\, \bar{\tau}_{\text{lng}}=4.0 \text{m})$ - At $t=5$s: $(\bar{\tau}_{\text{lat}}=1.8 \text{m},\, \bar{\tau}_{\text{lng}}=7.2 \text{m})$. The metric rewards agreement with optimal human-annotated plans and penalizes deviations outside the trust region, reflecting practical safety and legal requirements [2510.26125]. ## 4. Annotation Protocol and Quality Control Critical moments are selected by expert labelers identifying the earliest significant event. At each key frame, up to 64 candidate 5 s trajectories are auto-generated using a motion-forecasting model (e.g., Wayformer) and bucketed by lateral actions, velocity, and braking. Labelers select three representative trajectories: optimal (Rank 1), plausible alternative (Rank 2), and sub-optimal (Rank 3). Each trajectory receives a score on five dimensions: Safety, Legality, Reaction Time, Braking Necessity, and Efficiency; major infractions incur a –2 penalty, minor –1, with Rank 1 trajectories guaranteed at least 6 points. Quality is controlled via double-labeling (10% of segments; Cohen’s κ > 0.75) and rationales for each decision to maintain consistency and reproducibility. Initial filtering reduces the mined long-tail set to 0.03% from 0.1% of total miles [2510.26125]. ## 5. Baseline Model Performance and Evaluation Insights The dataset supports rigorous comparison of E2E architectures. Baselines include: | Model | RFS | ADE | Parameters | |----------------------|------------|-----------|--------------| | Swin-Trajectory | 7.543 | 2.814 | 36M | | DiffusionLTF | 7.717 | 2.977 | 60M | | UniPlan | 7.779 | 2.986 | 60M | | Gemini1 Nano | 7.528 | 3.018 | -- | | AutoVLA | 7.556 | 2.958 | -- | | HMVLM | 7.736 | 3.071 | -- | | Poutine | 7.986 | 2.741 | -- | Key observations include weak correlation between ADE and RFS—low displacement error does not guarantee human-preferred behavior in complex scenarios. Reinforcement learning (RL) targeting RFS leads to superior performance over ADE-optimized RL, with fine-tuning on WOD-E2E yielding moderate RFS gains (≈+0.08), and fusion of multi-camera inputs and test-time sampling further improving results. Qualitative exemplars demonstrate that trust-region adherence yields perfect RFS, while major errors invoke the minimum score penalty [2510.26125]. ## 6. Access Patterns, Best Practices, and Future Research Directions Data and code are accessible at https://waymo.com/open/data/e2e, with Python/TensorFlow API examples provided. For leaderboard participation, predicted 5 s trajectories are uploaded in prescribed JSON format. Recommended practices include: - Pre-training on nominal datasets, then fine-tuning on WOD-E2E to maximize exposure to long-tail distributions. - Spatial fusion of all eight camera modalities for occlusion-aware context. - Multi-sample test-time approaches to capture prediction multi-modality. - Aligning RL reward with RFS rather than ADE for safety-critical outcomes. Potential extensions involve integration with simulators (CARLA, NavSim), addition of map-based and LiDAR channels, leveraging large vision-language models for object-centric reasoning, conducting worst-case safety analyses, and curriculum learning for incremental exposure to rarer events [2510.26125]. A plausible implication is that multi-modal evaluation frameworks and rater-driven reference standards, as established in WOD-E2E, will catalyze new directions in reliable, generalizable E2E agent design. Source: https://www.emergentmind.com/topics/waymo-end-to-end-driving-dataset