BATON: A Multimodal Benchmark for Bidirectional Automation Transition Observation in Naturalistic Driving

Abstract: Existing driving automation (DA) systems on production vehicles rely on human drivers to decide when to engage DA while requiring them to remain continuously attentive and ready to intervene. This design demands substantial situational judgment and imposes significant cognitive load, leading to steep learning curves, suboptimal user experience, and safety risks from both over-reliance and delayed takeover. Predicting when drivers hand over control to DA and when they take it back is therefore critical for designing proactive, context-aware HMI, yet existing datasets rarely capture the multimodal context, including road scene, driver state, vehicle dynamics, and route environment. To fill this gap, we introduce BATON, a large-scale naturalistic dataset capturing real-world DA usage across 127 drivers, and 136.6 hours of driving. The dataset synchronizes front-view video, in-cabin video, decoded CAN bus signals, radar-based lead-vehicle interaction, and GPS-derived route context, forming a closed-loop multimodal record around each control transition. We define three benchmark tasks: driving action understanding, handover prediction, and takeover prediction, and evaluate baselines spanning sequence models, classical classifiers, and zero-shot VLMs. Results show that visual input alone is insufficient for reliable transition prediction: front-view video captures road context but not driver state, while in-cabin video reflects driver readiness but not the external scene. Incorporating CAN and route-context signals substantially improves performance over video-only settings, indicating strong complementarity across modalities. We further find takeover events develop more gradually and benefit from longer prediction horizons, whereas handover events depend more on immediate contextual cues, revealing an asymmetry with direct implications for HMI design in assisted driving systems.

• The paper introduces BATON, which offers a comprehensive, multimodal, and naturalistic dataset for analyzing bidirectional driver–automation transitions.
• It leverages synchronized front-view and in-cabin videos, CAN signals, radar, and GPS-derived features to capture both driver state and external road context.
• Empirical results demonstrate that multimodal fusion and temporal modeling significantly enhance predictions for driving actions, handovers, and takeovers.

BATON: A Multimodal Benchmark for Bidirectional Automation Transition Observation in Naturalistic Driving

Motivation and Context

The design of driver–automation interfaces in advanced driving automation (DA) systems fundamentally relies on the ability to predict and interpret control transitions between human drivers and automation. These transitions occur as drivers decide when to engage or disengage automation, often demanding complex situational judgment under substantial cognitive load. Existing public datasets are inadequate for this modeling challenge, often lacking synchronized multimodal context or concentrating on unidirectional transitions, simulator environments, or limited real-world diversity.

BATON, introduced in "BATON: A Multimodal Benchmark for Bidirectional Automation Transition Observation in Naturalistic Driving" (2604.07263), addresses this gap by offering a large-scale, naturalistic driving dataset structured specifically for bidirectional control transition study, supported by rigorous multimodal synchronization and comprehensive coverage of both internal and external driving conditions.

Figure 1: Data collection setup with comma device for synchronized front-view and in-cabin video streams, CAN signals, and GPS-derived route context.

Dataset Structure and Modalities

The BATON dataset comprises 136.6 hours of driving, spanning 380 routes, 127 drivers, and 84 car models, resulting in 2,892 annotated bidirectional control-transition events. The data collection utilizes comma devices for capturing synchronized multimodal time-series, including:

• Front-view video: 20 fps, capturing road scene, traffic, lane structure.
• In-cabin video: 20 fps, focused on driver pose, gaze, and readiness.
• CAN-decoded vehicle dynamics: 100 Hz, documenting speed, steering, pedal inputs, DA engagement state.
• Radar-based lead interaction: 20 Hz, tracking lead vehicle distance/speed.
• GPS/route context: Transformed to semantic spatial features (road type, speed limit, lane count), excluding raw coordinates to ensure privacy.

This multimodal setup supports context-rich modeling of both the external environment and the internal driver state, facilitating closed-loop analysis around control transitions.

Figure 2: Overview of BATON, showing global route distribution, driving time by driver, and composition statistics.

Benchmark Tasks

BATON defines three benchmark tasks, with all samples extracted as synchronized 5s observations and evaluated under cross-driver splits for generalization:

1. Driving Action Understanding: Classification of seven coarse actions (Cruising, Accelerating, Braking, Turning, Lane Change, Stopped, Car Following) using rule-based labels. The task comprises ~979k samples, reflecting everyday distributions with rare lane changes.
2. Handover Prediction: Given a 5s multimodal observation in manual driving, predict whether DA will be engaged within a future horizon (1/3/5s). Positive samples are pre-handover; negatives are stable manual-driving intervals. AUPRC and AUROC are primary metrics.
3. Takeover Prediction: Analogous to handover, but prediction is DA→human transition (driver takes control) in DA-active segments.

   Figure 3: Task distribution across BATON—driving action class frequencies, handover, and takeover positive/negative sample splits.

Multimodal Modeling and Baseline Results

Empirical evaluation utilizes sequence models (GRU, TCN), classical baselines (XGBoost, Logistic Regression), and zero-shot vision–LLMs (Gemini 2.0 Flash, GPT-4o), with video encoded via EfficientNet-B0 (PCA-reduced) and structured signals resampled at 50 Hz. Key findings include:

• Single modality limitations: Visual input (front or in-cabin) underperforms, particularly for transition prediction tasks. Front-view video is strong on external context; in-cabin is informative for driver state, but neither suffices alone for transitions.
• Multimodal fusion gains: Incorporating structured signals (vehicle, route, radar) elevates performance: Task 1 action understanding reaches 0.910 Macro-F1, and transition prediction AUPRC increases to ≥0.463 (handover) and ≥0.468 (takeover).
• Temporal context: Sequence models utilizing 5s histories substantially outperform single-frame baselines, especially for handover events (AUPRC drops from 0.631 to 0.449 for XGBoost when removing temporal context).
• Model comparisons: Tree-based XGBoost outperforms neural sequence models, suggesting temporal fusion architectures remain an open opportunity.

  Figure 4: Representative multimodal context for handover/takeover events—aligned cabin views, forward views, route maps, and synchronized radar, driver, and vehicle signals.

Asymmetric Transition Dynamics

Analysis reveals asymmetry between handover (manual→DA) and takeover (DA→manual) transitions:

• Takeover events: Develop more gradually, benefiting from longer prediction horizons and extended temporal context. Longer horizons improve both AUROC and AUPRC.
• Handover events: Depend more heavily on immediate contextual cues; predictive accuracy decreases as horizon increases, emphasizing relevance of short-term state indicators.

These emergent temporal patterns have direct HMI design implications: automation UIs may require anticipatory support for gradual takeovers but highly responsive signaling for handovers.

Practical Implications and Limitations

BATON enables robust algorithmic benchmarking for driver–automation interaction, with ramifications for both adaptive HMI design and proactive safety intervention. Multimodal fusion and temporal reasoning are indispensable, as shown by the empirical ablation results. However, the dataset lacks surround-view imagery for full situational awareness, exhibits some driver duration imbalance, and current baseline models do not exploit advanced temporal fusion or personalization, pointing toward future directions in model development and dataset augmentation.

Theoretical Implications and Future Trajectories

BATON provides the infrastructure to address fundamental questions in human–automation transitions: what contextual signals enable reliable prediction; how driver state interacts with environmental cues; and what temporal dynamics govern engagement/disengagement. As multimodal data and personalized modeling architectures mature, theoretical advances in modeling control loops, optimal fusion, and real-time anticipation will be enabled. Extension with richer scene context, larger driver cohorts, and integration of explainable AI for transparent intervention will further catalyze progress in intelligent driving assistance and HCI research.

Conclusion

BATON establishes a comprehensive, multimodal benchmark for bidirectional driver-automation control transitions in real-world conditions. Empirical results confirm the necessity of multimodal context and temporal modeling, while dataset design and task structure provide a foundation for future research into adaptive, behavior-aware automation interfaces. The asymmetry between handover and takeover transitions underscores the nuanced requirements for proactive HMI design, and the release protocol prioritizes privacy and practical reproducibility. BATON is positioned to accelerate both the practical deployment and theoretical understanding of intelligent assisted driving systems.