In-Cabin Driving Safety

• In-cabin driving behavior safety is defined as the integration of multimodal sensors (e.g., physiological monitors, dual cameras, CAN-Bus data) with AI inference to detect driver distraction, drowsiness, and risky maneuvers.
• Advanced sensor fusion and feature engineering synchronize diverse data streams, enabling robust real-time driver-state assessments and precise risk mapping.
• Emerging methodologies like deep spatio-temporal models, graph convolutional LSTM, and vision-language systems have demonstrated high accuracy in predicting unsafe behaviors and guiding timely interventions.

In-cabin driving behavior safety encompasses the detection, assessment, and mitigation of behaviors, physiological states, and environmental conditions inside a vehicle cabin that elevate crash risk or degrade driving performance. This domain integrates physiological monitoring, computer vision, AI/ML inference, sensor fusion, and domain-adapted human factors frameworks to deliver situational awareness, driver-state estimation, real-time intervention, and ultimately, crash prevention. Contemporary approaches leverage multimodal sensing—including vision, radar, wearable, and vehicular data—combined with hybrid AI models to address both acute (distraction, drowsiness) and chronic (habitual risky operation, cognitive-affective degradation) threats to occupant and road safety.

1. Sensing Modalities for In-Cabin Safety

Modern in-cabin safety frameworks deploy a heterogeneous sensor suite, organized as follows:

• Physiological Sensing: Radar units measure heart rate (HR) and respiration rate (RR), while wearables (e.g., GARMIN series) capture HR, RR, and heart-rate variability (HRV). Some wearables also run proprietary drowsiness predictors (Sini et al., 2023).
• Vision Systems: Dual-camera configurations monitor eye blinking (EB), blink rate, eye-gaze angles (EGA), PERCLOS (eyelid closure), and head pose (nods, pitch/roll/yaw) (Sini et al., 2023). Infrared and depth imaging provide robustness under low-light and occlusion conditions (Katrolia et al., 2021).
• Environmental and Cabin Sensing: CO₂, temperature, humidity, and ambient light measurements contextualize driver state and sensory effectiveness (Sini et al., 2023). Audio streams are utilized for both affective analysis and context recognition (Takato et al., 3 Aug 2024).
• Vehicular and Kinematic Data: CAN-Bus signals supply speed, acceleration (longitudinal/lateral), throttle/brake pressure, steering angle, and derived surrogates such as 2D-TTC (time-to-collision) and SDLP (standard deviation of lateral position) (Adhikari, 2023, Dong et al., 2023).
• Edge and Cloud Integration: Processors (e.g., on-vehicle SoC, Raspberry Pi), data brokers (MQTT), and persistence layers (MongoDB, S3, cloud DB) enable both real-time and offline fusion (Sini et al., 2023, Khosravinia et al., 2023, Huang et al., 2022).

This multimodal sensor architecture enables complementary feature capture: physiological signals yield early indicators of sleep onset, while vision systems excel at detecting brief distractions missed by inertial/biometric cues (Sini et al., 2023).

2. Feature Engineering and Data Preprocessing

Multi-channel sensor data necessitates rigorous temporal alignment, denoising, and feature construction:

• Synchronization: Timestamps from radar, wearables, cameras, and CAN-Bus/IMU streams are typically aligned via common system clocks, with compensation for variable sampling rates and drift (Sini et al., 2023).
• Derived Features: Key indicators include:
  • HRV (e.g., SDNN, RMSSD)
  • PERCLOS: $$\text{PERCLOS} = (\text{time eyelid} > 70\% \text{ closed})/(\text{window duration})$$
  • Eye Aspect Ratio (EAR), Head-pose angles (from facial landmarks)
  • Surrogate safety indicators: acceleration outliers, close-distance lane changes, 2D-TTC with $$\text{TTC}_{2D} = -(\Delta v \cdot \mathbf{D})/\|\Delta v\|^2$$ if $(\Delta v \cdot \mathbf{D}) > 0$ (Dong et al., 2023)
  • Emotion recognition: arousal–valence space from facial action units (Huang et al., 2022)
• Windowing and Smoothing: Sliding windows (typ. 1 s at 10–30 Hz) permit both real-time (on-device) and batched (offline) calculation; online smoothing (e.g., exponential moving average) is used for robust alerting (Tavakoli et al., 2021, Dong et al., 2023).
• Normalization: Features are standardized (zero-mean, unit-variance) within each channel before fusion or ML inference (Dong et al., 2023).
• Annotation and Ground Truth: Physiological ground truth is established via polysomnography, self-rating scales (ESS, KSS), or video-based manual annotation of activities and events (Sini et al., 2023, Tavakoli et al., 2021).

3. Inference, Classification, and Sensor Fusion Methodologies

In-cabin safety systems employ both classical and deep learning models, with several fusion paradigms:

• Early and Mid-Level Fusion: Concatenation of feature vectors from different modalities, optionally weighted by confidence. Example: $$F_{fused} = \alpha F_{physio} + (1-\alpha)F_{vision}$$ (Sini et al., 2023).
• Decision-Level Fusion: Aggregation of streamwise classifiers (e.g., raising a drowsiness alarm if either physiological or visual pipeline flags an event) (Sini et al., 2023).
• Hierarchical Extreme Learning Machines: HELM-based pipelines stack single-hidden-layer autoencoders for semi-supervised anomaly detection, using both raw kinematic/positional data and engineered safety indicators (Dong et al., 2023).
• Deep Spatio-Temporal Models: CEMFormer leverages a spatial-temporal transformer with episodic memory to fuse in-cabin and external video streams, utilizing context-consistency loss $\ell^{cc}$ to penalize infeasible maneuvers, thus increasing early anticipation accuracy (F1 score up to 0.87) (Ma et al., 2023).
• Graph Convolutional LSTM (GConvLSTM): CAN-Bus time series are modeled as sensor graphs, with graph convolutions capturing inter-signal dependencies and temporal evolution, yielding up to 98.7% accuracy for safe/unsafe classification on real-world data (Khosravinia et al., 2023).
• Vision-Language Modeling and Reasoning Systems: Video-LLaMA-style architectures process synchronized road- and driver-facing cameras, plus audio, to support both event recognition (AR up to 67.7%) and free-form, human-readability coaching via LLM outputs. Only Q-Former components are adapted during fine-tuning for model efficiency (Takato et al., 3 Aug 2024).
• Fuzzy Logic and Ensemble Approaches: Risk scores are determined by rules such as “IF PERCLOS is High AND SDLP is High THEN Drowsiness Risk is Very High,” often integrated with statistical models (LightGBM, boosting) for conflict prediction and mapping onto real-time risk heatmaps (Adhikari, 2023, Huang et al., 2022).

4. Data Resources, Benchmarks, and Performance Metrics

Advancing in-cabin safety depends on curated, annotated data and consensus metrics:

• Benchmark Datasets: Key resources include Brain4Cars (synchronized in/out-cabin video for intention prediction (Rong et al., 2020, Ma et al., 2023)), TICaM (multimodal ToF/RGB/IR, with activity, 2D/3D object/instance segmentation (Katrolia et al., 2021)), and DSBench (98K scene-safety QA pairs, with 3K in-cabin evaluation scenes, including 15 subcategories of risk (Meng et al., 18 Nov 2025)).
• Metrication:
  • Event Recognition Accuracy Rate: $\text{AR} = \#\text{Correct}/\#\text{Total}$ (Takato et al., 3 Aug 2024)
  • F1-score, ROC-AUC, precision, recall
  • LLM-based “virtual safety evaluation” (rating 0–100 on judgment, completeness, fluency) for VLMs (Meng et al., 18 Nov 2025)
  • For risk mapping: segment-level accuracy/F1, AUC on short-horizon conflict prediction (Huang et al., 2022)
• Baseline and Advanced Model Results:
  • HELM-based anomaly detection: 99.58% accuracy, 0.9913 F1; addition of 2D-TTC features gives a 3.4 pp jump in accuracy (Dong et al., 2023).
  • GConvLSTM for unsafe driving detection: 98.7% accuracy with ROC-AUC ~0.99 (Khosravinia et al., 2023).
  • CEMFormer for intention anticipation: 0.8709 F1 for dual-view spatial-temporal fusion, with meaningful ablations for episodic memory/context losses (Ma et al., 2023).
  • Vision-Language coaching pipeline: AR 67.7%, BLEU up to 8.1, BERTScore F1 0.899; real-time coaching reduced harsh braking by 15% over two weeks (Takato et al., 3 Aug 2024).
  • DSBench: baseline VLMs display low performance on in-cabin safety tasks, especially on cockpit environment (<30/100), improved to 80.1/100 after fine-tuning (Meng et al., 18 Nov 2025).

5. Practical Applications and System Architectures

Pragmatic implementation of in-cabin safety concepts is diverse:

• Edge and Real-Time Deployment: On-device execution is achieved with optimized model layers (Q-Former, heads) or efficient architectures (HELM, GConvLSTM) on embedded platforms (e.g., Raspberry Pi, vehicle SoC) (Khosravinia et al., 2023, Sini et al., 2023, Takato et al., 3 Aug 2024).
• Driver Coaching and Feedback: Live dashboards aggregate risk, coaching cues, and automated feedback, both for drivers (audio/visual warnings) and fleet managers (summary statistics) (Takato et al., 3 Aug 2024, Khosravinia et al., 2023).
• Alerting and Intervention: Immediate auditory/visual cues triggered by critical events (drowsiness, inattention, unsafe maneuver onset), often DDAW-compliant per regulatory standards (Sini et al., 2023, Khosravinia et al., 2023).
• Privacy-Preserving and Context-Adaptive Monitoring: Wearable-centric and sensor fusion solutions selectively rely on non-visual data when privacy or environmental (low-light) factors limit camera utility (Tavakoli et al., 2021).
• Risk Heat Mapping and Route Management: Individual and aggregated behavior-risk profiles inform dynamic safety maps for routing, law enforcement deployment, and user-tailored warnings (Huang et al., 2022).

6. Challenges, Limitations, and Open Research Questions

Despite significant progress, several technical and practical barriers persist:

• Sensor Integration: Heterogeneous sampling, time-stamp drift, and inter-device synchronization impair fusion reliability (Sini et al., 2023).
• Occlusion, Lighting, and Missing Values: Face/eye tracking fails with sunglasses or poor lighting; fallback to inertial/physiological sensing is common (Adhikari, 2023, Tavakoli et al., 2021).
• Semantic Gaps: VLM/LLM systems frequently underperform on cockpit-specific cues—seat-belt detection, mirror status, and affective state (Meng et al., 18 Nov 2025).
• Personalization and Domain Adaptation: Inter-driver variability is high; static model thresholds and labels induce drift and reduced sensitivity (Dong et al., 2023, Adhikari, 2023).
• Ethics, Privacy, and Acceptability: Continuous video and biometrics raise user consent, storage, and misuse concerns; wearables and non-visual sensing offer mitigation (Tavakoli et al., 2021).
• Label Scarcity and Generalization: Abnormal/risky events are rare and labor-intensive to annotate; semi-supervised, self-supervised, or anomaly-detection methods are being advanced (Dong et al., 2023, Adhikari, 2023).

A plausible implication is that future in-cabin safety pipelines will require more robust, context-aware fusion, improved reasoning on affective/cockpit cues by VLMs, privacy-preserving computation, and continuous adaptation to dynamic driver populations and conditions.

7. Trends and Future Directions

Anticipated research priorities as identified in the literature include:

• Formalization of End-to-End Multimodal Fusion: Ongoing development of cross-modal and graph-based attention mechanisms to unify vision, physiological, inertial, and context data (Sini et al., 2023, Adhikari, 2023, Rong et al., 2021).
• Fine-Tuned and Safety-Ready VLMs: Large, scenario-diverse benchmarks (e.g., DSBench) and LLM-centered evaluations are raising the bar for in-cabin cognition and instruction-following by vision-LLMs (Meng et al., 18 Nov 2025).
• Continual and Federated Learning: Edge-intelligent and federated approaches, enabling adaptation without centralized storage or retraining, are critical for generalization and privacy compliance (Adhikari, 2023).
• Ultra-Efficient On-Device Inference: Model pruning, quantization, and architectural distillation will drive true real-time safety analytics in cost- and power-constrained automotive environments (Takato et al., 3 Aug 2024).
• Comprehensive Real-World Evaluation: Expansion beyond simulator and curated datasets to large-scale, on-road trials and regulatory-compliant interfaces (e.g., DDAW) (Sini et al., 2023, Huang et al., 2022).
• Integrated Risk Scoring and Handover: Fusing in-cabin behavior prediction with vehicle- and infrastructure-sourced risk to orchestrate cooperative safety interventions and semi-autonomous handovers (Huang et al., 2022, Meng et al., 18 Nov 2025).

In sum, in-cabin driving behavior safety research is rapidly converging on hybrid, multimodal, and human-centric intelligence paradigms, with demonstrated improvements in drowsiness/danger detection, actionable feedback, and driver risk mitigation, but remains challenged by integration, privacy, and robustness under real-world heterogeneity (Sini et al., 2023, Meng et al., 18 Nov 2025, Takato et al., 3 Aug 2024, Ma et al., 2023, Dong et al., 2023, Tavakoli et al., 2021, Adhikari, 2023).