Spatiotemporal Attention for Unfit Driving Detection

• The paper introduces transformer architectures with spatiotemporal attention that integrate spatial features and temporal cues to robustly detect unfit driving conditions.
• It employs hierarchical video transformers, pose-fusion networks, and dual-level sensor attention to capture both short-term events and long-horizon trends in driver behavior.
• The study discusses challenges such as data variability, causal modeling, and real-time deployment, while proposing future directions like dynamic token selection and multimodal integration.

Spatiotemporal attention for unfit driving detection refers to the family of machine learning and deep learning techniques that explicitly model both spatial cues (such as facial expression, hand position, or vehicle control states) and temporal dynamics (such as blink rate, gesture sequence, or pedal/steering fluctuations) to robustly infer when a driver is impaired—whether by drowsiness, distraction, medical symptomatology, or other adverse states. The field has rapidly evolved from fixed-window feature engineering to multi-head, hierarchical transformer architectures capable of parsing complex and long-horizon dependencies among visual and sensor data streams. A central methodological axis is the use of attention mechanisms—at local or global spatiotemporal scales—to prioritize salient cues and capture the subtle, transient patterns that often differentiate unfit from normal driving behavior.

1. Problem Definition and Spatiotemporal Cues

Unfit driving encompasses a spectrum of functional (drowsiness, distraction) and pathological (e.g., Parkinsonism, intoxication) impairments. Key behavioral markers span both the spatial domain—such as eyelid closure, yawn, hand position, gaze angle, control inputs—and the temporal domain—such as blink rate, micro-sleeps, gesture rhythm, or rapid steering corrections. An accurate detector must:

• Localize features of interest (eyes, hands, wheel, pedals) within each frame,
• Model the temporal evolution of these cues, including both short events (e.g., a yawn or tremor burst) and long-term trends (growing fatigue, bradykinetic movement),
• Mitigate the confounds introduced by scene variability (illumination, occlusion), head pose, camera perspective, and inter-driver variability.

Spatiotemporal attention mechanisms directly address these requirements by enabling a model to dynamically reweight inputs and intermediate features based on learned saliency across both space and time (Lakhani, 2022, Chang et al., 6 Mar 2025).

2. Model Architectures Incorporating Spatiotemporal Attention

A range of architectures have been developed to operationalize spatiotemporal attention in the driving context:

2.1 Hierarchical Video Transformers

The Video Swin Transformer (VST) establishes a canonical design wherein input video is partitioned into non-overlapping 3D windows (tubelets) and multi-head self-attention (MSA) is computed within each window. Cross-window communication is achieved via shifted windowing across layers, and input “tubelets” are embedded using learnable 3D convolutions. Attention is computed as:

$$\text{head}(X) = \text{softmax}\left(\frac{Q K^\top}{\sqrt{d_k}} + B^r\right) V$$

where $X$ is the window token sequence and $B^r$ encodes relative position (Lakhani, 2022). Hierarchical pooling and patch merging yield a global sequence representation fed to a classifier.

2.2 Multi-Stream/Pose-Fusion Transformers

Transformer-based fusion of 2D pose and spatio-temporal appearance features enables granularity beyond raw pixel or 3D-CNN inputs. For instance, SlowFast backbones extract dense visual dynamics, while pose features (joint locations, velocities, distances) are projected to high-dimensional embeddings and added as dynamic positional encodings:

$Z_{ti}^{(0)} = f_{ti}^{st} + E_{ti}^{pose}$

Self-attention across these representations allows the model to focus on discriminative moments of distracted or unfit driving, with post-processing for temporal localization and false positive suppression (Akdag et al., 11 Mar 2024).

2.3 Causal and Permutation-Invariant STP Frameworks

Spatial-Temporal Perception (STP) networks use dual-branch backbones—temporal convolutions for motion patterns and GCNs for spatial relationships (distances between keypoints)—with attention-based fusion. Crucially, STP maximizes joint likelihood over all permutations of the order in which features are combined, removing inductive bias toward either spatial or temporal stream dominance:

$$\mathcal{L}_{\text{joint}} = - \frac{1}{2} \left[ \log p(F_s) + \log p(F_t | F_s) + \log p(F_t) + \log p(F_s | F_t) \right]$$

A causal-aware module restricts attention to permissible past frames ($M_{ij}$), avoiding “future-leakage” and emphasizing genuine temporally causal detection (Chang et al., 6 Mar 2025).

2.4 Channel and Frame-Wise Attention for Sensor Data

For detection of Parkinsonian (and other) anomalies, SAFE-D demonstrates a dual-level attention: channel attention over multiple control streams (steering, throttle, brake) and frame attention within a local window. The final per-window representation is:

$z_t = F_{\mathrm{Glo}}(t) + F_{\mathrm{Loc}}(t)$

where $F_{\mathrm{Glo}}$ aggregates across channels and $F_{\mathrm{Loc}}$ weights future/previous frames by learned softmax scores (Cao et al., 20 Oct 2025).

2.5 Coarse and Sequence-Based Temporal Attention

Coarse Temporal Attention Networks (CTA-Net) partition input sequences into semantically meaningful segments (e.g., before/during/after an action), learning separate branches for each and employing both spatial and temporal self-attention for contextualization. Alternatively, sequence-based temporal attention employs Mahalanobis-style measures in the correlation matrix for transformer attention, amplifying both differences and similarities between temporally offset frames, critical for detecting anomalies such as microsleeps or erratic motor events (Korban et al., 13 May 2024, Wharton et al., 2021).

3. Training Protocols and Datasets

Spatiotemporal attention models for unfit driving are typically trained using the following protocol dimensions:

• Input modalities: RGB video, pose keypoints, sensor data (steering angle, pedal force), or fused multi-stream.
• Preprocessing: Cropping, normalization, pose estimation (via HRNet, OpenPose), optical flow extraction, and sliding window segmentation are common.
• Datasets:
  • NTHU-DDD (drowsiness)—80/20 split, 30-frame clips (Lakhani, 2022)
  • DMD (distraction)—nine classes, 40+ hours, sequence length 30–64 frames (Akdag et al., 11 Mar 2024)
  • DriveAct, SynDD2—naturalistic RGB driver activity (Chang et al., 6 Mar 2025, Wharton et al., 2021)
  • Simulator-based (CARLA, Logitech G29) for pathological behaviors (PD-like) (Cao et al., 20 Oct 2025)
• Training: AdamW optimizer, learning rate schedules with warmup/cosine decay, batch sizes $16$–$32$, dropout (often $0.5$), focal or cross-entropy loss.
• Label Smoothing and Data Augmentation: Density-guided label smoothing and augmentations (color jitter, flips, occlusions) mitigate overfitting under limited training data (Akdag et al., 11 Mar 2024).

4. Quantitative Performance and Comparative Analysis

Performance is highly variable across application and data regime:

Model	Task/Dataset	Reported Accuracy/Score	Baseline/Comparison
Video Swin Transformer	Distraction (DMD)	97.5%	3D-CNN: 97.2% (Lakhani, 2022)
Video Swin Transformer	Drowsiness (NTHU-DDD)	44% (overfit)	3D-CNN: 75.4%
STP	DriveAct (fine, single-view)	Mean-1: 63.82%, Top-1: 78.32%	UniFormerV2: 61.79%/76.71% (Chang et al., 6 Mar 2025)
STP	SynDD2 (single-right)	AO-Score: 0.7823	Best prior: 0.7459
SAFE-D	Parkinson anomaly (CARLA)	96.8%	CNN-Transf: 94%; SVM: 65%
CTA-Net	Distracted Driver V2 (10 classes)	92.5%	Frame-wise: 90.1% (Wharton et al., 2021)
Pose-fusion Transformer	AI City Challenge A2	Overlap score: 0.5079	SlowFast: 0.3459, +pose: 0.4274

The principal determinants of accuracy are dataset scale/diversity, task framing (number of classes, granularity), architecture expressivity, and regularization. Models with insufficient temporal context or inadequate attention mechanisms systematically underperform on fine-grained or long-horizon markers (e.g., drowsiness) (Lakhani, 2022).

5. Interpretability, Failure Modes, and Analysis

Attention heatmaps and score trajectories provide non-trivial interpretability:

• Channel-wise attention identifies which sensor/feature is most discriminative during a temporal window (e.g., highlighting steering tremor for Parkinsonian detection (Cao et al., 20 Oct 2025)).
• Temporal attention pinpoints the frames most indicative of an event—such as the precise onset of eye closure in drowsiness, or the instant a secondary object (phone, cup) is manipulated (Akdag et al., 11 Mar 2024).
• Overfitting and poor generalization (notably on two-class drowsiness) can be linked to limited data, lack of intra-class variation, and failure to adequately regularize high-capacity transformer modules (Lakhani, 2022). A corollary is the critical importance of training set scale and diversity for robust unfit driving detection.

6. Extensions, Limitations, and Future Directions

Spatiotemporal attention remains an active domain with several frontiers:

• Dynamic Token Selection: Methods such as TokenLearner recommend dynamically sampling the most salient spatiotemporal regions for encoding, potentially increasing both efficiency and performance (Lakhani, 2022).
• Causal and Multimodal Integration: Extensions to physiological proxies (e.g., heart rate from PPG) and environmental signals (road context, lighting) are plausible, as is more nuanced causal masking to align with specific temporal dependencies (e.g., order of distraction, onset of micro-sleep) (Chang et al., 6 Mar 2025).
• Unified Multimodal Architectures: Fusing video, pose, and sensor streams—either early (motion-memory) or late (joint attention fusion)—enables more comprehensive detection frameworks (Korban et al., 13 May 2024).
• Real-Time and Embedded Deployment: Model pruning, quantization, and adaptive input techniques are necessary for deployment on resource-constrained hardware (Jetson Nano, mobile SoC) for in-vehicle use (Lakhani, 2022).
• Interpretability and Continuous State Modeling: Frame-level and aggregated attention visualizations offer routes for explainability, while continuous scale driver state (e.g., Karolinska Sleepiness Scale) labels support more physiologically orientated risk scoring (Chang et al., 6 Mar 2025).

This suggests that significant improvements in unfit driving detection are likely to originate from advances in both model architecture (flexible attention, multimodal fusion) and dataset construction (scale, richness, naturalistic diversity).

7. Representative Algorithms and Pseudocode (Selected Models)

The core computational routines for spatiotemporal attention in representative frameworks are as follows:

Self-attention in Video Swin Transformer (per 3D window):

$Q = X W_Q,\quad K = X W_K,\quad V = X W_V$

$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{Q K^T}{\sqrt{d_k}} + B^r\right) V$$

$$\text{MSA}(X) = \text{Concat}(\text{head}_1, ..., \text{head}_h) W_O$$

Channel-wise attention in SAFE-D (spatial/feature fusion):

$$\alpha_{t,i} = \frac{\exp(s_{t,i})}{\sum_{j=1}^C \exp(s_{t,j})},\quad s_{t,i} = w_s^T x_{t,i} + b_s$$

$$F_{\mathrm{Glo}}(t) = \sum_{i=1}^C \alpha_{t,i} x_{t,i}$$

Frame-wise attention in SAFE-D (temporal, per window):

$$\beta_{t,r} = \frac{\exp(e_{t,r})}{\sum_{k=1}^R \exp(e_{t,k})},\quad e_{t,r} = w_t^T h_{t,r} + b_t$$

$$F_{\mathrm{Loc}}(t) = \sum_{r=1}^R \beta_{t,r} h_{t,r}$$

Density-guided label smoothing (Transformer Fusion):

$$\mathcal{L} = -\sum_{t=1}^{T_n} \sum_{k=1}^C q''(k|t) \log p_t(k)$$

These equations exemplify the spectrum of attention mechanisms—across varying data streams and embedding levels—deployed in contemporary unfit driving detection research.

---

The emergence of spatiotemporal attention frameworks has advanced the precise, robust, and interpretable detection of unfit driving states. Whether through hierarchical transformer architectures, multimodal pose-fusion, causal-permutation decoders, or hybrid channel/frame attention as in control signal models, the principal challenge and opportunity remains: learning which cues, at which moments and locations, are most predictive of unsafe driver states under diverse and realistic operational conditions (Lakhani, 2022, Cao et al., 20 Oct 2025, Akdag et al., 11 Mar 2024, Chang et al., 6 Mar 2025, Wharton et al., 2021).