---
title: 'NTHU-DDD: Driver Drowsiness Dataset'
url: https://www.emergentmind.com/topics/nthu-ddd-dataset
type: topic
---

# NTHU-DDD: Driver Drowsiness Dataset

The National Tsing Hua University Drowsy Driving Dataset (NTHU-DDD) is a publicly available video dataset designed to benchmark the detection of driver drowsiness under diverse, quasi-realistic driving conditions. It features frontal, in-vehicle camera streams capturing fine-grained variations in driver vigilance, pose, and behavior, systematically labeled for supervised and unsupervised machine learning scenarios. NTHU-DDD supports rigorous evaluation of spatial, temporal, and spatiotemporal deep learning models focused on active road safety and driver state monitoring.

## 1. Dataset Structure and Data Collection

NTHU-DDD comprises multi-scenario video data acquired from volunteer participants simulating both alert and drowsy driving sessions in controlled settings. Recording protocols span:

- **Participants:** Up to 36 unique volunteer drivers, of various demographics, with canonical splits of 18 subjects for training/validation and 4–20 for held-out test in standard usage [2209.05269][1801.02325].
- **Session Scenarios:** Each participant is filmed across five primary conditions: day (with or without glasses), night (near-infrared, with or without glasses), and sunglasses. Scenarios cover both normal (“non-drowsy”) and simulated drowsy states (actors instructed to feign fatigue via yawning, slow blinking, head-nodding).
- **Camera Setup:** All sessions employ a single dashboard- or windshield-mounted camera, typically recording at 30 frames per second. Both visible-light and infrared (for night) modalities are present, but many studies use only one (e.g., only RGB or only IR) [2209.05269][2010.06235].
- **Data Scale:** 9.5–10.5 hours of annotated footage (for the 18- or 36-subject settings). Original data are segmented into temporally dense, overlapping fixed-length clips (e.g., 30 or 48 frames per clip), yielding tens of thousands of samples [2207.12148].

## 2. Annotation Scheme and Class Definitions

NTHU-DDD provides dense frame- and clip-level annotation to support both classification and temporal localization tasks:

- **Primary Class Labels:** The canonical two-class split is drowsy (1) and non-drowsy (0) at the frame or clip level. Drowsy behaviors are simulated rather than recorded via physiological sensors such as EEG; labels are assigned based on observable cues like eyelid closure, blink frequency, and head pose [2207.12148].
- **Auxiliary Labels:** Per-frame annotations cover eye status (normal/sleepy), head pose (normal/nodding/looking-aside), and mouth action (normal/yawn/talking). This enables fine-grained, action- or feature-level inference [1801.02325][2010.06235].
- **Labeling Protocol:** In all documented works, manual annotation is emphasized for high reliability; confidence-based aggregation at the clip level is used to control the threshold for assigning normal vs. anomaly class (e.g., ≥ ½ frames normal within a 48-frame window) [2209.05269].

## 3. Data Organization, Splitting, and Preprocessing

The dataset is partitioned to facilitate both within- and subject-independent evaluation protocols:

- **Standard Splits:** Core usage involves subject-adversarial partitioning; for instance: 12/6/4 drivers for train/validation/test, with no subject overlap—enabling strict subject-independence in evaluation [2209.05269]. Alternative studies employ simple random clip-level splits without subject holdout, which increases overfitting risk [2207.12148].
- **Clip Extraction:** Uniformly windowed temporal sampling is applied (e.g., 30 or 48 consecutive frames per clip, stride of 23, sampled at every second or every third frame for temporal redundancy control) [2207.12148][2209.05269].
- **Preprocessing Steps:** Face detection (MTCNN or OpenCV), facial landmark extraction, and patch cropping (global face, eyes, mouth, etc.). Image normalization includes resizing (224×224 or 64×64), per-pixel or per-channel normalization, and optional contrast enhancement via CLAHE [2209.05269][2010.06235][1801.02325].
- **Augmentation:** Many published works lack augmentation (rotation, jitter, flip), but its absence is explicitly cited as a limitation and recommendation for future work [2207.12148].

| Variant       | # Subjects | Recording Type         | Frame Rate | Typical Split              |
|---------------|------------|-----------------------|------------|---------------------------|
| Standard      | 18         | RGB (day/night)       | 30 fps     | 14/1/3 sets train/val/test|
| Extended      | 36         | IR (day/night)        | 30 fps     | 12/6/4 train/val/test     |

## 4. Model Usage and Evaluation Protocols

NTHU-DDD is employed to benchmark a range of architectures, from CNN-LSTM hybrids to spatiotemporal transformers and unsupervised anomaly-detection pipelines:

- **Framewise and Clipwise Training:** Models operate either on individual frames (CNN, MCNN) or short temporal clips (LSTM, transformers, 3D-CNNs). Sequence models take as input CNN-based feature maps or raw image crops [1801.02325][2207.12148].
- **Patch/Feature Granularity:** Multi-feature networks extract separate patches for eyes, mouth, and head (in various spatial resolutions), supporting parallel feature streams (RGB and optical flow) and patch-level attention [2010.06235].
- **Transformer Architectures:** Spatiotemporal attention models (e.g., Video Swin Transformer) employ tubelet embeddings—dividing input clips into 2×4×4×3 cubes transformed into 96-dim tokens for shift-window self-attention. No temporal down-sampling is used prior to attention [2207.12148].
- **Anomaly Detection:** LSTM autoencoders trained only on normal-class clips reconstruct the input sequence; clips with elevated reconstruction loss are classified as drowsy. Confidence thresholds are tunable per evaluation [2209.05269].
- **Metrics:** Principal metrics include overall classification accuracy, ROC AUC for detection, and runtime performance in frames per second (fps). Loss functions are cross-entropy for discriminative models and L2-reconstruction for autoencoder-based solutions.

| Method           | Feature Model         | Temporal Model | Accuracy / AUC |
|------------------|----------------------|---------------|----------------|
| MCNN + LSTM      | Multi-patch CNN      | 3-layer LSTM  | 90.05%         |
| Two-Stream       | RGB + Optical Flow   | 3-net fusion  | 94.46%         |
| Video Transformer| CNN → Transformer    | Spat.-temp.   | 44% (test acc) |
| LSTM Autoencoder | ResNet-34 features   | LSTM-AE       | AUC 0.8740     |

## 5. Reported Results and Performance Benchmarks

NTHU-DDD has enabled state-of-the-art benchmarks in both supervised and unsupervised drowsiness detection:

- **Best-Case Supervised Performance:** MCNN+LSTM achieves 90.05% frame-level accuracy, and a multi-feature two-stream network reports 94.46% accuracy on the official evaluation set (four subject holdout), with throughput up to 37–60 fps [1801.02325][2010.06235].
- **Transformers:** Vision transformer models achieve as high as 67% training accuracy but only 44% on the held-out test set, revealing significant overfitting attributable to the absence of subject-independent splits and data augmentation [2207.12148].
- **Anomaly Detection:** Unsupervised LSTM autoencoders, using strictly subject-independent splits, report a ROC AUC of 0.8740 and 81.6% accuracy at a 50/50 normal-anomalous clip split [2209.05269].
- **Impact of Preprocessing:** Contrast enhancement (via CLAHE) and feature patch extraction substantially mitigate the impact of varying lighting and occlusion (e.g., sunglasses), and are shown to improve model AUC by over 5% [2209.05269][2010.06235].

## 6. Limitations, Best Practices, and Recommendations

Multiple studies identify and address current dataset and protocol constraints:

- **Simulated vs. Real Drowsiness:** All drowsy states are actor-simulated; there is no EEG, PERCLOS, or other physiological ground truth. This limits the applicability of conclusions to genuine, real-world drowsiness.
- **Insufficient Data Scale:** The dataset size (9.5–10.5 hours, ≤ 36 subjects) is suboptimal for training high-capacity transformers and deep sequence models without overfitting [2207.12148].
- **Train/Test Leakage:** Clip-level splitting (without holding out subjects) enables memorization of subject-specific facial features; strict subject-independent evaluation is encouraged [2209.05269][2207.12148].
- **Absence of Augmentation:** Limited augmentation reduces the diversity of training data, impeding generalization to unseen settings [2207.12148].
- **Label Ambiguity:** Original frame-level labels, relying on “long-term memory,” can introduce temporal imprecision regarding transition points. Alternative datasets (e.g., FI-DDD) employ instant labeling with tighter thresholds [1801.02325].

Suggested improvements include increasing both the number of subjects (≥50), recording more genuine drowsiness and diverse road conditions, collecting aligned physiological ground truth, and merging with other open sets to cover further risk behaviors such as drunk or aggressive driving.

## 7. Research Impact and Use Cases

NTHU-DDD has established itself as a primary benchmark for multimodal and multigranularity driver state detection research. Key research outcomes include:

- **Deep Feature Development:** Encouraged architectural advances in multi-patch encoding, temporal-action recognition, and spatiotemporal self-attention approaches for behavior detection under adverse conditions.
- **Temporal Context Modeling:** Provided testbeds for long-horizon memory networks (LSTM, transformers), optical flow fusion, and anomaly detection models addressing the temporal ambiguity inherent to driver state transitions.
- **Standardized Evaluation:** Subject-independent splits with per-feature annotation facilitate rigorous, reproducible comparison across methods and encourage improved reporting on generalization.
- **Broader Relevance:** The dataset’s structure and benchmark protocols have influenced the design and evaluation criteria in both academic and industrial driver monitoring and automotive safety research. A plausible implication is that, by addressing current dataset limitations, next-generation driver state corpora will further bridge the gap between laboratory accuracy and real-world reliability.

**References:**  
[2207.12148], [1801.02325], [2010.06235], [2209.05269]

Source: https://www.emergentmind.com/topics/nthu-ddd-dataset