---
title: 'JAAD: Joint Attention in Autonomous Driving'
url: https://www.emergentmind.com/topics/joint-attention-in-autonomous-driving-jaad-dataset
type: topic
---

# JAAD: Joint Attention in Autonomous Driving

The Joint Attention in Autonomous Driving (JAAD) Dataset is a public resource specifically designed to support research on non-verbal communication, behavioral prediction, and joint attention dynamics between drivers and pedestrians in varied urban traffic scenarios. Through high-resolution video, detailed spatiotemporal annotations, and behavioral event streams, JAAD enables systematic analysis of visual cues and interaction protocols critical to the safety and functionality of both human-driven and autonomous vehicles [1702.03555, 1810.09805, 1609.04741].

## 1. Data Acquisition and Scope

JAAD comprises 346 high-resolution video clips, each spanning 5–15 seconds, sampled from approximately 240 hours of naturalistic driving. Two experimental vehicles—each fitted with a wide-angle dashboard camera mounted centrally just below the rear-view mirror—were used to collect footage across a range of environments:

- **Geographic Locations:** North York, Canada (55 clips, GoPro HERO+, 1920×1080); Kremenchuk, Ukraine (276 clips, Garmin GDR-35, 1920×1080); Hamburg, Germany (6 clips, Highscreen Black Box, 1280×720); Lviv, Ukraine (4 clips, Highscreen Black Box, 1280×720); New York, USA (5 clips, GoPro HERO+, 1920×1080).
- **Environmental Variation:** Clips cover urban downtown, suburban, a minority of residential/rural segments, and variable weather and daylight conditions including sunny, overcast, rainy, snowy, and sun-glare states. Over 90% of video sequences are set in urban and suburban streets [1702.03555, 1810.09805, 1609.04741].

## 2. Annotation Paradigms and Category Schema

JAAD provides comprehensive multi-layered annotation encompassing spatial, temporal, behavioral, and contextual aspects:

- **Bounding Boxes:** Manually labeled, per-frame tracks for all pedestrians and interacting vehicles, formatted as XML files (fields: frame index, x-y pixel coordinates, width, height, occlusion flag).
- **Behavioral Event Logs:** Event-based time-stamped state/action annotations (using BORIS²), e.g., {walking, standing, looking, glancing, nodding, hand_gesture, crossing}, structured per-subject (e.g., ped1, ped2, Driver).
- **Demographics:** Pedestrian age group (child, adult, elderly), gender.
- **Scene Context:** Crosswalk type (non-designated, zebra, signalized), weather, time-of-day.
- **Joint Attention Cues:** Defined as two-party interactions:
  - **Pedestrian:** Looking (gaze ≥ 1s toward oncoming vehicle), glancing (gaze < 1s), secondary cues (nods, hand gestures).
  - **Driver:** Speed maintenance/slight acceleration, slowing, full stop, explicit yielding (wheel turn, brake-light flash) [1702.03555].

Annotation quality control was conducted via manual cross-verification; inter-annotator agreement metrics (e.g., Cohen’s κ) were not reported but are flagged as future work [1702.03555, 1609.04741].

## 3. Metrics, Analytical Structures, and Modeling

JAAD is structured to facilitate empirical analysis of joint attention, pedestrian intent prediction, and driver-pedestrian negotiation:

- **Time-to-Collision (TTC):** Defined as
  $$
  \mathrm{TTC} = \frac{d}{v}
  $$
  where $d$ is the vehicle-to-crossing-point distance and $v$ is vehicle speed (assumed constant). Four TTC bins are provided: $<$3 s, 3–7 s, 7–15 s, $\geq$15 s. No crossing occurs without attention for $\mathrm{TTC}<2$ s; $\geq$50% of crossings without attention occur for $\mathrm{TTC}>10$ s [1702.03555].

- **Conditional Dependency Modeling:** Empirical histograms report conditional probabilities such as:
  $$
  P(\mathrm{Attention}\mid \mathrm{TTC}=t)
  $$
  and
  $$
  P(\mathrm{Cross} \mid \mathrm{Attention},\,\mathrm{CrosswalkType}=c,\,\mathrm{Response}=r)
  $$
  capturing dependencies among attention, crossing likelihood, TTC, crosswalk type, and driver response; no parametric regression was performed [1702.03555].

- **Behavior Recognition Baselines:** Using frame-aligned features (e.g., HOG, LBP, CNN embeddings from AlexNet) and classifiers (SVM, k-NN, ANN, DT):
  - Head orientation (HOG): 72% accuracy; (CNN): 70%
  - Motion detection (CNN): 85% accuracy [1810.09805]

- **Annotation Structure Overview Table:**

| Layer                 | Content (Examples)             | Format        |
|-----------------------|-------------------------------|---------------|
| Bounding Boxes        | Pedestrians, vehicles         | XML           |
| Behavioral Events     | Looking, walking, gestures    | CSV (BORIS2)  |
| Context Metadata      | Weather, crosswalk type, age  | CSV           |

## 4. Key Empirical Findings

Analysis of JAAD yields several robust findings regarding pedestrian and driver behavior at crossing points [1702.03555]:

- **Visual Attention Precedes Crossing:** In non-signalized crosswalks, $\gt$90% of pedestrians perform a looking action before crossing, with the rest split between glancing and absent attention (which only occurs at $\mathrm{TTC}\gt6$ s).
- **Mid-Crossing Monitoring:** Secondary attention behaviors (mid-crossing glance, look-back) arise in ~25% of crossings.
- **Attention Modulation by TTC:** The likelihood of pedestrian attention rises sharply as TTC drops below 7 s; inattentive crossings are never observed at $\mathrm{TTC}\lt2$ s.
- **Attention Duration by Age:** Elderly pedestrians average $\sim$1 s longer gaze than adults pre-crossing; children’s dwell times are modestly shorter.
- **Driver Response and Crosswalk Type:**
  - At non-designated crosswalks, if the driver slows/stops, $\approx$85% of pedestrians cross after attention. If the driver maintains speed, $\lt$10% cross (except $\mathrm{TTC}\gt25$ s).
  - At zebra-marked/signalized crossings, crossing rates after attention exceed 95%. When drivers accelerate, pedestrians hold unless $\mathrm{TTC}\gt20$ s.

## 5. File Structure, Access, and Recommended Processing

- **File and Directory Format:** Video (H.264 MP4, 1920×1080/1280×720, $\sim$30 fps), bounding boxes (per-clip XML), behavioral logs (CSV from BORIS2), e.g.:
  - `/JAAD/Videos/`
  - `/JAAD/Boxes/`
  - `/JAAD/Annotations/`
- **Usage License:** Publicly available to academic researchers under Creative Commons BY-NC-SA. Ethics approval: York University (#2016-203) [1702.03555].
- **Preprocessing Recommendations:**
  1. Undistort and crop to 16:9 aspect ratio.
  2. Frame sampling at 10 fps for behavior modeling.
  3. Per-channel mean normalization of pixel intensities.
  4. Use provided boxes to train pedestrian/vehicle detectors (e.g., SSD, YOLO).
  5. Extract head-pose/gaze features for in-box pedestrians (e.g., two-stage CNN).
  6. Build 1–3 s temporal windows before crossings for LSTM/GRU sequence models [1702.03555].

## 6. Research Applications and Limitations

JAAD serves both as a benchmarking resource and as a foundation for computational research in:

- **Joint Attention Inference:** By combining attention cues with action labels and TTC, the dataset enables probabilistic modeling of joint attention, negotiation, and right-of-way inference in traffic scenes [1702.03555, 1810.09805].
- **Behavioral Prediction:** Detection and prediction of pedestrian intent (e.g., imminent crossing) from visual cues and driver actions; supports both rule-based and data-driven approaches.
- **Limitations:** JAAD is monocular video only—no LiDAR, radar, or multi-view data. Event annotations may lack inter-annotator agreement; head orientation is binary (looking/not looking), limiting gaze granularity. Cyclist interactions are rare and not separately annotated. The moderate number of clips may not cover edge cases (e.g., severe occlusion, night scenarios) [1609.04741, 1810.09805].

## 7. Extensions and Suggested Improvements

Research utilizing JAAD highlights potential directions for dataset and methodology enrichment:

- **Higher Granularity Annotations:** Finer head-pose and gaze-direction labeling.
- **Sensor Diversity:** Inclusion of additional modalities (e.g., LiDAR, radar, optical flow).
- **Deeper Learning Architectures:** Application of advanced models (e.g., VGG, ResNet) and explicit pose-based descriptors for enhanced effect modeling [1810.09805].
- **Automated Semantic Parsing:** Automated annotation for additional contextual variables (e.g., crosswalk state, signalization) as future enhancement targets.

JAAD constitutes a comprehensive benchmark for the empirical study of joint attention, non-verbal communication, and pedestrian intention in realistic urban traffic, with strong relevance for autonomous driving research [1702.03555, 1810.09805, 1609.04741].

Source: https://www.emergentmind.com/topics/joint-attention-in-autonomous-driving-jaad-dataset