SUTD-TrafficQA Dataset Overview

• SUTD-TrafficQA is a large-scale, human-annotated video QA benchmark that assesses causal inference and event understanding in diverse traffic environments.
• It comprises 10,080 video clips and over 62,000 QA pairs across six reasoning tasks, facilitating robust visual-textual analysis.
• The dataset fosters advancements in efficient models like Eclipse and DRAX, addressing challenges in video-language fusion and real-world traffic comprehension.

SUTD-TrafficQA is a large-scale, human-annotated video question answering (QA) benchmark specifically designed to evaluate models on causal inference and event understanding in complex traffic scenarios. Comprising over 10,000 driving-scene videos and more than 62,000 manually-authored QA pairs, the dataset targets a diverse set of high-level cognitive reasoning tasks relevant to intelligent transportation, automated driving, and vision-based traffic analysis (Xu et al., 2021). It serves as a testbed for video-LLMs’ abilities to perform visual reasoning, anticipate events, understand causes, and generate introspective recommendations in realistic traffic environments (Dogra et al., 2023).

1. Dataset Composition and Annotation

SUTD-TrafficQA contains 10,080 short video clips sourced both online (YouTube, LiveLeak, Twitter, Bilibili) and offline from hand-held and car-mounted cameras. Clip durations range from 1 to 70 seconds. The benchmark features 62,535 human-written QA pairs covering a broad geographic, lighting, and weather spectrum to maximize situational diversity (Xu et al., 2021).

Annotation followed a structured methodology:

• Annotators were trained on six reasoning tasks and wrote at least three questions per clip, using freely paraphrased language to avoid annotation artifacts.
• Each QA pair includes a question and 2–12 answer candidates, with one or more correct choices; most commonly, the multiple-choice setting (MCQ) presents four candidates (one correct and three distractors) (Xu et al., 2021, Dogra et al., 2023).
• Weekly cross-checks and balanced distribution monitoring ensured even task coverage and QA quality.

In downstream usage, each video is typically partitioned into 8 equal-length clips, with 16 uniformly sampled frames per clip, yielding 128 frames per video as used in DRAX-based models (Dogra et al., 2023).

2. Reasoning Task Taxonomy

The dataset introduces six distinct reasoning tasks, each designed to probe a complementary facet of video event comprehension (Xu et al., 2021, Dogra et al., 2023):

1. Basic Understanding: Queries visual attributes, event presence, counts, and basic classification (e.g., “What color is the traffic light when the sedan passes?”).
2. Event Forecasting: Predicts near-future events immediately following the clip (e.g., “What happens next?”).
3. Reverse Reasoning: Infers events leading up to the first frame (e.g., “What just happened before this video started?”).
4. Counterfactual Inference: Posits alternate outcomes under hypothetical changes (e.g., “If the blue sedan had not braked, what would have occurred?”).
5. Introspection: Seeks preventive strategies and retrospective advice (e.g., “How could the driver have prevented the collision?”).
6. Attribution: Explains the causes of observed traffic events (e.g., “What caused the rear-end crash?”).

Task prevalence is strongly skewed toward Basic Understanding (≈62%) and Attribution (≈20%), with the other four tasks each contributing 4–5% of QA pairs (Dogra et al., 2023).

3. Question Formats and Answer Types

Each annotation consists of a natural-language question tied to a single video segment and a set of candidate answers. MCQs are the standard, especially four-way settings; some experiments also employ binary (true/false) formats (Xu et al., 2021). Answer types include:

• Binary (“yes/no”) for presence/absence queries.
• Categorical for vehicle, crash, or location identification.
• Counting for entity enumeration.
• Hypothetical/counterfactual outcomes and causal attributions.

Questions span from concrete (“Is there any traffic light violation?”) to abstract and counterfactual (“Would the accident still happen if all vehicles drive in their correct lane?”).

4. Evaluation Protocols and Metrics

The canonical evaluation metric is overall accuracy:

$$\mathrm{Acc} = \frac{1}{N} \sum_{i=1}^N \mathbf{1}[\hat{y}_i = y_i]$$

where $N$ is the number of QA pairs, $\hat{y}_i$ is the model prediction, and $y_i$ the ground truth (Xu et al., 2021, Dogra et al., 2023).

Category-wise (per-task) accuracy is routinely reported, with results often averaged over three random seeds for robustness. Human annotator performance provides an upper bound, consistently at ≈95.4% for the four-way MCQ variant (Xu et al., 2021, Dogra et al., 2023).

Computational efficiency is measured in GFLOPs per video for visual feature extraction. SUTD-TrafficQA benchmarks also report runtime (e.g., Eclipse runtime ≈16 ms per video on RTX 2080Ti) (Xu et al., 2021).

5. Baselines, State-of-the-art, and DRAX

A variety of baseline and advanced models have been evaluated. The table summarizes reported MCQ accuracy and GFLOPs for prominent methods (Xu et al., 2021, Dogra et al., 2023):

Model	Accuracy (MCQ)	GFLOPs
Q-type (random)	25.00%	n/a
QA-LSTM (text)	26.65%	n/a
Avgpooling	30.45%	252.7
BERT-VQA	33.68%	266.8
TVQA	35.16%	252.1
HCRN	36.49%	2051.0
Eclipse	37.05%	28.1
DRAX-base	39.63% ±0.24	not stated
DRAX-large	40.40% ±0.76	not stated
Human	95.43%	n/a

Notably, text-only models (e.g., QA-LSTM) perform near chance, demonstrating minimal language bias. Video-text joint models achieve higher scores, with the Eclipse model achieving both state-of-the-art efficiency (reducing GFLOPs by ~90% vs. HCRN/TVQA) and top-tier accuracy. The DRAX architecture further raises accuracy by 3–4 points, achieving 40.40% (DRAX-large) on this benchmark (Xu et al., 2021, Dogra et al., 2023).

6. Architectural Advancements: Eclipse and DRAX

Eclipse is a dynamically adaptive, end-to-end glimpse-based network. At each reasoning step $t$, it:

• Selects which frame to process next (skip-policy),
• Chooses between coarse or fine feature extraction (granularity-policy),
• Decides whether to terminate reasoning (exit-policy).

Major modules include:

• BiLSTM encoding of question/answer candidates,
• Context–query fusion with context-matching,
• LSTM-based interaction module,
• Prediction head for answer selection,
• Glimpse-determination and exit-policy modules.

Losses are computed for answer prediction, glimpse usefulness, feature granularity, and policy exit, with a combined objective optimized across sequential steps. To enable differentiability for discrete skip and granularity decisions, Eclipse implements a joint Gumbel-Softmax relaxation (Xu et al., 2021).

DRAX (“Distraction Removal and Attended Cross-Alignment”) leverages attention-guided removal of distractors in latent space, enhancing semantic cross-modal alignment and boosting performance in feature-query, temporal, forecasting, hypothesis, and causal reasoning tasks (Dogra et al., 2023).

7. Significance and Research Impact

SUTD-TrafficQA addresses a critical gap in evaluating video QA systems under complex, real-world causal, temporal, and hypothetical reasoning demands. Its diversity, large scale, and rigorous annotation protocols make it a standard for benchmarking multi-modal cognitive reasoning in traffic domains. The dataset has driven the development of computation-efficient and robust architectures (Eclipse, DRAX), clearly exposing limitations of offline fixed-feature and text-only models. High human–model performance gaps highlight the continued challenge of video-language causal reasoning, suggesting new opportunities for research in hierarchical, dynamic video-language fusion and robust question grounding (Xu et al., 2021, Dogra et al., 2023).