SafeDrive228K: Multimodal Safety Benchmark

• SafeDrive228K is a multimodal QA benchmark designed for safety-critical autonomous driving, featuring 228K QA pairs across accident, corner case, and commonsense tasks.
• It employs a semi-automated pipeline with curated video and image data, few-shot prompting, and rigorous expert review to ensure high-quality safety evaluations.
• Empirical results show that retrieval augmentation through SafeDriveRAG significantly improves model performance, highlighting current gaps in safety reasoning.

SafeDrive228K is a multimodal question-answering benchmark for safety-critical autonomous driving, introduced together with the SafeDriveRAG framework to evaluate whether vision-LLMs can understand and reason about hazardous, unusual, and rule-sensitive road situations (Ye et al., 29 Jul 2025). It comprises 228K QA pairs across 18 sub-tasks, organized into three main scenario groups—Traffic Accidents, Corner Cases, and Traffic Safety Commonsense—and supports both image QA and video QA. The benchmark is designed to probe not only dangerous-scene perception, but also situational understanding, safety reasoning, emergency response, legal and rule-based judgment, and commonsense knowledge for everyday driving.

1. Motivation and conceptual scope

SafeDrive228K was created in response to a specific limitation in prior autonomous-driving evaluation: strong performance on general multimodal benchmarks does not by itself establish reliability in safety-critical driving environments (Ye et al., 29 Jul 2025). The benchmark’s motivating argument is that existing driving datasets often focus on perception, planning, or generic scene understanding, while underrepresenting the safety problems that dominate real deployment concerns, particularly traffic accidents, rare corner cases, and traffic-safety commonsense.

The benchmark is positioned as a unified testbed for situations in which errors may be catastrophic. Its scope therefore extends beyond object recognition and scene captioning to questions about accident causes, prevention, emergency handling, right-of-way, legal regulation recognition, complex road condition decision-making, and practical safe behavior. A common misconception is that SafeDrive228K is primarily an accident-recognition dataset. In fact, accident understanding is only one of its three major scenario groups; the benchmark also includes image-based corner-case reasoning and commonsense traffic-safety evaluation.

Another common misconception is that SafeDrive228K measures only perceptual competence. The task design explicitly targets perception of dangerous scenes, situational understanding, safety reasoning, emergency response, legal and rule-based judgment, and commonsense knowledge. This broader scope is central to the benchmark’s role as a safety-centered evaluation suite rather than a conventional scene-understanding corpus.

2. Corpus composition and task taxonomy

The benchmark is partitioned into three major components with distinct source modalities and safety objectives (Ye et al., 29 Jul 2025).

Scenario group	Source modality	Scale
Traffic Accidents	9,331 traffic accident videos	102K QA pairs
Corner Cases	9,768 corner-case images	69K QA pairs
Traffic Safety Commonsense	26K images	57K QA pairs

These subsets jointly yield the headline statistics of 228K total QA pairs, 18 sub-tasks, 9,331 accident videos, and 35K images total across the image-based subsets. The organization is safety-oriented rather than merely semantic.

Within the Traffic Accident subset, the tasks are meant to evaluate the model’s ability to detect accident types, understand accident causes, predict or prevent accidents, suggest emergency actions, recognize relevant laws and regulations, determine right-of-way, reason about complex road conditions, and answer open-ended general questions. The paper notes that this subset covers the accident timeline from pre-accident risk prediction to post-accident response.

The Corner Case subset concentrates on rare and risky scenarios, especially “unknown hazards,” including unexpected objects, abrupt events, and visibility issues. Its task taxonomy includes object recognition, object localization, danger prevention, emergency handling, and general questions. The stated goal is to probe robustness when the environment is atypical or hard to predict.

The Traffic Safety Commonsense subset addresses ordinary road environments rather than extreme events. It focuses on traffic regulations, traffic sign recognition, safe driving behavior, legal compliance, and commonsense safety knowledge. This design makes the benchmark broad enough to test both exceptional hazards and routine rule-sensitive behavior.

3. Construction pipeline and quality control

SafeDrive228K is built through a semi-automated construction pipeline with three main stages: source data processing, QA pair generation, and data validation (Ye et al., 29 Jul 2025). In the accident portion, the authors selected 9K high-quality videos from CAP-DATA, filtering out low-quality clips. For corner cases, they selected 9K representative images from CODA-LM. For the commonsense subset, they processed around 1,100 documents totaling more than 2,600 pages, using OCR and layout detection to extract structured content; they also merged material from IDKB and translated relevant non-English content into English.

QA generation uses few-shot prompting to produce diverse questions. Different prompts were designed for each question category, and questions and answers were generated in structured formats. This is important because the benchmark spans multiple reasoning styles, including multiple-choice and open-ended responses, and therefore requires category-specific prompting rather than a single templated procedure.

Validation proceeds through a three-step filtering and review process. First, the authors apply script-based filtering of obviously bad entries. Second, they perform logical consistency checking using GPT-4o-mini. Third, they use expert review by 10 driving-experienced experts for contentious cases. The paper characterizes the result as a quality-controlled gold-standard dataset.

This pipeline suggests that SafeDrive228K is not simply a large synthetic QA dump layered onto existing media. Its construction emphasizes curated source selection, prompt specialization, and post-generation review, all of which are consequential for safety evaluation where annotation noise can distort model comparisons.

4. Formal task definition and evaluation protocol

SafeDrive228K is formalized as a multimodal QA problem over an input scene $I$, which may be a video $V$ or an image $S$, together with a question $Q$ (Ye et al., 29 Jul 2025). For multiple-choice QA, with candidate answers

$A = \{a_1, a_2, \cdots, a_n\},$

the objective is

$$a^* = \arg\max_{a \in A} \mathrm{VLM}(a \mid I, Q).$$

For open-ended QA, the model generates a response directly: $A = \mathrm{VLM}(Q, I).$

The evaluation protocol distinguishes sharply between these two regimes. For multiple-choice questions, the model output is parsed with regular expressions to extract selected options, and the prediction is counted correct only if it includes all valid options for the question. For open-ended questions, the benchmark uses ROUGE and SEMScore. ROUGE measures n-gram overlap, while SEMScore measures semantic similarity more directly.

The paper then combines multiple-choice and open-ended results into an overall composite metric, the SafeDrive Score, weighted by the number of items in each category. This weighting matters because the benchmark spans heterogeneous QA forms and sub-task sizes; a single unweighted average would distort the contribution of large and small components.

A further design point is that the evaluation uses a uniform prompt across models for fairness. This constrains one source of performance variance and makes the reported comparisons more directly attributable to the underlying model or retrieval method.

5. Reference baseline: SafeDriveRAG

SafeDrive228K is introduced together with SafeDriveRAG, a plug-and-play multimodal retrieval-augmented generation baseline intended to improve performance on the benchmark (Ye et al., 29 Jul 2025). The benchmark is therefore not only a measurement instrument for raw VLM ability, but also a testbed for whether external traffic-safety knowledge can improve driving-related reasoning.

The paper evaluates five mainstream open-source VLMs, all constrained to sizes suitable for in-vehicle deployment: Qwen2.5-VL-7B, Qwen2.5-VL-3B, LLaVA-OneVision-7B, LLaVA-OneVision-0.5B, and Phi-4-multimodal-instruct-4.5B. Each model is tested both in its original form and in a RAG-enhanced form using SafeDriveRAG.

SafeDriveRAG converts a large corpus of traffic safety documents into a heterogeneous multimodal knowledge graph

$G = (\{V_e, V_i, V_c\}, \{E_{ee}, E_{ec}\}),$

with entity nodes $V_e$, image entity nodes $V_i$, text chunk nodes $V$0, and entity-linked edge sets $V$1 and $V$2. This graph design allows retrieval over both textual and image-linked safety knowledge, which is relevant for traffic signs, diagrams, and visually grounded regulations.

Its retrieval module uses multi-scale subgraph retrieval. The pipeline extracts query keywords

$V$3

finds anchor entities above a similarity threshold,

$V$4

expands anchors through $V$5-hop traversal,

$V$6

and scores candidate chunks with a combined entity-chunk similarity,

$V$7

The intended role of this baseline is diagnostic as much as algorithmic. If retrieval materially improves performance, then benchmark difficulty is not reducible to visual perception alone; it also reflects missing traffic-safety knowledge and weak rule-sensitive reasoning.

6. Empirical findings

The main empirical conclusion is that contemporary mainstream VLMs struggle substantially on SafeDrive228K, and that retrieval augmentation improves performance consistently (Ye et al., 29 Jul 2025). Without RAG, most models do not exceed 50% SafeDrive Score on the sub-tasks, with only Qwen2.5-VL-7B slightly above 50 overall. Average baseline performance by subtask is reported as approximately 40.31% for Traffic Accidents, 45.02% for Corner Cases, and 40.71% for Traffic Safety Commonsense.

Adding SafeDriveRAG produces average gains of +4.73% on Traffic Accidents, +8.79% on Corner Cases, and +14.57% on Traffic Safety Commonsense. The paper interprets this as evidence that external structured safety knowledge is especially helpful for commonsense and corner-case reasoning, and still beneficial for accident understanding.

The model-specific observations are also notable. Qwen2.5-VL-7B is reported as the strongest and most stable overall. LLaVA-OneVision-7B shows the largest average gain after RAG, with a reported improvement of about +14.39 in some settings across corner cases and commonsense tasks. Phi-4-multimodal-instruct-4.5B also improves noticeably with retrieval. The 7B models generally outperform their smaller counterparts, but smaller models can close the gap once equipped with RAG.

The paper also reports a retrieval ablation on the traffic-safety-commonsense subset. Naïve RAG obtains 46.22 s retrieval time and 60.18% SafeDrive Score; MiniRAG obtains 9519.98 s and 61.26%; SafeDriveRAG obtains 884.10 s and 62.07%. This suggests that the proposed retrieval method is framed not only as more accurate than the alternatives in this comparison, but also as a more favorable balance between accuracy and efficiency.

These findings support one of the benchmark’s central claims: current VLMs still lack enough specialized traffic safety knowledge and reasoning ability. SafeDrive228K thus functions as a stress test for precisely those competencies that general multimodal benchmarks often fail to isolate.

7. Position within safe-driving evaluation research

SafeDrive228K occupies a specific place within the broader landscape of safe-driving evaluation. It is a multimodal QA benchmark for safety-centered comprehension and reasoning, rather than a closed-loop driving benchmark or a direct control-policy evaluation suite (Ye et al., 29 Jul 2025). This distinction is essential when comparing it with adjacent work.

A useful contrast is Safe2Drive, which evaluates end-to-end autonomous driving policies in 100 Bench2Drive-aligned scenarios centered on work zones, pedestrian jaywalking, and occluded vulnerable road users, and introduces the SafeDriving Score (SDS) to expose failures such as non-braking collisions, work-zone-object contact, route-corridor drift, and unsafe lane interpretation (Sahu et al., 29 May 2026). Safe2Drive therefore interrogates closed-loop behavior under safety-critical scenarios, whereas SafeDrive228K interrogates question answering, safety reasoning, and rule-sensitive multimodal understanding. A plausible implication is that the two benchmarks address complementary failure modes: one in embodied driving behavior, the other in multimodal reasoning about safety.

Another relevant comparison is DriveSafe, a caption-centered framework for risk detection and safety suggestion generation on DRAMA. DriveSafe constructs spatially grounded captions using motion, spatial, and depth cues, then uses an LLM to produce a risk label, refined risk caption, risk keywords, localized box, and a mapped safety action such as Stop, Slow down, or Yield (Artham et al., 16 May 2026). This indicates a neighboring research direction in which language-grounded risk reasoning is coupled directly to action suggestions. SafeDrive228K is closely aligned with that agenda insofar as it includes emergency response, legal judgment, and commonsense safety knowledge, but its primary function is benchmarking rather than end-task deployment.

A third comparison is SAFE-D, which focuses on pathology-induced abnormal driving detection using steering wheel angle, accelerator pedal position, and brake pedal position, with a spatiotemporal architecture combining ResNet, TCN, and multihead attention; it reports 96.8% average accuracy across three maps in distinguishing normal and Parkinson-affected driving patterns (Cao et al., 20 Oct 2025). This is a different modality and problem formulation, centered on vehicle-control time series and driver state rather than scene-grounded multimodal QA. The contrast underscores that “safe driving” benchmarks span multiple layers of the stack: closed-loop behavior, perception-and-language reasoning, and driver-state anomaly detection.

Taken together, these comparisons suggest that SafeDrive228K is best understood as a benchmark for safety-critical multimodal reasoning. Its distinctive contribution is to combine accident videos, corner-case imagery, and traffic-safety commonsense into a single evaluation framework, thereby testing whether VLMs can move from generic visual understanding toward the forms of reasoning required for safety-sensitive autonomous driving.