SafeDriveRAG: Safety-Driven Retrieval Framework

Updated 7 July 2026

SafeDriveRAG is a framework that integrates a heterogeneous multimodal knowledge graph with retrieval-augmented generation to enhance visual question answering in safety-critical autonomous driving scenarios.
The SafeDrive228K benchmark provides 228K multimodal QA pairs across 18 sub-tasks spanning Traffic Accidents, Corner Cases, and Traffic Safety Commonsense, enabling comprehensive performance evaluation.
By retrieving structured external knowledge and injecting it at inference time, SafeDriveRAG improves VLM reasoning in hazardous traffic conditions while exposing current model limitations.

SafeDriveRAG is a knowledge graph-based retrieval-augmented generation framework for safety-critical autonomous driving, introduced together with the SafeDrive228K benchmark to evaluate and improve vision-LLMs on multimodal traffic-safety question answering. Its stated scope spans perception, situational understanding, and path planning, but its implemented contribution is a plug-and-play VLM baseline for visual question answering that retrieves structured traffic-safety knowledge from a multimodal graph and injects it at inference time. The accompanying benchmark comprises 228K examples across 18 sub-tasks and is organized around three domains—Traffic Accidents, Corner Cases, and Traffic Safety Commonsense—so that model behavior can be assessed in scenarios that standard driving-oriented VLM evaluation often underrepresents (Ye et al., 29 Jul 2025).

1. Conceptual position within retrieval-augmented driving research

SafeDriveRAG belongs to a broader shift in autonomous-driving research from purely parametric models toward memory-augmented and retrieval-augmented systems. Within that broader landscape, however, its role is specific: it is not a closed-loop controller and not a direct motion planner, but a safety-oriented multimodal QA framework intended to strengthen VLM reasoning in hazardous or knowledge-intensive traffic scenarios (Ye et al., 29 Jul 2025).

This focus distinguishes it from several adjacent retrieval-based directions. “Driving-RAG” emphasizes scenario embedding, search, and graph-based reorganization for trajectory-planning-oriented RAG applications (Chang et al., 6 Apr 2025). “RAG-Driver” retrieves expert demonstrations to support explainable action prediction and control generation in a multimodal LLM (Yuan et al., 2024). “RealDrive” retrieves expert demonstrations as structured priors for a diffusion planner, targeting long-tail planning safety and controllability (Ding et al., 30 May 2025). “Towards Automated Safety Requirements Derivation Using Agent-based RAG” studies agent-based retrieval for safety requirements derived from standards and system documentation, rather than multimodal scene understanding (Balu et al., 15 Apr 2025). This suggests that SafeDriveRAG occupies the knowledge-heavy safety-reasoning end of the retrieval-augmented driving spectrum: it targets accident analysis, corner-case understanding, legal and commonsense knowledge, and question answering grounded in external traffic-safety material.

A common misconception is to treat SafeDriveRAG as a general autonomous-driving stack. The paper does motivate autonomous-driving safety broadly, but the concrete method and experiments are centered on multimodal VQA. Its contribution is therefore best understood as a benchmark-and-baseline framework for evaluating and improving safety reasoning in VLMs, rather than as an end-to-end driving policy (Ye et al., 29 Jul 2025).

2. SafeDrive228K benchmark

SafeDrive228K is presented as the first large-scale multimodal question-answering benchmark focused on traffic safety. It contains 228K multimodal QA pairs, 18 sub-tasks, both video QA and image QA, and both single-choice, multiple-choice, and open-ended QA. The benchmark is divided into Traffic Accidents, Corner Cases, and Traffic Safety Commonsense (Ye et al., 29 Jul 2025).

Domain	Source basis	Reported scale
Traffic Accidents	CAP-DATA	102K QA pairs from 9,331 videos
Corner Cases	CODA-LM	69K QA pairs from 9,768 scenarios/images
Traffic Safety Commonsense	Internet documents + IDKB-derived material	57K QA pairs from 26K images

The accident split evaluates understanding of dangerous real-world crash scenes and related safety reasoning. The paper explicitly lists seven accident question types: Accident type detection, Accident prevention, Emergency handling, Legal regulation recognition, Right-of-way determination, Complex road condition decision-making, and General questions. The corner-case split targets rare, high-risk, or atypical scenarios and explicitly lists five categories: Object recognition, Object localization, Danger prevention, Emergency handling, and General questions. For Traffic Safety Commonsense, the paper states that the tasks cover driving regulations, traffic sign recognition, and behavioral guidelines; since the benchmark totals 18 sub-tasks and the paper explicitly lists 7 accident types and 5 corner-case types, this suggests that the remaining 6 sub-tasks belong to the commonsense split, although the exact six names are not enumerated in the excerpt (Ye et al., 29 Jul 2025).

Benchmark construction uses a semi-automated pipeline. For traffic accidents, videos were selected from CAP-DATA and poor-quality clips were removed; an LLM then generated detailed video descriptions based on existing annotations. For corner cases, representative images were selected from CODA-LM. For traffic safety commonsense, approximately 1,100 relevant documents comprising more than 2,600 pages were collected from Chinese internet sources, supplemented with driving manuals and test data from IDKB, and converted into evaluation-ready material using layout detection and OCR. Question-answer pairs were generated with few-shot prompts, with malformed outputs regenerated after format inspection. Quality control proceeded in three stages: script filtering, GPT-4o-mini checking for logical consistency, and human expert review by 10 experts with driving experience, whose corrected answers became the gold standard (Ye et al., 29 Jul 2025).

The benchmark design is notable because it combines scene-grounded hazards with document-grounded safety knowledge. This makes it possible to evaluate both visual hazard comprehension and rule- or commonsense-heavy reasoning in a unified setting. It also creates a direct testbed for retrieval-augmented approaches, since many questions cannot be answered reliably from visual evidence alone.

3. Knowledge graph and indexing architecture

SafeDriveRAG uses a heterogeneous multimodal knowledge graph rather than a plain chunk index. The paper defines the graph as

$G=(\{V_e,V_i,V_c\},\{E_{ee},E_{ec}\}),$

where $V_e$ denotes entity nodes, $V_i$ image entity nodes, $V_c$ text chunk nodes, $E_{ee}$ entity–entity edges, and $E_{ec}$ entity–chunk edges (Ye et al., 29 Jul 2025).

Entity nodes represent key semantic units extracted from traffic-safety documents. Image entity nodes preserve document-linked visual content; the paper notes that a placeholder symbol <image> is inserted in text to indicate image location. Text chunk nodes preserve contextual paragraphs. Entity–entity edges encode semantic, spatial, or temporal associations, while entity–chunk edges preserve the link between structured concepts and their source context. This graph design is intended to retain both local semantic entities and broader textual coherence, instead of forcing retrieval to choose between symbolic fragments and unstructured paragraphs.

The graph is built from traffic safety legislation, driving guides, and accident response manuals collected from the Internet. The corpus contains over 1.32 million tokens. Documents are split into chunks of maximum 1200 tokens with a sliding-window overlap of 100 tokens. The framework is described as having a Multimodal Indexing Module and a Multi-Scale Subgraph Retrieval Module. The paper also formalizes the RAG system as

$M=\begin{pmatrix}G, R=(\varphi,\psi)\end{pmatrix},$

where $\varphi$ extracts the knowledge graph from raw documents and constructs an index, and $\psi(\cdot)$ is the retrieval module (Ye et al., 29 Jul 2025).

The indexing choice has a methodological implication. A plain text-RAG system can retrieve relevant chunks, but it does not explicitly preserve image-linked knowledge or entity-level relational structure. SafeDriveRAG instead assumes that safety reasoning benefits from a graph that can connect regulations, hazards, actions, and supporting contextual passages. This suggests that its core retrieval problem is not merely lexical relevance but subgraph construction around a safety query.

4. Multi-scale subgraph retrieval and inference workflow

At inference time, the input is a scene $I$ —either a video $V_e$ 0 or an image $V_e$ 1—plus a question $V_e$ 2. In the multiple-choice setting, the paper writes the decision problem as

$V_e$ 3

while open-ended answering is written as

$V_e$ 4

SafeDriveRAG alters this process by retrieving graph-grounded external knowledge before the VLM generates its answer (Ye et al., 29 Jul 2025).

The retrieval pipeline begins by extracting key information from the question, denoted

$V_e$ 5

These keywords are used to identify anchor entities in the graph:

$V_e$ 6

From these anchors, the system performs bounded multi-hop expansion:

$V_e$ 7

Candidate chunks linked to the expanded entities are then scored by a combined graph-and-text relevance function:

$V_e$ 8

where $V_e$ 9 is the query, $V_i$ 0 is a candidate chunk, $V_i$ 1 is query–entity similarity, $V_i$ 2 is query–chunk similarity, $V_i$ 3 is a path decay factor, $V_i$ 4 is the hop count from the anchor to entity $V_i$ 5, and $V_i$ 6 weights graph-expanded entity relevance against direct chunk relevance (Ye et al., 29 Jul 2025).

Operationally, the system uses top- $V_i$ 7 retrieval with entity-node top- $V_i$ 8 and chunk top- $V_i$ 9. The resulting subgraph contains selected entities, associated image nodes, and text chunks, which are then passed to the VLM as external knowledge. The paper emphasizes that this is plug-and-play: the RAG-enhanced models only incorporate external knowledge, such as entities and chunks, to assist understanding and generation, while the base VLM architecture remains unchanged (Ye et al., 29 Jul 2025).

A common misconception is that SafeDriveRAG performs unrestricted graph traversal. The paper instead describes a constrained retrieval strategy: anchor initialization, bounded multi-hop expansion, and top- $V_c$ 0 ranking. This design is intended to improve efficiency relative to more expensive graph-RAG methods while preserving enough structural context to support safety reasoning.

5. Empirical performance

The paper evaluates five mainstream VLMs in original and RAG-enhanced form: Qwen2.5-VL-7B, Qwen2.5-VL-3B, LLaVA-OneVision-7B, LLaVA-OneVision-0.5B, and Phi-4-multimodal-instruct-4.5B. The main summary metric is SafeDrive Score, which combines multiple-choice and open-ended performance by weighting each by the number of items (Ye et al., 29 Jul 2025).

Model	w/o RAG	w/ RAG
Qwen2.5-VL-3B	44.89%	57.38%
Qwen2.5-VL-7B	53.14%	60.18%
LLaVA-OneVision-0.5B	20.57%	29.52%
LLaVA-OneVision-7B	48.32%	57.24%
Phi-4-multimodal-instruct-4.5B	43.16%	51.55%

The best overall result is Qwen2.5-VL-7B with SafeDriveRAG at 60.18% SafeDrive Score. The paper further reports average gains across the benchmark domains of +4.73% on Traffic Accidents, +8.79% on Corner Cases, and +14.57% on Traffic Safety Commonsense. It also reports baseline weakness: average original-model performance is 40.31% on Traffic Accidents, 45.02% on Corner Cases, and 40.71% on Traffic Safety Commonsense, and most original models fail to exceed 50% across all tasks (Ye et al., 29 Jul 2025).

These numbers support two conclusions. First, external safety knowledge materially improves performance across all tested VLMs. Second, the absolute scores remain modest, which indicates substantial unresolved difficulty in safety-critical autonomous-driving QA. The paper explicitly notes that even the strongest compact model is far from reliable enough for safety-critical deployment.

An ablation on Traffic Safety Commonsense using Qwen2.5-VL-7B and a 10% test set compares Naïve RAG, MiniRAG, and SafeDriveRAG. The reported retrieval times are 46.22 s for Naïve RAG, 9519.98 s for MiniRAG, and 884.10 s for SafeDriveRAG; the corresponding SafeDrive Scores are 60.18%, 61.26%, and 62.07%. This indicates that SafeDriveRAG achieves the best score among the compared RAG variants while remaining much faster than MiniRAG, though still substantially slower than Naïve RAG (Ye et al., 29 Jul 2025).

The paper also reports a nuanced limitation: the smallest domain-level gain occurs on Traffic Accidents, which it attributes to input truncation. Accident tasks often combine video input with retrieved entities and chunks, making context length a practical bottleneck, especially for models at or below 7B parameters. This suggests that retrieval quality alone is not sufficient; context-budget management becomes a central systems issue for safety-critical multimodal RAG.

6. Interpretation, limitations, and significance

SafeDriveRAG shows that structured retrieval improves safety-oriented VLM reasoning, but it does not show that current VLMs are adequate for autonomous-driving safety in a strong operational sense. The benchmark’s highest reported SafeDrive Score is 60.18%, and the paper explicitly presents the benchmark as exposing substantial weaknesses in current lightweight, deployable VLMs (Ye et al., 29 Jul 2025).

Several limitations are either stated or visible from the specification. The retrieval pipeline omits some reproduction-critical details: the exact embedding model for graph similarity, the hop limit $V_c$ 1, the threshold $V_c$ 2, the weighting factor $V_c$ 3, the decay $V_c$ 4, the exact prompt template, and full split details are not specified in the excerpt. Commonsense material is collected from Chinese internet sources and supplemented with IDKB-derived content translated into English, which introduces dependence on source quality and translation normalization. The retrieval ablation shows that graph-based retrieval improves score over Naïve RAG, but also that the efficiency cost is nontrivial. These limitations do not invalidate the framework, but they constrain direct interpretation of SafeDriveRAG as a deployment-ready safety mechanism.

Another misconception is to view knowledge-graph retrieval as a complete solution to traffic-safety reasoning. The reported gains—especially +14.57% on Traffic Safety Commonsense—show that external knowledge is highly useful for knowledge-heavy tasks, yet the remaining performance gap shows that retrieval does not remove the need for stronger grounding, longer-context handling, and better multimodal reasoning. This suggests that SafeDriveRAG is best understood as infrastructure for safety-oriented evaluation and augmentation: it supplies a benchmark that makes failure modes visible and a structured retrieval baseline that improves VLM behavior without changing the underlying model (Ye et al., 29 Jul 2025).

In the wider literature, this places SafeDriveRAG alongside scenario-retrieval systems for planning (Chang et al., 6 Apr 2025), retrieval-augmented multimodal explanation models (Yuan et al., 2024), and retrieval-augmented generative planners (Ding et al., 30 May 2025), but with a distinct emphasis on traffic accidents, corner cases, safety commonsense, and graph-grounded VQA. Its encyclopedic significance lies in that dual contribution: SafeDrive228K establishes a safety-centered evaluation substrate, and SafeDriveRAG demonstrates that structured multimodal retrieval can measurably improve VLM performance on that substrate, while also revealing how far current models remain from robust safety reasoning in autonomous driving (Ye et al., 29 Jul 2025).