ADS-Edit: Multimodal Knowledge Editing Dataset

• ADS-Edit is a multimodal benchmark dataset that enables targeted knowledge editing for autonomous driving by integrating diverse real-world scenarios.
• It incorporates video sequences, multi-view images, and single images to comprehensively evaluate models on perception, contextual understanding, and decision-making tasks.
• The dataset uses metrics like reliability, generality, and locality to assess editing performance, ensuring precision and safety in updates.

ADS-Edit is a multimodal benchmark dataset crafted to advance research in knowledge editing for Autonomous Driving Systems (ADS). It enables rigorous evaluation of editing methods for large multimodal models challenged by real-world driving scenarios. The dataset incorporates varied data modalities and scenario types, alongside comprehensive metrics for reliability, generality, and locality, addressing both the safety-critical demands and evolving landscape of autonomous vehicle technologies.

1. Dataset Structure and Composition

ADS-Edit is designed for multimodal knowledge editing in the ADS domain by including data representative of diverse, real-world conditions. The dataset comprises three main scenario types:

• Perception: Basic object and obstacle detection, assessing models’ visual capabilities.
• Understanding: Semantic and rule-centered evaluation, testing competence in traffic regulations and contextual scene description.
• Decision Making: Requires the synthesis of perception and understanding to generate actionable driving decisions.

Each scenario is supported by three data modalities:

• Video Sequences: Instances consist of typically 5 temporally ordered frames, capturing dynamic and complex road environments.
• Multi-View Images: Simulate various sensor perspectives around the vehicle, representing fusion from different camera views.
• Single Images: Tasked with measuring recognition of static scene elements.

This design ensures comprehensive coverage from low-level perception to high-level inference, facilitating broad analysis of targeted knowledge edits within large multimodal models.

2. Knowledge Editing Paradigm

Knowledge editing in ADS-Edit refers to the targeted modification of a model’s parameters, specifically those encoding domain-specific knowledge, obviating the need for full retraining. This paradigm distinguishes itself from conventional fine-tuning by aiming to affect only targeted behaviors:

• Scope: Edits are localized to the specific scope $I(x_e, y_e)$ (inputs and desired outputs), defined formally as

$$f_{\theta_e}(x) = \begin{cases} y_e & \text{if } x \in I(x_e, y_e) \ f_{\theta}(x) & \text{if } x \notin I(x_e, y_e) \end{cases}$$

where $f_{\theta_e}$ denotes the post-edit model, $f_{\theta}$ the original, and $y_e$ the edited response.

• Contrast with Fine-Tuning: Targeted editing offers reduced computational cost and avoids catastrophic forgetting, preserving knowledge outside the edit’s scope.

Knowledge editing in this context focuses on adjusting factual, contextual, or rule-based information—typified by the insertion or updating of traffic rules, scene interpretation, or behavioral response templates.

3. Evaluation Methodology and Metrics

ADS-Edit introduces a multidimensional evaluation protocol tailored to the knowledge editing task. The principal metrics are:

Metric	Subtype	Description
Reliability	–	Edit is successfully applied in desired scenario (accuracy)
Generality	T-Generality, M-Generality	Generalization of edit to textual/multimodal queries
Locality	T-Locality, M-Locality	Preservation of behavior outside edit domain (text, modality)

• Reliability quantifies the accuracy of the modified output in the edit scope.
• Generality measures model’s ability to extrapolate updated knowledge to semantically similar but previously unseen queries, split for textual and multimodal contexts.
• Locality captures the extent to which knowledge outside the edit scope remains unaffected, crucial for maintaining operational safety and capability.

Evaluation spans both teacher-forced evaluation (token-level ground truth comparison) and free-form model generation, providing robustness and realistic assessment under unconstrained conditions.

4. Experimental Findings

Experiments employ both “single editing” (one-shot updates) and “lifelong editing” (sequential, incremental updates):

• Memory-Based Editing (GRACE, WISE): Achieve high reliability (100% tested modification rate). GRACE exhibits perfect edit reliability but comparatively lower generality; WISE balances reliability, generality, and locality.
• Prompt-Based Editing: Lower reliability/locality scores but strong free-form performance, suggesting unexpected resilience in open-ended queries.
• Model Comparison: Qwen2-VL retains original responses more robustly under editing than LLaVA-OneVision, indicating differences in edit receptivity among LMMs.
• Scenario Sensitivity: Decision-making scenarios pose greater difficulty for generality than perception or understanding, with some methods (e.g., AdaLora) scoring lower on complex synthesis tasks.
• Frame Rate Analysis: Reducing video frames (fewer visual tokens) can improve speed and, for methods such as WISE, even enhance generality.

These findings collectively illuminate strengths, weaknesses, and trade-offs inherent in existing knowledge editing approaches, as well as model-specific and scenario-specific challenges.

5. Data Processing and Technical Specification

Data preprocessing includes condensing verbose outputs from driving datasets—such as DriveLM, LingoQA, and CODA-LM—using the Deepseek-v3 model. This step ensures that driving suggestions and targets are represented concisely and unambiguously for robust evaluation. Each data instance is annotated with the expected edit response and scope definition.

Metrics are computed on both structured and free-form outputs. The formal editing behavior adheres strictly to the defined equation and scope, and all changes are benchmarked using both textual and multimodal criteria, ensuring method-agnostic and scenario-relevant assessment.

6. Implications and Prospects

ADS-Edit establishes a rigorous foundation for the development and benchmarking of knowledge editing methods in autonomous driving. Its comprehensive data composition and multifaceted metric system make it well-suited for future algorithmic refinement:

• Facilitates fast, precise knowledge updates responsive to dynamic traffic environments, new regulations, and edge-case scenarios.
• Enables comparative analysis among memory-based, parameter modification, and prompt-centric editing paradigms, clarifying trade-offs between reliability, generality, and preservation (locality).
• Provides a launch pad for hybrid or novel editing approaches, informed by empirical findings across scenario types, modalities, and model architectures.

A plausible implication is that continual improvements in knowledge editing prompted by ADS-Edit may lead to more agile, reliable, and context-aware autonomous driving systems, with rapid updating capabilities essential for real-world deployment in safety-critical domains. The capacity for safe, lifelong knowledge editing highlighted by the dataset will become increasingly central as ADS models interact with evolving traffic laws and operational contexts.