MTRDrive: Memory-Tool Autonomous Driving

• MTRDrive is a vision-language autonomous driving framework that integrates memory retrieval with callable perception tools to mitigate hallucinations and boost generalization.
• It employs a CLIP-based retrieval mechanism and a closed-loop decision pipeline to combine historical procedural experience with real-time tool outputs.
• Supervised fine-tuning and GRPO reinforcement learning optimize its planning accuracy, achieving robust metrics on benchmarks like NAVSIM and Roadwork-VLM.

MTRDrive is a vision-language-based autonomous driving framework that augments a core VLM with retrievable procedural driving experience and callable perception tools, with the explicit goal of improving generalization and proactive decision-making in corner cases and out-of-distribution scenarios such as roadworks. It is implemented on a 3B-parameter VLM and operates as a closed-loop system that combines a memory-based experience retrieval mechanism with dynamic toolkits, yielding what the paper terms memory-tool synergistic reasoning. On NAVSIM, the reported system achieves a PDMS of 88.3 without chain-of-thought, a driving metric score of 79.8%, and a planning accuracy of 82.6%; on the Roadwork-VLM benchmark, it reports a zero-shot driving metric score of 80.2% (Luo et al., 25 Sep 2025).

1. Definition and problem setting

MTRDrive is introduced to address two recurrent weaknesses of end-to-end autonomous driving with VLMs: hallucinations and poor generalization in OOD scenarios. The framework is motivated by the claim that current VLM-based driving systems remain fragile in complex settings, especially when they must reason about temporary road layouts, unusual signage, or other corner cases that are weakly represented in training data (Luo et al., 25 Sep 2025).

The system’s core premise is that one-shot visual reasoning is insufficient for robust driving. Instead of relying only on the current image and prompt, MTRDrive retrieves structured prior driving experiences and invokes external tools for grounded perception. This produces an interactive, closed-loop decision process in which historical procedural knowledge and real-time tool outputs are combined before final planning. The resulting design is distinct from motion-prediction systems in the MTR family, which focus on multimodal trajectory prediction from map and trajectory inputs (Shi et al., 2022), and also distinct from MTDrive, which uses multi-turn interactive reinforcement learning for trajectory refinement with a PDM Agent in NAVSIM (Li et al., 30 Jan 2026).

2. System architecture and closed-loop operation

At runtime, MTRDrive receives a forward-facing camera image and a textual prompt or scenario description. The core backbone is Qwen2.5-VL-3B, which processes the visual and textual inputs while interacting with two external subsystems: a memory module and a tool engine (Luo et al., 25 Sep 2025).

The closed-loop pipeline has seven stages. First, the image is encoded with CLIP for similarity-based retrieval. Second, the system searches a Driving Experience Base and retrieves top-$K$ Driving Scenario Documents. Third, the VLM decides whether and how to invoke tools. Fourth, tool outputs are returned to the VLM as additional context. Fifth, the model synthesizes current perception, retrieved experience, and tool outputs. Sixth, it emits a high-level speed plan and path plan. Seventh, those high-level actions are converted into trajectories that are executed in NAVSIM, enabling closed-loop evaluation with driving metrics and PDMS (Luo et al., 25 Sep 2025).

The retrieval stage is based on CLIP embeddings rather than the generative VLM itself. For an input image $I$, the visual encoder $\mathcal{F}_{enc}$ produces

$$v_I = \mathcal{F}_{enc}(I), \quad v_I \in \mathbb{R}^d$$

and scenario similarity is computed by cosine similarity: $$\text{Similarity}(I_1, I_2) = \frac{v_{I_1} \cdot v_{I_2}}{\|v_{I_1}\| \, \|v_{I_2}\|}.$$ This retrieval mechanism is intended to provide low-latency access to semantically similar prior cases while keeping the core planner compact (Luo et al., 25 Sep 2025).

3. Memory, tools, and memory-tool synergistic reasoning

A central object in MTRDrive is the Driving Scenario Document, represented as

$\langle SD, P, H, T, M \rangle.$

Its fields are defined as follows.

Component	Meaning
$SD$	Scenario description
$P$	Reasoning process
$H$	High-level decision
$T$	Tools used in $I$0
$I$1	Metadata

The memory bank stores procedural rather than merely descriptive records. Each document includes not only what happened in a scenario, but also how the system reasoned, which tools were used, and what final high-level decision was taken. Retrieved experiences are therefore used as contextual priors for both planning and tool selection (Luo et al., 25 Sep 2025).

The Vision Toolkit contains three callable functions: an Object Detection Tool, an Open-World Vocabulary Detection Tool, and an Image Cropping Tool. The first returns bounding boxes, classes, and confidences for standard traffic actors. The second accepts image-plus-text queries, such as a request to find a “No right turn” sign near a construction area, and returns matched regions with relevance scores. The third exposes a crop interface, crop(image_path, output_path, coordinates), allowing the agent to zoom into subtle regions before further analysis (Luo et al., 25 Sep 2025).

The paper formalizes experience-driven tool interaction as

$I$2

where $I$3 is the chosen tool action, $I$4 is the current visual observation, and $I$5 is the retrieved experience set. The intended effect is that tool usage is not driven purely by the current prompt, but by historically successful strategies for similar scenarios. This is the mechanism the paper identifies as memory-tool synergistic reasoning (Luo et al., 25 Sep 2025).

4. Training procedure and optimization objectives

MTRDrive is trained in two stages. The first stage is supervised fine-tuning, used as a warm-up to teach output formatting, high-level speed/path planning, and the integration of memory and tool calls. This produces an initial reference policy $I$6. The reported setup uses 2 epochs of SFT with AdamW, learning rate $I$7, batch size 2, a cosine learning-rate schedule, and a frozen CLIP encoder (Luo et al., 25 Sep 2025).

The second stage is reinforcement learning fine-tuning with GRPO. The objective is

$I$8

with

$I$9

Here $\mathcal{F}_{enc}$0 is the importance sampling ratio, $\mathcal{F}_{enc}$1 is the advantage, $\mathcal{F}_{enc}$2, and the KL coefficient is $\mathcal{F}_{enc}$3 (Luo et al., 25 Sep 2025).

The paper adds a format reward to train the meta-decision of whether experience should be used: $\mathcal{F}_{enc}$4 and combines it with a planning-accuracy reward: $\mathcal{F}_{enc}$5 This makes memory usage itself part of the optimization target, rather than a fixed inference heuristic (Luo et al., 25 Sep 2025).

5. Benchmarks, datasets, and reported results

MTRDrive is evaluated on NAVSIM and on a new benchmark called Roadwork-VLM. NAVSIM is used for closed-loop evaluation and for deriving vehicle speed profiles, angular changes, and high-level path and speed plans. Roadwork-VLM is built from ROADWork and re-annotated using Qwen2.5-VL-72B to generate detailed scenario descriptions, navigation instructions, and chain-of-thought reasoning (Luo et al., 25 Sep 2025).

The principal NAVSIM results reported for the 3B model are as follows. For high-level planning in the SFT setting, MTRDrive obtains Path 93.1%, Speed 84.6%, and Accuracy 82.6%. Its driving metrics are 79.8 for Risk Assessment, 78.8 for Reasoning, and 80.8 for Scene Awareness. In closed-loop trajectory evaluation on NAVSIM navtest, the model reports PDMS 88.3, matching WoTE at 88.3 and exceeding DiffusionDrive at 88.1, TransFuser at 84.0, UniAD at 83.4, QwenVL2.5-3B fine-tuned at 84.2, and the same base model with high-level decisions plus GRPO at 87.3 (Luo et al., 25 Sep 2025).

For zero-shot Roadwork-VLM evaluation, the reported driving metrics are Risk 80.2, Reason 79.6, and Scene 80.3. Planning metrics are Path 44.2, Speed 72.1, and Accuracy 33.5%, which exceeds the best baseline accuracy of 29.7 from Qwen2.5-VL-72B in the reported comparison (Luo et al., 25 Sep 2025).

The ablation study isolates the contribution of SFT, GRPO, and Driving Experience. The base Qwen2.5-VL-3B model yields NAVSIM Accuracy 1.6% and Roadwork Accuracy 0.5%. Adding SFT raises these to 79.9% and 13.3%. Adding GRPO without Driving Experience gives 79.6% and 17.3%. Full MTRDrive reaches 82.6% on NAVSIM and 33.5% on Roadwork-VLM. The paper interprets this as evidence that Driving Experience is the dominant factor for OOD robustness, particularly in complex roadworks (Luo et al., 25 Sep 2025).

6. Position within autonomous-driving research

MTRDrive belongs to the recent line of VLM-based end-to-end driving systems, but it differs from monolithic camera-to-action models by treating memory and tools as first-class components. The paper contrasts it with systems that rely mainly on chain-of-thought prompting, tool use guided by current text context, or generic retrieval without structured procedural experience (Luo et al., 25 Sep 2025).

This design places MTRDrive in a different part of the design space from trajectory-centric methods such as MTR and MTR++, which perform multimodal motion prediction with intention queries and transformer-based scene modeling (Shi et al., 2023), and from assistive-driving multitask perception systems such as TEM$\mathcal{F}_{enc}$6-Learning, which jointly optimize driver emotion recognition, driver behavior recognition, traffic context recognition, and vehicle behavior recognition under strict real-time constraints (Liu et al., 22 Jun 2025). It is also distinct from MTDrive, where the central mechanism is multi-turn trajectory refinement with mtGRPO and feedback from a PDM Agent rather than memory-tool synergy (Li et al., 30 Jan 2026).

A plausible implication is that MTRDrive shifts the locus of robustness from model scale alone to an interaction design in which a relatively small 3B VLM is coupled to retrieval and external grounding modules. The reported comparison with much larger Qwen2.5-VL variants, including 32B and 72B models, is presented as evidence for that claim (Luo et al., 25 Sep 2025).

7. Limitations and future directions

The paper identifies several limitations. Memory retrieval can fail when no sufficiently similar experience exists in the bank. Tool outputs can still be wrong on subtle or highly unusual objects. Scaling the memory bank and retrieval infrastructure to fleet-scale deployment raises storage and latency concerns. The current trajectory generation stage is described as relatively simple, and the authors suggest that specialized decoders such as those used in RecogDrive or AutoVLA could improve fine-grained waypoint quality and comfort (Luo et al., 25 Sep 2025).

The authors also frame real-world deployment as an open problem. They identify integration with LiDAR, radar, multiple cameras, and HD maps as future work, and note the need for deeper synergy between memory retrieval and tool use. The paper further emphasizes that safety certification and physical-vehicle validation remain future goals rather than present capabilities (Luo et al., 25 Sep 2025).

Within the broader taxonomy of similarly named systems, the terminology matters. MTDrive refers to a multi-turn interactive reinforcement learning framework for autonomous-driving trajectory planning (Li et al., 30 Jan 2026), whereas MTRDrive refers to memory-tool synergistic reasoning for robust autonomous driving in corner cases (Luo et al., 25 Sep 2025). The two systems address different failure modes, use different learning formulations, and should not be conflated.