OmniDrive-R1 Autonomous Driving VLM

• OmniDrive-R1 is an end-to-end vision-language framework that unifies visual perception with linguistic reasoning to address object hallucination in autonomous driving.
• It pioneers the interleaved Multi-modal Chain-of-Thought (iMCoT) mechanism and employs the Clip-GRPO algorithm for robust, annotation-free visual grounding.
• A two-stage reinforcement learning pipeline significantly improves overall reasoning and MCQ accuracy while optimizing tool usage in complex driving tasks.

OmniDrive-R1 is an end-to-end vision-language modeling (VLM) framework engineered for trustworthy autonomous driving. It addresses core reliability failures in existing VLMs—most notably object hallucination that arises from ungrounded, text-only chain-of-thought (CoT) reasoning—by tightly integrating multi-modal perception and reasoning within a single, jointly-optimized architecture. OmniDrive-R1 pioneers a reinforcement-driven, interleaved Multi-modal Chain-of-Thought (iMCoT) mechanism and introduces the Clip-GRPO algorithm to enable robust visual grounding without dense labels, resulting in substantial empirical advances over contemporary baselines (Zhang et al., 16 Dec 2025).

1. Model Architecture

OmniDrive-R1 leverages a multi-modal transformer backbone, implemented atop Qwen2.5VL-7B, and unifies visual perception with linguistic reasoning. The dual-stream encoder projects raw images, derived from up to six synchronized vehicular camera views, into patch embeddings and processes text inputs into word embeddings of matching hidden dimension $D$. These representations are concatenated and sequentially transformed by a shared stack of transformer layers, yielding deeply contextualized joint modality features.

Atop the main backbone, a lightweight classification and regression head operates at every reasoning step $t$. This head determines whether to (1) emit additional reasoning tokens, (2) invoke a zoom-in tool by predicting a bounding box $b_t$ and coarse class label $l_t$, or (3) terminate and output the final answer. The agent’s state at each timestep is formalized as:

$$s_t = \bigl\{(I_0,T_0), (I_1,T_1), \dots,(I_t,T_t)\bigr\},$$

where $I_0$ is the merged panoramic multi-view image, $T_0$ is the task prompt, and $(I_k,T_k)$ for $k \ge 1$ are successively cropped regions and their associated reasoning traces.

2. Interleaved Multi-modal Chain-of-Thought (iMCoT) Mechanism

The iMCoT paradigm enables iterative alternation between language-based reasoning and active visual perception, thereby embedding perception as an intrinsic part of the chain-of-thought process. At each reasoning step, the model consumes the sequence of existing cropped images and textual thoughts, and determines—via a modeled action—whether to continue linguistic reasoning, invoke the visual zoom tool (producing new attention regions), or output an answer:

• If the "zoom-in" tool is invoked, the model predicts a bounding box and a coarse class, crops the resultant region from the original panorama image, and appends the new image-text pair to the input sequence.
• Otherwise, the sequence is updated either with additional textual tokens or terminated.

This design ensures the transformer’s attention is continually refreshed with the most relevant visual evidence. The resulting end-to-end differentiability allows gradients originating from answer supervision to propagate both through the model’s perceptual localization and textual reasoning pathways.

3. Reinforcement-driven Visual Grounding: The Clip-GRPO Algorithm

To autonomously focus on task-relevant regions in the visual field, OmniDrive-R1 employs a policy $\pi_\theta$ trained using the Clip-GRPO algorithm. At each timestep $t$0, the policy stochastically selects an action $t$1 conditioned on state $t$2: output language, invoke a zoom-in with parameters $t$3, or terminate.

Visual grounding is reward-driven and annotation-free, operationalized by a process-based reward:

• Upon tool invocation, the model crops image $t$4 and predicts a coarse label embedding $t$5.
• Cosine similarity between $t$6 and $t$7 is computed in the CLIP embedding space:

$t$8

• A decaying sum over all zoom tool calls penalizes excessive invocation:

$t$9

where $b_t$0 and $b_t$1 is the total number of tool invocations in trajectory $b_t$2.

Joint optimization also incorporates outcome-based rewards: final answer accuracy $b_t$3, formatting $b_t$4, and a bonus $b_t$5 (if at least one tool call is made in correct outputs), weighted as:

$b_t$6

The total learning signal is $b_t$7, and the objective is maximized using Group Relative Policy Optimization (GRPO) in staged fashion.

4. Two-Stage Reinforcement Learning Pipeline

Training follows a label-free, two-stage RL approach to disentangle tool learning from domain generalization:

• Stage 1 (Tool Learning): Conducted on a subset from DeepEyes (14,452 samples) curated for scenarios benefitting from zoom-in localization. Here, Clip-GRPO is used, with both process and outcome rewards to train robust tool grounding and invocation strategies.
• Stage 2 (Domain Learning): Performed on the full DriveLMM-o1 dataset (18,507 samples), focusing on higher-level driving tasks such as risk assessment, rule adherence, and scene awareness. Stage 2 employs GRPO with outcome rewards only, teaching the model when and whether to apply perceptual tools to yield optimal final answers.

The framework is entirely annotation-free for region localization at both stages, exploiting the CLIP alignment as a proxy for correct visual focus.

5. Benchmarking and Empirical Results

Evaluation comprises both tool learning and broad domain reasoning using the DriveLMM-o1 benchmark. Main metrics include driving-related reasoning (Risk Assessment, Rule Adherence, Scene Awareness), scene detail (Relevance, Missing Details), overall reasoning score (mean of top-level categories), and MCQ answer accuracy, automatically evaluated by GPT-4o-mini.

Implementation details:

• Backbone: Qwen2.5VL-7B
• Hardware: 16 NVIDIA A800 GPUs
• Training: GRPO with 8 rollouts per sample, up to 5 tool calls

Model	Overall Reasoning	MCQ Accuracy
Qwen2.5VL-7B (zero-shot)	51.77	37.81
DriveLMM-o1 (SFT)	75.24	62.36
Agentthink (SFT+GRPO)	79.68	71.35
OmniDrive-R1	80.35	73.62

Compared to the zero-shot Qwen2.5VL-7B baseline, OmniDrive-R1 achieves a $b_t$8 percentage point increase in overall reasoning score and $b_t$9 in MCQ accuracy. Gains are statistically significant at $l_t$0 across three random seeds.

6. Ablation Studies and Reward Mechanism Analysis

Ablation experiments demonstrate the necessity of each pipeline component. Removing the process-based grounding reward ("+GRPO*, no process") leads to MCQ performance degradation of $l_t$1 percentage points, underscoring its importance for guiding appropriate tool usage and maintaining grounding fidelity. Two-stage RL yields $l_t$2 points in MCQ and $l_t$3 in reasoning metrics over single-stage procedural learning.

Variant	Stage 1	Stage 2	Grounding Reward	Reasoning	MCQ
Base (no RL)	–	–	–	51.77	37.81
+SFT (DriveLMM-o1)	S	–	–	72.36	62.95
+Clip-GRPO only	R	–	Yes	74.83	65.48
+GRPO* (no process)	R	R	No	70.18	57.19
+GRPO (SFT→RL)	S	R	Yes	76.58	64.38
OmniDrive-R1 (full)	R	R	Yes	80.35	73.62

(Legend: S = SFT; R = RL)

7. Limitations and Future Research Directions

Current constraints include a focus on single-frame, short-horizon reasoning, lacking temporal modeling necessary for video-based, long-term planning. The architecture does not yet address multi-agent coordination (e.g., pedestrian intent modeling) and imposes significant computational expense due to reinforcement learning, suggesting avenues for integrating more sample-efficient or offline RL paradigms.

Planned extensions include:

• Expanding iMCoT to support multi-frame video reasoning (spatiotemporal "zoom");
• Generalizing Clip-GRPO to additional tools (e.g., lane detection, depth estimation) via process-driven rewards;
• Applying hierarchical RL for seamless integration of high-level route planning with fine-grained perceptual decisions.

These directions aim to advance towards a fully end-to-end, interpretable, and trustworthy vision-language agent for autonomous driving, capable of explicit and verifiable multi-modal reasoning steps (Zhang et al., 16 Dec 2025).