LaViPlan: Visual Path Planning for Autonomous Driving

• LaViPlan is a language-guided visual path planning framework that post-trains a Vision-Language Model using Reinforcement Learning with Verifiable Rewards to mitigate vision–language–action misalignment in autonomous driving.
• It integrates explicit textual reasoning and numeric trajectory generation by outputting both descriptive analysis and 20 image-plane coordinate points in a single planning loop.
• Empirical evaluations show that LaViPlan significantly improves Average and Final Displacement Errors, resulting in smoother, safer trajectory predictions under hazardous conditions.

Searching arXiv for LaViPlan and closely related multimodal planning papers. LaViPlan is a language-guided visual path planning framework for autonomous driving that post-trains a Vision-LLM (VLM) with Reinforcement Learning with Verifiable Rewards (RLVR) to address a vision–language–action misalignment in out-of-distribution (OOD) scenarios (Oh, 17 Jul 2025). In this formulation, the VLM is not used merely for scene captioning or high-level command generation; it is placed directly in the planning loop and outputs both explicit reasoning and a future trajectory as 20 2D image-plane coordinates. The framework is motivated by the observation that existing VLM-based driving systems may recognize hazards and generate linguistically plausible reasoning while still producing trajectories that are unsafe or inconsistent with that reasoning, especially under road construction, unusual obstacles, or adverse weather (Oh, 17 Jul 2025). Within the broader multimodal planning literature, LaViPlan is part of a trend toward coupling visual grounding, language reasoning, and action generation more tightly, alongside related efforts in VLM-based trajectory planning and multimodal planning architectures such as LLaViDA, Vaiage, LLaViT, and LaVi (Liu et al., 20 Dec 2025, Liu et al., 16 May 2025, Kim et al., 13 Nov 2025, Yue et al., 20 Jun 2025).

1. Conceptual scope and planning problem

LaViPlan is defined for visual path planning in autonomous driving, with particular emphasis on OOD scenarios that deviate from the training domain and can induce hazardous behavior from planners trained purely by supervised learning (Oh, 17 Jul 2025). The motivating examples include road construction, unusual obstacles, and adverse weather. In such cases, conventional end-to-end planners may correctly perceive the scene yet still generate inappropriate trajectories.

The framework is situated against two common VLM usage patterns in driving. In one pattern, VLMs generate only high-level commands such as “go straight” or “turn right,” leaving trajectory realization to a separate planner. In the other, VLMs directly output trajectories or low-level controls but with limited explicit reasoning. LaViPlan identifies deficiencies in both patterns: heuristic command-to-trajectory mappings can fail in OOD cases, and direct trajectory outputs may be semantically inconsistent with the scene (Oh, 17 Jul 2025).

The central diagnosis is a vision–language–action misalignment. According to the formulation, a VLM can describe the scene well, detect hazards such as cones, workers, or blocked lanes, and even produce plausible reasoning about those hazards, while still outputting a trajectory that is not consistent with its own reasoning—for example, continuing straight through a blocked lane (Oh, 17 Jul 2025). The paper attributes this to supervised fine-tuning that aligns textual and trajectory labels separately rather than directly aligning the model to planning performance.

A plausible implication is that LaViPlan should be understood less as a generic multimodal assistant and more as a task-grounded post-training paradigm for planning-centric VLM agents. This interpretation is consistent with related work that seeks to make language-vision systems generate trajectories within a unified generative process, as in LLaViDA’s formulation of trajectory planning as explicit vision-language reasoning (Liu et al., 20 Dec 2025).

2. System formulation and output representation

LaViPlan uses a two-phase training framework. Phase 1 is Supervised Fine-Tuning (SFT), in which a pretrained VLM is fine-tuned on image–instruction–trajectory triplets. Phase 2 is RLVR-based Reinforcement Fine-Tuning (RFT), in which the same VLM is treated as a policy and optimized with verifiable rewards based on planning-oriented metrics and response formatting (Oh, 17 Jul 2025).

The input consists of a single image, from either the ROADWork dataset or CODA-LM, together with a language instruction prompt specifying the task: predict the ego vehicle’s future trajectory (Oh, 17 Jul 2025). The output is constrained to a fixed structure with a reasoning segment and an answer segment:

$o$0

Within this schema, > contains free-form textual reasoning about scene structure, hazards, and lane configuration, while <answer> contains a sequence of 20 image-plane coordinates $(x_i, y_i)$ that parameterize the planned trajectory (Oh, 17 Jul 2025). The VLM itself is the planner: there is no separate classical motion planner, and there is no low-level controller modeled in the system. The benchmark focus is trajectory prediction quality rather than vehicle dynamics.

This coupling of explicit reasoning and numeric trajectory generation places LaViPlan near a broader class of autoregressive multimodal planners. LLaViDA, for example, similarly uses a single VLM backbone to output a chain-of-thought, a discrete meta-action, and a numerically precise ego trajectory, though in bird’s-eye-view coordinates rather than image-plane coordinates (Liu et al., 20 Dec 2025). The difference is that LaViPlan’s design centers specifically on RLVR with ADE/FDE-based verifiable rewards.

3. RLVR and the alignment mechanism

LaViPlan’s RLVR objective maximizes expected reward while penalizing divergence from a reference policy, namely the frozen SFT model (Oh, 17 Jul 2025). If $q$ denotes the input query and $o$ the generated output, the objective is:

$\max_{\pi_\theta} \; \mathbb{E}_{o \sim \pi_\theta(q)} \left[ R_{\text{RLVR}(q, o) \right] = \left[ R(q, o) - \beta \, \mathrm{KL} \left( \pi_\theta(o \mid q) \,\|\, \pi_{\text{ref}(o \mid q) \right) \right]$

The policy is the VLM’s autoregressive token distribution, the action at each step is the next token, and the reward is sparse, being computed only after generation of the full output sequence (Oh, 17 Jul 2025). Optimization is implemented with Group Relative Policy Optimization (GRPO), a PPO variant with group-relative advantages.

The planning-oriented verifiable reward is defined from the ground-truth trajectory $T = \{p_1, \dots, p_N\}$ and the predicted trajectory $\hat{T} = \{\hat{p}_1, \dots, \hat{p}_N\}$, where the points are 2D image-plane coordinates. The planning reward is

$R_{\text{planning} = -\log \left( 1 + \frac{1}{N} \sum_{i=1}^{N} \left\| \hat{p}_i - p_i \right\|_2 \right) -\log \left( 1 + \left\| \hat{p}_N - p_N \right\|_2 \right)$

with the first term corresponding to Average Displacement Error (ADE) and the second to Final Displacement Error (FDE) (Oh, 17 Jul 2025). The logarithmic form smooths large errors and stabilizes the reward scale. A separate formatting reward enforces correct use of <think>... and <answer>...</answer> tags and a correct JSON-like list of 20 coordinate pairs. The total reward is

$R = R_{\text{format} + R_{\text{planning}$

This design is notable because the reward is verifiable: it depends only on measurable distances between numeric outputs and known ground-truth trajectories rather than on human preference judgments (Oh, 17 Jul 2025). The paper explicitly contrasts this with reward-model-based paradigms and argues that rule-based verification avoids a class of reward-model pathologies. This suggests that LaViPlan’s alignment strategy is primarily functional rather than semantic: the model is not rewarded for producing human-like explanations, but for trajectories that score well on planning metrics while remaining parseable.

GRPO uses groups of sampled responses per query, with group-relative advantages normalized by the mean and standard deviation of rewards in the group (Oh, 17 Jul 2025). Because tokens in both <think> and <answer> contribute to log-probabilities, policy gradients shape reasoning and trajectory generation jointly. The paper argues that the shared generation head is architecturally significant: improving trajectory tokens by RL necessarily affects the preceding reasoning tokens, yielding a shift toward planning-oriented reasoning.

4. Data, training procedure, and implementation details

LaViPlan is implemented on top of existing multimodal LLMs. The baseline architectures evaluated are Qwen2VL-2B, Qwen2VL-7B, and LLaMA3.2-Vision-11B (Oh, 17 Jul 2025). Each VLM comprises a vision encoder that produces image tokens, a LLM that receives text tokens plus projected visual tokens, and a unified output vocabulary used for both reasoning and trajectory segments.

The datasets are divided by role. ROADWork is the principal training and quantitative evaluation dataset. It is an OOD benchmark focused on road construction and contains 5,430 samples with image, scene description, and trajectory list. The SFT phase uses 4,344 samples, while the RFT phase uses 1,086 samples, approximately 20% of the total (Oh, 17 Jul 2025). ROADWork is also partitioned into Easy and Hard subsets based on the variance of x-coordinates in trajectories, with low lateral variance corresponding to straight paths and high lateral variance to curves or turns. CODA-LM extends CODA with natural language captions and is used only for qualitative OOD evaluation, without ground-truth trajectories.

The SFT objective is supervised next-token prediction over the concatenated reasoning and trajectory tokens. Hyperparameters for SFT include a global batch of 128, batch per GPU of 4, LoRA rank 64, LoRA $\alpha$ of 16, one epoch, learning rate $1 \times 10^{-5}$, and weight decay 0.1 (Oh, 17 Jul 2025). The RFT phase uses RLVR/GRPO with max response length 1024, batch size 128, GRPO mini-batch size 4, KL loss coefficient $q$0, group size $q$1, learning rate $q$2, top-$q$3 0.95, temperature 1.2, and repetition penalty 1.2. The SFT model is frozen as the reference policy.

The paper also provides a natural-language pseudocode description of dataset preprocessing and training. Samples are sorted by x-coordinate variance, split into turning and straight subsets, and then allocated to SFT and RFT partitions. The RL stage proceeds by sampling groups of responses, parsing the <answer> field into trajectories, computing ADE and FDE against ground truth, computing the log-based planning reward and formatting reward, normalizing rewards within groups, and then updating the policy with the GRPO objective (Oh, 17 Jul 2025).

A plausible implication is that LaViPlan’s contribution is not a new VLM architecture but a post-training method and task formulation. In this respect it differs from LLaViT, which modifies the transformer internals by introducing modality-specific QKV projections, bidirectional visual attention, and local-plus-global visual features to treat the LLM as an extended Vision Transformer (Kim et al., 13 Nov 2025), and from LaVi, which injects visual information through internal feature modulation of LayerNorm rather than visual token concatenation (Yue et al., 20 Jun 2025).

5. Empirical results and reasoning behavior

LaViPlan is evaluated primarily with ADE and FDE, computed as Euclidean distances in the image plane and expressed in tables as percentages normalized by image resolution (Oh, 17 Jul 2025). Evaluation is reported separately for Easy and Hard subsets.

The following table summarizes the key ROADWork results reported for the strongest planning numbers described in the main text.

Model setting	ADE	FDE
Qwen2VL-2B baseline (no SFT)	$q$4	$q$5
Qwen2VL-2B SFT, Easy / Hard	4.52 / 5.66	4.46 / 6.46
LaViPlan (SFT + RLVR), Easy / Hard	3.62 / 4.83	3.85 / 6.09

Relative to SFT, LaViPlan improves ADE by $q$6 on Easy and $q$7 on Hard, and improves FDE by $q$8 on Easy and $q$9 on Hard (Oh, 17 Jul 2025). Notably, these gains are achieved even though the LaViPlan setting uses fewer supervised samples for SFT than the pure SFT baselines, because only 4,344 of 5,430 ROADWork samples are used in the SFT phase before RLVR post-training.

The linguistic evaluation of reasoning shows a contrasting pattern. BERTScore drops from 0.748 in SFT to 0.712 in RFT, a decrease of 4.81%. Natural language inference label distributions against ground-truth captions shift from 44.8% to 37.7% entailment, 25.3% to 29.1% neutral, and 29.9% to 33.1% contradiction (Oh, 17 Jul 2025). The paper interprets this as evidence that RLVR moves the model away from linguistic mimicry and toward planning-oriented abstraction.

Qualitative analyses on CODA-LM support that interpretation. In construction zones, blocked-lane scenes, adverse weather, poor lighting, and scenes with unexpected pedestrians or obstacles, SFT often yields overly linear, risk-prone trajectories, whereas RLVR-fine-tuned LaViPlan produces smoother, safer, context-aware trajectories that deviate around cones, barriers, and vehicles (Oh, 17 Jul 2025). The reasoning style also changes: SFT outputs are described as verbose and human-like, whereas RFT outputs become more concise and hazard-centric, emphasizing cones, barriers, work vehicles, and blocked lanes.

This change in reasoning behavior is not treated as a defect by the authors. Rather, it is presented as a functional realignment: the reasoning channel is retained, but its content is no longer optimized for similarity to human captions. This interpretation parallels a broader theme in VLM planners such as LLaViDA, where chain-of-thought is explicitly trained to be predictive of action and trajectory quality rather than merely descriptive (Liu et al., 20 Dec 2025).

6. Ablations, limitations, and relation to adjacent systems

Several ablations clarify the conditions under which RLVR is effective. First, SFT versus SFT+RFT comparisons show that RLVR consistently improves ADE and FDE over supervised training alone (Oh, 17 Jul 2025). Second, when the total number of RFT samples is fixed at 1,086 but the easy:hard ratio is varied, a 6:4 allocation yields the best overall results, improving ADE and FDE in both easy and hard subsets. The paper concludes that including more difficult trajectories in RLVR produces richer gradients and better generalization. Third, reasoning ablations show that including <think> during SFT gives modest but consistent gains, and that RLVR with reasoning yields the best performance across all subsets, with larger gains on hard cases (Oh, 17 Jul 2025).

The framework’s limitations are explicitly stated. The reward is sparse because it is computed only after full sequence generation. ADE and FDE do not fully capture semantic safety, such as traffic-rule compliance or right-of-way. The system is evaluated in a prediction setting rather than in closed-loop simulation or in real vehicles. Trajectories are represented in image coordinates rather than metric world space. The method assumes single-image input, uses no temporal sequence or multi-view sensors, assumes ground-truth trajectories are safe and optimal, and models driving rules only implicitly through data (Oh, 17 Jul 2025).

These limitations define the boundary between LaViPlan and later multimodal planning work. LLaViDA, for example, addresses trajectory planning with a VLM that consumes six camera views plus structured ego and object states and produces object motion prediction, semantic reasoning, meta-action, and a 3-second bird’s-eye-view trajectory, achieving an average L2 trajectory error of 0.31 m and collision rate of 0.10% on the nuScenes test set after a two-stage pipeline of supervised fine-tuning and Trajectory Preference Optimization (Liu et al., 20 Dec 2025). Relative to LaViPlan, this suggests one possible progression from image-plane trajectory generation with RLVR toward more structured, state-rich, open-loop planning.

Other related systems illustrate different aspects of multimodal planning. Vaiage is a graph-structured, multi-agent travel planner that uses LLMs as both recommenders and sequential planners, grounded through external APIs, and attains an average GPT-4 rating of 8.5 out of 10 versus 7.2 for a no-strategy variant and 6.8 for a no-external-API variant (Liu et al., 16 May 2025). Although it targets travel planning rather than driving, it demonstrates a broader “LaViPlan-style” pattern in which grounding, strategy, and planning are separated across coordinated modules. LLaViT, by contrast, is relevant as an architectural argument that the LLM should build rich internal visual state through modality-specific QKV, bidirectional attention among visual tokens, and local-plus-global feature integration (Kim et al., 13 Nov 2025). LaVi contributes an efficiency-oriented alternative, replacing visual token concatenation with vision-conditioned internal feature modulation in LayerNorm and reducing FLOPs by 94.0% relative to LLaVA-OV-7B while improving inference speed by 3.1 times (Yue et al., 20 Jun 2025).

Taken together, these neighboring works suggest that “LaViPlan” can denote more than a single method. In the narrow sense, it is the autonomous-driving RLVR framework introduced in 2025 (Oh, 17 Jul 2025). In a broader methodological sense, it names a family of multimodal planning designs that attempt to align visual understanding, language reasoning, and action generation through either post-training objectives, architectural changes, or multi-agent orchestration. A plausible implication is that future extensions of LaViPlan would combine the task-grounded RLVR principle with richer visual state construction, multi-view or temporal perception, and evaluation metrics that better reflect semantic safety.