Wild-Drive: Autonomous Off-Road Driving

• Wild-Drive is an off-road autonomous driving framework that integrates multimodal sensor inputs and structured scene captioning to guide decision-making in unstructured, adverse environments.
• It leverages the MoRo-Former for adaptive modality routing between camera and LiDAR, ensuring robust performance under poor weather, low-light, and sensor noise conditions.
• Empirical evaluations on the OR-C2P benchmark demonstrate that Wild-Drive enhances scene captioning accuracy and trajectory planning efficiency compared to existing models.

Wild-Drive is an off-road autonomous driving framework that jointly performs structured scene captioning and path planning under degraded sensing conditions. It targets unstructured environments such as forests, fields, dirt roads, and construction areas, where there are no clear lanes, sparse traffic rules, and frequent adverse conditions including rain, fog, snow, darkness, and sensor noise. The system combines synchronized camera and LiDAR inputs, a task-conditioned modality-routing bridge called MoRo-Former, an efficient LLM, a planning token, and a GRU-based trajectory decoder to produce both structured natural-language scene descriptions and future trajectories. It is introduced together with the OR-C2P Benchmark, which covers structured off-road scene captioning and path planning under diverse sensor corruption conditions (Wang et al., 28 Feb 2026).

1. Problem formulation and structured off-road semantics

Wild-Drive is motivated by two coupled requirements in off-road autonomy: robustness under single-modality degradation and transparent decision-making. In contrast to structured urban driving, off-road navigation lacks lane markings, traffic lights, and standardized signage, and planning relies more heavily on terrain, traversability, and obstacles in natural scenes. The framework therefore treats scene captioning not as generic description generation but as a structured intermediate representation optimized for safety-critical planning (Wang et al., 28 Feb 2026).

The captioning interface is organized as templated question answering over five tasks. These tasks define a compact ontology for environmental conditions, drivable space, terrain, obstacles, and high-level action.

Task	Predicted fields	Representative vocabulary
Weather description	Weather, illumination	sunny, cloudy, rainy, snowy, foggy; bright light, daylight, twilight, darkness
Drivable area	Availability, free-space direction	clear, partially_blocked, blocked; front, front_left, front_right, left, right
Terrain traversability	Terrain, difficulty	dirt, gravel, grass, mud, sand, snow, rock; easy, moderate, hard, impassable
Obstacle detection	Category, direction, distance	vehicle, pedestrian, animal, rock, tree, pole, building, unknown; near, mid, far
Driving suggestion	High-level action	go_straight, turn_left, turn_right, stop

This representation directly couples perception and action. Weather and illumination summarize sensing conditions; drivable area and traversability encode local feasibility; obstacle fields expose the proximate cause of avoidance behavior; and driving suggestion provides a discrete intent prior. A plausible implication is that Wild-Drive treats language as a control-relevant latent interface rather than a post hoc explanation channel.

2. Multimodal architecture and MoRo-Former

Wild-Drive consumes a monocular RGB image $\mathbf{I}_c \in \mathbb{R}^{H \times W \times C}$ and a LiDAR point cloud $\mathbf{P}_l \in \mathbb{R}^{N \times 4}$. LiDAR is processed by VoxelNet followed by a BEV CNN to produce a BEV feature map $$\mathbf{F}_l \in \mathbb{R}^{H_l \times W_l \times D}$$, while the image is encoded by DINOv3 ViT-S/16 into $$\mathbf{F}_c \in \mathbb{R}^{H_c \times W_c \times D}$$. After flattening, the two streams are concatenated into

$$\mathbf{F}'_{lc} = \mathrm{Concat}\left(\mathbf{F}'_l, \mathbf{F}'_c\right),$$

which forms the shared token space for downstream routing and language conditioning (Wang et al., 28 Feb 2026).

The central bridge module, MoRo-Former, is designed to replace static multimodal fusion with task-conditioned routing. It maintains $T=5$ tasks and $K=64$ queries per task, giving $320$ learnable queries in total. Task identity is injected through a group embedding,

$$\tilde{\mathbf{q}}_{t,k} = \mathbf{q}_{t,k} + \mathbf{E}_g[t],$$

to reduce inter-task interference. For traversability, obstacles, and driving suggestion, each query is also associated with a 3D reference point and a locality-aware binary mask over LiDAR and camera tokens, restricting cross-attention to spatially relevant neighborhoods (Wang et al., 28 Feb 2026).

MoRo-Former predicts routing probabilities

$\mathbf{p}_i = [p_{i,l}, p_{i,c}, p_{i,lc}],$

for LiDAR, camera, and fusion experts, and uses hard routing at inference: $\mathbf{P}_l \in \mathbb{R}^{N \times 4}$0 Expert-specific cross-attention produces modality-specialized task tokens, which are then compressed and projected into compact task representations. During training, sensor dropout with probability $\mathbf{P}_l \in \mathbb{R}^{N \times 4}$1 exposes the router to camera-only, LiDAR-only, and dual-modality cases, with routing labels $\mathbf{P}_l \in \mathbb{R}^{N \times 4}$2, $\mathbf{P}_l \in \mathbb{R}^{N \times 4}$3, or $\mathbf{P}_l \in \mathbb{R}^{N \times 4}$4 respectively. This makes modality selection explicitly corruption-aware rather than implicitly learned through end-task loss alone (Wang et al., 28 Feb 2026).

The architectural significance of MoRo-Former lies in its selective fusion policy. Weather tends to be camera-dominant, whereas traversability and obstacle reasoning can shift toward LiDAR under low-light conditions. This suggests that Wild-Drive is organized around task-specific sensor reliability rather than globally shared fusion weights.

3. Language interface, planning token, and trajectory decoding

Wild-Drive uses Qwen2.5-Instruct as the LLM backbone, with Qwen2.5-0.5B and Qwen2.5-3B reported as the principal model scales. Fine-tuning is performed with LoRA, with rank and alpha both set to $\mathbf{P}_l \in \mathbb{R}^{N \times 4}$5, while the base weights remain frozen. The LLM input consists of instruction tokens $\mathbf{P}_l \in \mathbb{R}^{N \times 4}$6, compact task tokens $\mathbf{P}_l \in \mathbb{R}^{N \times 4}$7 produced by MoRo-Former, and a special planning token embedding $\mathbf{P}_l \in \mathbb{R}^{N \times 4}$8. Conditioned generation is written as

$\mathbf{P}_l \in \mathbb{R}^{N \times 4}$9

where $$\mathbf{F}_l \in \mathbb{R}^{H_l \times W_l \times D}$$0 is the structured answer sequence (Wang et al., 28 Feb 2026).

The planning token is the explicit interface between language and motion. After LLM processing, $$\mathbf{F}_l \in \mathbb{R}^{H_l \times W_l \times D}$$1 acts as a condensed planning latent summarizing weather, terrain difficulty, drivable free space, obstacle positions, and high-level suggestion. Trajectory generation is then performed by a GRU decoder: $$\mathbf{F}_l \in \mathbb{R}^{H_l \times W_l \times D}$$2 where $$\mathbf{F}_l \in \mathbb{R}^{H_l \times W_l \times D}$$3 is the waypoint at time step $$\mathbf{F}_l \in \mathbb{R}^{H_l \times W_l \times D}$$4. The model predicts future waypoints at fixed horizons, and B-spline interpolation can be applied to obtain a smooth trajectory (Wang et al., 28 Feb 2026).

This design makes explanation and planning structurally coupled. Caption tokens are not merely diagnostic outputs; they are part of the latent computational pathway that conditions trajectory prediction. The system’s quoted examples—such as reporting rainy twilight conditions, partially blocked free space in front_right, grass terrain with moderate difficulty, a tree at front_left and near distance, and then suggesting turn_right—illustrate that captions form a structured safety report aligned with the generated motion (Wang et al., 28 Feb 2026).

4. OR-C2P benchmark and optimization protocol

The OR-C2P Benchmark is built on the ORAD-3D dataset and repurposes it for joint caption-to-plan evaluation. It contains $$\mathbf{F}_l \in \mathbb{R}^{H_l \times W_l \times D}$$5 sequences and $$\mathbf{F}_l \in \mathbb{R}^{H_l \times W_l \times D}$$6 aligned LiDAR-camera pairs, split into $$\mathbf{F}_l \in \mathbb{R}^{H_l \times W_l \times D}$$7 training sequences with $$\mathbf{F}_l \in \mathbb{R}^{H_l \times W_l \times D}$$8 frames, $$\mathbf{F}_l \in \mathbb{R}^{H_l \times W_l \times D}$$9 validation sequences with $$\mathbf{F}_c \in \mathbb{R}^{H_c \times W_c \times D}$$0 frames, and $$\mathbf{F}_c \in \mathbb{R}^{H_c \times W_c \times D}$$1 test sequences with $$\mathbf{F}_c \in \mathbb{R}^{H_c \times W_c \times D}$$2 frames. The benchmark includes $$\mathbf{F}_c \in \mathbb{R}^{H_c \times W_c \times D}$$3 weather types, $$\mathbf{F}_c \in \mathbb{R}^{H_c \times W_c \times D}$$4 illumination types, $$\mathbf{F}_c \in \mathbb{R}^{H_c \times W_c \times D}$$5 terrain types, and $$\mathbf{F}_c \in \mathbb{R}^{H_c \times W_c \times D}$$6 frames with obstacles. The scenes span off-road environments across China from spring to winter and include real rain, snow, fog, and low-light degradations rather than synthetic corruptions (Wang et al., 28 Feb 2026).

Annotations are generated by Qwen2.5-VL-72B, then post-processed and human-verified. Driving suggestion labels are derived by clustering $$\mathbf{F}_c \in \mathbb{R}^{H_c \times W_c \times D}$$7-second future trajectories with $$\mathbf{F}_c \in \mathbb{R}^{H_c \times W_c \times D}$$8-means and mapping the resulting clusters to $$\mathbf{F}_c \in \mathbb{R}^{H_c \times W_c \times D}$$9. The overall training objective combines autoregressive text loss and waypoint regression: $$\mathbf{F}'_{lc} = \mathrm{Concat}\left(\mathbf{F}'_l, \mathbf{F}'_c\right),$$0 with

$$\mathbf{F}'_{lc} = \mathrm{Concat}\left(\mathbf{F}'_l, \mathbf{F}'_c\right),$$1

and

$$\mathbf{F}'_{lc} = \mathrm{Concat}\left(\mathbf{F}'_l, \mathbf{F}'_c\right),$$2

A second training phase freezes expert decoders and supervises routing using

$$\mathbf{F}'_{lc} = \mathrm{Concat}\left(\mathbf{F}'_l, \mathbf{F}'_c\right),$$3

Implementation details reported for training include an Intel Core i9-12900K CPU and an NVIDIA RTX 4090 GPU (Wang et al., 28 Feb 2026).

The benchmark’s structure emphasizes joint evaluation of semantic interpretation and action generation under realistic off-road sensing failures. This distinguishes it from caption-only and planner-only setups, where the relationship between intermediate semantics and downstream control is often left implicit.

5. Empirical performance, robustness, and efficiency

On OR-C2P scene captioning, Wild-Drive-3B reports BLEU-1 $$\mathbf{F}'_{lc} = \mathrm{Concat}\left(\mathbf{F}'_l, \mathbf{F}'_c\right),$$4, BLEU-2 $$\mathbf{F}'_{lc} = \mathrm{Concat}\left(\mathbf{F}'_l, \mathbf{F}'_c\right),$$5, BLEU-4 $$\mathbf{F}'_{lc} = \mathrm{Concat}\left(\mathbf{F}'_l, \mathbf{F}'_c\right),$$6, and BERT-P $$\mathbf{F}'_{lc} = \mathrm{Concat}\left(\mathbf{F}'_l, \mathbf{F}'_c\right),$$7. Wild-Drive-0.5B reports BLEU-1 $$\mathbf{F}'_{lc} = \mathrm{Concat}\left(\mathbf{F}'_l, \mathbf{F}'_c\right),$$8, BLEU-2 $$\mathbf{F}'_{lc} = \mathrm{Concat}\left(\mathbf{F}'_l, \mathbf{F}'_c\right),$$9, BLEU-4 $T=5$0, and BERT-P $T=5$1. The paper compares these results to BEV-LLM-8B and LiDAR-LLM, noting that Wild-Drive-3B exceeds BEV-LLM-8B by $T=5$2 BLEU-1, $T=5$3 BLEU-4, and $T=5$4 BERT-P, and that Wild-Drive-0.5B still surpasses LiDAR-LLM and BEV-LLM in BLEU-1 and BERT-P despite a substantially smaller parameter count (Wang et al., 28 Feb 2026).

On OR-C2P path planning, Wild-Drive reports FDE $T=5$5 and minADE $T=5$6, compared with BEV-LLM-GRU at FDE $T=5$7 and minADE $T=5$8. CoverNet, MTP, and MultiPath are weaker on both metrics, while TopoPath remains stronger, with FDE $T=5$9 and minADE $K=64$0. The ablation replacing MoRo-Former with Q-Former variants produces consistent drops in BLEU and BERT-P, indicating that modality routing and token compression are central to the observed captioning gains (Wang et al., 28 Feb 2026).

Setting	Captioning or planning result	Reported comparison
Wild-Drive-3B	BLEU-1 71.72, BLEU-4 49.26, BERT-P 98.13	Higher than BEV-LLM-8B on all three reported metrics
Wild-Drive-0.5B	BLEU-1 69.72, BLEU-4 47.21, BERT-P 95.79	Surpasses LiDAR-LLM and BEV-LLM in BLEU-1 and BERT-P
Wild-Drive	FDE 1.09, minADE 0.66	Improves over BEV-LLM-GRU; trails TopoPath

The robustness claim is qualitative and architectural rather than purely benchmark-specific. Under low-light or nighttime conditions, routing probabilities shift toward the LiDAR expert for traversability and obstacle queries; under sparse or noisy LiDAR, routing shifts toward camera or fusion. The fixed-fusion BEV-LLM baseline is reported to degrade more severely under such conditions (Wang et al., 28 Feb 2026).

Efficiency is also reported explicitly. Wild-Drive-0.5B has approximately $K=64$1B parameters in total and runs at about $K=64$2 s per sample in full precision and about $K=64$3 s with $K=64$4-bit quantization on an RTX 4090. Wild-Drive-3B has approximately $K=64$5B parameters in total and runs at about $K=64$6 s per sample in full precision and about $K=64$7 s with $K=64$8-bit quantization. On an RTX 2080Ti, runtime increases by about $K=64$9 (Wang et al., 28 Feb 2026).

6. Limitations and position within broader Wild-Drive research

Wild-Drive underperforms TopoPath in FDE and minADE, which the paper interprets as evidence that specialized geometric motion models still retain an advantage for pure path accuracy. The framework also depends on structured labels generated by Qwen2.5-VL-72B, so teacher-model biases can propagate into both captioning and planning. Extreme simultaneous corruption of both camera and LiDAR, as well as very long-horizon planning, are identified as remaining open issues (Wang et al., 28 Feb 2026).

Within the broader research literature, the name “Wild-Drive” aligns with a larger technical motif rather than a single methodology. WOD-E2E uses “wild” to denote rare, safety-critical long-tail on-road events and evaluates end-to-end policies on a curated subset of Waymo fleet logs with the human-aligned Rater Feedback Score (Xu et al., 30 Oct 2025). WROOM addresses off-road navigation through end-to-end reinforcement learning with Control Barrier Functions in a domain-randomized simulator and sim-to-real deployment on a buggy RC car (Kalaria et al., 2024). DRIVE focuses on rapid, standardized motion-model data collection for UGVs across diverse terrain and slip conditions (Baril et al., 2023). LFG treats large-scale unposed driving video as a source of label-free autonomy pretraining (Strong et al., 25 Feb 2026). Streaming video systems introduce proactive interaction, long-term memory, and sub-second processing for unbounded real-world streams, capabilities that are directly relevant to driving assistants (Yao et al., 7 Jun 2026). WorldDrive couples scene generation and planning through a unified vision-and-motion latent space (Gui et al., 16 Mar 2026), while Cosmos-Drive-Dreams uses world foundation models for controllable, multi-view synthetic driving data generation and long-tail scenario expansion (Ren et al., 10 Jun 2025).

This suggests that “Wild-Drive” has become a useful umbrella for research on autonomy under unstructured, degraded, long-tail, or off-nominal conditions. In that broader sense, the Wild-Drive framework occupies the explainable off-road caption-to-plan corner of the space: it formalizes structured semantics, routes among unreliable modalities, and uses language not only as explanation but also as a planning substrate.