---
title: 'OR-C2P: Off-Road Caption-to-Plan Benchmark'
url: https://www.emergentmind.com/topics/or-c2p-benchmark
type: topic
---

# OR-C2P: Off-Road Caption-to-Plan Benchmark

The OR-C2P Benchmark is a multimodal off-road benchmark introduced with "Wild-Drive: Off-Road Scene Captioning and Path Planning via Robust Multi-modal Routing and Efficient Large Language Model" [2603.00694]. OR-C2P stands for **Off-Road Caption-to-Plan** and is designed to jointly train and evaluate two coupled tasks: structured off-road scene captioning and downstream path planning. Its defining emphasis is robustness to naturally degraded sensing and explicit single-modality failures in unstructured environments such as rain, fog, snow, and darkness, where urban driving priors—lane markings, traffic rules, and stable sensor quality—do not reliably apply [2603.00694].

## 1. Definition, scope, and terminological boundaries

In [2603.00694], OR-C2P denotes a benchmark for **structured, task-oriented off-road scene captioning together with downstream path planning**, built to evaluate robustness under adverse environmental conditions and modality degradation. The benchmark couples interpretable language outputs with a concrete local trajectory prediction task rather than treating captioning as an isolated vision-language problem [2603.00694].

The acronym is not globally unique. A different 2026 paper, "Concept-to-Pixel: Prompt-Free Universal Medical Image Segmentation," concerns a universal medical segmentation framework called C2P and evaluates it on a unified medical benchmark, but that paper itself is not the off-road OR-C2P benchmark discussed here [2603.17746]. The OCT registration paper "Non-rigid Point Cloud Registration for Middle Ear Diagnostics with Endoscopic Optical Coherence Tomography" explicitly states that it **does not explicitly define or name a benchmark called “OR-C2P”** [2304.13618]. A separate MaxSAT benchmark on weighted AND/OR graphs is also described under the OR-C2P shorthand in the provided literature, but it targets identification of security-critical cyber-physical components rather than multimodal driving [1911.00516]. Within current off-road multimodal driving literature, the benchmark name refers specifically to the Off-Road Caption-to-Plan setting of [2603.00694].

## 2. Coupled task design and structured outputs

OR-C2P is organized around two coupled outputs: a **structured caption** and an **ego-frame future trajectory**. The captioning side uses concise, closed-set templates generated from five task heads, with canonical labels intended to improve robustness and reproducibility for efficient LLMs [2603.00694].

- **Weather and illumination**: Weather $\in \{\text{sunny, cloudy, rainy, snowy, foggy}\}$ and illumination $\in \{\text{bright\_light, daylight, twilight, darkness}\}$.
- **Drivable area**: Availability $\in \{\text{clear, partially\_blocked, blocked}\}$ and direction $\in \{\text{front, front\_left, front\_right, left, right}\}$.
- **Terrain traversability**: Terrain type $\in \{\text{dirt, gravel, grass, mud, sand, snow, rock}\}$; the paper lists these as examples, while the dataset summary states **10 terrain types**. Difficulty $\in \{\text{easy, moderate, hard, impassable}\}$.
- **Obstacle detection**: Category $\in \{\text{vehicle, pedestrian, animal, rock, tree, pole, building, unknown}\}$, with direction $\in \{\text{front, front\_left, front\_right, left, right}\}$ and distance $\in \{\text{near, mid, far}\}$.
- **Driving suggestion**: Action $\in \{\text{go\_straight, turn\_left, turn\_right, stop}\}$.

The discrete driving suggestion is derived by clustering future **10 s** ego-frame trajectories with K-means:

$$
\min_{\mu_1 \ldots \mu_K, a_1 \ldots a_N} \sum_{i=1}^N \| v_i - \mu_{a_i} \|_2^2
$$

where $v_i$ is the vectorized future trajectory of sample $i$ in the ego frame [2603.00694].

Path planning is supervised using trajectories built by **registering LiDAR with KISS-ICP** and then **smoothing with B-spline interpolation**. Future targets are defined at $t \in \{1, 2, 5, 10\}$ seconds from the current timestamp. The model-side planning interface uses a special planning token whose embedding conditions a GRU decoder:

$$
y_t = \mathrm{GRU}(x_p, y_{t-1}; \theta_{\mathrm{GRU}})
$$

where $x_p$ is the learned embedding of the planning token [2603.00694].

This task design ties semantics directly to control-relevant outputs. A plausible implication is that OR-C2P evaluates not only descriptive fidelity but also whether compact language abstractions remain aligned with local navigation behavior under sensing degradation.

## 3. Dataset composition, sensing modalities, and annotation pipeline

OR-C2P is built upon **ORAD-3D** and curates aligned LiDAR-camera samples with caption and planning supervision. The benchmark uses synchronized **monocular RGB** and **LiDAR** as its core modalities. RGB resolution is **1280×720**, and the LiDAR-camera pairs are calibrated and temporally aligned; the paper reports **57K aligned LiDAR–camera pairs** and also gives a table total of **57,808 frames** [2603.00694].

| Split | Sequences | Frames |
|---|---:|---:|
| Train | 100 | 39,727 |
| Val | 15 | 5,717 |
| Test | 29 | 12,164 |

The table reports **144 sequences / 57,808 frames** overall, while the narrative elsewhere mentions **145 sequences** and “57K” pairs [2603.00694]. The benchmark also reports **19,527 frames with obstacles**, **5 weather types**, **4 illumination types**, and **10 terrain types**. Collection spans **China from spring to winter**, with each sequence covering roughly **100 meters of off-road terrain**, including forest- and trail-like areas, mud, snow, gravel, grass, rock, and low-light or night scenes [2603.00694].

GNSS ground truth from the underlying ORAD-3D is available and used as reference, while KISS-ICP is additionally used to derive smooth pose sequences for trajectory supervision [2603.00694].

The annotation pipeline is structured rather than free-form. **Qwen2.5-VL-72B** is used to produce initial supervision through concise fixed-vocabulary Q&A templates, after which outputs are post-processed into canonical labels. Weather and terrain traversability are generated as **camera-based pseudo-labels**. Drivable area and obstacle detection are obtained by **image-first inference**, then refined with **calibrated LiDAR point clouds through clustering and filtering**. The paper reports **random human verification** for quality assurance [2603.00694].

## 4. Robustness protocol and evaluation methodology

A central design goal of OR-C2P is evaluation under naturally degraded sensing rather than purely synthetic corruption regimes. The benchmark includes **rain, snow, fog, and low-light/night scenes** collected from real-world off-road environments. It does **not** define artificial corruption severity levels [2603.00694].

Missing-modality robustness is explicitly built into the protocol. During training, inputs are randomly set to **camera-only**, **LiDAR-only**, or **dual-modality** with equal probability:

- **Camera-only**: probability $1/3$
- **LiDAR-only**: probability $1/3$
- **Dual-modality**: probability $1/3$

This is intended to induce robust routing and fusion under single-modality failures [2603.00694]. Additional sensing such as RADAR, thermal, event cameras, and IMU is **not part of the benchmark as described**.

Captioning is evaluated with **BLEU-1**, **BLEU-2**, **BLEU-4**, and **BERTScore**, reported in the paper as **BERT-P**. The BLEU definition is

$$
\mathrm{BLEU}_N = \mathrm{BP} \cdot \exp\!\left(\sum_{n=1}^{N} w_n \log p_n\right), \quad \sum w_n = 1,
$$

with brevity penalty

$$
\mathrm{BP} =
\begin{cases}
1, & c > r \\
\exp(1-r/c), & \text{otherwise}
\end{cases}
$$

where $p_n$ is clipped $n$-gram precision, $c$ is candidate length, and $r$ is reference length [2603.00694].

For BERTScore, the paper reports precision, while the standard formulation is

$$
P = \frac{1}{|X|} \sum_{x \in X} \max_{y \in Y} \cos(e(x), e(y)),
$$

$$
R = \frac{1}{|Y|} \sum_{y \in Y} \max_{x \in X} \cos(e(y), e(x)),
$$

$$
F1 = \frac{2PR}{P+R}.
$$

Planning is evaluated with **FDE** and **minADE**. The paper defines FDE at the final horizon as

$$
\mathrm{FDE} = \| s_{T-1} - s^*_{T-1} \|_2^2,
$$

and gives the multimodal minADE metric as

$$
\mathrm{minADE} = \min_{m \in \{1 \ldots M\}} \frac{1}{T} \sum_{t=1}^{T} \| y_t - \hat{y}^{(m)}_t \|_2.
$$

Evaluation is carried out on the provided train/val/test split, with results reported on the test set. Random seeds, repeated runs, and confidence intervals are **not reported**; the published values are single-run means [2603.00694].

## 5. Baselines and reported benchmark results

OR-C2P is evaluated against several baseline families. Captioning baselines include **MiniGPT-4**, **LLaVA-1.5**, **Instruct-BLIP**, **LLaMA-AdapterV2**, **LiDAR-LLM**, **BEV-LLM-1B**, and **BEV-LLM-8B**. Planning baselines include **BEV-LLM-GRU**, **TopoPath**, **CoverNet**, **MTP**, and **MultiPath** [2603.00694].

For captioning, the benchmark reports that **Wild-Drive-3B** attains the strongest scores among the listed methods, with **BLEU-1 71.72**, **BLEU-2 60.79**, **BLEU-4 49.26**, and **BERT-P 98.13** [2603.00694].

| Method | BLEU-4 | BERT-P |
|---|---:|---:|
| Wild-Drive-3B | 49.26 | 98.13 |
| Wild-Drive-0.5B | 47.21 | 95.79 |
| BEV-LLM-8B | 47.93 | 88.17 |
| LiDAR-LLM | 46.11 | 87.49 |
| Instruct-BLIP | 10.29 | — |

The reported trend is that robust multimodal fusion and routing improve both lexical and semantic captioning metrics relative to larger but fixed-fusion LLM systems [2603.00694].

Planning results show a more differentiated picture. **Wild-Drive** outperforms the LLM-based planning baselines and several classic multimodal predictors, but **TopoPath** remains the strongest dedicated planner with lower error [2603.00694].

| Method | FDE | minADE |
|---|---:|---:|
| TopoPath | 0.92 | 0.43 |
| Wild-Drive | 1.09 | 0.66 |
| BEV-LLM-GRU | 1.33 | 0.97 |
| MTP | 1.59 | 0.84 |
| MultiPath | 1.56 | 0.92 |
| CoverNet | 2.47 | 1.31 |

This benchmark outcome is important because it prevents a common overgeneralization: OR-C2P does not show that an LLM-centric system is universally best at planning. Rather, it shows that the benchmark can discriminate between **caption quality**, **LLM-based multimodal robustness**, and **specialized planning performance**.

## 6. Reference architecture, interpretive significance, and limitations

OR-C2P is closely tied to the Wild-Drive reference system, and the benchmark’s structure makes its intended algorithmic stresses explicit. Wild-Drive uses a **Modality-Routing Transformer (MoRo-Former)** with **$T=5$ task groups** and **$K=64$ queries per task**, yielding **320 learnable queries total**, fewer than **BEV-LLM (512)** and **LiDAR-LLM (576)**. RGB is encoded with **frozen DINOv3 ViT-S/16**, LiDAR BEV with **VoxelNet**, and the LLM backbone is **Qwen2.5-0.5B-Instruct** or **Qwen2.5-3B-Instruct** with **LoRA, rank = alpha = 16** [2603.00694].

The routing mechanism predicts branch assignment over **LiDAR-only**, **camera-only**, and **dual-modality** experts; hard routing is applied at inference. The multimodal output is then used both for structured text generation and for the GRU-based trajectory head. This coupling indicates that OR-C2P is not merely a dataset of paired captions and trajectories; it is a benchmark for **modality-aware caption-to-plan transfer** [2603.00694].

Ablations reinforce that interpretation. Full Wild-Drive achieves **BLEU-1/BLEU-4/BERT-P = 71.72 / 49.26 / 98.13**. Replacing MoRo-Former with Q-Former over **DINOv3 only** yields **63.41 / 44.59 / 86.24**; over **VoxelNet only**, **60.39 / 41.72 / 83.51**; and over **DINOv3+VoxelNet**, **66.79 / 45.92 / 92.74**. The paper summarizes the effect as modality-routing and token compression outperforming uniform query bridging by approximately **5.4% on BLEU/BERTScore** [2603.00694]. Qualitative analysis further reports that queries route preferentially toward LiDAR at night or in low light, whereas routing shifts toward camera or fusion when LiDAR becomes sparse or corrupted.

The benchmark also has explicit scope limits. It focuses on **camera–LiDAR** sensing and does not include RADAR, thermal, event cameras, or IMU. It uses **naturally degraded sensing** and **missing-modality settings**, but does not define synthetic corruption severities. The paper states that the **code and benchmark will be publicly available**, yet **license and detailed download instructions are not specified**, and **evaluation scripts are not explicitly detailed** [2603.00694].

Relative to urban benchmarks, OR-C2P emphasizes **closed-set, safety-centric captions tied to planning**, **ego-frame short-horizon trajectory prediction**, and **robust multimodal routing rather than fixed fusion** [2603.00694]. This makes it a focused benchmark for interpretable off-road perception-to-planning rather than a general-purpose autonomous driving benchmark.

Source: https://www.emergentmind.com/topics/or-c2p-benchmark