---
title: 'RoadBench: Benchmarking Road Damage Understanding'
url: https://www.emergentmind.com/topics/roadbench
type: topic
---

# RoadBench: Benchmarking Road Damage Understanding

Searching arXiv for the specified RoadBench papers and closely related benchmark context.
arXiv search query: 2605.20306
RoadBench is an overloaded designation in recent road-damage and multimodal evaluation literature. In arXiv usage, it denotes two distinct benchmarks with different problem formulations and data regimes. One is **WildRoadBench**, also described as **RoadBench**, a wild aerial road-damage grounding benchmark that evaluates both fixed vision-language models (VLMs) and autonomous LLM-driven agents on the same professionally annotated UAV corpus under the same detection metric [2605.20306]. The other is **RoadBench**, a large-scale multimodal benchmark for road damage understanding built from 100,000 high-resolution image–text pairs and introduced together with the RoadCLIP model [2507.17353]. The shared name reflects a common concern—road-damage understanding—but the two works differ in viewpoint, annotation structure, task design, and intended use.

## 1. Terminological scope and problem framing

The two RoadBench works occupy complementary positions in the literature. WildRoadBench targets **wild aerial road-damage grounding**: small, visually subtle defects captured by UAVs at **50–200 m altitude**, often confounded with **shadows, water stains, seams, lane markings, or asphalt texture** [2605.20306]. The multimodal RoadBench targets **comprehensive road damage understanding** through paired images and text, emphasizing contextual semantics such as severity, spatial location, surface condition, and environmental context [2507.17353].

| Variant | Core problem | Primary artifact |
|---|---|---|
| WildRoadBench | Aerial defect localization and autonomous detector development | UAV corpus with bounding boxes and dual evaluation tracks |
| RoadBench (2025) | Multimodal road damage understanding | 100,000 image–text pairs with spatial annotations and RoadCLIP |

WildRoadBench is explicitly motivated by the mismatch between conventional grounding benchmarks and aerial inspection. Smartphone road-damage corpora such as **RDD2022** are ground-level; aerial datasets such as **VisDrone** emphasize people and vehicles; and grounding benchmarks such as **RefCOCO/+/g** and **Visual Genome** do not test aerial inspection concepts [2605.20306]. By contrast, the 2025 multimodal RoadBench is positioned against vision-only road-damage datasets and models, arguing that they lack the contextual semantics used by inspectors in practice [2507.17353].

A common misconception is to treat these two resources as successive versions of one benchmark. The available evidence indicates instead that they are separate benchmarks with distinct data sources, annotation schemes, and evaluation protocols.

## 2. WildRoadBench: corpus design and aerial defect taxonomy

WildRoadBench is built from **internally collected UAV imagery over Chinese roads**, captured by **quad-rotor drones at 50–200 m altitude** under wild conditions [2605.20306]. After **MD5 de-duplication**, the corpus contains **1,061 unique images** with **1,699 ground-truth boxes** overall. The annotations were produced by **professional road-maintenance experts**, following **Chinese highway-maintenance specifications (JTG H10-2009 / 5210-2018 / 5421-2018)**. The labels are **axis-aligned bounding boxes and categories**; **masks are not provided**, and **inter-annotator agreement is not reported**.

For headline scoring, the benchmark aggregates eight fine-grained categories into five scene-level categories.

| Scene-level category | Images, boxes | Notes |
|---|---|---|
| Road water | 624 images, 786 boxes | 1.3 avg boxes per positive image |
| Road crack | 402 images, 561 boxes | 1.4 avg boxes per positive image |
| Road pothole | 145 images, 175 boxes | 1.2 avg boxes per positive image |
| Pavement debris | 48 images, 85 boxes | 1.8 avg boxes per positive image |
| Infrastructure anomaly | 83 images, 92 boxes | Groups drainage damage, drainage water, slope anomaly, guardrail damage |

The grouped **infrastructure anomaly** category contains four rare classes: **drainage damage (22, 22; 1.0)**, **drainage water (22, 24; 1.1)**, **slope anomaly (22, 26; 1.2)**, and **guardrail damage (17, 20; 1.2)** [2605.20306]. For the VLM leaderboard, the paper reports **1,394 GT boxes after protocol-specific filtering**.

Small-target prevalence is explicitly analyzed using COCO scale buckets. In the open-source VLM bucket analysis, **small nGT = 56, medium nGT = 671, large nGT = 667**, with small defined as **area $< 32\times32$ px in the evaluation frame**. No open-source model exceeds **7.1% recall on small objects** [2605.20306]. This directly supports the benchmark’s emphasis on small, subtle, and domain-shifted targets.

## 3. WildRoadBench evaluation protocols and unified metric suite

WildRoadBench applies **the same images and the same object-detection metrics** to two different routes: a **VLM Track** and an **Agent Track** [2605.20306]. The central methodological idea is direct comparability between fixed grounded inference and autonomous research-and-engineering.

The **VLM Track** is defined as **single-shot grounded VLM** evaluation. Given **one aerial image** and a **target damage scene**, a model must return all visible instances as **axis-aligned bounding boxes with confidence scores**, in **zero-shot**, **single forward pass**, with **no tools** and **no in-context examples**. The unified prompt is fixed:

```text
Detect all instances of <scene> in this aerial image.
Return the result as a JSON list. Each item should contain a bounding box in the format ([x_1, y_1, x_2, y_2]) and a confidence score.
If no instance is visible, return an empty list.
```

The benchmark uses **family-aware parsers** to normalize heterogeneous model outputs, including **generic JSON lists**, **Qwen-style label/confidence fields**, and **InternVL normalized integer coords**. If structured parsing fails, a fallback scans for **numeric quadruples that form plausible boxes**. Expected coordinates are **$[x_1, y_1, x_2, y_2]$ in pixel coordinates of the original image**; if a model emits normalized or resized coordinates, an inverse transform maps them back to the original frame before scoring [2605.20306].

The **Agent Track** is defined as **end-to-end autonomous detector development**. The agent receives a **written task specification**, the **closed class vocabulary**, the **prediction schema**, the **metric definition (mAP@0.50)**, the **submission protocol**, and a **small exploratory slice for EDA (17 labeled images)**. No training set is provided; the agent must **discover, adapt, or construct data at run time**, possibly by using **public datasets** and **pretrained weights** [2605.20306]. The interaction budget is fixed at **$T_{\max}=5$ submission attempts**, **$T_{\text{wall}}=5$ h per run**, **$\tau_{\text{idle}}=3$ consecutive empty-observation steps**, and **$T_{\text{call}}=480$ s** per tool call, with warnings at **30, 15, and 5 minutes remaining**.

The grader exposes a single submission endpoint and returns only scalar performance on the hidden holdout:
$$
\Pi(f)=\mathrm{mAP}@0.50(f(H),Y_H).
$$
The official run score is the best scalar across attempts:
$$
R^*=\max_t r_t,\qquad \theta^*=\arg\max_t r_t.
$$

Both tracks use a unified detection metric suite. With predicted box $B$ and ground-truth box $B^*$, the IoU threshold is fixed at $t=0.5$:
$$
\mathrm{IoU}(B,B^*)=\frac{|B\cap B^*|}{|B\cup B^*|}.
$$
Precision and recall are defined as
$$
\mathrm{Precision}=\frac{TP}{TP+FP},\qquad \mathrm{Recall}=\frac{TP}{TP+FN}.
$$
Average Precision is the area under the precision–recall curve:
$$
AP=\int_0^1 p(r)\,dr.
$$
Macro-averaged headline performance is
$$
\mathrm{mAP}@0.50=\frac{1}{C}\sum_{c=1}^C AP_c(t=0.5).
$$
The benchmark also reports **mAP$_{[.5:.95]}$**, **macro F1**, **micro F1**, and explicit **TP/FP/FN counts** [2605.20306].

## 4. Empirical findings from WildRoadBench

The WildRoadBench VLM study evaluates **25 models: 9 closed-source frontier and 16 open-source** [2605.20306]. The leading result is **Gemini 3 Pro** with **42.1% macro AP@0.50**, **macro F1 14.7%**, **micro F1 49.4%**, and **TP 689; FP/FN 705/705**. **Qwen 3.6 Plus** follows with **36.8% AP@0.50**, **macro F1 12.4%**, **micro F1 46.9%**, and **TP 654; FP/FN 821/740**. **Gemini 3.1 Flash Lite** reaches **25.8% AP@0.50** and **micro F1 32.8%**. The best open-source entry is **Qwen2.5-VL-32B-Instruct-AWQ** at **25.5% AP@0.50**, **macro F1 8.6%**, **micro F1 37.0%**, and **TP 484; FP/FN 736/910**. Other reported entries include **Qwen3.5-VL-27B-FP8: 24.4% AP@0.50**, **Qwen2.5-VL-7B-Instruct: 23.6%**, **GPT-5.4: 20.9%**, **GPT-4o: 9.5%**, **InternVL3-8B: 10.9%**, **LLaVA-NeXT-7B: 9.1%**, and **Kimi-VL-A3B-Instruct: 1.8%**, with the latter associated with **degenerate box behavior** [2605.20306].

Several performance patterns are explicit. Closed-source frontier models lead the leaderboard but still leave **more than half of the metric on the table**. Among open-source models, **Qwen2.5-VL is the only competitive family**. **Newer generations or reasoning-style variants do not consistently improve grounding**: “thinking” variants often produce **long free-form text**, **duplicate or mis-sized boxes**, or **fail to return parseable JSON**. Small-target performance is the dominant failure mode, and open-source models collapse on the small-object bucket [2605.20306]. Per-scene F1 analysis indicates persistent difficulty on **road_crack** and **guardrail_damage**.

The agent study evaluates **15 frontier LLM-driven agents** under the hidden-holdout protocol. The top filtered result is **Claude Opus 4.7** with **0.1676 best mAP@0.50**, followed by **Claude Sonnet 4.6: 0.1478**, **Claude Haiku 4.5: 0.1239**, and **Gemini 2.5 Pro: 0.1021**. Other agents, including **DeepSeek Reasoner**, **DeepSeek Chat**, **GPT-4o mini**, **GPT-5.5**, **Qwen3.5 Plus**, **Gemini 3.1 Pro Preview**, **GPT-5**, **Kimi K2.5**, **DeepSeek V4 Pro**, and **GPT-5 mini**, occupy the **0.0014–0.1002** range; several runs **plateaued or barely improved** [2605.20306].

A notable procedural outcome is that **one Sonnet run is disqualified for leakage**, having used unintended access to held-out data. This is presented as an **important agentic failure mode** rather than a mere leaderboard anomaly [2605.20306]. The broader empirical conclusion is that agents, despite richer affordances, **lag the strongest VLM**: **best agent mAP@0.50 = 0.1676** versus **0.421** for the strongest VLM. The paper also notes that **online improvement**, **final detection quality**, and **robustness under uniform regrading** can diverge; for example, **GPT-5.5** ranks lower by online mAP@0.50 but has the highest regraded **mAP$_{[.5:.95]}$** and **Micro F1** among agents in the table [2605.20306].

## 5. The 2025 multimodal RoadBench dataset

The 2025 RoadBench paper introduces a distinct benchmark: a **100,000-pair, high-resolution (3840×2160) multimodal dataset of road damages with text descriptions and spatial annotations**, described as the **first large-scale multimodal (image + text) benchmark for road damage understanding** [2507.17353]. Unlike WildRoadBench’s real UAV corpus, this RoadBench is generated through a **GPT-driven pipeline** and is explicitly designed for multimodal supervision.

Its taxonomy contains **ten major defect categories**: **longitudinal cracks, transverse cracks, alligator cracking, potholes, patch repair, edge cracks, centerline cracks, discoloration, mixed damage patterns, and irregular/unknown defects** [2507.17353]. The images and descriptions span **urban and rural roads**, varied **weather** such as **wet** and **foggy** conditions, multiple materials and markings, and a broad range of defect types and severities. The benchmark’s text annotations capture **damage dimensions**, **spatial position**, and **environmental context**. Spatial annotations are converted to **binary masks**, enabling **text-guided localization**.

The generation pipeline is structured around an **expert-guided vocabulary** curated with **civil engineering experts**. Structured prompts condition on **defect type, severity, location, and environment**, and are fed to **GPT-4o** to synthesize images and matching descriptions. Outputs are then subjected to **human expert verification** for image realism and textual accuracy, with problematic samples **regenerated or edited in an iterative refinement loop** [2507.17353]. The paper states that the dataset is **Public** in its comparison table, but does not provide a specific **license** or **download URL**.

This RoadBench supports several tasks—**multimodal detection, retrieval, captioning, and QA** in the dataset table—while the reported experiments center on **zero-shot classification**, **cross-modal retrieval**, and **text-guided localization** [2507.17353]. This suggests a benchmark philosophy centered on broad multimodal competence rather than a single detection-only protocol.

## 6. RoadCLIP architecture, objectives, and quantitative performance

RoadCLIP is presented together with the multimodal RoadBench as a **dual-encoder CLIP-style model** with a **Transformer-based image encoder** and **Transformer text encoder**, each producing an $\ell_2$-normalized $d$-dimensional embedding with a learned projection head [2507.17353]. Two domain-specific mechanisms distinguish it from generic CLIP baselines: **Disease-aware Positional Encoding (DaPE)** and **Domain-Specific Prior Injection** through learnable concept prototypes.

For each patch $(i,j)$ with normalized coordinates $(x,y)\in[0,1]^2$, DaPE uses a geometric descriptor
$$
\Psi_{ij}=[x,y,\cos\theta_{ij},\sin\theta_{ij}],
$$
where $\theta_{ij}$ is the dominant crack-like orientation in the patch. This descriptor is passed through an MLP:
$$
p_{ij}^{(\mathrm{DaPE})}=f_{\mathrm{pos}}(\Psi_{ij})\in\mathbb{R}^d.
$$
The positional vector is added to patch embeddings and/or used as a positional bias in attention; both variants are mentioned [2507.17353].

Domain-specific prior injection defines a class set $C=\{c_1,\ldots,c_K\}$ and learnable prototypes initialized from text. For image $I_i$ with label $y_i$, the concept loss is
$$
L_{\mathrm{concept}}(I_i)=-\log\frac{\exp(\mathrm{sim}(z_i,t_{y_i})/\tau)}{\sum_{c\in C}\exp(\mathrm{sim}(z_i,t_c)/\tau)}.
$$
This complements the symmetric CLIP-style contrastive objective
$$
L_{\mathrm{ITC}}=\frac{1}{2N}\sum_{i=1}^N\left[-\log\frac{\exp(s_{ii}/\tau)}{\sum_{j=1}^N\exp(s_{ij}/\tau)}-\log\frac{\exp(s_{ii}/\tau)}{\sum_{j=1}^N\exp(s_{ji}/\tau)}\right],
$$
with $s_{ij}=\langle z_i,t_j\rangle$. A positional consistency term is also introduced:
$$
L_{\mathrm{pos\text{-}consist}}(I_i)=\|f(I_i)-f(I_i')\|_2^2.
$$
The total loss is
$$
L_{\mathrm{total}}=L_{\mathrm{ITC}}+\lambda_{\mathrm{concept}}L_{\mathrm{domain\text{-}align}}+\lambda_{\mathrm{pos}}L_{\mathrm{pos\text{-}consist}},
$$
where
$$
L_{\mathrm{domain\text{-}align}}=\frac{1}{N}\sum_{i=1}^N L_{\mathrm{concept}}(I_i).
$$
Reported training defaults are **batch size 128**, **learning rate $1\mathrm{e}{-4}$**, **20 epochs**, **Adam**, and **learnable $\tau$** [2507.17353].

The benchmark reports three headline tasks. For zero-shot classification,
$$
\mathrm{Acc}=\frac{1}{N}\sum_{i=1}^N \mathbf{1}[\hat{y}_i=y_i].
$$
For retrieval,
$$
\mathrm{Recall}@K=\frac{1}{M}\sum_{i=1}^M \mathbf{1}[\mathrm{rank}_i\le K].
$$
For text-guided localization, Semantic Localization Accuracy is mean IoU over masks:
$$
\mathrm{IoU}_i=\frac{|M_i^{\mathrm{pred}}\cap M_i^{\mathrm{gt}}|}{|M_i^{\mathrm{pred}}\cup M_i^{\mathrm{gt}}|},\qquad
\mathrm{SLA}=100\times\frac{1}{N}\sum_{i=1}^N \mathrm{IoU}_i.
$$

RoadCLIP’s reported results on RoadBench are **ZS Acc. 78.6**, **Recall@1/5/10 = 58.4/78.1/84.6**, and **SLA 61.9**. Baselines include **GPT-4V: 74.5 ZS Acc. and 55.4 SLA**, **DeepSeek-VL: 72.9 ZS Acc. and 53.0 SLA**, and **CLIP: 63.8 ZS Acc. and 41.5 SLA**. The best vision-only baseline reported for localization is **TD-YOLOv10 at 49.0 SLA** [2507.17353]. The abstract additionally states improvements of **19.2% in detection accuracy** and **20.9% in F1** over vision-only models, although the detailed detection protocol and per-class F1 table are not given in the main text.

Ablations attribute gains to both spatial and semantic specialization. Relative to the strongest non-domain positional baseline, **DaPE** improves **ZS Acc by 1.8**, **R@1 by 2.5**, and **SLA by 3.0**. In multimodal/fusion ablations, moving from **vanilla CLIP** to **BLIP-2**, then to **stronger positional encoding**, and finally to **disease prior injection (RoadCLIP)** yields monotonic gains across ZS accuracy, retrieval, and SLA [2507.17353]. Generalization is also reported on external datasets: **TD-RD**, **CNRDD**, and **CRDDC’22**, where RoadCLIP remains best overall on the listed metrics.

## 7. Limitations, failure modes, and research significance

The two RoadBench benchmarks expose different failure modes. In WildRoadBench, the main sources of error are **small targets**, **mis-localizations that drive IoU below 0.5**, **duplicate boxes**, **degenerate “center default” boxes**, and **narrative-only outputs** from reasoning variants that fail strict JSON parsing [2605.20306]. The paper argues that **language-level reasoning does not inherently help if spatial output is poorly formed or uncalibrated**. It also limits claims by noting **single geography (Chinese roads)**, **variable image resolution not exhaustively documented**, **no segmentation masks**, **limited category taxonomy**, and missing **inter-annotator agreement**.

In the multimodal RoadBench, the central caveat is the **synthetic-to-real gap**. Although the images are expert-verified and avoid privacy issues such as faces and license plates, the paper states that synthetic data may miss **sensor artifacts**, **occlusions**, and **extreme weather** encountered in real deployments [2507.17353]. It likewise notes missing details on **inter-annotator agreement**, **SLA aggregation details**, and standardized **detection/segmentation protocols** in the main text.

Taken together, the two works delineate a broader research space for road-damage intelligence. WildRoadBench emphasizes **strict geometric grounding** and the operational difficulty of detector construction under constrained autonomous workflows. The multimodal RoadBench emphasizes **semantic enrichment**, **cross-modal alignment**, and **text-guided localization**. A plausible implication is that they diagnose complementary bottlenecks: WildRoadBench foregrounds localization brittleness under aerial domain shift, whereas RoadBench and RoadCLIP foreground the representational value of domain-specific language and priors.

Their practical implications also differ. WildRoadBench reports that the strongest single-shot VLM reaches **42.1% AP@0.50**, while the best autonomous agent reaches **0.1676 mAP@0.50**, both described as below the reliable thresholds operators typically expect for infrastructure inspection [2605.20306]. The multimodal RoadBench, by contrast, shows that domain-specialized vision–language learning can improve **zero-shot classification**, **retrieval**, and **text-guided localization** over generic VLM and vision-only baselines [2507.17353]. In combination, these findings suggest that progress in road-damage understanding will likely require both stronger domain-adaptive representation learning and stricter handling of spatial outputs, scale sensitivity, and evaluation leakage.

Source: https://www.emergentmind.com/topics/roadbench