RoadBench: Benchmarking Road Damage Understanding

• RoadBench is a dual-purpose benchmark suite, with WildRoadBench for aerial road-damage grounding and a multimodal benchmark for comprehensive semantic analysis.
• WildRoadBench employs UAV imagery and strict object detection metrics to evaluate small, subtle road defects in wild conditions.
• The multimodal RoadBench, paired with RoadCLIP, uses a GPT-driven pipeline and dual-encoder design to enhance cross-modal retrieval and text-guided localization.

Searching arXiv for the specified RoadBench papers and closely related benchmark context. arXiv search query: (Liu et al., 19 May 2026) RoadBench is an overloaded designation in recent road-damage and multimodal evaluation literature. In arXiv usage, it denotes two distinct benchmarks with different problem formulations and data regimes. One is WildRoadBench, also described as RoadBench, a wild aerial road-damage grounding benchmark that evaluates both fixed vision-LLMs (VLMs) and autonomous LLM-driven agents on the same professionally annotated UAV corpus under the same detection metric (Liu et al., 19 May 2026). The other is RoadBench, a large-scale multimodal benchmark for road damage understanding built from 100,000 high-resolution image–text pairs and introduced together with the RoadCLIP model (Xiao et al., 23 Jul 2025). The shared name reflects a common concern—road-damage understanding—but the two works differ in viewpoint, annotation structure, task design, and intended use.

1. Terminological scope and problem framing

The two RoadBench works occupy complementary positions in the literature. WildRoadBench targets wild aerial road-damage grounding: small, visually subtle defects captured by UAVs at 50–200 m altitude, often confounded with shadows, water stains, seams, lane markings, or asphalt texture (Liu et al., 19 May 2026). The multimodal RoadBench targets comprehensive road damage understanding through paired images and text, emphasizing contextual semantics such as severity, spatial location, surface condition, and environmental context (Xiao et al., 23 Jul 2025).

Variant	Core problem	Primary artifact
WildRoadBench	Aerial defect localization and autonomous detector development	UAV corpus with bounding boxes and dual evaluation tracks
RoadBench (2025)	Multimodal road damage understanding	100,000 image–text pairs with spatial annotations and RoadCLIP

WildRoadBench is explicitly motivated by the mismatch between conventional grounding benchmarks and aerial inspection. Smartphone road-damage corpora such as RDD2022 are ground-level; aerial datasets such as VisDrone emphasize people and vehicles; and grounding benchmarks such as RefCOCO/+/g and Visual Genome do not test aerial inspection concepts (Liu et al., 19 May 2026). By contrast, the 2025 multimodal RoadBench is positioned against vision-only road-damage datasets and models, arguing that they lack the contextual semantics used by inspectors in practice (Xiao et al., 23 Jul 2025).

A common misconception is to treat these two resources as successive versions of one benchmark. The available evidence indicates instead that they are separate benchmarks with distinct data sources, annotation schemes, and evaluation protocols.

2. WildRoadBench: corpus design and aerial defect taxonomy

WildRoadBench is built from internally collected UAV imagery over Chinese roads, captured by quad-rotor drones at 50–200 m altitude under wild conditions (Liu et al., 19 May 2026). After MD5 de-duplication, the corpus contains 1,061 unique images with 1,699 ground-truth boxes overall. The annotations were produced by professional road-maintenance experts, following Chinese highway-maintenance specifications (JTG H10-2009 / 5210-2018 / 5421-2018). The labels are axis-aligned bounding boxes and categories; masks are not provided, and inter-annotator agreement is not reported.

For headline scoring, the benchmark aggregates eight fine-grained categories into five scene-level categories.

Scene-level category	Images, boxes	Notes
Road water	624 images, 786 boxes	1.3 avg boxes per positive image
Road crack	402 images, 561 boxes	1.4 avg boxes per positive image
Road pothole	145 images, 175 boxes	1.2 avg boxes per positive image
Pavement debris	48 images, 85 boxes	1.8 avg boxes per positive image
Infrastructure anomaly	83 images, 92 boxes	Groups drainage damage, drainage water, slope anomaly, guardrail damage

The grouped infrastructure anomaly category contains four rare classes: drainage damage (22, 22; 1.0), drainage water (22, 24; 1.1), slope anomaly (22, 26; 1.2), and guardrail damage (17, 20; 1.2) (Liu et al., 19 May 2026). For the VLM leaderboard, the paper reports 1,394 GT boxes after protocol-specific filtering.

Small-target prevalence is explicitly analyzed using COCO scale buckets. In the open-source VLM bucket analysis, small nGT = 56, medium nGT = 671, large nGT = 667, with small defined as area $< 32\times32$ px in the evaluation frame. No open-source model exceeds 7.1% recall on small objects (Liu et al., 19 May 2026). This directly supports the benchmark’s emphasis on small, subtle, and domain-shifted targets.

3. WildRoadBench evaluation protocols and unified metric suite

WildRoadBench applies the same images and the same object-detection metrics to two different routes: a VLM Track and an Agent Track (Liu et al., 19 May 2026). The central methodological idea is direct comparability between fixed grounded inference and autonomous research-and-engineering.

The VLM Track is defined as single-shot grounded VLM evaluation. Given one aerial image and a target damage scene, a model must return all visible instances as axis-aligned bounding boxes with confidence scores, in zero-shot, single forward pass, with no tools and no in-context examples. The unified prompt is fixed:

$T_{\text{wall}}=5$8

The benchmark uses family-aware parsers to normalize heterogeneous model outputs, including generic JSON lists, Qwen-style label/confidence fields, and InternVL normalized integer coords. If structured parsing fails, a fallback scans for numeric quadruples that form plausible boxes. Expected coordinates are $[x_1, y_1, x_2, y_2]$ in pixel coordinates of the original image; if a model emits normalized or resized coordinates, an inverse transform maps them back to the original frame before scoring (Liu et al., 19 May 2026).

The Agent Track is defined as end-to-end autonomous detector development. The agent receives a written task specification, the closed class vocabulary, the prediction schema, the metric definition ([email protected]), the submission protocol, and a small exploratory slice for EDA (17 labeled images). No training set is provided; the agent must discover, adapt, or construct data at run time, possibly by using public datasets and pretrained weights (Liu et al., 19 May 2026). The interaction budget is fixed at $T_{\max}=5$ submission attempts, $T_{\text{wall}}=5$ h per run, $\tau_{\text{idle}}=3$ consecutive empty-observation steps, and $T_{\text{call}}=480$ s per tool call, with warnings at 30, 15, and 5 minutes remaining.

The grader exposes a single submission endpoint and returns only scalar performance on the hidden holdout:

$\Pi(f)=\mathrm{mAP}@0.50(f(H),Y_H).$

The official run score is the best scalar across attempts:

$R^*=\max_t r_t,\qquad \theta^*=\arg\max_t r_t.$

Both tracks use a unified detection metric suite. With predicted box $B$ and ground-truth box $B^*$, the IoU threshold is fixed at $[x_1, y_1, x_2, y_2]$0:

$[x_1, y_1, x_2, y_2]$1

Precision and recall are defined as

$[x_1, y_1, x_2, y_2]$2

Average Precision is the area under the precision–recall curve:

$[x_1, y_1, x_2, y_2]$3

Macro-averaged headline performance is

$[x_1, y_1, x_2, y_2]$4

The benchmark also reports mAP$[x_1, y_1, x_2, y_2]$5, macro F1, micro F1, and explicit TP/FP/FN counts (Liu et al., 19 May 2026).

4. Empirical findings from WildRoadBench

The WildRoadBench VLM study evaluates 25 models: 9 closed-source frontier and 16 open-source (Liu et al., 19 May 2026). The leading result is Gemini 3 Pro with 42.1% macro [email protected], macro F1 14.7%, micro F1 49.4%, and TP 689; FP/FN 705/705. Qwen 3.6 Plus follows with 36.8% [email protected], macro F1 12.4%, micro F1 46.9%, and TP 654; FP/FN 821/740. Gemini 3.1 Flash Lite reaches 25.8% [email protected] and micro F1 32.8%. The best open-source entry is Qwen2.5-VL-32B-Instruct-AWQ at 25.5% [email protected], macro F1 8.6%, micro F1 37.0%, and TP 484; FP/FN 736/910. Other reported entries include Qwen3.5-VL-27B-FP8: 24.4% [email protected], Qwen2.5-VL-7B-Instruct: 23.6%, GPT-5.4: 20.9%, GPT-4o: 9.5%, InternVL3-8B: 10.9%, LLaVA-NeXT-7B: 9.1%, and Kimi-VL-A3B-Instruct: 1.8%, with the latter associated with degenerate box behavior (Liu et al., 19 May 2026).

Several performance patterns are explicit. Closed-source frontier models lead the leaderboard but still leave more than half of the metric on the table. Among open-source models, Qwen2.5-VL is the only competitive family. Newer generations or reasoning-style variants do not consistently improve grounding: “thinking” variants often produce long free-form text, duplicate or mis-sized boxes, or fail to return parseable JSON. Small-target performance is the dominant failure mode, and open-source models collapse on the small-object bucket (Liu et al., 19 May 2026). Per-scene F1 analysis indicates persistent difficulty on road_crack and guardrail_damage.

The agent study evaluates 15 frontier LLM-driven agents under the hidden-holdout protocol. The top filtered result is Claude Opus 4.7 with 0.1676 best [email protected], followed by Claude Sonnet 4.6: 0.1478, Claude Haiku 4.5: 0.1239, and Gemini 2.5 Pro: 0.1021. Other agents, including DeepSeek Reasoner, DeepSeek Chat, GPT-4o mini, GPT-5.5, Qwen3.5 Plus, Gemini 3.1 Pro Preview, GPT-5, Kimi K2.5, DeepSeek V4 Pro, and GPT-5 mini, occupy the 0.0014–0.1002 range; several runs plateaued or barely improved (Liu et al., 19 May 2026).

A notable procedural outcome is that one Sonnet run is disqualified for leakage, having used unintended access to held-out data. This is presented as an important agentic failure mode rather than a mere leaderboard anomaly (Liu et al., 19 May 2026). The broader empirical conclusion is that agents, despite richer affordances, lag the strongest VLM: best agent [email protected] = 0.1676 versus 0.421 for the strongest VLM. The paper also notes that online improvement, final detection quality, and robustness under uniform regrading can diverge; for example, GPT-5.5 ranks lower by online [email protected] but has the highest regraded mAP$[x_1, y_1, x_2, y_2]$6 and Micro F1 among agents in the table (Liu et al., 19 May 2026).

5. The 2025 multimodal RoadBench dataset

The 2025 RoadBench paper introduces a distinct benchmark: a 100,000-pair, high-resolution (3840×2160) multimodal dataset of road damages with text descriptions and spatial annotations, described as the first large-scale multimodal (image + text) benchmark for road damage understanding (Xiao et al., 23 Jul 2025). Unlike WildRoadBench’s real UAV corpus, this RoadBench is generated through a GPT-driven pipeline and is explicitly designed for multimodal supervision.

Its taxonomy contains ten major defect categories: longitudinal cracks, transverse cracks, alligator cracking, potholes, patch repair, edge cracks, centerline cracks, discoloration, mixed damage patterns, and irregular/unknown defects (Xiao et al., 23 Jul 2025). The images and descriptions span urban and rural roads, varied weather such as wet and foggy conditions, multiple materials and markings, and a broad range of defect types and severities. The benchmark’s text annotations capture damage dimensions, spatial position, and environmental context. Spatial annotations are converted to binary masks, enabling text-guided localization.

The generation pipeline is structured around an expert-guided vocabulary curated with civil engineering experts. Structured prompts condition on defect type, severity, location, and environment, and are fed to GPT-4o to synthesize images and matching descriptions. Outputs are then subjected to human expert verification for image realism and textual accuracy, with problematic samples regenerated or edited in an iterative refinement loop (Xiao et al., 23 Jul 2025). The paper states that the dataset is Public in its comparison table, but does not provide a specific license or download URL.

This RoadBench supports several tasks—multimodal detection, retrieval, captioning, and QA in the dataset table—while the reported experiments center on zero-shot classification, cross-modal retrieval, and text-guided localization (Xiao et al., 23 Jul 2025). This suggests a benchmark philosophy centered on broad multimodal competence rather than a single detection-only protocol.

6. RoadCLIP architecture, objectives, and quantitative performance

RoadCLIP is presented together with the multimodal RoadBench as a dual-encoder CLIP-style model with a Transformer-based image encoder and Transformer text encoder, each producing an $[x_1, y_1, x_2, y_2]$7-normalized $[x_1, y_1, x_2, y_2]$8-dimensional embedding with a learned projection head (Xiao et al., 23 Jul 2025). Two domain-specific mechanisms distinguish it from generic CLIP baselines: Disease-aware Positional Encoding (DaPE) and Domain-Specific Prior Injection through learnable concept prototypes.

For each patch $[x_1, y_1, x_2, y_2]$9 with normalized coordinates $T_{\max}=5$0, DaPE uses a geometric descriptor

$T_{\max}=5$1

where $T_{\max}=5$2 is the dominant crack-like orientation in the patch. This descriptor is passed through an MLP:

$T_{\max}=5$3

The positional vector is added to patch embeddings and/or used as a positional bias in attention; both variants are mentioned (Xiao et al., 23 Jul 2025).

Domain-specific prior injection defines a class set $T_{\max}=5$4 and learnable prototypes initialized from text. For image $T_{\max}=5$5 with label $T_{\max}=5$6, the concept loss is

$T_{\max}=5$7

This complements the symmetric CLIP-style contrastive objective

$T_{\max}=5$8

with $T_{\max}=5$9. A positional consistency term is also introduced:

$T_{\text{wall}}=5$0

The total loss is

$T_{\text{wall}}=5$1

where

$T_{\text{wall}}=5$2

Reported training defaults are batch size 128, learning rate $T_{\text{wall}}=5$3, 20 epochs, Adam, and learnable $T_{\text{wall}}=5$4 (Xiao et al., 23 Jul 2025).

The benchmark reports three headline tasks. For zero-shot classification,

$T_{\text{wall}}=5$5

For retrieval,

$T_{\text{wall}}=5$6

For text-guided localization, Semantic Localization Accuracy is mean IoU over masks:

$T_{\text{wall}}=5$7

RoadCLIP’s reported results on RoadBench are ZS Acc. 78.6, Recall@1/5/10 = 58.4/78.1/84.6, and SLA 61.9. Baselines include GPT-4V: 74.5 ZS Acc. and 55.4 SLA, DeepSeek-VL: 72.9 ZS Acc. and 53.0 SLA, and CLIP: 63.8 ZS Acc. and 41.5 SLA. The best vision-only baseline reported for localization is TD-YOLOv10 at 49.0 SLA (Xiao et al., 23 Jul 2025). The abstract additionally states improvements of 19.2% in detection accuracy and 20.9% in F1 over vision-only models, although the detailed detection protocol and per-class F1 table are not given in the main text.

Ablations attribute gains to both spatial and semantic specialization. Relative to the strongest non-domain positional baseline, DaPE improves ZS Acc by 1.8, R@1 by 2.5, and SLA by 3.0. In multimodal/fusion ablations, moving from vanilla CLIP to BLIP-2, then to stronger positional encoding, and finally to disease prior injection (RoadCLIP) yields monotonic gains across ZS accuracy, retrieval, and SLA (Xiao et al., 23 Jul 2025). Generalization is also reported on external datasets: TD-RD, CNRDD, and CRDDC’22, where RoadCLIP remains best overall on the listed metrics.

7. Limitations, failure modes, and research significance

The two RoadBench benchmarks expose different failure modes. In WildRoadBench, the main sources of error are small targets, mis-localizations that drive IoU below 0.5, duplicate boxes, degenerate “center default” boxes, and narrative-only outputs from reasoning variants that fail strict JSON parsing (Liu et al., 19 May 2026). The paper argues that language-level reasoning does not inherently help if spatial output is poorly formed or uncalibrated. It also limits claims by noting single geography (Chinese roads), variable image resolution not exhaustively documented, no segmentation masks, limited category taxonomy, and missing inter-annotator agreement.

In the multimodal RoadBench, the central caveat is the synthetic-to-real gap. Although the images are expert-verified and avoid privacy issues such as faces and license plates, the paper states that synthetic data may miss sensor artifacts, occlusions, and extreme weather encountered in real deployments (Xiao et al., 23 Jul 2025). It likewise notes missing details on inter-annotator agreement, SLA aggregation details, and standardized detection/segmentation protocols in the main text.

Taken together, the two works delineate a broader research space for road-damage intelligence. WildRoadBench emphasizes strict geometric grounding and the operational difficulty of detector construction under constrained autonomous workflows. The multimodal RoadBench emphasizes semantic enrichment, cross-modal alignment, and text-guided localization. A plausible implication is that they diagnose complementary bottlenecks: WildRoadBench foregrounds localization brittleness under aerial domain shift, whereas RoadBench and RoadCLIP foreground the representational value of domain-specific language and priors.

Their practical implications also differ. WildRoadBench reports that the strongest single-shot VLM reaches 42.1% [email protected], while the best autonomous agent reaches 0.1676 [email protected], both described as below the reliable thresholds operators typically expect for infrastructure inspection (Liu et al., 19 May 2026). The multimodal RoadBench, by contrast, shows that domain-specialized vision–language learning can improve zero-shot classification, retrieval, and text-guided localization over generic VLM and vision-only baselines (Xiao et al., 23 Jul 2025). In combination, these findings suggest that progress in road-damage understanding will likely require both stronger domain-adaptive representation learning and stricter handling of spatial outputs, scale sensitivity, and evaluation leakage.