PaveBench: Unified Benchmark for Pavement Analysis

• The paper introduces PaveBench as a unified benchmark combining dense visual perception with structured vision-language Q&A for comprehensive pavement distress analysis.
• It employs real highway inspection imagery with top-down, geometry-preserving annotations for classification, detection, and semantic segmentation.
• It integrates expert-corrected multi-turn Q&A and agent-augmented approaches to enhance diagnostic reasoning and reduce numerical errors.

Searching arXiv for the primary paper and closely related benchmark papers to ground the article. {"^^^^^^^^2query2 A Versatile Benchmark for Pavement Distress Perception and Interactive Vision-Language Analysis\"","max_results":5,"sort_by":"submittedDate","sort_order":"descending"} {"^^^^^^^^2query2 pavement distress perception interactive vision-language analysis","max_results":^^^^2ti:\2query2^^^^,"sort_by":"relevance","sort_order":"descending"} {"^^^^^^^^2query2 pavement distress benchmark unified comprehensive dataset", "max_results":5, "sort_by":"relevance", "sort_order":"descending"} PaveBench is a unified, real-image benchmark for pavement distress perception and interactive vision-language analysis built from real highway inspection imagery captured with an orthographic line-scan system. It connects dense visual understanding—classification, object detection, and semantic segmentation—with single-turn, multi-turn, and expert-corrected vision-language question answering through PaveVQA, thereby linking geometry-preserving perception to quantitatively grounded diagnostic reasoning and interactive decision support (&&&^^^^2query2^^^^&&&).

^^^^2ti:\2^^^^. Motivation and conceptual scope

Pavement condition assessment is essential for road safety and maintenance, yet the benchmark landscape preceding PaveBench was oriented primarily toward unimodal computer vision tasks such as classification, detection, and segmentation. The motivating claim is that real inspection workflows require more than visual recognition: they also require quantitative analysis, explanation, and interactive decision support. In the same framing, general-purpose vision-LLMs trained on Internet images are described as lacking the domain knowledge, quantitative rigor, and grounding needed for pavement diagnosis, while earlier domain vision-language efforts such as RoadBench are characterized as relying on synthetic images and coarse language supervision without dense labels or interactive QA (&&&^^^^2query2^^^^&&&).

Within that problem setting, PaveBench is introduced as a coherent bridge from perception to interactive analysis. Its novelty is defined by three linked elements. First, it provides geometry-preserving, top-down perception on real inspection imagery with large-scale annotations for classification, detection, and pixel-level segmentation. Second, it introduces PaveVQA, a real-image, domain-specific VQA benchmark supporting single-turn, multi-turn, quantitative, and expert-corrected dialogues grounded in structured visual metadata derived from annotations. Third, it pairs these resources with unified task definitions and evaluation protocols, and with an agent-augmented VQA framework that integrates domain-specific vision tools with vision-LLMs in order to reduce hallucination and improve quantitative accuracy (&&&^^^^2query2^^^^&&&).

A plausible implication is that PaveBench should be read not merely as a dataset release, but as an attempt to redefine the evaluation unit for pavement inspection research: from isolated recognition tasks toward end-to-end, fact-grounded analysis.

2. Dataset composition and acquisition

PaveBench contains 2^^^^2query2^^^^,^^^^2ti:\22 high-resolution images at PRESERVED_PLACEHOLDER_^^^^2query2^^^^ pixels for perception. PaveVQA contains 32,^^^^2ti:\2sort_order2query2^^^^ image-grounded QA pairs, partitioned into ^^^^2ti:\2query2^^^^,^^^^2query2submittedDate2query2^^^^ single-turn, 2^^^^2query2^^^^,^^^^2ti:\2query2query2^^^^ multi-turn interactions, and 2,^^^^2query2ti:\2query2^^^^ error-correction pairs. The source domain is real-world highway inspection in Liaoning Province, China. Images were acquired by a high-resolution line-scan camera mounted on an inspection vehicle traveling at 8^^^^2query2^^^^ km/h, and the camera captures vertical, orthographic, top-down views intended to preserve geometric fidelity such as crack width and crack length (&&&^^^^2query2^^^^&&&).

The preprocessing pipeline consists of denoising, sharpening, contrast enhancement, and histogram equalization to improve visibility while suppressing background noise. The imagery includes diverse and challenging scenes with shadows, stains, and lighting variations common in highway environments. The paper explicitly notes that geographic coverage is currently limited, because it does not report multi-region collection beyond Liaoning Province (&&&^^^^2query2^^^^&&&).

The distress categories explicitly mentioned in the paper include alligator crack (AC), longitudinal crack (LC), and transverse crack (TC). The annotation pipeline is hierarchical across tasks, moving from classification to detection to segmentation. At the same time, the paper does not enumerate a full taxonomy table or a formal severity taxonomy for perception. Instead, severity and maintenance reasoning are represented at the QA level through PaveVQA. Structured per-instance attributes, including bounding box coordinates, areas, and skeleton lengths, are derived to support quantification and reasoning (&&&^^^^2query2^^^^&&&).

This composition matters because the benchmark couples image realism with geometric regularity. The top-down acquisition protocol differs from oblique or street-view imagery that can complicate direct measurement, and this suggests that PaveBench is particularly oriented toward workflows where measurement fidelity is as important as recognition fidelity.

3. Annotation design, grounding, and quality control

PaveBench includes three perception annotation types: image-level classification labels, bounding boxes for detection, and pixel-accurate semantic segmentation masks. Detection annotations were produced using LabelMe. Segmentation masks were manually traced by four domain experts in Photoshop, with an average effort of approximately ^^^^2ti:\2query2^^^^ minutes per image and up to ^^^^2ti:\2^^^^ hour for complex alligator crack clusters. Classification labels were cross-validated by multiple annotators, and all annotations underwent multi-stage expert verification to ensure boundary fidelity and consistency. The paper explicitly states that no formal inter-annotator agreement statistics such as Cohen’s kappa are reported, although a multi-expert review protocol is used (&&&^^^^2query2^^^^&&&).

On the multimodal side, PaveVQA is grounded in structured JSON metadata encoding geometric evidence such as coordinates, areas, and lengths. Its question types cover recognition or presence, localization, quantitative estimation, severity assessment, and maintenance reasoning. Answer formats include short text, categorical labels, and numbers. Negative queries about non-existent distress and adversarial or error-correction interactions are included to force factual rejection and corrections. Multi-turn QA uses approximately two rounds of dialogue per image, preserving context and history and carrying visual states forward, while expert-in-the-loop review corrects logic and ensures domain fidelity (&&&^^^^2query2^^^^&&&).

A distinctive component is the curated hard-distractor subset. The paper defines distractors as visual patterns such as shadows, stains, and superficial markings that resemble true distress and frequently co-occur with it, causing false positives and confusion. Rather than removing these cases, PaveBench retains and categorizes them for robustness-oriented evaluation. The stated curation criteria are co-occurrence with true distress and high visual confusability, with examples near AC, LC, and TC instances. The exact sample count for this subset is not stated in the text (&&&^^^^2query2^^^^&&&).

This annotation strategy places factual grounding at the center of the benchmark. The combination of pixel-accurate masks, structured geometric metadata, and expert-corrected dialogues is designed to constrain free-form language generation by tying it back to measurable visual evidence.

4. Tasks, splits, and evaluation protocols

PaveBench supports four core tasks: classification, object detection, semantic segmentation, and vision-language question answering. The unified perception tasks are defined as image-level distress presence or type prediction for classification, instance-level localization with bounding boxes for detection, and pixel-level masks for precise geometry in segmentation. PaveVQA provides single-turn QA for recognition, localization, and quantitative estimation; multi-turn QA to emulate interactive inspection with preserved context and history; and expert-corrected QA to probe reasoning robustness, including adversarial or false-premise queries that the model must reject (&&&^^^^2query2^^^^&&&).

The paper provides overall dataset and QA pair counts and unified evaluation protocols, but exact train, validation, and test split sizes per task or subset are not enumerated in the text. The release hosts the splits and task folders (&&&^^^^2query2^^^^&&&).

Task	Definition	Metrics
Classification	Image-level distress presence/type prediction	Top-^^^^2ti:\2^^^^ accuracy, macro precision, recall, F^^^^2ti:\2^^^^
Detection	Instance-level localization with bounding boxes	COCO-style mAP, AP5^^^^2query2^^^^, AP75, AR
Segmentation	Pixel-level masks for precise geometry	mPrecision, mRecall, mF^^^^2ti:\2^^^^, mIoU
VQA	Single-turn, multi-turn, and expert-corrected QA	Accuracy, token-level F^^^^2ti:\2^^^^, MAPE, ROUGE-L, BLEU, METEOR, BERTScore

For classification, the paper specifies: PRESERVED_PLACEHOLDER_^^^^2ti:\2^^^^

For detection, it uses COCO-style $\mathrm{mAP}$, $\mathrm{AP50}$, $\mathrm{AP75}$, and $\mathrm{AR}$, with $\mathrm{AP}@[.50:.95]$ averaged over IoU thresholds $0.50:0.95$ in steps of ^^^^2query2^^^^.^^^^2query2 For segmentation, it gives the foundational formulas

$$\mathrm{IoU}(A,B) = \frac{|A \cap B|}{|A \cup B|}, \qquad \mathrm{Dice}(A,B) = \frac{2|A \cap B|}{|A| + |B|}.$$

For VQA, the benchmark distinguishes numerical or factual evaluation from textual evaluation. Classification accuracy is used for categorical queries, token-level F^^^^2ti:\2^^^^ for localization descriptions, and quantitative estimation errors for numeric questions. Descriptive responses are evaluated with ROUGE-L, BLEU, METEOR, and BERTScore. The paper reports mean absolute percentage error for quantification in VQA tables: $$\mathrm{MAPE} = \frac{100}{N} \sum_i \left| \frac{y_i - \hat{y}_i}{y_i} \right|.$$ The text also mentions MAE and RMSE as standard quantification metrics, but it identifies MAPE as the metric reported in the VQA results tables (&&&^^^^2query2^^^^&&&).

5. Baselines, agent augmentation, and empirical behavior

The benchmark reports baseline results for state-of-the-art methods across perception and VQA. For classification, OverLoCK-T is the strongest among listed backbones, with Top-^^^^2ti:\2^^^^ Accuracy 93.8^^^^2ti:\2^^^^, Precision 93.8^^^^2ti:\2^^^^, Recall 93.8^^^^2ti:\2^^^^, and F^^^^2ti:\2^^^^ 93.76; ConvNeXt v2 reaches 9^^^^2ti:\2^^^^.^^^^2ti:\2 accuracy; TinyNeXt 9^^^^2query2^^^^.78; LSNet 88.8^^^^2query2^^^^; and FASTERVIT 87.36. For detection, DEIM obtains mAP 7^^^^2ti:\2^^^^.84, AP5^^^^2query2^^^^ 85.87, AP75 76.9^^^^2ti:\2^^^^, and AR 89.82, which is the best overall mAP among the reported models; RemDet reaches mAP 68.3^^^^2query2^^^^; YOLO26 reports AP5^^^^2query2^^^^ 83.6^^^^2query2^^^^ and AP75 68.53; and MI-DETR reports AR 87.7^^^^2query2^^^^. For segmentation, SCSegamba attains mPrecision 7^^^^2ti:\2^^^^.8^^^^2query2 mRecall 7^^^^2query2^^^^.78, mF^^^^2ti:\2^^^^ 7^^^^2ti:\2^^^^.^^^^2ti:\2 and mIoU 56.59, which is the best mIoU among listed methods; DeepLabV3+ reaches mPrecision 7^^^^2query2^^^^.26 and mIoU 54.^^^^2ti:\2query2^^^^; SegFormer reaches mF^^^^2ti:\2^^^^ 7^^^^2query2^^^^.59 and mIoU 55.44; and SOSNet reaches mRecall 7^^^^2query2^^^^.^^^^2ti:\2 and mIoU 54.85 (&&&^^^^2query2^^^^&&&).

The VQA baselines are reported for zero-shot, LoRA-tuned, and agent-augmented settings. For Qwen2.5-VL-3B, zero-shot performance is Cls Acc 65.^^^^2ti:\2^^^^8, Loc Token-F^^^^2ti:\2^^^^ ^^^^2ti:\2^^^^6.39, Quant MAPE ^^^^2ti:\2ti:\2^^^^6.2^^^^2query2 ROUGE-L 8.32, BLEU ^^^^2query2^^^^.79, METEOR ^^^^2ti:\2^^^^4.39, and BERTScore 83.69; with LoRA fine-tuning these become 88.24, 43.77, 47.^^^^2query2ti:\2^^^^, 52.4^^^^2query2^^^^, 2^^^^2query2^^^^.82, 44.66, and 92.86; with agent augmentation they become 89.68, 42.66, 35.4^^^^2query2^^^^, 53.4^^^^2ti:\2^^^^, 2^^^^2ti:\2^^^^.95, 44.22, and 92.78. For DeepSeek-VL2-small, the sequence is 55.48, 4^^^^2query2^^^^.36, 53^^^^2ti:\2^^^^.2^^^^2ti:\2 24.82, 4.77, 23.89, 88.^^^^2ti:\2^^^^7 in zero-shot; 92.98, 59.23, 48.69, 52.67, 2^^^^2query2^^^^.57, 43.4^^^^2query2^^^^, 93.2^^^^2ti:\2^^^^ with LoRA; and 92.8^^^^2query2^^^^, 42.^^^^2query2^^^^8, 26.9^^^^2ti:\2^^^^, 5^^^^2query2^^^^.39, ^^^^2ti:\2^^^^8.^^^^2query2 37.79, 92.65 with agent augmentation. For LLaVA-OneVision-7B, the corresponding values are 6^^^^2query2^^^^.9^^^^2ti:\2 27.7^^^^2ti:\2^^^^, ^^^^2ti:\2^^^^58.24, ^^^^2ti:\2ti:\2^^^^.^^^^2query2ti:\2 ^^^^2ti:\2^^^^.69, ^^^^2ti:\2^^^^5.95, 85.38; then 83.^^^^2query2^^^^4, 55.^^^^2ti:\2^^^^9, 66.95, 5^^^^2query2^^^^.82, ^^^^2ti:\2^^^^7.2^^^^2query2 38.52, 92.76; then 83.33, 46.64, 26.^^^^2ti:\2^^^^4, 46.^^^^2query2^^^^9, ^^^^2ti:\2^^^^3.27, 33.35, and 9^^^^2ti:\2^^^^.24 (&&&^^^^2query2^^^^&&&).

The agent-augmented VQA framework is described as follows. The VLM acts as an interactive planner or controller with tool-calling: it interprets the user’s question, decomposes it, selects tools, executes steps, and consolidates outputs into grounded answers. The integrated domain-specific tools are OverLoCK-T for classification, DEIM for detection, and SCSegamba for segmentation. Their outputs, including boxes and masks, are converted into explicit geometric quantities such as crack lengths via skeletonization and pixel areas, and these quantities are injected back into context as textual evidence. The planning and execution design is inspired by ReAct or tool-use paradigms and retains dialogue history and intermediate visual states to support multi-turn, stepwise analysis (&&&^^^^2query2^^^^&&&).

The reported empirical takeaways are consistent across the benchmark. Perception backbones perform well but still face ambiguity from real-world distractors. General VLMs underperform zero-shot on domain queries, whereas LoRA adaptation substantially improves both numerical and linguistic metrics. Agent augmentation markedly reduces numeric error, including relative to LoRA in several settings, and the paper highlights reductions such as DeepSeek-VL2-small PRESERVED_PLACEHOLDER_^^^^2ti:\2query2^^^^, LLaVA-OneVision-7B PRESERVED_PLACEHOLDER_^^^^2ti:\2ti:\2^^^^, and Qwen2.5-VL-3B PRESERVED_PLACEHOLDER_^^^^2ti:\22^^^^ (&&&^^^^2query2^^^^&&&).

A plausible interpretation is that the benchmark separates two failure modes often conflated in multimodal systems: domain adaptation in language generation and numerical grounding in visual reasoning. The LoRA results mainly indicate the value of adaptation, whereas the agent results emphasize explicit tool-mediated grounding.

6. Robustness, practical access, limitations, and position in the literature

Robustness in PaveBench is tied directly to realistic confounders. Shadows, stains, and superficial artifacts that mimic crack texture or topology and co-occur with true distress are explicitly retained and categorized rather than filtered out. Their purpose is to stress-test fine-grained discrimination and to discourage over-reliance on superficial textures. The paper reports that the presence of distractors contributes to reduced detection and segmentation performance relative to simpler benchmarks, but it does not report explicit per-subset performance deltas (&&&^^^^2query2^^^^&&&).

The benchmark also extends toward decision support. PaveVQA includes severity assessment and maintenance recommendation questions, and recommendations are grounded in structured evidence such as areas and lengths and expert-reviewed logic. At the same time, the paper explicitly states that it does not specify formal mappings to indices such as the Pavement Condition Index and does not provide domain-specific formulae for such indices (&&&^^^^2query2^^^^&&&).

Practical access is provided through a public Hugging Face release containing PRESERVED_PLACEHOLDER_^^^^2ti:\2^^^^3 top-down image patches, classification labels, detection boxes, segmentation masks, structured JSON metadata for geometric attributes, and PaveVQA single-turn, multi-turn, and expert-corrected QA pairs, including prompts and templates used for generation and negative or adversarial items. The text announces public availability but does not detail licensing terms, code, evaluation scripts, or training and inference recipes, which it says should be checked on the dataset card or repository (&&&^^^^2query2^^^^&&&).

The paper identifies several limitations. Current imagery originates from Liaoning Province and from a specific top-down line-scan system, so performance may degrade on oblique, smartphone, or different-region data. Realistic class imbalance means that rare but critical severe distresses pose learning challenges. Shadows and stains remain significant sources of confusion, and nighttime, adverse weather, occlusions, or extreme wear are difficult cases. Pixel-accurate masks, especially for complex alligator cracking, impose a heavy expert annotation burden. The future-work directions named in the paper are broader geographic and sensor coverage, richer taxonomies and severity schemas, expanded hard-distractor coverage and robustness diagnostics, and deeper connections to formal asset-management indices (&&&^^^^2query2^^^^&&&).

Within the broader literature, the paper positions crack-only benchmarks such as CFD and CRACK5^^^^2query2query2^^^^ as limited in category scope and often oblique in viewpoint; RDD-series datasets as broader in category coverage but mainly oblique and limited for pixel-level quantification; PaveDistress as top-down and segmentation-oriented but unimodal; and RoadBench as multimodal but synthetic, without segmentation and with only coarse descriptions. It further frames PaveBench, to its knowledge, as the first benchmark to unify real, top-down, geometry-preserving perception annotations across classification, detection, and segmentation with large-scale, structured, fact-grounded, single-turn and multi-turn VQA on the same images (&&&^^^^2query2^^^^&&&).

A useful contrast is provided by PaveSync, which standardizes multiple public sources into a globally representative detection benchmark of 52,747 images and ^^^^2ti:\2^^^^35,277 bounding boxes across ^^^^2ti:\2^^^^3 distress classes from eight countries, with unified class ontology and support for Pascal VOC XML, COCO JSON, and YOLO TXT formats. PaveSync focuses on detection and benchmark standardization across countries, viewpoints, and weather conditions, whereas PaveBench focuses on real top-down geometry-preserving imagery, dense perception annotations, and interactive vision-language analysis (&&&2^^^^2ti:\2^^^^&&&). This suggests that the two resources address different layers of the pavement-analysis stack: cross-domain detection standardization in one case, and tightly coupled perception-plus-reasoning in the other.