PaveBench: A Versatile Benchmark for Pavement Distress Perception and Interactive Vision-Language Analysis

Abstract: Pavement condition assessment is essential for road safety and maintenance. Existing research has made significant progress. However, most studies focus on conventional computer vision tasks such as classification, detection, and segmentation. In real-world applications, pavement inspection requires more than visual recognition. It also requires quantitative analysis, explanation, and interactive decision support. Current datasets are limited. They focus on unimodal perception. They lack support for multi-turn interaction and fact-grounded reasoning. They also do not connect perception with vision-language analysis. To address these limitations, we introduce PaveBench, a large-scale benchmark for pavement distress perception and interactive vision-language analysis on real-world highway inspection images. PaveBench supports four core tasks: classification, object detection, semantic segmentation, and vision-language question answering. It provides unified task definitions and evaluation protocols. On the visual side, PaveBench provides large-scale annotations and includes a curated hard-distractor subset for robustness evaluation. It contains a large collection of real-world pavement images. On the multimodal side, we introduce PaveVQA, a real-image question answering (QA) dataset that supports single-turn, multi-turn, and expert-corrected interactions. It covers recognition, localization, quantitative estimation, and maintenance reasoning. We evaluate several state-of-the-art methods and provide a detailed analysis. We also present a simple and effective agent-augmented visual question answering framework that integrates domain-specific models as tools alongside vision-LLMs. The dataset is available at: https://huggingface.co/datasets/MML-Group/PaveBench.

• The paper presents PaveBench, a benchmark that unifies pavement distress perception with interactive vision-language analysis to advance pavement diagnostics.
• It introduces a dataset of 20,124 high-resolution images with multi-level annotations including classification, detection, and segmentation, and incorporates challenging hard distractors.
• The study demonstrates that agent-augmented VQA and LoRA fine-tuning significantly improve performance metrics by reducing errors and hallucinated responses.

PaveBench: Unifying Pavement Distress Perception and Vision-Language Interactive Analysis

Motivation and Context

Automated assessment of pavement condition is fundamental for road infrastructure longevity, safety, and maintenance planning. Traditional research efforts have been limited to unimodal computer vision tasks such as distress classification, detection, and segmentation. However, real-world pavement inspection requires richer, context-aware analyses—quantitative measurement, multi-turn decision support, and domain-specific reasoning—underserved by existing datasets and methodologies, which commonly lack multimodal annotations, fine-grained interaction, or fact-grounded decision-making. The proposed benchmark, PaveBench (2604.02804), aims to address these deficiencies by unifying dense, realistic visual perception tasks with interactive vision-language analysis in a single cohesive framework, setting a new paradigm for both evaluation and model development in pavement distress understanding.

Figure 1: PaveBench integrates visual perception tasks and vision-language interaction, bridging the gap between traditional computer vision and domain-specific, interactive analysis in pavement distress inspection.

Dataset Construction and Characteristics

PaveBench is constructed from high-resolution, top-down pavement imagery, acquired via specialized inspection vehicles operating at 80 km/h, preserving geometric fidelity with orthographic views. The data pre-processing pipeline employs denoising, sharpening, contrast enhancement, and histogram equalization to produce visually consistent samples enabling robust downstream annotation.

Figure 2: The dataset construction pipeline, encompassing acquisition, pre-processing, hierarchical annotation, and curation of challenging distractors.

Annotations are performed through a multi-level pipeline:

• Classification: Labeling presence and types of distress.
• Detection: Bounding box localization for each distressed region, verified by multiple annotators.
• Segmentation: High-fidelity, pixel-accurate labeling, executed and verified by domain experts.
• Hard Distractor Curation: Explicit inclusion of confounding background patterns (e.g., shadows, stains) frequently mistaken for true distress, providing a realistic and demanding testbed for model robustness.

PaveBench comprises 20,124 high-resolution ($512 \times 512$) images with long-tailed, imbalanced class distributions that mirror real-world prevalence of rare but critical defect types. This structure effectively challenges models to detect sparse but high-impact events.

Figure 3: Distribution of distress categories, analysis types, and fine-grained condition labels reveals dataset diversity, class imbalance, and the deliberate inclusion of hard distractors.

Multimodal Vision-Language Component: PaveVQA

The multimodal extension, PaveVQA, encompasses 32,160 question-answer pairs, spanning:

• Single-turn and multi-turn dialogue
• Recognition, localization, quantitative measurement, severity grading
• Maintenance recommendation
• Explicit handling of both positive and negative (non-existent) distress scenarios
• Error-correction interactions to expose and repair model reasoning weaknesses

A structured dialogue generation pipeline leverages: (a) visual annotations, (b) structured metadata (JSON-formatted, with precise geometric properties), and (c) LLM-driven prompt templates, feeding ChatGPT with multimodal context for grounded generation. Domain expert review ensures logical consistency and domain correctness, yielding a high-fidelity dataset for comprehensive multimodal QA benchmarking.

Figure 4: PaveVQA's construction workflow, visual question typology, and representative samples in structured, expert-verified multi-turn dialogue.

Benchmarking and Experimental Protocol

PaveBench is evaluated via three core paradigms:

• Visual Perception: State-of-the-art architectures are benchmarked on image classification, object detection, and semantic segmentation. Despite high scores—best in class models achieve 93.81% accuracy, 71.84% mAP, and 76.0% mIoU—challenges persist due to co-occurring distractors and intra-class variability.
• Multimodal VQA: Three VLMs (Qwen2.5-VL, DeepSeek-VL2, LLaVA-OneVision) are assessed on categorical, locational, quantitative, and free-form reasoning using accuracy, token-F1, MAE, ROUGE-L, BLEU, METEOR, and BERTScore.
• Agent-Augmented VQA: Beyond model fine-tuning, the framework integrates VLM tool-calling with domain-specific perception models, delegating geometric and spatial tasks to visual tools while using the VLM for semantic interpretation.

Empirical highlights:

• Zero-shot VLMs perform poorly on domain-specialized queries (e.g., sub-65% classification accuracy; high quantification error).
• LoRA fine-tuning yields strong gains in both numerical and semantic metrics.
• Agent-augmented VQA outperforms or competes with fine-tuned VLMs, substantially reducing hallucinated responses and improving verifiability, all while requiring zero updates to VLM parameters.

Comparative Positioning

Relative to legacy pavement datasets (AigleRN, CFD, RDD series, PaveDistress, RoadBench), PaveBench is distinguished by:

• Geometrically consistent, real-world, top-down images (unlike RoadBench's synthetic imagery)
• Cohesive multi-task annotation (classification, detection, segmentation)
• Integration of language supervision, multi-turn and expert-corrected QA, and quantitative reasoning
• Deliberate challenge construction (hard distractors, class imbalance)

(Figure 1) previously illustrated how PaveBench bridges the divide between conventional vision benchmarks and interactive VLM benchmarks.

Implications and Future Directions

Practical implications:

PaveBench presents a realistic and challenging standardized platform for model comparison and ablation in the pavement inspection domain, catalyzing robust model development for deployment in national-scale infrastructure monitoring. Hard distractors and class imbalance in the dataset ensure that progress on PaveBench translates directly to improved real-world performance.

Theoretical implications:

The unified benchmarking approach exposes limitations in current unimodal and VLMs for expert-grounded, domain-specific reasoning. Agent-augmented VQA showcases the value of explicit tool integration for reducing hallucination, improving interpretability, and separating semantic reasoning from geometric operations. This approach could generalize to related industrial visual inspection domains.

Speculation on future developments:

PaveBench is likely to spur research into:

• More sophisticated tool-augmented VLM systems with learnable orchestration of visual and language modules
• Cross-domain transfer learning leveraging PaveBench's high-quality annotations and interactions
• Advanced hard distractor recognition and uncertainty quantification methodologies
• Real-time, in-vehicle interactive analysis agents for road maintenance and safety applications

Conclusion

PaveBench provides a unified, multi-task, and multimodal benchmark for advancing pavement distress analysis from visual recognition toward deep, interactive visual-language understanding. With its structured dataset, comprehensive task coverage, and agent-augmented evaluation paradigm, it significantly raises the bar for both academic evaluation and practical deployment in infrastructure diagnostics. The benchmark's design principles and results imply broad applicability for any domain requiring accurate, explainable, and robust vision-language reasoning, particularly where domain-specific quantification and interactive decision-making are critical.