PAVE Datasets for Pavement & AV Analysis

• PAVE Datasets are a collection of high-impact, multi-modal benchmarks designed for pavement distress analysis, condition assessment, and autonomous vehicle evaluation.
• These datasets integrate diverse imaging conditions, resolutions, and sensor streams while employing standardized annotation protocols such as COCO-style JSON for global consistency.
• They enable advanced applications including object detection, pixel segmentation, and vision-language reasoning, driving improvements in automated infrastructure assessment.

The term "PAVE Dataset" encompasses several distinct, high-impact datasets relevant to pavement distress analysis, pavement condition assessment, and autonomous vehicle evaluation. Each dataset, released under similar acronyms or names, targets specific sub-fields: from pixel-level segmentation, object detection, and condition scoring to vision-language reasoning and behavioral planning for autonomous vehicles. The following article systematically reviews five principal PAVE datasets and their technical scope as established in leading arXiv research.

1. PaveSync: Unified Benchmark for Pavement Distress Detection

PaveSync is a globally representative, comprehensively annotated benchmark developed to overcome limitations of previous fragmented, non-standardized pavement distress datasets. It consolidates seven existing sources into a single dataset comprising 52,747 images and 135,277 bounding-box annotations across 13 distinct distress types. These distress types—ranging from pothole, alligator cracking, rutting, to bleeding and edge cracking—reflect a harmonized taxonomy implemented according to strict definitions. Data sources span eight countries (Iran, China, United States, Japan, India, Czech Republic, Norway, Ghana), and cover diverse pavement environments, including urban arterials, rural highways, multi-lane expressways, and airport runways.

Key imaging parameters include native resolutions from 640×360 up to 4000×3000 px (standardized to 640×640 px for benchmarking), diverse viewpoints (ground, pavement-level, aerial/drone, top-down), and wide-ranging weather conditions (clear, dusk, rain, snow). The dataset exhibits a pronounced long-tailed class distribution, with distress types 'longitudinal cracking' and 'pothole' dominating the sample count.

Annotation unification aligns all labels to the COCO-style JSON format, ensuring global consistency of class IDs. Preprocessing involves resizing, padding (to preserve aspect ratio), and extensive data augmentation (random cropping, rotation, flipping, brightness/contrast variation, Gaussian noise). Annotation quality is systematically validated through stratified visual overlay and correction until the error rate falls below 1% (Kyem et al., 23 Dec 2025).

2. PAVE: End-to-End Dataset for Production Autonomous Vehicle Evaluation

Distinct from surface condition analysis, PAVE targets behavioral safety evaluation in autonomous driving. It is an end-to-end, real-world dataset collected entirely under production-grade autonomous vehicle (AV) control, comprising over 100 hours of driving data, 32,727 key frames, and multimodal sensory streams (four 2592×1944 RGB cameras, 0.8 cm-precision RTK GNSS/IMU). Key frames are defined as the pivotal image within each 11-second driving segment, supporting temporal reasoning over 6 seconds of past and 5 seconds of future 20 Hz vehicle trajectories.

Annotations include axis-aligned bounding boxes and tracking IDs for surrounding vehicles, pedestrians, traffic lights, and traffic signs. Each frame is richly attributed with scenario metadata: area type, driver intent, lighting, weather, surface type, and traffic density. The dataset is further organized into a normalized MySQL schema to facilitate complex querying.

A published end-to-end model baseline, which fuses multi-view vision and trajectory history in a BEV-style latent representation, achieves an Average Displacement Error (ADE) of 1.47 m and Final Displacement Error (FDE) of 8.16 m across the full dataset, with higher errors in night or highway scenarios (Li et al., 18 Nov 2025).

3. Pavementscapes (PAVE): Hierarchical Dataset for Pavement Damage Segmentation

The Pavementscapes dataset, also referenced as "PAVE," is dedicated to pixel-level segmentation of asphalt pavement damage. It contains 4,000 high-resolution (1024×2048) grayscale images recorded using top-down industrial cameras at ~60 km/h across 15 sites in China. The annotation protocol yields 8,680 instances labeled across six categories (longitudinal crack, lateral crack, alligator crack, pothole, rut, repair area) at image-level, bounding-box, and pixel (segmentation mask) granularity.

Quality control is rigorous: inter-annotator pixel-level consistency exceeds 92%. The dataset is explicitly designed to address pixel-class imbalance (distress pixels <1% per image) and non-iconic imaging conditions (cluttered, off-center, and occluded damages). Baseline segmentation model results are reported for architectures including FCN, U-Net, DeepLabv3+, Self-Attention Net, CC-Net, Double-Attention Net, and Segmentation Transformer, with the latter achieving mIoU of 59.7% and PA of 74.5% (Tong et al., 2022).

4. PaveCap: Multimodal Framework for Pavement Condition Assessment

PaveCap introduces a multimodal dataset from three U.S. cities, structured explicitly for both PCI (Pavement Condition Index) estimation and dense captioning. The dataset contains RGB top-down images labeled with scalar PCI scores (0–100, computed per ASTM D6433-07), bounding-box distress annotations for six defect classes (longitudinal, transverse, diagonal, block cracking, pothole, patching), and pavement-engineer-authored structured textual captions detailing defect presence/severity, absence, and PCI. The prevalence of each class exceeds 15% of images, with multi-label co-occurrences frequent.

Distinct pipeline components—YOLOv8 for object detection, SAM for segmentation, and a dense captioning transformer—enable end-to-end prediction of both numeric PCI and natural-language assessments. The test set demonstrates a Pearson correlation of 0.70 between predicted and annotated PCI, and high dense captioning performance (BLEU-1: 0.7445, METEOR: 0.7252) (Kyem et al., 2024).

5. PaveInstruct: Vision-Language Benchmark through Dataset Unification

PaveInstruct ("PAVE" in some references) exemplifies a next-generation instruction-tuned, vision-language dataset. It consists of 278,889 image–instruction–response triplets, curated by unifying annotations from nine diverse pavement datasets, producing 82,893 unique images. PaveInstruct operationalizes 32 tasks spanning spatial reasoning, PCI estimation (with explicit ASTM D6433 reasoning), severity classification, repair recommendation, and multi-turn inspection dialogues.

Data are formatted as JSON-L, containing bounding boxes, distress type lists, severity labels, and PCI where available. Prompts and model responses are designed via an automated LLM-based procedure, with 5% human expert review for terminological and engineering accuracy. Reported benchmarks highlight improvements of >20% in spatial grounding and reasoning versus non-instruction-tuned models. The dataset supports multi-modal foundation models—for instance, PaveGPT—for end-to-end, conversation-driven infrastructure assessment, with applications extending to training, report generation, and on-site inspector aid (Kyem et al., 9 Apr 2026).

6. PaveBench: Evaluation Suite for Perception and Vision-Language Analysis

PaveBench is a large-scale, line-scan, real-image pavement benchmark supporting object classification, detection, segmentation, and multi-stage vision-language VQA. It includes 20,124 orthorectified 512×512 image tiles from high-speed highway inspections in Liaoning Province, China, comprehensively labeled for ~10 surface distress types.

A distinctive “hard-distractor” subset challenges detection models with environmental confounders (e.g., stains, shadows). Tasks are structured via COCO-style detection, pixel-PNG segmentation, and a VQA module (PaveVQA) with 32,160 question–answer pairs spanning recognition, quantitative localization, maintenance reasoning, and adversarial/negative queries. The agent-augmented VQA framework integrates explicit tool calls to domain-specific perception models, enforcing fact grounding and minimizing LLM hallucination (Li et al., 3 Apr 2026).

7. Position within the Pavement Informatics Landscape

These PAVE-related datasets collectively define the critical axes of progress in pavement informatics. PaveSync and Pavementscapes set the global benchmark for standardized detection and pixel segmentation, respectively. PAVE (Autonomous Vehicle) uniquely enables empirical behavioral safety evaluation in real-world AV settings. PaveCap and PaveInstruct provide the first datasets optimized for multimodal and vision–language research, supporting not only perception but also explanation, PCI computation, and workflow reasoning. PaveBench bridges low-level perception and interactive, tool-grounded VQA.

The emergence of these datasets has directly enabled cross-domain foundation models, standardization of benchmarking practices, and robust zero-shot transfer validation—substantiated by results such as YOLOv8 achieving mAP@50=0.68 in cross-region testing and MiniCPM/PaveGPT attaining comparable or superior reasoning scores to generalist LLMs when tuned on PaveInstruct (Kyem et al., 23 Dec 2025, Kyem et al., 9 Apr 2026).

Future evolution is projected along three axes: longitudinal and regionally diverse sampling, expanding distress taxonomies, and tight standards compliance (ASTM D6433, GB 5768), as inferred from ongoing update plans and limitations cited in source papers. These developments serve both as the foundation and as catalysts for automated, explainable, and globally interoperable pavement management systems.