ROADWork Autonomous Driving Dataset

• ROADWork is a specialized autonomous driving dataset focused on work zones, featuring detailed annotations for objects, signs, and trajectories.
• It combines still images, video clips, and trajectory data from diverse U.S. cities to support tasks like detection, text reading, scene description, and path forecasting.
• The dataset underpins both supervised benchmarks and zero-shot VLM evaluations, addressing challenges in long-tailed, safety-critical work zone scenarios.

ROADWork is a work-zone–focused autonomous-driving dataset introduced to support the full pipeline of “Recognize → Observe → Analyze → Drive” in safety-critical, long-tailed road-repair and construction scenarios. It was created in response to the lack of open datasets for developing algorithms that can perceive and navigate work zones, a setting in which state-of-the-art foundation models perform poorly and can exhibit failures such as being stopped by cones or misreading signs (Ghosh et al., 2024). ROADWork combines still images, video-derived drivable trajectories, fine-grained object and sign annotations, semantic context, and benchmark tasks spanning detection, text reading, scene description, and path forecasting. Subsequent work has also used ROADWork as a held-out testbed for VLM-based trajectory planning and as the source corpus for the zero-shot Roadwork-VLM benchmark (Liao et al., 3 Oct 2025, Luo et al., 25 Sep 2025).

1. Motivation and problem setting

ROADWork targets a scenario class that is relatively rare in driving data but safety-critical and long-tailed: work zones, including road repairs and construction sites. The dataset paper identifies a central gap in existing autonomous-driving benchmarks: they almost never label the full variety of work-zone objects such as cones, drums, barriers, TTC signs or boards, arrow boards, vehicles, and workers (Ghosh et al., 2024).

The dataset is explicitly designed around four coupled capabilities. “Recognize” concerns long-tailed, fine-grained object recognition. “Observe” concerns sign detection and reading, including both text and graphics. “Analyze” concerns holistic scene description for planning. “Drive” concerns drivable-path forecasting under dynamic changes. This framing makes ROADWork broader than a conventional object-detection benchmark: it is intended to support perception, language-grounded scene understanding, and planning within the same domain (Ghosh et al., 2024).

A plausible implication is that ROADWork occupies a niche between conventional urban-scene datasets and end-to-end planning datasets. The provided tasks and annotations emphasize work-zone-specific irregularity rather than stable lane geometry, making the dataset particularly relevant for studying open-set and out-of-distribution behavior in autonomous driving.

2. Data acquisition, scale, and composition

ROADWork contains 7,416 still images, derived from 5,078 manually selected work-zone frames and 2,338 iPhone captures from Pittsburgh (Ghosh et al., 2024). The video source consists of 30 s clips at 5 FPS mined from the Michelin Mobility Intelligence Open Dataset, and the geographic coverage spans 18 U.S. cities including Denver, Phoenix, and Columbus (Ghosh et al., 2024). The data are described as smartphone-quality, for example approximately 12 MP, and dash or video imagery at approximately $1920\times1080$ resolution (Ghosh et al., 2024).

The dataset includes trajectory information extracted from monocular video. Specifically, sparse 3D reconstruction via COLMAP is used to produce metric 3D information and then 2D projected drivable paths (Ghosh et al., 2024). For benchmark construction, the paper does not fix a single train, validation, and test split; instead, baselines use standard 80/10/10 or cross-city leave-one-out for domain adaptation (Ghosh et al., 2024).

Later work characterizes ROADWork as the first large-scale, work-zone–focused collection of real driving videos aimed squarely at end-to-end trajectory planning, with 1,186 distinct “scenarios,” each corresponding to a short video clip of a vehicle navigating through a real work zone (Liao et al., 3 Oct 2025). In that description, the scenarios cover multiple city centers and highway environments, include layouts such as temporary cone delineations, dead-end detours, and shifting lanes, and involve temporary control devices including cones, drums, barricades, tubular markers, vertical panels, TTC signs, portable barriers, fencing, on-site vehicles, and human workers (Liao et al., 3 Oct 2025).

These two descriptions are consistent but operate at different granularities. The original ROADWork paper emphasizes a multimodal benchmark suite built from images and video-derived paths, whereas later trajectory-planning work foregrounds the 1,186 scenario units used for open-loop prediction and error analysis (Ghosh et al., 2024, Liao et al., 3 Oct 2025).

3. Annotation schema and data modalities

ROADWork provides annotations at multiple levels of granularity. At the scene level, image tags include weather, time of day, environment, alteration, and an active work-zone flag (Ghosh et al., 2024). Weather tags include conditions such as sunny, cloudy, wet, fog, and ice; time of day includes light, twilight, and dark; environment includes urban, suburban, rural, and highway; and alteration includes none, lane shift, and fully or partially blocked (Ghosh et al., 2024).

At the object level, the dataset contains bounding boxes and polygons for 15 categories: Cone, Drum, Tubular Marker, Vertical Panel, Barricade, Barrier, Fence, Work Vehicle, Work Equipment, Worker, Police Vehicle, Police Officer, TTC Sign, TTC Message Board, and Arrow Board (Ghosh et al., 2024). Polygons are manually drawn with Vishwalingam–Wyatt simplification and fully verified by a single annotator, and an “Occluded” flag marks truncated or partially blocked objects (Ghosh et al., 2024).

For traffic control devices, ROADWork includes fine-grained sign annotations. TTC Sign “graphics” cover 62 distinct pictorial motifs, and TTC Sign “text” covers 360 distinct phrases, with both text and graphics parsed and transcribed and with occlusion flags on sign elements (Ghosh et al., 2024). The examples explicitly include phrases such as “DETOUR AHEAD,” “PEDESTRIAN CROSS HERE,” and “NO RIGHT TURN” (Ghosh et al., 2024).

For scene context, ROADWork provides semantic segmentation masks for road, sidewalk, and sparse bike-lane regions (Ghosh et al., 2024). For planning, it provides drivable trajectories obtained automatically from video and GPS via COLMAP, ground-plane fitting, and road-mask filtering. These trajectories are sampled to 20 waypoints, with the first 5 as “observed,” the last 1 as “goal,” and the remaining 14 as “future path” (Ghosh et al., 2024).

A later trajectory-planning study describes a reduced planning-oriented annotation view of ROADWork in monocular form: instance-level bounding boxes for ten work-zone element classes $$C_{wz}=\{\text{drum, cone, work vehicle, ttc\_sign, fence, barricade, barrier, worker, tubular\_marker, vertical\_panel}\}$$, a “road mask” semantic segmentation of the drivable surface, virtual-lane assignments to every pixel in the road region, and per-frame future trajectories in pixel coordinates with $T=20$ discrete waypoints (Liao et al., 3 Oct 2025). This suggests a task-specific projection of the fuller ROADWork annotation space into a planning benchmark.

4. Benchmark tasks and evaluation methodology

ROADWork defines five benchmark task families. The first is object detection and segmentation over all 15 categories, evaluated with COCO Average Precision, including $AP$, $AP_{50}$, $AP_{75}$, and size-stratified $AP_S/M/L$ (Ghosh et al., 2024). The second is work-zone image discovery, formulated as classifying unlabeled driving frames as work-zone versus non-work-zone using a simple rule requiring at least 3 objects of at least 2 categories occupying at least 10% of the image; the reported metrics are Precision and number of discovered frames (Ghosh et al., 2024).

The third task is sign detection and reading. Detection uses COCO AP over fine-grained sign classes, while reading uses end-to-end text spotting evaluated by Normalized Edit Distance, Word Accuracy, and character-level recall, precision, and F1. The paper defines $1-\mathrm{NED}=1-\mathrm{ED}(s,s')/|s|$ (Ghosh et al., 2024). The fourth task is scene description, where the objective is to generate a short description of the work zone highlighting object layouts and spatial relations for planning; metrics are BLEU@4, METEOR, ROUGE-L, CIDEr, and SPICE, with SPICE defined as $F_1$ over scene-graph tuples (Ghosh et al., 2024).

The fifth task is drivable path and goal prediction. Given an image and 5 observed waypoints, the model predicts a goal point and full future path heatmaps. The principal metric is Angular Error percentage below a threshold $\theta$, with the paper also reporting pixel-level ADE and FDE for compatibility (Ghosh et al., 2024). The angular metric is defined through camera-calibrated ray comparison:

$$C_{wz}=\{\text{drum, cone, work vehicle, ttc\_sign, fence, barricade, barrier, worker, tubular\_marker, vertical\_panel}\}$$0

In subsequent work focused on VLM trajectory planning, ROADWork is evaluated in open-loop fashion under pixel-space ADE, FDE, and Collision Rate. There, the horizon is again $$C_{wz}=\{\text{drum, cone, work vehicle, ttc\_sign, fence, barricade, barrier, worker, tubular\_marker, vertical\_panel}\}$$1, and collision is defined by whether a predicted trajectory collides with any annotated obstacle or work-zone element (Liao et al., 3 Oct 2025). This indicates that ROADWork supports both multimodal benchmark evaluation in the original paper and later open-loop planning evaluation in trajectory-prediction studies.

5. Baselines and empirical findings

The ROADWork paper reports that state-of-the-art foundation models perform poorly on work zones, and then shows domain-specific gains obtained by training or adapting models on ROADWork (Ghosh et al., 2024). For object detection, open-vocabulary approaches such as Detic and OpenSeeD serve as baselines, while supervised Mask R-CNN with COCO initialization achieves $$C_{wz}=\{\text{drum, cone, work vehicle, ttc\_sign, fence, barricade, barrier, worker, tubular\_marker, vertical\_panel}\}$$2, improving by $$C_{wz}=\{\text{drum, cone, work vehicle, ttc\_sign, fence, barricade, barrier, worker, tubular\_marker, vertical\_panel}\}$$3 over Detic (Ghosh et al., 2024). Copy-paste augmentation adds $$C_{wz}=\{\text{drum, cone, work vehicle, ttc\_sign, fence, barricade, barrier, worker, tubular\_marker, vertical\_panel}\}$$4 AP overall but hurts small-object $$C_{wz}=\{\text{drum, cone, work vehicle, ttc\_sign, fence, barricade, barrier, worker, tubular\_marker, vertical\_panel}\}$$5, and pseudo-masks from SAM degrade AP by approximately $$C_{wz}=\{\text{drum, cone, work vehicle, ttc\_sign, fence, barricade, barrier, worker, tubular\_marker, vertical\_panel}\}$$6 relative to full manual polygons (Ghosh et al., 2024).

For work-zone discovery, the ROADWork detector discovers 5,245 frames at $$C_{wz}=\{\text{drum, cone, work vehicle, ttc\_sign, fence, barricade, barrier, worker, tubular\_marker, vertical\_panel}\}$$7 precision, compared with 410 frames at $$C_{wz}=\{\text{drum, cone, work vehicle, ttc\_sign, fence, barricade, barrier, worker, tubular\_marker, vertical\_panel}\}$$8 precision from a BDD100k-based Detic rule and 1,070 frames at $$C_{wz}=\{\text{drum, cone, work vehicle, ttc\_sign, fence, barricade, barrier, worker, tubular\_marker, vertical\_panel}\}$$9 precision from Mapillary, corresponding to 12.8 times more discoveries and a $T=20$0 precision improvement over the BDD100k setting (Ghosh et al., 2024).

For sign detection and reading, Mask R-CNN on fine-grained TTC graphics improves from $T=20$1 to $T=20$2 with copy-paste augmentation. For text spotting, Glass trained on TextOCR reaches $T=20$3 and Word Accuracy $T=20$4, while crop-rescale by a factor of two improves these to $T=20$5 and Word Accuracy $T=20$6; on small signs below $T=20$7 px, $T=20$8 increases from $T=20$9 to $AP$0 (Ghosh et al., 2024).

For scene description, pretrained BLIP and LLaVA obtain SPICE values of approximately 3.9 and 9.4, respectively. Fine-tuning raises BLIP to $AP$1 and LLaVA to $AP$2, while adding object context in the prompt yields SPICE $AP$3, and ground-truth context yields $AP$4 (Ghosh et al., 2024). For drivable path and goal prediction, YNet with Cityscapes segmentation obtains $AP$5 for goal prediction and $AP$6 for path prediction; replacing this with ROADWork segmentation raises these values to $AP$7 and $AP$8, respectively, while reducing FDE from $AP$9 to $AP_{50}$0 px and ADE from $AP_{50}$1 to $AP_{50}$2 px (Ghosh et al., 2024).

These results support the paper’s claim that work zones constitute a long-tail, open-set corner where generic models underperform, and that work-zone-specific supervision materially improves recognizing, reading, describing, and forecasting in this domain (Ghosh et al., 2024).

6. Reuse in later benchmarks and VLM-based planning research

ROADWork has been reused as an evaluation corpus for VLM trajectory planning. One study describes ROADWork as a monocular dataset recorded with a single front-facing camera, with typical frame resolution on the order of $AP_{50}$3 px and frame rate $AP_{50}$4 Hz, and emphasizes that no LiDAR or radar channels are provided and neither GPS nor IMU traces are released (Liao et al., 3 Oct 2025). Within that study, the entire ROADWork corpus is treated as a held-out testbed for VLM evaluation, and no official train, validation, and test split is prescribed in the paper (Liao et al., 3 Oct 2025).

The same study reports that mainstream VLMs fail to generate correct trajectories in $AP_{50}$5 of cases and that, across 1,186 total scenarios, 70–82% are “failure scenarios” under the criterion $AP_{50}$6 px and $AP_{50}$7 px (Liao et al., 3 Oct 2025). Eight canonical failure patterns, denoted P1–P8, are distilled via scene-graph subgraph mining and human verification, including “dead-end detours” and “lane-borrowing Across Zone” (Liao et al., 3 Oct 2025). Average unmitigated VLM performance is summarized as approximately $AP_{50}$8 px, $AP_{50}$9 px, and $AP_{75}$0, while the proposed REACT framework reduces these to approximately $AP_{75}$1 px, $AP_{75}$2 px, and $AP_{75}$3 (Liao et al., 3 Oct 2025).

ROADWork also underlies Roadwork-VLM, a re-annotation of the entire original ROADWork set introduced as a zero-shot benchmark for high-level planning and reasoning in complex construction scenarios (Luo et al., 25 Sep 2025). Roadwork-VLM retains RGB front-facing camera images and driving trajectories from the original dataset and adds newly generated textual annotations: scenario descriptions, high-level navigation instructions, and full Chain-of-Thought reasoning sequences (Luo et al., 25 Sep 2025). It is used purely as a zero-shot testing benchmark: models are supervised-fine-tuned on NAVSIM only and then evaluated directly on Roadwork-VLM with no further training, and the paper does not define an internal train, validation, or test partition (Luo et al., 25 Sep 2025).

Roadwork-VLM evaluates high-level planning accuracy by requiring both longitudinal and lateral components of the plan to exactly match ground truth, and it reports a “Driving Metric Score” formed by averaging Risk Assessment, Commonsense Reasoning, and Scene Awareness (Luo et al., 25 Sep 2025). In the reported zero-shot comparison, MTRDrive obtains Risk $AP_{75}$4, Reason $AP_{75}$5, Scene $AP_{75}$6, Path $AP_{75}$7, Speed $AP_{75}$8, and Plan Accuracy $AP_{75}$9, while the paper states that Roadwork-VLM is intended to stress OOD generalization for Language-Vision Agents (Luo et al., 25 Sep 2025).

A plausible implication is that ROADWork has evolved from a dataset for detection, sign understanding, captioning, and path forecasting into a reference stress test for failure analysis and zero-shot generalization in VLM-based autonomous-driving systems.

7. Limitations, release status, and research directions

The ROADWork paper notes several limitations. Data are primarily from smartphones and one camera, creating a domain gap relative to real AV rigs. Trajectories represent the single “longest continuous drivable path” rather than all possible forks. There is no simultaneous LiDAR or stereo, and tasks such as description are evaluated solely on ROUGE, CIDEr, and SPICE without human preference studies (Ghosh et al., 2024). Later work reiterates related constraints from the trajectory-planning perspective: only monocular camera data are available, depth must be estimated by an external model, there is no official train, validation, and test split or cross-validation procedure for the 1,186-scenario planning corpus, and extreme lighting or nighttime work zones are underrepresented (Liao et al., 3 Oct 2025).

Despite those limitations, the dataset is publicly released under an Open Data Commons Attribution License v1.0 (Ghosh et al., 2024). The original paper identifies future directions including generative or simulation-based augmentation for ultra-rare sign, text, and graphic combinations, retrieval-augmented and compositional “compound AI” that chains detectors, spotters, VLMs, and planners, unsupervised geographic or domain adaptation, and multi-camera, 3D perception to handle monocular depth ambiguity (Ghosh et al., 2024). In the Roadwork-VLM line, proposed extensions include adding LiDAR, semantic maps, traffic-signal states, richer metadata such as weather and time-of-day tags, human-verified annotations, official split definitions, and broader scenario coverage beyond roadwork (Luo et al., 25 Sep 2025).

Taken together, these directions indicate that ROADWork is both a dataset and a problem formulation: a benchmark centered on the long-tail geometry, temporary traffic control, and planning irregularities of work zones, and a basis for later work on robust VLM reasoning, retrieval-augmented planning, and construction-zone generalization (Ghosh et al., 2024, Banerjee, 20 Oct 2025)