WeatherProof: Paired Segmentation Benchmark

• WeatherProof is a paired semantic segmentation dataset offering clear and adverse weather image pairs with precise pixel-level annotations over 10 classes.
• Its controlled pairing ensures constant scene geometry, isolating the impact of weather conditions such as rain, fog, and snow on model performance.
• The dataset underpins novel training methodologies including paired training, consistency regularization, and language-guided conditioning to enhance weather-robust perception.

Searching arXiv for WeatherProof-related papers to ground the article. WeatherProof is a benchmark family for weather-robust visual perception whose best-known instantiation is a paired semantic segmentation dataset of clear-weather and adverse-weather images sharing the same underlying scene. In its paired-segmentation form, WeatherProof was introduced as the first semantic segmentation dataset with accurate clear and adverse weather image pairs, containing 174,000 RGB images organized into 87,000 paired scenes, with pixel-wise annotations over 10 semantic classes and a benchmark protocol centered on rain, snow, fog, and composite degradations (Gella et al., 2024). Its defining premise is controlled correspondence: by holding scene geometry and semantic content constant across the clear and degraded views, the benchmark is designed to attribute performance gaps to weather rather than to viewpoint or scene change, and it has consequently become a testbed for paired training, consistency regularization, language guidance, and challenge-style evaluation under adverse weather (Gella et al., 2023).

1. Historical emergence and naming

WeatherProof emerged from the observation that contemporary foundational segmentation architectures such as Swin, ConvNeXt, and InternImage retain strong clear-weather performance while exhibiting a large performance drop on rain-, fog-, and snow-degraded imagery. The dataset was therefore introduced together with a paired-training paradigm intended to separate the difficulty of semantic understanding from the difficulty of weather corruption, rather than conflating both within ordinary cross-scene evaluation (Gella et al., 2023).

The name is not unique in the literature. An earlier benchmark, also called the WeatherProof Dataset, was proposed for automotive multi-label classification of weather condition, light level, and street surface. That dataset contains 1,165 distilled images derived from 60,000 sampled stills, with labels for fog, rain, snow; bright, moderate, low light; and asphalt, grass, cobblestone (Dhananjaya et al., 2021). This suggests a potential source of bibliographic ambiguity: in current segmentation literature, “WeatherProof” generally denotes the paired clear/adverse semantic segmentation benchmark, whereas the 2021 work uses the same designation for a classification dataset.

A second important distinction concerns benchmark instantiations derived from the paired corpus. The original paired-dataset papers present WeatherProof as a research dataset for semantic segmentation and paired training, while later UG2+ challenge reports use the same corpus in competition settings with official splits, validation conventions, and model-specific preprocessing pipelines (Cao et al., 2024).

2. Data sources, pairing, and alignment

The paired semantic-segmentation version of WeatherProof is built from GT-RAIN and WeatherStream. Pairs are selected from sequences with minimal time lapse and consistent camera pose; one report specifies that only frame pairs captured within two seconds of one another are retained, after which clear/adverse pairs are automatically matched via shared GPS and time stamps and manually verified by overlay inspection (Gella et al., 2023).

Alignment is treated as a first-order design constraint. After coarse matching, homography refinements are computed on static background features such as buildings and lampposts, with the aim of ensuring pixel-precise correspondence. The paired-dataset report states that manual verification enforces fine alignment below approximately $\pm 2$ pixels shift, followed by a final human quality-control loop (Gella et al., 2023). The later WeatherProof paper frames the same objective more generally: pairs are drawn with minimal time lapse and consistent camera pose so that underlying scene geometry and semantic content are held constant, making the segmentation error gap attributable to weather degradations rather than scene variation (Gella et al., 2024).

This pairing strategy is central to the dataset’s methodological role. WeatherProof is not merely a collection of adverse-weather images; it is a controlled paired benchmark in which clear and degraded observations of the same scene enable direct study of weather-induced failure modes. A plausible implication is that the benchmark is particularly well suited for consistency-based objectives, because the supervision signal can be interpreted against a stable semantic substrate rather than against uncontrolled environmental variation.

3. Annotation schema, scale, and statistical structure

The canonical paired dataset defines 10 semantic classes: background, tree, structure, road, terrain-snow, terrain-grass, terrain-other, stone, building, and sky. Annotations are produced by expert human labelers on the clear frames, and the identical masks are reused on the aligned adverse frames. The published descriptions emphasize boundary quality: annotations are tightened to minimize “background” leakage, peer review is applied, and cross-validation against Cityscapes-style superpixel over-segmentation is used to detect under- or over-segmented regions (Gella et al., 2024).

Quality control is reported in multiple forms across the papers. One account states that over 95% of annotations passed an independent audit of boundary IoU greater than 98% between clear and adverse pairs; another reports cross-review of a random 5% subset by senior annotators and an automated consistency check requiring class-wise IoU between clear-frame and adverse-frame binary masks to exceed 99% before release (Gella et al., 2023). These descriptions are compatible in spirit, although they emphasize different validation stages.

The core reported properties of the paired-segmentation benchmark are as follows.

Property	Reported value	Note
Total RGB images	174,000	Organized as paired clear/adverse images
Paired scenes	87,000	One clear and one adverse image per scene
Train set	147,800 images	73,900 clear/adverse pairs
Test set	26,200 images	13,100 clear/adverse pairs
Primary weather phenomena	rain, snow, fog	Composite weather also present
Official classes	10	Includes background

The dataset papers describe the weather content in two complementary ways. One description states an approximately uniform distribution over the three primary phenomena, with about 58,000 pairs each for rain, fog, and snow (Gella et al., 2023). A later description provides a scene-level breakdown that explicitly includes composites: rain only at approximately 31% of scenes, fog only 28%, snow only 15%, rain plus fog 14%, snow plus fog 10%, and other composites 2% (Gella et al., 2024). This suggests that the “approximately uniform” statement pertains to the primary phenomenon counts, whereas the more granular breakdown reflects composite co-occurrence.

The same paper reports approximate per-class pixel shares of 33% background, 20% road, 12% building, 10% sky, 8% tree, 5% structure, 4% terrain-grass, 3% terrain-snow, 3% terrain-other, and 2% stone (Gella et al., 2024). Image resolution is diverse: raw data span common automotive and surveillance-style cameras, one report gives a mean resolution of $1500\times 900$, and the source collections are described as ranging from $1024\times 768$ to $1920\times 1080$ (Gella et al., 2023). In most benchmark studies, however, training is carried out on random crops rather than full-resolution images.

A nomenclature wrinkle arises in challenge reporting. The CVPR 2024 WeatherProof challenge report describes the benchmark as using nine semantic classes—building, structure, road, sky, stone, terrain-grass, terrain-other, terrain-snow, and tree—without enumerating background (Cao et al., 2024). By contrast, the dataset papers define 10 classes including background (Gella et al., 2024). The most precise interpretation is that the underlying dataset uses a 10-class taxonomy, while at least one challenge write-up reports a 9-class evaluation framing.

4. Formalization and benchmark protocols

The paired WeatherProof dataset is formalized as

$D=\{(x_i^c, x_i^a, y_i)\}_{i=1}^N,$

where $x_i^c$ is the clear image, $x_i^a$ its aligned adverse counterpart, and $y_i$ the shared ground-truth segmentation map over $C=10$ classes (Gella et al., 2023). This formulation makes explicit that the benchmark is not just paired at the image level; it is paired at the supervision level, since both observations inherit the same semantic mask.

Evaluation follows standard semantic-segmentation metrics. For class $c$,

$1500\times 900$0

and the mean score is

$1500\times 900$1

The original protocol performs inference on $1500\times 900$2 sliding-window crops to cover the full image and reports both absolute mIoU and the relative percentage drop from clear to adverse conditions (Gella et al., 2024).

WeatherProof’s paired structure also motivated a paired-training methodology. In the 2023 paired-dataset formulation, the baseline segmentation loss is augmented by a feature consistency loss,

$1500\times 900$3

and an output consistency loss,

$1500\times 900$4

where $1500\times 900$5 and $1500\times 900$6 are encoder features for the clear and adverse images and $1500\times 900$7 are the corresponding predictions. The same framework introduces a CLIP Injection Layer that encodes weather prompts and adverse images through a frozen ViT-B/32 encoder and injects the resulting representation into the backbone through cross-attention at three mid-level stages (Gella et al., 2023). In this sense, the benchmark helped define not only a dataset but also a family of evaluation practices centered on weather-invariant representation learning.

Later challenge-oriented work reinterpreted the same corpus in semi-supervised terms. The CVPR 2026 UG2+ Track 2 report describes the WeatherProof challenge as containing 174 K paired images, with 147.8 K training images and 26.2 K test images, and pixel-wise masks over the same 10 semantic classes plus an ignore label of 255. In that pipeline, clean images with labels are used for supervised cross-entropy, degraded-weather images are treated as unlabeled, pseudo-labels are produced by an EMA teacher, and test-time augmentation is applied at inference (Chai et al., 21 May 2026).

5. Reported empirical behavior

Published results on WeatherProof consistently show that adverse weather produces a measurable degradation relative to clear-weather performance, and that paired training or language-guided conditioning partially closes that gap. In the 2024 WeatherProof paper, adverse-test mIoU for InternImage-XL rises from 46.47 to 51.19 with the CLIP-Injection Layer, ConvNeXt-XL rises from 37.94 to 39.92, and Swin Transformer rises from 41.29 to 43.92 (Gella et al., 2024).

The 2023 paired-training paper reports a closely related progression under a different experimental protocol. On the adverse-weather test set, InternImage improves from 43.32 under “Adverse Only” training to 45.24 under “Paired” training and to 51.31 under “Paired + FCL+OCL+CLIP”; ConvNeXt improves from 40.07 to 40.92 to 43.92 under the same progression (Gella et al., 2023). The same study also reports that clear-weather accuracy is not harmed: InternImage improves from 52.96 to 55.04 and ConvNeXt from 48.04 to 54.30 when moving from “Adverse Only” to the full paired method.

A separate competitive benchmark report from the CVPR 2024 WeatherProof challenge describes a strong pipeline built from InternImage-H and Mask2Former, trained on clean, degraded, and DA-Clip-denoised views, and followed by Dense-CRF plus a $1500\times 900$8 morphological closing. The incremental results are 39.61% mIoU for the baseline, 43.60% with model ensembling, 44.48% after Dense-CRF, and 45.10% for the final submission (Cao et al., 2024). The same report identifies large homogeneous regions such as sky and building as relative strengths, while road and stone remain difficult under severe weather.

Error analysis across the WeatherProof literature is unusually concrete because the clear/adverse pairing permits controlled comparisons. Composite weather scenes are reported to be substantially harder than single-effect scenes: on rain-plus-fog and snow-plus-fog examples, the baseline InternImage mIoU drops approximately 14.9 percentage points below single-effect scenes, while CLIP guidance reduces that gap to approximately 3.5 points (Gella et al., 2024). The classes “stone” and “terrain-other” show the largest absolute IoU drops, and thick fog or heavy snowfall covering more than 50% of the image remains a failure mode even for the language-guided model.

The 2026 UG2+ Track 2 report shows that WeatherProof also supports semi-supervised segmentation research. Using UniMatch V2 with a frozen DINOv2-Base encoder and a lightweight DPT head, the authors report mIoU/mDice of 0.69/0.69 for fully supervised training on clean images only, 0.79/0.79 for semi-supervised training on clean plus degraded images, and 0.80/0.80 after test-time augmentation (Chai et al., 21 May 2026). That report also notes a limitation of the current challenge split: no per-weather quantitative breakdown is provided for rain versus fog versus snow.

6. Role within weather-robust perception research

WeatherProof occupies a specific niche within the broader ecosystem of adverse-weather benchmarks. Its distinctive contribution is not merely weather diversity, but paired semantic correspondence at the scene level. XWOD, for example, is an object-detection benchmark rather than a semantic-segmentation dataset, and it expands the taxonomy to seven extreme weather conditions—rain, snow, fog, haze/sand/dust, flooding, tornado, and wildfire—over 10,010 images and 42,924 bounding boxes (Chen et al., 12 May 2026). Panoptic-CUDAL, by contrast, is a rural rainy-weather point-cloud dataset for semantic and panoptic segmentation with LiDAR, cameras, and pose data (Tseng et al., 20 Mar 2025). Boreas contributes repeated-route multimodal sensing across seasons for odometry, localization, and 3D detection rather than paired dense semantic labels (Burnett et al., 2022).

Against this background, WeatherProof’s methodological importance is that it isolates the robustness problem at the level of dense prediction on matched scenes. That property has made it useful for studying whether robustness gains come from genuine weather invariance or from broader shifts in scene distribution. It has also served as a bridge between dataset design and model design: the paired corpus directly motivated feature consistency, output consistency, and language-guided conditioning, and later challenge systems reused the same paired structure for semi-supervised teacher-student learning (Gella et al., 2023).

Several future directions are already explicit in the literature. The paired-training paper proposes extension to nighttime, mixed meteorological conditions, instance-level and panoptic labels, and temporal sequences (Gella et al., 2023). The 2026 challenge report points to adaptive confidence thresholding, curriculum learning over weather severity, video or sequence modeling for temporal consistency, fine-tuning of the DINOv2 backbone, and integration of low-level deweathering preprocessing (Chai et al., 21 May 2026). Taken together, these proposals indicate that WeatherProof is best understood not as a completed endpoint but as a foundational paired benchmark around which more general all-weather dense-perception protocols may be built.