UAT-SAM: Uncertainty-Aware SAM in Adverse Weather

• UAT-SAM is an uncertainty-aware adaptation of SAM that integrates a CVAE-driven adapter to tackle ambiguous, weather-impaired car instance segmentation.
• It employs the Condition Modifies Sample Module (CMSM) to modulate features in each transformer block, effectively handling conditions like rain, fog, and snow.
• Empirical evidence shows UAT-SAM improves Dice scores by ~30% and IoU by ~42.7% over zero-shot SAM in adverse weather car instance extraction.

UAT-SAM denotes an uncertainty-aware adaptation of the Segment Anything Model (SAM) for adverse-weather autonomous-driving segmentation. In the formulation reported in the literature, it is the branch of a dual uncertainty-aware framework that targets extreme-weather, ambiguous car-instance segmentation by inserting an Uncertainty-Aware Adapter into SAM and conditioning feature computation on a sampled latent uncertainty code derived from a conditional variational autoencoder (CVAE). Its central premise is that rain, snow, fog, glare, low contrast, and lens occlusion induce ambiguous boundaries and multiple plausible masks, so segmentation should not be treated as a purely deterministic prediction problem (Ravindran et al., 5 Sep 2025).

1. Definition and problem scope

In the reported usage, UAT-SAM is the paper’s adaptation of the Uncertainty-Aware Adapter (UAT), originally designed for medical image segmentation, to the autonomous-driving and adverse-weather setting. It is described as a lightweight adapter-based finetuning method for SAM, with the adapter inserted into each transformer block of SAM and driven by a sampled uncertainty latent code $z$ produced by a CVAE with a Prior Net $P$ and a Posterior Net $Q$. The target application is extreme-weather, ambiguous instance segmentation, especially cars, rather than general scene parsing or broad multimodal reasoning (Ravindran et al., 5 Sep 2025).

The motivating failure mode is weather-induced ambiguity. Under rain, snow, fog, sleet, glare, low contrast, and lens occlusion, object boundaries become blurred, parts of objects disappear, structured noise appears, and small or distant objects become difficult to localize. The paper frames this as an uncertainty problem: standard SAM and SAM2 produce segmentation masks but do not explicitly model how confident they are, nor do they represent multiple plausible segmentations in ambiguous cases. This motivates a design in which segmentation features are modulated by uncertainty rather than inferred only through a single deterministic pathway.

Within that same paper, UAT-SAM is explicitly distinguished from a second method based on uncertainty-aware SAM2 finetuning. The two methods are presented as complementary rather than jointly trained parts of one architecture. UAT-SAM is the ambiguity-focused, adapter-based branch for severe weather and object-critical cases, whereas the SAM2 branch is a broader scene-level method using uncertainty inside the loss.

2. Architectural formulation

The core architectural claim is simple and specific: the Uncertainty-Aware Adapter is inserted into each transformer block of SAM. The adapter is described as a compact set of trainable parameters that incorporates uncertainty into SAM’s feature hierarchy rather than appending uncertainty only near the output. The mechanism used for this insertion is the CMSM, or Condition Modifies Sample Module, which takes a sampled latent code $z$, incorporates position vectors $p$, applies learnable attention-like transformations, and combines the transformed uncertainty features with the main features in a layer-specific manner (Ravindran et al., 5 Sep 2025).

The latent code $z$ comes from a CVAE with Prior Net $P$ and Posterior Net $Q$. The paper states that the CVAE employs both nets to encode observed uncertainty information from the input image. This positions the method as a probabilistic latent-variable adaptation of SAM. The uncertainty being modeled is primarily aleatoric or data-ambiguity uncertainty: degraded visibility, ambiguous boundaries, weather-induced occlusion, and the possibility of multiple valid segmentations for the same object.

A central conceptual distinction from standard SAM follows from this design. Standard SAM is treated as a pretrained, prompt-based, deterministic segmentation foundation model without explicit uncertainty representation. UAT-SAM adds trainable adapters, uncertainty-conditioned latent sampling, and training with multiple plausible annotations per image. The key difference is therefore not merely finetuning, but feature modulation by uncertainty at each transformer block.

The paper is more limited on some implementation details. It states that training starts with a pretrained SAM and uses selective parameter freezing to retain SAM’s pre-trained capabilities, but it does not specify the exact freeze or unfreeze map, and it does not report the precise SAM backbone variant such as ViT-B, ViT-L, or ViT-H. Likewise, the exact mathematical form of CMSM is not given in the paper.

3. Training pipeline and ambiguity synthesis

The training pipeline is tailored to the fact that UAT requires multiple plausible masks per image, whereas automotive datasets typically provide only a single annotation. The paper therefore simulates ambiguous supervision on CamVid by first applying a random weather filter to original images and then generating extra masks by elastic deformation. The weather filter is randomly selected from fog, rain, or snow, with strength sampled from 0 to 1, where 0 denotes a clear image and 1 a completely obscured image. This preprocessing is used for both training and testing on obscured images (Ravindran et al., 5 Sep 2025).

For all 1,419 human-segmented CamVid masks, the paper generates three additional annotations by elastic deformation. The mechanism randomly shifts pixel locations in $x$ and $y$, smooths the shift field with a Gaussian filter, and controls the deformation through $P$0 and $P$1. Three parameter sets are defined: fog-like deformations with $P$2 and $P$3, rain-like deformations with $P$4 and $P$5, and snow-like deformations with $P$6 and $P$7. Each image thus has one original human annotation and three deformed annotations, for a total of four segmentation masks per image.

The paper further narrows the task through instance cropping to focus on car segmentations. This makes UAT-SAM an object-patch segmentation method in the reported experiments rather than a full-scene semantic segmentation system. Training begins from pretrained SAM, applies selective parameter freezing and gradual adaptation, monitors Dice and IoU, uses early stopping to prevent overfitting, and uses TensorBoard for monitoring. The reported UAT-SAM section does not provide learning rate, optimizer, batch size, epoch count, scheduler, or exact prompt format.

The optimization description is similarly partial. The paper states that training incorporates a tailored loss function, primarily the Dice Coefficient Loss, to handle segmentation, and that the method uses a probabilistic latent uncertainty code $P$8 from a CVAE injected through CMSM. However, the paper does not provide the full UAT loss equation, the CVAE evidence lower bound, the KL term, or explicit posterior and prior formulas.

4. Relation to the paper’s uncertainty-aware SAM2 branch

The same paper proposes a second uncertainty-aware route based on SAM2, but its mechanism differs at every level from UAT-SAM. The SAM2 approach is built on SAM2 rather than SAM, is aimed at overall scene recognition or overall segmentation quality, uses repeated stochastic forward passes, computes pixel-wise standard deviation as uncertainty, and injects that uncertainty directly into a custom training loss. UAT-SAM, by contrast, is adapter-based, built on SAM, and designed for extreme-weather ambiguous instance segmentation with multiple plausible masks (Ravindran et al., 5 Sep 2025).

This distinction is important because the two methods are sometimes conflated. In the reported formulation, UAT-SAM is not a submodule of the uncertainty-aware SAM2 method, and there is no evidence of joint optimization between them. Their scopes are different as well. The SAM2 branch is evaluated more broadly on BDD100K, CamVid, and discussed with GTA, whereas UAT-SAM’s quantitative evidence is confined to adverse-weather CamVid car-instance segmentation.

A plausible implication is that the paper treats uncertainty in two incompatible but complementary ways. In the SAM2 branch, uncertainty is a variance-weighted training signal. In UAT-SAM, uncertainty is a latent representation that conditions intermediate features. The latter is closer to ambiguous-label learning, while the former is closer to loss-level reliability weighting.

5. Empirical evidence

The reported quantitative evaluation for UAT-SAM uses 177 heavy-weather-filtered CamVid car instance images, with the original human segmentations used as ground truth for evaluation. The baseline is zero-shot SAM, and the reported metrics are Dice coefficient and IoU (Ravindran et al., 5 Sep 2025).

Model	Dice Score	IoU
Zero-shot SAM	0.4809	0.3221
UA-SAM / UAT-SAM	0.6258	0.4598

The paper states that these results correspond to about 30% increase in Dice and about 42.7% increase in IoU. These are the main quantitative results specifically attributable to UAT-SAM in that study.

The qualitative findings sharpen the interpretation of those numbers. Under heavy weather filtering, standard SAM often fails to segment anything meaningful, fails to contour the target object, and can be halted by rain and snow; in heavy weather it may not localize any object at all. UAT-SAM generally localizes the car better, is more robust in heavily obscured images, and improves instance segmentation in ambiguous conditions. At the same time, the paper is explicit that both UAT-SAM and zero-shot SAM can still fail in particularly ambiguous or noisy regions, and it notes that UAT-SAM can be overconfident even when it localizes the car.

The empirical scope is narrow. No quantitative UAT-SAM table is reported for BDD100K or GTA, no full-scene semantic-segmentation metrics are given for UAT-SAM, and no calibration metrics such as ECE, NLL, or Brier score are reported, despite the uncertainty-centered framing.

6. Limitations, interpretation, and disambiguation

The reported limitations are substantial. The paper does not present formal ablations for adapter design, CMSM versus simpler fusion, latent uncertainty terms, freezing policy, insertion depth, prompt strategy, number of ambiguous masks, weather-augmentation severity, or sensitivity to the deformation parameters beyond the stated settings. It also omits the exact SAM backbone, optimizer, learning rate, batch size, epoch count, and exact prompt protocol for UAT-SAM (Ravindran et al., 5 Sep 2025).

These omissions affect reproducibility and interpretation. The reported gains establish a proof of concept that explicit uncertainty-aware adaptation can make SAM more robust to severe visual degradation, but they do not isolate how much of the improvement comes from the adapter architecture, the synthetic ambiguity generation, the weather filtering, or the car-instance cropping. A plausible implication is that the current evidence supports UAT-SAM most strongly as a targeted adverse-weather demonstration rather than a general uncertainty-aware SAM recipe.

The term also requires disambiguation. It is unrelated to “UAT: Unified Audio-Text Diffusion,” where UAT stands for Unified Audio-Text Diffusion and the paper explicitly states that there is no explicit component named “SAM” in the architecture, diagrams, equations, experiments, or appendices (Wang et al., 3 Jun 2026). It is also unrelated to automated user acceptance testing systems such as XUAT-Copilot, where “UAT” refers to user acceptance testing rather than uncertainty-aware adaptation (Wang et al., 2024).

In the current literature considered here, UAT-SAM therefore denotes a specific uncertainty-aware adapter insertion into SAM for extreme-weather, ambiguous car-instance segmentation. Its defining characteristics are adapter insertion into each transformer block, CVAE-derived latent uncertainty, CMSM-based feature modulation, training with multiple plausible masks generated through elastic deformation, and empirical gains over zero-shot SAM on heavy-weather CamVid car instances (Ravindran et al., 5 Sep 2025).