AD-SAM: Autonomous Driving Segmentation
- The paper introduces a supervised semantic segmentation framework that leverages a frozen SAM encoder alongside a trainable ResNet-50 and deformable feature fusion to enhance road-scene perception.
- AD-SAM is a family of adaptation strategies that repurpose SAM for autonomous driving by adding semantic branches, multimodal inputs, and uncertainty-aware techniques to address adverse conditions.
- Empirical results on benchmarks like Cityscapes and BDD100K demonstrate that AD-SAM outperforms vanilla SAM and traditional models, offering improved data efficiency, robustness, and local detail recovery.
Autonomous Driving Segment Anything Model (AD-SAM) denotes a line of research that adapts the Segment Anything Model (SAM) and related SAM-family models to autonomous-driving perception, and it is also the explicit name of a supervised semantic segmentation architecture introduced for road-scene understanding in 2025. Across this literature, the central problem is consistent: SAM is attractive because it is a large vision foundation model with promptable segmentation and strong mask generation, but autonomous driving requires more than generic mask extraction. Road-scene perception demands dense semantics, multi-scale spatial precision, robustness to adverse weather and degradations, compatibility with multimodal sensing, and reliable behavior in safety-critical edge cases such as occlusion, concealment, and reflective or transparent surfaces (Shan et al., 2023, Camarena et al., 30 Oct 2025).
1. Conceptual scope and problem formulation
In autonomous driving, SAM is rarely used in its original form as a complete perception system. The literature instead repurposes it in several distinct ways. One use treats SAM as a zero-shot or near-zero-shot mask generator and adds a semantic branch on top; another uses SAM as a pretrained encoder inside a supervised road-scene segmentation architecture; a third uses SAM as an auxiliary module for pseudo-label refinement, annotation generation, tracking initialization, or graph extraction. This suggests that AD-SAM is better understood as a family of specialization strategies than as a single canonical model.
A basic conceptual constraint appears repeatedly: SAM outputs masks, not semantic labels. In road scenes, this is insufficient because perception must distinguish road from sidewalk, drivable from non-drivable area, vehicle from background, and rare but safety-critical classes such as traffic signs, riders, and motorcycles (Shan et al., 2023). At the same time, autonomous-driving scenes exhibit strong multi-scale structure: large “stuff” regions such as road, sky, and building coexist with thin structures and small objects such as poles, traffic lights, bicycles, and distant pedestrians (Camarena et al., 30 Oct 2025).
Representative AD-SAM usages can be organized as follows.
| Mode | Mechanism | Representative work |
|---|---|---|
| Zero-shot semantic wrapper | SAM or MobileSAM combined with CLIP and a semantic branch, then semantic voting over SAM masks | (Yan et al., 2024) |
| Supervised specialization | SAM-family encoder adapted with task-specific fusion, decoder, or uncertainty-aware training | (Camarena et al., 30 Oct 2025) |
| Auxiliary prior or data engine | Frozen SAM masks used for UDA pseudo-label refinement or dense-label generation | (Yan et al., 2023) |
The literature therefore does not support the view that “segment anything” can simply be dropped into an autonomous-driving stack. Rather, it supports the narrower claim that SAM-style priors are useful when coupled to driving-specific semantics, feature fusion, or training protocols.
2. Why vanilla SAM is insufficient for road-scene perception
The most direct diagnostic evidence comes from the adverse-weather robustness study on BDD100k. That work evaluates vanilla SAM in automatic mask-generation mode on 100 validation images under synthetic corruptions, then uses an oracle matching protocol in which, for each ground-truth mask, the SAM-generated mask with highest IoU is selected. Under this favorable protocol, clean-weather performance is reported as mIoU $0.7161$, but performance falls with severity to $0.4395$ in rain severity 5, $0.4973$ in snow severity 5, $0.5392$ in frost severity 5, $0.6172$ in fog severity 5, and $0.6394$ in brightness severity 5. The paper interprets the strongest failures as arising from occlusion and blurred objects, and it explicitly warns that direct deployment of zero-shot SAM in safety-critical driving stacks is risky without adaptation (Shan et al., 2023).
A more optimistic result is reported by the zero-shot “Semantic-Segment-Anything” pipeline for Cityscapes, which combines SAM or MobileSAM with CLIP and a semantic branch implemented by SegFormer or OneFormer. In that setting, SAM-OneFormer reaches clean mIoU $80.0$ on the Cityscapes validation set and remains comparatively robust under FGSM and PGD-10 relative to several supervised baselines. However, the same study shows marked degradation under severe zoom blur, Gaussian noise, and pixelate, and it relies on a wrapper in which SAM provides mask geometry while a separate semantic model provides class evidence (Yan et al., 2024).
Failure modes become sharper in studies outside standard road-scene benchmarks. Under unprompted settings, SAM is described as “unskilled” in concealed scenes such as camouflaged objects, and the paper explicitly links SAM’s bottlenecks in granularity and uncertainty to application domains requiring high accuracy, including autonomous driving (Ji et al., 2023). A separate study on glass-related perception finds that SAM often fails to detect transparent objects and many mirror regions, frequently segmenting reflected or transmitted content instead of the physical surface itself. For driving, this is directly relevant to windshields, side windows, storefront glass, bus shelters, transparent barriers, and reflective façades (Han et al., 2023).
These results collectively establish a narrow but important conclusion: vanilla SAM has useful mask priors, but those priors are fragile when road-scene understanding depends on semantics, continuity of large regions, or physically difficult image formation conditions.
3. Adaptation strategies beyond vanilla SAM
A major adaptation route inserts SAM into an existing autonomous-driving learning pipeline without turning it into a standalone semantic segmenter. SAM4UDASS is the clearest example. It addresses unsupervised domain adaptation for driving segmentation by using frozen SAM automatic masks to refine target-domain pseudo-labels. Its key modules are Semantic-Guided Mask Labeling (SGML), which assigns semantics to unlabeled SAM masks using class-size priors and a road assumption, and fusion strategies that handle granularity mismatch between SAM masks and target-domain semantics. On GTA5 Cityscapes, DAFormer improves from $68.3$ to $71.3$ mIoU and MIC from $0.4395$0 to $0.4395$1; on Cityscapes $0.4395$2 ACDC, DAFormer improves from $0.4395$3 to $0.4395$4 and MIC from $0.4395$5 to $0.4395$6. The method is therefore less a full AD-SAM architecture than a training-time mask-prior mechanism for domain shift (Yan et al., 2023).
A second route makes SAM multimodal. FusionSAM is designed for aligned visible–infrared pairs in autonomous driving and introduces Latent Space Token Generation (LSTG) and Fusion Mask Prompting (FMP). Instead of using SAM as a generic single-image segmenter, it learns multimodal latent tokens, fuses them with cross-attention, and uses the resulting fusion features as prompts for precise pixel-level segmentation. On MFNet, FusionSAM reports $0.4395$7 mIoU versus $0.4395$8 for SAM and $0.4395$9 for SAM2; on FMB, it reports $0.4973$0 versus $0.4973$1 and $0.4973$2, respectively (Li et al., 2024).
A third route specializes SAM to object-centric, semantically rich subproblems. “Segment Any Vehicle” (SAV) is not a full-scene AD parser, but it is a prompt-free, semantically grounded vehicle-part segmenter built from an OVSAM-based encoder-decoder, a vehicle-part knowledge graph, and a retrieval-based visual context module. It predicts 13 semantic part masks and reaches $0.4973$3 mIoU / $0.4973$4 mAcc on VehicleSeg10K, outperforming the next-best mIoU by $0.4973$5 on that benchmark (Wang et al., 6 Aug 2025).
| Method | Core adaptation | Key reported result |
|---|---|---|
| SAM4UDASS | SGML plus fusion inside UDA self-training | $0.4973$6 on GTA5 $0.4973$7 Cityscapes with DAFormer |
| FusionSAM | RGB–infrared latent-token fusion and learned prompt generation | $0.4973$8 mIoU on MFNet |
| SAV | Knowledge-graph and retrieval-augmented prompt-free part decoding | $0.4973$9 mIoU on VehicleSeg10K |
Taken together, these works indicate that “AD-SAM” usually means adding semantics, multimodality, graph priors, or domain adaptation around SAM rather than merely prompting the original model more carefully.
4. The named AD-SAM architecture
The paper explicitly titled “AD-SAM: Fine-Tuning the Segment Anything Vision Foundation Model for Autonomous Driving Perception” defines AD-SAM as a supervised semantic segmentation framework built around a frozen SAM ViT-H image encoder, a trainable ResNet-50 backbone, deformable feature fusion, and a three-stage deformable decoder. The input image is resized to $0.5392$0. The SAM branch produces $0.5392$1 embeddings from a $0.5392$2 image with downsampling factor 16, while the ResNet-50 branch extracts four scales of features, projects them to 256 dimensions, and upsamples them to the same $0.5392$3 resolution. Fusion uses deformable convolution of the form
$0.5392$4
followed by channel attention, and the decoder performs progressive refinement through three stages: DeformConv(1024→256), DeformConv(256→128), and DeformConv(128→64). Training uses a hybrid loss
$0.5392$5
with AdamW, base learning rate $0.5392$6, weight decay $0.5392$7, cosine annealing, 100 epochs, and no additional data augmentation in the baseline configuration (Camarena et al., 30 Oct 2025).
The reported benchmarks are Cityscapes and BDD100K. In the main results table, AD-SAM attains $0.5392$8 mIoU on Cityscapes and $0.5392$9 on BDD100K. On Cityscapes, this is essentially on par with G-SAM’s $0.6172$0, while exceeding vanilla SAM’s $0.6172$1 and DeepLabV3’s $0.6172$2. On BDD100K, AD-SAM exceeds G-SAM’s $0.6172$3, SAM’s $0.6172$4, and DeepLabV3’s $0.6172$5. The same paper reports a retention score of $0.6172$6 for Cityscapes-to-BDD100K transfer, convergence around epochs 30–40 on BDD100K, and $0.6172$7 mIoU with only 1,000 Cityscapes samples, which it interprets as evidence of data efficiency (Camarena et al., 30 Oct 2025).
This specific AD-SAM model therefore represents one clear answer to the broader research question: instead of using SAM as a promptable mask generator, reuse its pretrained global semantic priors inside a dense road-scene segmentation architecture that restores local detail and boundary control.
5. Robustness, uncertainty, and adverse-condition specialization
A prominent strand of AD-SAM research addresses the fact that driving images are ambiguous under rain, fog, snow, low light, blur, and sensor noise. One response is uncertainty-aware supervised adaptation. The dual-method study on SAM optimization for adverse weather proposes, first, multistep finetuning of SAM2 with a custom loss built from Binary Cross-Entropy, IoU loss, and Monte Carlo uncertainty estimated from 10 stochastic forward passes; and second, a UAT-SAM variant that inserts Uncertainty-Aware Adapters into each SAM transformer block using a CVAE, Prior Net, Posterior Net, latent code $0.6172$8, and the Condition Modifies Sample Module. On BDD100K/CamVid overall segmentation, finetuned SAM2 reports IoU $0.6172$9 versus $0.6394$0 for zero-shot SAM2 and Dice $0.6394$1 versus $0.6394$2. On 177 heavily weather-filtered CamVid car crops, UA-SAM reports Dice $0.6394$3 and IoU $0.6394$4, versus Dice $0.6394$5 and IoU $0.6394$6 for zero-shot SAM. The paper also notes that UA-SAM can remain overconfident, so robustness gains do not by themselves solve calibration (Ravindran et al., 5 Sep 2025).
A second response is parameter-efficient robustification. GaRA-SAM inserts gated-rank adapters into the frozen SAM image encoder’s key, query, and value projections. Instead of a fixed LoRA rank, it dynamically selects rank-1 components through hierarchical gating, with $0.6394$7, $0.6394$8, and Gumbel-Sigmoid temperature $0.6394$9. The most driving-relevant result is the real-corruption setting: with ViT-L and box prompts, GaRA-SAM-Real, trained on real degraded BDD+LIS images, reaches $80.0$0 on BDD+LIS and $80.0$1 on ACDC, compared with $80.0$2 and $80.0$3 for RobustSAM, yielding the paper’s headline $80.0$4 IoU-point improvement on ACDC (Lee et al., 3 Jun 2025).
Robustness research also includes adversarial failure analysis. Region-Guided Attack (RGA) constructs a Region-Guided Map from SAM’s own segmentation output, deliberately fragmenting large regions and expanding small ones. On SAM-B, it reports mIoU $80.0$5, ASR@50 $80.0$6, and ASR@10 $80.0$7, and it transfers strongly to SAM-L, SAM-H, FastSAM, and Meta AI’s online SAM. For autonomous driving, the relevant implication is direct: large coherent regions such as road and sidewalk, and small critical regions such as pedestrians, poles, and traffic devices, are precisely the structures that region-topology attacks are designed to destabilize (Liu et al., 2024).
The robustness literature therefore converges on a common conclusion. Vanilla SAM-family segmentation is not sufficiently reliable under adverse conditions or structured attacks; explicit uncertainty modeling, parameter-efficient degradation adaptation, and topology-aware robustness evaluation are all active ingredients in current AD-SAM development.
6. Adjacent extensions in the AD-SAM ecosystem
Not all AD-SAM research uses SAM at inference time. Some of the most practically consequential work uses SAM as an annotation or structural prior. A representative example is the ZOD segmentation study, which introduces a SAM-based annotation pipeline that converts Zenseact Open Dataset bounding boxes into dense semantic masks. The pipeline processes over 100,000 frames, manually inspects 6,400 SAM outputs, and accepts a curated 2,300-frame subset, corresponding to a 36% acceptance rate. Those SAM-derived labels are then used to train downstream CLFT and DeepLabV3+ models. On ZOD, CLFT-Hybrid reaches up to $80.0$8 mIoU; on the Iseauto platform, CLFT-Hybrid fusion reaches $80.0$9 mIoU. The low acceptance rate is itself an important result: raw SAM output is useful, but not benchmark-grade autonomous-driving supervision without substantial curation (Tahves et al., 27 May 2026).
Temporal extension is addressed by SAM-Track, which combines SAM for promptable key-frame masks, DeAOT for video mask propagation, and Grounding-DINO for text-based selection. The framework supports click, stroke, and text interaction, periodic insertion of newly appearing objects, and qualitative deployment scenarios that include autonomous driving. On DAVIS-2016 Val it reports 0, and on DAVIS-2017 Test 1. Although it is not an autonomous-driving benchmark paper, it provides a clear template for turning static SAM masks into persistent video object tracks (Cheng et al., 2023).
Structured road representation appears in SAM-Road, which adapts SAM for vectorized road-network graph extraction from satellite imagery. The method fine-tunes a SAM ViT-B encoder, predicts dense road and intersection masks, extracts vertices by thresholding and non-maximum suppression, and predicts topology with a lightweight transformer-based graph decoder. It is not an onboard AD perception system, but its dense-to-graph decomposition is directly relevant to lane graph extraction, HD map construction, and BEV topology reasoning. On City-scale, it is reported as 40 times faster than RNGDet++ while remaining comparable in accuracy, and it attains APLS 2 (Hetang et al., 2024).
These adjacent extensions broaden the meaning of AD-SAM. In practice, SAM can serve as a deployed perception module, a label engine, a temporal initializer, or a source of geometry-aware embeddings for structured road representations.
7. Recurring limitations and open problems
Several limitations recur across the literature. First, many studies remain diagnostic rather than prescriptive. The adverse-weather robustness paper on BDD100k uses only 100 validation images, synthetic corruptions, and an optimistic oracle matching protocol, so it does not establish deployable end-to-end performance (Shan et al., 2023). Second, several adaptation papers remain weak on calibration, runtime, and reproducibility detail. The uncertainty-aware SAM/SAM2 paper does not report calibration metrics such as ECE, Brier score, or risk-coverage, and it does not provide a detailed runtime study despite using 10 stochastic forward passes for uncertainty estimation (Ravindran et al., 5 Sep 2025). Third, annotation-oriented work shows that raw SAM output is still far from plug-and-play in safety-critical road scenes, as indicated by the 36% acceptance rate in the ZOD label-generation pipeline (Tahves et al., 27 May 2026).
Scope limitations are equally important. SAV is highly effective for vehicle-part parsing, but it remains vehicle-only rather than a holistic autonomous-driving scene parser (Wang et al., 6 Aug 2025). SAM-Road operates in satellite imagery and addresses road topology rather than onboard perspective perception (Hetang et al., 2024). Even the specifically named AD-SAM model is still a supervised 2D semantic segmentation system; it does not provide temporal consistency, multi-camera fusion, LiDAR integration, occupancy reasoning, or planning-aware uncertainty logic (Camarena et al., 30 Oct 2025).
Open problems therefore remain substantial. A production-quality AD-SAM would likely need prompt-free dense semantics, strong small-object performance, temporal coherence, multimodal fusion, weather- and attack-aware robustness, richer calibration, and evaluation protocols tied to planner risk rather than mask overlap alone. Current work supports these requirements indirectly, but it does not yet show a single unified system that satisfies them all.
The field nonetheless exhibits a clear trajectory. Generic SAM is useful as a source of mask priors and pretrained visual semantics, but autonomous driving consistently forces it toward specialization: semantic decoders, uncertainty-aware losses, deformable fusion, multimodal prompting, graph reasoning, temporal tracking, and curated label-generation pipelines. In that sense, AD-SAM is best understood not as a single model family already stabilized into a standard form, but as an ongoing effort to convert a general segmentation foundation model into a road-scene perception system that is semantically explicit, structurally aware, and robust enough for safety-critical deployment.