AutoSAME: Adapting SAM for Driving Segmentation

• AutoSAME is a domain-adaptation strategy that modifies SAM for dense semantic segmentation in autonomous driving by combining a frozen global encoder with trainable local modules.
• It employs a dual-encoder architecture with deformable fusion and a progressive decoder to preserve fine spatial details through multi-scale, high-resolution processing.
• A hybrid loss function—integrating Focal, Dice, Lovasz, and Surface losses—ensures robust optimization to handle class imbalance, boundary precision, and domain shifts.

AutoSAME refers to automatically adapting the Segment Anything Model (SAM) for dense, domain-specific segmentation in autonomous driving without rearchitecting SAM from scratch. In the reported formulation, AD-SAM operationalizes AutoSAME by retaining SAM’s pretrained ViT-H image encoder for global semantics and adding trainable autonomous-driving-specific components—a ResNet-50 local encoder, deformable multi-scale fusion, a progressive deformable decoder, and a hybrid objective tailored to road-scene class imbalance and boundary precision. The resulting system is designed for dense semantic segmentation in road scenes, with emphasis on data efficiency, cross-domain robustness, and preservation of fine spatial detail through high-resolution, multi-scale processing (Camarena et al., 30 Oct 2025).

1. Definition and problem setting

AutoSAME is motivated by the mismatch between SAM’s original operating regime and the requirements of autonomous-driving perception. SAM is designed for promptable, instance-level segmentation, whereas autonomous driving requires dense semantic segmentation with robust handling of small objects such as poles and signs, thin and long structures such as lanes and curbs, and diverse contexts including weather, lighting, and occlusion. Road scenes also exhibit severe class imbalance, frequent domain shifts such as city-to-city and day-to-night variation, and a requirement for high boundary fidelity for planning, particularly for drivable area and sidewalks (Camarena et al., 30 Oct 2025).

Within this framing, AutoSAME denotes an adaptation strategy rather than a full replacement of SAM. The reported instantiation freezes SAM’s ViT-H encoder to preserve broad semantics learned on SA-1B and introduces auxiliary trainable modules for local spatial modeling, heterogeneous feature alignment, and dense decoding. The pipeline maintains high-resolution inputs of $1024 \times 1024$ and multi-scale feature usage in order to preserve fine spatial detail critical for autonomous driving. It also uses a hybrid loss that combines Focal, Dice, Lovasz-Softmax, and Surface losses to address class imbalance, region overlap, IoU optimization, and boundary accuracy.

A central technical point is that the model is not prompt-driven at inference time. The prompt strategy is explicitly “No prompting”; dense semantic segmentation is performed by leveraging frozen SAM encoder features together with trainable downstream modules. This distinguishes AutoSAME, as instantiated by AD-SAM, from prompt-centric SAM usage patterns.

2. Architectural composition

The AutoSAME architecture is organized around a dual-encoder backbone, a deformable fusion module, and a progressive deformable-attention decoder. Input images are resized to $1024 \times 1024$ and processed by a frozen ViT-H image encoder from SAM, yielding a $64 \times 64$ grid of 256-dimensional embeddings,

$$F_{\text{SAM}} \in \mathbb{R}^{B \times 64 \times 64 \times 256},$$

with the $1024 \times 1024$ input downsampled by 16 (Camarena et al., 30 Oct 2025).

Parallel to this global branch, a trainable ResNet-50 supplies local spatial detail through a four-scale feature pyramid. The scales are $F_1$ at stride $1/4$ with 256 channels, $F_2$ at stride $1/8$ with 512 channels, $F_3$ at stride $1024 \times 1024$0 with 1024 channels, and $1024 \times 1024$1 at stride $1024 \times 1024$2 with 2048 channels. Each scale is projected to 256 channels via $1024 \times 1024$3 convolution and bilinearly upsampled to $1024 \times 1024$4 so that it can be aligned with $1024 \times 1024$5. The projected ResNet features are then concatenated with SAM features and fused with deformable operations at each scale.

The deformable fusion module predicts offsets $1024 \times 1024$6 and modulation masks $1024 \times 1024$7 for a $1024 \times 1024$8 kernel with $1024 \times 1024$9. Feature aggregation is performed as

$64 \times 64$0

where $64 \times 64$1 are canonical kernel offsets, $64 \times 64$2 are learnable kernel weights, and $64 \times 64$3 modulates per-sample contribution. Channel attention then recalibrates the fused representation: $64 \times 64$4 with fused output

$64 \times 64$5

using reduction ratio $64 \times 64$6 in the shared MLP.

After per-scale fusion, the model concatenates the outputs across scales to obtain

$64 \times 64$7

This tensor is processed by a three-stage progressive decoder. Stage 1 applies DeformConv$64 \times 64$8 with GroupNorm(32), GELU, and Dropout(0.1). Stage 2 applies DeformConv$64 \times 64$9 with GroupNorm(16), GELU, and Dropout(0.1). Stage 3 applies DeformConv$$F_{\text{SAM}} \in \mathbb{R}^{B \times 64 \times 64 \times 256},$$0 with GroupNorm(8), GELU, and Dropout(0.1). A final $$F_{\text{SAM}} \in \mathbb{R}^{B \times 64 \times 64 \times 256},$$1 deformable convolution maps 64 channels to 19 semantic classes.

Mechanistically, the paper relates deformable convolution to content-dependent sampling analogous to attention and notes a common deformable-attention form,

$$F_{\text{SAM}} \in \mathbb{R}^{B \times 64 \times 64 \times 256},$$2

while emphasizing that AD-SAM realizes this principle through deformable convolutions rather than multi-head attention. The stated rationale is lower overhead with geometry-aware aggregation. The paper also notes that the GroupNorm group counts decrease from 32 to 16 to 8 to maintain feature diversity while stabilizing optimization during progressive refinement. This suggests that AutoSAME treats geometric adaptability and optimization stability as coupled design constraints rather than independent modules.

3. Objective functions and optimization regime

AutoSAME uses a hybrid objective with fixed weights: $$F_{\text{SAM}} \in \mathbb{R}^{B \times 64 \times 64 \times 256},$$3 Each term is assigned a distinct role in the reported formulation: Focal loss addresses class imbalance, Dice loss emphasizes region overlap, Lovasz-Softmax serves as a surrogate for mIoU, and Surface loss is boundary-aware through distance transforms (Camarena et al., 30 Oct 2025).

The Focal term is given by

$$F_{\text{SAM}} \in \mathbb{R}^{B \times 64 \times 64 \times 256},$$4

with $$F_{\text{SAM}} \in \mathbb{R}^{B \times 64 \times 64 \times 256},$$5 and $$F_{\text{SAM}} \in \mathbb{R}^{B \times 64 \times 64 \times 256},$$6. The Dice term is

$$F_{\text{SAM}} \in \mathbb{R}^{B \times 64 \times 64 \times 256},$$7

The Lovasz-Softmax component is

$$F_{\text{SAM}} \in \mathbb{R}^{B \times 64 \times 64 \times 256},$$8

and the Surface term is

$$F_{\text{SAM}} \in \mathbb{R}^{B \times 64 \times 64 \times 256},$$9

where $1024 \times 1024$0 is a signed distance transform to object boundaries. The paper states that AD-SAM uses a generalized surface loss that reduces Hausdorff distance-like errors at boundaries.

The training configuration is equally specific. Inputs are resized to $1024 \times 1024$1 and normalized with ImageNet mean and standard deviation. The optimizer is AdamW with base learning rate $1024 \times 1024$2, weight decay $1024 \times 1024$3, cosine annealing, no warm-up, 100 epochs, and batch size 2/GPU; gradient clipping is not used. SAM’s ViT-H encoder remains frozen, while ResNet-50, the fusion modules, decoder, and head are trainable. Learning-rate multipliers are 0.1 for ResNet-50 and 1.0 for the fusion and decoder components. Mixed precision is enabled with torch.cuda.amp. The reported software and hardware stack is PyTorch 2.7.0+cu118, torchvision 0.20.0, Ubuntu 22.04, and a single RTX 4090 (24GB). The baseline uses no data augmentation.

4. Datasets, protocol, and evaluation criteria

The reported evaluation uses two standard autonomous-driving segmentation benchmarks. Cityscapes consists of high-resolution urban scenes at $1024 \times 1024$4, 19 classes, and fine annotations for 5,000 images split into 2,975 training, 500 validation, and 1,525 test images. Coarse labels are excluded. BDD100K is described as a diverse driving dataset with $1024 \times 1024$5 videos; its segmentation subset contains 10,000 keyframes split into 7,000 training, 1,000 validation, and 2,000 test images, with 19 classes. The pipeline retained 5,968 valid image–mask pairs for training on BDD100K (Camarena et al., 30 Oct 2025).

The principal evaluation metric is mean Intersection over Union. Per-class IoU and mIoU are defined as

$1024 \times 1024$6

For cross-domain experiments from Cityscapes to BDD100K, the paper additionally defines a retention score: $1024 \times 1024$7

The forward pass is presented as a concise AutoSAME workflow. It consists of preprocessing; dual-encoder feature extraction with frozen ViT-H and trainable ResNet-50; deformable fusion with learned offsets and modulation; multi-scale concatenation into $1024 \times 1024$8; a three-stage progressive decoder; loss computation with the four-term hybrid objective; and evaluation via per-class IoU, mIoU, and retention for cross-domain experiments. This suggests that AutoSAME is intended as a structured adaptation recipe in which feature extraction, geometry-aware alignment, dense decoding, and optimization are jointly specified.

5. Reported empirical behavior

The reported results show that AD-SAM surpasses SAM, G-SAM, and DeepLabV3 on BDD100K, and surpasses SAM and DeepLabV3 while remaining essentially on par with G-SAM on Cityscapes (Camarena et al., 30 Oct 2025).

Benchmark	Reported mIoU
Cityscapes	AD-SAM 68.14; SAM 52.82; G-SAM 68.20; DeepLabV3 45.22
BDD100K	AD-SAM 59.50; SAM 40.26; G-SAM 46.06; DeepLabV3 43.03
Retention	AD-SAM 0.8732; SAM 0.7622; G-SAM 0.6754; DeepLabV3 0.9516

On Cityscapes, the stated margins are $1024 \times 1024$9 versus DeepLabV3 and $F_1$0 versus SAM, while the comparison with G-SAM is described as “on-par” at 68.14 versus 68.20. On BDD100K, the margins are $F_1$1 versus SAM, $F_1$2 versus G-SAM, and $F_1$3 versus DeepLabV3. The abstract summarizes these gains as up to $F_1$4 mIoU in structured road scenes and $F_1$5 mIoU in diverse road scenes.

The paper also reports favorable sample efficiency and optimization dynamics. With only 1,000 labeled Cityscapes samples, AD-SAM reaches 0.607 mIoU, compared with 0.561 for G-SAM, 0.471 for SAM, and 0.432 for DeepLabV3. It converges within 30–40 epochs and exhibits smoother validation loss and mIoU curves than SAM and G-SAM, especially on BDD100K. Training-time comparisons are said to indicate a favorable accuracy-efficiency balance as dataset size increases, although inference speed, latency, and memory footprint are not explicitly reported.

Per-class behavior follows a familiar pattern for road-scene segmentation. AD-SAM reports high mIoU on large or coherent classes, including road (0.976/0.924), building (0.900/0.838), sky (0.927/0.945), vegetation (0.894/0.843), and car (0.918/0.908). Performance is mid-range on smaller or more dynamic classes such as pole (0.386/0.439), motorcycle (0.470/0.549), bicycle (0.692/0.498), and signs or lights. The rare class train is reported as 0.000 on BDD100K due to sparsity. The paper attributes improvements qualitatively to the combination of frozen ViT-H semantics with trainable ResNet-50 local features, geometry-aware deformable fusion, progressive refinement, and the hybrid loss. A notable nuance is that DeepLabV3 has the highest reported retention score, 0.9516, despite lower absolute mIoU; this suggests that retention and absolute segmentation quality are not interchangeable criteria.

6. Limits, interpretations, and prospective extensions

The reported limitations are concentrated in exactly the regimes that often stress dense autonomous-driving perception: rare classes, small objects, domain shift, and deployment constraints. Rare or small object classes remain challenging, with explicit mention of train in BDD100K, poles, and riders. Proposed directions include class-aware sampling and instance-level refinement, weakly supervised or self-training approaches for night and weather shifts, unlabeled domain adaptation, and multimodal fusion with LiDAR or Radar (Camarena et al., 30 Oct 2025).

The paper also identifies a boundary between semantic segmentation and broader autonomous-driving perception stacks. Suggested extensions include joint panoptic or instance segmentation and integration with tracking and detection. For embedded deployment, the stated directions are pruning, distillation, or lighter decoders for real-time inference on resource-constrained hardware. Code and model availability are not provided; the paper does not reference a public repository, model weights, or license information.

Several interpretive clarifications follow directly from the reported evidence. First, AutoSAME should not be reduced to prompt engineering, since the implementation uses no prompting. Second, it is not equivalent to end-to-end retraining of SAM, because ViT-H remains frozen and adaptation is delegated to trainable auxiliary modules. Third, the results do not show universal dominance over every comparator on every benchmark: on Cityscapes, G-SAM is slightly higher in mIoU, whereas on BDD100K AD-SAM is clearly higher. A plausible implication is that AutoSAME is best understood as a domain-adaptation pattern for SAM-based dense segmentation in autonomous driving, with its effectiveness depending on the interaction among frozen global semantics, trainable local detail recovery, geometry-aware fusion, and a boundary-sensitive optimization objective.