Adaptive Semantic-Aware Mask (AdaSAM)

• The paper demonstrates that AdaSAM masks semantically rich, textured regions to force learning of both generative and content priors in low-level image restoration.
• It employs a multi-head attention mechanism and pixel-level importance propagation to adaptively generate masks, improving reconstruction over random masking.
• Empirical results show notable improvements in PSNR and SSIM, confirming AdaSAM’s effectiveness in handling diverse and unseen degradation scenarios.

Searching arXiv for AdaSAM and closely related methods to ground the article in current papers. Adaptive Semantic-Aware Mask (AdaSAM) most specifically denotes the pre-training masking strategy introduced in RAM++ for all-in-one image restoration, where pixel-level masks are applied to semantically rich and textured regions rather than sampled uniformly at random (Zhang et al., 15 Sep 2025). In that formulation, AdaSAM adapts Masked Image Modeling to low-level vision by forcing reconstruction of high-information regions under degradation, with the stated goal of enabling the network to learn both generative priors and image content priors from various degradations. The term also sits within a broader family of adaptive semantic-aware mechanisms in promptable segmentation and medical imaging, including CAM-derived self-prompting in ADA-SAM, mask-conditioned iterative refinement in M-SAM, and feature-level semantic conditioning in CLIP-Guided SAM, although those methods implement different mechanisms and do not use the same masking procedure (Ward et al., 2 Jul 2025).

1. Definition and problem setting

In RAM++, AdaSAM is the first-stage pre-training strategy in a two-stage framework whose second stage uses Mask Attribute Conductance (MAC) and Robust Feature Regularization (RFR) (Zhang et al., 15 Sep 2025). Its function is to construct a non-random mask over degraded images so that the restoration backbone is trained on regions that are semantically informative and structurally complex. The paper explicitly states that AdaSAM masks “semantically rich and textured regions” and is intended to “enable the network to learn both generative priors and image content priors from various degradations” (Zhang et al., 15 Sep 2025).

The motivation is framed against standard MIM strategies imported from high-level vision. Three deficiencies are emphasized. First, random masking can be too easy to solve in low-level image restoration, especially with pixel interpolation or local continuity. Second, large patch masking loses fine texture details, which are critical for restoration. Third, restoration should not only learn to remove degradations, but also learn intrinsic image content priors and generative priors for reconstructing missing structures under many degradations (Zhang et al., 15 Sep 2025).

AdaSAM therefore adopts a content-oriented rather than degradation-oriented perspective. Instead of organizing learning around degradation categories such as haze, rain, or noise, it masks high-information regions around main subjects and rich textures. This suggests that AdaSAM is intended to make restoration features sensitive to image content that must be preserved or regenerated, rather than primarily to degradation signatures.

2. Construction of the mask

AdaSAM takes a degraded image

$I_d \in \mathbb{R}^{C \times H \times W}$

and generates a pixel-level mask through four stated components: Patch Embedding, Multi-Head Attention, Pixel-level Importance Propagation, and Categorical Sampling (Zhang et al., 15 Sep 2025).

The image is first divided into non-overlapping tokens, with

$N = \frac{H}{h} \times \frac{W}{w},$

where $h,w$ are patch dimensions. These tokens form $T$, which are then passed through a multi-head attention module. Token-wise importance scores are computed as

$$S = \mathrm{Softmax}(\mathrm{Linear}(\mathrm{Atten}(T))).$$

The paper interprets $S$ as a measure of relational dependency among tokens: tokens that are more informative in context receive higher scores (Zhang et al., 15 Sep 2025).

A central design choice is that importance is estimated at token level but used for pixel-level masking. Each token’s importance score is uniformly propagated to all pixels inside that token’s spatial region, producing a pixel-level importance map $s$. The number of masked pixels is determined by a predefined mask ratio $\rho$,

$N_{\text{mask}} = H \times W \times \rho,$

and masked pixels are then sampled by multinomial sampling without replacement from $s$. The resulting binary mask $N = \frac{H}{h} \times \frac{W}{w},$0 is adaptive because it depends on semantic importance, semantic-aware because the importance originates from attention-based scores, pixel-level because selection occurs at pixel granularity, and stochastic because sampling introduces diversity (Zhang et al., 15 Sep 2025).

The targeted regions are described in several equivalent ways: semantically rich regions, texturally rich regions, high-information regions, regions around the main subjects, and regions that are structurally complex and difficult to reconstruct. The paper also states that AdaSAM masks “the degraded images’ semantically and texturally rich regions (i.e., high-information regions) at the pixel level with a 50% masking ratio” (Zhang et al., 15 Sep 2025).

3. Optimization and training dynamics

AdaSAM is trained jointly with a restoration network $N = \frac{H}{h} \times \frac{W}{w},$1 and a mask predictor with parameters $N = \frac{H}{h} \times \frac{W}{w},$2 (Zhang et al., 15 Sep 2025). The restoration objective is an $N = \frac{H}{h} \times \frac{W}{w},$3 loss applied to the masked part: $N = \frac{H}{h} \times \frac{W}{w},$4 where $N = \frac{H}{h} \times \frac{W}{w},$5 is the clean target image, $N = \frac{H}{h} \times \frac{W}{w},$6 is the AdaSAM mask, $N = \frac{H}{h} \times \frac{W}{w},$7, and $N = \frac{H}{h} \times \frac{W}{w},$8 denotes stop-gradient or detach.

The mask network is optimized with a weighted reconstruction loss: $N = \frac{H}{h} \times \frac{W}{w},$9 Here $h,w$0 is the pixel-level mask score map, and $h,w$1 is used to stabilize training. The optimization is described as adversarial-like: AdaSAM tries to choose harder regions to mask, while the restoration network tries to reconstruct them (Zhang et al., 15 Sep 2025).

Several implementation details are specified. Pre-training uses paired degraded-clean data; the paper notes that pre-training with only clean images is weaker. AdaSAM uses a mask ratio $h,w$2, the learning rate for AdaSAM is decayed from $h,w$3 to $h,w$4, and the loss weight for AdaSAM is set to $h,w$5 (Zhang et al., 15 Sep 2025). The stated effect of this setup is that the network learns intrinsic image priors and encodes diverse degradations into a more unified and semantically consistent feature space.

4. Empirical behavior and ablation evidence

The masking strategy comparison directly contrasts AdaSAM with two random masking baselines. The reported results are Random$h,w$6: 28.89 PSNR / 0.8967 SSIM, Random$h,w$7: 29.01 PSNR / 0.8968 SSIM, and AdaSAM: 29.46 PSNR / 0.8993 SSIM (Zhang et al., 15 Sep 2025). Within the paper’s framing, this is evidence that semantic-aware pixel masking is more effective than either random large-patch masking or fixed pixel-level random masking.

A second ablation isolates the contribution of AdaSAM within RAM++. The table reports w/o AdaSAM and RFR: 26.01 PSNR / 0.8007 SSIM, w/o AdaSAM: 28.40 / 0.8811, w/o RFR: 28.55 / 0.8897, and Full RAM++: 28.88 / 0.8895 (Zhang et al., 15 Sep 2025). The paper attributes AdaSAM’s gains to forcing the model to focus on hard-to-reconstruct textured regions, building a unified latent space across degradations, and enhancing robustness in both seen and unseen degradation scenarios.

Two clarifications are important. First, AdaSAM is not defined as a generic random-mask regularizer; its defining property is that masking probabilities are biased by semantic importance. Second, the reported benefits are tied to the RAM++ training regime based on paired degraded-clean data and the subsequent fine-tuning stage with MAC and RFR. A plausible implication is that AdaSAM’s contribution should be interpreted as part of a content-oriented representation-learning pipeline rather than as an isolated masking heuristic.

5. Relation to SAM-based adaptive semantic guidance

The phrase “adaptive semantic-aware” also appears in adjacent SAM literature, but the mechanisms differ substantially.

Method	Semantic mechanism	Task domain
AdaSAM in RAM++	pixel-level masking of semantically rich and textured regions	all-in-one image restoration
ADA-SAM	Grad-CAM region converted into a bounding-box prompt for SAM	medical MRI segmentation
M-SAM	coarse mask encoded and fused back into image representation	3D tumor lesion segmentation
CLIP-Guided SAM	CLIP text, vision, and similarity features injected into SAM’s image encoder	promptable segmentation

ADA-SAM is a semi-supervised, multitask framework for medical MRI segmentation of quadriceps muscles, specifically vastus lateralis and vastus medialis (Ward et al., 2 Jul 2025). It runs an auxiliary classifier, derives class activation maps from that classifier, converts the salient region into a bounding-box prompt for SAM, and then uses segmentation gradients as feedback to refine the shared representation. The prompt is therefore adaptive because it is generated dynamically from the image and semantic-aware because it is grounded in class-discriminative activations.

M-SAM, by contrast, is a medical-domain, 3D adaptation of SAM-Med3D with an explicit adaptive mask feedback loop for tumor lesion segmentation (Shi et al., 2024). Its Mask-Enhanced Adapter is inserted between the image encoder and the mask decoder, takes an image embedding and a mask embedding derived from the previous mask estimate, and produces an updated image embedding enriched with mask-derived semantic and positional cues. The refinement loop starts from an all-zero mask, predicts a coarse mask, re-encodes that mask, and repeats iteratively. In the paper’s own interpretation, the coarse mask acts as an adaptive semantic prior rather than as a static prompt.

CLIP-Guided SAM makes a related but distinct claim: SAM is semantically blind unless semantic information influences feature formation before mask decoding (Jalilian et al., 24 May 2026). Instead of using semantics only to generate prompts, it injects CLIP-derived text, vision, and similarity features directly into SAM’s image encoder via lightweight semantic adapters. The similarity signal is reported as the strongest among the semantic cues, and the method emphasizes train-test prompt consistency, especially in text-only inference.

These comparisons show that “semantic-aware” can refer to at least three different loci of adaptation: masked pre-training over pixels, automatic prompt generation, and internal feature conditioning. AdaSAM in the strict RAM++ sense belongs to the first category.

6. Conceptual significance, scope, and limitations

AdaSAM’s conceptual contribution is to redefine masking in low-level restoration as a semantic sampling problem rather than a uniform occlusion problem (Zhang et al., 15 Sep 2025). Standard random masking ignores image semantics, whereas AdaSAM first estimates which regions are semantically meaningful and then uses those estimates to bias pixel selection. The paper explicitly presents this as a combination of high-level semantics from attention with low-level pixel masking.

This positioning also clarifies a common misconception. AdaSAM is not primarily a method for identifying degradations; the paper contrasts it with degradation-oriented approaches that focus on degradation types or feature spaces separated by degradation categories. Instead, AdaSAM is presented as teaching “what image content should look like under missing or corrupted regions,” whereas degradation-oriented masking teaches “what degradation looks like” (Zhang et al., 15 Sep 2025). That distinction is central to the method’s claimed robustness in unseen, extreme, and mixed degradations.

At the same time, the term should not be universalized across all adaptive semantic methods. ADA-SAM, M-SAM, and CLIP-Guided SAM all use adaptive semantic signals, but they operate through CAM-derived bounding boxes, recurrent mask-conditioned refinement, or feature-level CLIP injection rather than through AdaSAM’s attention-guided pixel masking (Ward et al., 2 Jul 2025). This suggests that “Adaptive Semantic-Aware Mask” is best treated as a method-specific term anchored in RAM++, and only analogically extended to related architectures when the extension is made explicit.