SAM-BG: Foreground-Background Segmentation Challenges
- SAM-BG is defined as the regime where SAM struggles with low-contrast or entangled foregrounds, with studies quantifying performance using metrics like Dice score and Weber contrast.
- Researchers have developed prompt-engineering and test-time control methods, such as FN/FP correction and CAM-derived prompts, to alleviate SAM’s background interference.
- Empirical studies in medical imaging, camouflage, and shadow detection show that explicit foreground–background separation techniques markedly improve segmentation accuracy.
Searching arXiv for papers relevant to “SAM-BG” and SAM/background-foreground behavior. “SAM-BG” is not a standardized model name in the cited literature. As an Editor’s term, it is best understood as a cluster of questions and methods centered on the Segment Anything Model (SAM) when segmentation is governed by foreground–background separation: cases in which the foreground is difficult to distinguish from background, and cases in which SAM-derived masks, prompts, or memory are used to suppress, calibrate, or exploit background structure. Under that reading, the literature spans zero-shot medical segmentation, camouflaged object detection, shadow detection, prompt-sensitive correction schemes, background-suppressed representation learning, and SAM 2 memory mechanisms that explicitly suppress background-contaminated features (He et al., 2023, Tang et al., 2023, Jie et al., 2023, Li et al., 2024, Xu et al., 20 May 2025).
1. Terminology and scope
The literature does not introduce a single canonical “SAM-BG” framework. Instead, it repeatedly returns to the same technical issue: SAM is strongest when objectness, contour, and region separation are clear, and it weakens when the target is small, diffuse, weakly bounded, low-contrast, or visually entangled with surrounding tissue or clutter. A plausible interpretation is that “SAM-BG” denotes this background-governed regime of SAM behavior, together with methods that attempt to control it (He et al., 2023).
Within that interpretation, the relevant work falls into several recurring categories.
| Setting | BG-related role | Representative paper |
|---|---|---|
| Medical zero-shot segmentation | Performance tied to target/background separability | (He et al., 2023) |
| Camouflage and shadow benchmarks | Background confusion defeats generic objectness | (Tang et al., 2023, Jie et al., 2023) |
| Test-time prompt control | Prompt perturbation used to correct FN/FP leakage | (Yao et al., 2023) |
| Learned adaptation | Masks, CAMs, or memory used to suppress background artifacts | (Li et al., 2024, Ward et al., 2 Jul 2025, Xu et al., 20 May 2025, Liu et al., 29 Sep 2025) |
| Positive boundary-separated use cases | Brain/non-brain or player/background isolation can benefit from SAM | (Mohapatra et al., 2023, Ranasinghe et al., 9 Dec 2025) |
This scope matters because the cited papers do not merely report isolated failures. They collectively suggest that the central question is not whether SAM can segment “anything” in the abstract, but whether the foreground can be isolated from background under the assumptions inherited from SAM’s natural-image pretraining.
2. Foreground–background separability as the governing variable
The clearest empirical treatment of the issue appears in the benchmark “Computer-Vision Benchmark Segment-Anything Model (SAM) in Medical Images: Accuracy in 12 Datasets” (He et al., 2023). That study evaluates SAM in strict zero-shot mode on 12 public medical image segmentation datasets involving 7,451 subjects, spanning 10 organs, four 2D modalities—X-ray, colonoscopy, dermoscopy, ultrasound—and two 3D modalities—MRI and CT. It compares three operational variants, SAM-Semantic, SAM-Point, and SAM-Box, against U-Net, U-Net++, Attention U-Net, TransUNet, and UCTransNet. The central finding is unequivocal: SAM underperforms all five medical methods on all 12 datasets, often by 0.1–0.5 Dice and in some datasets by 0.6–0.7 Dice (He et al., 2023).
The paper’s most important contribution for a SAM-BG interpretation is not merely the performance gap, but the factor analysis explaining it. The authors explicitly compute Weber contrast between the target and its immediate neighboring background,
where is the average intensity inside the target region and is the average intensity in a 10-pixel dilated surrounding neighborhood. In the single-factor analysis, all three SAM variants had significantly higher Dice when contrast was higher (). Larger relative target regions also produced significantly higher Dice for all SAM variants (), and all three were significantly better on 2D than 3D images (). In the multivariable GLM, target size and U-Net Dice were the most significant jointly modeled factors for SAM-Point and SAM-Box, while modality remained significant for SAM-Semantic (He et al., 2023).
The same study further reports that all three SAM variations were more accurate in 2D medical images, larger target region sizes, easier cases with a higher Segmentation Ability score and higher U-Net Dice, and higher foreground-background contrast. This suggests that foreground–background distinctness is not a secondary nuisance factor but the dominant explanatory variable for zero-shot medical SAM behavior. When the target is large, high-contrast, and intrinsically easy to isolate, SAM can produce moderate results; when the target is small, low-contrast, diffuse, or dependent on 3D context, performance collapses.
3. Background-confusing regimes: camouflage and shadow
The same foreground–background story appears outside medicine in tasks that deliberately weaken objectness. “Can SAM Segment Anything? – When SAM Meets Camouflaged Object Detection” evaluates SAM on CAMO-Test, COD10K-Test, and NC4K, comparing it against 22 state-of-the-art COD methods (Tang et al., 2023). The protocol is generous to SAM: for each image, the evaluation can choose the SAM-generated mask with the highest overlap with ground truth under “maximum segmentation evaluation,” so the main result is effectively an oracle upper bound on SAM’s proposal quality. Even under that favorable setting, SAM remains substantially worse than dedicated COD methods on all three benchmarks. The more revealing result is “general location evaluation”: among all SAM-predicted masks, only about 1–3% exceed even modest thresholds. On CAMO, for example, the ratio is 2.55% at threshold 0.2 and 0.82% at threshold 0.7; analogous figures on COD10K and NC4K remain similarly small (Tang et al., 2023).
In a SAM-BG reading, those numbers are decisive. The problem is not merely that SAM draws imperfect boundaries around already-discovered objects. Rather, in highly background-confusing scenes, the proposal distribution itself is dominated by non-camouflage regions, salient clutter, or incomplete object parts. The paper therefore establishes camouflage as a failure regime in which foreground discovery fails because the background is too similar to the target.
“When SAM Meets Shadow Detection” provides a closely related negative result for non-object-like regions (Jie et al., 2023). The benchmark evaluates SAM in automatic no-prompt mode on four shadow datasets—SBU, UCF, ISTD, and CUHK—using two oracle selection rules: max F-measure selection and max IoU selection over SAM’s multiple automatically generated masks. Even under this favorable oracle setup, SAM remains far behind dedicated shadow detectors. The paper reports BER values roughly 25–30 on SBU, UCF, and ISTD, and 30–33 on CUHK, whereas specialized methods are around 3 on SBU, 6–7 on UCF, 1–2 on ISTD, and 8–12 on CUHK. Qualitatively, SAM tends to produce many tiny segmentations, often less than 10% of image area, and performs better only when the background is simple or the shadow has high contrast intensity (Jie et al., 2023).
Taken together, the camouflage and shadow studies show two complementary failure modes. In camouflage, foreground is object-like but visually assimilated into background. In shadow detection, the target is not a canonical object at all, but a contextual photometric region. In both cases, off-the-shelf SAM inherits an object-centric prior that becomes unreliable when background cues dominate or when the target does not present a clean object boundary.
4. Prompting, prompt sensitivity, and test-time control
One line of work treats SAM-BG not as a property of the data alone, but as a prompt-engineering problem. The medical benchmark already formalizes three prompt regimes: SAM-Semantic uses auto-prompted grid points and mask selection after non-maximum suppression; SAM-Point uses a single centroid point prompt; and SAM-Box uses a bounding-box prompt computed from ground truth and dilated by 20 pixels. Yet even with these favorable prompts, zero-shot SAM remains well below medical-image-specific models, which shows that better localization cues help but do not eliminate the background bottleneck (He et al., 2023).
SAMBA, a SAM-based framework for glioma segmentation on BraTS-Africa, sharpens that point by explicitly comparing prompt types (Barakat et al., 2023). The paper reports that point prompts were too unstable or sensitive, whole-brain bounding boxes were ineffective, and tumor-region bounding boxes worked best even before fine-tuning. For automatic prompt generation, the authors fine-tuned a YOLOv8 localizer for 150 epochs, reporting 96.7% accuracy in correctly differentiating slices with tumors from background slices, box loss , and average box confidence . But the same paper also shows how brittle the pipeline remains to prompt quality: on BraTS-Africa, binary tumor bounding boxes without YOLO yield 84.6% Dice, whereas the fully automatic YOLO-prompted version falls to 33.7%; direct multiclass segmentation with tumor bounding boxes without YOLO reaches 60.4% mean Dice over ET, TC, and WT (Barakat et al., 2023).
A more explicit control mechanism appears in “False Negative/Positive Control for SAM on Noisy Medical Images” (Yao et al., 2023). This is a training-free test-time method for noisy, low-contrast ultrasound. It perturbs a single bounding box into multiple box prompts, runs SAM on each, aggregates masks by majority vote with , estimates aleatoric uncertainty from prompt-induced disagreement, and then applies false-negative and false-positive correction: 0 The key idea is to use uncertainty to localize suspicious regions and then use image-intensity priors to decide whether uncertain pixels should be added back as false negatives or removed as false positives. On kidney ultrasound with coarse bounding boxes, Dice rises from 0.77 for single-box SAM and 0.70 for simple averaging to 0.88 for FNPC; on placenta ultrasound, Dice rises from 0.71 for SAM to 0.78 for FNPC. The same paper proposes Single-Slice-to-Volume (SS2V), enabling 3D placenta segmentation from one manually drawn 2D box, with Dice 0.72 (Yao et al., 2023). In encyclopedia terms, this is one of the clearest examples of a SAM-BG control scheme: the method explicitly suppresses background leakage and recovers missed foreground without retraining SAM.
5. Learned adaptations that operationalize background handling
A second line of work internalizes the background problem within the model rather than treating it only at test time. BC-SAM, a framework for cross-domain single blood-cell image classification, uses SAM in two roles: as an embedding extractor and as a segmentation/mask generator (Li et al., 2024). The mask is used to crop the original image to the square region corresponding to the masked area and to set the grayscale outside the masked region to 128. A cross-domain autoencoder then reconstructs this post-processed cell-focused image from the SAM embedding, with the stated goal of learning intrinsic blood-cell features while suppressing domain-specific artifacts. The encoder output is a 50-dimensional latent vector, and the best average accuracy across source/target scenarios is 63.54% with BC-SAM-ANN, compared with 53.98 for AE-CFE-RF and 49.30 for ResNext (Li et al., 2024). Although the paper does not name this “SAM-BG,” it is effectively a SAM-driven background-neutralization pipeline.
ADA-SAM makes the same idea more explicit in limited-label medical segmentation (Ward et al., 2 Jul 2025). It adds an auxiliary classification branch whose Grad-CAM or CAM heatmap is thresholded into a bounding-box prompt for SAM, thereby replacing manual prompts with automatically generated ones. The model is optimized with focal classification loss, Dice segmentation loss, and their weighted sum, and adapts SAM with LoRA, selecting rank 1 as best in ablations. The paper reports 26M trainable parameters instead of 91M for the original 2 SAM checkpoint, 132 ms/image inference, and very strong low-label results: with only 5 segmentation-labeled slices, ADA-SAM reaches 0.92 overall Dice, compared with 0.77 for SAM-Mix and 0.61 for SAM-PP (Ward et al., 2 Jul 2025). This is not an explicit background-supervision method, but it is a foreground-localization strategy in which CAM-derived prompts implicitly suppress background during prompt generation.
The most explicit internal BG suppression appears in FSSAM, a SAM 2-based few-shot segmentation method (Xu et al., 20 May 2025). The paper observes that pseudo query memory is likely to contain incomplete query foreground and unexpected query background features. It therefore introduces Support-Calibrated Memory Attention (SCMA), which uses support memory containing only foreground to calibrate cross-attention scores between query features and pseudo memory, suppressing memory locations unlikely to be foreground-consistent. Quantitatively, SCMA reduces the scores between query foreground pixels and unexpected background pixels in memory by 41.5% on average. The full method reaches 81.0 1-shot mIoU on PASCAL-53 and 62.3 on COCO-204 (Xu et al., 20 May 2025). In a strict SAM-BG sense, this is background suppression embedded inside memory attention itself.
BALR-SAM addresses the same issue through architecture rather than prompting or memory calibration (Liu et al., 29 Sep 2025). It is a prompt-free medical adaptation of SAM with three components: a Complementary Detail Enhancement Network (CDEN) for boundary-sensitive features, low-rank adapters in the image encoder’s ViT blocks, and low-rank tensor attention in the mask decoder. The method updates 11.7M parameters, reports a memory drop from 43,747MB to 11,748MB and inference time reduction from 196s to 148s when replacing decoder attention, and achieves 91.14 DSC on ISIC17 and 87.96 average Dice on Synapse (Liu et al., 29 Sep 2025). The paper frames this as boundary-aware adaptation rather than background modeling, but in practice it is another attempt to compensate for the same failure mode: natural-image SAM does not encode medical boundaries cleanly enough when foreground and background are weakly separated.
6. Positive counterexamples, applications, and open directions
The SAM-BG literature is not purely negative. In some settings, prompt-guided or mask-guided foreground/background separation is effective. “SAM vs BET” treats brain extraction as a practical form of brain/background segmentation in MRI (Mohapatra et al., 2023). Using inclusion markers, exclusion markers, and a custom bounding-box algorithm on 2D slices reconstructed into 3D, the study compares SAM with BET on 45 anonymized MR brain images across nine categories. SAM outperforms BET on Dice, IoU, and accuracy in eight of nine categories, with especially large gains on normalized FLAIR, WMH 3DT1, and fractional anisotropy images. The paper ties these gains to lesions near the outer boundary, meningeal ambiguity, signal inhomogeneity, and other cases where classical intensity-based skull stripping struggles (Mohapatra et al., 2023). This is an important counterweight to the broader failure story: when the prompt strategy aligns with the brain/non-brain boundary, SAM can act as an effective foreground–background separator.
A non-medical but operationally similar example appears in team-aware football player tracking (Ranasinghe et al., 9 Dec 2025). There SAM is used primarily for point-prompted segmentation at initialization and during recovery, with the resulting mask converted into a tighter CSRT tracker box and into a jersey-focused appearance descriptor. The upper 60% of the mask is used as the jersey area, helping suppress grass, field lines, shadows, and neighboring players before color histogram extraction. The system runs at 7.6–7.7 FPS with stable memory usage around 1880 MB, achieves 100 percent tracking success in light occlusions and 90 percent in crowded penalty-box scenarios with 5 or more players, recovers 50 percent of heavy occlusions, but only 8.66 percent re-acquisition success in long-term occlusion (Ranasinghe et al., 9 Dec 2025). Here again, SAM-BG is best understood as selective foreground extraction rather than as a standalone segmentation method.
Across these papers, the open directions are remarkably consistent. The medical benchmark argues for fine-tuning or medical-specific variants such as MedSAM and for models that explicitly address 3D structure, small foregrounds, low target-background contrast, and weakly defined boundaries (He et al., 2023). The camouflage study calls for stronger foundation models or architectural changes better suited to background-confusing scenes (Tang et al., 2023). The football tracking paper points toward stronger re-identification, motion prediction, multi-camera fusion, and temporal SAM variants such as SAM 2 or SAM-Track for long occlusions (Ranasinghe et al., 9 Dec 2025). BALR-SAM explicitly notes future work on better utilizing the prompt encoder (Liu et al., 29 Sep 2025).
The most defensible general conclusion is therefore narrow but robust. Off-the-shelf SAM is not a reliable solution when foreground is weakly object-like, poorly contrasted, diffuse, or heavily entangled with background. But the same literature shows that once background is modeled explicitly—through prompt design, uncertainty-guided FN/FP correction, CAM-derived prompting, support-calibrated memory attention, mask-guided artifact suppression, or boundary-aware low-rank adaptation—SAM can become a useful component in more specialized pipelines. Under that reading, “SAM-BG” names not a single architecture, but a recurring technical frontier: making SAM usable when segmentation quality is determined less by generic objectness than by the difficulty of separating foreground from background.