---
title: 'BoxSAM: Box-Prompt Segmentation & Refinement'
url: https://www.emergentmind.com/topics/boxsam
type: topic
---

# BoxSAM: Box-Prompt Segmentation & Refinement

Searching arXiv for papers directly using or discussing “BoxSAM” and closely related SAM-with-box-prompt terminology.
I’m querying the arXiv search index for “BoxSAM”, “RoBox-SAM”, and related prompt-based SAM terminology.
BoxSAM denotes a family of SAM usages organized around bounding-box prompts rather than a single uniformly defined model. In the cited literature, the term has two primary meanings. First, it can refer generically to SAM used with box prompts, as in the medical segmentation setting where the formal method name is **RoBox-SAM** rather than BoxSAM [2407.21284]. Second, it can denote a specific weakly supervised pseudo-label generation pipeline for camouflaged object detection, where **BoxSAM** is the paper’s explicit front-end module that converts box annotations into SAM-derived masks for downstream training [2605.25385]. Related works further broaden the surrounding design space by using SAM-generated masks to derive boxes, rotated boxes, or tracking boxes, but without adopting BoxSAM as their formal method name [2311.13128, 2504.04519].

## 1. Terminology and scope

The term is best understood as context-dependent. In medical segmentation, “BoxSAM” refers to **SAM used with box prompts**, while the authors’ actual method is **RoBox-SAM**, described as a robust box-prompt-based SAM extension [2407.21284]. In weakly supervised camouflaged object detection, by contrast, **BoxSAM** is the explicit name of the pseudo-label generation stage that uses **bounding-box annotations + SAM** to produce pixel-level supervision for **MGNet** [2605.25385]. Other papers discuss box-prompt evaluation, mask-to-box conversion, or BoxSAM-style systems, but do not introduce a standalone method named BoxSAM [2507.08765, 2504.04519, 2311.13128].

| arXiv id | Named method | Relation to “BoxSAM” |
|---|---|---|
| 2407.21284 | RoBox-SAM | Actual method name; “BoxSAM” refers to SAM with box prompts |
| 2605.25385 | BoxSAM | Named pseudo-label generation pipeline for COD |
| 2507.08765 | Birkhoff | Not BoxSAM; evaluates compressed SAMs with Box prompts |
| 2504.04519 | SAM2MOT | Described as BoxSAM-style via mask-to-box tracking |
| 2311.13128 | P2RBox | BoxSAM-adjacent prompt-to-box pipeline, not named BoxSAM |

A recurrent misconception is to treat BoxSAM as a canonical SAM variant with fixed architecture. The cited papers do not support that interpretation. Instead, they show that the phrase indexes a prompt modality and a class of downstream procedures built around box-conditioned segmentation, pseudo-label generation, or box recovery from masks.

## 2. RoBox-SAM: robust box-prompted SAM in medical image segmentation

“Robust Box Prompt based SAM for Medical Image Segmentation” introduces **RoBox-SAM**, motivated by the observation that SAM can achieve satisfactory segmentation performance under **high-quality box prompts**, but its robustness is compromised when box quality declines [2407.21284]. The central failure mode is that inaccurate boxes can cause the decoder’s cross-attention to activate incorrect regions, creating an **attention drift issue**. This is particularly consequential in medical images, where boundaries may be faint, contrast is low, and foreground/background separation is difficult.

RoBox-SAM preserves the standard SAM backbone structure—**image encoder, prompt encoder, and mask decoder**—but inserts three learnable modules around the prompt and feature pathways. The first is the **Prompt Refinement Module (PRM)**, which predicts coordinate offsets to transform a low-quality box into a better one. The formulation is:
$$
f^{*}=f_{img}+C_{Att}(f_{img},f_{p})
$$
$$
\Delta(o)=\left\{\Delta x_1,\Delta x_2,\Delta y_1,\Delta y_2\right\}
$$
$$
p_c^{*}=p_c+\Delta(o)
$$
with supervision
$$
L_o=L_1(p_c^{*},p_{gt}).
$$
PRM is therefore a box-coordinate regressor coupled to prompt-image feature fusion.

The second mechanism is an **online iterative refinement strategy**. At test time, the refined box from one iteration is reused as the next PRM input until either
$$
\sum \Delta(o)=0
$$
or the method reaches \(k=5\) iterations. This yields the iterative variant **RoBox-SAM (Iter)**, intended for cases in which a single refinement pass is insufficient.

The third mechanism is the **Prompt Enhancement Module (PEM)**, which automatically generates point prompts to assist box-promptable segmentation. The paper describes constructing a potential negative region
$$
(A \cup B) - C
$$
where \(A\) is the original box, \(B\) the refined box, and \(C\) the GT mask. PEM is defined as
$$
p_{pre}=\mathcal{M}(f_r^{*}-f^{*})
$$
and predicts \(n=5\) point prompts, optimized by
$$
L_p=\lVert(p_{pre},p_{gt})\rVert_2.
$$
Its stated role is to generate especially **negative points** that suppress over-segmentation.

RoBox-SAM also adds a **Self-Information Extractor (SIE)**, which mines internal image priors using **Canny edge maps, SLIC superpixels, and FFT-based frequency features**. The low-frequency amplitudes are masked to emphasize details and edges, and the extracted self-information is fused by
$$
f' = f^{*} + f_s.
$$
These features are then used to optimize image embeddings and attention calculation.

The experimental setting is explicitly large-scale: **99,299 images/slices**, **5 modalities**, **25 organs/targets**, and **14 public + 5 in-house datasets**. Training uses **Pytorch**, **NVIDIA A40**, **batch size 16**, **learning rate \(1\times10^{-4}\)**, **20 epochs**, **AdamW**, and losses combining **cross-entropy**, **Dice**, \(L_o\), and \(L_p\). Prompt corruption is generated by randomly shifting GT-derived boxes by **0–30%** while keeping box IoU \(>0.5\), and testing averages **5 random trials** [2407.21284].

The reported findings are robustness-centric. Under a **10% random shift**, RoBox-SAM achieves the best **DICE** and **prompt robustness (PR)** across all five modalities. Under **0–30% random shifts**, it remains stable while **MedSAM** degrades sharply. The iterative version improves performance further when prompt quality is very poor. The paper attributes about **6% DICE** and **8% PR** improvement over the baseline to PRM, reports that the full model is statistically significant with \(p<0.05\), and characterizes the system as lightweight, with **less than 1M trainable parameters** and about **0.6 s per test image**. During fine-tuning, the original SAM components are frozen and the backbone is **ViT-H**.

## 3. BoxSAM as weakly supervised pseudo-label generation for camouflaged object detection

In “Weakly Supervised Camouflaged Object Detection Based on the SAM Model and Mask Guidance,” **BoxSAM** is the name of the weak-supervision engine rather than a robustness modification of SAM [2605.25385]. Its purpose is to use **bounding-box annotations + SAM** to generate initial pixel-level pseudo-labels for **MGNet**, a mask-guided COD network designed to mitigate **edge ambiguity** and **missed detections**.

The annotation protocol is simple and explicit. If an image has **one camouflaged object**, the annotator marks the **smallest enclosing box**. If it has **multiple objects**, each is annotated separately; if boxes overlap because objects overlap, they are merged into one box. Given an image \(I\) and a bounding-box prompt \(B\), the initial pseudo-label is
$$
M = SAM(I, B).
$$
This defines the first stage of BoxSAM.

The paper emphasizes that SAM outputs on camouflaged scenes often contain **redundant information**, meaning background leakage induced by the visual similarity between object and surroundings. To address this, it introduces a **Redundancy Processing Strategy (RPS)**. The criterion is based on the number of connected components or bounding-box-like regions in the SAM mask. Pseudo-labels whose bounding-box count is **equal to the original annotation** are retained as initial training data; the mismatched cases are denoted \(F\) and processed further.

For each problematic mask \(F\), the method finds connected components \(F_i\), uses a trained **MGNet** to predict a segmentation map \(P\), and then keeps or removes each connected region according to
$$
F(m,n)=\left\{ \begin{aligned}
255 & , & \exists (m,n) \in F_{i}, P(m,n)>0,\\
0 & , & \forall (m,n) \in F_{i}, P(m,n)=0.
\end{aligned} \right.
$$
This establishes a self-correction loop: train with reliable pseudo-labels, use MGNet to refine noisy pseudo-labels, then retrain with the improved set.

The downstream network **MGNet** uses **PVTv2** as backbone and contains three modules: **CMD** (Cascaded Mask Decoder), **CEM** (Context Enhancement Module), and **MFAM** (Mask-guided Feature Aggregation Module). CMD progressively fuses multi-scale features to produce an initial mask. CEM employs multiple branches with different dilation rates and a **BA block** to reduce missed detection and recover high-frequency detail. MFAM uses the CMD mask to define foreground and background attention maps,
$$
S_{4}=\sigma(P_{4}), \qquad B_{4}=E-\sigma(P_{4}),
$$
and guides top-down feature aggregation to sharpen boundaries.

Training uses the hybrid loss from **F3Net**,
$$
L = L_{wBCE} + L_{wIoU},
$$
with total supervision
$$
L_{total}=\sum_{i=1}^{4} L(P_{i},G).
$$
Under weak supervision, \(G\) is the SAM-generated mask \(M\) or the RPS-refined mask.

The reported weakly supervised COD results are given for **CAMO**, **COD10K**, and **NC4K**. Under **bounding-box supervision**, BoxSAM achieves **CAMO**: \(S_\alpha = 0.859\), \(F_\beta = 0.842\), \(\mathcal{M} = 0.057\), \(E_\phi = 0.908\); **COD10K**: \(S_\alpha = 0.857\), \(F_\beta = 0.789\), \(\mathcal{M} = 0.027\), \(E_\phi = 0.919\); **NC4K**: \(S_\alpha = 0.877\), \(F_\beta = 0.854\), \(\mathcal{M} = 0.037\), \(E_\phi = 0.925\). The paper states that BoxSAM improves over the nearest weakly supervised competitor and specifically beats **SAM-COD** on bounding-box supervision. Removing **RPS** reduces \(F_\beta\) by about **0.7% on average**. For the full model, **MGNet with PVTv2\_B2** reaches **CAMO**: \(S_\alpha = 0.882\), \(F_\beta = 0.859\); **COD10K**: \(S_\alpha = 0.878\), \(F_\beta = 0.800\); **NC4K**: \(S_\alpha = 0.893\), \(F_\beta = 0.860\), with **PVTv2\_B4** performing even better. The same framework is also reported on **weakly supervised SOD with bounding boxes**, **polyp segmentation**, and **defect detection** [2605.25385].

## 4. Adjacent prompt-to-box and mask-to-box paradigms

Several adjacent works illuminate the broader technical space around BoxSAM without using that name as the formal method identifier. In oriented object detection, **P2RBox** uses a **single annotated point per object** as a prompt to SAM, selects among SAM-generated mask proposals with semantic and spatial scoring, and converts the best mask into a rotated bounding box for detector training [2311.13128]. Its final proposal score is
$$
Score=S_{mask}-\beta_1\cdot S_{offset}+\beta_2\cdot S_{smt},
$$
combining SAM proposal confidence, centrality guidance, and semantic quality. For symmetric categories such as **PL** and **HC**, it uses **Symmetry Axis Estimation (SAE)** rather than the minimum rotated rectangle. On **DOTA-v1.0**, the reported **SAM baseline** results are **47.91** for RetinaNet, **50.84** for FCOS, and **52.75** for Oriented R-CNN, while **P2RBox** reaches **55.50**, **58.40**, and **62.26**, respectively. The method therefore exemplifies prompt-conditioned mask generation followed by explicit box recovery.

In multi-object tracking, **SAM2MOT** proposes **Tracking by Segmentation**, in which detections are used only for initialization, re-initialization, and object management, while actual tracking is performed by **SAM2** and tracking boxes are extracted from predicted masks [2504.04519]. For predicted mask \(\mathcal{M}_i\), the implied box is the axis-aligned bounding rectangle enclosing the mask support:
$$
b_i = (x_{\min}, y_{\min}, x_{\max}, y_{\max}).
$$
The method further introduces a **trajectory manager system** and a **cross-object interaction module** to handle dynamic object addition and removal, prompt drift, and occlusion. On **DanceTrack**, the best reported result is **HOTA = 75.8**, **IDF1 = 83.9**, **MOTA = 88.5**, **AssA = 72.2**, with stated gains of **+2.1 HOTA**, **+4.5 IDF1**, and **+6.3 AssA** over the previous best. This is not a BoxSAM method in name, but it is explicitly framed as a **BoxSAM-style** segmentation-first system in which the box is derived from the mask rather than predicted directly.

These adjacent methods show that box-conditioned SAM workflows need not be limited to standard box-prompted segmentation. The same general pattern extends to pseudo-annotation generation, rotated-box induction, and tracking-box extraction.

## 5. Non-BoxSAM usages and scope delimitations

A separate source of ambiguity is the association of BoxSAM with papers that are about SAM more broadly but not about BoxSAM as a method. “Compress Any Segment Anything Model (SAM)” proposes **Birkhoff**, a **data-free compression framework** for SAMs and SAM variants, not a box-prompt segmentation method [2507.08765]. The method is based on **Hyper-Compression**, where a weight vector is approximated from a low-dimensional scalar trajectory, and introduces **HyperLinear** to fuse decompression and matrix multiplication.

The paper evaluates compressed SAMs on **COCO**, **LVIS**, and **SA-1B** using prompt types including **Box**, **Box-Center**, **Mask-Center**, and **Mask-Rand**, and for COCO uses box prompts obtained from **YOLOX** and **ViTDet-H** detections. However, the authors explicitly do not present BoxSAM as the method name. The headline results are **SAM2-B compression ratio: 5.17×**, **performance drop: <1%**, and **compression time: under 60 seconds for all models**, with examples such as **SAM-B: 5.17× in 20.83 s** and **EfficientSAM-Ti: 5.13× in 5.33 s**. These claims concern model compression rather than prompt robustness, pseudo-label generation, or box-derived segmentation.

A parallel clarification applies to RoBox-SAM: the 2024 medical paper does **not** introduce a distinct model named BoxSAM as its final method name. Its actual proposal is **RoBox-SAM**, and “BoxSAM” in that context means standard SAM used with box prompts rather than a separate architecture [2407.21284]. The distinction is essential because the scientific contribution lies in prompt refinement, prompt enhancement, and self-information extraction rather than in merely using boxes as prompts.

## 6. Cross-paper technical pattern

The cited works collectively indicate that box-conditioned SAM pipelines recur in three distinct forms [2407.21284, 2605.25385]. The first is **prompt robustness**, where the core problem is that low-quality boxes destabilize segmentation and must be corrected, iteratively refined, or complemented with auxiliary points and image priors. The second is **weak-supervision transfer**, where boxes are inexpensive annotations that can seed SAM masks, after which redundancy filtering or label refinement becomes the key issue. The third is **representation conversion**, where SAM masks are converted into boxes, rotated boxes, or tracking boxes for compatibility with downstream tasks.

This suggests that the operative variable in BoxSAM-related research is not merely the existence of a box prompt, but the engineering required to control the mismatch between the prompt-induced spatial prior and the target object. In RoBox-SAM, that mismatch appears as **attention drift** under prompt corruption. In the COD BoxSAM pipeline, it appears as **redundant information** in SAM-generated pseudo-labels. In P2RBox and SAM2MOT, it appears in the conversion from mask geometry to the box representation needed by the downstream task.

A plausible implication is that BoxSAM is most useful as a descriptor for a methodological family rather than a single algorithmic object. Within that family, the common denominator is the use of bounding boxes as prompts or derived outputs, while the decisive technical content lies in the refinement modules, filtering strategies, scoring rules, and mask-to-box transformations added around the SAM backbone.

Source: https://www.emergentmind.com/topics/boxsam