BoxSAM: Box-Prompt Segmentation & Refinement

• BoxSAM is a family of box-prompt based segmentation workflows that adapt SAM for diverse tasks including medical imaging and weakly supervised object detection.
• In RoBox-SAM, prompt refinement, iterative correction, and point enhancement modules mitigate attention drift and improve segmentation robustness.
• For camouflaged object detection, BoxSAM converts bounding-box annotations into refined pseudo-labels using redundancy processing strategies to enhance detection accuracy.

Searching arXiv for papers directly using or discussing “BoxSAM” and closely related SAM-with-box-prompt terminology. I’m querying the arXiv search index for “BoxSAM”, “RoBox-SAM”, and related prompt-based SAM terminology. BoxSAM denotes a family of SAM usages organized around bounding-box prompts rather than a single uniformly defined model. In the cited literature, the term has two primary meanings. First, it can refer generically to SAM used with box prompts, as in the medical segmentation setting where the formal method name is RoBox-SAM rather than BoxSAM (Huang et al., 2024). Second, it can denote a specific weakly supervised pseudo-label generation pipeline for camouflaged object detection, where BoxSAM is the paper’s explicit front-end module that converts box annotations into SAM-derived masks for downstream training (Li et al., 25 May 2026). Related works further broaden the surrounding design space by using SAM-generated masks to derive boxes, rotated boxes, or tracking boxes, but without adopting BoxSAM as their formal method name (Cao et al., 2023, Jiang et al., 6 Apr 2025).

1. Terminology and scope

The term is best understood as context-dependent. In medical segmentation, “BoxSAM” refers to SAM used with box prompts, while the authors’ actual method is RoBox-SAM, described as a robust box-prompt-based SAM extension (Huang et al., 2024). In weakly supervised camouflaged object detection, by contrast, BoxSAM is the explicit name of the pseudo-label generation stage that uses bounding-box annotations + SAM to produce pixel-level supervision for MGNet (Li et al., 25 May 2026). Other papers discuss box-prompt evaluation, mask-to-box conversion, or BoxSAM-style systems, but do not introduce a standalone method named BoxSAM (Fan et al., 11 Jul 2025, Jiang et al., 6 Apr 2025, Cao et al., 2023).

arXiv id	Named method	Relation to “BoxSAM”
(Huang et al., 2024)	RoBox-SAM	Actual method name; “BoxSAM” refers to SAM with box prompts
(Li et al., 25 May 2026)	BoxSAM	Named pseudo-label generation pipeline for COD
(Fan et al., 11 Jul 2025)	Birkhoff	Not BoxSAM; evaluates compressed SAMs with Box prompts
(Jiang et al., 6 Apr 2025)	SAM2MOT	Described as BoxSAM-style via mask-to-box tracking
(Cao et al., 2023)	P2RBox	BoxSAM-adjacent prompt-to-box pipeline, not named BoxSAM

A recurrent misconception is to treat BoxSAM as a canonical SAM variant with fixed architecture. The cited papers do not support that interpretation. Instead, they show that the phrase indexes a prompt modality and a class of downstream procedures built around box-conditioned segmentation, pseudo-label generation, or box recovery from masks.

2. RoBox-SAM: robust box-prompted SAM in medical image segmentation

“Robust Box Prompt based SAM for Medical Image Segmentation” introduces RoBox-SAM, motivated by the observation that SAM can achieve satisfactory segmentation performance under high-quality box prompts, but its robustness is compromised when box quality declines (Huang et al., 2024). The central failure mode is that inaccurate boxes can cause the decoder’s cross-attention to activate incorrect regions, creating an attention drift issue. This is particularly consequential in medical images, where boundaries may be faint, contrast is low, and foreground/background separation is difficult.

RoBox-SAM preserves the standard SAM backbone structure—image encoder, prompt encoder, and mask decoder—but inserts three learnable modules around the prompt and feature pathways. The first is the Prompt Refinement Module (PRM), which predicts coordinate offsets to transform a low-quality box into a better one. The formulation is:

$f^{*}=f_{img}+C_{Att}(f_{img},f_{p})$

$$\Delta(o)=\left\{\Delta x_1,\Delta x_2,\Delta y_1,\Delta y_2\right\}$$

$p_c^{*}=p_c+\Delta(o)$

with supervision

$L_o=L_1(p_c^{*},p_{gt}).$

PRM is therefore a box-coordinate regressor coupled to prompt-image feature fusion.

The second mechanism is an online iterative refinement strategy. At test time, the refined box from one iteration is reused as the next PRM input until either

$\sum \Delta(o)=0$

or the method reaches $k=5$ iterations. This yields the iterative variant RoBox-SAM (Iter), intended for cases in which a single refinement pass is insufficient.

The third mechanism is the Prompt Enhancement Module (PEM), which automatically generates point prompts to assist box-promptable segmentation. The paper describes constructing a potential negative region

$(A \cup B) - C$

where $A$ is the original box, $B$ the refined box, and $C$ the GT mask. PEM is defined as

$$\Delta(o)=\left\{\Delta x_1,\Delta x_2,\Delta y_1,\Delta y_2\right\}$$0

and predicts $$\Delta(o)=\left\{\Delta x_1,\Delta x_2,\Delta y_1,\Delta y_2\right\}$$1 point prompts, optimized by

$$\Delta(o)=\left\{\Delta x_1,\Delta x_2,\Delta y_1,\Delta y_2\right\}$$2

Its stated role is to generate especially negative points that suppress over-segmentation.

RoBox-SAM also adds a Self-Information Extractor (SIE), which mines internal image priors using Canny edge maps, SLIC superpixels, and FFT-based frequency features. The low-frequency amplitudes are masked to emphasize details and edges, and the extracted self-information is fused by

$$\Delta(o)=\left\{\Delta x_1,\Delta x_2,\Delta y_1,\Delta y_2\right\}$$3

These features are then used to optimize image embeddings and attention calculation.

The experimental setting is explicitly large-scale: 99,299 images/slices, 5 modalities, 25 organs/targets, and 14 public + 5 in-house datasets. Training uses Pytorch, NVIDIA A40, batch size 16, learning rate $$\Delta(o)=\left\{\Delta x_1,\Delta x_2,\Delta y_1,\Delta y_2\right\}$$4, 20 epochs, AdamW, and losses combining cross-entropy, Dice, $$\Delta(o)=\left\{\Delta x_1,\Delta x_2,\Delta y_1,\Delta y_2\right\}$$5, and $$\Delta(o)=\left\{\Delta x_1,\Delta x_2,\Delta y_1,\Delta y_2\right\}$$6. Prompt corruption is generated by randomly shifting GT-derived boxes by 0–30% while keeping box IoU $$\Delta(o)=\left\{\Delta x_1,\Delta x_2,\Delta y_1,\Delta y_2\right\}$$7, and testing averages 5 random trials (Huang et al., 2024).

The reported findings are robustness-centric. Under a 10% random shift, RoBox-SAM achieves the best DICE and prompt robustness (PR) across all five modalities. Under 0–30% random shifts, it remains stable while MedSAM degrades sharply. The iterative version improves performance further when prompt quality is very poor. The paper attributes about 6% DICE and 8% PR improvement over the baseline to PRM, reports that the full model is statistically significant with $$\Delta(o)=\left\{\Delta x_1,\Delta x_2,\Delta y_1,\Delta y_2\right\}$$8, and characterizes the system as lightweight, with less than 1M trainable parameters and about 0.6 s per test image. During fine-tuning, the original SAM components are frozen and the backbone is ViT-H.

3. BoxSAM as weakly supervised pseudo-label generation for camouflaged object detection

In “Weakly Supervised Camouflaged Object Detection Based on the SAM Model and Mask Guidance,” BoxSAM is the name of the weak-supervision engine rather than a robustness modification of SAM (Li et al., 25 May 2026). Its purpose is to use bounding-box annotations + SAM to generate initial pixel-level pseudo-labels for MGNet, a mask-guided COD network designed to mitigate edge ambiguity and missed detections.

The annotation protocol is simple and explicit. If an image has one camouflaged object, the annotator marks the smallest enclosing box. If it has multiple objects, each is annotated separately; if boxes overlap because objects overlap, they are merged into one box. Given an image $$\Delta(o)=\left\{\Delta x_1,\Delta x_2,\Delta y_1,\Delta y_2\right\}$$9 and a bounding-box prompt $p_c^{*}=p_c+\Delta(o)$0, the initial pseudo-label is

$p_c^{*}=p_c+\Delta(o)$1

This defines the first stage of BoxSAM.

The paper emphasizes that SAM outputs on camouflaged scenes often contain redundant information, meaning background leakage induced by the visual similarity between object and surroundings. To address this, it introduces a Redundancy Processing Strategy (RPS). The criterion is based on the number of connected components or bounding-box-like regions in the SAM mask. Pseudo-labels whose bounding-box count is equal to the original annotation are retained as initial training data; the mismatched cases are denoted $p_c^{*}=p_c+\Delta(o)$2 and processed further.

For each problematic mask $p_c^{*}=p_c+\Delta(o)$3, the method finds connected components $p_c^{*}=p_c+\Delta(o)$4, uses a trained MGNet to predict a segmentation map $p_c^{*}=p_c+\Delta(o)$5, and then keeps or removes each connected region according to

$p_c^{*}=p_c+\Delta(o)$6

This establishes a self-correction loop: train with reliable pseudo-labels, use MGNet to refine noisy pseudo-labels, then retrain with the improved set.

The downstream network MGNet uses PVTv2 as backbone and contains three modules: CMD (Cascaded Mask Decoder), CEM (Context Enhancement Module), and MFAM (Mask-guided Feature Aggregation Module). CMD progressively fuses multi-scale features to produce an initial mask. CEM employs multiple branches with different dilation rates and a BA block to reduce missed detection and recover high-frequency detail. MFAM uses the CMD mask to define foreground and background attention maps,

$p_c^{*}=p_c+\Delta(o)$7

and guides top-down feature aggregation to sharpen boundaries.

Training uses the hybrid loss from F3Net,

$p_c^{*}=p_c+\Delta(o)$8

with total supervision

$p_c^{*}=p_c+\Delta(o)$9

Under weak supervision, $L_o=L_1(p_c^{*},p_{gt}).$0 is the SAM-generated mask $L_o=L_1(p_c^{*},p_{gt}).$1 or the RPS-refined mask.

The reported weakly supervised COD results are given for CAMO, COD10K, and NC4K. Under bounding-box supervision, BoxSAM achieves CAMO: $L_o=L_1(p_c^{*},p_{gt}).$2, $L_o=L_1(p_c^{*},p_{gt}).$3, $L_o=L_1(p_c^{*},p_{gt}).$4, $L_o=L_1(p_c^{*},p_{gt}).$5; COD10K: $L_o=L_1(p_c^{*},p_{gt}).$6, $L_o=L_1(p_c^{*},p_{gt}).$7, $L_o=L_1(p_c^{*},p_{gt}).$8, $L_o=L_1(p_c^{*},p_{gt}).$9; NC4K: $\sum \Delta(o)=0$0, $\sum \Delta(o)=0$1, $\sum \Delta(o)=0$2, $\sum \Delta(o)=0$3. The paper states that BoxSAM improves over the nearest weakly supervised competitor and specifically beats SAM-COD on bounding-box supervision. Removing RPS reduces $\sum \Delta(o)=0$4 by about 0.7% on average. For the full model, MGNet with PVTv2_B2 reaches CAMO: $\sum \Delta(o)=0$5, $\sum \Delta(o)=0$6; COD10K: $\sum \Delta(o)=0$7, $\sum \Delta(o)=0$8; NC4K: $\sum \Delta(o)=0$9, $k=5$0, with PVTv2_B4 performing even better. The same framework is also reported on weakly supervised SOD with bounding boxes, polyp segmentation, and defect detection (Li et al., 25 May 2026).

4. Adjacent prompt-to-box and mask-to-box paradigms

Several adjacent works illuminate the broader technical space around BoxSAM without using that name as the formal method identifier. In oriented object detection, P2RBox uses a single annotated point per object as a prompt to SAM, selects among SAM-generated mask proposals with semantic and spatial scoring, and converts the best mask into a rotated bounding box for detector training (Cao et al., 2023). Its final proposal score is

$k=5$1

combining SAM proposal confidence, centrality guidance, and semantic quality. For symmetric categories such as PL and HC, it uses Symmetry Axis Estimation (SAE) rather than the minimum rotated rectangle. On DOTA-v1.0, the reported SAM baseline results are 47.91 for RetinaNet, 50.84 for FCOS, and 52.75 for Oriented R-CNN, while P2RBox reaches 55.50, 58.40, and 62.26, respectively. The method therefore exemplifies prompt-conditioned mask generation followed by explicit box recovery.

In multi-object tracking, SAM2MOT proposes Tracking by Segmentation, in which detections are used only for initialization, re-initialization, and object management, while actual tracking is performed by SAM2 and tracking boxes are extracted from predicted masks (Jiang et al., 6 Apr 2025). For predicted mask $k=5$2, the implied box is the axis-aligned bounding rectangle enclosing the mask support:

$k=5$3

The method further introduces a trajectory manager system and a cross-object interaction module to handle dynamic object addition and removal, prompt drift, and occlusion. On DanceTrack, the best reported result is HOTA = 75.8, IDF1 = 83.9, MOTA = 88.5, AssA = 72.2, with stated gains of +2.1 HOTA, +4.5 IDF1, and +6.3 AssA over the previous best. This is not a BoxSAM method in name, but it is explicitly framed as a BoxSAM-style segmentation-first system in which the box is derived from the mask rather than predicted directly.

These adjacent methods show that box-conditioned SAM workflows need not be limited to standard box-prompted segmentation. The same general pattern extends to pseudo-annotation generation, rotated-box induction, and tracking-box extraction.

5. Non-BoxSAM usages and scope delimitations

A separate source of ambiguity is the association of BoxSAM with papers that are about SAM more broadly but not about BoxSAM as a method. “Compress Any Segment Anything Model (SAM)” proposes Birkhoff, a data-free compression framework for SAMs and SAM variants, not a box-prompt segmentation method (Fan et al., 11 Jul 2025). The method is based on Hyper-Compression, where a weight vector is approximated from a low-dimensional scalar trajectory, and introduces HyperLinear to fuse decompression and matrix multiplication.

The paper evaluates compressed SAMs on COCO, LVIS, and SA-1B using prompt types including Box, Box-Center, Mask-Center, and Mask-Rand, and for COCO uses box prompts obtained from YOLOX and ViTDet-H detections. However, the authors explicitly do not present BoxSAM as the method name. The headline results are SAM2-B compression ratio: 5.17×, performance drop: <1%, and compression time: under 60 seconds for all models, with examples such as SAM-B: 5.17× in 20.83 s and EfficientSAM-Ti: 5.13× in 5.33 s. These claims concern model compression rather than prompt robustness, pseudo-label generation, or box-derived segmentation.

A parallel clarification applies to RoBox-SAM: the 2024 medical paper does not introduce a distinct model named BoxSAM as its final method name. Its actual proposal is RoBox-SAM, and “BoxSAM” in that context means standard SAM used with box prompts rather than a separate architecture (Huang et al., 2024). The distinction is essential because the scientific contribution lies in prompt refinement, prompt enhancement, and self-information extraction rather than in merely using boxes as prompts.

6. Cross-paper technical pattern

The cited works collectively indicate that box-conditioned SAM pipelines recur in three distinct forms (Huang et al., 2024, Li et al., 25 May 2026). The first is prompt robustness, where the core problem is that low-quality boxes destabilize segmentation and must be corrected, iteratively refined, or complemented with auxiliary points and image priors. The second is weak-supervision transfer, where boxes are inexpensive annotations that can seed SAM masks, after which redundancy filtering or label refinement becomes the key issue. The third is representation conversion, where SAM masks are converted into boxes, rotated boxes, or tracking boxes for compatibility with downstream tasks.

This suggests that the operative variable in BoxSAM-related research is not merely the existence of a box prompt, but the engineering required to control the mismatch between the prompt-induced spatial prior and the target object. In RoBox-SAM, that mismatch appears as attention drift under prompt corruption. In the COD BoxSAM pipeline, it appears as redundant information in SAM-generated pseudo-labels. In P2RBox and SAM2MOT, it appears in the conversion from mask geometry to the box representation needed by the downstream task.

A plausible implication is that BoxSAM is most useful as a descriptor for a methodological family rather than a single algorithmic object. Within that family, the common denominator is the use of bounding boxes as prompts or derived outputs, while the decisive technical content lies in the refinement modules, filtering strategies, scoring rules, and mask-to-box transformations added around the SAM backbone.