OpenWorldSAM: Universal Image Segmentation

• OpenWorldSAM is a framework that extends SAM2 for universal image segmentation using language prompts and a minimal set of trainable parameters.
• It integrates frozen SAM2 and BEiT-3 models with a 2-layer MLP, tie-breaker embeddings, and a 3-layer Transformer to align language and visual features.
• It supports diverse segmentation outputs—including semantic, instance, panoptic, and referring segmentation—achieving state-of-the-art open-vocabulary performance.

OpenWorldSAM denotes both a specific framework for language-prompted universal image segmentation and, in adjacent literature, a broader SAM-centered design pattern for open-world perception. In the named formulation, "OpenWorldSAM: Extending SAM2 for Universal Image Segmentation with Language Prompts" extends prompt-driven SAM2 to open-vocabulary scenarios by integrating multi-modal embeddings extracted from a lightweight vision-LLM, supports category-level and sentence-level language descriptions, and produces semantic, instance, panoptic, and referring-expression segmentations while training only 4.5 million parameters with frozen SAM2 and BEiT-3 (Xiao et al., 7 Jul 2025). In related work, the same conceptual pattern appears in systems that combine SAM with open-vocabulary detection, object priors, pseudo-label distillation, multi-view geometry, temporal propagation, or diffusion-based completion to address open-world detection, instance segmentation, 3D scene understanding, panoramic segmentation, video segmentation, and amodal completion (Lin et al., 25 May 2025, Wilms et al., 2024, Tai et al., 2024, Zhang et al., 2024, Guo et al., 2024, Ao et al., 2024).

1. Terminology and task scope

Across the literature, OpenWorldSAM is not tied to a single benchmark definition. In the specific SAM2-based framework, the operative setting is open-vocabulary, language-promptable segmentation: the model accepts text prompts, including category names and free-form referring expressions, and returns masks for semantic, instance, panoptic, or referring segmentation (Xiao et al., 7 Jul 2025). By contrast, VL-SAM-V2 distinguishes open-set detection, where a predefined category list is provided at inference, from open-ended detection, where no category list is provided and categories are generated post hoc; it uses the phrase open-world object detection for a framework that can operate in both modes (Lin et al., 25 May 2025). SOS defines Open-World Instance Segmentation (OWIS) as class-agnostic segmentation of all objects in an image when only a limited set of classes is annotated during training, with no requirement to predict semantic labels (Wilms et al., 2024). OV-SAM3D uses open-vocabulary 3D scene understanding and open-world to emphasize operation on any 3D scene without prior knowledge of the scene type or any 3D dataset-specific training (Tai et al., 2024). VideoSAM frames the task as class-agnostic, open-world video segmentation, in which all visually salient objects are discovered, segmented, and tracked over time without assuming a fixed category set (Guo et al., 2024).

This variation matters because common expectations about OpenWorldSAM can be misleading. It does not always mean a single end-to-end model that directly names and segments arbitrary objects from raw images. In some formulations it is prompt-conditional and per-query; in others it is class-agnostic; in others still it is a composition in which SAM supplies masks while separate modules provide labels, temporal continuity, or open-ended discovery. A plausible implication is that OpenWorldSAM is best understood as a family of SAM-anchored open-world perception strategies rather than a single canonical task.

2. Core architecture of the SAM2-based OpenWorldSAM framework

The specific framework "OpenWorldSAM: Extending SAM2 for Universal Image Segmentation with Language Prompts" keeps the heavy components frozen and inserts a small language adapter on top of SAM2 (Xiao et al., 7 Jul 2025). SAM2 contributes a hierarchical ViT encoder and a mask decoder; OpenWorldSAM uses SAM2 level-3 features at $64 \times 64$ resolution with 256 channels, denoted $$\mathbf{F}_{64\times64} \in \mathbb{R}^{4096 \times 256}$$. The language side is provided by frozen BEiT-3-Large, an early-fusion multimodal Transformer. For an image-text pair, the joint $[CLS]$ embedding is taken as the semantic prompt embedding,

$$\mathbf{p}_\mathrm{lang} = \mathbf{f}_{\text{[CLS]}} \in \mathbb{R}^{1024}.$$

A 2-layer MLP projects this representation into SAM prompt space,

$$\mathbf{u} = \mathrm{MLP}(\mathbf{p}_\mathrm{lang}) \in \mathbb{R}^{256}.$$

Instance awareness is introduced through positional tie-breaker embeddings and a small soft prompting Transformer. With $K=20$ learned tie-breakers $$\{\mathbf{t}_1,\dots,\mathbf{t}_K\} \subset \mathbb{R}^{256}$$, OpenWorldSAM forms $K$ language queries,

$$\mathbf{q}_i = \mathbf{u} + \mathbf{t}_i,\quad i = 1,\dots,K.$$

These queries pass through a 3-layer Transformer that alternates self-attention and cross-attention to SAM2 features. Using standard scaled dot-product attention,

$$\mathrm{Attn}(Q,K,V) = \mathrm{softmax}\left( \frac{QK^\top}{\sqrt{d}} \right)V,$$

the model lets the $$\mathbf{F}_{64\times64} \in \mathbb{R}^{4096 \times 256}$$0 queries diversify and align to spatial evidence in $$\mathbf{F}_{64\times64} \in \mathbb{R}^{4096 \times 256}$$1. The resulting queries $$\mathbf{F}_{64\times64} \in \mathbb{R}^{4096 \times 256}$$2 are then treated as prompt tokens for the frozen SAM2 mask decoder, which outputs $$\mathbf{F}_{64\times64} \in \mathbb{R}^{4096 \times 256}$$3 mask logits and $$\mathbf{F}_{64\times64} \in \mathbb{R}^{4096 \times 256}$$4 estimated IoU scores.

The architectural claim of OpenWorldSAM is therefore not that SAM2 is retrained for language. Rather, language is mapped into the prompt-token interface that SAM2 already exposes. Frozen components include the SAM2 Hiera encoder, the SAM2 mask decoder, and BEiT-3; trainable components are the 2-layer MLP projector, the $$\mathbf{F}_{64\times64} \in \mathbb{R}^{4096 \times 256}$$5 tie-breaker vectors, and the 3-layer Transformer, for a total of approximately 4.5 million parameters. The reported ablations further state that fine-tuning BEiT-3 or SAM2 degrades performance, so the adapter-only regime is presented not only as efficient but also as empirically preferable.

3. Prompting, supervision, and inference protocol

OpenWorldSAM uses a per-prompt inference pipeline. For each text prompt $$\mathbf{F}_{64\times64} \in \mathbb{R}^{4096 \times 256}$$6, BEiT-3 encodes the image and text jointly; the adapter produces $$\mathbf{F}_{64\times64} \in \mathbb{R}^{4096 \times 256}$$7 prompt tokens; SAM2 decodes $$\mathbf{F}_{64\times64} \in \mathbb{R}^{4096 \times 256}$$8 masks; and post-processing converts them into semantic, instance, or panoptic outputs (Xiao et al., 7 Jul 2025). Prompt types include single nouns such as “zebra” or “arcade machine,” multi-word descriptions such as “wooden chair,” and full referring expressions such as “the man in a red shirt holding a surfboard.” There is no prompt templating beyond raw text.

Training is performed on COCO-2017-Stuff with panoptic annotations, about 104k images, excluding images in the RefCOCOg-UMD validation split. For each prompt $$\mathbf{F}_{64\times64} \in \mathbb{R}^{4096 \times 256}$$9, ground-truth masks of class $[CLS]$0 are matched to the $[CLS]$1 predicted masks using Hungarian matching. The paper states that the objective is a focal loss on mask predictions,

$[CLS]$2

with no explicit classification loss because the class label is exactly the prompt text. Optimization uses AdamW, a learning rate of $[CLS]$3 for COCO pretraining and $[CLS]$4 for RefCOCOg fine-tuning, 25 epochs of COCO-Stuff pretraining, 15 epochs of RefCOCOg fine-tuning, batch size 8, one NVIDIA A100 (80 GB), and step decay with learning-rate reductions by 0.1 at 89% and 96% of training iterations. Image resolution is $[CLS]$5 for SAM2 and $[CLS]$6 for BEiT-3.

Task-specific outputs are derived by post-processing rather than by changing the backbone. For semantic segmentation, masks of the same text label are merged in a confidence-weighted manner. For instance segmentation, masks are thresholded by the SAM2 estimated IoU score at 0.7 and filtered with NMS at IoU 0.5. For panoptic segmentation, thing-versus-stuff selection is applied so that each pixel receives a single label. The same interface is also used for referring expression segmentation, with the difference that prompts are full natural-language descriptions rather than class names.

A notable aspect of the evaluation is the Oracle-prompts protocol. Instead of global matching over an entire label vocabulary, OpenWorldSAM is evaluated by providing the ground-truth class names for the dataset as prompts, one prompt per class, and measuring segmentation quality for the requested categories. This choice is motivated in the paper by the claim that global matching can unfairly penalize semantically close categories such as “building” and “skyscraper.” An optional two-stage refinement reuses the predicted masks as visual prompts back into SAM2’s standard decoder; the paper describes this as visually improving boundaries while having minor effect on mIoU and PQ.

4. Empirical performance and evaluation characteristics

Under Oracle-prompts evaluation, OpenWorldSAM reports 60.4 mIoU on ADE20K-150, 33.1 mIoU on ADE20K-857, 47.5 mIoU on PASCAL Context-459, and 55.6 mIoU on ScanNet-40; on smaller label sets such as VOC-20 and PASCAL Context-59, X-Decoder is slightly better (Xiao et al., 7 Jul 2025). For referring expression segmentation on RefCOCOg-UMD, OpenWorldSAM reports 74.0 cIoU, compared with 64.6 for X-Decoder, 65.6 for SEEM, 71.2 for PolyFormer, 74.4 for UNINEXT, 74.2 for GLaMM, and 76.8 for EVF-SAM. The paper also states that OpenWorldSAM achieves state-of-the-art performance in open-vocabulary semantic, instance, and panoptic segmentation across benchmarks including ADE20k, PASCAL, ScanNet, and SUN-RGBD.

The ablation studies isolate the mechanisms that distinguish OpenWorldSAM from earlier BEiT-3-plus-SAM systems. For VLM choice, CLIP-L text-only yields ADE-150 mIoU around 25.7, CLIP-L image+text around 26.5, BEiT-3 text-only around 26.3, and BEiT-3 image+text 60.4. This is used to argue that early image-text fusion is critical and that text-only prompting is insufficient. For trainable modules, the best configuration trains only the tie-breakers, cross-attention adapter, and MLP projector; fine-tuning BEiT-3 hurts performance massively. The tie-breakers are especially important for multi-instance output: on ADE-150 instance segmentation, AP rises from 1.0% without tie-breakers to 17.1% with tie-breakers. Cross-attention improves semantic alignment, with ADE-150 mIoU increasing from 56.8 to 60.4 and ADE-857 mIoU from 32.2 to 33.1.

These results clarify a recurrent misconception. OpenWorldSAM is not simply SAM2 with a text embedding appended to the decoder. The reported gains depend on image-conditioned language embeddings from BEiT-3, on multiplicity through $[CLS]$7 tied but differentiated prompt tokens, and on cross-attentional grounding into SAM2 feature maps. The empirical picture is also qualified by task structure: the strongest numbers are obtained when the label list is provided as prompts, not when the model must first infer which categories are present.

5. OpenWorldSAM as a broader SAM-centered design family

Several contemporaneous systems instantiate the same general principle—using SAM or SAM-like prompting as the spatial core of an open-world perception stack—but differ sharply in task definition, supervision, and output form.

System	Task	Defining mechanism
OpenWorldSAM	Universal image segmentation	Frozen SAM2 + frozen BEiT-3 + 4.5M adapter with tie-breakers and cross-attention (Xiao et al., 7 Jul 2025)
VL-SAM-V2	Open-world object detection / instance segmentation	General and specific query fusion; open-set and open-ended modes; optional SAM (Lin et al., 25 May 2025)
SOS	Open-world instance segmentation	DINO self-attention object priors prompt SAM; pseudo labels train class-agnostic Mask R-CNN (Wilms et al., 2024)
GoodSAM++	Panoramic semantic segmentation	SAM teacher, teacher assistant, distortion-aware rectification, multi-level knowledge adaptation (Zhang et al., 2024)
OV-SAM3D	Open-vocabulary 3D scene understanding	Superpoints + SAM + RAM + CLIP + ChatGPT; training-free 3D instance labeling (Tai et al., 2024)
VideoSAM	Open-world video segmentation	RADIO, Cycle-ack Pairs Propagation, memory, autoregressive object tokens (Guo et al., 2024)
Open-World Amodal Appearance Completion	Language-prompted amodal completion	LISA + Grounded-SAM + InstaOrderNet + Stable Diffusion v2 + RGBA output (Ao et al., 2024)

On the detection side, VL-SAM-V2 explicitly combines an open-set detector, LLMDet, with a VL-SAM-like open-ended proposal generator and can optionally plug into SAM for segmentation. Its core mechanism is a general and specific query fusion module that concatenates category-agnostic general queries and text-conditioned specific queries at each decoder layer, applies self-attention for interaction, and then updates them with shared cross-attention to image and text plus separate box heads. On LVIS minival, with Swin-T, it reports open-set AP 45.7 and AP$[CLS]$8 41.2, versus 44.7 and 37.3 for LLMDet; in open-ended mode it reports AP 29.5 and AP$[CLS]$9 29.8, versus 26.8 and 20.0 for GenerateU; and when combined with SAM for open-ended instance segmentation, it reports mask AP 28.7 and mask AP$$\mathbf{p}_\mathrm{lang} = \mathbf{f}_{\text{[CLS]}} \in \mathbb{R}^{1024}.$$0 27.7.

SOS represents a different branch of the design space. It is class-agnostic rather than language-labeled, and it uses SAM offline to generate pseudo annotations for unknown objects. The strongest object priors come from DINO self-attention maps, which guide point prompts for SAM; the resulting masks are filtered, deduplicated, merged with original annotations, and used to train a standard class-agnostic Mask R-CNN. In the COCO (VOC) $$\mathbf{p}_\mathrm{lang} = \mathbf{f}_{\text{[CLS]}} \in \mathbb{R}^{1024}.$$1 COCO (non-VOC) setting, SOS reports AP 8.9, AR$$\mathbf{p}_\mathrm{lang} = \mathbf{f}_{\text{[CLS]}} \in \mathbb{R}^{1024}.$$2 39.3, and $$\mathbf{p}_\mathrm{lang} = \mathbf{f}_{\text{[CLS]}} \in \mathbb{R}^{1024}.$$3 14.5, with an improvement in precision by up to 81.6% compared to the state of the art.

GoodSAM++ also uses SAM as a teacher rather than as a deployed inference module. In panoramic semantic segmentation, it addresses ERP distortion and capacity mismatch through a teacher assistant and a student. The Distortion-Aware Rectification (DARv2) module enforces consistency across overlapping windows and uses SAM boundaries to refine pseudo semantic maps, while Multi-level Knowledge Adaptation (MKA) distills both teacher-assistant predictions and pseudo labels into a compact panoramic student. Reported results include 61.20% mIoU on DensePASS and 65.97% on WildPASS for GoodSAM++-S, while the 3.7M-parameter GoodSAM++-M reaches 56.31% mIoU on DensePASS.

OV-SAM3D extends the motif into multi-view 3D. It is training-free, generates superpoints as initial 3D prompts, projects them into images to prompt SAM, back-projects masks into 3D, constructs an overlapping score table for coarse-mask formation and remerging, and uses RAM, ChatGPT, and CLIP for open-world labels. On ScanNet200 it reports AP 9.0, $$\mathbf{p}_\mathrm{lang} = \mathbf{f}_{\text{[CLS]}} \in \mathbb{R}^{1024}.$$4 13.6, and $$\mathbf{p}_\mathrm{lang} = \mathbf{f}_{\text{[CLS]}} \in \mathbb{R}^{1024}.$$5 19.4; on nuScenes, where training-based indoor 3D methods transfer poorly, it reports AP 8.9, $$\mathbf{p}_\mathrm{lang} = \mathbf{f}_{\text{[CLS]}} \in \mathbb{R}^{1024}.$$6 16.0, $$\mathbf{p}_\mathrm{lang} = \mathbf{f}_{\text{[CLS]}} \in \mathbb{R}^{1024}.$$7 29.1, $$\mathbf{p}_\mathrm{lang} = \mathbf{f}_{\text{[CLS]}} \in \mathbb{R}^{1024}.$$8 13.6, $$\mathbf{p}_\mathrm{lang} = \mathbf{f}_{\text{[CLS]}} \in \mathbb{R}^{1024}.$$9 16.2, and $$\mathbf{u} = \mathrm{MLP}(\mathbf{p}_\mathrm{lang}) \in \mathbb{R}^{256}.$$0 39.8.

VideoSAM adds the temporal machinery absent from image-based SAM. It replaces SAM’s encoder with RADIO, uses Cycle-ack Pairs Propagation and a memory mechanism for object association, and introduces an autoregressive object-token mechanism in the SAM decoder to maintain granularity consistency. On UVO it reports mAP 9.7 and AR100 29.8; on BURST it reports mAP 10.7, AP50 14.9, AP75 11.2, AR100 53.5, AR10 45.8, and AR1 18.1.

Open-World Amodal Appearance Completion pushes the design pattern beyond visible-mask extraction. It uses LISA for language-grounded visible masks, Grounded-SAM for object and background segmentation, InstaOrderNet for occlusion reasoning, CLIP for prompt selection, and Stable Diffusion v2 inpainting for iterative amodal reconstruction, outputting RGBA elements. On its 2379-image evaluation set with 2565 occluded instances and 553 distinct target classes, it reports overall human preference of 41.86%, CLIP score 28.181, LPIPS 0.320, VGG feature similarity 0.646, SSIM 0.731, and a complete failure rate of 4.1%.

6. Limitations, ambiguities, and research directions

The literature also makes clear that OpenWorldSAM is not a solved paradigm. Reported limitations include weaker zero-shot performance on Cityscapes and BDD100K and a fixed $$\mathbf{u} = \mathrm{MLP}(\mathbf{p}_\mathrm{lang}) \in \mathbb{R}^{256}.$$1 instance limit in the SAM2-based OpenWorldSAM; inherited VLM weaknesses such as hallucinations, incorrect responses, and relatively slow inference in VL-SAM-V2; dependence on the object prior in SOS; pseudo-label quality bounded by SAM and the teacher assistant in GoodSAM++; scene-level runtimes of 4–6 minutes on ScanNet or 30–60 seconds on nuScenes and sensitivity to thresholds in OV-SAM3D; difficulty with frequent object in/out events and camera transitions in VideoSAM; and dependence on multiple large sequential modules, proxy metrics, and pre-trained generative models in open-world amodal completion (Xiao et al., 7 Jul 2025, Lin et al., 25 May 2025, Wilms et al., 2024, Zhang et al., 2024, Tai et al., 2024, Guo et al., 2024, Ao et al., 2024).

A second ambiguity concerns evaluation. Some systems are judged under Oracle-prompts, some under class-agnostic mask AP or $$\mathbf{u} = \mathrm{MLP}(\mathbf{p}_\mathrm{lang}) \in \mathbb{R}^{256}.$$2, some under open-ended category generation aligned by CLIP similarity, and some under human preference because ground-truth occluded appearance is unavailable. This suggests that “open-world” currently bundles several partially overlapping desiderata: prompt flexibility, category generalization, object discovery without category lists, class-agnostic objectness, and domain transfer without retraining. A plausible implication is that future work will continue to split between two strategies. One strategy keeps SAM or SAM2 in the inference loop and augments it with language, memory, geometry, or reasoning modules. The other uses SAM mainly as a teacher or pseudo-label generator and distills its open-world spatial prior into lighter downstream models. Both strategies preserve the central insight shared across the literature: SAM supplies a strong class-agnostic mask prior, but open-world perception requires additional machinery for semantics, discovery, temporal or geometric consistency, and, in some cases, completion beyond the visible image.