Semantic-Fast-SAM: Efficient Segmentation Framework

• Semantic-Fast-SAM (SFS) is a segmentation framework that leverages FastSAM for rapid class-agnostic mask generation and an SSA labeling strategy for assigning semantic categories.
• It achieves competitive mIoU scores—70.33% on Cityscapes and 48.01% on ADE20K—while operating up to 20× faster and using significantly less GPU memory than traditional SAM implementations.
• The modular design supports both closed-set and open-vocabulary segmentation, though open-vocab mode is currently limited by BLIP captioning speed, inviting further optimization for real-time deployment.

Searching arXiv for the primary SFS paper and the directly related FastSAM/SSA/open-vocabulary segmentation works mentioned in the provided data. Semantic-Fast-SAM (SFS) is a semantic segmentation framework that combines the Fast Segment Anything model with a semantic labeling pipeline in order to achieve real-time performance without sacrificing accuracy. In the reported formulation, FastSAM provides rapid class-agnostic mask generation, and a Semantic-Segment-Anything (SSA) labeling strategy assigns semantic categories to those masks. The resulting system is described as producing high-quality semantic segmentation maps at a fraction of the computational cost and memory footprint of the original SAM-based approach, with reported results of approximately $70.33$ mIoU on Cityscapes and $48.01$ on ADE20K, alongside approximately $20\times$ faster inference than SSA in the closed-set setting. The same framework is also used for open-vocabulary segmentation through CLIP-based semantic heads (Kim, 22 Apr 2026).

1. Terminology and conceptual scope

Within the SFS literature, acronym disambiguation is necessary. In "Semantic-Fast-SAM: Efficient Semantic Segmenter," SFS denotes Semantic-Fast-SAM, a segmentation framework centered on FastSAM and semantic labeling (Kim, 22 Apr 2026). A distinct 2025 paper, "One Swallow Does Not Make a Summer: Understanding Semantic Structures in Embedding Spaces," uses SFS to denote Semantic Field Subspace, a geometry-preserving, context-aware representation for embedding-space analysis (Sun et al., 30 Nov 2025). The two uses are unrelated in method and application.

Semantic-Fast-SAM is positioned as an efficient alternative to original SAM-based semantic segmentation pipelines. Its defining idea is to preserve the "segment-anything" capability at the mask-generation stage while replacing the heavier transformer-based SAM backbone with FastSAM, an efficient CNN-based re-implementation of SAM that runs much faster than the original transformer-based SAM. This suggests that SFS is best understood not as a monolithic end-to-end semantic segmenter, but as a modular assembly in which mask proposal and semantic assignment are separated.

2. Architectural composition

The mask generator in SFS is FastSAM. According to the reported architecture, FastSAM is based on a YOLOv8-seg-style CNN rather than a ViT, uses a C2f backbone with stacked convolutions and Cross-Stage Partial connections, and employs a lightweight FPN to produce multi-scale feature maps (Kim, 22 Apr 2026).

Mask prediction follows a YOLACT-inspired prototype formulation. A small convolutional prototype head generates $K$ global mask bases,

$P \in \mathbb{R}^{H \times W \times K},$

while a parallel mask coefficient head predicts, for each detected object, a vector

$c_i \in \mathbb{R}^{K},$

together with a detection box and objectness score. The final mask is then

$m_i = \sigma(P \cdot c_i).$

Relative to SAM with a ViT-H backbone, the reported structural and computational differences are explicit: FastSAM uses a single forward pass, does not require iterative prompt decoding, and is reported as approximately $20$–$50\times$ faster on an RTX 3090, with approximately $20$–$48.01$0 versus approximately $48.01$1–$48.01$2. The model size is reported as approximately $48.01$3 parameters, compared with approximately $48.01$4 for ViT-H, and GPU memory is reported as approximately $48.01$5 for $48.01$6 masks, compared with approximately $48.01$7–$48.01$8 for SAM in "everything" mode (Kim, 22 Apr 2026).

These properties define the computational substrate on which the semantic pipeline is built. A plausible implication is that the main contribution of SFS lies in converting FastSAM’s efficient class-agnostic masks into semantically labeled outputs without reintroducing the full computational burden associated with original SAM-based pipelines.

3. Semantic labeling pipeline

After FastSAM produces $48.01$9 class-agnostic masks $20\times$0, SFS labels them through two parallel branches: a closed-set semantic branch and an open-vocabulary semantic branch (Kim, 22 Apr 2026).

In the closed-set branch, the full image is passed to a pre-trained segmentation model, such as OneFormer with ConvNeXt-L, trained on a fixed taxonomy such as COCO or ADE20K. This produces a dense per-pixel map

$20\times$1

For each mask $20\times$2, the assigned closed-set label is the majority vote over the masked pixels:

$20\times$3

In the open-vocabulary branch, each mask region is cropped and resized, BLIP is used to generate a caption, and candidate noun phrases $20\times$4 are extracted from that caption. SFS then computes CLIP embeddings for the image crop and text candidates:

$20\times$5

$20\times$6

Candidate labels are scored by cosine similarity:

$20\times$7

The pipeline keeps the top-$20\times$8 highest-scoring labels, with the reported example $20\times$9.

Fusion is rule-based. If the closed-set label $K$0 is confident, with score $K$1, and is not contradicted by any open-vocabulary suggestion, it is retained. Otherwise, the open-vocabulary label with highest $K$2 is selected. Unlabeled masks are assigned "unidentified."

The CLIP-based semantic heads are described as off-the-shelf CLIP image and text encoders, such as ViT-B/16, used without any fine-tuning. The head itself is simply the cosine-similarity module, with no additional fully connected layers. Because closed-set labels can also be embedded through the same text encoder, the framework supports mixing closed-set labels with open-vocabulary captions in a unified embedding space (Kim, 22 Apr 2026).

4. Training formulation and evaluation protocol

SFS is described as an inference-only assembly of pretrained modules, but the FastSAM component itself is trained with a YOLACT-style objective (Kim, 22 Apr 2026). The reported total loss is

$K$3

where

$K$4

is cross-entropy on objectness or class logits,

$K$5

is the box regression loss, and

$K$6

is binary cross-entropy over prototypes and coefficients.

A semantic-head loss is also specified as a conditional extension for fine-tuning:

$K$7

with combined objective

$K$8

Because the framework is presented primarily as an inference-time composition of pretrained subsystems, this loss is best interpreted as a possible training extension rather than the default operating mode.

Evaluation uses mean Intersection-over-Union:

$K$9

The empirical discussion in the paper centers on Cityscapes, ADE20K, and contextual comparison on ADE20K novel classes.

5. Reported empirical performance

The central empirical claim is that SFS matches the accuracy of prior SAM-based methods while substantially improving efficiency in the closed-set regime, and that it remains competitive in open-vocabulary segmentation (Kim, 22 Apr 2026).

Setting	SFS result	Reported comparison
Cityscapes, closed-set	$P \in \mathbb{R}^{H \times W \times K},$0 mIoU	SSA $P \in \mathbb{R}^{H \times W \times K},$1; OneFormer $P \in \mathbb{R}^{H \times W \times K},$2
ADE20K, closed-set	$P \in \mathbb{R}^{H \times W \times K},$3 mIoU	SSA $P \in \mathbb{R}^{H \times W \times K},$4; OneFormer $P \in \mathbb{R}^{H \times W \times K},$5
ADE20K novel classes, open-vocabulary	$P \in \mathbb{R}^{H \times W \times K},$6 zero-shot	CLIPSeg $P \in \mathbb{R}^{H \times W \times K},$7–$P \in \mathbb{R}^{H \times W \times K},$8; GroupViT $P \in \mathbb{R}^{H \times W \times K},$9; MaskCLIP $c_i \in \mathbb{R}^{K},$0

For a single $c_i \in \mathbb{R}^{K},$1 image on an RTX 3090, the reported inference times are: FastSAM mask-only at $c_i \in \mathbb{R}^{K},$2, SFS closed-set only at $c_i \in \mathbb{R}^{K},$3, OneFormer at $c_i \in \mathbb{R}^{K},$4, SSA closed-set at $c_i \in \mathbb{R}^{K},$5, SFS open-vocab at $c_i \in \mathbb{R}^{K},$6, and SSA open-vocab at $c_i \in \mathbb{R}^{K},$7. The reported GPU memory footprint is approximately $c_i \in \mathbb{R}^{K},$8 for the FastSAM mask generator, approximately $c_i \in \mathbb{R}^{K},$9 for the full SFS pipeline, and approximately $m_i = \sigma(P \cdot c_i).$0 for the SSA pipeline.

The ablations further separate the contribution of fusion and mask budget. For fusion, the reported closed-set-only configuration reaches $m_i = \sigma(P \cdot c_i).$1 mIoU, while the full model reaches $m_i = \sigma(P \cdot c_i).$2 mIoU. For open-vocabulary mask budget, the reported trade-offs are: Top-100 masks at $m_i = \sigma(P \cdot c_i).$3 and $m_i = \sigma(P \cdot c_i).$4, Top-50 masks at $m_i = \sigma(P \cdot c_i).$5 and $m_i = \sigma(P \cdot c_i).$6, and Top-25 masks at $m_i = \sigma(P \cdot c_i).$7 and $m_i = \sigma(P \cdot c_i).$8.

A common misconception would be to read the phrase "without sacrificing accuracy" as implying parity with fully supervised semantic segmentation. The reported numbers do not support that interpretation: OneFormer remains higher on both Cityscapes and ADE20K. The more precise reading is that SFS approximately preserves the performance of prior SAM-based methods while materially reducing latency and memory use.

6. Practical significance, deployment profile, and limitations

The reported practical implications are oriented toward deployment. In closed-set mode, SFS is described as enabling real-time semantic segmentation at approximately $m_i = \sigma(P \cdot c_i).$9 FPS on a single high-end GPU; in open-vocabulary mode with BLIP, the paper reports feasible operation at approximately $20$0 FPS. The total memory requirement of approximately $20$1 is presented as making on-device deployment possible on $20$2–$20$3 GPUs (Kim, 22 Apr 2026).

The framework’s output is characterized as rich "segment-anything" output: instance-aware masks together with open-world labels, and it does so without per-dataset retraining. This broadens the applicability of foundation segmentation models in robotics scenarios. This suggests a deployment niche in systems that require prompt-free dense perception, instance-sensitive region extraction, and label spaces that can extend beyond a closed benchmark taxonomy.

The reported limitations are equally specific. Open-vocabulary speed is still dominated by BLIP captioning. Proposed future directions include replacing BLIP with lighter CLIP-only heads, distilling the fusion logic into one network, and using a shared backbone for mask and semantic heads to further improve end-to-end throughput. These directions indicate that, in the present design, the principal bottleneck has shifted away from mask generation and toward semantic interpretation.

Taken together, Semantic-Fast-SAM is most accurately described as a modular efficiency-oriented reinterpretation of SAM-style semantic segmentation: FastSAM supplies rapid class-agnostic regions, SSA-style semantic labeling converts them into closed-set and open-vocabulary predictions, and CLIP-based scoring provides the unifying mechanism for broad label assignment (Kim, 22 Apr 2026).