PartSAM: Promptable 3D Part Segmentation

• PartSAM is a promptable 3D part segmentation model that leverages a dual-branch triplane encoder to natively learn from large-scale 3D data.
• It employs interactive prompt encoding and cross-attention decoding to achieve fine-grained part decomposition with significant IoU improvements over prior methods.
• Its scalable design and model-in-the-loop curation pipeline enable fast, accurate segmentation across diverse domains, from surgical to industrial applications.

PartSAM denotes a family of models and methodologies for part-level segmentation built upon the promptable paradigm of the Segment Anything Model (SAM), enabling part decomposition in both 2D and 3D domains across diverse application areas. In the most significant recent instance, PartSAM specifically refers to "PartSAM: A Scalable Promptable Part Segmentation Model Trained on Native 3D Data” (Zhu et al., 26 Sep 2025), establishing the first foundation model for 3D part segmentation that is natively trained on vast labeled 3D data and adheres to prompt-driven segmentation principles reminiscent of 2D SAM. Complementary approaches include SurgicalPart-SAM (SP-SAM) for fine-grained part segmentation in surgical vision (Yue et al., 2023), and part-aware extensions such as P²SAM for one-shot adaptive segmentation in medical and natural imaging (Zhao et al., 2024).

1. Background and Motivation

Traditional 3D part segmentation approaches have relied on either shape-analytic methods (e.g., Shape Diameter Function (Tang et al., 2024)), which require domain-specific feature engineering and tuning, or learning-based models trained on limited 3D datasets. The rise of large-scale foundation models for 2D image segmentation—exemplified by SAM—has enabled label-efficient, promptable segmentation with strong generalization in 2D. Attempts to leverage 2D foundation models for 3D segmentation have typically involved "2D-to-3D lifting," where multiview image masks predicted by SAM or its successors are fused back onto a 3D shape (Zhu et al., 26 Sep 2025, Tang et al., 2024). However, this indirect paradigm struggles to capture intrinsic 3D geometry, delivers only surface-level understanding, and is constrained by the limitations of the underlying 2D models.

The primary motivation for PartSAM (Zhu et al., 26 Sep 2025) is to realize open-world, promptable 3D part segmentation by learning part-aware 3D representations directly from massive native 3D data, thus providing consistent, scalable, and flexible decomposition of objects beyond taxonomy-constrained or single-part scenarios.

2. Architecture and Prompting Mechanisms

The canonical PartSAM (Zhu et al., 26 Sep 2025) is structured as follows:

• Input and Encoder: PartSAM accepts a 3D shape represented as a point cloud (coordinates, optional normals and RGB) along with part prompts, typically 3D points indicating positive or negative assignment.
• Dual-Branch Triplane Encoder: The input point cloud is projected onto three axis-aligned 2D planes in two parallel branches. One branch is initialized and then frozen (distilled from PartField, itself trained with 2D SAM priors via contrastive learning), preserving 2D segmentation knowledge; the other is learnable, initialized to zero, ingesting geometric attributes and learning 3D-native representations. Features from both branches are summed to yield a continuous 3D feature field $\phi(x, y, z)$.
• Context Token Extraction: Farthest-point sampling (FPS) selects $N_c=2000$ center points. Local patches are formed via $K$-NN and aggregated into context tokens $F_c = \text{MLP}(\{\phi(p)\})$ for each patch, which serve as the basis for mask decoding.
• Prompt Encoding and Cross-Attention Decoder: Prompts are embedded using triplane features and positional encodings. A single cross-attention block allows bidirectional refinement between context tokens, prompt tokens, and two special tokens ($T_\text{out}$ for output mask logits, $T_\text{IoU}$ for quality prediction). The decoder returns mask logits per point and mask quality scores for up to $K=3$ candidate masks per prompt, with non-maximum suppression resolving ambiguity.
• Multi-Round Interaction: Outputs from one round (mask logits per point) may be included as input tokens in subsequent rounds for interactive refinement.

This design enables both single-prompt (identify specific part) and "segment-every-part" (automatic decomposition) operation modes. At present, PartSAM accepts 3D point-based prompts; support for box and text prompts is a future direction.

3. Model-in-the-Loop Data Curation and Training Schema

PartSAM achieves the scale required for robust open-world part segmentation via a unique data pipeline:

• Stage A: Native 3D Annotated Asset Integration: Shapes are collected from Objaverse/Objaverse-XL, leveraging artist-created groupings or extracting connected components as candidate parts. Rigorous filtering retains objects with $3$–$50$ parts, yielding $\sim 180\text{k}$ shapes with $N_c=2000$0 million part annotations.
• Stage B: Model-in-the-Loop Expansion: Over-fragmented shapes are re-annotated using pseudo-labels from PartField clustering. PartSAM is simulated with up to ten interactive prompts per shape; pseudo-labels are accepted if predicted mask IoU exceeds threshold values ($N_c=2000$1 or $N_c=2000$2, with at least five valid parts per shape). This extends the corpus to $N_c=2000$3 million part annotations.
• Training Objective: For mask prediction, the loss aggregates focal loss and Dice loss between each candidate mask and ground truth, with an additional encoder triplet contrastive loss (from PartField) weighted by $N_c=2000$4. Optimization employs AdamW at an initial learning rate of $N_c=2000$5, decayed stepwise, over $N_c=2000$6k iterations.

Data augmentation includes geometric (rotation, scaling, flips) and color (auto-contrast, jitter) transformations to enhance robustness.

4. Emergent Capabilities and Benchmark Results

PartSAM demonstrates strong open-world part segmentation:

• Promptable Segmentation: On benchmarks including PartObjaverse-Tiny and PartNetE, a single positive 3D click yields $N_c=2000$7 and $N_c=2000$8 respectively, vastly outperforming previous point-based methods (e.g., Point-SAM: $N_c=2000$9 and $K$0).
• Segment-Every-Part Mode: Sampling up to $K$1 prompt points enables automatic, exhaustive decomposition. Redundant or low-IoU masks are filtered via non-maximum suppression, and mesh faces are assigned by majority-vote among predicted masks. Internal structures, invisible in standard views or to 2D-based lifting methods, are discovered and segmented.
• Comparative Performance: PartSAM yields substantial IoU gains:

| Benchmark | Prior Best Model (IoU %) | PartSAM (IoU %) | Absolute Gain (pp) | |-----------------------|-------------------------|-----------------|--------------------| | PartObjaverse-Tiny | SAMesh (56.9) | 69.5 | +12.6 | | PartNetE | PartField* (59.1) | 72.4 | +13.3 |

• Ablations: The triplane encoder and large-scale 3D training are indispensable. Downgrading to Point-SAM's point-transformer degrades $K$2 to $K$3; using only PartNet data drops performance to $K$4.
• Efficiency: PartSAM requires $K$5 per shape for full segmentation, compared to $K$6 for traditional 2D-lifting or post-hoc fusion approaches, at a model size of $K$7M parameters.

5. Related Methodologies and Extensions

Several parallel or complementary extensions of the SAM paradigm exist:

• SurgicalPart-SAM (SP-SAM) (Yue et al., 2023): Tailored for fine-grained part segmentation in surgical instrument images, SP-SAM fuses expert knowledge via "collaborative prompts"—text prompts encoding category and part hierarchy. A cross-modal prompt encoder processes both CLIP text and image embeddings, with part-level features adaptively fused using both category and image-level attention. Hierarchical decoding delivers part-aware masks. SP-SAM achieves state-of-the-art results on EndoVis instrument segmentation challenges with only $K$8M trainable parameters.
• P²SAM (Part-aware Prompted SAM) (Zhao et al., 2024): Oriented towards one-shot adaptive segmentation, P²SAM extracts part prototypes by clustering reference mask features and generates part-aware prompt sets via similarity maps. A distribution-guided scheme selects the optimal number of parts by minimizing the Wasserstein distance between feature distributions on reference and target images. P²SAM is not restricted to clinical imaging—it generalizes to COCO, LVIS, and personalized segmentation tasks, yielding up to $K$9 percentage points mean Dice improvement over direct transfer.
• ParticleSAM (Zhou et al., 5 Aug 2025): Specializes in dense, small instance segmentation in industrial and materials contexts. By modulating inference heuristics (e.g., maximum-area filtering, enhanced feature extraction), ParticleSAM adapts the original SAM for segmentation of thousands of small objects per high-resolution image.

6. Limitations, Open Challenges, and Future Directions

PartSAM and its related methods are subject to important limitations:

• Prompt Scope: PartSAM as of (Zhu et al., 26 Sep 2025) is solely point-promptable in 3D; support for box and text prompts remains undeveloped.
• Class-Agnostic Output: Mask outputs are not semantic; instance masks are not labeled with object part types (e.g., "axle," "handle”). Annotation of large paired datasets (mask, text label) is suggested as a future direction.
• Catastrophic Forgetting Risk: The dual-branch mechanism preserves 2D priors but could diverge with extended training. Future work in continual learning or parameter-efficient regularization is proposed.
• Annotation and Computation: While curation is partially automated, 3D datasets with fine-grained, high-quality part labels remain difficult to assemble at scale. Inference—especially in interactive modes—can impose nontrivial computational overhead.
• Extensibility: Integration of additional prompt types, more advanced graph-based or spectral segmentation approaches, and distilled 3D architectures for real-time inference are highlighted as promising avenues.

7. Summary Table: Comparison of Recent SAM-Based Part Segmentation Methods

Model	Input Modality / Domain	Prompt Types	Training Data	Notable Strengths	Reference
PartSAM	3D shape (point cloud)	3D point	$F_c = \text{MLP}(\{\phi(p)\})$05M 3D part pairs	Native 3D training, open-world, fast feedforward	(Zhu et al., 26 Sep 2025)
SP-SAM	2D surgical images	Text (part + category)	Surgical datasets (EndoVis)	Fine-grained part semantic prompting	(Yue et al., 2023)
P²SAM	2D medical/natural images	Multi-point (part-wise)	One-shot, any promptable SAM	Distribution-based part prompt selection	(Zhao et al., 2024)
ParticleSAM	2D RGB/high-res images	Dense grid (heuristics)	Synthetic industrial data	Dense small object segmentation, zero-shot	(Zhou et al., 5 Aug 2025)

These advances collectively cement promptable part segmentation as a foundational direction for both 2D and 3D perceptual tasks. Each approach targets distinct domains or technical challenges, further reinforcing the adaptability and extensibility of the prompt-driven segmentation paradigm.