Part-Aware Instance Segmentation

Updated 7 July 2026

Part-aware instance segmentation is a method that integrates object-level masks with detailed part decomposition, enabling hierarchical and semantic analysis of data.
It employs diverse architectures—ranging from anchor-free detectors and rotated proposals to transformer-based methods and 3D point cloud pipelines—to capture both global and local features.
This approach improves segmentation accuracy in applications like surgical robotics, industrial automation, and microscopy by leveraging multi-level supervision and latent-part discovery.

Part-aware instance segmentation (PIS) denotes a family of formulations in which instance segmentation is coupled to an explicit internal decomposition of each instance into parts. In the surgical-instrument formulation, PIS predicts, for each instance $i$ , a binary mask $y_i$ , an instrument label $c_i$ , and disjoint binary part masks $\{m_{i,1},\dots,m_{i,K}\}$ summing to $y_i$ ; related formulations decompose industrial objects into approximate convex parts, discover class-shared latent parts for few-shot segmentation, or cluster 3D point clouds into multi-level part instances under semantic guidance (Wei et al., 25 Jul 2025, Feng et al., 2022, Nguyen et al., 2021, Sun et al., 2022).

1. Formal scope and relation to adjacent tasks

In the most explicit formulation, instrument-level Instance Segmentation (IIS) predicts a set of disjoint masks $y_i$ and instrument-class labels $c_i$ , while Part-level Semantic Segmentation (PSS) predicts a per-pixel one-hot or soft semantic map $s[k]$ over $K$ part classes. PIS combines both: for each instance $i$ , it predicts the instance mask and label together with a set of disjoint part masks $y_i$ 0 that sum to the instance mask. The resulting representation is hierarchical, because it localizes complete instances and simultaneously decomposes each one into constituent parts (Wei et al., 25 Jul 2025).

The cited literature uses “part” in several technically distinct senses. In FAPIS, the $y_i$ 1 latent parts are the $y_i$ 2 output channels of PartNet, and no part labels are ever provided; the parts are learned as shared latent basis elements regularized to resemble sparse non-negative matrix factorization bases over training masks (Nguyen et al., 2021). In industrial bin picking, parts are approximate convex components produced by automatic solidity-based label decoupling from full-object masks (Feng et al., 2022). In the 3D PartNet setting, part instances are organized in a three-level hierarchy—coarse, middle, and fine—and are segmented from point clouds by combining semantic segmentation and instance-center regression (Sun et al., 2022). This suggests that PIS is best understood as a structural constraint on instance segmentation rather than as a single fixed annotation protocol.

A recurrent misconception is that PIS is merely ordinary instance segmentation with extra labels. The formulations above indicate otherwise. IIS alone cannot distinguish parts within each instrument, and PSS alone does not separate different instrument instances; PIS is introduced precisely to resolve that gap by binding part predictions to specific parent instances (Wei et al., 25 Jul 2025).

2. Canonical architectural patterns in 2D PIS

The 2D literature represented here spans anchor-free detection, rotated-proposal pipelines, detect-then-segment refinement, and transformer-based mask classification.

Formulation	Domain	Key part mechanism
FAPIS (Nguyen et al., 2021)	Few-shot COCO-20i	Latent parts via PartNet and Part Assembling Module
Industrial PIS (Feng et al., 2022)	Bin picking	Approximate convex parts and part-to-part offsets
Sperm morphology (Chen et al., 2024)	Microscopy	Attention-based refinement of part masks
SurgPIS (Wei et al., 25 Jul 2025)	Robot-assisted surgery	Part-specific queries derived from object queries

FAPIS builds on a two-stream backbone plus FPN and attaches two major modules: an Anchor-Free Detector (AFD) and a Part-Based Instance Segmenter (PIS). At each location $y_i$ 3 on each FPN level, AFD predicts a figure-ground score $y_i$ 4, a box-regression vector $y_i$ 5, and a part-importance vector $y_i$ 6. After non-maximum suppression, the downstream PartNet produces part-activation maps $y_i$ 7, and the Part Assembling Module fuses ROI-aligned part maps with the predicted part weights according to

$y_i$ 8

where $y_i$ 9. This makes mask prediction an explicit non-negative factorization into basis maps and coefficients (Nguyen et al., 2021).

The industrial bin-picking pipeline also inherits the Mask R-CNN paradigm, but replaces axis-aligned RoIs with rotated proposals and adds a part-correlation head. For each part proposal, the network predicts a binary part mask and a structured offset vector containing a visibility-correction offset $c_i$ 0 and inter-part offsets $c_i$ 1. The final instance is not predicted directly as one mask; instead, parts are assembled through candidate matching by offset thresholding, a bidirectional consistency test, and connected-component clustering on the induced graph. The reported motivation is specific to “low solidity” objects whose mask area is only a small fraction of the bounding box, making whole-object segmentation unstable under clutter and occlusion (Feng et al., 2022).

A different 2D pattern appears in automated sperm morphology analysis. There, the starting point is explicitly a “detect-then-segment” pipeline: backbone plus FPN, Region Proposal Network, RoIAlign on $c_i$ 2, and a mask head producing preliminary part masks. The technical novelty lies in a second-pass attention-based refinement module. Preliminary per-instance part masks are re-projected into full-image coordinates, converted by a $c_i$ 3 convolution and sigmoid into attention maps, and multiplied with fused full-image multi-scale features,

$c_i$ 4

before dilated convolutions and a $c_i$ 5 convolution produce refined masks. This design is introduced to reconstruct lost context outside bounding boxes and correct feature distortion due to bounding-box cropping and resizing during RoIAlign (Chen et al., 2024).

SurgPIS adopts a transformer-based mask classification design built on Mask2Former. A set of learnable instrument queries $c_i$ 6 is decoded into instrument embeddings, each of which produces an instrument class distribution and an instrument mask. Part predictions are generated by feeding each object query through a small MLP to obtain $c_i$ 7 part-specific queries $c_i$ 8, each producing a part mask by dot-product with the pixel-decoder feature map. The explicit transformation from an object query to its part queries is the mechanism that links part predictions to their parent instrument instance (Wei et al., 25 Jul 2025).

3. Supervision regimes, latent structure, and label generation

The supervision required by PIS varies substantially across formulations. FAPIS is notable for using no part labels at all. Training supervision consists of support masks for modulating query features, box locations derived from query ground-truth masks for learning AFD, and final instance-mask supervision inside each predicted box for learning both the $c_i$ 9-head and PartNet via back-propagation through the Part Assembling Module. To impose a shared decomposition across classes, FAPIS precomputes one NMF decomposition $\{m_{i,1},\dots,m_{i,K}\}$ 0 over resized instance masks and penalizes disagreement between predicted part channels and NMF part prototypes through a permutation-matching Dice loss,

$\{m_{i,1},\dots,m_{i,K}\}$ 1

with the assignment $\{m_{i,1},\dots,m_{i,K}\}$ 2 obtained by the Hungarian algorithm maximizing IoU. This regularizer is intended to push the network toward sparse parts shared across all training classes (Nguyen et al., 2021).

The industrial formulation likewise avoids manual part annotation, but does so through automatic label decoupling. Each instance mask is recursively split by approximate convex decomposition until each part has concavity below a threshold $\{m_{i,1},\dots,m_{i,K}\}$ 3, where $\{m_{i,1},\dots,m_{i,K}\}$ 4 is the shorter side of the parent bounding box. Part centers are then computed from the decoupled masks, and ground-truth inter-part offsets $\{m_{i,1},\dots,m_{i,K}\}$ 5 together with visibility-correction offsets $\{m_{i,1},\dots,m_{i,K}\}$ 6 are derived automatically. The multi-task loss sums classification, rotated-box regression, part-mask loss, and offset regression loss (Feng et al., 2022).

SurgPIS makes the supervision problem itself central. Stage 1 assumes fully supervised PIS data $\{m_{i,1},\dots,m_{i,K}\}$ 7, and training uses part-aware bipartite matching with a cost that combines instrument classification, instrument mask loss, and average part mask loss, followed by a supervised loss

$\{m_{i,1},\dots,m_{i,K}\}$ 8

Stage 2 addresses disjoint datasets labeled either for IIS only or PSS only. PIS predictions are aggregated into IIS outputs or into semantic part maps $\{m_{i,1},\dots,m_{i,K}\}$ 9, so that losses can be computed against partial labels. A student–teacher consistency mechanism, with EMA teacher updates $y_i$ 0 and pseudo-label filtering, is then used to maintain prediction consistency for missing PIS information (Wei et al., 25 Jul 2025).

These examples challenge another common misconception: exhaustive, instance-aligned part annotation is not a prerequisite for PIS. The literature includes latent-part discovery without part labels, automatic part-label generation from instance masks, and weakly supervised learning from disjoint IIS and PSS datasets (Nguyen et al., 2021, Feng et al., 2022, Wei et al., 25 Jul 2025).

4. Three-dimensional and hierarchical PIS

The 3D formulation addresses part instance segmentation from point clouds, with particular emphasis on the fine-grained, hierarchical annotations of PartNet. The network is built on octree-based U-Nets and splits after a shared encoder into a semantic decoder $y_i$ 1 and an instance decoder $y_i$ 2. The semantic branch predicts per-point semantic probabilities $y_i$ 3 and is trained with cross-entropy. The instance branch predicts offsets $y_i$ 4 to ground-truth instance centers and offsets $y_i$ 5 to semantic region centers, both trained with $y_i$ 6 regression (Sun et al., 2022).

The central mechanism is semantic segmentation-assisted instance feature fusion. Instead of concatenating semantic and instance features pointwise, semantic probabilities are used to aggregate instance features nonlocally over all points associated with a semantic label: $y_i$ 7 The fused feature for regression is $y_i$ 8. In the three-level PartNet hierarchy, this is extended to cross-level fusion: semantic probabilities from every level are used to aggregate instance features for the target level, allowing fine-level instance regression to leverage coarse semantic groupings (Sun et al., 2022).

The “semantic region center” prediction task is introduced because mean-shift clustering can collapse spatially adjacent instances, such as the two blades of a scissor, into a single cluster. Each point predicts an offset not only to its instance center but also to the centroid of all instance centers of the same semantic class. At inference, the shifted point is

$y_i$ 9

with $y_i$ 0 empirically set to $y_i$ 1. Mean-shift clustering with fixed bandwidth $y_i$ 2 is then applied within each semantic label (Sun et al., 2022).

This 3D line of work broadens the meaning of PIS in two ways. First, the representation is multi-level rather than flat. Second, part awareness is not implemented as per-instance pixel masks but as a joint semantic-instance geometry over sparse 3D points. A plausible implication is that PIS should be viewed as a modality-agnostic structural prediction problem rather than as a purely 2D mask-decoding architecture.

5. Evaluation protocols and reported empirical behavior

Reported gains are strong across all represented domains, but the metrics are domain-specific and not directly comparable.

Work	Evaluation setting	Reported result
FAPIS (Nguyen et al., 2021)	COCO-20i few-shot instance segmentation	Significantly outperforms the state of the art
3D multi-level PIS (Sun et al., 2022)	PartNet, averaged over 24 categories and three levels	$y_i$ 3 vs $y_i$ 4 prior best
Industrial PIS (Feng et al., 2022)	ILS dataset	$y_i$ 5
Sperm morphology (Chen et al., 2024)	Held-out 64-image test set	$y_i$ 6 with ResNet-101
SurgPIS (Wei et al., 25 Jul 2025)	EndoVis2018/2017, SAR-RARP50, GraSP	PartPQ $y_i$ 7 weakly supervised with Swin-B

On PartNet, the 3D method reports $y_i$ 8, compared with $y_i$ 9 for the prior best, with coarse-level $c_i$ 0 vs $c_i$ 1, middle-level $c_i$ 2 vs $c_i$ 3, and fine-level $c_i$ 4 vs $c_i$ 5. The ablation study further reports a progression from approximately $c_i$ 6 AP50 for the baseline, to $c_i$ 7 with single-level fusion only, $c_i$ 8 with multi-level fusion only, $c_i$ 9 with full cross-level fusion, and $s[k]$ 0 after adding semantic region-center prediction (Sun et al., 2022).

In industrial bin picking, the reported comparison on the ILS dataset is $s[k]$ 1, $s[k]$ 2, and $s[k]$ 3, versus Mask R-CNN at $s[k]$ 4, $s[k]$ 5, and $s[k]$ 6. The valve ablation isolates part-level supervision and aggregation: a baseline Mask R-CNN yields AP $s[k]$ 7, adding rotated proposals and part masks raises this to $s[k]$ 8, adding the offset head with Hungarian matching yields $s[k]$ 9, and the full PIS system with bidirectional aggregation reaches $K$ 0. Replacing bidirectional aggregation with one-way Hungarian matching is reported to cost approximately 4 percentage points in AP $K$ 1 and to raise runtime from $K$ 2 to $K$ 3 (Feng et al., 2022).

In sperm morphology analysis, the attention-based refinement module improves over top-down RP-R-CNN by $K$ 4 in $K$ 5. On a held-out test set of 64 images, the ResNet-101 variant reports mIoU $K$ 6, $K$ 7, $K$ 8, and $K$ 9. The centerline-based tail measurement method, evaluated on 50 sperm against ImageJ-based manual reference, reports approximately $i$ 0 length accuracy, $i$ 1 width accuracy, and $i$ 2 curvature accuracy (Chen et al., 2024).

In robot-assisted surgery, fully supervised SurgPIS on EndoVis2018 with ResNet-50 achieves PartPQ $i$ 3, reported as $i$ 4 percentage points over a strong baseline. Under weak supervision on EndoVis2018, EndoVis2017, and SAR-RARP50, the Swin-B model achieves PartPQ $i$ 5, $i$ 6 percentage points versus baseline. The same framework reports IIS $i$ 7 on EndoVis2018, PSS PartIoU $i$ 8 on SAR-RARP50, and cross-dataset generalization on GraSP with PartPQ $i$ 9 and $y_i$ 00 without fine-tuning (Wei et al., 25 Jul 2025).

6. Limitations, misconceptions, and prospective directions

The surveyed literature makes clear that PIS is not synonymous with a top-down RoI pipeline. The cited methods include an anchor-free detector with downstream part assembly, a rotated-proposal graph-aggregation system, a full-image attention-refinement architecture, a transformer-based mask-classification framework, and a 3D point-cloud pipeline with mean-shift clustering (Nguyen et al., 2021, Feng et al., 2022, Chen et al., 2024, Wei et al., 25 Jul 2025, Sun et al., 2022).

The same literature also exposes task-specific limitations. The 3D method explicitly notes reliance on good semantic predictions; in cases of severe semantic noise, fusion can be misled. It also identifies the fixed mean-shift bandwidth and the hand-tuned $y_i$ 01 in semantic-region-center pushing as limitations, and proposes differentiable clustering and more cyclical semantic–instance co-training as natural extensions (Sun et al., 2022). In surgery, the weakly supervised setting is sufficiently sensitive to pseudo-label quality that removing pseudo-label filtering reduces PartPQ on EndoVis2018 by $y_i$ 02 percentage points, but reduces PQ on EndoVis2017 by $y_i$ 03 percentage points and PartIoU on SAR-RARP50 by $y_i$ 04 percentage points; removing strong-versus-weak augmentation consistency also causes substantial drops (Wei et al., 25 Jul 2025).

At the same time, several papers argue for portability beyond their original domain. The sperm study states that two-stage mask refinement via attention can be applied wherever top-down instance segmentation suffers from context loss, explicitly mentioning long poles, ropes, and vessels, while its endpoint reconstruction strategy is said to generalize to curvilinear structures whose endpoint normals are corrupted by intersecting edges (Chen et al., 2024). The industrial work reports that high-solidity parts can be handled by setting part count $y_i$ 05, yielding on-par accuracy with standard methods (Feng et al., 2022). FAPIS concludes that pushing part discovery from detection to segmentation encourages a class-agnostic, compositional representation that generalizes to new categories from only a handful of examples (Nguyen et al., 2021).

Taken together, these formulations present PIS as a broad research program rather than a single algorithmic template: a program in which instance localization is coupled to internal structure, and in which that structure may be semantic, geometric, latent, hierarchical, or weakly supervised, depending on the data regime and application domain.