Rigid Part Segmentation in 3D

• Rigid part segmentation is the process of decomposing a 3D object into geometrically coherent, rigid components that behave as subassemblies under motion.
• It employs diverse methodologies such as native 3D learning, promptable segmentation, and motion-based clustering to accurately partition static shapes and dynamic scenes.
• Evaluation relies on metrics like IoU, AP, and scene flow consistency while addressing challenges like occlusion, thin features, and fused geometries.

Searching arXiv for papers on rigid part segmentation and the cited works. Rigid part segmentation is the decomposition of an object into components that are treated as rigid under a chosen modeling assumption. In current literature, the term spans several closely related settings: for static 3D assets it often denotes a class-agnostic partition into geometrically coherent, piecewise-contiguous components with clean boundaries and volumetric consistency; for CAD-like parsing it may additionally require semantic instance labels, oriented bounding boxes, and hierarchical relations; for dynamic scenes it denotes partitioning points or pixels into subsets that each follow a single rigid motion in $SE(3)$ over time. Recent work has expanded the field from supervised taxonomy-bound part labeling toward promptable, open-world, motion-driven, and training-free formulations, while also exposing a central divide between surface-oriented methods derived from 2D lifting and native 3D methods that learn directly from volumetric geometry (Zhu et al., 26 Sep 2025).

1. Definitions and problem formulations

Rigid part segmentation is not a single task with a universal formalization. In native 3D open-world segmentation, it is defined as decomposing a 3D object into geometrically coherent, piecewise-contiguous components that would behave as rigid subassemblies under motion, while emphasizing clean boundaries, compactness, and volumetric consistency rather than predefined semantic names (Zhu et al., 26 Sep 2025). In CAD-oriented parsing, the same phrase can mean partitioning a static shape’s point cloud into semantic, instance-level parts that are geometrically rigid and consistently bounded by oriented boxes, then embedding those parts in an n-ary hierarchical tree with parent-child and sibling relations such as translational, rotational, reflective symmetry, and adjacency (Kim et al., 2022). In industrial bin picking, it is framed as recovering accurate pixel masks for constituent rigid parts and then assembling them into full object instances under clutter and occlusion (Feng et al., 2022).

A parallel motion-centric formulation defines rigid parts through shared rigid-body motion. In dynamic point clouds, points assigned to the same part share a rigid transform $T_k=(R_k,t_k)$ such that scene flow satisfies $s_i \approx R_k x_i + t_k - x_i$ for all points in that part (Zhong et al., 2023). MotionBits sharpens this notion further by introducing the MotionBit as the smallest unit in motion-based segmentation defined by kinematic spatial twist equivalence in a fixed world frame, independent of semantics (Qian et al., 6 Mar 2026). Earlier pair-based articulated analysis similarly sought rigid parts and motions explaining how one shape transforms into another, with per-part rigid transformations in $SE(3)$ inferred jointly with segmentation (Yi et al., 2018).

These formulations differ in what they regard as the source of evidence. Static 3D methods privilege intrinsic geometry, topology, and learned feature fields. Structure-aware parsers add hierarchical priors and box consistency. Motion-based approaches derive parts from relative motion, scene flow, or spatial twists. Image-based methods often reduce rigid part segmentation to semantic part segmentation or prompt-guided part localization, but typically without explicit rigidity constraints. This suggests that “rigid part segmentation” functions as an umbrella term whose exact meaning is determined by the representation—mesh, point cloud, RGB image, RGB-D video, or dynamic scene—and by whether rigidity is treated as an implicit geometric prior, an explicit kinematic constraint, or an assembly requirement.

2. Representations, priors, and the role of native 3D geometry

A major distinction in recent work is between methods trained natively on 3D data and methods that derive 3D parts by lifting supervision from 2D segmentation systems. Native 3D approaches argue that direct supervision on 3D shape–part pairs captures intrinsic geometry, including interior volumes, avoids per-view inconsistencies and occlusion artifacts inherent in 2D lifting, and enforces volumetric consistency (Zhu et al., 26 Sep 2025). PartSAM exemplifies this position through a prompt-guided encoder–decoder built around a dual-branch triplane encoder whose frozen branch preserves 2D priors distilled from SAM via PartField, while a learnable branch adapts to native 3D part supervision and can ingest normals and RGB through a zero-conv layer (Zhu et al., 26 Sep 2025). P$^3$-SAM adopts a different native 3D design: a PointTransformerV3-based feature extractor, a two-stage multi-head point-promptable segmentor, and an IoU predictor operating on point clouds sampled from mesh surfaces (Ma et al., 8 Sep 2025).

PartField provides a related but feedforward formulation of native 3D part understanding. It learns a continuous 3D feature field $f:\mathbb{R}^3\rightarrow\mathbb{R}^d$ with $d=448$, represented through a PVCNN encoder, orthogonal projection to triplanes, a 2D CNN plus Transformer stack, and bilinear querying of triplane features at arbitrary 3D points (Liu et al., 15 Apr 2025). Rather than directly outputting masks, the learned field is clustered to obtain hierarchical part decompositions. COPS occupies an intermediate position: it is training-free and multi-view, but explicitly imposes 3D geometric structure by back-projecting dense 2D DINOv2 features to points and then applying spatially and semantically consistent geometric feature aggregation using FPS and KNN in coordinate space and feature space (Garosi et al., 2024).

By contrast, methods based on lifting multi-view 2D masks into 3D are characterized in the recent literature as often producing surface-only groupings detached from intrinsic 3D structure and as struggling with interior components and occlusions (Zhu et al., 26 Sep 2025). The distinction is not only architectural but epistemic. Surface-derived pipelines infer parts from visible regions and then reconcile them in 3D; native 3D models learn directly in volumetric or point-set representations. This suggests why native 3D systems are repeatedly associated with segmentation of occluded regions, interior cavities, and internal structures, whereas 2D-lifted pipelines are repeatedly described as more brittle on such cases.

A different family of priors comes from structure and motion. Seg&Struct represents a shape by a hierarchy $S=(P,H,R)$ whose node semantics, oriented boxes $\theta_i=(t_i,s_i,q_i)$, and relation matrix over sibling subsets support plausible rigid layouts (Kim et al., 2022). Motion-centric methods instead encode rigidity through piecewise-$SE(3)$ transformations, rigid scene flow consistency, or temporal global energies over RGB-D sequences (Zhong et al., 2023). In manufacturing-oriented partitioning, Surface2Volume goes further and treats rigid parts as volumetric subassemblies constrained by collision-free translational disassembly and exact conformity to a prescribed surface segmentation (Araújo et al., 2019).

3. Core methodological families

One methodological family is promptable segmentation. PartSAM follows the design philosophy of SAM but in native 3D: the encoder turns the input shape into spatially structured tokens, 3D point prompts are mapped to prompt embeddings by sampling a continuous feature field and adding position embeddings, and a lightweight prompt-conditioned decoder returns binary masks and an IoU confidence (Zhu et al., 26 Sep 2025). The decoder introduces special tokens $T_k=(R_k,t_k)$0 and $T_k=(R_k,t_k)$1, performs bidirectional cross-attention,

$T_k=(R_k,t_k)$2

and computes point-level mask logits by a dot product with the refined output token. P$T_k=(R_k,t_k)$3-SAM is also promptable, but it restricts prompts to a single positive 3D point and uses three stage-1 heads, three stage-2 refinement heads, and an IoU head that predicts three quality scores for the candidate masks (Ma et al., 8 Sep 2025). In 2D images, Guided SAM learns positional prompts from coarse patch annotations by training DINOv2-feature SVMs that form ROIs and point prompts for SAM, thereby addressing SAM’s tendency to over-segment parts when used naïvely (Rooij et al., 13 Jan 2025).

A second family is feedforward part-field learning and clustering. PartField trains a continuous feature field by distilling 2D and 3D part proposals and optimizing an InfoNCE-like contrastive objective with uniform negatives, 3D-hard negatives, and feature-hard negatives (Liu et al., 15 Apr 2025). At inference, segmentation is produced by clustering point or face features using cosine distance, often with agglomerative clustering constrained by mesh adjacency to obtain crisp connected parts and a hierarchy. COPS also separates decomposition from labeling: K-means yields $T_k=(R_k,t_k)$4 geometric regions from aggregated features, and CLIP-based zero-shot labeling is deferred to the end, with Hungarian alignment used to match clusters to textual masks (Garosi et al., 2024). Joint neural SDF reconstruction and segmentation pursues a related one-pass idea for CAD meshes: a SIREN-based implicit network predicts both signed distance $T_k=(R_k,t_k)$5 and part logits $T_k=(R_k,t_k)$6, allowing semantically structured meshes with any number of parts to be extracted by Marching Cubes and per-vertex label evaluation (Fan et al., 4 Oct 2025).

A third family combines segmentation with structure inference. Seg&Struct begins with an off-the-shelf 3D instance segmenter $T_k=(R_k,t_k)$7 based on PointGroup, maps segments to StructureNet hierarchy nodes, predicts boxes and relations by hierarchical message passing, and then rectifies segmentation by detecting box-overlap confusion and applying directed merges classified from structure-aware features (Kim et al., 2022). The forward path segmentation$T_k=(R_k,t_k)$8structure and the backward path structure$T_k=(R_k,t_k)$9segmentation refinement are presented as a closed loop in which explicit structural priors regularize rigid part boundaries.

A fourth family is motion-driven unsupervised rigid segmentation. OGC discovers rigid objects and object parts from sequential point clouds by jointly training a single-frame multi-object mask predictor, a self-supervised scene flow estimator, and an object geometry consistency objective. Its central dynamic rigid loss fits a differentiable weighted-Kabsch transform per mask and penalizes the discrepancy between rigidly transformed points and scene-flow-displaced points (Song et al., 2022). The SE(3)-equivariant framework of multi-body rigid segmentation uses an EPN backbone with $s_i \approx R_k x_i + t_k - x_i$0 rotational channels, a point-level invariant segmentation head, and a part-level equivariant motion head, optimized jointly through scene flow, mask agreement, and weighted Kabsch estimation (Zhong et al., 2023). MotionBits is learning-free: it estimates local 2D rigid transforms from optical flow, converts them into spatial twists in a fixed frame, builds a graph weighted by Mahalanobis twist similarity, diffuses labels, and uses Markov clustering and SAM2 refinement to produce masks (Qian et al., 6 Mar 2026).

A fifth family addresses specialized domains. In industrial bin picking, a Mask R-CNN–inspired pipeline predicts rotated part proposals, binary part masks, and part-to-part offsets, then assembles parts into object instances via reciprocal offset consistency; ground-truth part labels are obtained automatically through concavity-driven approximate convex decomposition (Feng et al., 2022). In RGB-D video, temporally consistent motion segmentation formulates global sequence-level energies over label assignments and rigid motions, alternates between object-motion optimization and graph-cut segmentation, and fuses consistently labeled segments into object reconstructions (Bertholet et al., 2016). Surface2Volume addresses a manufacturing-specific variant in which the goal is not only to segment rigid volumetric parts but to make them assemblable under linear translational motions while strictly conforming to an exterior surface segmentation (Araújo et al., 2019).

4. Objectives, metrics, and algorithmic criteria

Loss functions vary according to the chosen representation and source of supervision. Native 3D promptable segmentation in PartSAM uses focal and dice losses plus a triplet contrastive term on encoder features,

$s_i \approx R_k x_i + t_k - x_i$1

with

$s_i \approx R_k x_i + t_k - x_i$2

$s_i \approx R_k x_i + t_k - x_i$3

and

$s_i \approx R_k x_i + t_k - x_i$4

The paper explicitly notes that it does not introduce boundary consistency or compactness constraints; clean rigid-like parts emerge from promptable supervision, IoU-based candidate selection, and large-scale native 3D training (Zhu et al., 26 Sep 2025). P$s_i \approx R_k x_i + t_k - x_i$5-SAM likewise combines dice and focal terms in a two-stage min-over-heads loss and supervises its IoU predictor with MSE against the true IoU of binarized masks (Ma et al., 8 Sep 2025).

Structure-guided parsing and CAD-oriented SDF segmentation use different objectives. Seg&Struct optimizes a structure inference loss

$s_i \approx R_k x_i + t_k - x_i$6

with $s_i \approx R_k x_i + t_k - x_i$7 and $s_i \approx R_k x_i + t_k - x_i$8, and a separate focal merge loss for segmentation rectification with $s_i \approx R_k x_i + t_k - x_i$9 and $SE(3)$0 (Kim et al., 2022). Joint neural SDF reconstruction and segmentation minimizes

$SE(3)$1

with $SE(3)$2, $SE(3)$3, $SE(3)$4, $SE(3)$5, and $SE(3)$6, where $SE(3)$7 is cross-entropy on labeled manifold samples and the remaining terms enforce SDF reconstruction, Eikonal regularity, and curvature control (Fan et al., 4 Oct 2025).

Motion-based methods use rigidity-consistency objectives. OGC’s dynamic loss penalizes disagreement between motion induced by per-mask rigid transforms and observed scene flow,

$SE(3)$8

and supplements it with a smoothness regularizer over neighboring points and a geometry invariance loss across rigidly augmented views (Song et al., 2022). The multi-body SE(3)-equivariant approach defines a segmentation loss weighted by a motion–flow consensus factor,

$SE(3)$9

and couples it to weighted Kabsch estimation of per-part rigid transformations (Zhong et al., 2023). MotionBits does not train a model, but its graph weights are explicit functions of local twist similarity through a Mahalanobis kernel (Qian et al., 6 Mar 2026).

Evaluation is correspondingly heterogeneous. IoU and mean IoU recur across static segmentation, often with class-agnostic matching between predicted and ground-truth parts (Zhu et al., 26 Sep 2025). Interactive segmentation in promptable systems commonly reports IoU@i after $^3$0 prompts (Zhu et al., 26 Sep 2025). Structure-aware systems add AP at IoU 0.25 for part prediction and edge prediction error for relations (Kim et al., 2022). CAD reconstruction-plus-segmentation reports mIoU, accuracy, and a Segmentation Consistency metric defined as the fraction of k-nearest neighbors sharing the same predicted label, with $^3$1 and $^3$2 anchors (Fan et al., 4 Oct 2025). Motion-centric methods report AP, PQ, F1, precision, recall, mIoU, RI, EPE3D, or macro-averaged overlap and boundary metrics depending on whether the domain is dynamic point clouds, videos, or motion benchmarks (Zhong et al., 2023).

5. Empirical behavior, capabilities, and trade-offs

Native 3D promptable systems report strong gains in both interactive and automatic settings. On PartObjaverse-Tiny, PartSAM attains IoU@1 of $^3$3 versus Point-SAM’s $^3$4, and on PartNet-E it reaches $^3$5 versus $^3$6; in automatic “Segment-Every-Part” mode it reports mIoU $^3$7 on PartObjaverse-Tiny and $^3$8 on PartNet-E, outperforming SAMesh, SAMPart3D, and PartField* on the cited benchmarks (Zhu et al., 26 Sep 2025). P$^3$9-SAM reports full-segmentation averages of $f:\mathbb{R}^3\rightarrow\mathbb{R}^d$0 on PartObj-Tiny without connectivity, $f:\mathbb{R}^3\rightarrow\mathbb{R}^d$1 on PartObj-Tiny with connectivity, and $f:\mathbb{R}^3\rightarrow\mathbb{R}^d$2 on PartNetE point clouds, alongside interactive averages of $f:\mathbb{R}^3\rightarrow\mathbb{R}^d$3, $f:\mathbb{R}^3\rightarrow\mathbb{R}^d$4, and $f:\mathbb{R}^3\rightarrow\mathbb{R}^d$5 on PartObj-Tiny, PartObj-Tiny-WT, and PartNetE respectively (Ma et al., 8 Sep 2025). PartField reports class-agnostic mIoU $f:\mathbb{R}^3\rightarrow\mathbb{R}^d$6 on PartObjaverse-Tiny and $f:\mathbb{R}^3\rightarrow\mathbb{R}^d$7 on PartNetE, with runtime around $f:\mathbb{R}^3\rightarrow\mathbb{R}^d$8 seconds per shape (Liu et al., 15 Apr 2025). COPS, in zero-shot evaluation, reports $f:\mathbb{R}^3\rightarrow\mathbb{R}^d$9 mIoUI and $d=448$0 mIoUC on ShapeNetPart, $d=448$1 mIoUI and $d=448$2 mIoUC on PartNet, and $d=448$3 aIoUI and $d=448$4 aIoUC on PartNetE (Garosi et al., 2024).

These systems differ in what their quantitative behavior implies. PartSAM and P$d=448$5-SAM emphasize promptability, candidate-mask ranking, and automatic decomposition of arbitrary shapes. PartField emphasizes continuous feature consistency, hierarchical clustering, and cross-shape transfer. COPS emphasizes zero-shot decomposition before text-based part labeling. Joint neural SDF reconstruction emphasizes single-pass mesh reconstruction plus coherent labels for arbitrary part counts, with segmentation results uniformly high on the reported 50-shape evaluation: mIoU $d=448$6, accuracy $d=448$7, and consistency $d=448$8 across tested heads (Fan et al., 4 Oct 2025). This suggests that “state of the art” in rigid part segmentation is not a single frontier but several overlapping ones: promptable native 3D decomposition, feedforward class-agnostic clustering, zero-shot multi-view aggregation, and joint geometry-plus-label implicit modeling.

Structure-aware and motion-aware methods reveal different trade-offs. Seg&Struct improves structure inference dramatically, reaching average AP $d=448$9 and EE $S=(P,H,R)$0, and yields a smaller but consistent segmentation refinement from PointGroup’s $S=(P,H,R)$1 mAP to $S=(P,H,R)$2 overall on PartNet categories (Kim et al., 2022). OGC reports unsupervised AP $S=(P,H,R)$3, PQ $S=(P,H,R)$4, F1 $S=(P,H,R)$5, mIoU $S=(P,H,R)$6, and RI $S=(P,H,R)$7 on SAPIEN part instance segmentation, while also achieving strong performance on OGC-DR and OGC-DRSV and reasonable transfer to KITTI-SF and LiDAR evaluation (Song et al., 2022). The multi-body SE(3)-equivariant model reports unsupervised AP $S=(P,H,R)$8, PQ $S=(P,H,R)$9, F1 $\theta_i=(t_i,s_i,q_i)$0, mIoU $\theta_i=(t_i,s_i,q_i)$1, RI $\theta_i=(t_i,s_i,q_i)$2, and EPE3D $\theta_i=(t_i,s_i,q_i)$3 on SAPIEN, with a supervised variant reducing EPE3D to $\theta_i=(t_i,s_i,q_i)$4 (Zhong et al., 2023). MotionBits reports a $\theta_i=(t_i,s_i,q_i)$5 macro-averaged mIoU improvement over baselines on the MoRiBo benchmark and a $\theta_i=(t_i,s_i,q_i)$6 improvement over the two strongest baselines, while also demonstrating downstream gains for stack-building and manipulation (Qian et al., 6 Mar 2026).

Application-specific pipelines further clarify the field’s operating regimes. In industrial bin picking, the part-aware instance segmentation pipeline achieves average AP@50 $\theta_i=(t_i,s_i,q_i)$7, AP@75 $\theta_i=(t_i,s_i,q_i)$8, and mIoU $\theta_i=(t_i,s_i,q_i)$9, markedly exceeding retrained Mask R-CNN and SOLOv2 on thin, concave, tightly packed objects (Feng et al., 2022). In two-frame rigid motion segmentation from monocular video, the geometric-modular network reports object IoU around $SE(3)$0 and background IoU around $SE(3)$1 on KITTI, while the inferred rigid motions improve scene flow and disparity estimates (Yang et al., 2021). For semantic image parsing rather than class-agnostic decomposition, LGFormer reports joint-training part/object mIoU of $SE(3)$2 on PartImageNet and $SE(3)$3 on Pascal-Part, but the paper explicitly states that rigidity is not modeled (Xie et al., 2024). Guided SAM, likewise, is label-efficient and part-focused in images, improving average IoU to $SE(3)$4 on car parts with patch-level training on $SE(3)$5 images, yet it is still fundamentally a prompt-guided semantic part localizer rather than a native 3D rigid decomposition model (Rooij et al., 13 Jan 2025).

6. Limitations, misconceptions, and open directions

A recurrent misconception is that rigid part segmentation always entails explicit rigidity or kinematics. Several recent systems explicitly reject that interpretation. PartSAM targets rigid-like decomposition but “does not explicitly model kinematics or rigid-body constraints”; it learns rigid-like parts implicitly from geometry and large-scale part supervision (Zhu et al., 26 Sep 2025). LGFormer performs semantic part segmentation but “does not impose kinematic constraints or shape rigidity priors” (Xie et al., 2024). PartField learns general “part-ness,” not physical rigidity, even though its supervision sources and empirical behavior align well with CAD and articulation boundaries (Liu et al., 15 Apr 2025). This suggests that many high-performing methods achieve rigid part behavior as an emergent property of geometry, data scale, or motion consistency rather than from a hard-coded mechanical model.

Another misconception is that semantics and rigid decomposition are equivalent. PartSAM outputs masks without semantic names, like 2D SAM (Zhu et al., 26 Sep 2025). PartField deliberately avoids predefined templates or text labels and instead clusters a continuous field (Liu et al., 15 Apr 2025). COPS explicitly disentangles decomposition from semantic labeling and can be evaluated in an unlabeled “$SE(3)$6” mode (Garosi et al., 2024). By contrast, Seg&Struct and many image-based part segmentation systems rely on semantic labels as core supervision (Kim et al., 2022). The field therefore contains both semantic and non-semantic notions of parts, and confusion between them obscures methodological differences.

Failure modes are also domain-specific. Native 3D systems still struggle on extremely thin features, overlapping assemblies, highly fragmented artist meshes, or heavily fused geometry with weak surface delineations (Zhu et al., 26 Sep 2025). Joint neural SDF segmentation inherits per-face supervision limits and can exhibit label leaks or frayed borders on thin structures and tight fillets (Fan et al., 4 Oct 2025). Structure-guided methods are sensitive to the initial segmentation backbone and to hierarchy-rule mismatches (Kim et al., 2022). Motion-driven systems cannot separate parts that never exhibit relative motion within the observation window, and may merge bodies sharing identical twists or suffer under occlusion, blur, biased flow, or non-rigid backgrounds (Qian et al., 6 Mar 2026). Two-frame monocular motion segmentation remains vulnerable to pure rotation, insufficient parallax, poor depth priors, and textureless regions (Yang et al., 2021). Surface2Volume, finally, is limited by its translational-motion assumption and by the complexity of finely segmented, high-curvature inputs (Araújo et al., 2019).

Open directions are stated explicitly across several works. PartSAM identifies integration of part semantics as future work (Zhu et al., 26 Sep 2025). Joint neural SDF segmentation proposes boundary-aware training, higher-resolution supervision, and adaptive sampling near part boundaries (Fan et al., 4 Oct 2025). Seg&Struct points toward explicit rigid-body constraints, motion-based rigid segmentation across frames, and kinematic model integration with hinge or slider relations (Kim et al., 2022). MotionBits identifies full $SE(3)$7 camera compensation, depth-aware twist estimation, learning-based refinement, and multi-frame optimization as future extensions (Qian et al., 6 Mar 2026). Collectively, these directions suggest a plausible convergence: native 3D geometric representations, motion or structure cues when available, promptable or automatic interaction, and optional semantic grounding layered on top of a class-agnostic rigid decomposition core.