CoSMo3D: Open-World Promptable 3D Semantic Part Segmentation through LLM-Guided Canonical Spatial Modeling

Abstract: Open-world promptable 3D semantic segmentation remains brittle as semantics are inferred in the input sensor coordinates. Yet, humans, in contrast, interpret parts via functional roles in a canonical space -- wings extend laterally, handles protrude to the side, and legs support from below. Psychophysical evidence shows that we mentally rotate objects into canonical frames to reveal these roles. To fill this gap, we propose \methodName{}, which attains canonical space perception by inducing a latent canonical reference frame learned directly from data. By construction, we create a unified canonical dataset through LLM-guided intra- and cross-category alignment, exposing canonical spatial regularities across 200 categories. By induction, we realize canonicality inside the model through a dual-branch architecture with canonical map anchoring and canonical box calibration, collapsing pose variation and symmetry into a stable canonical embedding. This shift from input pose space to canonical embedding yields far more stable and transferable part semantics. Experimental results show that \methodName{} establishes new state of the art in open-world promptable 3D segmentation.

• The paper introduces CoSMo3D, a dual-branch framework that combines geometry-text alignment with canonical space modeling for promptable 3D semantic segmentation.
• It integrates a Point Transformer for 3D feature extraction and SigLIP for text semantics, using semantic contrastive alignment and novel canonical losses.
• Experimental results show significant mIoU improvements over baselines across diverse datasets, demonstrating robust segmentation performance in open-world settings.

Summary of "CoSMo3D: Open-World Promptable 3D Semantic Part Segmentation through LLM-Guided Canonical Spatial Modeling"

Introduction to CoSMo3D

The paper introduces CoSMo3D, a method for open-world promptable 3D semantic segmentation, which harnesses canonical space perception to better understand and interpret 3D shapes across varying poses and categories. The authors critique prior methods, such as Find3D, that rely on direct geometric-text matching, stating they fail to address semantic inconsistencies across different poses and categories. CoSMo3D introduces a dual-branch architecture that combines the advantages of geometry-text alignment with canonical space regularities to improve semantic part localization and overall segmentation performance.

Methodology

The proposed framework, depicted in (Figure 1), employs a dual-branch approach. The primary feature extraction branch utilizes a Point Transformer to encode 3D shape features, and a SigLIP model for text semantic features, enabling robust cross-modal part segmentation. The secondary canonical embedding branch is dedicated to training, leveraging semantic contrastive alignment, canonical map anchoring, and canonical box calibration losses to enforce consistent canonical space perception. This method aligns part semantics across varying shapes and poses into a stable canonical embedding.

Figure 1: We propose a dual-branch framework for open-world promptable 3D semantic segmentation: the feature extraction branch encodes 3D shape features (via Point Transformer) and text semantic features (via SigLIP) to enable cross-modal part segmentation. A training-only canonical embedding branch then enforces consistent canonical space perception via semantic contrastive alignment, canonical map anchoring, and canonical box calibration losses, ensuring robust reasoning across any shape in any pose.

Canonical Dataset Construction

The authors constructed a unified cross-category canonical dataset to support the learning of transferable canonical space priors, addressing existing datasets' limitations regarding cross-category alignment. Using an LLM-guided process, this dataset aligns 200 diverse categories into a universal canonical frame (Figure 2). This involves intra-category canonicalization using existing methods and a novel hierarchical alignment strategy for cross-category canonicalization, significantly improving scalability in open-world settings.

Figure 2: (a) Prior works perform category-level canonicalization, aligning intra-category shapes but neglecting cross-category consistency. (b) We cluster categories via LLM and align different categories relying on key semantic parts and functional consistency.

Addressing Symmetry and Pose Variation

Handling symmetrical objects within the canonical map anchoring framework is a critical challenge. The authors innovate with an order-invariant set loss on RGB-encoded canonical coordinates, circumventing the limitations of previous methods that required manual symmetry annotations (Figure 3). This loss aligns distributions rather than individual points, enabling consistent semantic part localization despite pose ambiguities.

Figure 3: Handling Symmetric Objects for Canonical Map Anchoring. (a) For symmetric shapes, multiple valid poses induce ambiguous point-wise canonical labels, making direct point-wise supervision unreliable. (b) Prior works rely on category-specific processing with manual symmetry annotation, limiting open-world scalability. (c) We instead apply an order-invariant set loss on RGB-encoded canonical coordinates that matches the overall layout of semantic parts in canonical space while remaining robust to symmetric pose ambiguities.

Experimental Results

The experimental results demonstrate CoSMo3D's superiority over existing methods in open-world 3D segmentation tasks. The model establishes a new state of the art, significantly outperforming baselines like Find3D across various datasets, including 3Dcompat-Coarse, 3Dcompat-Fine, ShapeNet-Part, and PartNet-E, particularly in terms of mean Intersection-over-Union (mIoU) scores.

Figure 4: Qualitative comparison of promptable 3D part segmentation. Across challenging cases (similar geometry with different semantics, noise-prone objects, cross-category semantics, and arbitrary poses), our method produces more accurate and consistent part localizations than existing baselines.

Conclusion

CoSMo3D represents a significant advancement in open-world 3D semantic segmentation by integrating canonical space reasoning with promptable segmentation models. The framework circumvents the constraints of direct geometry-text alignment by instilling a canonical reference frame shared across categories, achieving consistent spatial semantics that mimic human-like 3D perception. The implications of this work extend beyond segmentation, offering potential pathways for canonical space-based 3D reasoning, cross-modal grounding, and applications in next-stage 3D agent development. Future research may explore leveraging this framework for tasks demanding sophisticated 3D understanding and agent interactions within canonical spaces.