Point-Consistent Part Encoder

• The paper introduces a novel neural architecture that generates globally coherent, part-aware embeddings from 3D point clouds using multi-view 2D distillation and 3D contrastive learning.
• It leverages a tri-plane representation with transformer blocks to integrate 2D segmentation cues and enforce semantic consistency across views.
• Empirical evaluations demonstrate significant improvements in segmentation performance, with clear gains in mIoU on benchmarks compared to ablated model configurations.

The Point-Consistent Part Encoder is a neural architecture designed for part-level segmentation of 3D point clouds, introduced in the S²AM3D framework. Its core aim is to generate globally coherent, part-aware per-point embeddings from unstructured 3D point data by aggregating multi-view 2D features and enforcing 3D part consistency through contrastive learning. The encoder bridges the gap between powerful 2D segmentation priors, such as those from the Segment Anything Model (SAM), and volumetric 3D representation, producing point features that are consistent across views and aligned with true part boundaries (Su et al., 30 Nov 2025).

1. Architectural Composition and Data Flow

The encoder operates on an input point cloud $P = \{p_1, \ldots, p_N\}$, with each point $p_n \in \mathbb{R}^3$ and $N \approx 10,000$. The processing pipeline begins by encoding the point cloud into a volumetric grid via a sparse-voxel Point-Voxel CNN (PVCNN), following Liu et al. (2019). This volumetric embedding is then collapsed into three orthogonal axis-aligned feature planes $$T_{xy}, T_{yz}, T_{zx} \in \mathbb{R}^{D \times H \times W}$$, where $D=448$ and $H=W=512$, converted into a tri-plane representation $T \in \mathbb{R}^{3 \times D \times H \times W}$.

Global contextualization is achieved by flattening each plane and applying $L_\tau$ transformer blocks (multi-head self-attention and MLP layers). Typically, $L_\tau = 8$, though experiments found similar efficacy for $6 \leq L_\tau \leq 12$. After refinement, these tri-planes remain in the same shape but encode enhanced global features.

Un-projection from the tri-plane to point-wise embeddings is performed via trilinear sampling on each feature plane at the coordinates of $p_n$, summing the sampled feature vectors:

$$F_n = T_{xy}(x_n, y_n) + T_{yz}(y_n, z_n) + T_{zx}(z_n, x_n) \in \mathbb{R}^D$$

Aggregating across all points yields $F \in \mathbb{R}^{N \times D}$.

2. Multi-View 2D Feature Distillation

The tri-plane representation serves as an implicit 3D field that enables rendering of any 2D view without re-encoding the point cloud. During training, a random camera pose is sampled, rays are projected for each pixel, and features are aggregated from the tri-planes to form a feature map $Z \in \mathbb{R}^{D \times H_i \times W_i}$ in image space.

A lightweight 2D segmentation head (a $1 \times 1$ convolution or MLP per location) predicts per-pixel logits, and a standard cross-entropy distillation loss is computed against pseudo-masks from a frozen SAM model. This process compels the tri-plane features to embed strong, SAM-like semantic part cues. However, relying solely on multi-view 2D distillation can result in inconsistencies across views, particularly in cases of occlusions or intricate structures.

3. 3D Contrastive Learning for Consistent Part Embeddings

To address the limitations of view consistency, a supervised 3D contrastive learning head is added. Each training batch processes a single object and uniformly samples a subset $\hat{P}$ of $\hat{N}=1000$ points. For each anchor $i$, positives $\hat{P}(i)$ are all other sampled points sharing the same part label $y_i$, and negatives are points with different labels.

All feature vectors are $\ell_2$-normalized, and similarities are computed as $s_{ij} = \frac{f_i^\top f_j}{\tau}$ with temperature parameter $\tau = 0.07$. The supervised InfoNCE loss is defined as:

$$L_{\text{contrast}} = \frac{1}{|\hat{P}|} \sum_{i \in \hat{P}} -\log \frac{\sum_{j \in \hat{P}(i)} \exp(s_{ij})}{\sum_{k \in \hat{P} \setminus \{i\}} \exp(s_{ik})}$$

This loss explicitly compacts features of same-part points and pushes apart features across part boundaries. It is weighted equally with the 2D cross-entropy loss from the distillation step.

4. Feature Fusion and Implementation Details

Fusion from the tri-plane representation to point-wise feature embeddings is performed in a view-agnostic, parameter-free manner. For each point $p_n$,

$$F_n = T_{xy}(x_n, y_n) + T_{yz}(y_n, z_n) + T_{zx}(z_n, x_n)$$

Grid sampling is performed using bilinear or trilinear interpolation on the planes. No additional learned weights are involved in the fusion process, contributing to model efficiency and interpretability.

5. Training Protocol and Hyperparameters

The encoder is pre-trained with a mixture of 2D distillation and 3D contrastive losses. Key hyperparameters include:

• Optimizer: AdamW
• Learning rate: $1 \times 10^{-5}$
• Batch size: 1 object per GPU, with $\hat{N}=1000$ sampled points
• Contrastive temperature: $\tau=0.07$
• Number of epochs: 15 on 8$\times$NVIDIA A6000 GPUs (approximately 1 day)
• Tri-plane dimensions: $D=448$, $H=W=512$
• Number of transformer blocks: $L_\tau=8$
• 2D view size: $512 \times 512$ pixels
• Segmentation head: 1$\times$1 convolution + softmax
• Loss weights: 1.0 for cross-entropy, 1.0 for contrastive loss

6. Empirical Effectiveness and Ablation Analysis

The impact of the point-consistent part encoder components was validated through ablation studies on the Interactive mIoU metric, using datasets PartObj-Tiny and PartNet-E. Table 1 summarizes results:

Setting	PartObj-Tiny	PartNet-E	Avg
Full (2D+3D + data + scale)	53.75	70.63	62.19
–– w/o 3D Contrastive	47.34	58.01	52.68
–– w/o Curated Data	46.83	59.40	53.12
Full (no scale input)	41.70	54.39	48.05
–– w/o 3D Contrastive	31.04	49.93	40.49
–– w/o Curated Data	40.79	53.80	47.30
–– w/o Scale Embedding	40.31	53.28	46.80

Qualitative analysis (Figure 1 in (Su et al., 30 Nov 2025)) demonstrates that removing the 3D contrastive branch causes feature clusters to bleed across part boundaries and leads to fragmented segmentation. The fully realized encoder yields tight, well-separated clusters and crisp, spatially and semantically coherent masks.

7. Significance and Implications

The Point-Consistent Part Encoder provides an effective pipeline for producing part-aware 3D point embeddings that are robust to input variability and consistent across views. Its three-stage design—incorporating (1) multi-view 2D prior distillation via tri-plane architectures and transformer-based global context, (2) a simple but effective unprojection-to-point mechanism, and (3) supervised 3D InfoNCE contrastive regularization—results in features that are both SAM-aware and spatially organized in accordance with true object parts. This directly addresses the previously unsolved challenge of achieving segmentation consistency throughout 3D space, even in the presence of complex geometries, occlusions, and size variation.