---
title: 'PinPoint3D: 3D Part Segmentation Framework'
url: https://www.emergentmind.com/topics/pinpoint3d
type: topic
---

# PinPoint3D: 3D Part Segmentation Framework

PinPoint3D denotes two distinct entities in recent research usage. In computer vision and embodied AI, it refers primarily to an interactive framework for fine-grained, multi-granularity 3D segmentation in scene-level point clouds, designed to generate precise part-level masks from only a few user point clicks [2509.25970]. In medical physics, the same name appears as the PinPoint3D T31022, a small-volume vented thimble ionization chamber investigated for ultra-high dose-per-pulse, FLASH-like electron-beam dosimetry [2512.14525]. This suggests that the term is strongly context-dependent, with the segmentation framework forming the dominant computational meaning and the dosimetry chamber representing a separate instrument-specific usage.

## 1. Terminology and conceptual scope

The computational PinPoint3D solves **interactive fine-grained 3D part segmentation in scene-level point clouds**. Its target is not merely object-instance isolation such as “cabinet,” but segmentation of specific functional or structural parts such as a drawer, handle, wheel, door panel, or chair leg. The paper frames the desired perception hierarchy as **scene → area → instance → part**, emphasizing that embodied agents often need sub-object understanding rather than object-level masks alone [2509.25970].

| Usage | Domain | Defining description |
|---|---|---|
| PinPoint3D | 3D scene understanding | interactive fine-grained, multi-granularity 3D segmentation from a few user point clicks |
| PinPoint3D T31022 | Dosimetry | small-volume vented thimble ionization chamber |

Within the segmentation literature, PinPoint3D is explicitly designed for **scene-level input**, **multiple parts per object**, and **multi-granularity segmentation**, meaning that the system aims to preserve both object-level and part-level consistency and can produce object masks as well as finer part masks. Operationally, the output is a set of binary masks over points, one mask per queried object or part, with a background channel [2509.25970].

The work is motivated by limitations of earlier interactive 3D segmentation systems. The paper states that **InterObject3D** segments one object at a time, **AGILE3D** supports multi-object scene segmentation but remains object-oriented, **PointSAM** can be repurposed for parts but is not purpose-built for fine-grained part decomposition, and **iSeg** performs interactive 3D part segmentation on single objects in isolation rather than sparse scene-level scans. Non-interactive part and hierarchical segmentation methods are described as functioning relatively well on clean CAD objects while degrading on sparse point clouds, noisy real-world scans, and scene-level data [2509.25970].

## 2. Problem formulation and interactive task model

PinPoint3D takes as input a **3D scene point cloud** \(P\) and a set of **sparse 3D point prompts** \(S\). “From a few clicks” is operationalized as user-provided clicks directly on the point cloud, with evaluation under **first-click performance**, **multi-click performance** after 3 and 5 clicks per part, and **NoC** thresholds measuring the number of clicks needed to reach a target IoU [2509.25970].

The system supports iterative refinement. A user clicks on a target part region, the model predicts a mask, and if errors remain the user adds more clicks and the mask is refined. The main text emphasizes positive clicks, while the appendix clarifies that negative clicks are also supported in training and implementation notes. The interaction protocol includes annotation mode, selection mode, and navigation mode through a web UI implemented in Vue.js and Three.js, with undo and visual feedback [2509.25970].

The click representation is query-based rather than distance-transform-based. Each click is encoded using **Fourier positional encoding** for 3D location and **1D temporal encoding** for click order. Foreground queries come from positive clicks; background queries are composed of learned anchors and, in the appendix formulation, optionally user negatives. Query groups are fused by per-point max, which the paper describes as permutation-invariant and conducive to specialization [2509.25970].

A central design choice is hierarchical conditioning. Instead of predicting a fine part directly from the entire scene, PinPoint3D first performs **scene-to-instance localization** and then **instance-to-part decomposition**. The paper presents this as a coarse-to-fine hierarchy: object-level decoding recovers the target object region, that region defines the target interior, and part-level decoding then reasons only within that interior. This reduces the difficulty of isolating a small part from a large sparse scene using only a few clicks [2509.25970].

## 3. Architecture and inference pipeline

The geometry representation is a **sparse 3D point cloud** processed on a **sparse voxel grid**. The backbone is a **3D sparse convolutional backbone** based on the **Minkowski Engine**; the architecture figure describes it as **Minkowski U-Net with a 1×1 adapter**. The backbone is kept frozen to preserve object-level semantics, while a lightweight residual adapter specializes features for part segmentation [2509.25970].

The adapter is given as
\[
\mathbf{F}_{\text{mask}} \;=\; \mathbf{F}_{\text{pcd}} \;+\; \alpha \cdot \text{Conv}_{1\times1}^{2}\Big(\text{ReLU}\big(\text{Conv}_{1\times1}^{1}(\mathbf{F}_{\text{pcd}})\big)\Big),
\]
where \(\mathrm{Conv}_{1\times1}^{1}\) reduces channels, \(\mathrm{Conv}_{1\times1}^{2}\) expands them, and \(\alpha \in (0,1]\) stabilizes the residual update. The stated purpose is to preserve stable object semantics from the frozen backbone while making features more part-sensitive [2509.25970].

The decoder is dual-level. First, a **Scene–Instance decoder** performs bidirectional attention between click queries and scene features. The appendix describes its block structure as AGILE3D-style **C2S**, **C2C**, **FFN**, and **S2C**. A mask head then predicts object-level masks. For query \(i\) at point \(p\), the logit is
\[
z_{i,p} = \mathbf{F}_{\text{mask}(p)}^\top \mathbf{w}_i.
\]
These logits are converted to assignments with a max operation across queries [2509.25970].

The hierarchy is enforced by **Targeted Attention Masking (TAM)**. From object predictions \(\hat{y}_n \in \{0,\dots,M\}\), the model derives a binary attention mask
\[
\mathbf{A}^{(t)} \in \{0,1\}^{Q\times N},
\]
where \(\mathbf{A}^{(t)}_{q,n}=0\) if query \(q\) may attend to point \(n\), and \(\mathbf{A}^{(t)}_{q,n}=1\) otherwise. Masked attention is then
\[
\alpha_{q,n} \;=\; \frac{\exp(s_{q,n}) \cdot \mathds{1}[\mathbf{A}^{(t)}_{q,n}=0]} {\sum_{n': \mathbf{A}^{(t)}_{q,n'}=0} \exp(s_{q,n'})}.
\]
Foreground part queries for object \(t\) can therefore attend only to points inside object \(t\), while background queries attend outside that target region. The appendix generalizes this to a user-specified target set \(\mathcal{S}\) and its union interior \(\Omega(\mathcal{S})\) [2509.25970].

The second stage is an **Instance–Part decoder**. For selected object \(i\), part queries
\[
\mathbf{P}^{i}\!\in\!\mathbb{R}^{N_p^{i}\times d}
\]
interact with object-restricted points \(\mathbf{O}^{i}\) through masked cross-attention, self-attention, and feed-forward blocks. When new clicks arrive, click queries are updated, TAM is recomputed from the current object prediction, and the part decoder reruns. The appendix states that this refinement occurs **without re-encoding the whole scene**, since scene features remain available [2509.25970].

## 4. PartScan data synthesis and training regime

A major contribution of PinPoint3D is **PartScan**, a large-scale scene-level dataset with dense part annotations. The paper argues that progress in fine-grained interactive 3D segmentation has been blocked by the lack of scene-level point clouds with dense part labels. PartScan is constructed by combining realism from **ScanNet** and part fidelity from **PartNet** [2509.25970].

The first component is pseudo-label generation on ScanNet. For decomposable object instances, the authors use **PartField** to compute continuous point-wise part features and then cluster those features into part segments. The clustering objective is
\[
\min_{\{\boldsymbol{\mu}_j\}_{j=1}^K} \sum_{j=1}^K \sum_{i \in C_j} \lVert \mathbf{f}_i - \boldsymbol{\mu}_j \rVert_2^2,
\]
with the number of clusters chosen by silhouette score:
\[
k^{*} = \arg\max_{k \in \{2,\dots,10\}} \frac{1}{N} \sum_{i=1}^N s_i.
\]
This produces pseudo part labels on real scanned scenes [2509.25970].

The second component inserts **PartNet** objects into ScanNet scenes. The paper selects **12 object categories**: Table, Refrigerator, StorageFurniture, Chair, Dishwasher, Microwave, Bag, Mug, Bottle, Lamp, Vase, and Faucet. Selection is based on indoor frequency, scale compatibility with ScanNet, and semantically meaningful part structure. To align domains, the authors analyze object size, volume, and point density across the two datasets, scale each PartNet object to match ScanNet statistics for its category, downsample the point cloud, and apply **farthest point sampling independently to each annotated part**. Synthetic objects are then inserted by estimating the floor plane, partitioning it into a 2D grid, and randomly placing objects in unoccupied cells until space is exhausted or an object limit is reached [2509.25970].

The model is trained with **cross-entropy + Dice losses** on both object- and part-level predictions. The appendix gives default coefficients \(\lambda_{\text{CE}} = 1.0\) and \(\lambda_{\text{Dice}} = 2.0\), with auxiliary supervision from intermediate decoder layers and click-adaptive weighting near clicked regions. Optimization uses **AdamW**, initial learning rate **\(1\times10^{-4}\)**, weight decay **\(1\times10^{-4}\)**, gradient clipping **0.1**, multi-step learning-rate decay after **1000 epochs**, total training length **1100 epochs**, validation every **50 epochs**, batch size **5** for training and **1** for validation [2509.25970].

Interactive training simulates **0–19 rounds** per episode, with only the final round contributing gradients. Positive clicks are sampled from mis-segmented regions of the target part, negatives from background, and active parts are capped or sampled in the range **3–10** or up to **10**, depending on the passage cited. The broad intention is to mimic iterative correction while controlling compute [2509.25970].

## 5. Empirical performance, ablations, and user study

On the main **PartScan all-part** setting, PinPoint3D reports **IoU1 55.8**, **IoU3 68.4**, **IoU5 71.3**, **NoC50 1.68**, **NoC65 3.46**, **NoC80 6.43**, **AP25 96.9**, and **AP50 85.7**. Under the same setting, **AGILE3D** reports **39.1**, **61.1**, **66.7**, **2.67**, **5.18**, **8.12**, **96.7**, and **78.2**, while **PointSAM** reports **48.4**, **52.6**, **52.7**, **74.1**, and **51.0** for the metrics listed in the table. The clearest gains over AGILE3D are **+16.7 points** in IoU1, **+7.3** in IoU3, **+4.6** in IoU5, improvement in NoC50 from **2.67** to **1.68**, and **+7.5** in AP50 [2509.25970].

On **PartScan random-part**, PinPoint3D reports **IoU1 50.0**, **IoU3 65.9**, **IoU5 69.7**, **NoC50 2.12**, **NoC65 3.92**, **NoC80 6.92**, **AP25 95.1**, and **AP50 81.5**. On **MultiScan all-part**, it reports **IoU1 44.4**, **IoU3 62.7**, **IoU5 68.1**, **NoC50 2.28**, **NoC65 4.53**, **NoC80 7.66**, **AP25 93.9**, and **AP50 78.9**; on **MultiScan random-part**, the corresponding numbers are **44.0**, **60.8**, **66.8**, **2.71**, **4.93**, **7.74**, **94.0**, and **77.3**. The paper notes that gains on MultiScan are modest relative to PartScan, attributing this to MultiScan’s relatively less fine-grained part annotations [2509.25970].

PinPoint3D is also evaluated at the object level. On **PartScan**, it reports **86.7**, **97.0**, and **98.0** for IoU1, IoU3, and IoU5, compared with **83.64**, **96.87**, and **97.69** for AGILE3D. On **MultiScan**, however, AGILE3D remains higher, with **58.46**, **75.04**, and **81.02**, while PinPoint3D reports **57.1**, **72.3**, and **78.6**. This indicates that the architecture preserves or slightly improves coarse performance in-domain, but incurs a cross-domain object-level drop [2509.25970].

The category-wise PartScan mIoU values are: **Bag 61.8**, **Bottle 65.8**, **Chair 74.4**, **Dishwasher 59.1**, **Faucet 65.4**, **Lamp 63.1**, **Microwave 49.6**, **Mug 71.9**, **Refrigerator 62.3**, **StorageFurniture 72.2**, **Table 74.1**, and **Vase 66.5**. The best categories are therefore **Chair**, **Table**, **StorageFurniture**, and **Mug**, while **Microwave** is the hardest category at **49.6 mIoU** [2509.25970].

The ablations isolate the hierarchical design. Removing the dedicated **Part-Transformer** drops performance from baseline **IoU1 55.8**, **IoU3 68.4**, **IoU5 71.6**, **NoC50 1.70**, **NoC65 3.83**, **NoC80 7.10** to **43.7**, **60.2**, **65.0**, **2.54**, **5.46**, **8.60**. Replacing the frozen backbone plus adapter with direct backbone fine-tuning yields similar part metrics at first click but badly degrades object segmentation: **A+F** gives object **IoU@1 86.7**, **IoU@3 97.0**, **AP25 90.6**, **AP50 89.9**, while **NA+UF** gives **58.9**, **92.2**, **88.5**, **85.6**. Multi-object training is also better than single-object training, and combined **PartScan** training outperforms training on **PartNet (in-scene)** or **ScanNet-PartField** alone [2509.25970].

The user study includes **6 non-expert participants**, each given written instructions, a verbal demo, and a **15-minute practice session**, then asked to annotate **20 scene point clouds** containing **5–10 parts** each. On **PartScan**, human performance is \(\overline{\text{IoU}@3} = 61.9\) and \(\overline{\text{IoU}@5} = 74.7\), compared with simulated performance of **54.7** and **73.7**. On **MultiScan**, humans achieve **64.5** and **73.7**, compared with simulated **55.1** and **74.3**. The paper interprets this as evidence that simulated interaction is reasonably realistic and that the interface is usable by non-experts [2509.25970].

## 6. Distinct usages and adjacent research directions

Outside interactive segmentation, **PinPoint3D** also appears as the **PinPoint3D T31022 (PP3D)** in dosimetry. In that context it is a **small-volume vented thimble ionization chamber** with **inner radius \(0.30\ \mathrm{mm}\)**, **outer radius \(1.45\ \mathrm{mm}\)**, **height \(2.90\ \mathrm{mm}\)**, and **sensitive volume \(12\ \mathrm{mm^3}\)**. The chamber was studied under **20 MeV** electron beams with **0.1 Gy to 9.3 Gy per pulse**, and the paper concludes that thimble ionization chambers under ultra-high DPP exhibit strong volume recombination and a large polarity effect. For PP3D specifically, inclusion of the **conductive guard ring** in the complete **2D** finite-element geometry is described as **mandatory** for accurate simulation; simplified geometry can overestimate CCE by up to **18%** at high DPP [2512.14525].

Several neighboring computational works address superficially similar but technically distinct problems. **TAPIP3D** addresses long-term **3D point tracking** in monocular RGB and RGB-D videos using a camera-stabilized spatio-temporal 3D feature cloud, Local Pair Attention, and world-space tracking [2504.14717]. “**Commodifying Pointing in HRI**” addresses **3D pointing gesture detection** from RGB-D images by estimating a face-to-hand ray and intersecting it with a ground plane, operating at **better than 30 Hz** on commodity hardware [1902.02636]. The work summarized under arXiv **2403.13438** is explicitly noted as **not about “PinPoint3D” directly**; instead it studies zero-shot task hallucination from a single RGB image through object discovery, partial 3D reconstruction, waypoint planning, and rendered task-execution videos, with relevance mainly to object-level 3D grounding and waypoint-based manipulation planning rather than point-level pinpointing [2403.13438].

Taken together, these distinctions delimit the specific meaning of PinPoint3D in computer vision. It is not a general label for 3D pointing, 3D point tracking, or geometry-aware manipulation planning. Rather, in its principal computational usage, it denotes a hierarchical interactive framework for **scene-level fine-grained 3D part segmentation** from sparse clicks, combining a frozen sparse-convolutional backbone, adapter-based feature specialization, scene-to-instance decoding, instance-to-part decoding, and Targeted Attention Masking to turn minimal 3D interaction into actionable part-level masks [2509.25970].

Source: https://www.emergentmind.com/topics/pinpoint3d