PinPoint3D: 3D Part Segmentation Framework

• PinPoint3D is an interactive framework for scene-level fine-grained 3D part segmentation that generates precise object and part masks from a few user clicks.
• It employs a hierarchical decoding process using a frozen sparse convolutional backbone, lightweight adapters, and targeted attention masking to refine segmentation results.
• Beyond computer vision, the term also denotes a dosimetry instrument, highlighting its context-dependent usage in both 3D segmentation and medical physics.

PinPoint3D denotes two distinct entities in recent research usage. In computer vision and embodied AI, it refers primarily to an interactive framework for fine-grained, multi-granularity 3D segmentation in scene-level point clouds, designed to generate precise part-level masks from only a few user point clicks (Zhang et al., 30 Sep 2025). In medical physics, the same name appears as the PinPoint3D T31022, a small-volume vented thimble ionization chamber investigated for ultra-high dose-per-pulse, FLASH-like electron-beam dosimetry (Paz-Martín et al., 16 Dec 2025). This suggests that the term is strongly context-dependent, with the segmentation framework forming the dominant computational meaning and the dosimetry chamber representing a separate instrument-specific usage.

1. Terminology and conceptual scope

The computational PinPoint3D solves interactive fine-grained 3D part segmentation in scene-level point clouds. Its target is not merely object-instance isolation such as “cabinet,” but segmentation of specific functional or structural parts such as a drawer, handle, wheel, door panel, or chair leg. The paper frames the desired perception hierarchy as scene → area → instance → part, emphasizing that embodied agents often need sub-object understanding rather than object-level masks alone (Zhang et al., 30 Sep 2025).

Usage	Domain	Defining description
PinPoint3D	3D scene understanding	interactive fine-grained, multi-granularity 3D segmentation from a few user point clicks
PinPoint3D T31022	Dosimetry	small-volume vented thimble ionization chamber

Within the segmentation literature, PinPoint3D is explicitly designed for scene-level input, multiple parts per object, and multi-granularity segmentation, meaning that the system aims to preserve both object-level and part-level consistency and can produce object masks as well as finer part masks. Operationally, the output is a set of binary masks over points, one mask per queried object or part, with a background channel (Zhang et al., 30 Sep 2025).

The work is motivated by limitations of earlier interactive 3D segmentation systems. The paper states that InterObject3D segments one object at a time, AGILE3D supports multi-object scene segmentation but remains object-oriented, PointSAM can be repurposed for parts but is not purpose-built for fine-grained part decomposition, and iSeg performs interactive 3D part segmentation on single objects in isolation rather than sparse scene-level scans. Non-interactive part and hierarchical segmentation methods are described as functioning relatively well on clean CAD objects while degrading on sparse point clouds, noisy real-world scans, and scene-level data (Zhang et al., 30 Sep 2025).

2. Problem formulation and interactive task model

PinPoint3D takes as input a 3D scene point cloud $P$ and a set of sparse 3D point prompts $S$. “From a few clicks” is operationalized as user-provided clicks directly on the point cloud, with evaluation under first-click performance, multi-click performance after 3 and 5 clicks per part, and NoC thresholds measuring the number of clicks needed to reach a target IoU (Zhang et al., 30 Sep 2025).

The system supports iterative refinement. A user clicks on a target part region, the model predicts a mask, and if errors remain the user adds more clicks and the mask is refined. The main text emphasizes positive clicks, while the appendix clarifies that negative clicks are also supported in training and implementation notes. The interaction protocol includes annotation mode, selection mode, and navigation mode through a web UI implemented in Vue.js and Three.js, with undo and visual feedback (Zhang et al., 30 Sep 2025).

The click representation is query-based rather than distance-transform-based. Each click is encoded using Fourier positional encoding for 3D location and 1D temporal encoding for click order. Foreground queries come from positive clicks; background queries are composed of learned anchors and, in the appendix formulation, optionally user negatives. Query groups are fused by per-point max, which the paper describes as permutation-invariant and conducive to specialization (Zhang et al., 30 Sep 2025).

A central design choice is hierarchical conditioning. Instead of predicting a fine part directly from the entire scene, PinPoint3D first performs scene-to-instance localization and then instance-to-part decomposition. The paper presents this as a coarse-to-fine hierarchy: object-level decoding recovers the target object region, that region defines the target interior, and part-level decoding then reasons only within that interior. This reduces the difficulty of isolating a small part from a large sparse scene using only a few clicks (Zhang et al., 30 Sep 2025).

3. Architecture and inference pipeline

The geometry representation is a sparse 3D point cloud processed on a sparse voxel grid. The backbone is a 3D sparse convolutional backbone based on the Minkowski Engine; the architecture figure describes it as Minkowski U-Net with a 1×1 adapter. The backbone is kept frozen to preserve object-level semantics, while a lightweight residual adapter specializes features for part segmentation (Zhang et al., 30 Sep 2025).

The adapter is given as

$$\mathbf{F}_{\text{mask}} \;=\; \mathbf{F}_{\text{pcd}} \;+\; \alpha \cdot \text{Conv}_{1\times1}^{2}\Big(\text{ReLU}\big(\text{Conv}_{1\times1}^{1}(\mathbf{F}_{\text{pcd}})\big)\Big),$$

where $\mathrm{Conv}_{1\times1}^{1}$ reduces channels, $\mathrm{Conv}_{1\times1}^{2}$ expands them, and $\alpha \in (0,1]$ stabilizes the residual update. The stated purpose is to preserve stable object semantics from the frozen backbone while making features more part-sensitive (Zhang et al., 30 Sep 2025).

The decoder is dual-level. First, a Scene–Instance decoder performs bidirectional attention between click queries and scene features. The appendix describes its block structure as AGILE3D-style C2S, C2C, FFN, and S2C. A mask head then predicts object-level masks. For query $i$ at point $p$, the logit is

$$z_{i,p} = \mathbf{F}_{\text{mask}(p)}^\top \mathbf{w}_i.$$

These logits are converted to assignments with a max operation across queries (Zhang et al., 30 Sep 2025).

The hierarchy is enforced by Targeted Attention Masking (TAM). From object predictions $\hat{y}_n \in \{0,\dots,M\}$, the model derives a binary attention mask

$S$0

where $S$1 if query $S$2 may attend to point $S$3, and $S$4 otherwise. Masked attention is then

$S$5

Foreground part queries for object $S$6 can therefore attend only to points inside object $S$7, while background queries attend outside that target region. The appendix generalizes this to a user-specified target set $S$8 and its union interior $S$9 (Zhang et al., 30 Sep 2025).

The second stage is an Instance–Part decoder. For selected object $$\mathbf{F}_{\text{mask}} \;=\; \mathbf{F}_{\text{pcd}} \;+\; \alpha \cdot \text{Conv}_{1\times1}^{2}\Big(\text{ReLU}\big(\text{Conv}_{1\times1}^{1}(\mathbf{F}_{\text{pcd}})\big)\Big),$$0, part queries

$$\mathbf{F}_{\text{mask}} \;=\; \mathbf{F}_{\text{pcd}} \;+\; \alpha \cdot \text{Conv}_{1\times1}^{2}\Big(\text{ReLU}\big(\text{Conv}_{1\times1}^{1}(\mathbf{F}_{\text{pcd}})\big)\Big),$$1

interact with object-restricted points $$\mathbf{F}_{\text{mask}} \;=\; \mathbf{F}_{\text{pcd}} \;+\; \alpha \cdot \text{Conv}_{1\times1}^{2}\Big(\text{ReLU}\big(\text{Conv}_{1\times1}^{1}(\mathbf{F}_{\text{pcd}})\big)\Big),$$2 through masked cross-attention, self-attention, and feed-forward blocks. When new clicks arrive, click queries are updated, TAM is recomputed from the current object prediction, and the part decoder reruns. The appendix states that this refinement occurs without re-encoding the whole scene, since scene features remain available (Zhang et al., 30 Sep 2025).

4. PartScan data synthesis and training regime

A major contribution of PinPoint3D is PartScan, a large-scale scene-level dataset with dense part annotations. The paper argues that progress in fine-grained interactive 3D segmentation has been blocked by the lack of scene-level point clouds with dense part labels. PartScan is constructed by combining realism from ScanNet and part fidelity from PartNet (Zhang et al., 30 Sep 2025).

The first component is pseudo-label generation on ScanNet. For decomposable object instances, the authors use PartField to compute continuous point-wise part features and then cluster those features into part segments. The clustering objective is

$$\mathbf{F}_{\text{mask}} \;=\; \mathbf{F}_{\text{pcd}} \;+\; \alpha \cdot \text{Conv}_{1\times1}^{2}\Big(\text{ReLU}\big(\text{Conv}_{1\times1}^{1}(\mathbf{F}_{\text{pcd}})\big)\Big),$$3

with the number of clusters chosen by silhouette score: $$\mathbf{F}_{\text{mask}} \;=\; \mathbf{F}_{\text{pcd}} \;+\; \alpha \cdot \text{Conv}_{1\times1}^{2}\Big(\text{ReLU}\big(\text{Conv}_{1\times1}^{1}(\mathbf{F}_{\text{pcd}})\big)\Big),$$4 This produces pseudo part labels on real scanned scenes (Zhang et al., 30 Sep 2025).

The second component inserts PartNet objects into ScanNet scenes. The paper selects 12 object categories: Table, Refrigerator, StorageFurniture, Chair, Dishwasher, Microwave, Bag, Mug, Bottle, Lamp, Vase, and Faucet. Selection is based on indoor frequency, scale compatibility with ScanNet, and semantically meaningful part structure. To align domains, the authors analyze object size, volume, and point density across the two datasets, scale each PartNet object to match ScanNet statistics for its category, downsample the point cloud, and apply farthest point sampling independently to each annotated part. Synthetic objects are then inserted by estimating the floor plane, partitioning it into a 2D grid, and randomly placing objects in unoccupied cells until space is exhausted or an object limit is reached (Zhang et al., 30 Sep 2025).

The model is trained with cross-entropy + Dice losses on both object- and part-level predictions. The appendix gives default coefficients $$\mathbf{F}_{\text{mask}} \;=\; \mathbf{F}_{\text{pcd}} \;+\; \alpha \cdot \text{Conv}_{1\times1}^{2}\Big(\text{ReLU}\big(\text{Conv}_{1\times1}^{1}(\mathbf{F}_{\text{pcd}})\big)\Big),$$5 and $$\mathbf{F}_{\text{mask}} \;=\; \mathbf{F}_{\text{pcd}} \;+\; \alpha \cdot \text{Conv}_{1\times1}^{2}\Big(\text{ReLU}\big(\text{Conv}_{1\times1}^{1}(\mathbf{F}_{\text{pcd}})\big)\Big),$$6, with auxiliary supervision from intermediate decoder layers and click-adaptive weighting near clicked regions. Optimization uses AdamW, initial learning rate $$\mathbf{F}_{\text{mask}} \;=\; \mathbf{F}_{\text{pcd}} \;+\; \alpha \cdot \text{Conv}_{1\times1}^{2}\Big(\text{ReLU}\big(\text{Conv}_{1\times1}^{1}(\mathbf{F}_{\text{pcd}})\big)\Big),$$7, weight decay $$\mathbf{F}_{\text{mask}} \;=\; \mathbf{F}_{\text{pcd}} \;+\; \alpha \cdot \text{Conv}_{1\times1}^{2}\Big(\text{ReLU}\big(\text{Conv}_{1\times1}^{1}(\mathbf{F}_{\text{pcd}})\big)\Big),$$8, gradient clipping 0.1, multi-step learning-rate decay after 1000 epochs, total training length 1100 epochs, validation every 50 epochs, batch size 5 for training and 1 for validation (Zhang et al., 30 Sep 2025).

Interactive training simulates 0–19 rounds per episode, with only the final round contributing gradients. Positive clicks are sampled from mis-segmented regions of the target part, negatives from background, and active parts are capped or sampled in the range 3–10 or up to 10, depending on the passage cited. The broad intention is to mimic iterative correction while controlling compute (Zhang et al., 30 Sep 2025).

5. Empirical performance, ablations, and user study

On the main PartScan all-part setting, PinPoint3D reports IoU1 55.8, IoU3 68.4, IoU5 71.3, NoC50 1.68, NoC65 3.46, NoC80 6.43, AP25 96.9, and AP50 85.7. Under the same setting, AGILE3D reports 39.1, 61.1, 66.7, 2.67, 5.18, 8.12, 96.7, and 78.2, while PointSAM reports 48.4, 52.6, 52.7, 74.1, and 51.0 for the metrics listed in the table. The clearest gains over AGILE3D are +16.7 points in IoU1, +7.3 in IoU3, +4.6 in IoU5, improvement in NoC50 from 2.67 to 1.68, and +7.5 in AP50 (Zhang et al., 30 Sep 2025).

On PartScan random-part, PinPoint3D reports IoU1 50.0, IoU3 65.9, IoU5 69.7, NoC50 2.12, NoC65 3.92, NoC80 6.92, AP25 95.1, and AP50 81.5. On MultiScan all-part, it reports IoU1 44.4, IoU3 62.7, IoU5 68.1, NoC50 2.28, NoC65 4.53, NoC80 7.66, AP25 93.9, and AP50 78.9; on MultiScan random-part, the corresponding numbers are 44.0, 60.8, 66.8, 2.71, 4.93, 7.74, 94.0, and 77.3. The paper notes that gains on MultiScan are modest relative to PartScan, attributing this to MultiScan’s relatively less fine-grained part annotations (Zhang et al., 30 Sep 2025).

PinPoint3D is also evaluated at the object level. On PartScan, it reports 86.7, 97.0, and 98.0 for IoU1, IoU3, and IoU5, compared with 83.64, 96.87, and 97.69 for AGILE3D. On MultiScan, however, AGILE3D remains higher, with 58.46, 75.04, and 81.02, while PinPoint3D reports 57.1, 72.3, and 78.6. This indicates that the architecture preserves or slightly improves coarse performance in-domain, but incurs a cross-domain object-level drop (Zhang et al., 30 Sep 2025).

The category-wise PartScan mIoU values are: Bag 61.8, Bottle 65.8, Chair 74.4, Dishwasher 59.1, Faucet 65.4, Lamp 63.1, Microwave 49.6, Mug 71.9, Refrigerator 62.3, StorageFurniture 72.2, Table 74.1, and Vase 66.5. The best categories are therefore Chair, Table, StorageFurniture, and Mug, while Microwave is the hardest category at 49.6 mIoU (Zhang et al., 30 Sep 2025).

The ablations isolate the hierarchical design. Removing the dedicated Part-Transformer drops performance from baseline IoU1 55.8, IoU3 68.4, IoU5 71.6, NoC50 1.70, NoC65 3.83, NoC80 7.10 to 43.7, 60.2, 65.0, 2.54, 5.46, 8.60. Replacing the frozen backbone plus adapter with direct backbone fine-tuning yields similar part metrics at first click but badly degrades object segmentation: A+F gives object IoU@1 86.7, IoU@3 97.0, AP25 90.6, AP50 89.9, while NA+UF gives 58.9, 92.2, 88.5, 85.6. Multi-object training is also better than single-object training, and combined PartScan training outperforms training on PartNet (in-scene) or ScanNet-PartField alone (Zhang et al., 30 Sep 2025).

The user study includes 6 non-expert participants, each given written instructions, a verbal demo, and a 15-minute practice session, then asked to annotate 20 scene point clouds containing 5–10 parts each. On PartScan, human performance is $$\mathbf{F}_{\text{mask}} \;=\; \mathbf{F}_{\text{pcd}} \;+\; \alpha \cdot \text{Conv}_{1\times1}^{2}\Big(\text{ReLU}\big(\text{Conv}_{1\times1}^{1}(\mathbf{F}_{\text{pcd}})\big)\Big),$$9 and $\mathrm{Conv}_{1\times1}^{1}$0, compared with simulated performance of 54.7 and 73.7. On MultiScan, humans achieve 64.5 and 73.7, compared with simulated 55.1 and 74.3. The paper interprets this as evidence that simulated interaction is reasonably realistic and that the interface is usable by non-experts (Zhang et al., 30 Sep 2025).

6. Distinct usages and adjacent research directions

Outside interactive segmentation, PinPoint3D also appears as the PinPoint3D T31022 (PP3D) in dosimetry. In that context it is a small-volume vented thimble ionization chamber with inner radius $\mathrm{Conv}_{1\times1}^{1}$1, outer radius $\mathrm{Conv}_{1\times1}^{1}$2, height $\mathrm{Conv}_{1\times1}^{1}$3, and sensitive volume $\mathrm{Conv}_{1\times1}^{1}$4. The chamber was studied under 20 MeV electron beams with 0.1 Gy to 9.3 Gy per pulse, and the paper concludes that thimble ionization chambers under ultra-high DPP exhibit strong volume recombination and a large polarity effect. For PP3D specifically, inclusion of the conductive guard ring in the complete 2D finite-element geometry is described as mandatory for accurate simulation; simplified geometry can overestimate CCE by up to 18% at high DPP (Paz-Martín et al., 16 Dec 2025).

Several neighboring computational works address superficially similar but technically distinct problems. TAPIP3D addresses long-term 3D point tracking in monocular RGB and RGB-D videos using a camera-stabilized spatio-temporal 3D feature cloud, Local Pair Attention, and world-space tracking (Zhang et al., 20 Apr 2025). “Commodifying Pointing in HRI” addresses 3D pointing gesture detection from RGB-D images by estimating a face-to-hand ray and intersecting it with a ground plane, operating at better than 30 Hz on commodity hardware (Azari et al., 2019). The work summarized under arXiv (Ma et al., 2024) is explicitly noted as not about “PinPoint3D” directly; instead it studies zero-shot task hallucination from a single RGB image through object discovery, partial 3D reconstruction, waypoint planning, and rendered task-execution videos, with relevance mainly to object-level 3D grounding and waypoint-based manipulation planning rather than point-level pinpointing (Ma et al., 2024).

Taken together, these distinctions delimit the specific meaning of PinPoint3D in computer vision. It is not a general label for 3D pointing, 3D point tracking, or geometry-aware manipulation planning. Rather, in its principal computational usage, it denotes a hierarchical interactive framework for scene-level fine-grained 3D part segmentation from sparse clicks, combining a frozen sparse-convolutional backbone, adapter-based feature specialization, scene-to-instance decoding, instance-to-part decoding, and Targeted Attention Masking to turn minimal 3D interaction into actionable part-level masks (Zhang et al., 30 Sep 2025).