PointVDP: View-Dependent 3D Segmentation

• PointVDP is a view-dependent 3D segmentation framework that learns adaptive 'fireworks rays' to map 3D point clouds to 2D images.
• It replaces fixed projection schemes with learnable distortion parameters, improving semantic segmentation and reducing computational redundancy.
• The method integrates lightweight ray parameter prediction, color regularization, and fusion of 2D-3D features to enhance segmentation performance.

Searching arXiv for the primary PointVDP paper and closely related uses of similar terminology. PointVDP is a projection-based 3D semantic segmentation framework that replaces view-independent, hand-designed projection rays with learned, view-dependent “fireworks rays” for 3D-to-2D mapping (Chen et al., 9 Jul 2025). It is designed for point cloud segmentation in settings where 2D backbones can be exploited efficiently, but conventional projection schemes rely on fixed ray formulas, require multiple projections per view plane, and produce semantically redundant images with large black regions (Chen et al., 9 Jul 2025). In PointVDP, the projection itself becomes a learned component: for each view plane, a lightweight module predicts view-specific distortion parameters from the 3D point distribution, a mutation mechanism diversifies the rays, and a color regularization term encourages high semantic occupancy in the projected image (Chen et al., 9 Jul 2025).

1. Problem formulation and departure from view-independent projection

PointVDP addresses 3D point cloud semantic segmentation by first projecting 3D points into 2D images, then using strong 2D backbones such as Swin and SG-Former together with a 3D encoder-decoder to segment points (Chen et al., 9 Jul 2025). Its starting point is the observation that projection-based works such as TAP, PointCLIP, MirageRoom, MVCNN, and RangeViT use view-independent projection, meaning that the mapping from 3D space to the image plane is defined by a fixed, human-chosen formula and reused across scenes and view planes (Chen et al., 9 Jul 2025).

For a $YZ$ view, the straight-line orthographic mapping is written as

$$h_n = \frac{z_n-\min(Z)}{\max(Z)-\min(Z)}H,\quad w_n = \frac{y_n-\min(Y)}{\max(Y)-\min(Y)}W,$$

while Mirage-style projection replaces $z_n$ by $z_n+\kappa x_n^2$, with $\kappa \in \{0,1,2\}$ fixed, in order to reduce occlusions (Chen et al., 9 Jul 2025). The limitations identified for such view-independent projection are fixed human-defined parameters, view-independence across different scene layouts, redundancy from multiple projections per view, poor 2D space usage, and the computational overhead induced by processing more images through the 2D backbone (Chen et al., 9 Jul 2025).

PointVDP is therefore motivated by a different objective: generate a single, highly informative projected image per view by learning projection rays from the actual 3D point distribution, and let those rays distort in both height and width directions rather than only along the vertical axis (Chen et al., 9 Jul 2025). A plausible implication is that the framework treats projection as an adaptive geometric optimization problem rather than as a fixed preprocessing step.

2. View-dependent projection and fireworks rays

The core projection in PointVDP is parameterized by two scalars for each view plane $m$, $\tilde{\kappa}_m^H$ and $\tilde{\kappa}_m^W$, which define nonlinear distortion in the vertical and horizontal directions (Chen et al., 9 Jul 2025). For a point $p_n=(x_n,y_n,z_n)$, the final mapping is

$$\begin{aligned} h_n &= \frac{z_n + \tilde{\kappa}_m^H x_n^2 - \min(Z)}{\max(Z)-\min(Z)}\,H, \ w_n &= \frac{y_n + \tilde{\kappa}_m^W x_n^2 - \min(Y)}{\max(Y)-\min(Y)}\,W. \end{aligned}$$

This construction makes the projection explicitly view-dependent and allows rays to bend upward or downward in height and left or right in width (Chen et al., 9 Jul 2025).

The paper describes these learned rays as “fireworks rays” because they can deviate in multiple directions rather than following a single upward-curved pattern (Chen et al., 9 Jul 2025). Toy examples include straight lines with $$h_n = \frac{z_n-\min(Z)}{\max(Z)-\min(Z)}H,\quad w_n = \frac{y_n-\min(Y)}{\max(Y)-\min(Y)}W,$$0, upward curves with $$h_n = \frac{z_n-\min(Z)}{\max(Z)-\min(Z)}H,\quad w_n = \frac{y_n-\min(Y)}{\max(Y)-\min(Y)}W,$$1, upward-leftward curves with $$h_n = \frac{z_n-\min(Z)}{\max(Z)-\min(Z)}H,\quad w_n = \frac{y_n-\min(Y)}{\max(Y)-\min(Y)}W,$$2, and downward-rightward curves with $$h_n = \frac{z_n-\min(Z)}{\max(Z)-\min(Z)}H,\quad w_n = \frac{y_n-\min(Y)}{\max(Y)-\min(Y)}W,$$3 (Chen et al., 9 Jul 2025). The quadratic term $$h_n = \frac{z_n-\min(Z)}{\max(Z)-\min(Z)}H,\quad w_n = \frac{y_n-\min(Y)}{\max(Y)-\min(Y)}W,$$4 increases displacement with depth, so points farther from the view plane are distorted more strongly.

The mutation stage is applied after deterministic prediction:

$$h_n = \frac{z_n-\min(Z)}{\max(Z)-\min(Z)}H,\quad w_n = \frac{y_n-\min(Y)}{\max(Y)-\min(Y)}W,$$5

with

$$h_n = \frac{z_n-\min(Z)}{\max(Z)-\min(Z)}H,\quad w_n = \frac{y_n-\min(Y)}{\max(Y)-\min(Y)}W,$$6

and experimental bounds $$h_n = \frac{z_n-\min(Z)}{\max(Z)-\min(Z)}H,\quad w_n = \frac{y_n-\min(Y)}{\max(Y)-\min(Y)}W,$$7, $$h_n = \frac{z_n-\min(Z)}{\max(Z)-\min(Z)}H,\quad w_n = \frac{y_n-\min(Y)}{\max(Y)-\min(Y)}W,$$8 (Chen et al., 9 Jul 2025). Boundary constraints are imposed so that deformed coordinates remain consistent with the image domain in the $$h_n = \frac{z_n-\min(Z)}{\max(Z)-\min(Z)}H,\quad w_n = \frac{y_n-\min(Y)}{\max(Y)-\min(Y)}W,$$9 and $z_n$0 planes (Chen et al., 9 Jul 2025).

The empirical rationale for fireworks rays is geometric coverage. In the toy analysis reported in the paper, straight-line projection yields approximately $z_n$1 2D space utilization, a basic upward curve reaches $z_n$2, and multi-directional curves reach up to $z_n$3 (Chen et al., 9 Jul 2025). This suggests that bidirectional distortion in both axes is not merely decorative; it is used to expose occluded objects and reduce empty image regions.

3. Prediction of ray parameters and color regularization

The view-dependent parameters are predicted by a lightweight VDP framework operating on each view plane (Chen et al., 9 Jul 2025). The pipeline begins with farthest point sampling to select $z_n$4 central points, followed by BallQuery within radius $z_n$5, and pooled local coordinates

$z_n$6

These pooled descriptors are partitioned into $z_n$7 and $z_n$8 components,

$z_n$9

so that the framework can learn geometry relevant to height and width separately (Chen et al., 9 Jul 2025).

The model then computes self-attention and cross-attention:

$z_n+\kappa x_n^2$0

and combines them with mixing weight $z_n+\kappa x_n^2$1:

$z_n+\kappa x_n^2$2

Finally, an MLP and pooling regress the ray parameters,

$z_n+\kappa x_n^2$3

where the MLP uses channels $z_n+\kappa x_n^2$4 with Sigmoid and pooling averages over $z_n+\kappa x_n^2$5 balls (Chen et al., 9 Jul 2025).

A second defining component is color regularization. PointVDP constructs a real-color image for the 2D backbone and a semantic-color image used during training, where black pixels correspond to no projected point (Chen et al., 9 Jul 2025). From the semantic-color image, it defines 2D space utilization

$z_n+\kappa x_n^2$6

with $z_n+\kappa x_n^2$7 controlling the awareness level (Chen et al., 9 Jul 2025).

This utilization term is then used as the standard deviation of a Gaussian regularizer on ray parameters:

$z_n+\kappa x_n^2$8

The total sparks loss is

$z_n+\kappa x_n^2$9

The stated effect is that low-utilization projections induce a stronger penalty near trivial straight rays, pushing the system toward more informative distortions, whereas high-utilization projections incur weaker pressure and can preserve the current rays (Chen et al., 9 Jul 2025).

4. Segmentation architecture and training pipeline

PointVDP uses the learned projection as the front end of a hybrid 2D-3D segmentation system (Chen et al., 9 Jul 2025). For each view plane $\kappa \in \{0,1,2\}$0, the model predicts $\kappa \in \{0,1,2\}$1, projects the points into a single $\kappa \in \{0,1,2\}$2 image, and processes that image with either Swin-T or SG-Former-S plus an FPN (Chen et al., 9 Jul 2025). In parallel, the raw 3D point cloud is processed by a 3D encoder-decoder identical to MirageRoom, with three encoder layers of channels $\kappa \in \{0,1,2\}$3 (Chen et al., 9 Jul 2025).

The 2D branch produces multi-scale feature maps, and the FPN aligns them with the 3D encoder channels (Chen et al., 9 Jul 2025). The two streams are fused by masked reprojection: per-pixel 2D features are reprojected back to 3D points using the known $\kappa \in \{0,1,2\}$4 mapping and fused with 3D point features before the final semantic prediction (Chen et al., 9 Jul 2025). This preserves the standard projection-based advantage of using strong 2D image models while avoiding the redundancy of multiple fixed-projection images.

Training is end-to-end. The semantic loss is standard cross-entropy,

$\kappa \in \{0,1,2\}$5

and the total objective is

$\kappa \in \{0,1,2\}$6

On S3DIS, the paper uses $\kappa \in \{0,1,2\}$7 view planes, $\kappa \in \{0,1,2\}$8, $\kappa \in \{0,1,2\}$9, $m$0, $m$1, and $m$2; on ScanNet it uses $m$3 and $m$4 (Chen et al., 9 Jul 2025). Optimization uses AdamW with initial learning rate $m$5, weight decay $m$6, and $m$7 epochs on a single RTX 3090 24G (Chen et al., 9 Jul 2025).

5. Empirical performance, efficiency, and ablations

On S3DIS Area 5 and ScanNet val, PointVDP reports consistent gains over MirageRoom under matched projection-based settings (Chen et al., 9 Jul 2025). The main quantitative results are as follows.

Dataset / backbone	Baseline	PointVDP
S3DIS, Swin-T	61.8 mIoU	67.8 mIoU
S3DIS, SG-Former-S	68.2 mIoU	71.0 mIoU
ScanNet val, Swin-T	60.4 mIoU	69.7 mIoU
ScanNet val, SG-Former-S	70.3 mIoU	72.1 mIoU

For S3DIS with Swin-T, MirageRoom obtains OA $m$8, mAcc $m$9, and mIoU $\tilde{\kappa}_m^H$0, whereas PointVDP reaches OA $\tilde{\kappa}_m^H$1, mAcc $\tilde{\kappa}_m^H$2, and mIoU $\tilde{\kappa}_m^H$3 (Chen et al., 9 Jul 2025). With SG-Former-S, MirageRoom$\tilde{\kappa}_m^H$4 reaches OA $\tilde{\kappa}_m^H$5, mAcc $\tilde{\kappa}_m^H$6, and mIoU $\tilde{\kappa}_m^H$7, while PointVDP$\tilde{\kappa}_m^H$8 reaches OA $\tilde{\kappa}_m^H$9, mAcc $\tilde{\kappa}_m^W$0, and mIoU $\tilde{\kappa}_m^W$1 (Chen et al., 9 Jul 2025). On ScanNet val, the corresponding gains are $\tilde{\kappa}_m^W$2 mIoU with Swin-T and $\tilde{\kappa}_m^W$3 mIoU with SG-Former-S (Chen et al., 9 Jul 2025).

Class-wise gains are substantial. On S3DIS with SG-Former-S, PointVDP$\tilde{\kappa}_m^W$4 has the best IoU in $\tilde{\kappa}_m^W$5 classes and improves over MirageRoom$\tilde{\kappa}_m^W$6 by $\tilde{\kappa}_m^W$7 on window, $\tilde{\kappa}_m^W$8 on door, $\tilde{\kappa}_m^W$9 on sofa, $p_n=(x_n,y_n,z_n)$0 on bookcase, and $p_n=(x_n,y_n,z_n)$1 on board (Chen et al., 9 Jul 2025). On ScanNet with Swin-T, improvements include bed $p_n=(x_n,y_n,z_n)$2, cabinet $p_n=(x_n,y_n,z_n)$3, table $p_n=(x_n,y_n,z_n)$4, window $p_n=(x_n,y_n,z_n)$5, bookshelf $p_n=(x_n,y_n,z_n)$6, fridge $p_n=(x_n,y_n,z_n)$7, and bath $p_n=(x_n,y_n,z_n)$8 (Chen et al., 9 Jul 2025).

Efficiency is a central result rather than a side effect. MirageRoom uses $p_n=(x_n,y_n,z_n)$9 images per scene on S3DIS, while PointVDP uses $$\begin{aligned} h_n &= \frac{z_n + \tilde{\kappa}_m^H x_n^2 - \min(Z)}{\max(Z)-\min(Z)}\,H, \ w_n &= \frac{y_n + \tilde{\kappa}_m^W x_n^2 - \min(Y)}{\max(Y)-\min(Y)}\,W. \end{aligned}$$0, a $$\begin{aligned} h_n &= \frac{z_n + \tilde{\kappa}_m^H x_n^2 - \min(Z)}{\max(Z)-\min(Z)}\,H, \ w_n &= \frac{y_n + \tilde{\kappa}_m^W x_n^2 - \min(Y)}{\max(Y)-\min(Y)}\,W. \end{aligned}$$1 reduction in 2D inputs (Chen et al., 9 Jul 2025). Training time decreases from $$\begin{aligned} h_n &= \frac{z_n + \tilde{\kappa}_m^H x_n^2 - \min(Z)}{\max(Z)-\min(Z)}\,H, \ w_n &= \frac{y_n + \tilde{\kappa}_m^W x_n^2 - \min(Y)}{\max(Y)-\min(Y)}\,W. \end{aligned}$$2 ms/sample to $$\begin{aligned} h_n &= \frac{z_n + \tilde{\kappa}_m^H x_n^2 - \min(Z)}{\max(Z)-\min(Z)}\,H, \ w_n &= \frac{y_n + \tilde{\kappa}_m^W x_n^2 - \min(Y)}{\max(Y)-\min(Y)}\,W. \end{aligned}$$3 ms/sample, and inference decreases from $$\begin{aligned} h_n &= \frac{z_n + \tilde{\kappa}_m^H x_n^2 - \min(Z)}{\max(Z)-\min(Z)}\,H, \ w_n &= \frac{y_n + \tilde{\kappa}_m^W x_n^2 - \min(Y)}{\max(Y)-\min(Y)}\,W. \end{aligned}$$4 ms/sample to $$\begin{aligned} h_n &= \frac{z_n + \tilde{\kappa}_m^H x_n^2 - \min(Z)}{\max(Z)-\min(Z)}\,H, \ w_n &= \frac{y_n + \tilde{\kappa}_m^W x_n^2 - \min(Y)}{\max(Y)-\min(Y)}\,W. \end{aligned}$$5 ms/sample (Chen et al., 9 Jul 2025). With SG-Former-S, the comparison is $$\begin{aligned} h_n &= \frac{z_n + \tilde{\kappa}_m^H x_n^2 - \min(Z)}{\max(Z)-\min(Z)}\,H, \ w_n &= \frac{y_n + \tilde{\kappa}_m^W x_n^2 - \min(Y)}{\max(Y)-\min(Y)}\,W. \end{aligned}$$6 ms/sample during training and $$\begin{aligned} h_n &= \frac{z_n + \tilde{\kappa}_m^H x_n^2 - \min(Z)}{\max(Z)-\min(Z)}\,H, \ w_n &= \frac{y_n + \tilde{\kappa}_m^W x_n^2 - \min(Y)}{\max(Y)-\min(Y)}\,W. \end{aligned}$$7 ms/sample during inference (Chen et al., 9 Jul 2025). Under a 24G GPU budget, batch size increases from $$\begin{aligned} h_n &= \frac{z_n + \tilde{\kappa}_m^H x_n^2 - \min(Z)}{\max(Z)-\min(Z)}\,H, \ w_n &= \frac{y_n + \tilde{\kappa}_m^W x_n^2 - \min(Y)}{\max(Y)-\min(Y)}\,W. \end{aligned}$$8 to $$\begin{aligned} h_n &= \frac{z_n + \tilde{\kappa}_m^H x_n^2 - \min(Z)}{\max(Z)-\min(Z)}\,H, \ w_n &= \frac{y_n + \tilde{\kappa}_m^W x_n^2 - \min(Y)}{\max(Y)-\min(Y)}\,W. \end{aligned}$$9 relative to MirageRoom, and with SG-Former-S from $$h_n = \frac{z_n-\min(Z)}{\max(Z)-\min(Z)}H,\quad w_n = \frac{y_n-\min(Y)}{\max(Y)-\min(Y)}W,$$00 to $$h_n = \frac{z_n-\min(Z)}{\max(Z)-\min(Z)}H,\quad w_n = \frac{y_n-\min(Y)}{\max(Y)-\min(Y)}W,$$01 (Chen et al., 9 Jul 2025).

Ablations isolate the contributions of the fireworks mutation and color regularization. On S3DIS with SG-Former-S, vanilla VDP yields mAcc $$h_n = \frac{z_n-\min(Z)}{\max(Z)-\min(Z)}H,\quad w_n = \frac{y_n-\min(Y)}{\max(Y)-\min(Y)}W,$$02 and mIoU $$h_n = \frac{z_n-\min(Z)}{\max(Z)-\min(Z)}H,\quad w_n = \frac{y_n-\min(Y)}{\max(Y)-\min(Y)}W,$$03; adding mutation yields $$h_n = \frac{z_n-\min(Z)}{\max(Z)-\min(Z)}H,\quad w_n = \frac{y_n-\min(Y)}{\max(Y)-\min(Y)}W,$$04 and $$h_n = \frac{z_n-\min(Z)}{\max(Z)-\min(Z)}H,\quad w_n = \frac{y_n-\min(Y)}{\max(Y)-\min(Y)}W,$$05; adding color regularization yields $$h_n = \frac{z_n-\min(Z)}{\max(Z)-\min(Z)}H,\quad w_n = \frac{y_n-\min(Y)}{\max(Y)-\min(Y)}W,$$06 and $$h_n = \frac{z_n-\min(Z)}{\max(Z)-\min(Z)}H,\quad w_n = \frac{y_n-\min(Y)}{\max(Y)-\min(Y)}W,$$07 (Chen et al., 9 Jul 2025). In the comparison between pre-defined and data-driven rays, the best VIP single up-curve reaches mIoU $$h_n = \frac{z_n-\min(Z)}{\max(Z)-\min(Z)}H,\quad w_n = \frac{y_n-\min(Y)}{\max(Y)-\min(Y)}W,$$08, whereas full fireworks reach up to $$h_n = \frac{z_n-\min(Z)}{\max(Z)-\min(Z)}H,\quad w_n = \frac{y_n-\min(Y)}{\max(Y)-\min(Y)}W,$$09 (Chen et al., 9 Jul 2025). On ScanNet, varying the awareness exponent shows $$h_n = \frac{z_n-\min(Z)}{\max(Z)-\min(Z)}H,\quad w_n = \frac{y_n-\min(Y)}{\max(Y)-\min(Y)}W,$$10 is best at $$h_n = \frac{z_n-\min(Z)}{\max(Z)-\min(Z)}H,\quad w_n = \frac{y_n-\min(Y)}{\max(Y)-\min(Y)}W,$$11 mIoU, outperforming $$h_n = \frac{z_n-\min(Z)}{\max(Z)-\min(Z)}H,\quad w_n = \frac{y_n-\min(Y)}{\max(Y)-\min(Y)}W,$$12 at $$h_n = \frac{z_n-\min(Z)}{\max(Z)-\min(Z)}H,\quad w_n = \frac{y_n-\min(Y)}{\max(Y)-\min(Y)}W,$$13 and $$h_n = \frac{z_n-\min(Z)}{\max(Z)-\min(Z)}H,\quad w_n = \frac{y_n-\min(Y)}{\max(Y)-\min(Y)}W,$$14 at $$h_n = \frac{z_n-\min(Z)}{\max(Z)-\min(Z)}H,\quad w_n = \frac{y_n-\min(Y)}{\max(Y)-\min(Y)}W,$$15 (Chen et al., 9 Jul 2025).

6. Scope, terminology, and limitations

In the literature represented here, “PointVDP” specifically denotes the view-dependent projection framework of Yang et al., not a generic point-cloud diffusion model or a generic decoder module (Chen et al., 9 Jul 2025). This distinction matters because adjacent papers use related abbreviations or “PointVDP-style” language for materially different mechanisms.

Several nearby concepts are distinct. The “Trainable Pointwise Decoder Module” is a point-level decoder for range-image segmentation, and the paper explicitly states that it does not use the name “PointVDP” (Chen et al., 2024). “PointDP” is a diffusion-driven purification defense against adversarial attacks on 3D point cloud recognition (Sun et al., 2022). “Sparse Point-Voxel Diffusion” belongs to the point-voxel diffusion line for generative modeling and is discussed as “PointVDP / PVD-style” in the accompanying explanation, but its task is 3D shape generation rather than semantic segmentation (Romanelis et al., 2024). The “Voxel Diffusion Module” for 3D object detection is also described as matching “PointVDP-style ideas” only in the sense of voxel diffusion or propagation before serialized backbones (Liu et al., 22 Aug 2025). A common misconception is therefore to read PointVDP as a diffusion model; in the named paper, it is a learned projection mechanism for segmentation.

The limitations stated for PointVDP are equally specific (Chen et al., 9 Jul 2025). The method is evaluated on indoor datasets, with one pair of ray parameters per view plane rather than finer per-object or per-region control. Its distortion model is quadratic in $$h_n = \frac{z_n-\min(Z)}{\max(Z)-\min(Z)}H,\quad w_n = \frac{y_n-\min(Y)}{\max(Y)-\min(Y)}W,$$16, which constrains expressivity, and—as with projection-based methods generally—segmentation quality remains dependent on whether the selected views expose the relevant surfaces (Chen et al., 9 Jul 2025). The paper proposes several directions: per-object or per-region VDP, more expressive ray parameterizations, integration with more advanced 2D and 3D backbones, real-time optimization, and extension to instance segmentation, panoptic segmentation, detection, and outdoor datasets such as SemanticKITTI and nuScenes (Chen et al., 9 Jul 2025).

PointVDP’s significance lies in shifting the design emphasis from choosing a projection formula to learning the projection itself (Chen et al., 9 Jul 2025). Within projection-based 3D segmentation, it operationalizes a precise claim: view-dependent, data-driven rays combined with a utilization-aware regularizer can replace redundant multi-projection heuristics with a smaller set of more informative images, while improving semantic segmentation accuracy and reducing compute.