Projected Normalized Coordinate Code

• PNCC is a representation that encodes normalized 3D coordinates into three image channels, enabling straightforward use of 2D convolutional architectures.
• It supports tasks such as single-view 3D super-resolution and dense face alignment by converting 3D mesh or depth map data into a standardized, reversible tensor format.
• PNCC enhances geometric fidelity and execution speed by providing efficient, direct mapping of spatial information compared to traditional RGB or depth representations.

The Projected Normalized Coordinate Code (PNCC) is a structured three-channel representation that encodes normalized 3D coordinates—typically X, Y, and Z—within a pixelwise image format. PNCC is designed to transfer geometric information (from 3D mesh vertices or camera‐space depth maps) into a 2D tensor, supporting tasks such as single-view 3D super-resolution and dense 3D face alignment. By embedding world or canonical 3D coordinates into image channels, PNCC enables direct use of established 2D convolutional architectures and brings geometric fidelity and reversibility to image-based pipelines. PNCC is central in contemporary approaches to unifying geometric and appearance-based analysis in both computer vision and 3D reconstruction.

1. Mathematical Formulation and Canonical Construction

PNCC converts pointwise or mesh-based 3D coordinates into a three-channel image. In the single-view depth scenario (Mas et al., 11 Nov 2025), given a depth map $d(u,v)$ and camera intrinsics $(f_x, f_y, c_x, c_y)$, for each valid pixel $(u, v)$, compute the camera-coordinate 3D point: $$\begin{aligned} X &= (u - c_x) \cdot d(u, v) / f_x \ Y &= (v - c_y) \cdot d(u, v) / f_y \ Z &= d(u, v) \end{aligned}$$ Each coordinate is normalized by a scalar $s$ (often the maximum scene depth in training), yielding: $$\begin{aligned} u_{\mathrm{norm}}(u, v) &= X / s = [(u - c_x) \cdot d(u, v)] / (f_x s) \ v_{\mathrm{norm}}(u, v) &= Y / s = [(v - c_y) \cdot d(u, v)] / (f_y s) \ w_{\mathrm{norm}}(u, v) &= Z / s = d(u, v) / s \end{aligned}$$ Thus, the PNCC image is defined as $$PNCC(u, v) = [u_{\mathrm{norm}}, v_{\mathrm{norm}}, w_{\mathrm{norm}}]$$, mapping visible surface geometry as regular H×W×3 tensors.

In dense 3D face alignment (Zhu et al., 2018), PNCC is constructed by assigning normalized canonical 3D coordinates to mesh vertices: $$\mathrm{NCC}_d(i) = [\bar S_d(i) - \min_j \bar S_d(j)] / [\max_j \bar S_d(j) - \min_j \bar S_d(j)]$$ Stacked vectors over axes ($x, y, z$) yield NCC, which are rendered into a 2D image via barycentric interpolation in a Z-buffer pipeline, so the resulting PNCC channel for pixel $(u, v)$ encodes the interpolated canonical coordinates of the visible mesh triangle.

2. Pipeline and Implementation Workflows

The PNCC construction pipeline (Mas et al., 11 Nov 2025) consists of:

1. Masking valid pixels ($d(u,v) > 0$) in the input depth map.
2. For each valid pixel, compute X, Y, Z as above and fill the PNCC channels (R = X, G = Y, B = Z), each normalized by $s$.
3. Fill missing (invalid) pixels using nearest-neighbor for convolutional stability.
4. The resulting PNCC image (low-resolution) is upsampled using a 2D super-resolution (SR) network (e.g., Swin Transformer or Vision Mamba).
5. The high-resolution PNCC is decoded back to a depth map or a set of 3D camera coordinates by inverting the normalization.

In 3DDFA’s cascaded CNN framework (Zhu et al., 2018), PNCC is recalculated at each refinement iteration:

1. Given PCA model parameters $\mathbf p^k$, compute the current 3D mesh.
2. Project to image space using orthographic projection.
3. Z-buffer render the canonical mesh, encoding normalized coordinates as pixel colors.
4. Concatenate PNCC with RGB input as the feature map.
5. Predict parameter updates $\Delta \mathbf p^k$ with the regressor CNN, and repeat.

3. Channel Semantics and Occlusion Handling

The standard channel assignments are:

• Channel 0 (R): normalized X (horizontal camera coordinate or canonical mesh X)
• Channel 1 (G): normalized Y (vertical camera coordinate or canonical mesh Y)
• Channel 2 (B): normalized Z (camera depth or canonical mesh Z)

PNCC only covers visible surfaces from the active viewpoint, so occluded or undefined regions yield invalid pixels (zero or null values). During training, such pixels are excluded from loss computations. For convolutional stability, missing regions within the PNCC tensor are filled by spatial nearest-neighbor propagation—these placeholders do not contribute to supervision. In mesh-based PNCC (face alignment), visibility is enforced via Z-buffering; only the front-most mesh is rendered per pixel.

4. Comparative Advantages over Alternative Representations

PNCC outperforms conventional representations in several dimensions (Mas et al., 11 Nov 2025, Zhu et al., 2018):

• Against point-based upsampling (e.g., PU-Net, Grad-PU): PNCC supports a one-shot, raster-based, feed-forward pipeline, eschewing iterative optimization and graph convolutions. Inference is dramatically faster (for example, 0.16s vs. 9.9s for NYUv2 ×4 scale) due to full exploitation of 2D hardware acceleration.
• Against RGB-guided depth SR: It avoids reliance on RGB/depth pixel-level registration, high-res color frames, or color-depth artifacts. Entirely geometric, PNCC eliminates texture-bleeding and maintains simpler input pipelines.
• Against pure depth upscaling: Raw depth does not encode spatial lateral context (X/Y); PNCC supplies full camera space, boosting boundary sharpness and 3D scene consistency.
• Experimental metrics: PNCC-based models achieve competitive or superior root mean square error (RMSE), run up to 18× faster, and utilize 3–5× fewer parameters than RGB-guided baselines like SGNet.

5. Key Hyperparameters, Losses, and Network Variants

Critical hyperparameter choices include:

• The scale factor $s$, set to the maximum depth or coordinate norm in the training dataset to ensure normalized range fits [0,1] or [0,255].
• Loss functions are usually pixelwise Charbonnier across all PNCC channels (or restricted to Z for depth-only variants).
• Two output variants: XYZ-predict (all three channels regressed), or Z-only (SR network only predicts depth, with X/Y recomputed from geometry). Z-only variants may yield slightly improved RMSE.
• Network input resolutions are matched exactly to downsampled depth dimensions, with no padding tricks.
• Invalid pixels (regions) are masked entirely during loss evaluation.
• In practice, SR networks used include SwinIR (“SwinT-PNCC”) for maximal accuracy and Vision Mamba (“VM-PNCC”) for real-time deployment.

6. The Role of PNCC in Model Feedback, Supervision, and Pose Robustness

PNCC delivers dense, geometric feedback throughout iterative model pipelines (Zhu et al., 2018). In cascaded fitting (3DDFA), the PNCC image changes as model parameters are refined—PNCC offers explicit, continuous correspondence between the current mesh geometry and observed appearance. This mechanism supports supervision at each iteration by directly comparing PNCC to RGB, accelerating convergence and suppressing pose-induced ambiguities. Because PNCC always encodes the same underlying 3D mean shape, it enables learning pose-dependent corrections and generalizes across large viewpoint changes, including extreme yaw angles (up to 90°).

7. Visualization and Interpretability

Visual analysis of PNCC images reveals clear coordinate gradients aligned with scene or facial structure (Mas et al., 11 Nov 2025). Features such as contours and edges manifest as smooth transitions in channel intensity, facilitating interpretable geometry supervision. Comparative figures demonstrate that upsampled PNCC restores sharper and more consistent surface geometry (compared to bicubic or RGB-guided SR), and aligns anatomical regions with their true spatial context.

---

In sum, the Projected Normalized Coordinate Code provides a reversible, geometric representation that bridges 3D spatial structure and efficient, convolution-friendly 2D processing pipelines. PNCC’s applicability to real-time 3D super-resolution and robust face alignment illustrates its role as a central mechanism for transferring geometric information into modern vision architectures.