Canonical Facial Point Prediction

• Canonical Facial Point Prediction defines techniques to estimate semantically consistent facial landmarks by mapping pixels to a shared 3D reference space.
• It encompasses paradigms such as sparse regression, heatmap-based localization, and dense embedding approaches, each applied to face alignment, tracking, and reconstruction.
• Evaluation using metrics like NME, RMSE, and AUC demonstrates improved accuracy and robustness in landmark localization and dense correspondence.

Searching arXiv for recent and foundational papers on canonical facial point prediction, dense canonical embeddings, and facial landmark localization. Canonical facial point prediction denotes the estimation of semantically consistent facial points or coordinates across images, videos, and 3D surfaces. In its classical form, it predicts indexed landmarks such as eye corners, nose tip, mouth corners, and jawline points; in newer formulations, it predicts a dense canonical coordinate for every pixel or sampled 3D point, so that correspondence is defined in a shared reference space rather than by frame-to-frame motion alone. This broader notion supports face alignment, tracking, reconstruction, animation, anthropometry, registration, and geometry-aware matching, and it spans protocols from sparse 5-, 15-, 21-, 29-, 68-, 106-, and 194-point schemes to dense per-pixel canonical maps and canonical radiance-field coordinates (Wang et al., 2014, Kocasari et al., 21 Apr 2026, Pozdeev et al., 4 Nov 2025).

1. Canonicality, semantic consistency, and landmark protocols

In the landmark-centric tradition, “canonical” means that landmark index $i$ carries the same semantic meaning across subjects and views. The survey literature defines canonical facial points as semantically consistent, anatomically meaningful points on facial components such as eyes, eyebrows, nose, mouth, and jawline, and lists commonly used protocols including 5-point, 21-point, 29-point, 39/49/58/66/68-point, 76/98/130-point, and 194-point schemes (Wang et al., 2014). This convention underlies many benchmarks and applications because it converts facial analysis into a structured correspondence problem.

Several works make this canonicality explicit by refining the landmark vocabulary itself. The JD-landmark benchmark was introduced to move beyond 68-point mark-ups, which were described as incompetent to depict the structure of facial components; its 106-point schema adds structural coverage such as the lower boundary of the eyebrow and the wing of the nose, and it is evaluated as a 2D coordinate prediction problem under large variations of pose, expression, and occlusion (Liu et al., 2019). PAL-Net adopts a different notion of canonicality grounded in clinical anthropometry: it predicts 50 standardized soft-tissue anatomical landmarks on 3D stereo-photogrammetry meshes, with placement following Ferrario (2003) and Sforza (2012), and with intra-observer variability measured on 20 subjects twice over two weeks (Yazdi et al., 1 Oct 2025).

A second, denser meaning of canonicality appears in recent correspondence and reconstruction systems. In "Face Anything" (Kocasari et al., 21 Apr 2026), canonical facial point prediction assigns each image pixel a 3D canonical coordinate in FLAME’s coordinate system, where the origin is located at the center of the face. In "DenseMarks" (Pozdeev et al., 4 Nov 2025), every pixel in a 2D head image is mapped to a 3D coordinate in a canonical unit cube, with the aim that pixels depicting the same semantic facial or head part across frames, poses, and identities map to the same cube location. This suggests that canonical facial point prediction is no longer restricted to sparse fiducials; it also denotes dense category-level canonicalization.

2. Major representational paradigms

A first paradigm is sparse direct landmark regression. NaimishNet uses the 15 canonical facial keypoints of the Kaggle Facial Keypoints Detection task and trains 15 separate regressors, one per keypoint, each outputting a 2D vector from a $96\times96$ grayscale face image (Agarwal et al., 2017). A later efficiency-oriented study retains the same 15-point target set but predicts all 30 coordinates jointly, comparing MobileNetV2, NASNetMobile, and custom CNNs with an emphasis on model size, parameters, and inference time (Dileep et al., 2022). These systems canonicalize through fixed index semantics rather than through an explicit shared 3D space.

A second paradigm is heatmap-based dense localization for sparse canonical landmarks. The 106-point challenge reports top systems based on stacked hourglass models, DU-Net plus Refine-Net with integral regression, and hierarchical modules with dual soft-argmax, all of which predict 2D landmark locations under a fixed 106-point make-up (Liu et al., 2019). ArtFacePoints follows the 68-point 300-W definition for artworks and combines a global ResNet-based encoder–decoder at $256\times256$ with regional refinement networks for the eyes, nose, and mouth, using spatial softargmax to convert heatmaps to coordinates (Sindel et al., 2022). PAL-Net instead operates directly on 3D local patches from registered meshes, uses point-wise convolutions plus attention, and regresses 50 anatomical landmark coordinates while preserving inter-landmark geometry (Yazdi et al., 1 Oct 2025).

A third paradigm is dense canonical coordinate prediction. In "Face Anything" (Kocasari et al., 21 Apr 2026), the model jointly predicts depth, ray maps, and canonical facial maps so that each pixel $u$ at time $t$ is associated with a canonical point $C_t(u)\in\mathbb{R}^3$. Correspondence estimation then reduces to nearest-neighbor matching in canonical space instead of explicit motion estimation. In "DenseMarks" (Pozdeev et al., 4 Nov 2025), the canonical output is a 3D embedding $e(x)\in[0,1]^3$ for each pixel, which becomes a semantic descriptor only after indexing a smoothed latent cube feature field by trilinear interpolation. The canonical space bottleneck is central: semantics are compared only through cube-indexed descriptors, which regularizes the coordinate field and makes the cube queryable for ears, forehead, hair, and other regions.

A fourth paradigm maps observed 3D points into a canonical deformation space prior to rendering or reconstruction. PartNerFace defines a FLAME-based canonical space, computes a coarse inverse-skinning mapping for each sampled 3D point in a monocular video frame, and refines that point with a part-based deformation field before querying a canonical radiance field (Yu et al., 15 Apr 2026). Here canonical facial point prediction is embedded in a rendering pipeline rather than a stand-alone landmark detector.

3. Mathematical formulations

Dense canonical facial point prediction is often expressed as joint dense prediction. "Face Anything" formulates the problem as

$(D_i,R_i,C_i)=f_\theta(I_1,\dots,I_N),$

where $D_i$ is depth, $R_i$ is a ray map, and $96\times96$0 is the canonical facial map for image $96\times96$1 (Kocasari et al., 21 Apr 2026). Dense correspondences between images $96\times96$2 and $96\times96$3 are then computed by nearest-neighbor search in canonical space:

$96\times96$4

This replaces explicit motion fields with a canonical matching rule. The same work also defines regression, confidence-weighted regression, and gradient penalties for $96\times96$5, with $96\times96$6, $96\times96$7, and $96\times96$8 (Kocasari et al., 21 Apr 2026).

"DenseMarks" formulates canonicalization as a pixel-to-cube mapping

$96\times96$9

with coordinate $256\times256$0 and descriptor

$256\times256$1

The method combines contrastive supervision on tracked correspondences, landmark anchoring to fixed cube coordinates, and segmentation supervision, with $256\times256$2 and $256\times256$3 (Pozdeev et al., 4 Nov 2025). A notable design choice is that smoothness is not imposed as an explicit loss; instead, it is enforced by Gaussian filtering of the latent cube feature grid.

Sparse landmark systems typically use coordinate regression or heatmap-to-coordinate conversion. PAL-Net optimizes a composite loss

$256\times256$4

with $256\times256$5 and $256\times256$6, explicitly coupling pointwise localization to inter-landmark distance preservation for clinical use (Yazdi et al., 1 Oct 2025). ArtFacePoints instead supervises softargmax-derived coordinates from global and regional heatmaps, using a weighted sum of global 68-point and regional losses with $256\times256$7 (Sindel et al., 2022).

Evaluation metrics are likewise protocol-dependent. The 106-point challenge defines per-image normalized mean error as

$256\times256$8

and ranks methods by AUC of the cumulative error distribution up to $256\times256$9, with failure rate defined by $u$0 (Liu et al., 2019). Other works report RMSE, AbsRel, EPE, MAE/RMSE against reconstructed meshes, pixel mean Euclidean error, or clinically relevant distance- and angle-wise errors.

4. Supervision, data construction, and training regimes

The supervision used for canonical facial point prediction varies sharply with the representation. For dense 4D facial reconstruction, "Face Anything" derives ground-truth depth and canonical maps from NeRSemble, a 16-camera synchronized and calibrated video dataset. Multi-view geometry is reconstructed with COLMAP at downsample$u$1, FLAME-based tracking is run per timestamp, shape and static offsets are enforced consistent across time with outlier removal, and per-vertex deformation between tracked and canonical FLAME meshes is transferred to COLMAP points via nearest FLAME surface points, yielding rendered per-pixel canonical coordinates in FLAME space (Kocasari et al., 21 Apr 2026). Training proceeds in two stages: DAViD pretraining for facial priors via monocular depth, followed by joint prediction of depth, ray maps, and canonical maps. The model has 1.2B parameters and uses multi-resolution training, gradient checkpointing, bfloat16, gradient clipping, AdamW, cosine decay from $u$2 to $u$3, 90 epochs, and alternating sampling between multi-view images at one timestamp and single-camera images across multiple timestamps (Kocasari et al., 21 Apr 2026).

Dense canonical embeddings can instead be learned from point tracks. DenseMarks constructs its supervision from CelebV-HQ, starting with 35K in-the-wild talking-head videos. GroundedSAM2, prompted with “person,” provides a foreground mask in the first frame; points are sampled in the foreground and tracked with CoTracker3; videos with fewer than 80 valid tracks or failed segmentation are discarded, leaving 32K videos (Pozdeev et al., 4 Nov 2025). The method augments these tracked correspondences with 70 manually selected landmarks extracted via MediaPipe and parsing masks obtained via FaRL and refined at boundaries using face-parsing models. The backbone is a Vision Transformer initialized from DINOv3, followed by a DPT-style upsampling head to $u$4, and a learnable latent cube feature field that is smoothed by a 3D Gaussian filter (Pozdeev et al., 4 Nov 2025).

Sparse 3D anatomical landmark prediction relies on explicit annotation and preprocessing. PAL-Net is trained on 214 annotated stereo-photogrammetry scans from healthy adults, uses rigid global registration via FPFH features plus RANSAC, local rigid ICP, atlas-based seed placement by mean landmarks, and patch extraction around each expected landmark location (Yazdi et al., 1 Oct 2025). The best patch setting uses $u$5 nearest points per landmark, ordered by Euclidean distance to a fixed origin near the nasal region. Training uses Adam, an initial learning rate of $u$6, batch size 16, ReduceLROnPlateau with 8-epoch patience, and early stopping with 30-epoch patience (Yazdi et al., 1 Oct 2025).

Cross-domain canonical facial point prediction requires domain-specific augmentation. ArtFacePoints builds a high-resolution artwork dataset with real paintings and prints plus AdaIN- and CycleGAN-generated stylizations and geometric landmark-group warps using thin-plate splines, while YOLOv4 is used for face detection and high-resolution cropping (Sindel et al., 2022). In lower-resolution 2D landmark regression, NaimishNet handles missing labels by training 15 separate models and uses horizontal flipping with left-right keypoint swapping (Agarwal et al., 2017), whereas the later efficiency study explicitly compares augmentation against forward-fill and KNN label imputation and reports augmentation as consistently superior in RMSE without affecting inference time (Dileep et al., 2022).

5. Tasks, benchmarks, and empirical behavior

The practical outputs of canonical facial point prediction range from alignment and tracking to dense 4D reconstruction. "Face Anything" reports state-of-the-art depth accuracy on NeRSemble and Ava-256, with NeRSemble RMSE/AbsRel of $u$7 for image input and $u$8 for video input, and Ava-256 RMSE/AbsRel of $u$9 and $t$0 (Kocasari et al., 21 Apr 2026). On 3D correspondence, it achieves EPE $t$1 versus $t$2 for V-DPM over margins 1–10, which the paper summarizes as approximately $t$3 lower correspondence error; on FLAME tracking it reports CD-L1 $t$4 versus $t$5; and on efficiency it reports 5 s runtime, 19 GB peak memory, and 470 max images/GPU versus 160 s, 40 GB, and 74 images for V-DPM (Kocasari et al., 21 Apr 2026). KD-Tree nearest-neighbor matching in canonical space typically takes less than 0.2 s per image pair on CPU.

DenseMarks evaluates canonical embeddings as geometry-aware point matching, cross-person consistency, 3DMM tracking, and stereo reconstruction. On Nersemble same-person matching it reports MAE $t$6 and RMSE $t$7, outperforming DINOv3, Fit3D, Hyperfeatures, and Sapiens; for cross-person consistency it reports ArcFace $t$8 and Met3R $t$9, both best in the comparison; and its ablation study shows MAE $C_t(u)\in\mathbb{R}^3$0 without canonical space, $C_t(u)\in\mathbb{R}^3$1 without pretrain, and $C_t(u)\in\mathbb{R}^3$2 for the full model (Pozdeev et al., 4 Nov 2025). The reported qualitative behavior emphasizes robust localization of hair, ear centers, forehead, and eyebrow corners using only 3D embeddings.

Clinical and structured 3D landmarking are evaluated differently. PAL-Net reports on LAFAS a mean point-wise error of $C_t(u)\in\mathbb{R}^3$3 mm and a mean distance-wise error of $C_t(u)\in\mathbb{R}^3$4 mm for 50 anatomical landmarks, with low errors in well-defined midline regions such as Subnasale at $C_t(u)\in\mathbb{R}^3$5 mm and Stomion at $C_t(u)\in\mathbb{R}^3$6 mm (Yazdi et al., 1 Oct 2025). On FaceScape it reports $C_t(u)\in\mathbb{R}^3$7 mm point-wise and $C_t(u)\in\mathbb{R}^3$8 mm distance-wise error, along with a total runtime of 0.304 s per subject and a memory footprint of approximately 2.5 GiB (Yazdi et al., 1 Oct 2025). These results are framed relative to intra-observer variability and to competing methods such as MVLM and 2S-SGCN.

In 2D benchmark settings, the 106-point challenge ranks methods by AUC up to 8% NME and reports the top three test results as follows: Baidu VIS achieves AUC 84.01%, FR 0.10%, and NME 1.31%; USTC achieves AUC 82.68%, FR 0.05%, and NME 1.41%; Meituan achieves AUC 82.22%, FR 0.00%, and NME 1.42% (Liu et al., 2019). For high-resolution artworks, ArtFacePoints reports on prints a mean Euclidean error of $C_t(u)\in\mathbb{R}^3$9 pixels over all 68 landmarks and $e(x)\in[0,1]^3$0 pixels over the inner 51 landmarks for the full model, and shows that coarse-to-fine regional refinement substantially improves inner landmarks relative to the global-only network (Sindel et al., 2022). On the Kaggle 15-point task, the efficiency study reports that fine-tuned MobileNetV2 yields the lowest RMSE and inference time, with approximately 0.82–0.88 s per 100 images on CPU, corresponding to roughly 8.2–8.9 ms per image (Dileep et al., 2022).

6. Limitations, misconceptions, and broader research context

A common misconception is that canonical facial point prediction is synonymous with sparse landmark detection. The literature does not support that restriction. Survey work organizes facial feature point detection into CLM-based, AAM-based, regression-based, and deep learning-based methods (Wang et al., 2014), but more recent systems extend the notion of a canonical facial point to per-pixel 3D coordinates in FLAME space, 3D embeddings in a unit cube, or canonicalized points along rays in a radiance field (Kocasari et al., 21 Apr 2026, Pozdeev et al., 4 Nov 2025, Yu et al., 15 Apr 2026). A plausible implication is that canonicality is best viewed as a correspondence structure rather than as a fixed annotation cardinality.

The dominant failure modes also differ by formulation. "Face Anything" is specialized to faces, does not canonicalize nearby non-facial objects such as microphones, hands, or accessories, and may degrade under strong occlusions, extreme viewpoints, or limited visibility (Kocasari et al., 21 Apr 2026). DenseMarks notes that very few valid tracks under training-domain shift can weaken cube consistency and that ambiguous hairstyles or accessories may map to nearby cube regions (Pozdeev et al., 4 Nov 2025). PAL-Net reports larger errors in ear and hairline regions because stereo-photogrammetry often yields sparse or noisy geometry there, and it is sensitive to rigid registration accuracy and atlas bias (Yazdi et al., 1 Oct 2025). ArtFacePoints identifies prints, jawline prediction, extreme poses, and severe occlusions as especially difficult (Sindel et al., 2022).

Another misconception is that dense temporal correspondence necessarily requires explicit motion modeling. "Face Anything" directly contests this by transforming dense tracking and dynamic reconstruction into a canonical reconstruction problem, in which correspondence is recovered by nearest-neighbor matching in canonical space rather than by predicted motion fields (Kocasari et al., 21 Apr 2026). DenseMarks advances a related argument through the canonical space bottleneck, where semantics are only compared via cube-indexed descriptors (Pozdeev et al., 4 Nov 2025). Taken together, these works suggest a broader shift from appearance-tied or frame-tied prediction toward identity-consistent coordinate systems that can support tracking, dense warping, 3DMM fitting, and 4D reconstruction within a single representational framework.

Future directions are already visible in the surveyed material. The survey highlights 3D-aware canonicalization, temporal models, semi- or weakly supervised learning, domain adaptation, and uncertainty estimation as important trajectories (Wang et al., 2014). Recent dense models indicate how such directions may materialize: through multi-view geometry supervision, point-track supervision in the wild, queryable canonical spaces that include hair and ears, and hybrid systems that combine parametric priors such as FLAME with learned dense correspondence fields (Kocasari et al., 21 Apr 2026, Pozdeev et al., 4 Nov 2025, Yu et al., 15 Apr 2026).