Face2PPG Pipeline Overview

• The Face2PPG pipeline is a modular system that transforms facial video into rPPG/BVP signals by integrating steps such as face detection, alignment, and frequency analysis.
• It employs robust techniques like rigid mesh normalization, dynamic multi-region selection, and advanced signal transformations (e.g., OMIT) to enhance signal stability and quality.
• The approach extends to privacy-preserving and biometric applications, with deep learning variants that intelligently fuse identity and physiological features for improved performance.

Searching arXiv for recent and foundational papers on Face2PPG-style pipelines and related facial rPPG systems. Face2PPG Pipeline denotes a family of computational pipelines that transform facial video into remote photoplethysmography (rPPG), blood volume pulse (BVP), or related physiological representations. In its original formulation, Face2PPG is an unsupervised, modular pipeline that treats rPPG as a full system rather than as only an RGB-to-PPG conversion step, spanning database interfacing, face detection and alignment, ROI selection, RGB extraction, preprocessing, RGB-to-PPG transformation, frequency analysis, and evaluation (Casado et al., 2022). Subsequent work preserved the same face-to-physiology logic while modifying specific stages with rigid mesh normalization, dynamic multi-region selection, Orthogonal Matrix Image Transformation (OMIT), contour-guided multi-task learning, orientation-conditioned UV facial texture mapping, privacy-preserving perturbation, and background-referenced transformer disentanglement (Casado et al., 2022, Zhu et al., 2024, Cantrill et al., 2024, Gupta et al., 2023, Shao et al., 14 Mar 2025).

1. Canonical formulation

The original Face2PPG pipeline is explicitly described as an 8-stage sequence: database interface, face detection and alignment, ROI selection, RGB extraction, pre-processing, RGB-to-PPG transformation, frequency analysis, and evaluation (Casado et al., 2022). Its central methodological claim is that many rPPG studies over-focus on the final signal-combination step while leaving earlier stages under-optimized or inconsistently evaluated.

Stage	Function
Database interface	Reads videos and synchronized reference signals
Face detection and alignment	Detects the face and estimates landmarks
ROI selection	Chooses facial regions for signal extraction
RGB extraction	Computes mean RGB time series over ROIs
Pre-processing	Filters raw RGB to keep the heart-rate band
RGB-to-PPG transformation	Converts RGB to a 1D rPPG waveform
Frequency analysis	Estimates heart rate from spectra
Evaluation	Compares predicted HR against ground truth

This formulation established Face2PPG as a configurable processing chain rather than a single algorithm. The paper also introduced a comparative protocol across six public datasets—PURE, COHFACE, LGI-PPGI, UBFC1, UBFC2, and MAHNOB-HCI—and emphasized that reproducible comparison requires matched spectral analysis, signal synchronization correction, and recognition of imperfections in the reference signals (Casado et al., 2022).

A recurring implication of this design is that the quality of the final BVP estimate is jointly determined by geometric stabilization, region definition, temporal filtering, channel transformation, and evaluation choices. Later Face2PPG-style systems inherited this systems view even when the transformation stage was replaced by deep networks.

2. Spatial normalization and region formation

A major structural contribution of Face2PPG is the use of rigid mesh normalization for face stabilization. The method extends detected facial landmarks from 68 to 85 points by interpolation, constructs a fixed mesh of 131 triangles, and maps each frame’s facial mesh triangle-by-triangle to a normalized template (Casado et al., 2022). This produces a spatio-temporal normalized face representation in which the same facial locations are sampled over time despite motion, pose, or expression changes.

The second major contribution is Dynamic Multi-Region Selection (DMRS) in the Multi-region pipeline. After normalization, the face is divided into an $n \times n$ grid, with 9 × 9 as the default. Each candidate region yields its own raw RGB or rPPG-like time series over a temporal window. Regions with zero variance are removed immediately, and the remaining regions are screened with statistics including mean, standard deviation, variance, signal-to-noise ratio, Katz Fractal Dimension, zero crossings, sample entropy, detrended fluctuation analysis, and local power spectral density energy (Casado et al., 2022). The relative Katz criterion is defined as

$D_{KFD} = \frac{KFD_{region}}{KFD_{global}},$

and regions are retained when

$D_{KFD} \ge 0.85.$

The DFA threshold is reported around 0.75 to 1.0, and the final selection is capped at $r_{max}=32$ regions (Casado et al., 2022).

Variants of Face2PPG-style region design pursue different objectives:

Strategy	Mechanism	Reported purpose
Rigid mesh normalization	68→85 landmarks, 131 triangles	Stabilize facial measurement across motion
DMRS	9×9 grid, KFD/DFA screening, $r_{max}=32$	Keep statistically reliable regions
Cheek–forehead RoI	Left cheek, right cheek, forehead, resized to $64 \times 64$	Preserve rPPG while reducing identity information
Multi-ROI stacking	100 rectangular facial ROIs	Improve robustness to ROI-specific variance

The privacy-preserving pipeline in "Privacy-Preserving Remote Heart Rate Estimation from Facial Videos" (Gupta et al., 2023) uses only the left cheek, right cheek, and forehead. The cheeks are concatenated horizontally, the forehead is concatenated vertically, and the final RoI is resized to $64 \times 64$. The stated rationale is that these regions contain stronger rPPG signals and less identity-related information than the full face. By contrast, the kinship-verification pipeline in "Facial Kinship Verification from remote photoplethysmography" (Wu et al., 2023) selects 100 rectangular facial ROIs and stacks the recovered rPPG traces as a multichannel input, on the grounds that different facial skin regions have different reflection characteristics and different vulnerability to motion or illumination artifacts.

A common misconception is that one uniform facial ROI is sufficient. The Face2PPG literature argues the opposite: dynamic region selection is preferable to fixed patches in unconstrained settings, and ROI design may be optimized either for signal quality, for privacy, or for downstream biometric discrimination depending on the task (Casado et al., 2022, Gupta et al., 2023, Wu et al., 2023).

3. Signal extraction, transformation, and physiological inference

In the canonical unsupervised setting, Face2PPG extracts mean RGB traces from the selected facial regions and applies preprocessing to keep the heart-rate band while removing trends and noise (Casado et al., 2022). The paper evaluates a wide set of RGB-to-PPG transformations—GREEN, ICA, PCA, CHROM, PBV, 2SR, Lab, POS, LGI, and OMIT—and argues that transformation quality cannot be separated from stabilization, region selection, and fair spectral analysis.

The most distinctive new transform in the original paper is Orthogonal Matrix Image Transformation (OMIT), which uses thin QR factorization with Householder reflections. Given an RGB matrix $A \in \mathbb{R}^{n \times 3}$,

$A = QR.$

Using the first orthonormal vector $S$, the projection matrix is

$D_{KFD} = \frac{KFD_{region}}{KFD_{global}},$0

and the projected signal is

$D_{KFD} = \frac{KFD_{region}}{KFD_{global}},$1

The BVP signal is then extracted from the second column of $D_{KFD} = \frac{KFD_{region}}{KFD_{global}},$2 rather than the first, because the first reflects the dominant variation already removed (Casado et al., 2022). OMIT is described as efficient, stable, robust to corrupted or noisy matrices, and especially useful under compression artifacts, with particularly strong results on the highly compressed MAHNOB-HCI dataset (Casado et al., 2022).

Heart-rate inference in Face2PPG-style systems is consistently spectral, but not fully uniform across implementations. The original unsupervised pipeline uses 10 s sliding windows with 1 s step size, Kaiser-window FIR filtering with $D_{KFD} = \frac{KFD_{region}}{KFD_{global}},$3, and a 0.75–4 Hz bandpass for evaluation (Casado et al., 2022). The privacy-preserving pipeline derives HR from the predicted rPPG using the Welch power spectrum method (Gupta et al., 2023). In rFaceNet, HR is defined as the maximum spectral peak of the FFT-transformed BVP in the physiological band

$D_{KFD} = \frac{KFD_{region}}{KFD_{global}},$4

(Zhu et al., 2024). The orientation-conditioned UV pipeline detrends the signal using Tarvainen’s method, applies a 2nd-order Butterworth bandpass filter with cutoff frequencies 0.75–2.50 Hz, computes the FFT, finds the dominant frequency, and converts it to BPM (Cantrill et al., 2024).

These differences are methodologically significant. The Face2PPG literature explicitly notes that direct waveform comparison is often not ideal because facial rPPG and fingertip or contact PPG differ in delay, scale, morphology, measurement site, and filtering effects (Casado et al., 2022). For that reason, many studies emphasize HR agreement, error metrics, and correlation rather than exact waveform identity.

4. End-to-end deep architectures and geometry-aware variants

Later Face2PPG-style pipelines replace the handcrafted transformation stage with deep architectures while preserving the logic of face-to-physiology conversion. Three representative lines are contour-guided multi-task learning, 3D-surface-conditioned UV mapping, and interference-aware transformer disentanglement.

Model	Input representation	Key mechanism
rFaceNet	Temporally normalized RGB frames	TCU + CTFC + multi-task supervision
PhysNet-UV	Orientation-conditioned UV texture video	3D facial surface mapping + orientation masking
Outdoor transformer rPPG	Face, background, and global STMaps	Global interference sharing + self-supervised disentanglement

In "rFaceNet: An End-to-End Network for Enhanced Physiological Signal Extraction through Identity-Specific Facial Contours" (Zhu et al., 2024), the pipeline takes face-cropped and stabilized facial video, represented during training as 64 × 128 × 128 in the main implementation and 128 × 72 × 72 for cross-dataset experiments. The network splits into an identity or contour branch and an rPPG or BVP branch. The contour branch applies a Temporal Compressor Unit (TCU), then a 2D feature extractor and identity classifier; the physiological branch applies a 3D feature extractor, a Cross-Task Feature Combiner (CTFC), and a BVP predictor. The TCU shifts dimensions, applies 3D Adaptive Average Pooling to compress temporal information while preserving spatial resolution, and flattens the result into a 2D tensor. The CTFC upsamples the identity feature map, replicates it across the temporal axis, and performs linear superposition with the physiological feature map: $D_{KFD} = \frac{KFD_{region}}{KFD_{global}},$5 The model is trained with an uncertainty-weighted multi-task loss over BVP regression, heart-rate regression, and identity classification (Zhu et al., 2024).

The paper’s geometric premise is that contour and physiological pixels have minimal positional overlap, yet the facial contour surrounds or constrains the regions from which physiological information is extracted. That assumption motivates a fusion design that does not simply concatenate all features everywhere. The ablation study reports that the best result occurs when both TCU and CTFC are used, with CTFC applied at the Stage-3 BVP feature output (Zhu et al., 2024).

"Orientation-conditioned Facial Texture Mapping for Video-based Facial Remote Photoplethysmography Estimation" (Cantrill et al., 2024) takes a different route. Instead of ordinary XY face crops, it constructs a UV texture video from a 3D facial mesh built from MediaPipe FaceMesh with 468 3D facial landmarks. Facial pixels are warped into a common UV texture space through piecewise affine transformations over triangles, with bilinear interpolation and undefined areas filled with 0. The method then computes a per-vertex relative angle $D_{KFD} = \frac{KFD_{region}}{KFD_{global}},$6 between the vertex normal and the camera normal $D_{KFD} = \frac{KFD_{region}}{KFD_{global}},$7, interpolates these angles into UV space to form $D_{KFD} = \frac{KFD_{region}}{KFD_{global}},$8, and masks unreliable regions. The best masking threshold is reported as

$D_{KFD} = \frac{KFD_{region}}{KFD_{global}},$9

which gave the best trade-off between removing distorted geometry and preserving signal (Cantrill et al., 2024). The masked UV video is then resized, processed by first-order normalized frame difference, pixel outlier clipping, and standardization, and fed into PhysNet.

"Remote Photoplethysmography in Real-World and Extreme Lighting Scenarios" (Shao et al., 14 Mar 2025) extends the Face2PPG logic to outdoor and extreme-lighting conditions. Its representation is a set of STMaps rather than raw 3D video tensors. The face is partitioned by 68 facial landmarks into 64 non-overlapping regions, producing a face STMap; a background STMap is built from the area outside the face bounding box; and a global STMap is obtained by averaging blocks over the full frame. These are processed by a U-shaped Swin video transformer with global interference sharing, subject background reference, self-supervised disentanglement, reconstruction guidance, and frequency-domain and biological prior constraints. The key refinement step is

$D_{KFD} \ge 0.85.$0

which suppresses foreground components correlated with the background (Shao et al., 14 Mar 2025). This directly opposes the common assumption that the background is always irrelevant; here it is treated as a reference for interference.

5. Privacy, kinship, and biometric extensions

The Face2PPG pipeline has also been reinterpreted as a privacy-preserving and biometric representation pipeline. In the privacy-preserving variant (Gupta et al., 2023), the pipeline detects and aligns the face, crops only the cheek–forehead RoI, applies pixel shuffling, blurs the shuffled image with a $D_{KFD} \ge 0.85.$1 Gaussian kernel, and then feeds the perturbed clip into a 3D CNN for rPPG estimation. Each frame is flattened into 4096 pixels, the same random shuffle key is used across all frames in the same clip, and the key space is noted as $D_{KFD} \ge 0.85.$2. The model is trained with Smooth L1 loss with $D_{KFD} \ge 0.85.$3. The stated objective is to preserve overall color intensity, which remains useful for rPPG, while destroying spatial facial structure needed for recognition (Gupta et al., 2023).

The reported privacy–utility trade-off is quantitative. For the proposed RoI + Sh + B setting with unlimited key space, the paper reports on PURE MAE 0.96, RMSE 1.30, $D_{KFD} \ge 0.85.$4, and ID 46.19%, and on UBFC MAE 0.89, RMSE 1.24, $D_{KFD} \ge 0.85.$5, and ID 36.14% (Gupta et al., 2023). The same work states that face-recognition accuracy is reduced by over 60% in some settings, and also notes an important limitation: privacy evaluation is based mainly on recognition accuracy and reconstruction difficulty rather than formal information-theoretic guarantees (Gupta et al., 2023).

In the kinship-verification reinterpretation (Wu et al., 2023), facial video is first converted to ROI-based rPPG signals and then processed by a Siamese-like 1D CNN with a channel-attention module and contrastive loss. For each pair, the inputs are multichannel tensors

$D_{KFD} \ge 0.85.$6

where the channels are derived from 100 facial ROIs. Each branch contains two 1D convolution blocks with kernel sizes $D_{KFD} \ge 0.85.$7, strides $D_{KFD} \ge 0.85.$8, and output channels $D_{KFD} \ge 0.85.$9, followed by a squeeze-and-excitation-style channel attention mechanism with reduction ratio $r_{max}=32$0 and a three-layer embedding head. The contrastive loss is

$r_{max}=32$1

with margin $r_{max}=32$2 (Wu et al., 2023). On the UvANEMO Smile Database, the full multichannel attention model achieved average AUCs of approximately 60.88 ± 6.80 for LGI, 64.65 ± 8.66 for OMIT, 67.93 ± 9.45 for GREEN, 68.82 ± 11.84 for CHROM, and 69.28 ± 3.41 for POS. The ablation study reports that multi-channel input improved average AUC by 3.39%, and channel attention added about 1.66% AUC on average (Wu et al., 2023).

A further biometric direction appears in "A Hybrid Deep Learning Model for Robust Biometric Authentication from Low-Frame-Rate PPG Signals" (Rahman et al., 6 Nov 2025). That work is based on low-frame-rate fingertip videos rather than facial videos, but it states that in a Face2PPG setting the same logic would apply conceptually: a camera stream is converted into a pulsatile physiological waveform, then detrended, artifact-suppressed, bandpass-filtered, resampled, normalized, transformed into scalograms, and classified into an identity. The model combines a CVT branch, a ConvMixer branch, and an LSTM branch, and reports about 98% authentication accuracy on the CFISHR dataset (Rahman et al., 6 Nov 2025). This suggests a broader interpretation of Face2PPG as a representation pipeline in which physiology, rather than facial appearance, becomes the primary biometric substrate.

6. Benchmarks, empirical performance, and methodological issues

The original Face2PPG paper reports that Face2PPG-Multi achieves the best performance among unsupervised, non-learning methods across all six datasets, and that in some databases it is very close to supervised, learning-based methods (Casado et al., 2022). It also reports efficiency of about 17 ms per frame, or under 33 ms including face detection and alignment (Casado et al., 2022). These results established the pipeline as a practical unsupervised baseline and as a methodological reference point.

Among later end-to-end models, rFaceNet reports strong benchmark performance on VIPL-HR, with SD = 4.87, MAE = 5.21, RMSE = 7.13, and $r_{max}=32$3, outperforming prior methods in SD, RMSE, and correlation $r_{max}=32$4 and remaining competitive in MAE (Zhu et al., 2024). In cross-dataset evaluation on UBFC-rPPG, it reports MAE = 1.05 ± 0.35, RMSE = 2.51 ± 2.55, and $r_{max}=32$5, while on PURE it reports MAE = 3.74 ± 1.52, RMSE = 11.95 ± 71.94, and $r_{max}=32$6 (Zhu et al., 2024). The ablation study shows that only CTFC without TCU already improves performance, only TCU without CTFC hurts performance, and the best configuration uses both modules at Stage-3 (Zhu et al., 2024).

The UV-texture variant reports its headline cross-dataset result for PURE $r_{max}=32$7 MMPD as an 18.2% reduction in MAE relative to standard face crops, from 14.905 BPM MAE for PhysNet-XY to 12.187 BPM MAE for PhysNet-UV (Cantrill et al., 2024). It also reports RMSE improvement of 11.9%, correlation improvement of 89.8% from $r_{max}=32$8 to $r_{max}=32$9, and scenario-wise MAE improvements on MMPD of 21.5% for stationary, 10.2% for stationary after exercise, 29.6% for rotation, 7.5% for talking, and 20.8% for walking (Cantrill et al., 2024). The same paper argues that cross-dataset testing is the meaningful evaluation because intra-dataset results on PURE can be misleadingly optimistic or unstable (Cantrill et al., 2024).

The outdoor transformer paper reports that its model outperforms standard chrominance methods, CNN-based methods, and recent transformer methods on multiple datasets, especially MR-NIRP-DRV, and emphasizes robustness in real-world driving scenarios with day–night and illumination changes (Shao et al., 14 Mar 2025). Its design choices—background reference, coarse reconstruction, disentanglement, and biological priors—are explicitly targeted at conditions in which interference may occupy or exceed the subtle biosignal amplitude (Shao et al., 14 Mar 2025).

Several methodological cautions recur across the literature. First, better face cropping is not equivalent to surface-aware alignment: the UV-mapping work reports that dynamic face cropping alone can hurt due to jitter, whereas UV mapping gives a temporally consistent facial surface representation (Cantrill et al., 2024). Second, privacy-preserving perturbation should not be conflated with formal privacy guarantees (Gupta et al., 2023). Third, the kinship-verification study states that robustness on noisy real-world data remains unexplored, that the approach depends on traditional unsupervised rPPG extraction quality, and that the current evidence comes from a controlled, front-facing indoor dataset (Wu et al., 2023). Fourth, several pipelines depend critically on reliable landmarking or 3D facial reconstruction; the UV-texture paper explicitly notes that accurate and stable 3D landmark detection is required (Cantrill et al., 2024).

Taken together, these works define the Face2PPG Pipeline not as one fixed implementation but as a research program. Its invariant structure is the conversion of facial video into physiological observables; its main axes of variation are geometric normalization, ROI policy, signal transformation, fusion strategy, privacy perturbation, and downstream task design. The original unsupervised Face2PPG paper framed this as a modular systems problem (Casado et al., 2022). Later work extended that framing into contour-guided multi-task estimation, 3D-surface-conditioned motion robustness, interference-disentangled transformers for outdoor illumination extremes, and physiology-based biometric applications (Zhu et al., 2024, Cantrill et al., 2024, Shao et al., 14 Mar 2025, Wu et al., 2023, Gupta et al., 2023).