Patch-Level Posterior Sampling Technique

• Patch-Level Posterior Sampling Technique is a method that leverages patch-scale diffusion priors to reconstruct high-fidelity facial reflectance maps from low-quality video.
• The approach divides full-resolution maps into overlapping, UV-conditioned patches, using tiled posterior sampling to blend local details and eliminate seams.
• It enhances data efficiency by multiplying training examples from limited studio scans and enables practical, studio-quality facial reconstructions for AR/VR and digital avatar applications.

A patch-level posterior sampling technique is a strategy for solving high-resolution inverse problems (notably, facial reflectance capture) by sampling from the posterior distribution of reflectance maps using a prior learned at the image-patch scale. It is designed to address the challenge of reconstructing seamless, high-fidelity facial reflectance maps from everyday smartphone video, while leveraging the strong statistical regularities present in datasets from studio Light Stage scans. The approach combines a UV-conditional diffusion model trained on local image patches (the “patch-level diffusion prior”) with a novel, tiled posterior sampling procedure tailored to full-resolution face asset reconstruction, enabling practical restoration of studio-quality results in data-limited, resource-constrained settings.

1. Patch-Level Posterior Sampling Mechanism

The technique involves two principal components:

• Patch-level diffusion prior: The prior is trained by extracting random 256×256 patches from high-resolution Light Stage scans, with each patch comprising concatenated diffuse albedo (3 channels), specular albedo (1 channel), normal (3 channels), and UV coordinates (2 channels). The model is a conditional diffusion model (i.e., a UNet predicting noise at each timestep), which is conditioned on the patch’s UV region to preserve both appearance and pose context.
• Patch-wise posterior sampling for inverse problem: Given multi-view smartphone video (with fixed geometry and lighting), the task is to infer full-resolution reflectance maps such that their renderings match all captured images as well as possible, but restrict the possible outputs to the studio distribution defined by the patch-level prior.

Due to hardware and dataset constraints, direct diffusion sampling over the entire 4K map is infeasible. The proposed procedure splits the current map into overlapping patches, applies the patch-level DPS (Diffusion Posterior Sampling) step to each patch independently (using UV coordinates for spatial context), and then blends overlaps (with weighted averaging as in Tiled Diffusion) to construct a seamless candidate for the next diffusion iteration.

At each reverse diffusion step $t$:

1. Patch splitting: The full map is divided into overlapping patches of size $p^+ = p + 2p_{pad}$.
2. Patch denoising and guidance: For each patch, a reverse diffusion update is performed:

$$x_{t-1}^{ij+} = \frac{1}{\sqrt{\alpha_t}} \left[ x_t^{ij+} - \frac{1-\alpha_t}{\sqrt{1-\bar{\alpha}_t}} \epsilon_\theta(x_t^{ij+}, t \mid y^{ij+}) \right] + \sigma_t z$$

followed by a gradient update along the photometric loss:

$$x_{t-1}^{ij+} \leftarrow x_{t-1}^{ij+} - \zeta_t \nabla_{x_t^{ij+}} \mathcal{L}_{pho}(\hat{x}_t^{ij+})$$

where $$\mathcal{L}_{pho}(x) = \sum_{i=1}^V \| \mathcal{R}(x, i) - I_i \|_2^2$$, with $V$ being the number of views, $\mathcal{R}$ a differentiable renderer, and $I_i$ the captured images.

1. Patch blending: Overlapping patches are averaged to reconstruct the full-resolution candidate for the next step.
2. Iteration: Steps 1–3 are repeated for each diffusion timestep $t=T \to 0$.

This process results in a reconstructed reflectance map that is globally seamless and statistically consistent with the studio prior, but also photometrically matches the observed low-cost data.

2. Statistical and Practical Benefits for Facial Reflectance Capture

The patch-level posterior sampling approach confers several key benefits:

• Data efficiency and improved generalization: Using a patch-based prior multiplied the effective dataset size (since each scan can yield thousands of unique, spatially localized patches), greatly mitigating the limitation imposed by the small number of available studio scans while retaining the ability to train high-capacity diffusion models.
• Realism and detail preservation: By modeling plausible appearance at the patch level (with the UV coordinate conveying spatial context), the model effectively constrains the inverse problem’s solution to lie within the empirical data distribution of studio captures. This enables the recovery of high-frequency detail and person-specific features under-constrained by the noisy low-cost video.
• Stability and seamlessness: Overlapping, UV-conditioned patchwise updates followed by blending eliminate seam artifacts that would otherwise appear if non-overlapping patches or naive sliding window approaches were used.
• Hardware feasibility: The approach makes studio-quality, full-resolution inversion possible on consumer hardware with moderate GPU memory, since only small patches are processed at a time and global solution updates are achieved via local blending.

3. Comparison with Previous Facial Appearance Capture Approaches

Approach	Prior/Generative Model	Resolution/Training Regime	Patch Coordination	Seamlessness/Quality
PCA (e.g. HiFi3DFace)	Low-capacity, linear Gaussian	Full/global, limited data	N/A	Blurry, loses local details
GANs (prior works)	Moderate, often global or patch	Moderate, may overfit	Poor/limited	Inconsistent, may have artifacts
Patchwise optimization	Weak/no prior, local search	Local/patch only	None	Strong seams, low realism
Patch-level diffusion (this work)	High-capacity, patch-level diffusion	Patch—trained, global—sampled	Overlap + UV conditioning	Seamless, high realism, efficient

The patch-level diffusion technique outperforms PCA-based and GAN-based priors in terms of realism, local detail, and seamlessness. Unlike prior patch-based approaches, UV conditioning enables spatially consistent outputs even when global structure must be reconstructed from limited or ambiguous evidence.

4. Training and Role of the Diffusion Prior at Patch Level

The diffusion prior is trained on 256×256 patches from Light Stage scans, concatenated with the corresponding UV coordinate map and other appearance channels. The UV information (provided with positional encoding) is present at both training and sampling time, anchoring each patch in its correct spatial context.

The rationale for patch-wise training arises from the scarcity of full-head, high-resolution scans: studio datasets are small in number but massive in per-sample spatial extent. This strategy achieves both training stability (obviating overfitting/global collapse) and generalization across facial regions, identities, and expressions.

At inference, the prior conditions each patch sample not only on its local appearance but also the UV-mapped spatial context, ensuring reconstruction of globally plausible facial structure even from local observations.

5. Experimental Validation and Ablation Studies

Experimental validation shows that the method achieves state-of-the-art performance for smartphone facial reflectance capture:

• Objective metrics: Improvements in PSNR, SSIM, and LPIPS over previous low-cost methods (e.g., HiFi3DFace, CoRA) are reported. Table 1 (see the paper) quantifies these advances.
• Visual quality: Figures in the work show that the method closes the perceptual gap with Light Stage ground truth, recovering pores, skin texture, and landmarks that are lost in previous methods.
• Ablations: Removing UV conditioning degrades consistency, and omitting tile blending introduces seams. Larger patch overlaps and patch sizes further improve output quality and efficiency; increased diffusion step sizes can trade off between data fidelity and prior adherence.
• Generalization: Despite training on only 48 Light Stage scans, the system generalizes to diverse subjects and facial expressions not present in the training set, suggesting the prior captures broadly valid appearance regularities.

6. Applications, Limitations, and Broader Impact

The patch-level posterior sampling paradigm enables studio-quality reflectance map recovery using only low-cost, easy-to-acquire data, with immediate applications in digital avatar creation, AR/VR, telepresence, and entertainment. The method demonstrates that fine-grained, data-driven priors—when trained at the correct local scale and paired with explicit spatial encodings—can be successfully harnessed via patch-level posterior sampling pipelines to realize seamless, high-fidelity reconstructions in otherwise data- and resource-limited settings.

Potential limitations include the need for consistent and accurate UV mapping of the target face, and the reliance on sufficient overlap and noise scheduling in the sampling process to avoid artifacts. While a plausible implication is that advancements in UV mapping and scalable diffusion models will further empower such frameworks, success remains bounded by the coverage of the training data and fidelity of the forward modeling (particularly the differentiable renderer used in photometric loss).

Summary Table of the Process

Stage	Inputs	Processing	Outputs
Patch-level diffusion prior training	256×256 Light Stage patches, UV	Conditioned diffusion model training (with positional encoding)	UV-conditional patch-level prior
Inference (posterior sampling)	User multi-view images, geometry, UV	Patchwise DPS with UV context, photometric loss, tiled blending	Seamless, photo-consistent reflectance maps