FaceRefiner: High-Fidelity Facial Texture Refinement

• The paper presents a novel framework that integrates multi-stage style transfer with photometric supervision to refine oversmoothed UV facial textures.
• FaceRefiner utilizes a differentiable renderer and VGG-16 based hypercolumn features with STROTSS to restore fine details and preserve identity.
• Quantitative results show significant improvements in PSNR and SSIM over baseline methods, confirming its state-of-the-art performance.

FaceRefiner is a facial texture refinement framework that leverages differentiable rendering-based style transfer to produce high-fidelity UV textures from single in-the-wild facial images. Designed as a post-processing "plug-in" for existing UV-based face texture generators, FaceRefiner integrates multi-level style transfer with photometric supervision from the input image to rectify over-smoothing, identity drift, and loss of detail introduced by generative models. The method explicitly preserves visible features, skin structure, and true shading, achieving state-of-the-art results in both quantitative and qualitative benchmarks (Li et al., 8 Jan 2026).

1. System Overview and Motivation

FaceRefiner addresses the limitation of current face texture generation pipelines, where deep networks often synthesize UV textures that lack precise correspondence to the subject's details and identity—especially for real-world, unconstrained images. Typically, a baseline face generator (e.g., Deep3DFace, OSTEC) first recovers a 3D mesh, camera pose, and a complete but oversmoothed UV texture. FaceRefiner takes as input: (1) the original image; (2) a reconstructed mesh and pose; (3) an incomplete, direct UV sample (style image); and (4) a baseline-completed UV (content image). The objective is to recover a refined UV texture consistent with the global plausibility of the generator's result, while restoring fine details and identity from the original input.

The core procedure alternates style transfer—using the incomplete UV map as "style" and the generator output as "content"—with differentiable rendering that forces visible regions to match the input photograph at the pixel level. The process is staged, progressing through increasing resolutions and dynamic loss reweighting, to balance detail recovery and smoothness.

2. Architecture and Differentiable Rendering

FaceRefiner's neural architecture consists of three interconnected components:

• Feature Extraction: VGG-16 hypercolumns (n ≈ 2179 channels) are computed from the input, style, and content images. Each pixel is mapped to a rich local descriptor, aggregating features across multiple layers for multi-scale analysis.
• Style Transfer Backbone: The STROTSS algorithm, featuring a cost matrix between feature hypercolumns and employing a one-way Relaxed Earth Mover’s Distance (EMD) for style loss, moment matching for feature distributions, and direct pixel-space color matching. STROTSS supports binary masking, ensuring that UV "holes" or invisibilities in the sampled style do not corrupt the synthesized result.
• Differentiable Renderer: Implemented via [Laine et al. ’20], it rasterizes mesh triangles under the current texture and pose with simple Lambertian shading and antialiasing. The renderer computes the projection of the 3D mesh with barycentric interpolation, enabling gradients from pixel-wise losses in the rendered image to flow efficiently back to the UV texture space.

The direct UV sampling for the style map uses 3D-2D mesh correspondences and coarse visibility masking, setting invisible regions to black to prevent contamination during loss evaluation.

3. Multi-stage Style Transfer and Loss Optimization

The core optimization loop is split into 5 progressive stages. Each stage performs the following:

• Resolution: Begins at 256×256 pixels, concluding at 512×512.
• Objective: At each stage, FaceRefiner minimizes a composite loss:

$$L_\mathrm{st}(I_X) = \alpha\,L_\mathrm{content}(I_X,I_C) + \beta\,L_\mathrm{style}(I_X,I_S) + \gamma\,L_\mathrm{render}(I_X,I)$$

where: - $L_\mathrm{content}$: Self-similarity loss on VGG-16 features between the solution and the generator’s UV (content). - $L_\mathrm{style}$: STROTSS-based EMD and moment/color matching between the solution and the reconstructed UV (style). - $L_\mathrm{render}$: $L_1$ difference in the pixel space between the rendered image (from the current UV, mesh, and pose) and the input photograph over the visible mask.

• Stage-wise scheduling: $\alpha$ is decreased and $\gamma$ increased across stages, shifting importance from global/structural similarity to fine-grained, pixel-level accuracy.

Optimization is conducted with stochastic gradient descent (SGD, lr = 0.3, momentum = 0.9, 150 iters per stage), initializing the variable as a Laplacian pyramid for rapid convergence. No network retraining is needed; each image is refined in a self-supervised procedure.

4. Loss Formulation and Information Transfer

FaceRefiner's objective decomposes into multi-level constraints:

• Content Preservation:

$$L_\mathrm{content} = \frac{1}{n^2} \sum_{i,j} | D^{X}_{ij} - D^{C}_{ij} |,$$

where $D^{X}_{ij}$ is the cosine feature distance between VGG-16 hypercolumns in the current solution $I_X$; encourages the refined UV to maintain the high-level structures of the (possibly oversmoothed) generator's output.

• Style Transfer:

$$L_\mathrm{style} = L_r + L_m + \frac{1}{\max(\alpha, 1)} L_p,$$

comprising: - $L_r$: Relaxed EMD term based on the minimal feature cost between $I_X$ and $I_S$. - $L_m$: Channel-wise feature mean/variance matching. - $L_p$: Direct L1 color matching.

• Photometric/Rendering Supervision: For visible regions, the rendering loss

$$L_\mathrm{render} = \sum_p | (I_R(p) - I(p)) \cdot M_F(p) |$$

matches the differentially rendered image to the original, preserving exact chrominance, shadowing and fine markings such as spots and creases.

This multi-level design ensures high-level semantic, mid-level structural, and low-level pixel cues are all recovered from the original image.

5. Experimental Validation

Extensive experiments demonstrate substantial improvements over previous state-of-the-art methods:

• Datasets: Multi-PIE (pose-varied ground-truth UVs), CelebA (in-the-wild), and FFHQ.
• Metrics: Peak Signal-to-Noise Ratio (PSNR), Structural Similarity (SSIM), mean absolute error (MAE), identity similarity via LightCNN and evoLVe.

Key numerical results (Multi-PIE, mean across poses):

Method	PSNR (dB)	SSIM	Identity Similarity*
Deep3DFace baseline	18.24	0.8257	–
OSTEC baseline	22.40	0.8391	–
+FaceRefiner (Deep3D)	24.78	0.8739	–
+FaceRefiner (OSTEC)	24.79	0.8717	–

*Identity metrics (LightCNN/evoLVe) provided for CelebA.

On CelebA, FaceRefiner, when applied to OSTEC as the base generator, improved PSNR from 25.69 dB to 30.20 dB, SSIM from 0.8841 to 0.9375, and LightCNN identity similarity from 0.8319 to 0.9853. Qualitatively, FaceRefiner exhibited recovery of freckles, precise lip shading, hair highlights, and overall higher photorealism. Ablation studies showed that removal of any core loss term or reduction of the number of transfer stages significantly degraded both PSNR and SSIM, confirming the necessity of multi-level supervision and multi-stage optimization.

6. Implementation Details and Usage

• Training Regimen: No end-to-end network training; per-image optimization takes ~2 minutes on an NVIDIA RTX 3090.
• Optimization: Laplacian pyramid variables, SGD-based minimization, progressive upsampling across stages.
• Integration: Can post-process any UV-based facial texture generator that outputs a UV map and coarse 3D geometry; requires only the original image, mesh, baseline UV, and camera parameters.
• Software Stack: PyTorch; custom differentiable renderer for efficient gradient backpropagation through UV space.

7. Significance, Limitations, and Extensions

By combining classical hypercolumn-based style transfer with differentiable rendering-based pixel supervision, FaceRefiner bridges the gap between plausible generative priors and photorealistic, identity-preserving detail recovery. The result is a robust system capable of extracting both visible and semantic identity cues from unconstrained inputs—even in the presence of occlusions, poor lighting, or generator-related biases (Li et al., 8 Jan 2026).

Current limitations include reliance on provided, accurate 3D geometry and pose, as well as the effectiveness of the base generator in extrapolating occluded regions. Future improvements could incorporate adaptive guidance for ambiguous or occluded regions, joint optimization over geometry and texture, or temporally-consistent pipelines for video inputs.

FaceRefiner's methodology sets a new standard for post-hoc refinement in facial texture synthesis, with applications in facial reenactment, relighting, digital humans, and forensic analysis.