Face Retouching Restoration (FRR) Overview

• Face Retouching Restoration (FRR) is the process of recovering natural faces by reversing complex beautification edits that alter low-frequency structure such as geometry and tone.
• Key methods leverage conditional models and coarse-to-fine pipelines, using label-guided and diffusion-based approaches to separate and restore structure and high-frequency details.
• Recent FRR research employs paired and synthetic datasets across multiple retouching operations to enhance identity preservation and adapt to diverse beautification techniques.

Face Retouching Restoration (FRR) is the task of reconstructing an original, un-retouched or more natural face from a retouched counterpart, with emphasis on authenticity, identity fidelity, and the reversal of beautification-induced changes in geometry, tone, and texture rather than only classical signal degradations such as blur or noise. In the recent literature, FRR is formulated either as blind reconstruction of the original face $Y$ from a retouched image $X$, via a conditional model $p_\theta(Y \mid X)$, or as a label- or reference-guided inverse mapping that seeks to recover both geometry and appearance from edited inputs (Liu et al., 26 Jul 2025, Zhao et al., 2024, Xing et al., 2024).

1. Definition, scope, and relation to adjacent tasks

FRR differs from generic image restoration because facial retouching changes low-frequency and semantic facial content, not only high-frequency detail. The MoFRR formulation states that FRR addresses “complex retouching operations with various types and degrees,” and that it “focuses more on the restoration of the low-frequency information of the faces,” including geometric reshaping and global tone changes rather than only denoising or deblurring (Liu et al., 26 Jul 2025). Re-Face likewise defines the objective as recovering an image close to the original un-retouched face “including both geometry and appearance (color/texture),” explicitly targeting deformation-style retouching such as eye enlargement, face lifting, and skin smoothing (Zhao et al., 2024).

This places FRR near blind face restoration, but with semantic or stylistic degradations replacing purely physical ones. The face restoration survey formalizes this perspective by rewriting a retouched image as $$I_{ret} = D_{\text{retouch}}(I_{nat}; \delta_{ret})$$, where the degradation is a retouch pipeline rather than blur, noise, or compression; it further argues that FRR is closest in complexity to blind face restoration because the transformation is unknown, diverse, and identity-affecting (Li et al., 2023). In the same sense, FRR is distinct from makeup removal, which typically assumes aligned geometry and mainly removes cosmetic color or texture overlays, whereas FRR must also invert non-linear geometric edits such as face slimming and eye enlargement (Zhao et al., 2024).

Concrete retouching operations vary by benchmark. Re-Face uses a three-dimensional label vector $$\mathbf{m}_l = [m_{\text{eye}}, m_{\text{face}}, m_{\text{smooth}}] \in \{0,1,2,3\}^3$$, with four discrete degrees for eye enlargement, face lifting, and skin smoothing (Zhao et al., 2024). MoFRR organizes the task around whitening, smoothing, face lifting, and eye enlarging, and also includes multi-operation images where several such manipulations coexist (Liu et al., 26 Jul 2025). FRRffusion targets six Face++ beautification operations—Eye Enlarging, Face Slimming, Skin Whitening, Skin Smoothing, Eyebrow Shaping, and Face Shrinking—applied simultaneously at maximum level to emphasize strong deceptive beautification (Xing et al., 2024).

2. Datasets and empirical task construction

The emergence of FRR as a trainable problem has been coupled to the creation of paired retouched/original datasets. These datasets differ in data source, retouching engine, and whether the emphasis is label prediction, paired inversion, or cross-API generalization.

Dataset	Source and scale	Role in FRR research
deepFRR	50,000 StyleGAN-generated 1024×1024 faces and corresponding Face++-retouched counterparts	Paired training and evaluation for FRRffusion (Xing et al., 2024)
RetouchingFFHQ	Large-scale synthetic retouching dataset with labels for eye, face, and smoothing operations; the full dataset contains more than 500,000 conditioned retouched images	Detector training and label-guided reversion in Re-Face (Zhao et al., 2024)
RetouchingFFHQ++	57,910 original images, 404,542 single-operation retouches, 612,886 multi-operation retouches, for 1,075,338 retouched images in total, built with Portrait, Megvii, Tencent, and Alibaba APIs	Large-scale benchmark for MoFRR, including intra-API and cross-API evaluation (Liu et al., 26 Jul 2025)

deepFRR is notable because it is entirely synthetic at the identity level: its raw faces are generated by StyleGAN2 from the SeePrettyFace StyleGAN dataset, then retouched through the Face++ Beautify API. The authors justify this by privacy, legal, and scalability considerations, arguing that synthetic faces allow large-scale paired supervision without portrait-right issues while still providing realistic identity variation (Xing et al., 2024). Re-Face instead builds on RetouchingFFHQ and explicitly filters its FaceR training subset to 19,757 paired samples after excluding occlusions, closed eyes, and infants, while still training the retouching detector on the full dataset (Zhao et al., 2024). MoFRR extends earlier retouching data by adding multi-operation images and multiple commercial APIs, then uses an 8:1:1 split for both intra-API and cross-API experiments (Liu et al., 26 Jul 2025).

These datasets also encode different views of FRR difficulty. deepFRR fixes all six Face++ operations at level 100, creating a strong mixed-retouching regime (Xing et al., 2024). Re-Face uses discrete operation labels as explicit conditions for inversion (Zhao et al., 2024). RetouchingFFHQ++ derives degree bins from PSNR distributions rather than raw API values, thereby defining a content-based notion of retouch intensity that is consistent across APIs (Liu et al., 26 Jul 2025).

3. Architectural paradigms specific to FRR

Three architectural patterns currently dominate explicit FRR systems: label-guided diffusion reversion, coarse-to-fine diffusion with a dedicated detail generator, and mixture-of-experts diffusion.

Re-Face is a label-guided system built from a Facial Retouching Detector, a ControlNet-based FaceR reversion model, and a color correction module called Hierarchical Adaptive Instance Normalization (H-AdaIN) (Zhao et al., 2024). Its detector predicts the retouching label $\mathbf{m}_l$, then FaceR conditions a frozen Stable Diffusion v1.5-style latent diffusion model on three signals: a fixed text prompt (“a human face”), a latent encoding of the retouched image, and an MLP embedding of $\mathbf{m}_l$. Only the ControlNet branch and small conditioning modules are trained. This makes the restoration explicitly dependent on the detected operation types and degrees, rather than forcing the model to infer them implicitly.

FRRffusion adopts a coarse-to-fine decomposition (Xing et al., 2024). Its first stage, the Facial Morpho-Architectonic Restorer (FMAR), is a conditional DDPM trained on paired raw/retouched faces at 128×128, so that diffusion reconstructs the authentic low-resolution face $x_0$ given the retouched image $y_0$. Its second stage, the Hyperrealistic Facial Detail Generator (HFDG), is a Transformer-based super-resolution network built on HAT-B. FMAR is responsible for global morphology, contours, and coarse shading, while HFDG adds high-resolution details such as skin texture, hair strands, wrinkles, and eyebrow detail. This separation reflects a core FRR assumption: geometric and low-frequency corrections should precede high-frequency refinement.

MoFRR frames FRR as a sparse mixture of diffusion experts (Liu et al., 26 Jul 2025). A router predicts which retouching types are present; activated specialized experts then reverse individual operations, while a shared expert remains always on to capture universal retouching traces. Each specialized expert is a WaveFRR model with a dual-branch structure. The low-frequency branch applies a DDIM-based restoration guided by an Iterative Distortion Evaluation Module (IDEM), and the high-frequency branch uses a Cross-Attention-based High-Frequency branch (HFCAM) for detail refinement. The image is explicitly decomposed with a discrete wavelet transform into $x_{LL}$ and $X$0, which operationalizes the claim that FRR is primarily a low-frequency restoration problem with high-frequency refinement as a secondary stage (Liu et al., 26 Jul 2025).

Despite these differences, all three paradigms share a common principle: FRR benefits from separating operation recognition, low-frequency or structural inversion, and appearance refinement, rather than treating the task as a single undifferentiated image translation.

4. Semantic, identity, and prior-based foundations

Although explicit FRR formulations are recent, several broader face restoration lines provide mechanisms that directly inform FRR. HiFaceGAN formulates “Face Renovation” as a “dual-blind” problem with unknown heterogeneous degradations and no external structural guidance, then uses collaborative suppression and replenishment (CSR) to extract internal semantic maps and progressively restore structure, textures, illumination, and color (Yang et al., 2020). This is not an FRR system, but it establishes a useful prior: restoration can be guided by learned internal facial semantics rather than fixed degradation models or explicit labels. In FRR terms, this suggests that retouch-induced distortions may likewise be reversed through internally learned hierarchical semantics.

PSFR-GAN makes this facial-semantic idea explicit through multi-scale parsing-conditioned style transformation and a semantic-aware style loss computed per semantic region (Chen et al., 2020). It requires parsing maps, but its conditioning mechanism—region-wise modulation of features with skin, eyes, lips, and hair treated differently—maps closely to FRR, where skin smoothing, eye enlargement, or lip enhancement are localized and heterogeneous edits.

A second major foundation is identity-aware restoration. MGFR uses a multi-modal diffusion design with attribute text prompts, high-quality reference images, and ArcFace-based identity embeddings to reduce “false facial attributes and identities” in restoration (Tao et al., 2024). RestorerID introduces a tuning-free reference-guided diffusion framework with a Face ID Rebalancing Adapter (FIR-Adapter) and an Adaptive ID-Scale Adjusting strategy, showing that reference-guided identity preservation can be achieved without per-identity test-time finetuning (Ying et al., 2024). Reference-Guided Identity Preserving Face Restoration introduces Composite Context, combining ArcFace and FaRL features, and Hard Example Identity Loss to better exploit reference faces and preserve identity (Zhou et al., 28 May 2025). RIDFR then extends this logic by introducing Alignment Learning, explicitly aiming to suppress “ID-irrelevant face semantics (e.g. pose, expression, make-up, hair style)” from identity references (Fang et al., 15 Jul 2025). Because make-up is listed among the unwanted reference attributes, this provides a direct conceptual bridge to FRR, where retouch style and personal identity must be disentangled rather than co-transferred.

InstantRestore shows a further efficiency-oriented identity pathway: a single-step diffusion model with shared-image attention over a small reference set and a landmark attention loss for semantic correspondence, achieving near real-time personalized face restoration with about four references (Zhang et al., 2024). This suggests a practical deployment route for FRR when identity-preserving reversal is needed at interactive speed.

5. Objectives, metrics, and reported performance

FRR evaluation combines pixel-level, perceptual, and identity-oriented metrics, but the relative weight of these criteria differs across papers. Re-Face reports PSNR, SSIM, FID, LPIPS, and DISTS (Zhao et al., 2024). FRRffusion adds VGG score (VGGS) and CLIP score (CLIPS), and also analyzes cosine similarity with VGG-Face, FaceNet, OpenFace, and ArcFace (Xing et al., 2024). MoFRR uses PSNR, SSIM, and cosine similarity in AdaFace and ArcFace spaces to reflect its forensic emphasis on faithful identity recovery (Liu et al., 26 Jul 2025).

Method	Setting	Representative reported result
Re-Face	Full Face2Face with ground-truth labels on the 1000-image RetouchingFFHQ validation subset	FID 23.48, SSIM 0.8556, PSNR 28.24, LPIPS 0.1677, DISTS 0.0737 (Zhao et al., 2024)
FRRffusion	deepFRR intra-dataset evaluation at 512×512	SSIM 0.884, PSNR 33.05, VGGS 0.991, CLIPS 0.973 (Xing et al., 2024)
MoFRR	RetouchingFFHQ++ intra-API, multi-operation subset	34.47 dB PSNR and SSIM 0.959; best baseline ResDiff reports 28.98 dB and 0.889 (Liu et al., 26 Jul 2025)

FRRffusion reports that on deepFRR the retouched-versus-raw baseline already has SSIM 0.876 and PSNR 31.94, but FRRffusion improves these to SSIM 0.884 and PSNR 33.05 while also raising VGGS from 0.936 to 0.991 (Xing et al., 2024). It further presents a user study with 85 subjects in which FRRffusion receives 83.27 average votes, compared with 2.09 for GP-UNIT and 0.04 for Stable Diffusion, indicating that human observers strongly preferred its de-retouched outputs as closest to the raw faces (Xing et al., 2024).

Re-Face shows a different pattern: the jump from FaceR alone to FaceR plus H-AdaIN is large, with FID dropping from about 44.83–44.97 to about 23.48–23.60, depending on whether ground-truth or detected labels are used (Zhao et al., 2024). This underlines a practical point for FRR: even when geometry is reversed correctly, diffusion color drift can dominate final quality if not corrected explicitly.

MoFRR emphasizes cross-operation and cross-API robustness. On RetouchingFFHQ++ it reports 36.65 PSNR and 0.971 SSIM on single-operation cross-API testing, and 31.28 PSNR and 0.938 SSIM on multi-operation cross-API testing, both outperforming retrained baselines (Liu et al., 26 Jul 2025). It also shows, through identity-similarity density plots, that restored images cluster much closer to the original faces than the retouched images themselves, especially in difficult multi-operation settings (Liu et al., 26 Jul 2025).

6. Limitations, open questions, and likely research directions

A central limitation is domain realism. deepFRR uses 50,000 StyleGAN-generated faces retouched by a single commercial API, which gives strong paired supervision and avoids privacy problems, but also means that the raw-image distribution is synthetic and the retouching source is narrow (Xing et al., 2024). Re-Face, by contrast, works on a large real-face retouching benchmark but only models three retouching dimensions—eye, face, and smoothing—so its label space does not yet cover whitening, makeup, eyebrow shaping, or more complex composite filters (Zhao et al., 2024). MoFRR broadens the operation and API space substantially, yet still notes performance drops under unseen retouching styles and highlights demographic bias and privacy concerns as unresolved issues (Liu et al., 26 Jul 2025).

Another unresolved issue is ambiguity. MoFRR explicitly remarks that aggressive whitening, smoothing, or reshaping can destroy information, so reconstruction becomes ill-posed and the model can only approximate the original face (Liu et al., 26 Jul 2025). This is partly why identity-centric restoration continues to attract attention in the broader restoration literature. Reward-based methods such as DiffusionReward introduce a Face Reward Model and structural consistency constraints to align restoration outputs with human preferences while preventing reward hacking; this suggests a possible FRR direction in which authenticity and human-judged naturalness are jointly optimized rather than treated as separate objectives (Wu et al., 23 May 2025). Likewise, the video restoration work IP-FVR combines reference-guided identity conditioning, cosine-similarity reward signals, and temporal blending to reduce intra-clip and inter-clip identity drift; this suggests a direct route toward temporally consistent video FRR once retouch-specific degradations are modeled (Han et al., 14 Jul 2025).

A different line of future work is physically grounded modeling. CGFR models blemishes through chromophore decomposition into melanin and haemoglobin channels and uses a Sum-of-Gaussians representation with per-chromophore gains $X$1 to generate realistic gradual retouching trajectories (Shuai et al., 2024). Although this is a forward retouching method rather than an inversion system, it suggests a physically interpretable augmentation model for FRR, especially in dermatological or blemish-specific settings where “natural” recovery paths matter.

Taken together, the literature indicates that FRR is moving from ad hoc de-makeup or blind image-translation formulations toward a dedicated restoration field with its own datasets, operation taxonomies, conditioning mechanisms, and evaluation criteria. The dominant technical themes are now explicit retouch-type modeling, low-frequency structural inversion, identity preservation, and cross-domain robustness. This suggests that future FRR systems will likely combine the explicit task formalization of MoFRR (Liu et al., 26 Jul 2025), the label and color-control strategies of Re-Face (Zhao et al., 2024), the coarse-to-fine synthesis of FRRffusion (Xing et al., 2024), and the identity-guided conditioning frameworks developed in adjacent restoration research (Tao et al., 2024, Ying et al., 2024, Fang et al., 15 Jul 2025).