BeautyGRPO: RL for Aesthetic Face Retouching

• BeautyGRPO is a reinforcement learning framework for face retouching that aligns generative editing with human aesthetic preferences and preserves facial identity.
• It leverages the FRPref-10K dataset and a specialized reward model to assess subtle differences in texture, blemish removal, and clarity.
• Dynamic Path Guidance (DPG) stabilizes the flow-matching editor by correcting stochastic drift, ensuring detailed and structurally faithful results.

BeautyGRPO is a reinforcement-learning framework for face retouching that aligns generative editing with human aesthetic preferences while preserving high visual fidelity and facial identity. It is centered on three components: FRPref-10K, a fine-grained face-retouching preference dataset; a specialized reward model trained to evaluate subtle perceptual differences; and Dynamic Path Guidance (DPG), a sampling mechanism that stabilizes GRPO-based online RL on flow-matching image editors by correcting stochastic drift during training (Yang et al., 1 Mar 2026). Within the longer history of computational beautification, BeautyGRPO differs from earlier systems that emphasized makeup transfer, latent-space traversal, or regression-guided beauty optimization, because it treats retouching as preference alignment over subtle blemish removal, texture realism, clarity, and identity preservation rather than as pixel-level label imitation or a single scalar beauty ascent (Liu et al., 2019, Zhou et al., 2020, Nguyen et al., 1 Jan 2025).

1. Conceptual scope and problem setting

BeautyGRPO addresses face retouching as a high-precision, high-subjectivity image editing problem. The task is not merely to smooth skin, but to remove subtle imperfections such as acne, blemishes, and small scars while preserving identity cues including moles, pores, natural skin texture, and facial structure, and to do so in a way that improves overall aesthetics according to human taste (Yang et al., 1 Mar 2026).

The framework is motivated by two limitations identified in prior paradigms. First, supervised retouching models optimize pixel-level reconstruction losses against retouched labels, which ties them to label mimicry rather than human preference; the result can be outputs that reproduce over-smoothing, residual blemishes, or rigid stylistic conventions present in the training set (Yang et al., 1 Mar 2026). Second, standard online RL for generative models introduces stochastic exploration noise that is useful for preference optimization but problematic for portrait editing, because accumulated stochastic drift can produce grain, identity distortions, and degradation of micro-texture over a short denoising trajectory (Yang et al., 1 Mar 2026).

This problem framing places BeautyGRPO in contrast with earlier beautification systems. "Face Beautification: Beyond Makeup Transfer" integrated style-based beauty representation and beauty score prediction to support many-to-many beautification conditioned on a reference face and a target beauty score, with explicit identity and beauty losses in a GAN translation framework (Liu et al., 2019). "GAN-Based Facial Attractiveness Enhancement" instead inverted a portrait into StyleGAN latent space and edited it along an InterFaceGAN beauty direction while preserving identity and fidelity through reconstruction losses (Zhou et al., 2020). "Regression Guided Strategy to Automated Facial Beauty Optimization through Image Synthesis" replaced hand-crafted beauty rules with a learned scalar regressor and optimized StyleGAN2 latent codes with CMA-ES under a perceptual similarity constraint (Nguyen et al., 1 Jan 2025). BeautyGRPO inherits the ambition of holistic, identity-preserving aesthetic enhancement, but relocates the optimization target from latent-space beauty direction search to online preference alignment for retouching trajectories (Yang et al., 1 Mar 2026).

2. Framework architecture and optimization target

BeautyGRPO is implemented as an online RL framework on top of a flow-matching generative editor, specifically a LoRA-adapted Flux.1-Kontext or Qwen-Image-Edit model. The generative policy $\pi_\theta$ is the flow-matching model that predicts a vector field $v_\theta(x_t,t)$ over latent image states, and the terminal reward is a scalar aesthetic score $R_\phi(I_{\text{in}}, x_0)$ assigned by the specialized reward model to the input portrait and final retouched sample (Yang et al., 1 Mar 2026).

The RL state is the latent image state $x_t$ at time $t$ along the flow trajectory, conditioned on the input portrait $I_{\text{in}}$ and, potentially, a retouching prompt. The action is the stochastic transition from $x_t$ to $x_{t-\Delta t}$. The environment is the flow integration process that maps noise $x_1$ to the final sample $x_0$. Rewards are terminal and are normalized into standardized advantages that are broadcast over the trajectory (Yang et al., 1 Mar 2026).

The system is initialized from a supervised editing backbone. Flux.1-Kontext is LoRA-adapted at resolution $v_\theta(x_t,t)$0 with LoRA rank $v_\theta(x_t,t)$1 and $v_\theta(x_t,t)$2, using AdamW, learning rate $v_\theta(x_t,t)$3, weight decay $v_\theta(x_t,t)$4, cosine decay, $v_\theta(x_t,t)$5 warmup steps, batch size $v_\theta(x_t,t)$6 per device, and $v_\theta(x_t,t)$7 gradient accumulation steps. Online RL is then run at $v_\theta(x_t,t)$8 with per-device batch size $v_\theta(x_t,t)$9, $R_\phi(I_{\text{in}}, x_0)$0 images per prompt, $R_\phi(I_{\text{in}}, x_0)$1 sampling steps, noise level $R_\phi(I_{\text{in}}, x_0)$2, and $R_\phi(I_{\text{in}}, x_0)$3 DPG-guided steps per trajectory (Yang et al., 1 Mar 2026).

The core optimization principle is not label reconstruction but reward maximization under a trust-region-like constraint. BeautyGRPO uses GRPO, a PPO-style policy-gradient method, together with standardized advantages, PPO-style clipping, and a KL-style regularization effect induced by clipping the likelihood ratio. This suggests that the framework is best understood as a preference-aligned control layer over a preexisting high-capacity editor, rather than as a from-scratch retouching model (Yang et al., 1 Mar 2026).

3. FRPref-10K and fine-grained preference modeling

FRPref-10K is a 10,000-pair high-resolution face-retouching preference dataset designed to model nuanced human judgments. It draws from the FFHQR (AutoRetouch) dataset and a proprietary high-resolution portrait collection intended to diversify demographics such as age and race. For each input portrait $R_\phi(I_{\text{in}}, x_0)$4, multiple candidate retouches are generated using RetouchFormer, NanoBanana, Flux.1-Kontext with LoRA, and different random seeds and settings; preference pairs are then constructed both as output-vs-output comparisons and output-vs-label comparisons (Yang et al., 1 Mar 2026).

The annotation space is explicitly factorized into five retouching dimensions.

Dimension	Annotation focus
Skin Smoothing	Remove roughness without producing a plastic look
Blemish Removal	Remove acne, scars, and spots without erasing identity features such as moles
Texture Quality	Preserve pores, fine wrinkles, and skin grain realistically
Clarity	Maintain sharpness without unnecessary blur or noise
Identity Preservation	Retain the same persona, facial features, shape, and moles

Annotations are produced by a hybrid VLM-human pipeline. First, GPT-4o, Qwen2.5-VL-72B, and Gemini 2.5 Pro provide dimension-wise scores, structured chain-of-thought reasoning, and an overall preference decision for each pair; these outputs are aggregated into an initial consensus. Second, trained human annotators review and correct the assessments, and disputed cases are resolved by senior experts (Yang et al., 1 Mar 2026).

The reward model $R_\phi(I_{\text{in}}, x_0)$5 is based on Qwen2.5-VL-7B-Instruct and is trained in three stages following UnifiedReward-Thinking ideas. Stage 1 performs SFT on approximately $R_\phi(I_{\text{in}}, x_0)$6K carefully curated samples, teaching the model to output a > block reasoning over the five dimensions and an <answer> block containing the preference decision. Stage 2 performs self-training on the remaining approximately $R_\phi(I_{\text{in}}, x_0)$7K pairs by generating multiple reasoning trajectories and filtering them for both preference correctness and reasoning coherence. Stage 3 uses GRPO on hard or inconsistent samples, with rewards composed of an outcome reward based on label agreement and a process reward based on reasoning coherence, computed by a verifier such as DeBERTa-V3 (Yang et al., 1 Mar 2026).

The result is a reward model intended to score subtle perceptual distinctions that general-purpose reward models miss. The paper reports that in user studies the model’s dimension-wise agreement with humans is higher than GPT-4o and Qwen2.5-VL-72B, which is used as evidence that the reward signal is sensitive to distinctions such as over-smoothed versus naturally soft skin or complete blemish removal with preserved pores (Yang et al., 1 Mar 2026).

4. GRPO objective and Dynamic Path Guidance

BeautyGRPO uses group-relative standardized advantages. For a condition $R_\phi(I_{\text{in}}, x_0)$8 and $R_\phi(I_{\text{in}}, x_0)$9 sampled outputs $x_t$0, the reward model assigns scalar rewards $x_t$1, which are normalized as

$x_t$2

The policy objective is

$x_t$3

with stepwise likelihood ratio

$x_t$4

This is the standard PPO-style clipped surrogate objective adapted to flow trajectories (Yang et al., 1 Mar 2026).

The distinctive technical contribution is DPG. In FlowGRPO, each reverse step adds Gaussian noise,

$x_t$5

which yields stochastic drift over the denoising horizon. DPG introduces a stability anchor $x_t$6, defined as a high-quality retouched exemplar for the input, selected from FRPref-10K and used only during sampling. At each timestep, DPG defines an anchor-path target

$x_t$7

computes the corresponding correction noise

$x_t$8

and mixes it with standard Gaussian exploration noise using a time-dependent coefficient $x_t$9,

$t$0

Substituting this into the update yields

$t$1

The induced transition distribution remains Gaussian, with mean $t$2 and standard deviation $t$3, which makes the stepwise likelihood computable for GRPO updates (Yang et al., 1 Mar 2026).

Early reverse steps therefore receive strong anchor guidance, stabilizing global structure and identity, while late steps are less constrained and allow fine-grained exploration of texture and micro-detail. For efficiency, DPG is applied to one randomly selected timestep in each of $t$4 trajectory segments, while the remaining steps use ODE updates; ablation shows that $t$5 performs nearly the same as $t$6 at lower cost (Yang et al., 1 Mar 2026).

A common misconception is that DPG is a form of supervised reference guidance. The method explicitly avoids that interpretation: anchors are never used in the loss, never act as pixel- or feature-level targets, and are discarded at inference time, which uses standard deterministic ODE sampling of the improved policy (Yang et al., 1 Mar 2026).

5. Experimental profile and quantitative findings

BeautyGRPO is evaluated on FFHQR, using 1,000 portraits from AutoRetouch, and on an in-the-wild set of 1,000 internet portrait images. Baselines include specialized retouching systems such as ABPN, RestoreFormer, RestoreFormer++, VRetouchEr, and RetouchFormer; general editors such as ICEdit, SeedDream 4.0, NanoBanana, and Flux.1-Kontext + LoRA; and an RL baseline consisting of FluxKontext + LoRA with FlowGRPO, i.e., the same reward model but without DPG (Yang et al., 1 Mar 2026).

The evaluation emphasizes no-reference perceptual and aesthetic metrics—NIQE, NRQM, NIMA, MUSIQ, MANIQA, and TOPIQ—together with ArcFace similarity for identity preservation and FID on FFHQR for distribution realism. The paper explicitly argues that full-reference metrics such as PSNR, SSIM, and LPIPS are inappropriate for this task because the perception-distortion tradeoff makes high PSNR compatible with aesthetically inferior over-smoothed outputs (Yang et al., 1 Mar 2026).

On FFHQR, BeautyGRPO with Flux.1-Kontext + LoRA achieves the best scores across all six no-reference metrics: NIMA $t$7, MUSIQ $t$8, MANIQA $t$9, NRQM $I_{\text{in}}$0, TOPIQ $I_{\text{in}}$1, and NIQE $I_{\text{in}}$2. Identity preservation remains high, with ArcFace similarity $I_{\text{in}}$3 on FFHQR and $I_{\text{in}}$4 on in-the-wild images. Its FID is slightly higher than that of the best supervised baseline, which the paper interprets as a consequence of RL moving beyond the label distribution toward more preferred styles (Yang et al., 1 Mar 2026).

User studies reinforce the metric profile. In a 100-participant overall-preference questionnaire, BeautyGRPO attains a win rate of $I_{\text{in}}$5, whereas other methods each remain at or below $I_{\text{in}}$6, including RetouchFormer at $I_{\text{in}}$7 and NanoBanana at $I_{\text{in}}$8. In a separate dimension-wise questionnaire, the specialized reward model shows the highest agreement with human ratings across skin smoothing, blemish removal, texture quality, clarity, and identity preservation (Yang et al., 1 Mar 2026).

Ablations isolate the contributions of reward specialization and DPG. Replacing the retouch-specific reward with generic edit rewards such as EditReward, EditScore, or UnifiedReward-Edit improves the RL-tuned model relative to the supervised baseline but remains inferior to the retouch-specific reward; on FFHQR, for example, EditReward yields NIMA $I_{\text{in}}$9, MUSIQ $x_t$0, and MANIQA $x_t$1, whereas the specialized reward yields NIMA $x_t$2, MUSIQ $x_t$3, MANIQA $x_t$4, and TOPIQ $x_t$5 (Yang et al., 1 Mar 2026). On Qwen-Image-Edit, BeautyGRPO raises NIMA from $x_t$6 in the pretrained model to $x_t$7 with LoRA and to $x_t$8 after BeautyGRPO, while MUSIQ rises from $x_t$9 to $x_{t-\Delta t}$0 to $x_{t-\Delta t}$1 (Yang et al., 1 Mar 2026).

Qualitatively, the paper reports that supervised retouchers often mis-detect blemishes and over-smooth skin, general editors may alter eye shape or facial structure and create glossy plastic skin, and FlowGRPO introduces visible noise artifacts due to stochastic drift. BeautyGRPO is described as removing acne and blemishes while preserving moles, pores, fine wrinkles, natural gloss, skin shading, and facial identity (Yang et al., 1 Mar 2026).

6. Broader significance, limitations, and relation to adjacent research

BeautyGRPO can be read as a convergence point between facial beautification research and GRPO-based visual alignment. Earlier beautification systems established several core motifs: reference-conditioned many-to-many beautification with explicit beauty prediction and AdaIN-based style control (Liu et al., 2019), latent-space attractiveness enhancement via StyleGAN inversion and InterFaceGAN beauty directions (Zhou et al., 2020), and regression-guided optimization in StyleGAN2 $x_{t-\Delta t}$2 using a FaceNet-ensemble beauty regressor and CMA-ES (Nguyen et al., 1 Jan 2025). BeautyGRPO retains the emphasis on identity-preserving aesthetic improvement but changes the optimization substrate from latent editing to reward-aligned policy optimization over flow trajectories (Yang et al., 1 Mar 2026).

Its GRPO component also belongs to a broader family of post-2025 visual RL methods. "Fine-Tuning Next-Scale Visual Autoregressive Models with Group Relative Policy Optimization" showed that GRPO can optimize next-scale VAR models using LAION Aesthetic Predictor V2 and CLIP rewards, increasing AES from $x_{t-\Delta t}$3 to $x_{t-\Delta t}$4 on VAR-d16 and from $x_{t-\Delta t}$5 to $x_{t-\Delta t}$6 on VAR-d30, while also enabling style transfer toward prompts such as “a painting” beyond the original ImageNet distribution (Gallici et al., 29 May 2025). "Image Aesthetic Reasoning via HCM-GRPO" adapted GRPO to multi-image aesthetic screening and reached $x_{t-\Delta t}$7 with a compact 2B model, far above its carefully computed random baseline of $x_{t-\Delta t}$8, by combining dense partial-credit reward with hard-case mining (Hu et al., 13 Nov 2025). "GRPO++" modified grouped relative policy optimization for low-resource dermatological reasoning by introducing a confidence-aware penalty regime when all sampled outputs are bad, in order to avoid gradient vanishing and error reinforcement (Swapnil et al., 23 Sep 2025). These results suggest a broader methodological pattern: GRPO variants become especially attractive when the target criterion is perceptual, structured, or weakly supervised, but the exact reward geometry and exploration control must be domain-specific.

BeautyGRPO’s own limitations are explicit. DPG currently assumes access to at least one high-quality retouched exemplar per input for use as an anchor during training. FRPref-10K, although more diverse than FFHQR alone, still reflects the preferences of a specific annotator and VLM configuration, so cultural or demographic biases may be encoded in what counts as good retouching. Online RL at high resolution is computationally intensive; the reported experiments use $x_{t-\Delta t}$9 H20 GPUs. The framework is also specialized to face retouching; extending the same recipe to other domains would require new datasets and new reward models (Yang et al., 1 Mar 2026).

A plausible implication is that BeautyGRPO should be understood less as a universal theory of facial beauty than as a domain-specific aesthetic alignment system whose notion of quality is operationalized through FRPref-10K and the learned reward model. That reading is consistent with both its empirical strengths—superior texture quality, controlled blemish removal, and strong human preference alignment—and its central caution: whichever data and annotators define the reward will also define the system’s effective standard of aesthetic improvement (Yang et al., 1 Mar 2026).