---
title: 'BeautyGRPO: RL for Aesthetic Face Retouching'
url: https://www.emergentmind.com/topics/beautygrpo
type: topic
---

# BeautyGRPO: RL for Aesthetic Face Retouching

BeautyGRPO is a reinforcement-learning framework for face retouching that aligns generative editing with human aesthetic preferences while preserving high visual fidelity and facial identity. It is centered on three components: FRPref-10K, a fine-grained face-retouching preference dataset; a specialized reward model trained to evaluate subtle perceptual differences; and Dynamic Path Guidance (DPG), a sampling mechanism that stabilizes GRPO-based online RL on flow-matching image editors by correcting stochastic drift during training [2603.01163]. Within the longer history of computational beautification, BeautyGRPO differs from earlier systems that emphasized makeup transfer, latent-space traversal, or regression-guided beauty optimization, because it treats retouching as preference alignment over subtle blemish removal, texture realism, clarity, and identity preservation rather than as pixel-level label imitation or a single scalar beauty ascent [1912.03630][2006.02766][2501.00811].

## 1. Conceptual scope and problem setting

BeautyGRPO addresses face retouching as a high-precision, high-subjectivity image editing problem. The task is not merely to smooth skin, but to remove subtle imperfections such as acne, blemishes, and small scars while preserving identity cues including moles, pores, natural skin texture, and facial structure, and to do so in a way that improves overall aesthetics according to human taste [2603.01163].

The framework is motivated by two limitations identified in prior paradigms. First, supervised retouching models optimize pixel-level reconstruction losses against retouched labels, which ties them to label mimicry rather than human preference; the result can be outputs that reproduce over-smoothing, residual blemishes, or rigid stylistic conventions present in the training set [2603.01163]. Second, standard online RL for generative models introduces stochastic exploration noise that is useful for preference optimization but problematic for portrait editing, because accumulated stochastic drift can produce grain, identity distortions, and degradation of micro-texture over a short denoising trajectory [2603.01163].

This problem framing places BeautyGRPO in contrast with earlier beautification systems. "Face Beautification: Beyond Makeup Transfer" integrated style-based beauty representation and beauty score prediction to support many-to-many beautification conditioned on a reference face and a target beauty score, with explicit identity and beauty losses in a GAN translation framework [1912.03630]. "GAN-Based Facial Attractiveness Enhancement" instead inverted a portrait into StyleGAN latent space and edited it along an InterFaceGAN beauty direction while preserving identity and fidelity through reconstruction losses [2006.02766]. "Regression Guided Strategy to Automated Facial Beauty Optimization through Image Synthesis" replaced hand-crafted beauty rules with a learned scalar regressor and optimized StyleGAN2 latent codes with CMA-ES under a perceptual similarity constraint [2501.00811]. BeautyGRPO inherits the ambition of holistic, identity-preserving aesthetic enhancement, but relocates the optimization target from latent-space beauty direction search to online preference alignment for retouching trajectories [2603.01163].

## 2. Framework architecture and optimization target

BeautyGRPO is implemented as an online RL framework on top of a flow-matching generative editor, specifically a LoRA-adapted Flux.1-Kontext or Qwen-Image-Edit model. The generative policy $\pi_\theta$ is the flow-matching model that predicts a vector field $v_\theta(x_t,t)$ over latent image states, and the terminal reward is a scalar aesthetic score $R_\phi(I_{\text{in}}, x_0)$ assigned by the specialized reward model to the input portrait and final retouched sample [2603.01163].

The RL state is the latent image state $x_t$ at time $t$ along the flow trajectory, conditioned on the input portrait $I_{\text{in}}$ and, potentially, a retouching prompt. The action is the stochastic transition from $x_t$ to $x_{t-\Delta t}$. The environment is the flow integration process that maps noise $x_1$ to the final sample $x_0$. Rewards are terminal and are normalized into standardized advantages that are broadcast over the trajectory [2603.01163].

The system is initialized from a supervised editing backbone. Flux.1-Kontext is LoRA-adapted at resolution $1024 \times 1024$ with LoRA rank $32$ and $\alpha = 32$, using AdamW, learning rate $1 \times 10^{-5}$, weight decay $0.01$, cosine decay, $500$ warmup steps, batch size $4$ per device, and $2$ gradient accumulation steps. Online RL is then run at $1024 \times 1024$ with per-device batch size $2$, $8$ images per prompt, $T = 10$ sampling steps, noise level $\eta = 0.7$, and $K = 3$ DPG-guided steps per trajectory [2603.01163].

The core optimization principle is not label reconstruction but reward maximization under a trust-region-like constraint. BeautyGRPO uses GRPO, a PPO-style policy-gradient method, together with standardized advantages, PPO-style clipping, and a KL-style regularization effect induced by clipping the likelihood ratio. This suggests that the framework is best understood as a preference-aligned control layer over a preexisting high-capacity editor, rather than as a from-scratch retouching model [2603.01163].

## 3. FRPref-10K and fine-grained preference modeling

FRPref-10K is a 10,000-pair high-resolution face-retouching preference dataset designed to model nuanced human judgments. It draws from the FFHQR (AutoRetouch) dataset and a proprietary high-resolution portrait collection intended to diversify demographics such as age and race. For each input portrait $I_{\text{in}}$, multiple candidate retouches are generated using RetouchFormer, NanoBanana, Flux.1-Kontext with LoRA, and different random seeds and settings; preference pairs are then constructed both as output-vs-output comparisons and output-vs-label comparisons [2603.01163].

The annotation space is explicitly factorized into five retouching dimensions.

| Dimension | Annotation focus |
|---|---|
| Skin Smoothing | Remove roughness without producing a plastic look |
| Blemish Removal | Remove acne, scars, and spots without erasing identity features such as moles |
| Texture Quality | Preserve pores, fine wrinkles, and skin grain realistically |
| Clarity | Maintain sharpness without unnecessary blur or noise |
| Identity Preservation | Retain the same persona, facial features, shape, and moles |

Annotations are produced by a hybrid VLM-human pipeline. First, GPT-4o, Qwen2.5-VL-72B, and Gemini 2.5 Pro provide dimension-wise scores, structured chain-of-thought reasoning, and an overall preference decision for each pair; these outputs are aggregated into an initial consensus. Second, trained human annotators review and correct the assessments, and disputed cases are resolved by senior experts [2603.01163].

The reward model $R_\phi$ is based on Qwen2.5-VL-7B-Instruct and is trained in three stages following UnifiedReward-Thinking ideas. Stage 1 performs SFT on approximately $2$K carefully curated samples, teaching the model to output a `<think>` block reasoning over the five dimensions and an `<answer>` block containing the preference decision. Stage 2 performs self-training on the remaining approximately $8$K pairs by generating multiple reasoning trajectories and filtering them for both preference correctness and reasoning coherence. Stage 3 uses GRPO on hard or inconsistent samples, with rewards composed of an outcome reward based on label agreement and a process reward based on reasoning coherence, computed by a verifier such as DeBERTa-V3 [2603.01163].

The result is a reward model intended to score subtle perceptual distinctions that general-purpose reward models miss. The paper reports that in user studies the model’s dimension-wise agreement with humans is higher than GPT-4o and Qwen2.5-VL-72B, which is used as evidence that the reward signal is sensitive to distinctions such as over-smoothed versus naturally soft skin or complete blemish removal with preserved pores [2603.01163].

## 4. GRPO objective and Dynamic Path Guidance

BeautyGRPO uses group-relative standardized advantages. For a condition $c$ and $G$ sampled outputs $\{x_0^i\}_{i=1}^G$, the reward model assigns scalar rewards $R(x_0^i)$, which are normalized as

$$
\hat{A}^i = \frac{R(x_0^i) - \text{mean}(\{R(x_0^j)\})}{\text{std}(\{R(x_0^j)\})}.
$$

The policy objective is

$$
\mathcal{L}(\theta) =
\mathbb{E}_{c \sim \mathcal{C}, \{x^i\}\sim\pi_{\theta_{\text{old}}}}
\left[
\frac{1}{G}\sum_{i=1}^{G}\frac{1}{T}\sum_{t=0}^{T-1}
\min\big(r_t^i(\theta)\hat{A}_t^i,\; \text{clip}(r_t^i(\theta), 1-\varepsilon, 1+\varepsilon)\hat{A}_t^i\big)
\right],
$$

with stepwise likelihood ratio

$$
r_t^i(\theta) = \frac{p_\theta(x_{t-\Delta t}^i|x_t^i)}{p_{\theta_{\text{old}}}(x_{t-\Delta t}^i|x_t^i)}.
$$

This is the standard PPO-style clipped surrogate objective adapted to flow trajectories [2603.01163].

The distinctive technical contribution is DPG. In FlowGRPO, each reverse step adds Gaussian noise,

$$
x_{t-\Delta t} = \mu_t + \sigma_{\text{step}} z_t,\qquad z_t \sim \mathcal{N}(0,I),
$$

which yields stochastic drift over the denoising horizon. DPG introduces a stability anchor $x_0^{\text{anchor}}$, defined as a high-quality retouched exemplar for the input, selected from FRPref-10K and used only during sampling. At each timestep, DPG defines an anchor-path target

$$
x^{*}_{t-\Delta t} = \Big(\frac{\Delta t}{t}\Big)\,x_0^{\text{anchor}} + \Big(1-\frac{\Delta t}{t}\Big)\,x_t,
$$

computes the corresponding correction noise

$$
z^{\text{anchor}}_t = \frac{x^{*}_{t-\Delta t} - \mu_t}{\sigma_{\text{step}}},
$$

and mixes it with standard Gaussian exploration noise using a time-dependent coefficient $\lambda(t)$,

$$
z^{\text{mix}}_t = \lambda(t)\,z^{\text{anchor}}_t + (1-\lambda(t))\,z^{\text{std}}_t,\qquad
\lambda(t) = \frac{t}{\max(1, T-1)}.
$$

Substituting this into the update yields

$$
x^{\text{guided}}_{t-\Delta t}
= (1-\lambda)\,\mu_t + \lambda\,x^{*}_{t-\Delta t}
+ (1-\lambda)\,\sigma_{\text{step}}\,z_t^{\text{std}}.
$$

The induced transition distribution remains Gaussian, with mean
$\mu_{\text{new}} = (1-\lambda)\mu_t + \lambda x_{t-\Delta t}^{*}$ and standard deviation $\sigma_{\text{new}} = (1-\lambda)\sigma_{\text{step}}$, which makes the stepwise likelihood computable for GRPO updates [2603.01163].

Early reverse steps therefore receive strong anchor guidance, stabilizing global structure and identity, while late steps are less constrained and allow fine-grained exploration of texture and micro-detail. For efficiency, DPG is applied to one randomly selected timestep in each of $K=3$ trajectory segments, while the remaining steps use ODE updates; ablation shows that $K=3$ performs nearly the same as $K=5$ at lower cost [2603.01163].

A common misconception is that DPG is a form of supervised reference guidance. The method explicitly avoids that interpretation: anchors are never used in the loss, never act as pixel- or feature-level targets, and are discarded at inference time, which uses standard deterministic ODE sampling of the improved policy [2603.01163].

## 5. Experimental profile and quantitative findings

BeautyGRPO is evaluated on FFHQR, using 1,000 portraits from AutoRetouch, and on an in-the-wild set of 1,000 internet portrait images. Baselines include specialized retouching systems such as ABPN, RestoreFormer, RestoreFormer++, VRetouchEr, and RetouchFormer; general editors such as ICEdit, SeedDream 4.0, NanoBanana, and Flux.1-Kontext + LoRA; and an RL baseline consisting of FluxKontext + LoRA with FlowGRPO, i.e., the same reward model but without DPG [2603.01163].

The evaluation emphasizes no-reference perceptual and aesthetic metrics—NIQE, NRQM, NIMA, MUSIQ, MANIQA, and TOPIQ—together with ArcFace similarity for identity preservation and FID on FFHQR for distribution realism. The paper explicitly argues that full-reference metrics such as PSNR, SSIM, and LPIPS are inappropriate for this task because the perception-distortion tradeoff makes high PSNR compatible with aesthetically inferior over-smoothed outputs [2603.01163].

On FFHQR, BeautyGRPO with Flux.1-Kontext + LoRA achieves the best scores across all six no-reference metrics: NIMA $5.123$, MUSIQ $4.906$, MANIQA $1.079$, NRQM $8.401$, TOPIQ $0.676$, and NIQE $10.831$. Identity preservation remains high, with ArcFace similarity $0.952$ on FFHQR and $0.944$ on in-the-wild images. Its FID is slightly higher than that of the best supervised baseline, which the paper interprets as a consequence of RL moving beyond the label distribution toward more preferred styles [2603.01163].

User studies reinforce the metric profile. In a 100-participant overall-preference questionnaire, BeautyGRPO attains a win rate of $63.25\%$, whereas other methods each remain at or below $12\%$, including RetouchFormer at $8.5\%$ and NanoBanana at $9.75\%$. In a separate dimension-wise questionnaire, the specialized reward model shows the highest agreement with human ratings across skin smoothing, blemish removal, texture quality, clarity, and identity preservation [2603.01163].

Ablations isolate the contributions of reward specialization and DPG. Replacing the retouch-specific reward with generic edit rewards such as EditReward, EditScore, or UnifiedReward-Edit improves the RL-tuned model relative to the supervised baseline but remains inferior to the retouch-specific reward; on FFHQR, for example, EditReward yields NIMA $4.754$, MUSIQ $4.462$, and MANIQA $1.028$, whereas the specialized reward yields NIMA $5.123$, MUSIQ $4.906$, MANIQA $1.079$, and TOPIQ $0.676$ [2603.01163]. On Qwen-Image-Edit, BeautyGRPO raises NIMA from $4.571$ in the pretrained model to $4.824$ with LoRA and to $5.351$ after BeautyGRPO, while MUSIQ rises from $4.535$ to $4.745$ to $5.133$ [2603.01163].

Qualitatively, the paper reports that supervised retouchers often mis-detect blemishes and over-smooth skin, general editors may alter eye shape or facial structure and create glossy plastic skin, and FlowGRPO introduces visible noise artifacts due to stochastic drift. BeautyGRPO is described as removing acne and blemishes while preserving moles, pores, fine wrinkles, natural gloss, skin shading, and facial identity [2603.01163].

## 6. Broader significance, limitations, and relation to adjacent research

BeautyGRPO can be read as a convergence point between facial beautification research and GRPO-based visual alignment. Earlier beautification systems established several core motifs: reference-conditioned many-to-many beautification with explicit beauty prediction and AdaIN-based style control [1912.03630], latent-space attractiveness enhancement via StyleGAN inversion and InterFaceGAN beauty directions [2006.02766], and regression-guided optimization in StyleGAN2 $\mathcal{W}^{+}$ using a FaceNet-ensemble beauty regressor and CMA-ES [2501.00811]. BeautyGRPO retains the emphasis on identity-preserving aesthetic improvement but changes the optimization substrate from latent editing to reward-aligned policy optimization over flow trajectories [2603.01163].

Its GRPO component also belongs to a broader family of post-2025 visual RL methods. "Fine-Tuning Next-Scale Visual Autoregressive Models with Group Relative Policy Optimization" showed that GRPO can optimize next-scale VAR models using LAION Aesthetic Predictor V2 and CLIP rewards, increasing AES from $4.64$ to $5.55$ on VAR-d16 and from $4.80$ to $5.80$ on VAR-d30, while also enabling style transfer toward prompts such as “a painting” beyond the original ImageNet distribution [2505.23331]. "Image Aesthetic Reasoning via HCM-GRPO" adapted GRPO to multi-image aesthetic screening and reached $64.74$ with a compact 2B model, far above its carefully computed random baseline of $34.27$, by combining dense partial-credit reward with hard-case mining [2511.10055]. "GRPO++" modified grouped relative policy optimization for low-resource dermatological reasoning by introducing a confidence-aware penalty regime when all sampled outputs are bad, in order to avoid gradient vanishing and error reinforcement [2510.01236]. These results suggest a broader methodological pattern: GRPO variants become especially attractive when the target criterion is perceptual, structured, or weakly supervised, but the exact reward geometry and exploration control must be domain-specific.

BeautyGRPO’s own limitations are explicit. DPG currently assumes access to at least one high-quality retouched exemplar per input for use as an anchor during training. FRPref-10K, although more diverse than FFHQR alone, still reflects the preferences of a specific annotator and VLM configuration, so cultural or demographic biases may be encoded in what counts as good retouching. Online RL at high resolution is computationally intensive; the reported experiments use $8$ H20 GPUs. The framework is also specialized to face retouching; extending the same recipe to other domains would require new datasets and new reward models [2603.01163].

A plausible implication is that BeautyGRPO should be understood less as a universal theory of facial beauty than as a domain-specific aesthetic alignment system whose notion of quality is operationalized through FRPref-10K and the learned reward model. That reading is consistent with both its empirical strengths—superior texture quality, controlled blemish removal, and strong human preference alignment—and its central caution: whichever data and annotators define the reward will also define the system’s effective standard of aesthetic improvement [2603.01163].

Source: https://www.emergentmind.com/topics/beautygrpo