Reinforcement Learning from Aesthetic Feedback

• RL-AF is a machine learning framework that formulates visual and multimodal tasks as MDPs, using learned reward models to capture subjective aesthetics.
• It leverages architectures like ResNet-based estimators, vision-language transformers, and adversarial discriminators, stabilized by techniques such as GRPO and RAPO.
• Empirical findings in photo capture, face retouching, and design tasks demonstrate improved human preference ratings and robust performance under diverse conditions.

Reinforcement Learning from Aesthetic Feedback (RL-AF) is a class of machine learning methodologies in which agents optimize controllable actions in image, vision-language, or multimodal generation environments by receiving feedback—scalar, vectorial, or structured—quantifying the aesthetic quality of generated outputs. Unlike traditional perceptual optimization regimes built on hand-crafted criteria or rigid heuristic metrics, RL-AF leverages learned reward models, human (or proxy) preference signals, and sometimes adversarially-updated discriminators to align output distributions with complex, subjective aesthetic values. RL-AF frameworks are used for tasks including photo capture, image enhancement, generative modeling, personalized aesthetic assessment, face retouching, and visual design, among others.

1. Fundamental Problem Formulations in RL-AF

RL-AF methods universally encode the downstream task as a Markov Decision Process (MDP) or, equivalently, a (Partially Observable) Markov Decision Process (POMDP), parameterized as follows:

• State ($s_t$): Typically the current visual or multimodal observation (e.g., camera view, image composite, generator latent, slide), often augmented with task or exploration memory (e.g., trajectory history via LSTM).
• Action ($a_t$): Executed controls (physical movement, retouch parameters, token generation, etc.).
• Transition ($P(s_{t+1}|s_t,a_t)$): Deterministic or stochastic, depending on task environment (e.g., AI Habitat simulator (Alzayer et al., 2021), sequential photo collage editing (Zhang et al., 2021), autoregressive or SDE-based image synthesis (Mao et al., 25 Nov 2025, Yang et al., 1 Mar 2026)).
• Reward ($r(s,a)$): Derived from the change in (or absolute) value of an aesthetic estimator, preference model, adversarial discriminator, or vectorial attribute agreement.

Each RL-AF formulation is task-specific. For instance:

• In AutoPhoto (Alzayer et al., 2021), the agent navigates scenes, maximizing a learned $\phi(s)$ score of photo aesthetics, with per-step rewards $$r(s_t,a) = [\phi(s_{t+1})-\phi(s_t)] + 0.1\Gamma(\zeta) - \beta t$$ (with a terminal $+1/-1$ reward on CAPTURE if $\phi(s_T)$ exceeds a local threshold).
• In Aesthetic Photo Collage (Zhang et al., 2021), immediate rewards combine subjective proposal counts and blank space penalties, $r_t = [s(C_{t+1})-s(C_t)] - 0.01\cdot(t+1)$.
• In diffusion-based image generation (Mao et al., 25 Nov 2025, Yang et al., 1 Mar 2026), rewards hinge on discriminator outputs or explicit human preference models, sometimes integrated into dense visual rewards (foundation model embeddings) rather than a scalar.

The core challenge is defining and learning $\phi$ or $a_t$0 that both reflects human aesthetics and provides stable, informative gradients for RL optimization.

2. Architectures and Reward Model Training

Aesthetic reward estimators underpin RL-AF. Their design, training, and calibration critically affect agent behavior:

• Image-based Estimators: Typical architecture involves ResNet-18 backbones (with anti-aliasing kernels for composition invariance), global-average-pooling, and aesthetic heads yielding scalar outputs $a_t$1 (Alzayer et al., 2021).
• Multi-task VL Transformers: For generalizable feedback across design, defect adjustment, and reasoning, vision-language transformers such as Qwen-2.5-VL-7B are used, fine-tuned on multimodal datasets (e.g., slides with labeled defects for presentation design (Liu et al., 7 Oct 2025), or image–rationale pairs for Aes-R1 (Liu et al., 26 Sep 2025)).
• Preference-based Discriminators: In adversarial RL-AF, discriminators $a_t$2 are trained on reference distributions and generator samples via binary cross-entropy or Bradley–Terry/logistic losses, often using vision foundation model features (e.g., DINOv2) for dense, high-dimensional aesthetic feedback (Mao et al., 25 Nov 2025).
• Personalized Feedback: User-guided RL-AF integrates online retouch actions or manual rankings as immediate rewards—using measures like SSIM or normalized Spearman-ρ for ranking agreement (Lv et al., 2021).

Losses combine ranking, regression, and robustness penalties. For example: $a_t$3 Crucially, no post-normalization is applied; raw scores directly define reward signals.

3. Group-Relative and Relative-Absolute Policy Optimization

Standard RL algorithms are adapted to handle the unique properties of aesthetic feedback:

• Group-Relative Policy Optimization (GRPO): Key in modern RL-AF, GRPO normalizes reward signals across a sampled group, stabilizing training and reflecting comparative human judgments (Liu et al., 7 Oct 2025, Mao et al., 25 Nov 2025, Yang et al., 1 Mar 2026). The advantage for a group is computed as

$a_t$4

with clipped objective and (optionally) a KL-divergence penalty to a reference policy.

• Relative-Absolute Policy Optimization (RAPO): Extends PPO with a dual reward: continuous regression for per-instance error plus probabilistic pairwise ranking for ordinal consistency. The overall reward is $a_t$5, combining deviation from ground-truth and ranking accuracy. This approach achieves improved PLCC/SRCC and generalization under limited supervision (Liu et al., 26 Sep 2025).
• Direct Preference Optimization (DPO): Used for reward model refinement, optimizing for outcome reward (pairwise preference) and process reward (coherent reasoning), as in BeautyGRPO (Yang et al., 1 Mar 2026).

Exploration/exploitation is handled by intrinsic bonuses (e.g., decaying exploration terms in (Alzayer et al., 2021)), attention-based action selection (e.g., collage arrangement (Zhang et al., 2021)), or anchor guidance in flow-matching generative models (Yang et al., 1 Mar 2026).

4. Task-Specific RL-AF Pipelines and Application Domains

RL-AF frameworks have been adapted for heterogeneous visual and multimodal tasks, as summarized below:

Application	RL-AF Approach / Reward Source	Key Result/Metric
Autonomous photo capture	PPO w/ ResNet-18 φ(s) trained on CPC/AVA; reward = Δφ(s) + bonus	81.7% Gibson success rate (Alzayer et al., 2021)
Face retouching	Flow-matching, DPG-guided GRPO; fine-grained human prefs. (FRPref)	63.3% user win rate (Yang et al., 1 Mar 2026)
Photo collage arrangement	A2C; reward blends proposal count and blank space penalty	110.6 score (Hollywood2), user study top
Personalized aesthetic enhancement/rank	Actor–Critic (enhance), Double DQN (rank); SSIM/ρ feedback	SSIM ~0.67, ρ = 0.692 (Lv et al., 2021)
Presentation slide generation/design	GRPO on multi-task slide benchmark; tag/format/accuracy rewards	F1=0.389/Acc=87.8%/MAE=1.33 (Liu et al., 7 Oct 2025)
Multimodal aesthetic reasoning	RAPO in MLLMs; reward = absolute+relative, with CoT explanations	PLCC↑47.9%, OOD strong (Liu et al., 26 Sep 2025)
Diffusion image generation	Adv-GRPO; adversarial reward from ref. images + foundation models	Human: 70.0–75.2% win (Mao et al., 25 Nov 2025)

In all cases, the RL-AF agent is coupled to a reward estimator calibrated on preference data or high-quality exemplars, and the learning pipeline is built around parallel rollouts, batch-wise policy optimization, and regular reevaluation against reference distributions.

5. Methodological Advancements: From Scalar Rewards to Dense and Structure-Preserving Feedback

A defining trend in RL-AF literature is the progression from scalar, sometimes hackable aesthetic or preference proxies to richer, structure-preserving feedback:

• Scalar vs. Dense Visual Rewards: Early RL-AF used scalar outputs from learned networks as direct reward; however, such models are vulnerable to reward hacking (i.e., generation of high-scoring but poor images) (Mao et al., 25 Nov 2025). By incorporating high-dimensional foundation model features (e.g., DINO embeddings), and weighting global ([CLS]) and local (patch) components, RL-AF algorithms now leverage dense signals to guide synthesis, improving both quality and alignment.
• Preference Model Robustness: Adversarial training (alternate generator/discriminator updates) prevents the static reward model from being exploited by the generator (Mao et al., 25 Nov 2025). Human-in-the-loop or multi-VLM scoring increases robustness and reliability for subjective editing tasks (Yang et al., 1 Mar 2026).
• Fidelity-Exploration Tradeoff: For high-fidelity domains (e.g., face retouching), naive RL induces stochastic drift/artifacting. Dynamic Path Guidance (DPG) corrects the exploration trajectory via anchor ODE paths sampled from high-preference examples, stabilizing synthesis while preserving diversity (Yang et al., 1 Mar 2026).

6. Practical Evaluation and Empirical Findings

Quantitative and qualitative assessment of RL-AF systems involve both automatic aesthetic metrics (PLCC, SRCC, NIMA, MUSIQ, FID, etc.) and large-scale human studies (pairwise preferences, AMT ratings, subjective win rates).

Key results:

• Photo capture: AutoPhoto achieves 81.7% success rate (φ(s_T)>τ_aes), and a 0.63 human preference score over initial views (Alzayer et al., 2021).
• Aesthetic reasoning: Aes-R1 attains +47.9% PLCC and +34.8% SRCC over baselines, and strong out-of-domain performance (Liu et al., 26 Sep 2025).
• Face retouching: BeautyGRPO yields a 63.3% win rate in 5-way human comparisons, outperforming all specialized and generic editors (Yang et al., 1 Mar 2026).
• Image generation: Adv-GRPO improves aesthetics by +70% win vs. state-of-the-art and achieves persistent gains even in OOD style transfer when using foundation-model rewards (Mao et al., 25 Nov 2025).

Ablation studies across works show that the exclusion of core RL-AF components (e.g., LSTM memory, attention fusion, exploration bonuses, DPG) measurably degrades final outcomes.

7. Generalization, Limitations, and Future Directions

RL-AF has shown effective transferability to diverse domains: face retouching, color grading, HDR, slide design, and beyond. The central requirements are (1) a reliable aesthetic or preference reward (learned from data contemporaneous with agent training), (2) a control policy adaptable to complex, often sequential, actions, and (3) a design that limits reward hacking and overfitting to imperfect metrics. Dynamic guidance and architecture innovations underpin recent advances for high-fidelity and subjectivity-sensitive tasks.

Limitations include the cost and coverage of preference annotation, sensitivity to reward misspecification, and, in high-resolution settings, scaling challenges for backbone models and policy optimization. The adversarial and foundation-model reward paradigms mitigate but do not remove these risks.

This suggests that the trajectory of RL-AF is toward finer-grained, multi-dimensional, and personalized aesthetic alignment, leveraging ongoing advances in reward model robustness, foundation-model representations, and optimization theory. Continued progress will likely depend on larger, higher-quality preference datasets and integration of richer human-in-the-loop signals.

---

References:

• (Alzayer et al., 2021) AutoPhoto: Aesthetic Photo Capture using Reinforcement Learning
• (Zhang et al., 2021) Aesthetic Photo Collage with Deep Reinforcement Learning
• (Lv et al., 2021) User-Guided Personalized Image Aesthetic Assessment based on Deep Reinforcement Learning
• (Liu et al., 7 Oct 2025) Presenting a Paper is an Art: Self-Improvement Aesthetic Agents for Academic Presentations
• (Liu et al., 26 Sep 2025) Unlocking the Essence of Beauty: Advanced Aesthetic Reasoning with Relative-Absolute Policy Optimization
• (Mao et al., 25 Nov 2025) The Image as Its Own Reward: Reinforcement Learning with Adversarial Reward for Image Generation
• (Yang et al., 1 Mar 2026) BeautyGRPO: Aesthetic Alignment for Face Retouching via Dynamic Path Guidance and Fine-Grained Preference Modeling