Relative-Absolute Policy Optimization (RAPO)

• Relative-Absolute Policy Optimization (RAPO) is a reinforcement learning method that integrates both relative ranking and absolute score calibration to evaluate image aesthetics.
• It employs a dual reward system—using a FRank-based relative reward and a Gaussian absolute-error reward—to enhance both ordinal correctness and score precision in multimodal models.
• Experimental results show that RAPO improves PLCC and SRCC metrics on datasets like AVA and TAD66K, demonstrating robust performance and out-of-distribution generalization.

Searching arXiv for the RAPO paper and closely related aesthetic-reasoning work. Relative-Absolute Policy Optimization (RAPO) is the reinforcement learning core of the Aes-R1 framework for multimodal aesthetic reasoning. In this formulation, RAPO jointly optimizes two complementary facets of human aesthetic judgment: an absolute dimension that calibrates predicted scores to ground-truth mean opinion scores (MOSs), and a relative dimension that enforces correct ordinal relationships across images. The method is designed for multimodal LLMs (MLLMs) that generate both a structured aesthetic critique and a final scalar score for an image, and it is positioned as a remedy for the limitations of pure regression and rank-only optimization in image aesthetic assessment (Liu et al., 26 Sep 2025).

1. Motivation and problem setting

RAPO is introduced for image aesthetic assessment under two constraints emphasized by Aes-R1: the scarcity of multimodal aesthetic reasoning data and the inherently subjective nature of aesthetic judgment. The paper argues that “directly applying RL to IAA fails to inspire reasoning patterns without prior aesthetic knowledge” and that “modeling an appropriate reward proxy to evaluate aesthetic score is intricate,” because aesthetics lacks unified criteria. Within this setting, the target model is a pretrained MLLM with parameters $\theta$ and policy $\pi_\theta(\cdot \mid \mathcal{I}, \mathcal{P})$, which produces a structured critique $c$ and a final score $s$ for image $\mathcal{I}$ under prompt $\mathcal{P}$ (Liu et al., 26 Sep 2025).

The central motivation is that human aesthetic judgment is expressed both as direct scoring and as comparative preference. Pure regression, in the paper’s characterization, restricts models to implicit reasoning and can overfit curated datasets, reducing entropy and hindering further optimization. Rank-only RL, exemplified in the paper by comparison to rank-based methods, can separate high- and low-quality images but “fails to calibrate scores,” which depresses PLCC. RAPO is therefore constructed to “comprise two complementary facets: a relative dimension that ranks images by comparative appeal and an absolute dimension that assesses intrinsic aesthetic merit.” This suggests that the algorithm is not merely a reward engineering variant, but an attempt to align optimization with the dual structure of the evaluation problem itself (Liu et al., 26 Sep 2025).

Formally, the model generates a trajectory

$$\tau = (c, s) \sim \pi_{\theta}(\cdot \mid \mathcal{I}, \mathcal{P}),$$

and is optimized to maximize expected reward:

$$\mathcal{J}(\theta)=\mathbb{E}_{\tau \sim\pi_{\theta}(\cdot\mid\mathcal{I},\mathcal{P})}\left[R(\tau)\right].$$

The policy-gradient expression used in the paper is

$$\nabla_{\theta} \mathcal{J}(\theta)= \mathbb{E}_{\tau \sim \pi_{\theta}(\cdot\mid\mathcal{I},\mathcal{P})}\left[ R(\tau)\sum_{t=1}^{T}\nabla_{\theta}\log \pi_{\theta}\left(a_t \mid s_t\right)\right],$$

where $a_t \in \mathcal{V}$ and $\pi_\theta(\cdot \mid \mathcal{I}, \mathcal{P})$0 (Liu et al., 26 Sep 2025).

2. Dual reward design: relative ranking and absolute calibration

RAPO uses group sampling and defines two rewards for each generated output $\pi_\theta(\cdot \mid \mathcal{I}, \mathcal{P})$1, the $\pi_\theta(\cdot \mid \mathcal{I}, \mathcal{P})$2-th output for image $\pi_\theta(\cdot \mid \mathcal{I}, \mathcal{P})$3: a FRank-based pairwise relative reward and a continuous absolute-error reward. The relative component begins from the assumption that an image’s aesthetic score follows a Gaussian $\pi_\theta(\cdot \mid \mathcal{I}, \mathcal{P})$4, so that the score difference $\pi_\theta(\cdot \mid \mathcal{I}, \mathcal{P})$5 follows $\pi_\theta(\cdot \mid \mathcal{I}, \mathcal{P})$6. For output $\pi_\theta(\cdot \mid \mathcal{I}, \mathcal{P})$7, the pairwise comparison probability against image $\pi_\theta(\cdot \mid \mathcal{I}, \mathcal{P})$8 is

$\pi_\theta(\cdot \mid \mathcal{I}, \mathcal{P})$9

where $c$0 is the standard normal CDF and $c$1 prevents division-by-zero (Liu et al., 26 Sep 2025).

The relative rank reward is then defined as

$c$2

with binary ground-truth preference label

$c$3

The paper describes this term as a fidelity-based signal that aligns pairwise ranking consistency (Liu et al., 26 Sep 2025).

To calibrate absolute scoring, RAPO adds a Gaussian-shaped absolute-error reward toward the MOS $c$4:

$c$5

where $c$6 is a hyperparameter and $c$7 avoids zero reward. The total reward is additive:

$c$8

The stated intuition is that the rank term optimizes SRCC, while the absolute term optimizes PLCC; the joint reward is presented as a way to stabilize exploration and exploitation while avoiding the imbalance observed in pure ranking or pure regression (Liu et al., 26 Sep 2025).

This reward decomposition addresses a common misconception that ordinal correctness alone is sufficient for aesthetic assessment. In the Aes-R1 analysis, rank-only optimization indeed improves ordering, but it does not by itself produce calibrated score distributions. Conversely, score-only optimization lacks explicit comparative structure. RAPO’s defining feature is that it treats these as complementary rather than substitutable objectives (Liu et al., 26 Sep 2025).

3. Policy optimization objective

For each prompt-image pair $c$9, RAPO computes rewards over $s$0 sampled outputs $s$1 and uses group-wise normalization to obtain token-level advantages:

$s$2

This normalization is part of the PPO-style optimization adopted in Aes-R1 (Liu et al., 26 Sep 2025).

Following DAPO, RAPO uses a higher clipping threshold and a reduced KL penalty. The clipped objective is

$s$3

with probability ratio

$s$4

The KL term is approximated in the paper by

$s$5

Here $s$6 is the pretrained reference policy, $s$7 is the KL coefficient, and $s$8 are clipping thresholds (Liu et al., 26 Sep 2025).

The significance of this construction is methodological rather than cosmetic. RAPO is not defined solely by its reward function; it is defined by the coupling of that dual reward with group-wise advantage normalization and PPO-style clipped updates over outputs that include both reasoning and score tokens. A plausible implication is that the method’s reported gains depend on this full optimization stack rather than on the additive reward alone (Liu et al., 26 Sep 2025).

4. Role within Aes-R1: cold-start, prompts, and training pipeline

RAPO is embedded in a two-stage Aes-R1 pipeline: cold-start supervised fine-tuning with AesCoT, followed by RAPO reinforcement learning. The paper uses a combined train set of 15K samples drawn from AVA, TAD66K, and FLICKR-AES in a 2:2:1 ratio, with MOSs normalized to $s$9 (Liu et al., 26 Sep 2025).

The cold-start stage is built on AesCoT. Starting from image-score pairs $\mathcal{I}$0, the framework conceals scores and prompts a strong external expert model to produce chain-of-thought critiques along five dimensions: Light and Shadow, Mood and Narrative, Composition, Color, and Exposure. Critiques are matched to three aesthetic levels—bad $\mathcal{I}$1, fair $\mathcal{I}$2, and good $\mathcal{I}$3—and concatenated with ground-truth scores, after filtering for score leakage, reasoning–score mismatch, and factual errors. The supervised warm-up objective is

$\mathcal{I}$4

This stage is trained for 1 epoch to “equip MLLMs with aesthetic cognitive behaviors” and “teach explanation-before-score” (Liu et al., 26 Sep 2025).

The prompt format is itself part of the method. For aesthetic scoring, the model is instructed to place all reasoning inside one pair of > ... tags and the final scalar rating inside one pair of <answer>...</answer> tags. The paper presents this enforced structure as a way to stabilize output format and prepare the model for RL over reasoning-plus-score sequences (Liu et al., 26 Sep 2025).

After warm-up, RAPO proceeds batchwise. For each batch of $\mathcal{I}$5 images $\mathcal{I}$6, the old policy samples $\mathcal{I}$7 outputs per input. Pairwise preference labels are computed from MOSs, $\mathcal{I}$8 is estimated as the mean over the $\mathcal{I}$9 outputs of image $\mathcal{P}$0, and $\mathcal{P}$1 and $\mathcal{P}$2 are computed for each output. Rewards are grouped by image, advantages are normalized within group, and the policy is updated with the clipped objective. RAPO is then trained for 10 epochs (Liu et al., 26 Sep 2025).

An important empirical claim in the paper is that warm-up is not optional in practice. “Without cold-start SFT, RL can fail to activate aesthetic reasoning patterns,” and the case study reports that Aes-R1-Zero yields superficial explanations. This suggests that RAPO is best understood as a second-stage optimizer over already induced reasoning behavior, not as a standalone mechanism for eliciting aesthetic cognition from an unprepared model (Liu et al., 26 Sep 2025).

5. Experimental performance and ablation results

Aes-R1 is evaluated on TAD66K, AVA, FLICKR-AES, and on out-of-distribution datasets PARA and AADB, using PLCC and SRCC. With Qwen2.5-VL-7B-Instruct as the backbone, Aes-R1 reports average PLCC/SRCC of 0.6337/0.6186. The paper states that this corresponds to a 47.9%/34.8% improvement over the backbone’s average 0.4285/0.4589, and that Aes-R1 shows the closest alignment to the ground-truth score distribution with well-matched peak and reduced tail deviation (Liu et al., 26 Sep 2025).

Dataset	PLCC	SRCC
TAD66K	0.4513	0.4248
AVA	0.6702	0.6619
FLICKR	0.7243	0.6973
PARA	0.7842	0.7666
AADB	0.5386	0.5423

The out-of-distribution results on PARA and AADB are highlighted by the paper as evidence of robust generalization, with scores of 0.7842/0.7666 and 0.5386/0.5423 respectively. Compared to a retrained rank-only baseline cited in the paper, which obtains average PLCC/SRCC of 0.4429/0.5930, RAPO improves calibration while preserving strong rank performance. This directly matches the method’s stated objective: rank-only optimization improves SRCC but under-calibrates PLCC, whereas the combined reward produces a more balanced result (Liu et al., 26 Sep 2025).

The reward ablations are especially diagnostic. Binary-only reward yields the weakest results, with average 0.4255/0.4433. Error-only reward markedly outperforms binary-only, at 0.5655/0.5600. Rank-only reward achieves average SRCC 0.5908 but depresses PLCC to 0.4542. Combining error and rank yields the best averages without cold-start, at 0.6297/0.6102, while the full Aes-R1 pipeline with 1-epoch SFT reaches 0.6337/0.6186 (Liu et al., 26 Sep 2025).

The entropy analysis adds a further qualification. Moderate SFT, specifically 1 epoch, raises average token entropy and downstream performance, while excessive SFT from 2 to 10 epochs depresses entropy, overfits the data, narrows policy exploration, and reduces RL gains and OOD generalization. The paper explicitly states that “maintaining appropriately high entropy can foster exploration, stabilize training and mitigate entropy collapse,” and it reports stable entropy traces for RAPO without collapse (Liu et al., 26 Sep 2025).

6. Positioning, acronym ambiguity, and limitations

Within the paper’s own positioning, RAPO differs from regression-only baselines, preference-based rank methods, and generic PPO-like RL. Regression-only methods optimize score calibration but lack explicit ranking supervision. Rank-only methods improve ordinal discernment but do not calibrate scores adequately. RAPO adopts a PPO-style clipped ratio objective and follows DAPO in using higher clip thresholds and reduced KL penalty, but its defining characteristic is the dual reward tailored to aesthetics: a FRank fidelity term for comparative order and a Gaussian absolute-error term for MOS calibration (Liu et al., 26 Sep 2025).

The acronym itself is ambiguous in the broader 2025–2026 literature. In the Aes-R1 paper, RAPO means Relative-Absolute Policy Optimization (Liu et al., 26 Sep 2025). In unrelated 2026 papers, RAPO denotes Regret-Aware Policy Optimization for environment-level replay suppression under delayed harm (Hiremath, 8 Apr 2026), Reaction Aware Policy Optimization for dialogue agents trained from simulated user reactions (Ye et al., 16 Mar 2026), and Retrieval-Augmented Policy Optimization for step-level exploration in LLM agents (Zhang et al., 3 Mar 2026). This acronym collision is a recurrent source of confusion and makes paper-level disambiguation necessary.

The limitations reported for Relative-Absolute Policy Optimization are specific. First, the subjectivity and absence of unified criteria in aesthetics complicate reward design, so RAPO still depends on MOS proxies. Second, without cold-start SFT, RL can fail to activate aesthetic reasoning patterns, which creates a dependency on curated AesCoT data. Third, excessive SFT reduces entropy, hinders exploration, and harms OOD generalization, so the balance between warm-up and RL is consequential. Finally, while RAPO improves both PLCC and SRCC, the paper does not report confidence intervals or detailed convergence proofs; broader theoretical guarantees and more diverse OOD studies are identified as future directions (Liu et al., 26 Sep 2025).

Taken together, Relative-Absolute Policy Optimization is best characterized as a domain-specific RL design for multimodal aesthetic reasoning rather than a generic policy-optimization template. Its central claim is that aesthetic judgment requires simultaneous optimization of calibrated scalar assessment and comparative preference structure, and that this combination is most effective when applied after a cold-start stage that teaches explanation-before-score behavior (Liu et al., 26 Sep 2025).