HPSv3++: Scaling Reward Models Across the Full Spectrum of Diffusion Model Capabilities

Abstract: Reward models guide text-to-image (T2I) systems toward outputs aligned with human preferences. However, typical reward models such as HPSv3 are trained on pre-annotated data from earlier T2I models, without accounting for quality discriminative shifts arising from evolving model capabilities and reinforcement learning (RL) iterations, limiting their broader applicability. In this work, we propose HPSv3++, a reward model framework that elevates the HPSv3 model for varying T2I model capabilities and their RL iteration changes across the full capability-iteration spectrum. Specifically, we first introduce HPDv3++, a 212K dual-dimension preference dataset annotated for text fidelity and aesthetic quality using a recent high-capability (Qwen-Image) model with human supervision. We then propose a two-stage training framework. Stage 1 employs data-aware orthogonal gradient projection to incorporate diverse aesthetic perception from HPDv3++ while preserving the original effective human preference knowledge in HPSv3. Stage 2 further leverages unlabeled data from T2I models spanning different capability levels and RL iterations, and introduces a joint capability-iterations conditioned signal for the reward model together with a standard deviation-driven unsupervised guidance mechanism, strengthening reward model across the capability-iteration spectrum. HPSv3++ achieves state-of-the-art preference prediction, outperforming HPSv3 9.8% on HPDv3, 5.5% on GenAI-Bench, while achieving 79.1%/88.1% on our proposed HPDv3++. When used for T2I RL training, it consistently improves GenEval scores across diverse T2I models, demonstrating its wide-range capabilities. The code is available at https://github.com/PlantPotatoOnMoon/HPSv3-PlusPlus.

• The paper presents a novel reward model that conditions scoring on both T2I generator capability and RL iteration, improving dynamic discriminability.
• It employs a two-stage training pipeline using Orthogonal Gradient Descent and semi-supervised adaptive learning to integrate new aesthetic data without catastrophic forgetting.
• State-of-the-art results are achieved, evidenced by an 86.7% pairwise prediction accuracy and significant RL improvements in GenEval, HPSv2, and CLIPScore metrics.

HPSv3++: Scaling Reward Models for Text-to-Image Diffusion Across Capability and Optimization Axes

Motivation and Problem Statement

Reward models (RMs) are central to RLHF-driven T2I generation, providing scalar signals that shape post-training behavior toward outputs preferred by humans. However, models such as HPSv3 and prior RMs have been trained on static, legacy distributions and do not generalize across the rapidly increasing capability of modern T2I backbones or the continually shifting distributions induced by RL optimization. The static, unconditional scoring regime fundamentally limits their dynamic discriminability, leading to a loss of score variance and ranking reliability as RL progresses, especially for high-quality, low-diversity rollouts (see (Figure 1)). HPSv3++ addresses these deficiencies by explicitly conditioning scoring on model capability and RL iteration, and by introducing a robust, dual-dimension dataset annotated for both semantic text-following and aesthetic quality using frontier generators.

Figure 1: Score standard deviation behavior across RL iterations reveals HPSv3++'s ability to preserve discriminability, unlike HPSv3.

Dual-Dimension Dataset: HPDv3++

A foundational component is HPDv3++, a large-scale dataset curated with 185K prompts yielding 1.1M+ candidate images from Qwen-Image. Human annotators provide pairwise judgments on text-following fidelity and aesthetic quality, explicitly decoupling semantic accuracy from perceptual appeal. A dual-vote system employing a VLM judge and RM judge filters noise, resulting in 95K text-following and 117K aesthetic pairs, distributed across more than 20 diverse scenarios (see (Figure 2)). Prompt statistics confirm extensive coverage of semantics, languages, and compositional complexity (see (Figure 3), (Figure 4)), with an intentional emphasis on activity, sports, and text-rich prompts that reflect the trajectory of modern generation models.

Figure 2: Data pipeline and annotation strategy underpinning HPDv3++ for robust preference supervision.

Figure 3: Broad semantic coverage in HPDv3++ prompts, with enrichment for activity and person-centric concepts.

Figure 4: Language and length statistics of HPDv3++ prompts, evidencing compositional diversity and long-tail distribution.

Two-Stage Training with Conditioning and Continual Learning

HPSv3++ employs a two-stage pipeline:

Stage 1: Continual learning via Orthogonal Gradient Descent (OGD) integrates new aesthetic perceptions from HPDv3++ while retaining HPSv3's original preference scope. OGD projects HPDv3++ gradients onto the orthogonal complement of HPDv3 reference gradients, mitigating catastrophic forgetting and enhancing discriminative range on unseen frontier distributions.

Stage 2: Semi-supervised adaptive training exploits both labeled HPDv3++ data and unlabeled rollouts from diverse T2I models and RL stages. The architecture extends reward prediction to condition on inferred model capability (via a Capability Encoder from image features) and explicit RL iteration (via normalized input), combined through FiLM-based conditioning in the reward head (see (Figure 5)). Score standard deviation-based objectives maximize intra-group discriminability and adaptively calibrate sensitivity across capability/iteration axes, preventing score collapse and enhancing dynamic ranking.

Figure 5: HPSv3++ training—OGD-based continual learning and semi-supervised capability-iteration conditioning.

Numerical Results and Empirical Evaluation

HPSv3++ demonstrates state-of-the-art pairwise preference prediction, achieving 86.7% on HPDv3 (+9.8% over HPSv3), 76.3% on GenAI-Bench (+5.5%), and 79.1%/88.1% on HPDv3++ aesthetic/text-following splits. Controlled RL fine-tuning experiments on Qwen-Image, FLUX.1-dev, and SDXL reveal consistent improvements in GenEval, HPSv2, Aesthetic Score, and CLIPScore metrics (see (Figure 6), (Figure 7), (Figure 8)), with HPSv3++ maintaining and in many cases increasing intra-group score standard deviation across RL stages (see (Figure 9)). A user study confirms a 77.5% human win rate for HPSv3++ over HPSv3, substantiating improved alignment.

Figure 10: HPSv3++'s evaluation scope encompasses the full [model capability × RL iteration] space, unlike HPSv3.

Figure 6: Qualitative RL fine-tuning comparison: HPSv3++ yields visually superior results across models and RL stages.

Figure 7: GenEval score trajectories during RL, evidencing robust improvements with HPSv3++ rewards.

Figure 9: Score std analysis, showing monotonic growth and adaptive response to RL iteration conditioning.

Figure 8: Controlled qualitative comparisons confirm better prompt fidelity and quality with HPSv3++ guidance.

Ablations and Analysis

Ablation studies validate the necessity of both OGD-based continual learning and adaptive semi-supervised conditioning. The Capability Encoder strategy is critical; direct joint training from random initialization yields the strongest results, outstripping frozen or pretrained variants. Iteration-aware std analysis shows mean intra-group score std increases monotonically with RL iteration across all capability tiers, confirming successful dynamic modulation via FiLM conditioning.

Practical and Theoretical Implications

HPSv3++ substantiates the need for context-conditioned reward modeling, moving beyond static single-score paradigms. By calibrating scoring to the joint capability–iteration space, HPSv3++ delivers reliable reward signals for heterogeneous T2I generators throughout RL optimization. This approach reduces reward hacking susceptibility and improves preference alignment, especially in high-quality and low-diversity regimes that challenge legacy RMs. Its dual-dimension dataset architecture sets new standards for annotation scope and compositional diversity.

Future research directions include scaling capability-iteration conditioning further—potentially with expanded axes for style, safety, and domain expertise—and developing dynamically adaptive RMs that respond to even longer RL horizons and novel generation paradigms. The methodology is directly extensible to other generative modalities (e.g., text-to-video, image-to-video) and to emerging, continually-improving diffusion architectures.

Conclusion

HPSv3++ provides a robust, capability- and iteration-aware RM for T2I generation, supported by the HPDv3++ dataset and a two-stage training scheme that incorporates both continual learning and adaptive conditioning. Empirical evaluations confirm strong improvements in both numerical preference accuracy and visual RL outcomes across a broad spectrum of generator capabilities and optimization stages. The approach enables stable, context-sensitive reward modeling that is essential for modern RLHF workflows in generative vision models (2606.14657).