IPRO: Identity-Preserving Reward Optimization

• IPRO is a framework that optimizes generative models with reward signals while explicitly preserving semantic, behavioral, and perceptual identity.
• It leverages ensemble-based conservative methods like worst-case and uncertainty-weighted objectives alongside KL-regularized PPO and best-of-n sampling to prevent overoptimization.
• IPRO demonstrates robust performance under noisy feedback, ensuring reliable model alignment and scalability across language and vision domains.

Identity-Preserving Reward-guided Optimization (IPRO) encompasses methodologies and frameworks for optimizing generative models using reward signals, with explicit constraints or objectives to maintain the identity—semantic, behavioral, or perceptual—of the base model or generated outputs. Research in this area spans reinforcement learning from human feedback, multi-objective optimization, preference modeling, reward shaping, and feedback-driven alignment in both language and vision domains. The central challenge addressed by IPRO methods is the risk of overoptimization, reward hacking, and identity drift, particularly when proxy reward models are imperfect or noisy. Recent work formalizes identity preservation through conservative objectives, regularization strategies, ensemble modeling, adversarial robustness, and explicit multi-identity handling, positioning IPRO as a cornerstone of safe, reliable, and human-aligned model optimization.

1. Foundations: Overoptimization, Identity Drift, and the Need for IPRO

IPRO arose from the observed tendency of reward-guided or reinforcement learning optimization to exploit flaws in proxy reward models, resulting in policies or outputs that diverge from the intended "identity" of the base system. This phenomenon, documented by Gao et al., demonstrates that learned reward models are inevitably imperfect approximations of the true objective, making them susceptible to reward hacking and overoptimization even when increasing model and dataset sizes (Coste et al., 2023).

The central question addressed by IPRO is how to optimize for high reward—be it human preference, aesthetic fidelity, or behavioral correctness—without sacrificing identity consistency. In vision applications, this refers to the preservation of unique facial features across images and videos, while in LLMs it pertains to the retention of desirable characteristics such as helpfulness or harmlessness.

2. Ensemble-Based Conservative Optimization: Worst-Case and Uncertainty-Weighted Objectives

Ensemble-based conservative optimization is a key technical approach underpinning robust IPRO. Rather than relying on a single proxy reward model, ensembles of reward models trained on identical data with different random seeds form the basis for decision-making.

Worst-Case Optimization (WCO):

Optimizes policies using the minimum reward from the ensemble:

$R_{WCO}(q, a) = \min_{i} R_i(q, a)$

This conservative estimate ensures that as long as one reward model member is not overestimating, the policy is less likely to overexploit spurious reward signals.

Uncertainty-Weighted Optimization (UWO):

Downweights responses with high disagreement among the ensemble, incorporating mean reward and variance penalty:

$$R_{UWO}(q, a) = \frac{1}{k}\sum_i R_i(q, a) - \lambda \frac{1}{k}\sum_i (R_i(q, a) - \frac{1}{k}\sum_j R_j(q, a))^2$$

This discourages the optimization of uncertain outputs, enhancing stability and preserving the original behavioral intent (Coste et al., 2023).

These methods are empirically shown to nearly eliminate overoptimization, with performance gains up to 70% in best-of-n sampling regimes and improved robustness under label noise.

3. Optimization Methods: Best-of-n Sampling and PPO with KL Regularization

IPRO leverages two principal optimization paradigms for reward-guided learning:

Best-of-n Sampling (BoN):

The model generates $n$ responses per prompt, selecting the maximum-scoring output. Degree of optimization is quantified by KL divergence (e.g., $\text{KL} \approx \log n - \frac{n-1}{n}$). With ensemble conservative objectives, BoN achieves substantial increases in gold reward scores without the drift typically observed under mean reward optimization.

Proximal Policy Optimization (PPO):

Rewards are regularized by a KL penalty to constrain deviation from the initial (reference) model:

$$R_{PPO}(q, a) = R(q, a) - \beta \log \frac{\pi_{PPO}(a|q)}{\pi_{init}(a|q)}$$

Ensemble-based methods (WCO/UWO) with KL penalties (e.g., $\beta=0.01$) consistently reduce overoptimization, ensuring high reward without identity loss (Coste et al., 2023).

4. Robustness to Label Noise and Real-World Applicability

A distinguishing feature of IPRO strategies is robustness under noisy feedback. Introducing 25% label noise to proxy reward model training reflects the realities of human annotation (typical agreement rates 60–75%). Conservative ensemble objectives maintain reward integrity and stability, outperforming mean optimization which is susceptible to reward overestimation and drift.

This robustness is critical for deployment in scenarios where noisy or ambiguous human labels are unavoidable, ensuring that identity-preserving optimization is not compromised by labeling imperfections.

5. Performance Metrics, Quantitative Results, and Implementation Guidelines

IPRO frameworks are evaluated through:

• Gold reward scores: Benchmarked using a "true" large reward model.
• Win-rate statistics: Proportion of cases where ensemble policies outperform single reward model optimizations.
• KL divergence: Analytical and empirical measures track policy deviation from the base distribution.
• Label noise scenarios: Sensitivity analysis quantifies preservation under noisy conditions.

Extensive experiments report:

Optimization Method	Conservative Objective	Performance Gain	Overoptimization Eliminated
BoN (noiseless)	WCO/UWO Ensemble	Up to ~30%	Yes
BoN (noise 25%)	WCO/UWO Ensemble	Up to ~75%	Yes
PPO	WCO/UWO + KL Penalty	Consistent gains	Yes (with $\beta \approx 0.01$)

Implementation recommendations include ensemble formation via multiple pretrained RMs (minimizing cost), integration of KL regularization for PPO, and scalability across RM sizes and datasets.

6. Broader Implications and Future Directions

Ensemble-based IPRO features prominently in strategies for preference-based LLM alignment, image and video synthesis, and multi-objective reinforcement learning. Key implications:

• Identity Preservation: Formal mechanisms maintain the behavioral or perceptual core of models, preventing reward hacking and collapse to degenerate solutions.
• Scalability: Methods apply across domains and model scales, critical for real-world RLHF and generative tasks.
• Integration with Off-Policy Optimization: Off-policy methods (e.g., MPO, RPO) further enhance stability and data efficiency while retaining alignment with reference policies.
• Potential for Multi-Agent and Multi-Identity Systems: Extensions into assignments and group-based optimization (e.g., UMO, Identity-GRPO) allow for scalable handling of identity in multimodal and multi-subject environments.

Research continues toward adaptive regularization schemes, enhanced ensemble integration, and broader metrics for qualitative and quantitative identity assessment, anchoring IPRO as a foundational methodology for robust, alignment-safe model optimization.