Sign-Certified Policy Optimization

• The paper introduces SignCert-PO, a method that computes a certified sign-preservation radius to ensure reliable gradient updates in RLHF.
• It employs a closed-form certificate based on linear reward models, selectively down-weighting completions with fragile advantage signs.
• The approach mitigates reward hacking and improves policy performance, as shown by enhanced gold reward metrics on summarization and instruction tasks.

Sign-Certified Policy Optimization (SignCert-PO) is a policy optimization method for reinforcement learning from human feedback (RLHF) that mitigates reward hacking by down-weighting completions whose advantage signs are not robust to perturbations of the reward model (RM) parameters. Its central premise is that reward hacking is often caused by flipped advantage signs: instead of reducing the likelihood of a bad response, a flipped sign causes the update to increase it. Rather than relying on multiple reward models or access to RM training data, SignCert-PO operates purely at the policy optimization stage using only the RM parameters and on-policy completions, and introduces a per-completion certified sign-preservation radius that quantifies how much the RM can be perturbed before the advantage sign changes (Ono et al., 3 Apr 2026).

1. Problem setting and motivation

In RLHF, a reward model is trained as a proxy for human preferences, and the policy is then optimized against that proxy. The proxy is only an approximation, however, and as the policy diverges from the training distribution, the RM can be exploited in unforeseen ways: proxy reward continues to increase, but actual human-aligned quality plateaus or drops. This is the reward-hacking failure mode targeted by SignCert-PO (Ono et al., 3 Apr 2026).

The method is motivated by the observation that policy-gradient updates are sensitive not merely to reward magnitude but to the sign of each sampled completion’s advantage. If the RM incorrectly assigns the sign of the advantage—for example, treating a bad response as better than the group average—the update increases the probability of generating undesirable outputs. SignCert-PO therefore treats advantage sign reliability per completion as the key object of robustness analysis. The underlying claim is not that all RM errors are equally harmful, but that sign errors are especially consequential because they reverse the direction of the update itself (Ono et al., 3 Apr 2026).

This emphasis places SignCert-PO within a robustness-oriented strand of RLHF research, but its granularity is distinctive. The method does not apply a single pessimistic correction to a batch; instead, it quantifies sign robustness separately for each completion and suppresses only those gradient contributions whose direction is fragile under reward-model perturbation. A plausible implication is that SignCert-PO is best understood as a direction-of-update certification method rather than a global uncertainty estimator.

2. Certified sign-preservation radius

For a prompt $x$, suppose $K$ responses $y^{(1)}, \ldots, y^{(K)}$ are sampled. SignCert-PO defines the group-relative advantage of completion $j$ under RM parameters $\theta$ as

$$A_j(\theta) := r_\theta(x, y^{(j)}) - \frac{1}{K}\sum_{k=1}^K r_\theta(x, y^{(k)}).$$

To formalize sign robustness, the method considers an uncertainty set in parameter space,

$$\mathcal{U}_\tau^\theta = \{\theta' : \|\theta' - \theta\|_2 \leq \tau\},$$

and defines the certified sign-preservation radius for completion $j$ as

$$\Delta_j := \sup \left\{ \tau \geq 0 : \mathrm{sign}(A_j(\theta')) = \mathrm{sign}(A_j(\theta)), \ \forall \theta' \in \mathcal{U}_\tau^\theta \right\}.$$

This is the largest parameter perturbation under which the advantage sign is provably unchanged (Ono et al., 3 Apr 2026).

The interpretation is explicit in the formulation. A large $\Delta_j$ means the advantage sign is robust: many nearby reward models would yield the same favor-or-reject verdict for that completion. A small $K$0 means the completion is fragile: a small RM perturbation would reverse whether it is considered good or bad. SignCert-PO uses this quantity as a certification signal for whether a completion’s gradient contribution should be trusted.

The paper’s analysis focuses on linear head reward models commonly used in practice,

$K$1

where $K$2 and $K$3 parameterize the final layer and $K$4 are frozen features from a backbone. Restricting uncertainty to the linear head yields the closed-form certificate

$K$5

This closed form makes the certification computationally practical while preserving the core notion of sign robustness (Ono et al., 3 Apr 2026).

3. Policy update and optimization rule

The baseline group-based policy-gradient objective used for comparison is written as

$K$6

SignCert-PO modifies this update by computing, for each completion, a conservative weight derived from the certified radius. The per-completion worst-case advantage is

$K$7

The resulting SignCert-PO policy update is

$K$8

If $K$9 is small, $y^{(1)}, \ldots, y^{(K)}$0 is reduced toward zero, so the completion is almost ignored. If $y^{(1)}, \ldots, y^{(K)}$1 is large, $y^{(1)}, \ldots, y^{(K)}$2, and the completion’s contribution is largely preserved (Ono et al., 3 Apr 2026).

Operationally, each optimization step proceeds as follows: for each prompt, sample $y^{(1)}, \ldots, y^{(K)}$3 completions from the current policy; compute $y^{(1)}, \ldots, y^{(K)}$4 and last-layer features for all completions; compute $y^{(1)}, \ldots, y^{(K)}$5; compute $y^{(1)}, \ldots, y^{(K)}$6; and use $y^{(1)}, \ldots, y^{(K)}$7 as the coefficient in the policy-gradient update. The conservativeness hyperparameter $y^{(1)}, \ldots, y^{(K)}$8 can be set adaptively, for example via quantiles over observed $y^{(1)}, \ldots, y^{(K)}$9 per batch. Because only last-layer features are required, the method adds negligible overhead compared to standard policy gradients (Ono et al., 3 Apr 2026).

A key conceptual distinction is that the method does not replace the reward model or retrain it. It performs a post hoc robustness correction at the policy optimization stage. This makes the procedure lightweight relative to ensemble-based uncertainty methods and methods that depend on RM training data.

4. Empirical evaluation

The reported experiments evaluate SignCert-PO on two RLHF-style benchmarks: TL;DR, a Reddit summarization task, and AlpacaFarm, an instruction-following benchmark simulating RLHF. The policy backbones are Pythia (1B and 2.8B) and Qwen2.5 (1.5B and 3B). A stronger reward model, such as Skywork-Reward-Llama or GPT-4.1 Nano, is used as a “gold RM” reference for evaluation. The baselines are supervised fine-tuning (SFT), Dr.GRPO, UWO, BSPO, and AdvPO (Ono et al., 3 Apr 2026).

On gold-RM win rate, SignCert-PO is reported to achieve the highest, or competitive, performance in nearly all settings. On TL;DR, its win rates are 60.0 for Pythia 1B, 66.0 for Qwen 1.5B, 73.8 for Pythia 2.8B, and 91.8 for Qwen 3B. On AlpacaFarm, its win rates are 47.9 for Qwen 1.5B and 52.3 for Qwen 3B. The paper states that the benefit is most pronounced with limited preference data or smaller proxy reward models, where reward hacking is most problematic (Ono et al., 3 Apr 2026).

The baseline comparisons are heterogeneous rather than uniformly dominated. For example, BSPO attains 78.3 on TL;DR with Qwen 1.5B, but SignCert-PO attains the top value in the Pythia 1B, Pythia 2.8B, Qwen 3B, Alpaca Qwen 1.5B, and Alpaca Qwen 3B settings. The empirical claim is therefore not universal superiority in every configuration, but consistent improvement together with broad competitiveness across summarization and instruction-following regimes.

These results are used to support the paper’s main argument: suppressing updates from completions with fragile advantage signs improves downstream quality as judged by a stronger evaluator. In that sense, the benchmark evidence is not merely about robustness diagnostics; it is directly tied to policy improvement outcomes.

5. Training dynamics, robustness, and reward-hacking mitigation

The training-dynamics analysis centers on how RM reliability changes as the policy departs from the supervised initialization. Under standard Dr.GRPO, as the policy diverges and the KL from SFT increases, RM accuracy on on-policy completions drops toward random chance, and reward hacking emerges as rising proxy RM reward diverges from falling gold RM reward. SignCert-PO is reported to maintain higher RM accuracy at comparable KL divergence and to achieve higher gold reward without over-optimizing the proxy (Ono et al., 3 Apr 2026).

The paper also reports that early stopping does not eliminate the gap. Even with an oracle for stopping Dr.GRPO at peak gold reward, SignCert-PO achieves higher final gold reward. This matters because it suggests the gains are not solely due to slowed optimization or implicit regularization; they are linked to the selective suppression of unreliable update directions.

A second robustness claim concerns the meaning of the certified radius itself. The empirically measured certified radius $j$0 is reported to correlate well with the actual sign preservation rate under broader RM perturbations: completions with large radii remain robust even under more drastic changes. This does not convert the certificate into a full guarantee of human preference alignment, but it does support the intended interpretation of $j$1 as a practically informative robustness proxy (Ono et al., 3 Apr 2026).

The method’s low overhead is part of the empirical story rather than only an implementation detail. Because it only requires norms of feature differences and advantages, the added computation is described as negligible relative to standard methods. This suggests that the principal trade-off is between conservativeness and retained gradient signal, controlled through $j$2, rather than between robustness and computational feasibility.

6. Relation to policy certificates and sign-gated optimization

The term certification in SignCert-PO differs from the notion of policy certificates developed in accountable reinforcement learning. In the IPOC framework of Dann et al., algorithms output, before each episode, a policy $j$3, a return certificate $j$4, and an optimality certificate $j$5, with guarantees that $j$6 and $j$7 together with cumulative or mistake-style global constraints (Dann et al., 2018). Those certificates are policy-level and episode-wise. By contrast, SignCert-PO certifies the preservation of the sign of a completion-level advantage under perturbations of reward-model parameters. This places the two approaches in related but distinct parts of the certification landscape.

A second nearby development is SG-OPD, which studies sign-conditioned gating in on-policy distillation rather than RLHF reward-model robustness. SG-OPD uses a binary verifier as a trust signal for the teacher at two complementary granularities: phased teacher sampling mixes in verifier-endorsed teacher rollouts at cold-start, and a sign-consistency gate extrapolates the distillation update on tokens where the teacher agrees with the verifier-correct direction and interpolates it where it disagrees. On competition-level mathematical reasoning benchmarks, it consistently outperforms standard OPD, with average gains of 1.98 at the per-sample level and 7.50 at the per-question level (Xu et al., 8 Jun 2026).

Taken together, these lines of work indicate a broader research pattern in which optimization is conditioned not only on reward or preference magnitude but on whether the direction of a proposed update is certified by an auxiliary criterion. In SignCert-PO that auxiliary criterion is local robustness of the reward-model advantage sign; in SG-OPD it is agreement between teacher preference and verifier signal; in IPOC-style policy certificates it is episode-level confidence about return and suboptimality. This suggests a broader family of sign-conditioned or certificate-conditioned policy optimization methods, though the concrete guarantees and operating assumptions differ substantially across the three settings.