Certified Sign-Preservation Radius in RLHF

• Certified Sign-Preservation Radius is a robustness measure in RLHF that ensures the sign of a completion’s advantage remains unchanged under controlled parameter perturbations.
• It is defined as the maximum ℓ2 perturbation in reward model parameters where the advantage sign stays consistent, linking update stability to feature sensitivity.
• Applied in Sign-Certified Policy Optimization, this radius helps suppress non-robust completions and mitigates reward hacking by adjusting policy gradients.

Searching arXiv for the cited paper and closely related usage of the term. The certified sign-preservation radius is a completion-level robustness quantity for reinforcement learning from human feedback (RLHF). It was introduced to formalize the claim that a major local mechanism of reward hacking is flipped advantage signs during policy optimization: when a proxy reward model assigns an advantage with the wrong sign, optimization reinforces a bad completion or suppresses a good one. For a sampled completion, the radius is the largest perturbation of reward-model parameters under which the sign of its advantage is guaranteed to remain unchanged. In "Mitigating Reward Hacking in RLHF via Advantage Sign Robustness" (Ono et al., 3 Apr 2026), this quantity is used to construct Sign-Certified Policy Optimization (SignCert-PO), a lightweight post-hoc modification of Dr.GRPO that down-weights non-robust completions during policy optimization.

1. RLHF setting and the role of advantage signs

In RLHF, one trains a proxy reward model $r_\theta(x,y)$ from preference data and then optimizes a policy $\pi_\phi(y\mid x)$ against that proxy rather than the unknown true reward $r^*(x,y)$. The standard KL-regularized RLHF objective is

$$J(\phi, \theta) = \mathbb{E}_{x\sim P(x), y\sim\pi_\phi}\bigl[r_\theta(x, y) - \beta D_\mathrm{KL}(\pi_\phi(\cdot|x) \| \pi_\text{ref}(\cdot | x))\bigr],$$

with policy gradient

$$\nabla_\phi J(\phi, \theta) = \mathbb{E}_{x \sim P, y \sim \pi_\phi(\cdot|x)} [A^{\pi_\phi}_\theta(x,y) \nabla_\phi \log \pi_\phi(y|x)],$$

where

$$A_\theta^{\pi_\phi}(x,y) := r'_\theta(x,y) - V_\theta^{\pi_\phi}(x), \qquad r'_\theta(x,y):=r_\theta(x,y)-\beta D_\mathrm{KL}(\pi_\phi(\cdot\vert x)\Vert\pi_\mathrm{ref}(\cdot\vert x)).$$

The sign of the advantage determines whether a sampled completion is reinforced or suppressed. If $A>0$, the policy update increases that completion’s log-probability; if $A<0$, it decreases it. The central premise of (Ono et al., 3 Apr 2026) is that reward hacking often arises when this sign is wrong. That premise shifts attention away from absolute reward values and toward the binary update decision induced by the reward model.

The paper specializes this analysis to Dr.GRPO. For a prompt $x$, one samples $K$ completions $\pi_\phi(y\mid x)$0, scores them with the reward model, and defines the group-relative advantage

$\pi_\phi(y\mid x)$1

The policy gradient becomes

$\pi_\phi(y\mid x)$2

Under a first-order step $\pi_\phi(y\mid x)$3,

$\pi_\phi(y\mid x)$4

In isolation, the sign of $\pi_\phi(y\mid x)$5 therefore determines whether the $\pi_\phi(y\mid x)$6-th completion is upweighted or downweighted. This is the direct optimization-theoretic motivation for certifying sign stability.

2. Formal definition of the radius

Because language inputs are discrete, (Ono et al., 3 Apr 2026) does not perturb the input pair $\pi_\phi(y\mid x)$7. Instead, it models uncertainty as perturbation in reward-model parameter space. For a general reward-model parameter vector $\pi_\phi(y\mid x)$8, it defines the $\pi_\phi(y\mid x)$9-ball uncertainty set

$r^*(x,y)$0

For completion $r^*(x,y)$1, the certified sign-preservation radius is

$r^*(x,y)$2

This is the largest $r^*(x,y)$3 parameter perturbation radius around the current reward model such that the completion’s advantage sign is guaranteed unchanged (Ono et al., 3 Apr 2026).

The paper explicitly frames this as an analogue of a certified radius in randomized smoothing. The difference is that the preserved binary label is not a classifier prediction on perturbed input, but $r^*(x,y)$4 under perturbed reward-model parameters. A large $r^*(x,y)$5 indicates that the update direction for that completion is robust; a small $r^*(x,y)$6 indicates that small reward-model changes could reverse whether the policy reinforces or suppresses it.

For a general differentiable reward model, a first-order Taylor expansion around $r^*(x,y)$7,

$r^*(x,y)$8

yields the approximate certificate

$r^*(x,y)$9

This has a margin-over-sensitivity form: larger $$J(\phi, \theta) = \mathbb{E}_{x\sim P(x), y\sim\pi_\phi}\bigl[r_\theta(x, y) - \beta D_\mathrm{KL}(\pi_\phi(\cdot|x) \| \pi_\text{ref}(\cdot | x))\bigr],$$0 and smaller $$J(\phi, \theta) = \mathbb{E}_{x\sim P(x), y\sim\pi_\phi}\bigl[r_\theta(x, y) - \beta D_\mathrm{KL}(\pi_\phi(\cdot|x) \| \pi_\text{ref}(\cdot | x))\bigr],$$1 imply greater sign robustness. The paper also notes that computing per-completion full-parameter gradient norms is too expensive for practical large reward models, which motivates a more tractable specialization.

3. Exact closed form under a linear reward head

To obtain an exact and cheap certificate, (Ono et al., 3 Apr 2026) assumes the standard reward-model architecture

$$J(\phi, \theta) = \mathbb{E}_{x\sim P(x), y\sim\pi_\phi}\bigl[r_\theta(x, y) - \beta D_\mathrm{KL}(\pi_\phi(\cdot|x) \| \pi_\text{ref}(\cdot | x))\bigr],$$2

where $$J(\phi, \theta) = \mathbb{E}_{x\sim P(x), y\sim\pi_\phi}\bigl[r_\theta(x, y) - \beta D_\mathrm{KL}(\pi_\phi(\cdot|x) \| \pi_\text{ref}(\cdot | x))\bigr],$$3 is the last-layer feature representation and $$J(\phi, \theta) = \mathbb{E}_{x\sim P(x), y\sim\pi_\phi}\bigl[r_\theta(x, y) - \beta D_\mathrm{KL}(\pi_\phi(\cdot|x) \| \pi_\text{ref}(\cdot | x))\bigr],$$4 is a linear scalar head. Instead of perturbing all parameters $$J(\phi, \theta) = \mathbb{E}_{x\sim P(x), y\sim\pi_\phi}\bigl[r_\theta(x, y) - \beta D_\mathrm{KL}(\pi_\phi(\cdot|x) \| \pi_\text{ref}(\cdot | x))\bigr],$$5, the paper perturbs only the head: $$J(\phi, \theta) = \mathbb{E}_{x\sim P(x), y\sim\pi_\phi}\bigl[r_\theta(x, y) - \beta D_\mathrm{KL}(\pi_\phi(\cdot|x) \| \pi_\text{ref}(\cdot | x))\bigr],$$6

Let

$$J(\phi, \theta) = \mathbb{E}_{x\sim P(x), y\sim\pi_\phi}\bigl[r_\theta(x, y) - \beta D_\mathrm{KL}(\pi_\phi(\cdot|x) \| \pi_\text{ref}(\cdot | x))\bigr],$$7

Under $$J(\phi, \theta) = \mathbb{E}_{x\sim P(x), y\sim\pi_\phi}\bigl[r_\theta(x, y) - \beta D_\mathrm{KL}(\pi_\phi(\cdot|x) \| \pi_\text{ref}(\cdot | x))\bigr],$$8, linearity gives

$$J(\phi, \theta) = \mathbb{E}_{x\sim P(x), y\sim\pi_\phi}\bigl[r_\theta(x, y) - \beta D_\mathrm{KL}(\pi_\phi(\cdot|x) \| \pi_\text{ref}(\cdot | x))\bigr],$$9

The smallest perturbation that can flip the sign is the smallest $$\nabla_\phi J(\phi, \theta) = \mathbb{E}_{x \sim P, y \sim \pi_\phi(\cdot|x)} [A^{\pi_\phi}_\theta(x,y) \nabla_\phi \log \pi_\phi(y|x)],$$0 such that there exists $$\nabla_\phi J(\phi, \theta) = \mathbb{E}_{x \sim P, y \sim \pi_\phi(\cdot|x)} [A^{\pi_\phi}_\theta(x,y) \nabla_\phi \log \pi_\phi(y|x)],$$1 with $$\nabla_\phi J(\phi, \theta) = \mathbb{E}_{x \sim P, y \sim \pi_\phi(\cdot|x)} [A^{\pi_\phi}_\theta(x,y) \nabla_\phi \log \pi_\phi(y|x)],$$2 and $$\nabla_\phi J(\phi, \theta) = \mathbb{E}_{x \sim P, y \sim \pi_\phi(\cdot|x)} [A^{\pi_\phi}_\theta(x,y) \nabla_\phi \log \pi_\phi(y|x)],$$3. By Cauchy-Schwarz, the worst perturbation aligns against the current sign, so the sign flips when

$$\nabla_\phi J(\phi, \theta) = \mathbb{E}_{x \sim P, y \sim \pi_\phi(\cdot|x)} [A^{\pi_\phi}_\theta(x,y) \nabla_\phi \log \pi_\phi(y|x)],$$4

Hence the exact closed form of the certified sign-preservation radius is

$$\nabla_\phi J(\phi, \theta) = \mathbb{E}_{x \sim P, y \sim \pi_\phi(\cdot|x)} [A^{\pi_\phi}_\theta(x,y) \nabla_\phi \log \pi_\phi(y|x)],$$5

This is Theorem 1 in (Ono et al., 3 Apr 2026). The numerator $$\nabla_\phi J(\phi, \theta) = \mathbb{E}_{x \sim P, y \sim \pi_\phi(\cdot|x)} [A^{\pi_\phi}_\theta(x,y) \nabla_\phi \log \pi_\phi(y|x)],$$6 is the margin from zero; the denominator $$\nabla_\phi J(\phi, \theta) = \mathbb{E}_{x \sim P, y \sim \pi_\phi(\cdot|x)} [A^{\pi_\phi}_\theta(x,y) \nabla_\phi \log \pi_\phi(y|x)],$$7 is the sensitivity of that advantage to head perturbation. The paper interprets a small radius as arising either from small advantage magnitude or from being a feature-space outlier.

The same section connects the certificate to randomized smoothing. If $$\nabla_\phi J(\phi, \theta) = \mathbb{E}_{x \sim P, y \sim \pi_\phi(\cdot|x)} [A^{\pi_\phi}_\theta(x,y) \nabla_\phi \log \pi_\phi(y|x)],$$8, then the sign-preservation probability is

$$\nabla_\phi J(\phi, \theta) = \mathbb{E}_{x \sim P, y \sim \pi_\phi(\cdot|x)} [A^{\pi_\phi}_\theta(x,y) \nabla_\phi \log \pi_\phi(y|x)],$$9

and the randomized-smoothing radius

$$A_\theta^{\pi_\phi}(x,y) := r'_\theta(x,y) - V_\theta^{\pi_\phi}(x), \qquad r'_\theta(x,y):=r_\theta(x,y)-\beta D_\mathrm{KL}(\pi_\phi(\cdot\vert x)\Vert\pi_\mathrm{ref}(\cdot\vert x)).$$0

recovers the same margin form. The paper also lists related alternatives,

$$A_\theta^{\pi_\phi}(x,y) := r'_\theta(x,y) - V_\theta^{\pi_\phi}(x), \qquad r'_\theta(x,y):=r_\theta(x,y)-\beta D_\mathrm{KL}(\pi_\phi(\cdot\vert x)\Vert\pi_\mathrm{ref}(\cdot\vert x)).$$1

This suggests that the certified sign-preservation radius is best understood as a task-specific margin certificate tailored to policy-gradient update direction rather than to data-space label invariance.

4. Worst-case advantage and SignCert-PO

The paper uses the radius not only as a diagnostic but as a training coefficient. For completion $$A_\theta^{\pi_\phi}(x,y) := r'_\theta(x,y) - V_\theta^{\pi_\phi}(x), \qquad r'_\theta(x,y):=r_\theta(x,y)-\beta D_\mathrm{KL}(\pi_\phi(\cdot\vert x)\Vert\pi_\mathrm{ref}(\cdot\vert x)).$$2, it defines a worst-case conservative advantage

$$A_\theta^{\pi_\phi}(x,y) := r'_\theta(x,y) - V_\theta^{\pi_\phi}(x), \qquad r'_\theta(x,y):=r_\theta(x,y)-\beta D_\mathrm{KL}(\pi_\phi(\cdot\vert x)\Vert\pi_\mathrm{ref}(\cdot\vert x)).$$3

If the nominal reward model wants to reinforce a sample, the adversary makes its advantage as small as possible; if the nominal reward model wants to suppress it, the adversary makes it as large as possible. In both cases, the adversary tries to corrupt the sign-consistent update.

Under the linear-head analysis,

$$A_\theta^{\pi_\phi}(x,y) := r'_\theta(x,y) - V_\theta^{\pi_\phi}(x), \qquad r'_\theta(x,y):=r_\theta(x,y)-\beta D_\mathrm{KL}(\pi_\phi(\cdot\vert x)\Vert\pi_\mathrm{ref}(\cdot\vert x)).$$4

If $$A_\theta^{\pi_\phi}(x,y) := r'_\theta(x,y) - V_\theta^{\pi_\phi}(x), \qquad r'_\theta(x,y):=r_\theta(x,y)-\beta D_\mathrm{KL}(\pi_\phi(\cdot\vert x)\Vert\pi_\mathrm{ref}(\cdot\vert x)).$$5 is large relative to $$A_\theta^{\pi_\phi}(x,y) := r'_\theta(x,y) - V_\theta^{\pi_\phi}(x), \qquad r'_\theta(x,y):=r_\theta(x,y)-\beta D_\mathrm{KL}(\pi_\phi(\cdot\vert x)\Vert\pi_\mathrm{ref}(\cdot\vert x)).$$6, then $$A_\theta^{\pi_\phi}(x,y) := r'_\theta(x,y) - V_\theta^{\pi_\phi}(x), \qquad r'_\theta(x,y):=r_\theta(x,y)-\beta D_\mathrm{KL}(\pi_\phi(\cdot\vert x)\Vert\pi_\mathrm{ref}(\cdot\vert x)).$$7. If $$A_\theta^{\pi_\phi}(x,y) := r'_\theta(x,y) - V_\theta^{\pi_\phi}(x), \qquad r'_\theta(x,y):=r_\theta(x,y)-\beta D_\mathrm{KL}(\pi_\phi(\cdot\vert x)\Vert\pi_\mathrm{ref}(\cdot\vert x)).$$8 is near $$A_\theta^{\pi_\phi}(x,y) := r'_\theta(x,y) - V_\theta^{\pi_\phi}(x), \qquad r'_\theta(x,y):=r_\theta(x,y)-\beta D_\mathrm{KL}(\pi_\phi(\cdot\vert x)\Vert\pi_\mathrm{ref}(\cdot\vert x)).$$9, the weight approaches zero. If $A>0$0, then $A>0$1, and the paper’s interpretation is that non-robust completions are effectively suppressed; its overview figure states that it “suppress[es] completions below $A>0$2” (Ono et al., 3 Apr 2026).

The resulting policy gradient is

$A>0$3

Setting $A>0$4 recovers standard Dr.GRPO.

Algorithmically, Sign-Certified Policy Optimization requires only the current reward-model parameters, specifically the linear head $A>0$5 and hidden features $A>0$6, together with on-policy completions. It does not require multiple reward models, access to the reward-model training dataset at policy optimization time, retraining the reward model, or special uncertainty labels (Ono et al., 3 Apr 2026). For each prompt, it samples $A>0$7 completions, computes $A>0$8, rewards $A>0$9, advantages

$A<0$0

then computes

$A<0$1

chooses a perturbation budget $A<0$2, forms

$A<0$3

and updates the policy using $A<0$4 instead of $A<0$5.

The practical choice in (Ono et al., 3 Apr 2026) is to set $A<0$6 adaptively from the batch as the $A<0$7-th quantile of

$A<0$8

equivalently the $A<0$9-th quantile of $x$0. Higher $x$1 therefore yields a larger effective uncertainty budget and more aggressive conservative downweighting.

5. Why the certificate is completion-specific

A distinctive feature of the certified sign-preservation radius is that it is defined per completion rather than as a global robustification of the RLHF objective. The paper considers the shared-adversary objective

$x$2

and derives

$x$3

The correction is a single completion-agnostic direction. The authors describe this as “uniform pessimism,” since it does not distinguish reliable from unreliable completions (Ono et al., 3 Apr 2026).

By contrast, the certified sign-preservation radius is inherently completion-specific. It selectively down-weights fragile gradient terms while leaving robust ones largely unchanged. The paper also notes that a direct robustification of aggregate GRPO advantage is vacuous because group-relative advantages sum to zero, so a standard $x$4 provides no useful discrimination. This is the technical reason the per-sample adversary is both meaningful and tractable, even though it is more conservative than a shared-adversary formulation.

This suggests that the radius is not merely a generic robustness score. It is a certificate aligned to the update geometry of group-relative policy optimization: it measures the stability of the sign decision that determines whether a completion is pushed up or down in probability.

6. Empirical behavior, scope, and related usages of “sign preservation”

On TL;DR summarization and AlpacaFarm, SignCert-PO consistently achieves a better win rate than baselines and reduces reward hacking (Ono et al., 3 Apr 2026). The main metric is win rate against a human-generated reference response as judged by a stronger held-out gold reward model. Reported results include the following.

Benchmark / model	Baselines	SignCert-PO
TL;DR, Pythia 1B	SFT $x$5, Dr.GRPO $x$6, UWO $x$7, BSPO $x$8, AdvPO $x$9	$K$0
TL;DR, Qwen2.5 1.5B	SFT $K$1, Dr.GRPO $K$2, UWO $K$3, BSPO $K$4, AdvPO $K$5	$K$6
AlpacaFarm, Qwen2.5 1.5B	SFT $K$7, Dr.GRPO $K$8, UWO $K$9, BSPO $\pi_\phi(y\mid x)$00, AdvPO $\pi_\phi(y\mid x)$01	$\pi_\phi(y\mid x)$02

The runtime table on TL;DR reports average wall-clock time per policy optimization step of $\pi_\phi(y\mid x)$03s / $\pi_\phi(y\mid x)$04s for Dr.GRPO, $\pi_\phi(y\mid x)$05s / $\pi_\phi(y\mid x)$06s for UWO, $\pi_\phi(y\mid x)$07s / $\pi_\phi(y\mid x)$08s for BSPO, $\pi_\phi(y\mid x)$09s / $\pi_\phi(y\mid x)$10s for AdvPO, and $\pi_\phi(y\mid x)$11s / $\pi_\phi(y\mid x)$12s for SignCert-PO. The paper therefore characterizes the method as nearly identical to Dr.GRPO in wall-clock cost while avoiding multiple reward models or access to reward-model training data during policy optimization (Ono et al., 3 Apr 2026).

The empirical argument for the radius itself is twofold. First, SignCert-PO maintains higher proxy reward-model accuracy on on-policy samples as optimization progresses. On TL;DR with Pythia-1B, Dr.GRPO’s proxy reward-model accuracy drops toward $\pi_\phi(y\mid x)$13 at high KL, whereas SignCert-PO maintains accuracy around $\pi_\phi(y\mid x)$14. Second, larger $\pi_\phi(y\mid x)$15 correlates with higher sign-preservation or agreement rates even under broader perturbations than the theorem assumes: the paper reports Spearman correlations of $\pi_\phi(y\mid x)$16 for whole-model perturbation and $\pi_\phi(y\mid x)$17 for input-embedding perturbation (Ono et al., 3 Apr 2026). This does not extend the theorem, but it indicates that the head-derived radius captures broader sensitivity structure.

The scope of the formal guarantee remains narrow. The exact certificate is derived only for head-only perturbations under a linear head; the full-parameter analogue

$\pi_\phi(y\mid x)$18

is acknowledged to be computationally infeasible at scale. The method is also tied to sign robustness, not full reward accuracy: a sign can be robust and still wrong relative to the true reward if the proxy reward model is systematically biased. The per-sample adversary is more conservative than a shared-adversary robust formulation, and the denominator $\pi_\phi(y\mid x)$19 makes the radius sensitive to feature-space outliers (Ono et al., 3 Apr 2026).

Outside RLHF, the phrase sign-preserving appears in different senses that should not be conflated with the certified sign-preservation radius. In "Sign-Preserving Property for Some Fourth-Order Elliptic Operators in One Dimension and Radial Symmetry" (Laurencot et al., 2013), the result is a qualitative theorem: under structural hypotheses on fourth-order operators with clamped boundary conditions, nonpositive nontrivial forcing yields a strictly negative solution. The paper does not provide a quantitative geometric “certified sign-preservation radius”; its central contribution is a strong sign-preserving property, not a perturbation radius (Laurencot et al., 2013). Likewise, "Certified Signed Graph Unlearning" (Zhao et al., 18 Nov 2025) provides certified unlearning, a triadic sign-aware certification region, sensitivity bounds, and utility/error guarantees, but it does not define a formal object called a certified sign-preservation radius and does not certify that sign predictions are preserved inside a radius (Zhao et al., 18 Nov 2025).

In current arXiv usage, the certified sign-preservation radius therefore refers most precisely to the RLHF quantity introduced in (Ono et al., 3 Apr 2026): $\pi_\phi(y\mid x)$20 Under a linear reward head, this becomes the exact closed form

$\pi_\phi(y\mid x)$21

and underlies a conservative policy-gradient rule that trusts completions whose update directions are robust under reward-model perturbation while suppressing those whose signs are easy to flip.