---
title: Certified Sign-Preservation Radius in RLHF
url: https://www.emergentmind.com/topics/certified-sign-preservation-radius
type: topic
---

# Certified Sign-Preservation Radius in RLHF

Searching arXiv for the cited paper and closely related usage of the term.
The **certified sign-preservation radius** is a completion-level robustness quantity for reinforcement learning from human feedback (RLHF). It was introduced to formalize the claim that a major local mechanism of reward hacking is **flipped advantage signs** during policy optimization: when a proxy reward model assigns an advantage with the wrong sign, optimization reinforces a bad completion or suppresses a good one. For a sampled completion, the radius is the largest perturbation of reward-model parameters under which the sign of its advantage is guaranteed to remain unchanged. In "Mitigating Reward Hacking in RLHF via Advantage Sign Robustness" [2604.02986], this quantity is used to construct Sign-Certified Policy Optimization (SignCert-PO), a lightweight post-hoc modification of Dr.GRPO that down-weights non-robust completions during policy optimization.

## 1. RLHF setting and the role of advantage signs

In RLHF, one trains a proxy reward model \(r_\theta(x,y)\) from preference data and then optimizes a policy \(\pi_\phi(y\mid x)\) against that proxy rather than the unknown true reward \(r^*(x,y)\). The standard KL-regularized RLHF objective is
\[
J(\phi, \theta) = \mathbb{E}_{x\sim P(x), y\sim\pi_\phi}\bigl[r_\theta(x, y) - \beta D_\mathrm{KL}(\pi_\phi(\cdot|x) \| \pi_\text{ref}(\cdot | x))\bigr],
\]
with policy gradient
\[
\nabla_\phi J(\phi, \theta) = \mathbb{E}_{x \sim P, y \sim \pi_\phi(\cdot|x)} [A^{\pi_\phi}_\theta(x,y) \nabla_\phi \log \pi_\phi(y|x)],
\]
where
\[
A_\theta^{\pi_\phi}(x,y) := r'_\theta(x,y) - V_\theta^{\pi_\phi}(x), \qquad r'_\theta(x,y):=r_\theta(x,y)-\beta D_\mathrm{KL}(\pi_\phi(\cdot\vert x)\Vert\pi_\mathrm{ref}(\cdot\vert x)).
\]

The sign of the advantage determines whether a sampled completion is reinforced or suppressed. If \(A>0\), the policy update increases that completion’s log-probability; if \(A<0\), it decreases it. The central premise of [2604.02986] is that reward hacking often arises when this sign is wrong. That premise shifts attention away from absolute reward values and toward the binary update decision induced by the reward model.

The paper specializes this analysis to Dr.GRPO. For a prompt \(x\), one samples \(K\) completions \(y^{(1)},\dots,y^{(K)}\sim \pi_\phi(\cdot|x)\), scores them with the reward model, and defines the group-relative advantage
\[
A_j(\theta) := r_\theta(x,y^{(j)}) - \frac{1}{K}\sum_{k=1}^K r_\theta(x,y^{(k)}).
\]
The policy gradient becomes
\[
\nabla_\phi J(\phi, \theta) = \mathbb{E}_{x\sim P, \{y^{(1)},\dots,y^{(K)}\} \sim \pi_\phi} \!\left[ \sum_{j=1}^{K} A_j(\theta) \, \nabla_\phi \log \pi_\phi(y^{(j)} | x) \right].
\]
Under a first-order step \(\phi'=\phi+\eta \nabla_\phi J(\phi,\theta)\),
\[
\log \pi_{\phi'}(y^{(j)} | x) - \log \pi_\phi(y^{(j)} | x) \approx \eta A_j(\theta) \| \nabla_\phi \log \pi_\phi(y^{(j)} | x) \|_2^2.
\]
In isolation, the sign of \(A_j(\theta)\) therefore determines whether the \(j\)-th completion is upweighted or downweighted. This is the direct optimization-theoretic motivation for certifying sign stability.

## 2. Formal definition of the radius

Because language inputs are discrete, [2604.02986] does not perturb the input pair \((x,y)\). Instead, it models uncertainty as perturbation in reward-model parameter space. For a general reward-model parameter vector \(\theta\), it defines the \(\ell_2\)-ball uncertainty set
\[
\mathcal{U}_\tau^\theta:=\{\theta':\|\theta'-\theta\|_2\le\tau\}.
\]

For completion \(j\), the certified sign-preservation radius is
\[
\Delta_j := \sup\bigl\{\tau \geq 0 : \mathrm{sign}(A_j(\theta')) = \mathrm{sign}(A_j(\theta))\ \forall \theta' \in \mathcal{U}_\tau^\theta\bigr\}.
\]
This is the largest \(\ell_2\) parameter perturbation radius around the current reward model such that the completion’s advantage sign is guaranteed unchanged [2604.02986].

The paper explicitly frames this as an analogue of a certified radius in randomized smoothing. The difference is that the preserved binary label is not a classifier prediction on perturbed input, but \(\operatorname{sign}(A_j)\) under perturbed reward-model parameters. A large \(\Delta_j\) indicates that the update direction for that completion is robust; a small \(\Delta_j\) indicates that small reward-model changes could reverse whether the policy reinforces or suppresses it.

For a general differentiable reward model, a first-order Taylor expansion around \(\theta\),
\[
A_j(\theta') \approx A_j(\theta) + \nabla_\theta A_j(\theta)^\mathsf{T}(\theta' - \theta),
\]
yields the approximate certificate
\[
\Delta_j(\theta)\gtrsim\frac{|A_j(\theta)|} {\|\nabla_\theta A_j(\theta)\|_2}.
\]
This has a margin-over-sensitivity form: larger \(|A_j|\) and smaller \(\|\nabla_\theta A_j\|_2\) imply greater sign robustness. The paper also notes that computing per-completion full-parameter gradient norms is too expensive for practical large reward models, which motivates a more tractable specialization.

## 3. Exact closed form under a linear reward head

To obtain an exact and cheap certificate, [2604.02986] assumes the standard reward-model architecture
\[
r_\theta(x,y) = w^\mathsf{T} h_\psi(x,y) + b,
\]
where \(h_\psi(x,y)\in\mathbb{R}^d\) is the last-layer feature representation and \((w,b)\) is a linear scalar head. Instead of perturbing all parameters \((\psi,w,b)\), the paper perturbs only the head:
\[
\mathcal{U}_\epsilon^w := \{w' \in \mathbb{R}^d : \|w' - w\|_2 \le \epsilon\}.
\]

Let
\[
h_j := h_\psi(x,y^{(j)}), \qquad \bar{h}:=\frac{1}{K}\sum_{k=1}^Kh_\psi(x,y^{(k)}).
\]
Under \(w' = w+\delta\), linearity gives
\[
A_j(w') = A_j(w) + \delta^\mathsf{T}(h_j - \bar{h}).
\]
The smallest perturbation that can flip the sign is the smallest \(\epsilon\) such that there exists \(\delta\) with \(\|\delta\|_2\le \epsilon\) and \(\operatorname{sign}(A_j(w+\delta))\neq \operatorname{sign}(A_j(w))\). By Cauchy-Schwarz, the worst perturbation aligns against the current sign, so the sign flips when
\[
\epsilon \ge \frac{|A_j(w)|}{\|h_j-\bar h\|_2}.
\]

Hence the exact closed form of the certified sign-preservation radius is
\[
\Delta_j = \frac{|A_j(w)|}{\|h_\psi(x,y^{(j)}) - \bar{h}\|_2}, \quad \text{where} \ \ \bar{h}:=\frac{1}{K}\sum_{k=1}^K h_\psi(x,y^{(k)}).
\]
This is Theorem 1 in [2604.02986]. The numerator \(|A_j(w)|\) is the margin from zero; the denominator \(\|h_j-\bar h\|_2\) is the sensitivity of that advantage to head perturbation. The paper interprets a small radius as arising either from small advantage magnitude or from being a feature-space outlier.

The same section connects the certificate to randomized smoothing. If \(\xi\sim\mathcal N(0,\sigma^2 I)\), then the sign-preservation probability is
\[
p = \Phi(|A_j|/(\sigma c)),
\]
and the randomized-smoothing radius
\[
R=\sigma\Phi^{-1}(p)
\]
recovers the same margin form. The paper also lists related alternatives,
\[
\Delta_j^{\mathrm{full} } = \frac{|A_j|}{\|\nabla_\theta A_j\|_2}, \qquad \Delta_j^{\mathrm{feat} } = \frac{|A_j|}{\|w\|_2}.
\]
This suggests that the certified sign-preservation radius is best understood as a task-specific margin certificate tailored to policy-gradient update direction rather than to data-space label invariance.

## 4. Worst-case advantage and SignCert-PO

The paper uses the radius not only as a diagnostic but as a training coefficient. For completion \(j\), it defines a worst-case conservative advantage
\[
\tilde{A}_j := \begin{cases} \inf_{w' \in \mathcal{U}_\epsilon^w} A_j(w') & \text{if } A_j(w) > 0, \\[1mm]
\sup_{w' \in \mathcal{U}_\epsilon^w} A_j(w') & \text{if } A_j(w) < 0. \end{cases}
\]
If the nominal reward model wants to reinforce a sample, the adversary makes its advantage as small as possible; if the nominal reward model wants to suppress it, the adversary makes it as large as possible. In both cases, the adversary tries to corrupt the sign-consistent update.

Under the linear-head analysis,
\[
\tilde{A}_j = \rho_j^* \cdot A_j(w), \quad \text{where} \quad \rho_j^* := 1 - \frac{\epsilon}{\Delta_j} = 1 - \frac{\epsilon\,\|h_\psi(x,y^{(j)}) - \bar{h}\|_2}{|A_j(w)|}.
\]
If \(\Delta_j\) is large relative to \(\epsilon\), then \(\rho_j^*\approx 1\). If \(\Delta_j\) is near \(\epsilon\), the weight approaches zero. If \(\Delta_j<\epsilon\), then \(\rho_j^*<0\), and the paper’s interpretation is that non-robust completions are effectively suppressed; its overview figure states that it “suppress[es] completions below \(\Delta=\epsilon\)” [2604.02986].

The resulting policy gradient is
\[
\widehat{\nabla}_\phi J(\phi, \theta) = \mathbb{E}_{x}\left[ \sum_{j=1}^K \rho_j^* \cdot A_j(w) \cdot \nabla_\phi \log\pi_\phi(y^{(j)}|x) \right].
\]
Setting \(\epsilon=0\) recovers standard Dr.GRPO.

Algorithmically, Sign-Certified Policy Optimization requires only the current reward-model parameters, specifically the linear head \(w\) and hidden features \(h_\psi(x,y)\), together with on-policy completions. It does not require multiple reward models, access to the reward-model training dataset at policy optimization time, retraining the reward model, or special uncertainty labels [2604.02986]. For each prompt, it samples \(K\) completions, computes \(h_j\), rewards \(r_j=w^\top h_j+b\), advantages
\[
A_j=r_j-\frac{1}{K}\sum_{k=1}^Kr_k,
\]
then computes
\[
\bar h=\frac1K\sum_k h_k, \qquad \Delta_j=|A_j|/\|h_j-\bar h\|_2,
\]
chooses a perturbation budget \(\epsilon\), forms
\[
\rho_j^* = 1-\epsilon/\Delta_j, \qquad \widetilde A_j = \rho_j^* A_j,
\]
and updates the policy using \(\widetilde A_j\) instead of \(A_j\).

The practical choice in [2604.02986] is to set \(\epsilon\) adaptively from the batch as the \(q_t\)-th quantile of
\[
\left\{\frac{\|h_\psi(x,y^{(j)}) - \bar{h}\|_2}{|A_j(w)|}\right\}_{j=1}^{B\cdot K},
\]
equivalently the \(q_t\)-th quantile of \(\{1/\Delta_j\}\). Higher \(q_t\) therefore yields a larger effective uncertainty budget and more aggressive conservative downweighting.

## 5. Why the certificate is completion-specific

A distinctive feature of the certified sign-preservation radius is that it is defined per completion rather than as a global robustification of the RLHF objective. The paper considers the shared-adversary objective
\[
J_\mathrm{global}(\phi,\theta):=\inf_{w'\in\mathcal{U}_\epsilon^w}J(\phi,\{\psi, w', b\}),
\]
and derives
\[
\nabla_\phi J_{\mathrm{global}(\phi,\theta) = \mathbb{E}_x \left[ \sum_{j=1}^{K} A_j(w)  \nabla_\phi \log \pi_\phi(y^{(j)} | x) \right] - \epsilon  \nabla_\phi \bigl\| \mathbb{E}_{x,y}[h_\psi(x, y)] \bigr\|_2.
\]
The correction is a single completion-agnostic direction. The authors describe this as “uniform pessimism,” since it does not distinguish reliable from unreliable completions [2604.02986].

By contrast, the certified sign-preservation radius is inherently completion-specific. It selectively down-weights fragile gradient terms while leaving robust ones largely unchanged. The paper also notes that a direct robustification of aggregate GRPO advantage is vacuous because group-relative advantages sum to zero, so a standard \(\inf_{w'} \sum_j A_j(w')\) provides no useful discrimination. This is the technical reason the per-sample adversary is both meaningful and tractable, even though it is more conservative than a shared-adversary formulation.

This suggests that the radius is not merely a generic robustness score. It is a certificate aligned to the update geometry of group-relative policy optimization: it measures the stability of the sign decision that determines whether a completion is pushed up or down in probability.

## 6. Empirical behavior, scope, and related usages of “sign preservation”

On TL;DR summarization and AlpacaFarm, SignCert-PO consistently achieves a better win rate than baselines and reduces reward hacking [2604.02986]. The main metric is win rate against a human-generated reference response as judged by a stronger held-out gold reward model. Reported results include the following.

| Benchmark / model | Baselines | SignCert-PO |
|---|---|---|
| TL;DR, Pythia 1B | SFT \(15.7\pm0.2\), Dr.GRPO \(21.0\pm5.4\), UWO \(6.1\pm3.6\), BSPO \(40.4\pm1.2\), AdvPO \(47.0\pm2.6\) | \(60.0\pm2.0\) |
| TL;DR, Qwen2.5 1.5B | SFT \(27.3\pm0.3\), Dr.GRPO \(66.9\pm3.1\), UWO \(59.6\pm5.3\), BSPO \(78.3\pm0.7\), AdvPO \(58.1\pm1.0\) | \(66.0\pm1.9\) |
| AlpacaFarm, Qwen2.5 1.5B | SFT \(37.8\), Dr.GRPO \(32.6\), UWO \(38.3\), BSPO \(31.3\), AdvPO \(37.5\) | \(47.9\) |

The runtime table on TL;DR reports average wall-clock time per policy optimization step of \(7.54\)s / \(10.67\)s for Dr.GRPO, \(7.48\)s / \(10.76\)s for UWO, \(9.92\)s / \(13.37\)s for BSPO, \(10.93\)s / \(16.32\)s for AdvPO, and \(7.47\)s / \(10.78\)s for SignCert-PO. The paper therefore characterizes the method as nearly identical to Dr.GRPO in wall-clock cost while avoiding multiple reward models or access to reward-model training data during policy optimization [2604.02986].

The empirical argument for the radius itself is twofold. First, SignCert-PO maintains higher proxy reward-model accuracy on on-policy samples as optimization progresses. On TL;DR with Pythia-1B, Dr.GRPO’s proxy reward-model accuracy drops toward \(50\%\) at high KL, whereas SignCert-PO maintains accuracy around \(60\%\). Second, larger \(\Delta_j\) correlates with higher sign-preservation or agreement rates even under broader perturbations than the theorem assumes: the paper reports Spearman correlations of \(0.72\) for whole-model perturbation and \(0.67\) for input-embedding perturbation [2604.02986]. This does not extend the theorem, but it indicates that the head-derived radius captures broader sensitivity structure.

The scope of the formal guarantee remains narrow. The exact certificate is derived only for head-only perturbations under a linear head; the full-parameter analogue
\[
\Delta_j \approx |A_j|/\|\nabla_\theta A_j\|_2
\]
is acknowledged to be computationally infeasible at scale. The method is also tied to **sign robustness**, not full reward accuracy: a sign can be robust and still wrong relative to the true reward if the proxy reward model is systematically biased. The per-sample adversary is more conservative than a shared-adversary robust formulation, and the denominator \(\|h_j-\bar h\|_2\) makes the radius sensitive to feature-space outliers [2604.02986].

Outside RLHF, the phrase **sign-preserving** appears in different senses that should not be conflated with the certified sign-preservation radius. In "Sign-Preserving Property for Some Fourth-Order Elliptic Operators in One Dimension and Radial Symmetry" [1303.2237], the result is a qualitative theorem: under structural hypotheses on fourth-order operators with clamped boundary conditions, nonpositive nontrivial forcing yields a strictly negative solution. The paper does **not** provide a quantitative geometric “certified sign-preservation radius”; its central contribution is a strong sign-preserving property, not a perturbation radius [1303.2237]. Likewise, "Certified Signed Graph Unlearning" [2511.14168] provides certified unlearning, a triadic sign-aware certification region, sensitivity bounds, and utility/error guarantees, but it does **not** define a formal object called a certified sign-preservation radius and does **not** certify that sign predictions are preserved inside a radius [2511.14168].

In current arXiv usage, the certified sign-preservation radius therefore refers most precisely to the RLHF quantity introduced in [2604.02986]:
\[
\Delta_j := \sup\bigl\{\tau \geq 0 : \mathrm{sign}(A_j(\theta')) = \mathrm{sign}(A_j(\theta))\ \forall \theta' \in \mathcal{U}_\tau^\theta\bigr\}.
\]
Under a linear reward head, this becomes the exact closed form
\[
\Delta_j = \frac{|A_j(w)|}{\|h_\psi(x,y^{(j)}) - \bar{h}\|_2},
\]
and underlies a conservative policy-gradient rule that trusts completions whose update directions are robust under reward-model perturbation while suppressing those whose signs are easy to flip.

Source: https://www.emergentmind.com/topics/certified-sign-preservation-radius