Binarized Reward Optimization (BRO)

• BRO is a framework that recasts binary supervision as an expected reward maximization problem, defining actions as binary decisions in a one-step RL setting.
• Its methodology employs tailored objectives like BinaryPPO, Beta normalization, and Bayesian estimators to stabilize reward signals and mitigate variance issues.
• Empirical results show significant gains in classification accuracy, improved reasoning metrics, enhanced protein design outputs, and robust performance in offline RL tasks.

Binarized Reward Optimization (BRO) denotes a family of policy-optimization schemes in which supervision is represented as binary or binarized reward signals and the learning problem is reformulated as reward maximization rather than direct likelihood fitting. In current arXiv usage, the term spans several closely related settings: offline binary decision learning for LLM classifiers, where BinaryPPO is presented as an explicit instantiation of BRO; group-based reinforcement learning with verifiable pass/fail rewards for reasoning models; offline post-training of protein LLMs using positive/negative labels derived from proxy rewards; and offline preference-based reinforcement learning in which pairwise preferences are converted into per-step binary rewards (Pandey et al., 2 Feb 2026, Kim et al., 19 Mar 2026, Li et al., 17 Jun 2026, Xu et al., 2024).

1. Formal scope and problem classes

In the binary-classification formulation, BRO recasts supervised binary prediction as a one-step reinforcement-learning problem. The input text is the state $x$, the binary decision is the action $a \in \{0,1\}$, the policy is a stochastic classifier $\pi_\theta(a\mid x)$, and the objective is the expected reward

$$J(\theta)=\mathbb{E}_{x\sim\mathcal D,\;a\sim\pi_\theta(\cdot\mid x)}[r(x,a,y)].$$

BinaryPPO emphasizes that this setting is strictly offline: there is no online environment interaction, all learning occurs from a static dataset $\mathcal D=\{(x_i,y_i)\}_{i=1}^N$, and the reward is computed from $(x,y)$ and the model’s own predicted probability $\pi(a\mid x)$ (Pandey et al., 2 Feb 2026).

In RLVR, BRO appears as on-policy, bandit-like post-training with verifiable rewards. For each prompt $q$, the policy samples responses $o \sim \pi_\theta(\cdot\mid q)$, and a verifier returns a Bernoulli reward

$$r(o,q)= \begin{cases} 1 & \text{if the response contains the correct answer}\ 0 & \text{otherwise.} \end{cases}$$

GRPO is treated as a canonical BRO method because it uses groups of sampled responses per prompt, binary rewards, and group-relative normalized advantages (Kim et al., 19 Mar 2026).

In unsupervised protein design, BRO is embedded in a KL-regularized RLHF objective

$a \in \{0,1\}$0

but the reward is not human-labeled. Instead, it is derived from task-agnostic proxy scores based on intrinsic model uncertainty and extrinsic semantic consistency from protein representation models. BRO then uses binary labels over generated sequences, rather than continuous rewards, to steer the fine-tuned model relative to the reference model $a \in \{0,1\}$1 (Li et al., 17 Jun 2026).

In offline preference-based RL, binary reward labeling converts trajectory preferences into scalar rewards on transitions. For a preference dataset $a \in \{0,1\}$2, the non-overlapping case admits an exact binary solution: every step in a chosen trajectory receives $a \in \{0,1\}$3, and every step in a rejected trajectory receives $a \in \{0,1\}$4. This reduction allows any reward-based offline RL algorithm to be applied without modifying the underlying optimizer (Xu et al., 2024).

2. Objective functions and canonical formulations

BinaryPPO instantiates BRO through a PPO-style objective adapted to a binary action space and an offline dataset. Its core reward is

$a \in \{0,1\}$5

where $a \in \{0,1\}$6 if $a \in \{0,1\}$7 and $a \in \{0,1\}$8 if $a \in \{0,1\}$9. The policy update uses the clipped PPO surrogate with importance ratio

$\pi_\theta(a\mid x)$0

and a one-step value baseline $\pi_\theta(a\mid x)$1 with advantage

$\pi_\theta(a\mid x)$2

The same framework includes a value loss, a supervised CE regularizer, and entropy regularization, yielding a trust-region-style offline BRO objective rather than naive 0/1 reward maximization (Pandey et al., 2 Feb 2026).

In the protein-language setting, BRO is a binary logistic objective against a reference policy. Sequences are binarized into positive and negative sets, $\pi_\theta(a\mid x)$3 and $\pi_\theta(a\mid x)$4, by ranking generated samples with a proxy reward and keeping only the top-4 and bottom-4 sequences per prompt. The practical loss is

$\pi_\theta(a\mid x)$5

where $\pi_\theta(a\mid x)$6 for $\pi_\theta(a\mid x)$7 and $\pi_\theta(a\mid x)$8 for $\pi_\theta(a\mid x)$9. Positives are thus pushed to have higher relative likelihood under $$J(\theta)=\mathbb{E}_{x\sim\mathcal D,\;a\sim\pi_\theta(\cdot\mid x)}[r(x,a,y)].$$0 than under $$J(\theta)=\mathbb{E}_{x\sim\mathcal D,\;a\sim\pi_\theta(\cdot\mid x)}[r(x,a,y)].$$1, and negatives are pushed in the opposite direction (Li et al., 17 Jun 2026).

In offline preference-based RL, BRO appears in an even more literal form. Under the no-overlap assumption, the optimal binary labels are

$$J(\theta)=\mathbb{E}_{x\sim\mathcal D,\;a\sim\pi_\theta(\cdot\mid x)}[r(x,a,y)].$$2

and standard reward-based offline RL is then run on the labeled dataset. For Q-learning, the per-pair Bellman objective becomes

$$J(\theta)=\mathbb{E}_{x\sim\mathcal D,\;a\sim\pi_\theta(\cdot\mid x)}[r(x,a,y)].$$3

This construction makes BRO a general interface between pairwise preferences and reward-based offline RL algorithms such as CQL, IQL, MOPO, and COMBO (Xu et al., 2024).

Taken together, these formulations show that BRO is not a single optimizer. It is a design pattern in which binary supervision is converted into either expected-return maximization with a binarized reward or a logistic comparison between a trainable policy and a reference policy.

3. Reward shaping, estimation, and normalization

A central theme in BRO research is that binary supervision rarely remains a raw $$J(\theta)=\mathbb{E}_{x\sim\mathcal D,\;a\sim\pi_\theta(\cdot\mid x)}[r(x,a,y)].$$4 signal inside the optimizer. BinaryPPO explicitly contrasts its confidence-weighted reward with naive

$$J(\theta)=\mathbb{E}_{x\sim\mathcal D,\;a\sim\pi_\theta(\cdot\mid x)}[r(x,a,y)].$$5

arguing that hard 0/1 rewards are flat with respect to confidence and poorly calibrated for PPO. Its logarithmic shaping $$J(\theta)=\mathbb{E}_{x\sim\mathcal D,\;a\sim\pi_\theta(\cdot\mid x)}[r(x,a,y)].$$6 yields a reward that is binary in correctness through $$J(\theta)=\mathbb{E}_{x\sim\mathcal D,\;a\sim\pi_\theta(\cdot\mid x)}[r(x,a,y)].$$7 but continuous in the policy’s own probability, thereby penalizing overconfident errors and giving ambiguous cases smaller magnitudes (Pandey et al., 2 Feb 2026).

BNPO addresses the same binary-reward regime from a distributional-normalization perspective. It models the per-question success probability $$J(\theta)=\mathbb{E}_{x\sim\mathcal D,\;a\sim\pi_\theta(\cdot\mid x)}[r(x,a,y)].$$8 as a Beta random variable and defines the normalized advantage

$$J(\theta)=\mathbb{E}_{x\sim\mathcal D,\;a\sim\pi_\theta(\cdot\mid x)}[r(x,a,y)].$$9

where $\mathcal D=\{(x_i,y_i)\}_{i=1}^N$0 is a Beta density. Under the assumption that $\mathcal D=\{(x_i,y_i)\}_{i=1}^N$1, Theorem 1 gives the variance-minimizing normalization parameters

$\mathcal D=\{(x_i,y_i)\}_{i=1}^N$2

REINFORCE with baseline is recovered by $\mathcal D=\{(x_i,y_i)\}_{i=1}^N$3, while GRPO corresponds to $\mathcal D=\{(x_i,y_i)\}_{i=1}^N$4 up to a constant factor, so BNPO generalizes both under binary-valued rewards (Xiao et al., 3 Jun 2025).

DBB reformulates group-based RLVR as finite-sample Bernoulli estimation. For each prompt and epoch, rewards are modeled as $\mathcal D=\{(x_i,y_i)\}_{i=1}^N$5, and the usual group mean and variance are replaced by discounted Beta-Bernoulli statistics

$\mathcal D=\{(x_i,y_i)\}_{i=1}^N$6

The estimator

$\mathcal D=\{(x_i,y_i)\}_{i=1}^N$7

is biased but lower-variance than the point estimator, and its variance is multiplied by $\mathcal D=\{(x_i,y_i)\}_{i=1}^N$8. Because the Beta counts remain strictly positive, DBB avoids estimated variance collapse in small-$\mathcal D=\{(x_i,y_i)\}_{i=1}^N$9 groups and provides a non-degenerate advantage signal even when all current rollouts succeed or fail (Kim et al., 19 Mar 2026).

ReDit changes the reward itself by injecting zero-mean noise,

$(x,y)$0

with $(x,y)$1 or $(x,y)$2. The expected gradient remains unbiased,

$(x,y)$3

but the reward distribution becomes continuous at training time. This produces non-zero gradients in flat all-zero or all-one regions and softens abrupt 0-to-1 transitions that otherwise generate extreme normalized advantages (Wei et al., 23 Jun 2025).

4. Stability mechanisms and optimization dynamics

BRO methods devote substantial machinery to preventing policy collapse, variance collapse, and pathological gradient behavior. In BinaryPPO, three stabilizers are explicit: PPO clipping, entropy regularization, and supervised CE regularization. The full loss is

$(x,y)$4

The ablation “Removing Entropy Regularization” collapses performance across tasks: on Qwen-2.5-3B, average accuracy drops from approximately $(x,y)$5 to approximately $(x,y)$6, with a similar collapse on Gemma-2-2B. BinaryPPO also reports that advantage peaks at intermediate confidence $(x,y)$7–$(x,y)$8, reflecting the interaction among confidence-weighted reward, value baseline, and PPO clipping (Pandey et al., 2 Feb 2026).

Group-based BRO for RLVR faces a different instability: empirical variance collapse. In naive GRPO, if all rewards in a group are identical, the sample variance is zero and the normalized advantages become undefined or are practically set to zero. DBB addresses this statistically rather than heuristically. Its discounted Beta posterior ensures $(x,y)$9, so the posterior variance estimate remains positive in finite training. The practical effect is that zero-variance groups cease to be dead batches, and the reward estimator becomes more sample-efficient in the low-rollout regime typical of RLVR (Kim et al., 19 Mar 2026).

ReDit diagnoses “gradient anomaly” under discrete rewards: gradient vanishing when batches are uniform and gradient explosion when a few samples flip from $\pi(a\mid x)$0 to $\pi(a\mid x)$1. It reports many red vanishing-gradient points and purple exploding-gradient spikes without dithering, and much smoother gradient norms when noise is added. BNPO reaches a related conclusion from a different angle: adaptive Beta normalization yields the most stable gradient norms over time relative to REINFORCE, GRPO, and REINFORCE++, supporting the claim that binary-reward training requires normalization that tracks the evolving distribution of success probabilities (Wei et al., 23 Jun 2025, Xiao et al., 3 Jun 2025).

A broader implication is that BRO systems are usually engineered around variance control rather than reward definition alone. The binary signal identifies success or failure, but stable learning depends on how that signal is centered, normalized, smoothed, or regularized.

5. Empirical domains and reported results

Reported BRO results span classification, reasoning, protein design, and offline control, and they differ not only in optimizer but also in how the binary signal is constructed and used (Pandey et al., 2 Feb 2026, Kim et al., 19 Mar 2026, Xiao et al., 3 Jun 2025, Wei et al., 23 Jun 2025, Li et al., 17 Jun 2026, Xu et al., 2024).

Domain	Representative BRO formulation	Reported results
Binary classification	BinaryPPO	Up to $\pi(a\mid x)$2 accuracy; typically $\pi(a\mid x)$3–$\pi(a\mid x)$4 percentage-point improvements over SFT
RLVR reasoning	GRPO-DBB, BNPO, ReDit	DBB: average Acc@8 gains of $\pi(a\mid x)$5 in-distribution and $\pi(a\mid x)$6 out-of-distribution on 1.7B/8B; ReDit reaches GRPO-like performance with approximately $\pi(a\mid x)$7 training steps
Protein design	BRO for PLMs	Pfam700 average keyword recovery: $\pi(a\mid x)$8 vs base $\pi(a\mid x)$9 on 151M; $q$0 vs base $q$1 on 764M
Offline preference-based RL	BRL + offline RL	D4RL total score $q$2 vs Oracle $q$3, RM $q$4, IPL $q$5

BinaryPPO reports gains across eight binary classification benchmarks, including CLadder, SciRIFF, BoolQ, FEVER, IMDB, OpenAI Moderation, Detect-Jailbreak, and JailbreakBench. A concrete example on Qwen-2.5-3B for SciRIFF is: Base $q$6, SFT(20) $q$7, PPO $q$8, and BinaryPPO $q$9–$o \sim \pi_\theta(\cdot\mid q)$0. Its OOD tests include reconstructed CLadder, where Gemma-2-2B reaches $o \sim \pi_\theta(\cdot\mid q)$1 OOD accuracy and Qwen reaches $o \sim \pi_\theta(\cdot\mid q)$2, and cross-dataset moderation transfer between JB and OpenAI Moderation that remains significantly above chance (Pandey et al., 2 Feb 2026).

In RLVR reasoning, GRPO-DBB is evaluated on DAPO-Math-17k with 8 rollouts per prompt. On six in-distribution benchmarks, the 1.7B model improves from $o \sim \pi_\theta(\cdot\mid q)$3 to $o \sim \pi_\theta(\cdot\mid q)$4 average Acc@8 and the 8B model from $o \sim \pi_\theta(\cdot\mid q)$5 to $o \sim \pi_\theta(\cdot\mid q)$6. On three out-of-distribution benchmarks, gains are larger: $o \sim \pi_\theta(\cdot\mid q)$7 to $o \sim \pi_\theta(\cdot\mid q)$8 for 1.7B and $o \sim \pi_\theta(\cdot\mid q)$9 to $$r(o,q)= \begin{cases} 1 & \text{if the response contains the correct answer}\ 0 & \text{otherwise.} \end{cases}$$0 for 8B. BNPO reports the best average performance across four math benchmarks for Qwen2.5-Math-7B, with average score $$r(o,q)= \begin{cases} 1 & \text{if the response contains the correct answer}\ 0 & \text{otherwise.} \end{cases}$$1 versus $$r(o,q)= \begin{cases} 1 & \text{if the response contains the correct answer}\ 0 & \text{otherwise.} \end{cases}$$2 for REINFORCE, $$r(o,q)= \begin{cases} 1 & \text{if the response contains the correct answer}\ 0 & \text{otherwise.} \end{cases}$$3 for ReMax, $$r(o,q)= \begin{cases} 1 & \text{if the response contains the correct answer}\ 0 & \text{otherwise.} \end{cases}$$4 for GRPO, and $$r(o,q)= \begin{cases} 1 & \text{if the response contains the correct answer}\ 0 & \text{otherwise.} \end{cases}$$5 for REINFORCE++. ReDit reports that it achieves performance comparable to vanilla GRPO with only approximately $$r(o,q)= \begin{cases} 1 & \text{if the response contains the correct answer}\ 0 & \text{otherwise.} \end{cases}$$6 the training steps and still shows a $$r(o,q)= \begin{cases} 1 & \text{if the response contains the correct answer}\ 0 & \text{otherwise.} \end{cases}$$7 performance improvement over vanilla GRPO when trained for a similar duration; on MATH, GRPO ends at $$r(o,q)= \begin{cases} 1 & \text{if the response contains the correct answer}\ 0 & \text{otherwise.} \end{cases}$$8 while ReDit reaches $$r(o,q)= \begin{cases} 1 & \text{if the response contains the correct answer}\ 0 & \text{otherwise.} \end{cases}$$9 (Kim et al., 19 Mar 2026, Xiao et al., 3 Jun 2025, Wei et al., 23 Jun 2025).

For protein LLMs, BRO is evaluated on compositional OOD Func2Seq tasks constructed from seven Pfam families. On the 151M ProGen2 model, the average pass@1 keyword recovery is $a \in \{0,1\}$00 for BRO, $a \in \{0,1\}$01 for SRO, $a \in \{0,1\}$02 for DPO, $a \in \{0,1\}$03 for KTO, and $a \in \{0,1\}$04 for the base model. On the 764M model, BRO reaches $a \in \{0,1\}$05, SRO $a \in \{0,1\}$06, DPO $a \in \{0,1\}$07, KTO $a \in \{0,1\}$08, and the base model $a \in \{0,1\}$09. Figure-level analysis further indicates that BRO and SRO improve pass@k strongly for small $a \in \{0,1\}$10 and then converge toward the base model at large $a \in \{0,1\}$11, which the paper interprets as re-prioritization of controllable sequences rather than severe support collapse (Li et al., 17 Jun 2026).

In offline preference-based RL, binary reward labeling is evaluated on D4RL HalfCheetah, Hopper, and Walker2d tasks. The aggregate total scores are Oracle $a \in \{0,1\}$12, BRL $a \in \{0,1\}$13, RM $a \in \{0,1\}$14, and IPL $a \in \{0,1\}$15. BRL is close to Oracle on several medium-expert tasks, including HalfCheetah ME $a \in \{0,1\}$16 versus Oracle $a \in \{0,1\}$17, Hopper ME $a \in \{0,1\}$18 versus $a \in \{0,1\}$19, and Walker2d ME $a \in \{0,1\}$20 versus $a \in \{0,1\}$21. The paper also reports that BRL usually outperforms reward modeling under overlapping-trajectory settings and degrades more gracefully when the preference dataset is reduced (Xu et al., 2024).

6. Limitations, misconceptions, and open directions

A recurring misconception is that BRO is synonymous with naive hard $a \in \{0,1\}$22 reward optimization. The cited formulations contradict that view. BinaryPPO uses sign-binarized correctness with logarithmic confidence shaping; BNPO replaces static normalization with an adaptive Beta density; DBB uses discounted Bayesian estimation of Bernoulli success probabilities; and ReDit perturbs discrete rewards with zero-mean noise to make the training signal continuous at the batch level (Pandey et al., 2 Feb 2026, Xiao et al., 3 Jun 2025, Kim et al., 19 Mar 2026, Wei et al., 23 Jun 2025).

Another misconception is that BRO is intrinsically online or intrinsically sequence-level. BinaryPPO is strictly offline and operates on one-step binary decisions from a static dataset. BRL is also offline and turns trajectory preferences into per-step $a \in \{0,1\}$23 rewards. By contrast, GRPO-style RLVR is on-policy and sequence-level, while protein BRO is an offline relative-likelihood objective against a reference model. BRO is therefore better understood as a reward-design and optimization family than as a single training protocol (Pandey et al., 2 Feb 2026, Xu et al., 2024, Li et al., 17 Jun 2026).

The limitations are equally domain-specific. DBB is tailored to binary verifiable rewards, assumes per-prompt independence, and uses a constant discount factor $a \in \{0,1\}$24, which may be mismatched to changing non-stationarity across training. BNPO’s theory depends on binary rewards and on an uncorrelation assumption between $a \in \{0,1\}$25 and the normalized reward term. ReDit is explicitly helpful for discrete or sparse rewards and shows no benefit when the reward model is already continuous in $a \in \{0,1\}$26. Protein BRO depends on proxy-reward selection, uses a heuristic top-4/bottom-4 binarization rule, and is validated only in silico. BRL is limited to the offline setting and does not address active preference querying (Kim et al., 19 Mar 2026, Xiao et al., 3 Jun 2025, Wei et al., 23 Jun 2025, Li et al., 17 Jun 2026, Xu et al., 2024).

The current literature identifies several extension paths. These include adaptive or learned dithering, multi-level discrete rewards, confidence-aware binarization, uncertainty-aware optimization, combining unsupervised signals with minimal human feedback, and extending Beta-based theory beyond strict Bernoulli/Beta assumptions. A plausible implication is that future BRO systems will place even greater emphasis on estimator design and reward calibration, rather than treating binary rewards as intrinsically self-sufficient (Wei et al., 23 Jun 2025, Xu et al., 2024, Li et al., 17 Jun 2026, Xiao et al., 3 Jun 2025).