Soft Reward Optimization

• Soft Reward Optimization is a family of methods that replace hard accept-reject objectives with continuous, graded reward signals for guiding policy learning.
• These approaches tilt a reference distribution using KL-regularized exponential factors, achieving improved performance in domains like protein generation and diffusion-based design.
• The framework balances intrinsic and extrinsic reward signals while addressing key challenges such as reward calibration, uncertainty mitigation, and safe optimization.

Soft Reward Optimization (SRO) denotes a class of optimization procedures in which learning is driven by a softened reward signal rather than by a purely hard accept–reject objective. In the narrowest explicit usage, SRO is an offline algorithm for protein LLMs that optimizes a KL-regularized policy objective with continuous proxy rewards, yielding an exponentially tilted policy of the form $\pi^*(y\mid x)=Z(x)^{-1}\pi_{\mathrm{ref}}(y\mid x)\exp(r(x,y)/\beta)$ (Li et al., 17 Jun 2026). In broader recent usage, the same structural idea appears under other names: diffusion-based design is optimized by KL-regularized soft value guidance (Keramati et al., 2 Aug 2025), preference optimization converges to a softmax over latent rewards under Bradley–Terry assumptions (Sharifnassab et al., 2024), and semi-supervised learning can be interpreted as learning a soft reward over pseudo labels even when the final downstream decision is hard thresholding (Li et al., 2023). The phrase is therefore best understood as a family resemblance rather than a single universally standardized algorithm.

1. Scope, terminology, and boundary cases

The term is not used uniformly across the literature. Some papers instantiate what is naturally read as SRO without naming it explicitly, while others use nearby terminology for conceptually different purposes. The clearest explicit use appears in controllable protein generation, where SRO and BRO are presented as offline algorithms that optimize the classical RLHF objective induced by unsupervised proxy rewards (Li et al., 17 Jun 2026). By contrast, “Soft Preference Optimization” is reward-model-free in derivation, yet under the Bradley–Terry model its minimizer is $\mathrm{Softmax}(r(\cdot\mid x)/\alpha)$, making it reward-like in its asymptotic behavior (Sharifnassab et al., 2024). “SemiReward” is even more partial: it learns a continuous reward model $\mathcal R(x^u,y^u)\in[0,1]$ for pseudo-label quality, but the reward is mostly converted into a binary accept–reject rule for SSL training (Li et al., 2023).

Two recurrent sources of confusion are terminological. In “Learning to Design Soft Hands using Reward Models,” the word “soft” refers to soft robotic hands rather than to softened rewards or entropy-regularized optimization (Bai et al., 20 Oct 2025). In “Differentiable inverse design of short-range order in high-entropy alloys,” the abbreviation SRO denotes short-range order, not soft reward optimization (Ding et al., 2 Jul 2026). These boundary cases are useful because they isolate what is distinctive about reward softness in the machine-learning sense: the reward remains graded, decomposed, KL-regularized, uncertainty-penalized, or otherwise non-binary during optimization, even if the final deployed decision rule may later be hardened.

2. Canonical mathematical structure

A recurring formulation is KL-regularized reward maximization against a reference policy. In the explicit protein-language-model formulation, the objective is

$\max_{\pi_\theta}\; \mathbb{E}_{x\sim \mathcal D,\; y\sim \pi_\theta(\cdot\mid x)} [r(x,y)] \;-\; \beta D_{\mathrm{KL}}\!\left[\pi_\theta(y\mid x)\,\|\,\pi_{\mathrm{ref}}(y\mid x)\right],$

with optimal policy

$\pi^*(y\mid x) = \frac{1}{Z(x)}\, \pi_{\mathrm{ref}}(y\mid x)\, \exp\!\left(\frac{r(x,y)}{\beta}\right).$

This same exponential tilting of a reference distribution by reward reappears in diffusion design, where the soft-optimal reverse kernel is

$p_t^*(x_{t-1}\mid x_t) := \frac{\exp(v_{t-1}(x_{t-1})/\alpha)\, p_t^{\mathrm{pre}}(x_{t-1}\mid x_t)}{\exp(v_t(x_t)/\alpha)},$

and the soft Bellman relation takes the exponential form

$\exp(v_t(x_t)/\alpha)=\mathbb E\!\left[\exp(r(x_0)/\alpha)\mid x_t\right].$

There the soft value is explicitly a log-moment-generating object rather than a plain expected reward (Keramati et al., 2 Aug 2025).

A second recurrent structure is reward-softmax equivalence. In Soft Preference Optimization, if the Bradley–Terry model holds with latent rewards $r(\cdot\mid x)$, then the unique minimizer of the $\alpha$-SPO loss is $\mathrm{Softmax}(r(\cdot\mid x)/\alpha)$, reducing to $\mathrm{Softmax}(r(\cdot\mid x)/\alpha)$0 at $\mathrm{Softmax}(r(\cdot\mid x)/\alpha)$1 (Sharifnassab et al., 2024). In combinatorial optimization, preference optimization begins from the entropy-regularized objective $\mathrm{Softmax}(r(\cdot\mid x)/\alpha)$2 and derives the Boltzmann trajectory policy

$\mathrm{Softmax}(r(\cdot\mid x)/\alpha)$3

after which reward differences are reparameterized by log-policy differences and optimized through pairwise preference likelihoods rather than through REINFORCE magnitudes (Pan et al., 13 May 2025).

These formulations suggest that SRO is best characterized not by a single optimizer but by a common geometry: the target distribution is a reward-tilted deformation of a baseline policy or generator, with softness controlled by $\mathrm{Softmax}(r(\cdot\mid x)/\alpha)$4, $\mathrm{Softmax}(r(\cdot\mid x)/\alpha)$5, or an equivalent temperature parameter.

3. Reward construction: continuous, decomposed, and intrinsic signals

The reward itself varies widely across SRO instantiations. In the protein-language-model setting, the reward is unsupervised and task-agnostic, combining intrinsic model uncertainty with extrinsic semantic consistency from protein representation models. The intrinsic part uses negative sequence entropy or normalized entropy, while the extrinsic part uses negative embedding-space distance to other prompt-conditioned samples. The final reward is temperature-dependent: $\mathrm{Softmax}(r(\cdot\mid x)/\alpha)$6 with $\mathrm{Softmax}(r(\cdot\mid x)/\alpha)$7 near the best-performing generation regime (Li et al., 17 Jun 2026).

Another major construction is decomposed checklist reward. Soft-RLVR converts a prompt into a checklist $\mathrm{Softmax}(r(\cdot\mid x)/\alpha)$8, scores each item with an LLM verifier, and defines the soft reward as

$\mathrm{Softmax}(r(\cdot\mid x)/\alpha)$9

The paper formalizes the resulting tradeoff with the strict target $\mathcal R(x^u,y^u)\in[0,1]$0, the partial-credit target $\mathcal R(x^u,y^u)\in[0,1]$1, and the relaxation gap $\mathcal R(x^u,y^u)\in[0,1]$2. It further shows that averaging item-level verifier outputs can reduce reward variance by roughly $\mathcal R(x^u,y^u)\in[0,1]$3, while partial credit introduces a bias term proportional to $\mathcal R(x^u,y^u)\in[0,1]$4 (Dash et al., 27 May 2026).

In semi-supervised learning, the reward is neither human preference nor future utility but pseudo-label quality. SemiReward defines a continuous similarity target

$\mathcal R(x^u,y^u)\in[0,1]$5

with scaled cosine similarity

$\mathcal R(x^u,y^u)\in[0,1]$6

The rewarder architecture computes $\mathcal R(x^u,y^u)\in[0,1]$7 by cross-attention between embedded input features and embedded pseudo labels, so the reward is a learned compatibility score on $\mathcal R(x^u,y^u)\in[0,1]$8, not simply a confidence heuristic (Li et al., 2023).

A third construction dispenses with any external reward model at all. SPRO derives process-level intrinsic rewards from the policy itself through the token-level log-probability ratio

$\mathcal R(x^u,y^u)\in[0,1]$9

and accumulates these into cumulative process rewards

$\max_{\pi_\theta}\; \mathbb{E}_{x\sim \mathcal D,\; y\sim \pi_\theta(\cdot\mid x)} [r(x,y)] \;-\; \beta D_{\mathrm{KL}}\!\left[\pi_\theta(y\mid x)\,\|\,\pi_{\mathrm{ref}}(y\mid x)\right],$0

This makes the reward signal self-guided, stepwise, and explicitly tied to a soft-RL derivation (Fei et al., 2 Jul 2025).

4. Algorithmic realizations

One major realization of SRO is reward-weighted fitting of a generator. In reward-directed diffusion for engineering design, a pretrained DDPM is first fine-tuned by reward-weighted maximum likelihood with weights $\max_{\pi_\theta}\; \mathbb{E}_{x\sim \mathcal D,\; y\sim \pi_\theta(\cdot\mid x)} [r(x,y)] \;-\; \beta D_{\mathrm{KL}}\!\left[\pi_\theta(y\mid x)\,\|\,\pi_{\mathrm{ref}}(y\mid x)\right],$1, and then guided at inference time by “Soft Value-based Decoding in Diffusion model (SVDD).” At each reverse step, multiple candidates are sampled, each candidate’s implied clean design $\max_{\pi_\theta}\; \mathbb{E}_{x\sim \mathcal D,\; y\sim \pi_\theta(\cdot\mid x)} [r(x,y)] \;-\; \beta D_{\mathrm{KL}}\!\left[\pi_\theta(y\mid x)\,\|\,\pi_{\mathrm{ref}}(y\mid x)\right],$2 is estimated, a soft value $\max_{\pi_\theta}\; \mathbb{E}_{x\sim \mathcal D,\; y\sim \pi_\theta(\cdot\mid x)} [r(x,y)] \;-\; \beta D_{\mathrm{KL}}\!\left[\pi_\theta(y\mid x)\,\|\,\pi_{\mathrm{ref}}(y\mid x)\right],$3 is computed, and one candidate is resampled according to weights proportional to $\max_{\pi_\theta}\; \mathbb{E}_{x\sim \mathcal D,\; y\sim \pi_\theta(\cdot\mid x)} [r(x,y)] \;-\; \beta D_{\mathrm{KL}}\!\left[\pi_\theta(y\mid x)\,\|\,\pi_{\mathrm{ref}}(y\mid x)\right],$4 (Keramati et al., 2 Aug 2025). This is a direct diffusion-specific instance of reward-tilted generation.

A second realization is preference optimization as a tractable surrogate for soft reward optimization. In Soft Preference Optimization, the model’s induced pairwise preference probability is

$\max_{\pi_\theta}\; \mathbb{E}_{x\sim \mathcal D,\; y\sim \pi_\theta(\cdot\mid x)} [r(x,y)] \;-\; \beta D_{\mathrm{KL}}\!\left[\pi_\theta(y\mid x)\,\|\,\pi_{\mathrm{ref}}(y\mid x)\right],$5

and the $\max_{\pi_\theta}\; \mathbb{E}_{x\sim \mathcal D,\; y\sim \pi_\theta(\cdot\mid x)} [r(x,y)] \;-\; \beta D_{\mathrm{KL}}\!\left[\pi_\theta(y\mid x)\,\|\,\pi_{\mathrm{ref}}(y\mid x)\right],$6-parameterized loss replaces raw probabilities by $\max_{\pi_\theta}\; \mathbb{E}_{x\sim \mathcal D,\; y\sim \pi_\theta(\cdot\mid x)} [r(x,y)] \;-\; \beta D_{\mathrm{KL}}\!\left[\pi_\theta(y\mid x)\,\|\,\pi_{\mathrm{ref}}(y\mid x)\right],$7, so the softness parameter directly controls the entropy of the resulting policy (Sharifnassab et al., 2024). In combinatorial optimization, the reward function is reparameterized as $\max_{\pi_\theta}\; \mathbb{E}_{x\sim \mathcal D,\; y\sim \pi_\theta(\cdot\mid x)} [r(x,y)] \;-\; \beta D_{\mathrm{KL}}\!\left[\pi_\theta(y\mid x)\,\|\,\pi_{\mathrm{ref}}(y\mid x)\right],$8, and the optimizer learns from pairwise comparisons $\max_{\pi_\theta}\; \mathbb{E}_{x\sim \mathcal D,\; y\sim \pi_\theta(\cdot\mid x)} [r(x,y)] \;-\; \beta D_{\mathrm{KL}}\!\left[\pi_\theta(y\mid x)\,\|\,\pi_{\mathrm{ref}}(y\mid x)\right],$9 rather than from vanishing reward magnitudes (Pan et al., 13 May 2025).

A third realization keeps the reward soft but operationalizes it through hard selection. SemiReward replaces the handcrafted selector $\pi^*(y\mid x) = \frac{1}{Z(x)}\, \pi_{\mathrm{ref}}(y\mid x)\, \exp\!\left(\frac{r(x,y)}{\beta}\right).$0 with a learned rewarder $\pi^*(y\mid x) = \frac{1}{Z(x)}\, \pi_{\mathrm{ref}}(y\mid x)\, \exp\!\left(\frac{r(x,y)}{\beta}\right).$1, yet the actual student update uses

$\pi^*(y\mid x) = \frac{1}{Z(x)}\, \pi_{\mathrm{ref}}(y\mid x)\, \exp\!\left(\frac{r(x,y)}{\beta}\right).$2

The rewarder is trained online in two stages with a generator $\pi^*(y\mid x) = \frac{1}{Z(x)}\, \pi_{\mathrm{ref}}(y\mid x)\, \exp\!\left(\frac{r(x,y)}{\beta}\right).$3 and a subsampling strategy, and after the warm-up stage the batch-mean reward is used as a dynamic threshold for pseudo-label acceptance (Li et al., 2023). This is a partial SRO instance: the reward model is soft, but the downstream policy is largely hard.

The explicit protein SRO algorithm is fully offline. It builds a dataset

$\pi^*(y\mid x) = \frac{1}{Z(x)}\, \pi_{\mathrm{ref}}(y\mid x)\, \exp\!\left(\frac{r(x,y)}{\beta}\right).$4

normalizes rewards within each prompt group, and minimizes

$\pi^*(y\mid x) = \frac{1}{Z(x)}\, \pi_{\mathrm{ref}}(y\mid x)\, \exp\!\left(\frac{r(x,y)}{\beta}\right).$5

whose closed-form optimizer satisfies

$\pi^*(y\mid x) = \frac{1}{Z(x)}\, \pi_{\mathrm{ref}}(y\mid x)\, \exp\!\left(\frac{r(x,y)}{\beta}\right).$6

With $\pi^*(y\mid x) = \frac{1}{Z(x)}\, \pi_{\mathrm{ref}}(y\mid x)\, \exp\!\left(\frac{r(x,y)}{\beta}\right).$7, zero-centered normalized reward leaves the policy locally unchanged (Li et al., 17 Jun 2026).

5. Robustness, misspecification, and controlled optimization pressure

A central issue in SRO is not only how to densify reward, but how to prevent overoptimization against an imperfect proxy. Reward-model ensembles are one practical answer. In RLHF, worst-case optimization uses $\pi^*(y\mid x) = \frac{1}{Z(x)}\, \pi_{\mathrm{ref}}(y\mid x)\, \exp\!\left(\frac{r(x,y)}{\beta}\right).$8, while uncertainty-weighted optimization uses

$\pi^*(y\mid x) = \frac{1}{Z(x)}\, \pi_{\mathrm{ref}}(y\mid x)\, \exp\!\left(\frac{r(x,y)}{\beta}\right).$9

In the reported synthetic-overseer experiments, conservative optimization practically eliminates overoptimization and improves performance by up to 70% for best-of-$p_t^*(x_{t-1}\mid x_t) := \frac{\exp(v_{t-1}(x_{t-1})/\alpha)\, p_t^{\mathrm{pre}}(x_{t-1}\mid x_t)}{\exp(v_t(x_t)/\alpha)},$0 sampling, while in PPO the combination of ensemble-based conservative optimization with a small KL penalty prevents overoptimization at no performance cost (Coste et al., 2023).

A more formal answer is regret-robust reward optimization. DRRO replaces worst-case value with worst-case regret relative to the best policy under the same plausible reward perturbation. In the promptwise $p_t^*(x_{t-1}\mid x_t) := \frac{\exp(v_{t-1}(x_{t-1})/\alpha)\, p_t^{\mathrm{pre}}(x_{t-1}\mid x_t)}{\exp(v_t(x_t)/\alpha)},$1-ambiguity case, the paper derives the soft relaxed objective

$p_t^*(x_{t-1}\mid x_t) := \frac{\exp(v_{t-1}(x_{t-1})/\alpha)\, p_t^{\mathrm{pre}}(x_{t-1}\mid x_t)}{\exp(v_t(x_t)/\alpha)},$2

whose gradient induces a sampled robust bonus on top of the nominal reward (Wang et al., 30 Apr 2026). This makes the optimization softer than standard DRO not by adding entropy directly, but by redistributing optimization pressure toward regret-vulnerable responses instead of globally flattening the policy.

Self-generated soft rewards create a different pathology: reward inflation. In Soft-SVeRL, the policy also acts as verifier, and naive self-verification causes an always-yes collapse in which $p_t^*(x_{t-1}\mid x_t) := \frac{\exp(v_{t-1}(x_{t-1})/\alpha)\, p_t^{\mathrm{pre}}(x_{t-1}\mid x_t)}{\exp(v_t(x_t)/\alpha)},$3 for nearly every checklist item. The paper stabilizes this with verifier co-training, replay pseudo-labeling, repeated verifier voting, and an anti-collapse partition penalty, showing that self-verified soft reward optimization is possible but fragile (Dash et al., 27 May 2026). Safe RL exposes another variant of the same problem: reward improvement and safety improvement may have conflicting gradients. PCRPO responds with soft switching, projecting conflicting reward and safety gradients onto each other’s normal planes and using slack bands around the constraint boundary to move among reward-only, mixed, and safety-only regimes (Gu et al., 2024).

Taken together, these works show that softened rewards do not by themselves solve Goodharting, verifier permissiveness, or safety interference. They shift the problem from reward sparsity to reward calibration, uncertainty, and optimization geometry.

6. Empirical domains, observed gains, and open limits

Empirical evidence spans a wide range of domains. In engineering design, reward-directed diffusion produces samples that go beyond the training distribution and reports a greater 25 percent reduction in resistance for ship design and over 10 percent improvement in lift-to-drag ratio for 2D airfoils (Keramati et al., 2 Aug 2025). In protein generation, explicit SRO improves average pass@1 on the Pfam700 compositional OOD benchmark from 0.281 to 0.427 for the 151M ProGen2 model and from 0.385 to 0.602 for the 764M model, essentially matching the oracle average of 0.603 in the larger setting (Li et al., 17 Jun 2026). In partially verifiable instruction following, checklist-based Soft-RLVR improves IFEval by up to 11.1 points using only learned verifier rewards (Dash et al., 27 May 2026). In process reinforcement learning for reasoning, SPRO reports 3.4x higher training efficiency, a 17.5% test accuracy improvement, stable elevated policy entropy, and average response length reduced by approximately $p_t^*(x_{t-1}\mid x_t) := \frac{\exp(v_{t-1}(x_{t-1})/\alpha)\, p_t^{\mathrm{pre}}(x_{t-1}\mid x_t)}{\exp(v_t(x_t)/\alpha)},$4 (Fei et al., 2 Jul 2025).

Evidence for partial or adjacent SRO also appears in semi-supervised learning and combinatorial optimization. SemiReward reports improvements across 13 SSL benchmarks, including faster convergence and gains over Pseudo Label, FlexMatch, and Free/SoftMatch; on CIFAR-100 with 400 labels, cosine-similarity reward yields $p_t^*(x_{t-1}\mid x_t) := \frac{\exp(v_{t-1}(x_{t-1})/\alpha)\, p_t^{\mathrm{pre}}(x_{t-1}\mid x_t)}{\exp(v_t(x_t)/\alpha)},$5 accuracy versus $p_t^*(x_{t-1}\mid x_t) := \frac{\exp(v_{t-1}(x_{t-1})/\alpha)\, p_t^{\mathrm{pre}}(x_{t-1}\mid x_t)}{\exp(v_t(x_t)/\alpha)},$6 for $p_t^*(x_{t-1}\mid x_t) := \frac{\exp(v_{t-1}(x_{t-1})/\alpha)\, p_t^{\mathrm{pre}}(x_{t-1}\mid x_t)}{\exp(v_t(x_t)/\alpha)},$7-derived targets, and $p_t^*(x_{t-1}\mid x_t) := \frac{\exp(v_{t-1}(x_{t-1})/\alpha)\, p_t^{\mathrm{pre}}(x_{t-1}\mid x_t)}{\exp(v_t(x_t)/\alpha)},$8 is best in the two-stage schedule ablation (Li et al., 2023). Preference Optimization for combinatorial optimization reports that PO converges 1.5x to 2.5x faster than REINFORCE baselines on TSP-100 and improves solution gaps for TSP, CVRP, and FFSP, while its local-search fine-tuning treats improved trajectories as preference winners rather than as post-processing only (Pan et al., 13 May 2025).

The limitations are correspondingly heterogeneous. SemiReward does not preserve soft reward information all the way into the student loss, since the final update is thresholded (Li et al., 2023). Soft-RLVR assumes tasks can be decomposed into atomic checklist items and shows that poor checklist quality directly degrades downstream RL (Dash et al., 27 May 2026). Diffusion-based SRO depends strongly on reward quality and approximates the soft value by $p_t^*(x_{t-1}\mid x_t) := \frac{\exp(v_{t-1}(x_{t-1})/\alpha)\, p_t^{\mathrm{pre}}(x_{t-1}\mid x_t)}{\exp(v_t(x_t)/\alpha)},$9, which may be inaccurate early in denoising (Keramati et al., 2 Aug 2025). Protein SRO remains fully in silico and can reweight existing capabilities more readily than create entirely new ones (Li et al., 17 Jun 2026). These limits suggest that SRO is not a single solution to reward design, but a general strategy for replacing brittle hard supervision with smoother reward-mediated optimization, whose success depends on the fidelity of the reward signal and on how carefully optimization pressure is regularized.