---
title: Saturating Additive Rewards for Geometric RL
url: https://www.emergentmind.com/topics/saturating-additive-rewards-sar
type: topic
---

# Saturating Additive Rewards for Geometric RL

Searching arXiv for the cited papers to ground the article.
Saturating Additive Rewards (SAR) is a reward design for reinforcement learning in precision-critical geometric generation in which a residual vector over constraints is decomposed into bounded per-constraint terms rather than first being collapsed by a global norm. In the formulation introduced for geometric constraint solving, SAR is designed to address a failure mode called Outlier Gradient Masking: under rewards such as $\exp(-\|\mathbf{r}\|_2^2)$ or $\exp(-\mathrm{MSE})$, a single badly violated constraint can drive the scalar reward toward zero and thereby suppress policy-gradient signal for all other constraints, including those on which the model is making progress [2606.09278]. SAR replaces that global coupling with an additive construction over individual residuals, so that each constraint contributes independently through a monotonically decreasing bounded kernel. In the reported experiments, this design preserves partial-progress signal, improves hard-tier solving rate relative to MSE-based rewards, and reduces token usage when training a language model to generate geometry-solving code [2606.09278].

## 1. Problem setting and motivation

SAR was introduced in the context of open-ended geometric synthesis from natural language, where a model must emit Python code producing exact coordinates and radii that satisfy many simultaneous geometric constraints [2606.09278]. The setting is explicitly described as precision-critical: outputs are not judged by semantic plausibility alone, but by whether the generated construction satisfies strict geometric constraints as measured by a verifier.

The paper formalizes each candidate solution by a residual vector
\[
\mathbf{r} = [r_1, \ldots, r_C], \qquad r_i \ge 0,
\]
where each residual corresponds to the violation of one constraint [2606.09278]. A standard dense-reward instinct in such settings is to aggregate the residuals through a single global norm or norm-like transformation, for example
\[
\exp(-\|\mathbf{r}\|_2^2), \qquad \exp(-\text{MSE}), \qquad \phi(\|\mathbf{r}\|_p).
\]
The paper identifies a specific weakness of this strategy, termed **Outlier Gradient Masking**: if even one constraint is badly violated, the global norm becomes large, the reward collapses toward zero, and the policy update becomes uninformative because the reward multiplies the log-probability gradient
\[
\nabla_\theta J(\theta) = \mathbb{E}_{\mathbf{x}\sim\mathcal D,\ \mathbf{y}\sim\pi_\theta(\cdot\mid \mathbf{x})} \left[ \mathcal{R}(\mathbf{x},\mathbf{y})\cdot \nabla_\theta \log \pi_\theta(\mathbf{y}\mid \mathbf{x}) \right].
\]
This makes partial correctness difficult to exploit: a sample that satisfies many constraints can still receive nearly zero reward if a small subset of residuals is large [2606.09278].

The paper’s argument is both geometric and optimization-theoretic. It states that signal volume collapses for global norms in high dimensions, and that gradient coupling is global, because sensitivity to one residual depends on the total error level rather than only on that residual [2606.09278]. SAR is proposed to avoid that global veto effect.

## 2. Formal definition

SAR is defined by transforming each residual separately and then summing:
\[
\mathcal{R}_{\mathrm{SAR}}(\mathbf{r}) = \sum_{i=1}^{C} \phi(r_i),
\]
where $\phi : \mathbb{R}_{\ge 0}\to[0,1]$ is monotonically decreasing and satisfies $\lim_{r\to\infty}\phi(r)=0$ [2606.09278]. The essential structural distinction is therefore not merely that SAR is dense, but that it is **additive after per-constraint saturation**, whereas global-norm rewards are **transform-after-aggregation** objectives.

The default instantiation in the paper is the Boltzmann kernel
\[
\phi(r) = e^{-r/T}.
\]
Two other bounded kernels are also mentioned:
\[
\phi(r) = \frac{1}{1 + (r/\gamma)^2}
\]
and
\[
\phi(r) = \frac{1}{1 + e^{k(r-\tau)}}.
\]
This kernel family expresses the “saturating” aspect of SAR: large residuals asymptotically contribute little, but they do not erase positive contributions from other constraints [2606.09278].

The comparison class is the global-norm reward
\[
\mathcal{R}_G = \phi(\|\mathbf{r}\|_p).
\]
Under that formulation, all constraints are coupled through a single scalar norm. Under SAR, they are decoupled at the reward-contribution level. This is the central mathematical distinction.

## 3. Gradient structure and partial-progress preservation

The paper makes the contrast between SAR and global-norm rewards explicit at the gradient level. For a global-norm reward, the magnitude of the derivative with respect to a single residual is
\[
\|\nabla_{r_i}\mathcal{R}_G\| = |\phi'(L)| \cdot \left(\frac{r_i}{L}\right)^{p-1}, \qquad L = \|\mathbf{r}\|_p.
\]
For SAR, the corresponding quantity is
\[
\|\nabla_{r_i}\mathcal{R}_{\mathrm{SAR}}\| = |\phi'(r_i)|.
\]
This means that the global-norm gradient is coupled to the total error $L$, whereas the SAR gradient is local to $r_i$ [2606.09278].

This locality is the basis for the claim that SAR preserves partial progress. If one constraint is badly violated, its own term in $\mathcal{R}_{\mathrm{SAR}}$ goes to zero, but the other constraints retain their contributions. The paper describes this as removal of the veto power that global norms grant to outlier residuals [2606.09278]. In its intended regime, geometric solving is a combinatorial assembly problem in which some constraints are often solved earlier than others; SAR is designed so that these partial substructures remain visible to reinforcement learning rather than being erased by one aggregate failure.

Empirical evidence is reported through an analysis of **3,893 partially correct solutions**. The reward distributions are summarized as follows [2606.09278]:

| Reward scheme | Near-zero collapse | Useful range $[0.1,0.9]$ | Near one |
|---|---:|---:|---:|
| SAR | 2.9% | 97.1% | — |
| MSE | 60.4% | 19.0% | 20.6% |
| MSE+S+D | 63.5% | 36.5% | — |

These numbers are used in the paper to argue that SAR preserves ranking information across partially correct outputs, whereas MSE-based schemes often collapse distinct partial solutions into nearly indistinguishable rewards [2606.09278].

## 4. Composite reward and training objective

The paper is explicit that SAR alone is not sufficient for exact geometric solving. Two additional problems are identified: **reward plateau**, in which a partially correct solution may appear sufficiently good under a dense reward, and **geometric degeneracy**, in which trivial collapsed constructions satisfy many constraints but are not valid solutions [2606.09278].

For that reason, the training objective uses a composite reward:
\[
\mathcal{R} = \frac{w}{C}\sum_{i=1}^{C} e^{-r_i/T} + \mathbb{I}_{\text{suc}} \cdot R_{\text{bonus}} - \min(4, C_{\text{deg}}).
\]
The reported settings are [2606.09278]:

- $w = 6.0$
- $T = 0.1$
- $R_{\text{bonus}} = 4.0$
- $\mathbb{I}_{\text{suc}} = 1$ when
  \[
  \|\mathbf{r}\|_2^2 < 10^{-3}
  \]
- $C_{\text{deg}}$ counts detected degenerate substructures, capped at 4

The paper states that this bounds the reward in $[-4,10]$ and creates a sharp landscape that still requires high precision [2606.09278]. It also reports that reward is clipped to $[-10,10]$ during RL. The best-performing condition is therefore not SAR in isolation, but **SAR+S+D**, meaning SAR together with a sparse success bonus and a degeneracy penalty [2606.09278].

A plausible implication is that SAR functions as a dense shaping component inside a broader verifier-driven reward stack, rather than as a complete objective by itself.

## 5. Verification environment, benchmarks, and empirical results

SAR was evaluated in the PyGeoX environment, a programmable geometric DSL and verifier used both for data generation and reward evaluation [2606.09278]. PyGeoX compiles declarative geometry into symbolic equations and then into differentiable residuals. The paper reports support for about **35 object types**, **38 relationships**, equalities, inequalities, not-equal constraints, and arbitrary algebraic constraints on object properties [2606.09278]. The model does not call PyGeoX as a tool during inference; the verifier is used off-policy for validation and reward computation.

Evaluation is conducted on **PyGeoX-Bench**, a suite of **300 problems** stratified into **100 Easy**, **100 Medium**, and **100 Hard**, with correctness defined by
\[
\|\mathbf{r}\|_2^2 < 10^{-3}.
\]
The metric is **one-shot solving rate (SR)** [2606.09278]. The paper also introduces **PyGeoX-Wild**, an OOD set of **86** problems adapted from a published middle-school geometry benchmark, using human-authored prose, constraint types absent from training, and unseen combinations of geometric identities [2606.09278].

The main ablation results for Qwen3-8B are organized around SFT and RL. The hard-tier results reported in the paper are [2606.09278]:

| Setting | Sparse baseline | SAR+S+D | MSE+S+D | SAR only | MSE only |
|---|---:|---:|---:|---:|---:|
| RL Hard SR | 0.35 | 0.41 | 0.18 | 0.09 | 0.10 |
| SFT Hard SR | 0.23 | 0.32 | 0.04 | 0.23 | 0.22 |

The headline RL comparison is
\[
0.41 / 0.18 \approx 2.3\times,
\]
which the paper reports as **SAR+S+D improves the hard-tier solving rate by 2.3× over MSE+S+D** [2606.09278]. Cross-distribution results on PyGeoX-Wild are also reported:

- **RL, Sparse:** 0.59
- **RL, SAR+S+D:** 0.66
- **RL, MSE+S+D:** 0.60
- **SFT, Sparse:** 0.57
- **SFT, SAR+S+D:** 0.65
- **SFT, MSE+S+D:** 0.48 [2606.09278]

The paper further reports token-efficiency gains: SAR+S+D uses about **4,060 tokens/task** versus about **5,260 tokens/task** for the sparse baseline, a **22.8% reduction** [2606.09278]. This is interpreted as evidence that the model reaches geometric solutions more directly.

A context table also places the resulting **Qwen3-8B-RL** system against larger frontier systems on the Hard tier: **DeepSeek-V3.2: 0.25**, **Proprietary-A: 0.37**, **Proprietary-B: 0.38**, **Proprietary-C: 0.51**, and **Qwen3-8B-RL (Ours): 0.41** [2606.09278]. The paper presents this as calibration rather than as a controlled comparison.

## 6. Limitations, scope, and implementation conditions

Several caveats are integral to the interpretation of SAR. First, the paper states explicitly that **SAR alone is not sufficient**. In RL, **SAR only** reaches Hard SR **0.09**, compared with **0.41** for **SAR+S+D**; in SFT, **SAR only** reaches **0.23**, compared with **0.32** for **SAR+S+D** [2606.09278]. The dense additive signal improves learning from partial progress, but an exact-solution threshold still requires the sparse success bonus, while avoidance of trivial constructions requires the degeneracy penalty.

Second, the setting is restricted to **2D static geometry** [2606.09278]. Extending SAR’s demonstrated utility to 3D CAD or kinematic synthesis is not established in the paper; additional symbolic primitives would be required.

Third, the reported experiments depend on a relatively strong base model. The paper states that experiments with **Qwen3-1.7B** and **Llama-3.1-8B-Instruct** yielded near-zero performance and unstable training across reward settings [2606.09278]. This suggests that the effect of SAR is conditional on the model already possessing sufficient mathematical and instruction-following competence.

Fourth, the training setup is specific: **Qwen3-8B** as the main base model, **SFT with LoRA** using $r=8$, $\alpha=32$, dropout $0.05$, and **RL with GRPO via OpenRLHF**, using **10k medium-difficulty problems**, **8 rollouts per problem**, a **90-second sandbox execution timeout**, and **numpy, scipy, sympy** in the sandbox [2606.09278]. Benchmarking is **one-shot**, so solving rates are not boosted by repeated sampling.

A plausible implication is that SAR should be understood as a verifier-compatible reward shaping strategy whose efficacy depends materially on the residual decomposition, the success criterion, and the surrounding training recipe.

## 7. Relation to other reward formulations and terminological distinctions

SAR is most directly contrasted in its source paper with MSE- and SSE-based rewards such as $\exp(-\mathrm{MSE})$, which the paper describes as the natural dense baseline for geometry solvers [2606.09278]. The conceptual distinction is that those rewards aggregate first and saturate later, whereas SAR saturates locally and aggregates afterward.

The broader RL literature on reward aggregation is relevant but distinct. “Recursive Reward Aggregation” develops an algebraic framework in which one keeps the per-step reward fixed and changes how the reward sequence is aggregated over time, encompassing discounted sum, discounted max, discounted min, mean, variance, top-$k$, range, and Sharpe ratio [2507.08537]. That paper states that SAR is **not explicitly discussed** and **not one of the paper’s showcased operators**, though a saturating or clipped additive accumulator could be expressible in principle if encoded through an appropriate recursive update rule and post-processing map [2507.08537]. This suggests a conceptual relationship at the level of aggregation semantics, but not an explicit equivalence.

SAR should also be distinguished from **Self-Aligned Reward**, another “SAR” acronym introduced for reasoning LLMs. Self-Aligned Reward is defined as a relative perplexity difference between an answer conditioned on the query and the standalone answer, and is used to complement verifiable rewards in PPO and GRPO [2509.05489]. Despite the shared acronym, it is a different construction, operates on query-conditioned language-model perplexity rather than per-constraint residuals, and targets efficiency–correctness trade-offs in reasoning rather than precision-critical geometric generation.

A further acronym collision occurs in network economics, where **SAR** denotes **Subscription-Aware Rewarding**, a Stackelberg-game scheme under which only subscribers can earn mobile-data rewards by watching ads [1911.12023]. That usage is unrelated to saturating additive rewards in reinforcement learning.

These distinctions matter because “SAR” is not a unique technical label across arXiv. In current usage, **Saturating Additive Rewards** refers specifically to the bounded per-constraint additive reward design introduced for geometric residual learning [2606.09278].

## 8. Significance

Within its stated scope, SAR is a reward-design response to a general difficulty in verifier-based RL for structured generation: scalar global objectives can be too brittle when correctness is conjunctive and partial progress is common. The paper’s formal argument is that global norms lose informative signal in high-dimensional residual spaces and couple gradient magnitude to unrelated errors, whereas SAR preserves locality by assigning each constraint an independent bounded contribution [2606.09278].

Its practical significance in the reported experiments lies in three linked outcomes: improved learning from partially correct outputs, better exact-solution rates when combined with sparse and degeneracy terms, and lower token consumption [2606.09278]. This suggests that, in domains where solutions are evaluated by multi-constraint residual vectors rather than by single scalar scores, the choice between aggregate-first and decompose-first reward design can materially affect both optimization dynamics and inference behavior.

The current evidence is specific to PyGeoX and 2D static geometry, but the underlying design principle is clear in the source formulation: if the task exposes verifier residuals at the level of individual constraints, then a bounded additive reward can preserve signal that global norms may erase [2606.09278].

Source: https://www.emergentmind.com/topics/saturating-additive-rewards-sar