Saturating Additive Rewards for Geometric RL

• Saturating Additive Rewards (SAR) is a reinforcement learning design that decomposes constraint residuals into independent, bounded contributions to mitigate outlier gradient masking.
• By replacing a global-norm aggregation with a per-constraint additive approach, SAR preserves partial progress and robustly guides policy updates in precision-critical settings.
• Empirical results demonstrate that SAR combined with sparse bonuses and degeneracy penalties improves hard-tier solving rates and reduces token usage compared to MSE-based rewards.

Searching arXiv for the cited papers to ground the article. Saturating Additive Rewards (SAR) is a reward design for reinforcement learning in precision-critical geometric generation in which a residual vector over constraints is decomposed into bounded per-constraint terms rather than first being collapsed by a global norm. In the formulation introduced for geometric constraint solving, SAR is designed to address a failure mode called Outlier Gradient Masking: under rewards such as $\exp(-\|\mathbf{r}\|_2^2)$ or $\exp(-\mathrm{MSE})$, a single badly violated constraint can drive the scalar reward toward zero and thereby suppress policy-gradient signal for all other constraints, including those on which the model is making progress (Cabral et al., 8 Jun 2026). SAR replaces that global coupling with an additive construction over individual residuals, so that each constraint contributes independently through a monotonically decreasing bounded kernel. In the reported experiments, this design preserves partial-progress signal, improves hard-tier solving rate relative to MSE-based rewards, and reduces token usage when training a LLM to generate geometry-solving code (Cabral et al., 8 Jun 2026).

1. Problem setting and motivation

SAR was introduced in the context of open-ended geometric synthesis from natural language, where a model must emit Python code producing exact coordinates and radii that satisfy many simultaneous geometric constraints (Cabral et al., 8 Jun 2026). The setting is explicitly described as precision-critical: outputs are not judged by semantic plausibility alone, but by whether the generated construction satisfies strict geometric constraints as measured by a verifier.

The paper formalizes each candidate solution by a residual vector

$\mathbf{r} = [r_1, \ldots, r_C], \qquad r_i \ge 0,$

where each residual corresponds to the violation of one constraint (Cabral et al., 8 Jun 2026). A standard dense-reward instinct in such settings is to aggregate the residuals through a single global norm or norm-like transformation, for example

$$\exp(-\|\mathbf{r}\|_2^2), \qquad \exp(-\text{MSE}), \qquad \phi(\|\mathbf{r}\|_p).$$

The paper identifies a specific weakness of this strategy, termed Outlier Gradient Masking: if even one constraint is badly violated, the global norm becomes large, the reward collapses toward zero, and the policy update becomes uninformative because the reward multiplies the log-probability gradient

$$\nabla_\theta J(\theta) = \mathbb{E}_{\mathbf{x}\sim\mathcal D,\ \mathbf{y}\sim\pi_\theta(\cdot\mid \mathbf{x})} \left[ \mathcal{R}(\mathbf{x},\mathbf{y})\cdot \nabla_\theta \log \pi_\theta(\mathbf{y}\mid \mathbf{x}) \right].$$

This makes partial correctness difficult to exploit: a sample that satisfies many constraints can still receive nearly zero reward if a small subset of residuals is large (Cabral et al., 8 Jun 2026).

The paper’s argument is both geometric and optimization-theoretic. It states that signal volume collapses for global norms in high dimensions, and that gradient coupling is global, because sensitivity to one residual depends on the total error level rather than only on that residual (Cabral et al., 8 Jun 2026). SAR is proposed to avoid that global veto effect.

2. Formal definition

SAR is defined by transforming each residual separately and then summing: $$\mathcal{R}_{\mathrm{SAR}}(\mathbf{r}) = \sum_{i=1}^{C} \phi(r_i),$$ where $\phi : \mathbb{R}_{\ge 0}\to[0,1]$ is monotonically decreasing and satisfies $\lim_{r\to\infty}\phi(r)=0$ (Cabral et al., 8 Jun 2026). The essential structural distinction is therefore not merely that SAR is dense, but that it is additive after per-constraint saturation, whereas global-norm rewards are transform-after-aggregation objectives.

The default instantiation in the paper is the Boltzmann kernel

$\phi(r) = e^{-r/T}.$

Two other bounded kernels are also mentioned: $\phi(r) = \frac{1}{1 + (r/\gamma)^2}$ and

$\exp(-\mathrm{MSE})$0

This kernel family expresses the “saturating” aspect of SAR: large residuals asymptotically contribute little, but they do not erase positive contributions from other constraints (Cabral et al., 8 Jun 2026).

The comparison class is the global-norm reward

$\exp(-\mathrm{MSE})$1

Under that formulation, all constraints are coupled through a single scalar norm. Under SAR, they are decoupled at the reward-contribution level. This is the central mathematical distinction.

3. Gradient structure and partial-progress preservation

The paper makes the contrast between SAR and global-norm rewards explicit at the gradient level. For a global-norm reward, the magnitude of the derivative with respect to a single residual is

$\exp(-\mathrm{MSE})$2

For SAR, the corresponding quantity is

$\exp(-\mathrm{MSE})$3

This means that the global-norm gradient is coupled to the total error $\exp(-\mathrm{MSE})$4, whereas the SAR gradient is local to $\exp(-\mathrm{MSE})$5 (Cabral et al., 8 Jun 2026).

This locality is the basis for the claim that SAR preserves partial progress. If one constraint is badly violated, its own term in $\exp(-\mathrm{MSE})$6 goes to zero, but the other constraints retain their contributions. The paper describes this as removal of the veto power that global norms grant to outlier residuals (Cabral et al., 8 Jun 2026). In its intended regime, geometric solving is a combinatorial assembly problem in which some constraints are often solved earlier than others; SAR is designed so that these partial substructures remain visible to reinforcement learning rather than being erased by one aggregate failure.

Empirical evidence is reported through an analysis of 3,893 partially correct solutions. The reward distributions are summarized as follows (Cabral et al., 8 Jun 2026):

Reward scheme	Near-zero collapse	Useful range $\exp(-\mathrm{MSE})$7	Near one
SAR	2.9%	97.1%	—
MSE	60.4%	19.0%	20.6%
MSE+S+D	63.5%	36.5%	—

These numbers are used in the paper to argue that SAR preserves ranking information across partially correct outputs, whereas MSE-based schemes often collapse distinct partial solutions into nearly indistinguishable rewards (Cabral et al., 8 Jun 2026).

4. Composite reward and training objective

The paper is explicit that SAR alone is not sufficient for exact geometric solving. Two additional problems are identified: reward plateau, in which a partially correct solution may appear sufficiently good under a dense reward, and geometric degeneracy, in which trivial collapsed constructions satisfy many constraints but are not valid solutions (Cabral et al., 8 Jun 2026).

For that reason, the training objective uses a composite reward: $\exp(-\mathrm{MSE})$8 The reported settings are (Cabral et al., 8 Jun 2026):

• $\exp(-\mathrm{MSE})$9
• $\mathbf{r} = [r_1, \ldots, r_C], \qquad r_i \ge 0,$0
• $\mathbf{r} = [r_1, \ldots, r_C], \qquad r_i \ge 0,$1
• $\mathbf{r} = [r_1, \ldots, r_C], \qquad r_i \ge 0,$2 when

$\mathbf{r} = [r_1, \ldots, r_C], \qquad r_i \ge 0,$3

• $\mathbf{r} = [r_1, \ldots, r_C], \qquad r_i \ge 0,$4 counts detected degenerate substructures, capped at 4

The paper states that this bounds the reward in $\mathbf{r} = [r_1, \ldots, r_C], \qquad r_i \ge 0,$5 and creates a sharp landscape that still requires high precision (Cabral et al., 8 Jun 2026). It also reports that reward is clipped to $\mathbf{r} = [r_1, \ldots, r_C], \qquad r_i \ge 0,$6 during RL. The best-performing condition is therefore not SAR in isolation, but SAR+S+D, meaning SAR together with a sparse success bonus and a degeneracy penalty (Cabral et al., 8 Jun 2026).

A plausible implication is that SAR functions as a dense shaping component inside a broader verifier-driven reward stack, rather than as a complete objective by itself.

5. Verification environment, benchmarks, and empirical results

SAR was evaluated in the PyGeoX environment, a programmable geometric DSL and verifier used both for data generation and reward evaluation (Cabral et al., 8 Jun 2026). PyGeoX compiles declarative geometry into symbolic equations and then into differentiable residuals. The paper reports support for about 35 object types, 38 relationships, equalities, inequalities, not-equal constraints, and arbitrary algebraic constraints on object properties (Cabral et al., 8 Jun 2026). The model does not call PyGeoX as a tool during inference; the verifier is used off-policy for validation and reward computation.

Evaluation is conducted on PyGeoX-Bench, a suite of 300 problems stratified into 100 Easy, 100 Medium, and 100 Hard, with correctness defined by

$\mathbf{r} = [r_1, \ldots, r_C], \qquad r_i \ge 0,$7

The metric is one-shot solving rate (SR) (Cabral et al., 8 Jun 2026). The paper also introduces PyGeoX-Wild, an OOD set of 86 problems adapted from a published middle-school geometry benchmark, using human-authored prose, constraint types absent from training, and unseen combinations of geometric identities (Cabral et al., 8 Jun 2026).

The main ablation results for Qwen3-8B are organized around SFT and RL. The hard-tier results reported in the paper are (Cabral et al., 8 Jun 2026):

Setting	Sparse baseline	SAR+S+D	MSE+S+D	SAR only	MSE only
RL Hard SR	0.35	0.41	0.18	0.09	0.10
SFT Hard SR	0.23	0.32	0.04	0.23	0.22

The headline RL comparison is

$\mathbf{r} = [r_1, \ldots, r_C], \qquad r_i \ge 0,$8

which the paper reports as SAR+S+D improves the hard-tier solving rate by 2.3× over MSE+S+D (Cabral et al., 8 Jun 2026). Cross-distribution results on PyGeoX-Wild are also reported:

• RL, Sparse: 0.59
• RL, SAR+S+D: 0.66
• RL, MSE+S+D: 0.60
• SFT, Sparse: 0.57
• SFT, SAR+S+D: 0.65
• SFT, MSE+S+D: 0.48 (Cabral et al., 8 Jun 2026)

The paper further reports token-efficiency gains: SAR+S+D uses about 4,060 tokens/task versus about 5,260 tokens/task for the sparse baseline, a 22.8% reduction (Cabral et al., 8 Jun 2026). This is interpreted as evidence that the model reaches geometric solutions more directly.

A context table also places the resulting Qwen3-8B-RL system against larger frontier systems on the Hard tier: DeepSeek-V3.2: 0.25, Proprietary-A: 0.37, Proprietary-B: 0.38, Proprietary-C: 0.51, and Qwen3-8B-RL (Ours): 0.41 (Cabral et al., 8 Jun 2026). The paper presents this as calibration rather than as a controlled comparison.

6. Limitations, scope, and implementation conditions

Several caveats are integral to the interpretation of SAR. First, the paper states explicitly that SAR alone is not sufficient. In RL, SAR only reaches Hard SR 0.09, compared with 0.41 for SAR+S+D; in SFT, SAR only reaches 0.23, compared with 0.32 for SAR+S+D (Cabral et al., 8 Jun 2026). The dense additive signal improves learning from partial progress, but an exact-solution threshold still requires the sparse success bonus, while avoidance of trivial constructions requires the degeneracy penalty.

Second, the setting is restricted to 2D static geometry (Cabral et al., 8 Jun 2026). Extending SAR’s demonstrated utility to 3D CAD or kinematic synthesis is not established in the paper; additional symbolic primitives would be required.

Third, the reported experiments depend on a relatively strong base model. The paper states that experiments with Qwen3-1.7B and Llama-3.1-8B-Instruct yielded near-zero performance and unstable training across reward settings (Cabral et al., 8 Jun 2026). This suggests that the effect of SAR is conditional on the model already possessing sufficient mathematical and instruction-following competence.

Fourth, the training setup is specific: Qwen3-8B as the main base model, SFT with LoRA using $\mathbf{r} = [r_1, \ldots, r_C], \qquad r_i \ge 0,$9, $$\exp(-\|\mathbf{r}\|_2^2), \qquad \exp(-\text{MSE}), \qquad \phi(\|\mathbf{r}\|_p).$$0, dropout $$\exp(-\|\mathbf{r}\|_2^2), \qquad \exp(-\text{MSE}), \qquad \phi(\|\mathbf{r}\|_p).$$1, and RL with GRPO via OpenRLHF, using 10k medium-difficulty problems, 8 rollouts per problem, a 90-second sandbox execution timeout, and numpy, scipy, sympy in the sandbox (Cabral et al., 8 Jun 2026). Benchmarking is one-shot, so solving rates are not boosted by repeated sampling.

A plausible implication is that SAR should be understood as a verifier-compatible reward shaping strategy whose efficacy depends materially on the residual decomposition, the success criterion, and the surrounding training recipe.

7. Relation to other reward formulations and terminological distinctions

SAR is most directly contrasted in its source paper with MSE- and SSE-based rewards such as $$\exp(-\|\mathbf{r}\|_2^2), \qquad \exp(-\text{MSE}), \qquad \phi(\|\mathbf{r}\|_p).$$2, which the paper describes as the natural dense baseline for geometry solvers (Cabral et al., 8 Jun 2026). The conceptual distinction is that those rewards aggregate first and saturate later, whereas SAR saturates locally and aggregates afterward.

The broader RL literature on reward aggregation is relevant but distinct. “Recursive Reward Aggregation” develops an algebraic framework in which one keeps the per-step reward fixed and changes how the reward sequence is aggregated over time, encompassing discounted sum, discounted max, discounted min, mean, variance, top-$$\exp(-\|\mathbf{r}\|_2^2), \qquad \exp(-\text{MSE}), \qquad \phi(\|\mathbf{r}\|_p).$$3, range, and Sharpe ratio (Tang et al., 11 Jul 2025). That paper states that SAR is not explicitly discussed and not one of the paper’s showcased operators, though a saturating or clipped additive accumulator could be expressible in principle if encoded through an appropriate recursive update rule and post-processing map (Tang et al., 11 Jul 2025). This suggests a conceptual relationship at the level of aggregation semantics, but not an explicit equivalence.

SAR should also be distinguished from Self-Aligned Reward, another “SAR” acronym introduced for reasoning LLMs. Self-Aligned Reward is defined as a relative perplexity difference between an answer conditioned on the query and the standalone answer, and is used to complement verifiable rewards in PPO and GRPO (Han et al., 5 Sep 2025). Despite the shared acronym, it is a different construction, operates on query-conditioned language-model perplexity rather than per-constraint residuals, and targets efficiency–correctness trade-offs in reasoning rather than precision-critical geometric generation.

A further acronym collision occurs in network economics, where SAR denotes Subscription-Aware Rewarding, a Stackelberg-game scheme under which only subscribers can earn mobile-data rewards by watching ads (Yu et al., 2019). That usage is unrelated to saturating additive rewards in reinforcement learning.

These distinctions matter because “SAR” is not a unique technical label across arXiv. In current usage, Saturating Additive Rewards refers specifically to the bounded per-constraint additive reward design introduced for geometric residual learning (Cabral et al., 8 Jun 2026).

8. Significance

Within its stated scope, SAR is a reward-design response to a general difficulty in verifier-based RL for structured generation: scalar global objectives can be too brittle when correctness is conjunctive and partial progress is common. The paper’s formal argument is that global norms lose informative signal in high-dimensional residual spaces and couple gradient magnitude to unrelated errors, whereas SAR preserves locality by assigning each constraint an independent bounded contribution (Cabral et al., 8 Jun 2026).

Its practical significance in the reported experiments lies in three linked outcomes: improved learning from partially correct outputs, better exact-solution rates when combined with sparse and degeneracy terms, and lower token consumption (Cabral et al., 8 Jun 2026). This suggests that, in domains where solutions are evaluated by multi-constraint residual vectors rather than by single scalar scores, the choice between aggregate-first and decompose-first reward design can materially affect both optimization dynamics and inference behavior.

The current evidence is specific to PyGeoX and 2D static geometry, but the underlying design principle is clear in the source formulation: if the task exposes verifier residuals at the level of individual constraints, then a bounded additive reward can preserve signal that global norms may erase (Cabral et al., 8 Jun 2026).