Bipolar Float Reward in Reinforcement Learning

• Bipolar Float Reward (BFR) is a reinforcement learning mechanism that provides a +1 reward for perfect responses and graded negative penalties for all imperfect outputs.
• It mitigates reward sparsity and the Non-negative Reward Trap by offering a clear, error-proportional signal that steers models toward global optimum solutions.
• BFR integrates with policy-gradient methods using normalized advantages, showing improved performance across benchmarks in logical reasoning tasks.

Bipolar Float Reward (BFR) is a reward-shaping mechanism for reinforcement learning that implements a strictly bipolar reward signal, providing a sharp positive reward exclusively for perfect responses and graded negative penalties for all imperfect outputs. Introduced in the context of the UltraLogic framework for enhancing reasoning in LLMs, BFR is designed to address shortcomings in traditional binary and graded reward schemes, such as sparse gradients and local optima associated with the Non-negative Reward Trap. By leveraging problem-specific partial correctness metrics, BFR increases the discriminative signal available during policy optimization and guides the model more directly toward global logical accuracy (Liu et al., 6 Jan 2026).

1. Formal Definition

Let $y$ denote the model-generated answer and $y^*$ the ground-truth. A partial correctness score $S(y, y^*) \in [0, 1]$ is first computed using a task-appropriate metric—examples include Accuracy, F1-score, token-edit similarity, or Absolute Difference Rate.

The Bipolar Float Reward $r(y, y^*)$ is defined as: $$r(y, y^*) = \begin{cases} 1, & S(y, y^*) = 1 \ S(y, y^*) - 1, & 0 \leq S(y, y^*) < 1 \end{cases}$$ Thus, a perfect response ($S=1$) receives a reward of $+1$, while any imperfect response ($S<1$) is assigned a negative penalty in the range $[-1, 0)$. The penalty scales linearly with the degree of error, providing a proportional signal for the policy to correct flawed reasoning.

2. Addressing Reward Sparsity and the Non-negative Reward Trap

Conventional binary rewards ($r\in\{0,1\}$) are highly sparse, yielding no informative signal for nearly correct outputs—a critical limitation for complex reasoning tasks. Alternatively, non-negative float rewards ($y^*$0) permit a positive gradient even for consistently flawed responses within a batch, potentially causing the learning process to plateau at sub-optimal performance—a phenomenon termed the Non-negative Reward Trap.

BFR mitigates these deficiencies by imposing a reward cliff at the $y^*$1 boundary. Only perfect outputs exert a positive "pull" in the policy gradient, while all others provide a "push" away from flawed solutions, proportionate to their incorrectness. This dynamic redistributes the advantage signal, discouraging convergence to local optima of near-perfect but incorrect strategies and facilitating progress toward global logical optimality.

3. Integration with Policy Gradient Methods

Within the UltraLogic framework, BFR is applied in the context of policy-gradient optimization using Grouped Rollout Policy Optimization (GRPO). For each sample $y^*$2 in a rollout group of size $y^*$3, the advantage is computed as

$y^*$4

The policy parameters $y^*$5 are then updated using gradient ascent on the expected log-probability weighted by the normalized advantage: $y^*$6 The reward mapping and advantage computation are performed for each rollout, followed by the standard update step. In UltraLogic, four scoring functions are used (Accuracy, F1-score, Similarity, Absolute Difference Rate) and a small formatting bonus (0.1) is attached separately to isolate logic correctness from cosmetic concerns.

4. Design Parameters and Calibration

BFR operates within the reward range $y^*$7. Positive reward is exclusive to perfect correctness; all flawed outputs incur a negative penalty scaled linearly by $y^*$8. Supported scoring metrics include:

• Accuracy: Fraction of exactly matched elements in structured outputs.
• F1-score: $y^*$9.
• Similarity: Token-edit or embedding-based similarity, $S(y, y^*) \in [0, 1]$0.
• Absolute Difference Rate: $S(y, y^*) \in [0, 1]$1.

UltraLogic applies identical GRPO hyperparameters across all reward schemes: learning rate $S(y, y^*) \in [0, 1]$2, 16 rollouts per prompt, maximum response length of 32,768 tokens, temperature 1.0, and top-$S(y, y^*) \in [0, 1]$3 of 1.0.

Tasks are stratified into ten difficulty levels by automatically calibrated input parameters, targeting success rates of approximately 100%, 70%, 50%, 30%, and 0% at anchor levels (1, 3, 5, 7, 10). BFR is deployed uniformly across all difficulty strata.

5. Empirical Analysis and Comparative Evaluation

UltraLogic provides an ablation study comparing Binary Reward, Graded Float reward, and Bipolar Float Reward on Qwen3-8B over two epochs on 50 "Easy" tasks (10K samples). The efficacy across multiple benchmarks is summarized in the following table:

Reward Scheme	AIME24	AIME25	HMMT25	BBH	BBEH	ARC-AGI
Baseline model (no RL)	75.2	66.1	47.1	88.2	29.2	4.0
Binary Reward $S(y, y^*) \in [0, 1]$4	81.7	69.1	52.3	90.2	31.1	4.6
Graded Float $S(y, y^*) \in [0, 1]$5	76.9	66.3	53.0	90.4	31.0	4.3
Bipolar Float $S(y, y^*) \in [0, 1]$6	82.6	71.3	56.6	91.1	32.5	4.7

BFR attains the highest accuracy on all reasoning-intensive metrics, with gains of 1.5–4.5 points on HMMT25 and AIME25 relative to the binary scheme. Training-curve diagnostics show that BFR produces a steadily increasing critic/mean score, while Graded Float plateaus early and Binary Reward exhibits greater noise. This suggests the graded penalization and unique reward structure of BFR accelerate convergence and improve ultimate performance across diverse logical tasks.

6. Significance and Implications

The Bipolar Float Reward mechanism provides a principled resolution to structural challenges in reward design for reinforcement learning based reasoning. By combining a discriminative, error-proportional penalty for sub-optimal outputs with an isolated positive reward for fully correct solutions, BFR sharpens the learning signal. This effectively breaks the Non-negative Reward Trap, injects continuous negative guidance proportionate to error magnitude, and preserves a strong attractor at global optima. A plausible implication is that BFR’s approach to sparse, high-precision reward shaping could generalize to other domains where “almost correct” outputs are insufficient for task success, but dense informative feedback is required for efficient exploration and convergence (Liu et al., 6 Jan 2026).