What is the objective of reasoning with reinforcement learning?

Abstract: We show that several popular algorithms for reinforcement learning in LLMs with binary rewards can be viewed as stochastic gradient ascent on a monotone transform of the probability of a correct answer given a prompt. In particular, the transformation associated with rejection sampling algorithms is the logarithm and that associated with the GRPO algorithm is the arcsine of the square root.

• The paper demonstrates that popular RL algorithms like REINFORCE, rejection sampling, and GRPO optimize monotone transforms of the correct answer probability.
• It employs Bernstein polynomial approximation to unify diverse RL fine-tuning methods and provides theoretical convergence guarantees.
• The analysis reveals that objective scaling primarily influences optimization dynamics and sample efficiency without changing the optimal solutions.

Objective Functions in Reinforcement Learning for Reasoning with LLMs

This paper provides a rigorous analysis of the objective functions implicitly optimized by popular reinforcement learning (RL) algorithms in the post-training of LLMs with binary rewards. The authors demonstrate that these algorithms, including REINFORCE, rejection sampling, and GRPO, can be interpreted as stochastic gradient ascent on monotone transformations of the probability of generating a correct answer given a prompt. The work unifies disparate RL fine-tuning approaches under a common mathematical framework, clarifying the relationship between algorithmic choices and the underlying optimization objectives.

Meta-Algorithm and Objective Formulation

The central abstraction is a meta-algorithm for RL-based LLM fine-tuning, which consists of sampling prompts, generating responses, labeling correctness, and updating model parameters via weighted supervised learning steps. The authors formalize the objective as maximizing a monotone function $h$ of the probability $p_\theta(C|x)$ that the model $\pi_\theta$ produces a correct answer $y \in C(x)$ for prompt $x$:

$$J_h(\theta) = \mathbb{E}_{x \sim Q} \left[ h\left( \sum_{y \in C(x)} \pi_\theta(y \mid x) \right) \right]$$

The choice of $h$ is determined by the weighting scheme $Z_i$ applied to each sampled response in the gradient update. The authors show that for a broad class of weights, the induced objective $h_M$ is a Bernstein polynomial approximation of $h$, parameterized by the number of samples $M$.

Analysis of Popular Algorithms

REINFORCE

The vanilla REINFORCE algorithm uses $Z_i = 1_{y_i \in C(x)}$, directly maximizing the expected probability of correctness ($h(t) = t$). This corresponds to standard policy gradient updates and is equivalent to maximizing accuracy averaged over the corpus.

Rejection Sampling

Rejection sampling-based fine-tuning, where only correct responses are used for updates, induces an objective close to $h(t) = \log(t)$. The authors derive the exact form of $h_M$ and show its convergence to the logarithmic function as $M \to \infty$.

Figure 1: Function $h_M$ induced from rejection sampling, illustrating its convergence to $\log(t)$ as $M$ increases.

This approach is analogous to multilabel supervised learning, where multiple answers may be correct, but lacks a maximum likelihood interpretation unless $|C(x)| = 1$.

GRPO

The GRPO algorithm normalizes the reward by the empirical standard deviation, amplifying the gradient signal when correct answers are rare. The induced objective approaches $h(t) = \arcsin(\sqrt{t})$ for large $M$ and small regularization $\varepsilon$.

Figure 2: The effect of $M$ and $\varepsilon$ on the function $h_{M, \varepsilon}/h_{M, \varepsilon}(1)$, showing convergence to $h(t) = \frac{2}{\pi}\arcsin\sqrt{t}$ for large $M$ and small $\varepsilon$.

The authors provide a detailed derivation of the gradient estimator and its connection to the arcsine transformation, highlighting the impact of normalization on the optimization landscape.

Comparative Visualization

Figure 3: GRPO loss vs. REINFORCE loss vs. log loss, comparing the monotone transforms of the probability of correctness targeted by each algorithm.

This figure succinctly illustrates the differences in objective scaling, which can affect optimization dynamics and sample efficiency.

Generalization to Arbitrary Objectives

The framework allows for the construction of gradient estimators targeting any monotone function $h$ via Bernstein polynomial approximation. The authors provide explicit formulas for the coefficients $a_s$ and $b_s$ in the weighting scheme to approximate $h'$ by $h_M'$, with convergence guarantees for smooth $h$. This generality enables principled exploration of alternative objectives, such as log-odds or beta CDF-based scalings, and facilitates the design of new RL fine-tuning algorithms tailored to specific desiderata.

Practical and Theoretical Implications

The analysis reveals that all monotone rescalings of the probability of correctness induce equivalent optimization problems in the limit of expressive models and sufficient data, analogous to the equivalence of hinge and logistic loss in separable classification. The choice of $h$ primarily affects optimization dynamics and sample efficiency, not the set of optimal solutions. However, in practical settings with limited data, model capacity, or nontrivial reward structures, the scaling function can influence convergence rates and generalization.

The work cautions against overinterpreting the superiority of any particular objective, emphasizing that the best choice is context-dependent and task-specific. The framework also clarifies that RL fine-tuning cannot compensate for a base model incapable of generating correct answers, as all algorithms rely on the existence of positive reward signals.

Conclusion

This paper provides a unified mathematical perspective on RL-based fine-tuning of LLMs with binary rewards, showing that popular algorithms optimize monotone transforms of the probability of correctness. The framework enables systematic design and analysis of new objectives via Bernstein polynomial approximation, with theoretical guarantees on convergence. The practical impact of objective scaling is nuanced, affecting optimization but not the set of optimal solutions in expressive regimes. Future work may explore empirical trade-offs between different objectives, especially in low-data or low-capacity settings, and extend the analysis to more complex reward structures and multi-step reasoning tasks.