A First-Principles Derivation of LLM Policy Optimization: From Expected Reward to GRPO and Its Structural Extensions

Abstract: Policy gradient algorithms for LLMs optimize the same objective $J(θ) = \mathbb{E}{τ\sim pθ(τ)}[R(τ)]$, which has exactly two factors: the trajectory probability $p_θ(τ)$ and the reward $R(τ)$. Every method from REINFORCE to PPO to GRPO and their descendants modifies one or both factors to address a specific failure in the preceding formulation. Existing surveys organize these methods by domain or chronology, which obscures the rationale behind each design choice and the precise location of its intervention within the gradient estimator. This survey revisits the landscape of LLM policy optimization from $J(θ)$ on first principles and uses the trajectory side, induced by $p_θ(τ)$, and the reward side, induced by $R(τ)$, as the two axes along which methods are located. It covers the path from REINFORCE and PPO to GRPO, as well as post-GRPO variants, Agentic RL, and GRPO-OPD. The resulting framework is unified, diagnostic, and extensible: it analyzes methods from a shared objective, identifies which side each method modifies and why, and applies the same trajectory and reward axes across these settings. Across these settings, the framework also exposes compound failures that no single-side fix resolves and that therefore require joint design of the trajectory side and the reward side. The boundary cases and coupled failures identified by this map mark where existing solutions run out and provide a principled starting point for designing the next generation of LLM policy optimization algorithms.

• The paper derives a unified framework linking LLM policy optimization methods, transitioning from REINFORCE to PPO and advancing to GRPO.
• It demonstrates how structural interventions on trajectory probability and reward functional can stabilize reinforcement learning training and address specific failure modes.
• The study extends the framework to agentic RL and GRPO-OPD hybrids, highlighting opportunities for future research in multi-turn and hybrid reward scenarios.

A First-Principles Framework for LLM Policy Optimization: From Expected Reward to GRPO and Extensions

Introduction and Motivation

This paper presents a unified, first-principles framework for understanding and systematizing the modern landscape of policy gradient methods for LLM optimization. All such methods maximize the expected reward objective $$J(\theta) = \mathbb{E}_{\tau \sim p_\theta(\tau)}[R(\tau)]$$, with the two fundamental axes of algorithmic design and diagnosis being the trajectory probability $p_\theta(\tau)$ ("trajectory side") and the reward functional $R(\tau)$ ("reward side"). The authors argue that most reinforcement learning (RL) algorithms and recent structural variants—ranging from REINFORCE to PPO to the modern critic-free GRPO and its extensions—are best understood as principled, minimal interventions on one or both of these axes in response to specific failures in the base estimator.

Figure 1: A first-principles landscape of LLM policy optimization delineating the trajectory axis ($p_\theta(\tau)$) and reward axis ($R(\tau)$), tracing method transitions (REINFORCE $\to$ PPO $\to$ GRPO) and forking into trajectory-side, reward-side, and coupled extensions; Agentic RL generalizes the trajectory side, GRPO-OPD hybrids generalize the reward side.

From Policy Gradient to GRPO: Structural Organization

The paper begins by reconstructing the classic policy gradient derivation in the language modeling context. The log-derivative trick converts the intractable gradient into a sampling-based estimator, and the REINFORCE and actor-critic architectures implement unbiased or variance-reduced estimators based on reward-to-go and advantage estimation. Proximal Policy Optimization (PPO) further stabilizes and improves data efficiency via trust-region clipping and importance-reweighted data reuse.

A key argument is that LLM generation is an MDP with deterministic transitions and tractable policy gradients. The typical RLHF pipeline (supervised fine-tuning, reward modeling, RL fine-tuning) can thus be mapped precisely onto the two-factor frame, with per-token credit assignment issues made explicit in the token-level reward construction.

The Rise of GRPO and the Two-Axis View

Group Relative Policy Optimization (GRPO) emerges as a critical innovation: by replacing the high-variance, compute-intensive learned critic with a group-relative advantage computed from group-normalized rewards, GRPO achieves strong sample efficiency and mathematical reasoning performance with minimal infrastructure overhead. The paper positions this as a "pure reward-side substitution," leaving the rest of the trajectory logic from PPO or RLHF-PPO intact.

Subsequent variants, surveyed in detail, systematically address trajectory-side (e.g., sample collection, compression/reuse, diversity management, repair) or reward-side (e.g., reward density, bias correction, credit assignment) failures, or deal with coupled failures which require simultaneous intervention. This results in a taxonomy where every method can be precisely located as its modification target and corresponding failure.

Trajectory-Side Variants

Trajectory-axis interventions span methods to increase signal-to-noise ratio via prompt or trajectory selection (GRESO, Prompt Replay/PODS, adaptive allocation), amortize rollout cost (ThinkPrune/DAST, replay-based approaches), or augment group diversity (TreeRPO, DIVER, GAPO, dual-scale regularization). Structural mismatches and inefficiencies, such as per-token versus sequence-level granularity in importance ratios, are addressed by update rule modifications (DAPO, GSPO) and more sophisticated clipping strategies.

Reward-Side Variants

Reward-axis interventions seek to densify, shape, or multi-objectify the reward signal (e.g., process rewards, step/segment-level signals), correct normalization/statistical biases (e.g., Dr.GRPO, AGPO, CPPO), and target the credit assignment pathologies of uniform scalar broadcast (TreeRPO, HighEnt, segment-level or subproblem decomposition). The authors emphasize that the greatest training instabilities (entropy collapse, gradient dilution) are compound phenomena, requiring careful co-design or entirely novel operator placement.

Structural Extensions: Agentic RL and GRPO-OPD Hybrids

The framework is shown to scale to settings where either:

• The trajectory side is expanded—to agentic RL, where trajectories are multi-turn, interleaving tool-use decisions and environment feedback (WebRL, AgentRL, Search-R1). Here, new failure modes such as rollout validity, importance ratio staleness, and high environment cost appear, as do new reward-side challenges, notably temporal credit assignment and complex reward specification.
• The reward side is expanded—to hybrids of GRPO and On-Policy Distillation (OPD). These methods introduce per-token, teacher-driven signals into the reward pipeline without discarding the $J(\theta)$ frame. The paper develops an operator-based taxonomy ($\mathcal{T}_A$ importance ratio, $\mathcal{T}_B$ advantage, $p_\theta(\tau)$0 in-expectation distillation, $p_\theta(\tau)$1 external regularizer), showing how different hybrids stabilize or moderate the regimes where either verifier or teacher signals are degenerate.

Unification, Technical Implications, and Open Problems

The major technical claim supported in the paper is that the two-factor framework is simultaneously unified, diagnostic, and extensible:

• Unification: All surveyed methods (including very recent ones) and their boundaries (e.g., DPO, pure-KL distillation) can be precisely mapped along the two axes, with clear separation of what each fix resolves or leaves open.
• Diagnosis: Every instability—be it entropy collapse, over-optimization, sparse/degenerate learning signal—can be exactly localized, preventing misinterpretation of symptoms and supporting minimal, targeted intervention design.
• Extensibility: The analytical axes support transfer to agentic RL, to reward/teacher hybrids, and (with caveats) to multimodal or diffusion LLMs.

The paper highlights persistent gaps and directions for future work:

• Absence of joint design theory: There is no principled recipe for when or how trajectory and reward side interventions must be paired—most combined approaches are ad hoc, and rigorous characterization of cross-operator transfer or stability is missing.
• Boundary crossings: Mixing or transferring methods (e.g., clipping strategies, advantage bias corrections) between single-turn and agentic or hybrid contexts is nontrivial, with many pathologies newly emergent (e.g., teacher signal degradation, joint signal collapse).
• Operator-level design: The full principled space of how teacher and verifier signals should be mixed or fused remains largely unexplored, particularly with regards to proper in-expectation and out-of-expectation placements.

Conclusion

This work formalizes a two-axis, first-principles framework for LLM policy optimization, unifying recent advances in trajectory and reward factorization, systematic diagnostic of failures, and expansion to multi-turn and hybrid reward settings. The framework provides actionable structure for both practical method development and theoretical investigation and identifies precise locations where further innovation or theoretical support is required.