Generalized Advantage Estimation

• Generalized Advantage Estimation (GAE) is a method for estimating advantage functions in reinforcement learning, which combines temporal-difference residuals over multiple future steps to reduce the variance of policy-gradient estimates while controlling bias through parameters γ and λ, used in algorithms like TRPO and PPO.
• GAE reduces bias and variance tradeoffs by using exponentially weighted combinations of multi-step estimators, where the discount factor γ emphasizes near-term consequences and the trace decay parameter λ controls reliance on bootstrapping from the value function
• Modifications of GAE, including segmented GAE, distribute GAE and UGAE address specific challenges such as long-context reasoning and distributional critics, emphasizing the flexibility in balancing bias and variance in reinforcement learning.

Generalized Advantage Estimation (GAE) is an exponentially weighted estimator of the advantage function used in policy-gradient and actor–critic reinforcement learning. It combines temporal-difference residuals over multiple future steps to reduce the variance of policy-gradient estimates while controlling bias through the parameters $\gamma$ and $\lambda$. For a value-function approximation $V$, the standard finite-horizon estimator is

$$\hat A_t^{\mathrm{GAE}(\gamma,\lambda)} = \sum_{l=0}^{T-t-1}(\gamma\lambda)^l \delta_{t+l}^{V}, \qquad \delta_t^{V}=r_t+\gamma V(s_{t+1})-V(s_t).$$

GAE is used in algorithms including TRPO, PPO, and related actor–critic methods, while subsequent work has extended or modified it for non-exponential discounting, distributional critics, multi-agent credit assignment, truncated rollouts, data augmentation, and long-context language-model reinforcement learning.

1. Definition and motivation

A stochastic policy-gradient method updates policy parameters according to an expectation of score-function terms,

$$\nabla_\theta J(\theta) = \mathbb E\left[ \sum_t \Psi_t\nabla_\theta\log\pi_\theta(a_t\mid s_t) \right],$$

where $\Psi_t$ is a learning signal associated with the action. Using the complete trajectory return produces an unbiased estimator, but its variance can be high because an action’s score is multiplied by rewards occurring far in the future, including rewards only weakly related to that action.

The ideal learning signal is the advantage function,

$A^\pi(s,a)=Q^\pi(s,a)-V^\pi(s),$

where $Q^\pi(s,a)$ is the expected return after taking action $a$ in state $s$, and $\lambda$0 is the expected return under the policy from that state. Positive-advantage actions should become more probable, whereas negative-advantage actions should become less probable.

A state-dependent baseline does not alter the expected policy gradient: $\lambda$1 The value function is therefore a useful baseline: it removes the state-dependent average return while retaining information about whether the selected action was unusually good or bad. Exact $\lambda$2 and $\lambda$3 functions are generally unavailable, so GAE addresses the problem of estimating advantages using an imperfect learned value function while avoiding the variance of full Monte Carlo returns. The original formulation introduced GAE together with trust-region optimization for both policy and value networks and evaluated it on high-dimensional simulated locomotion tasks (Schulman et al., 2015).

The one-step temporal-difference residual,

$\lambda$4

is a natural one-step advantage estimate. If $\lambda$5, then

$\lambda$6

With an imperfect value function, however, the residual can be biased.

2. Multi-step construction and recursion

Summing $\lambda$7 discounted temporal-difference residuals produces the $\lambda$8-step advantage estimator

$\lambda$9

Expanding and telescoping the value terms gives

$V$0

Thus, a $V$1-step advantage is a $V$2-step bootstrapped return minus the baseline $V$3.

For $V$4,

$V$5

As $V$6 increases, the estimator depends more on observed rewards and less on intermediate value predictions. This generally reduces bias caused by an inaccurate value function but increases variance. In the infinite-horizon limit,

$V$7

which is the Monte Carlo return minus the learned value baseline.

GAE forms an exponentially weighted combination of these multi-step estimators: $V$8 Rearranging yields the operational form

$V$9

For a finite trajectory,

$$\hat A_t^{\mathrm{GAE}(\gamma,\lambda)} = \sum_{l=0}^{T-t-1}(\gamma\lambda)^l \delta_{t+l}^{V}, \qquad \delta_t^{V}=r_t+\gamma V(s_{t+1})-V(s_t).$$0

The estimator is usually computed backward through the recursion

$$\hat A_t^{\mathrm{GAE}(\gamma,\lambda)} = \sum_{l=0}^{T-t-1}(\gamma\lambda)^l \delta_{t+l}^{V}, \qquad \delta_t^{V}=r_t+\gamma V(s_{t+1})-V(s_t).$$1

with $$\hat A_t^{\mathrm{GAE}(\gamma,\lambda)} = \sum_{l=0}^{T-t-1}(\gamma\lambda)^l \delta_{t+l}^{V}, \qquad \delta_t^{V}=r_t+\gamma V(s_{t+1})-V(s_t).$$2 at the end of a relevant trajectory segment. At a genuine terminal state, the bootstrap term is normally masked or $$\hat A_t^{\mathrm{GAE}(\gamma,\lambda)} = \sum_{l=0}^{T-t-1}(\gamma\lambda)^l \delta_{t+l}^{V}, \qquad \delta_t^{V}=r_t+\gamma V(s_{t+1})-V(s_t).$$3 is set to zero. If a trajectory is merely truncated by a finite rollout horizon, the final value estimate can instead be used for bootstrapping.

GAE is closely analogous to TD$$\hat A_t^{\mathrm{GAE}(\gamma,\lambda)} = \sum_{l=0}^{T-t-1}(\gamma\lambda)^l \delta_{t+l}^{V}, \qquad \delta_t^{V}=r_t+\gamma V(s_{t+1})-V(s_t).$$4. TD$$\hat A_t^{\mathrm{GAE}(\gamma,\lambda)} = \sum_{l=0}^{T-t-1}(\gamma\lambda)^l \delta_{t+l}^{V}, \qquad \delta_t^{V}=r_t+\gamma V(s_{t+1})-V(s_t).$$5 uses exponentially weighted temporal-difference errors to estimate a value function, whereas GAE applies a residual trace to estimate an advantage function for policy-gradient learning.

3. The roles of $$\hat A_t^{\mathrm{GAE}(\gamma,\lambda)} = \sum_{l=0}^{T-t-1}(\gamma\lambda)^l \delta_{t+l}^{V}, \qquad \delta_t^{V}=r_t+\gamma V(s_{t+1})-V(s_t).$$6 and $$\hat A_t^{\mathrm{GAE}(\gamma,\lambda)} = \sum_{l=0}^{T-t-1}(\gamma\lambda)^l \delta_{t+l}^{V}, \qquad \delta_t^{V}=r_t+\gamma V(s_{t+1})-V(s_t).$$7

The discount factor and trace parameter appear jointly in the coefficient $$\hat A_t^{\mathrm{GAE}(\gamma,\lambda)} = \sum_{l=0}^{T-t-1}(\gamma\lambda)^l \delta_{t+l}^{V}, \qquad \delta_t^{V}=r_t+\gamma V(s_{t+1})-V(s_t).$$8, but they have different meanings.

The original objective may be the undiscounted total reward,

$$\hat A_t^{\mathrm{GAE}(\gamma,\lambda)} = \sum_{l=0}^{T-t-1}(\gamma\lambda)^l \delta_{t+l}^{V}, \qquad \delta_t^{V}=r_t+\gamma V(s_{t+1})-V(s_t).$$9

In that formulation, $$\nabla_\theta J(\theta) = \mathbb E\left[ \sum_t \Psi_t\nabla_\theta\log\pi_\theta(a_t\mid s_t) \right],$$0 is an algorithmic parameter rather than part of the task definition. The discounted value function is

$$\nabla_\theta J(\theta) = \mathbb E\left[ \sum_t \Psi_t\nabla_\theta\log\pi_\theta(a_t\mid s_t) \right],$$1

Therefore, changing $$\nabla_\theta J(\theta) = \mathbb E\left[ \sum_t \Psi_t\nabla_\theta\log\pi_\theta(a_t\mid s_t) \right],$$2 changes the scale and definition of the value and advantage functions themselves.

A smaller $$\nabla_\theta J(\theta) = \mathbb E\left[ \sum_t \Psi_t\nabla_\theta\log\pi_\theta(a_t\mid s_t) \right],$$3 emphasizes near-term consequences, generally reduces variance, and suppresses delayed rewards. Relative to an undiscounted objective, it introduces bias even when the value function is learned perfectly. The effective temporal range is described as roughly $$\nabla_\theta J(\theta) = \mathbb E\left[ \sum_t \Psi_t\nabla_\theta\log\pi_\theta(a_t\mid s_t) \right],$$4, although the useful value depends on how long action effects persist.

The parameter $$\nabla_\theta J(\theta) = \mathbb E\left[ \sum_t \Psi_t\nabla_\theta\log\pi_\theta(a_t\mid s_t) \right],$$5 controls reliance on bootstrapping from the value function:

• $$\nabla_\theta J(\theta) = \mathbb E\left[ \sum_t \Psi_t\nabla_\theta\log\pi_\theta(a_t\mid s_t) \right],$$6 gives the one-step TD residual,

$$\nabla_\theta J(\theta) = \mathbb E\left[ \sum_t \Psi_t\nabla_\theta\log\pi_\theta(a_t\mid s_t) \right],$$7

This has relatively low variance but is sensitive to value-function error.

• $$\nabla_\theta J(\theta) = \mathbb E\left[ \sum_t \Psi_t\nabla_\theta\log\pi_\theta(a_t\mid s_t) \right],$$8 gives

$$\nabla_\theta J(\theta) = \mathbb E\left[ \sum_t \Psi_t\nabla_\theta\log\pi_\theta(a_t\mid s_t) \right],$$9

The value terms telescope, producing a Monte Carlo return minus a baseline. It is unbiased for the $\Psi_t$0-discounted advantage regardless of the accuracy of $\Psi_t$1, but can have high variance.

• Intermediate $\Psi_t$2 values interpolate between one-step TD and Monte Carlo estimation.

The two parameters should not be treated as interchangeable. $\Psi_t$3 changes the policy-gradient target relative to an undiscounted objective, whereas $\Psi_t$4 primarily introduces bias through reliance on an imperfect value function. The original experiments reported that the best $\Psi_t$5 was generally lower than the best $\Psi_t$6, with approximate best regions of $\Psi_t$7, $\Psi_t$8 on cart-pole and $\Psi_t$9, $A^\pi(s,a)=Q^\pi(s,a)-V^\pi(s),$0 for 3D biped locomotion (Schulman et al., 2015).

4. Integration with policy optimization

Given estimated advantages, a policy-gradient estimate is

$A^\pi(s,a)=Q^\pi(s,a)-V^\pi(s),$1

In trust-region policy optimization, the importance-weighted surrogate objective uses

$A^\pi(s,a)=Q^\pi(s,a)-V^\pi(s),$2

subject to a constraint on the average policy KL divergence. GAE supplies the action-quality estimates, while the trust region limits the size of each policy update. The trust region does not remove estimator bias, but can prevent imperfect advantage estimates from causing excessively large policy changes.

In PPO, GAE is inserted into the clipped surrogate objective,

$A^\pi(s,a)=Q^\pi(s,a)-V^\pi(s),$3

where

$A^\pi(s,a)=Q^\pi(s,a)-V^\pi(s),$4

The advantage determines whether the probability of the sampled action should be increased or decreased.

The value function is used as a baseline and bootstrap predictor, not as the policy. In the original algorithm, the policy update uses the old value function $A^\pi(s,a)=Q^\pi(s,a)-V^\pi(s),$5 to compute advantages, after which the value function is updated. Updating the value function first can introduce additional bias. If the value function is overfit to sampled transitions so that

$A^\pi(s,a)=Q^\pi(s,a)-V^\pi(s),$6

the resulting policy-gradient estimate can also become nearly zero.

The value network is commonly fitted by regression to discounted empirical returns: $A^\pi(s,a)=Q^\pi(s,a)-V^\pi(s),$7 using an objective such as

$A^\pi(s,a)=Q^\pi(s,a)-V^\pi(s),$8

The original work reported no performance difference between a TD$A^\pi(s,a)=Q^\pi(s,a)-V^\pi(s),$9-style value target and a $Q^\pi(s,a)$0 Monte Carlo-style target for value regression. Advantage normalization is common in later implementations, but it was not presented as part of the reported original GAE algorithm (Schulman et al., 2015).

5. Empirical scope, limitations, and variants

The original evaluation combined GAE with TRPO, neural value fitting, large batches, and trust-region value optimization. It covered cart-pole and MuJoCo continuous-control tasks, including 3D bipedal locomotion, quadrupedal locomotion, and bipedal standing up from the ground. Neural policies mapped directly from raw kinematics to joint torques. The reported experiments used approximately $Q^\pi(s,a)$1 timesteps per biped locomotion batch and $Q^\pi(s,a)$2 timesteps per quadrupedal locomotion and standing batch. These results established that the combined method could learn stable locomotion and stand-up behaviors, but did not establish a universal optimal $Q^\pi(s,a)$3 or attribute all improvement to GAE alone.

Later work has examined several limitations of the standard construction.

Truncated rollouts. Fixed-length PPO segments may end before environmental termination. Advantage estimates near the artificial boundary have access to fewer future rewards and can be strongly affected by the bootstrap convention. Partial GAE computes ordinary truncated GAE but retains only an initial portion of the segment for the PPO update, discarding estimates close to the artificial boundary. This was evaluated in MuJoCo and $Q^\pi(s,a)$4RTS, where partial GAE improved empirical results relative to standard truncated GAE (Song et al., 2023).

Alternative aggregation. GAE is a linear combination of multi-step estimates. Order-statistics methods replace this linear aggregation with maximum, minimum, maximum-absolute-value, or intermediate order-statistics operators. The maximum estimator is intended to preserve promising partial behaviors in sparse-reward tasks; the minimum estimator is intended to penalize actions with catastrophic downstream consequences; and maximum-absolute-value estimation exaggerates strong positive or negative evidence. These methods are deliberately biased and are not equivalent to robust MDP optimization or formal risk-sensitive reinforcement learning. Their reported performance was environment-dependent (Lei et al., 2019).

Direct advantage modeling. Direct Advantage Estimation models a centered advantage function directly rather than obtaining it indirectly as a difference between estimated $Q^\pi(s,a)$5- and $Q^\pi(s,a)$6-functions or from TD residuals. Its objective minimizes the variance of a transformed return subject to the centering constraint

$Q^\pi(s,a)$7

DAE does not require an explicit value function in its basic Monte Carlo form, although a bootstrapped extension can incorporate one. In discrete-control experiments, DAE outperformed GAE on a majority of the evaluated environments, while remaining sensitive to data volume, function-approximation capacity, and the on-policy setting (Pan et al., 2021).

Data augmentation. Bootstrap Advantage Estimation averages $Q^\pi(s,a)$8-step advantage estimates computed from the original observation and semantically transformed observations before applying the GAE-style weighting: $Q^\pi(s,a)$9 It is intended to reduce sensitivity to nuisance variation in image and vector observations. BAE retains the original rewards and action distribution and uses augmentation only for advantage computation. Experiments in Procgen, DeepMind Control, and PyBullet reported improved sample efficiency and generalization relative to GAE-PPO, although the benefit depends on the transformation preserving reward semantics (Rahman et al., 2022).

6. Extensions beyond scalar single-agent GAE

GAE has also been generalized by changing the value representation, temporal discounting, or credit-assignment structure.

Non-exponential discounting. Universal Generalized Advantage Estimation extends GAE to an arbitrary discount sequence $a$0, replacing the geometric weighting $a$1 with direct reward and bootstrap weights. For generalized $a$2-step advantages,

$a$3

UGAE forms

$a$4

When $a$5, UGAE reduces exactly to standard GAE. Beta-weighted discounting provides a continuous family between exponential and hyperbolic discounting. UGAE is generally computed through vectorized sums rather than the standard one-pass recursion and has $a$6 cost without truncation (Kwiatkowski et al., 2023).

Distributional critics. Distributional GAE replaces scalar TD residuals with a signed comparison between return distributions using a Wasserstein-like directional metric. The resulting DGAE estimator is

$a$7

where $a$8 compares the Bellman-updated return distribution with the current distributional value estimate. DGAE preserves the familiar backward recursion, but its directional metric may collapse to a signed mean difference when the transport cost is linear. Consequently, it does not necessarily capture differences in variance, skewness, or tail risk (Shaik et al., 23 Jul 2025).

Multi-agent credit assignment. Multi-agent methods modify the baseline rather than necessarily changing the trace mechanism. Marginal advantage estimation averages counterfactual advantages over the policies of other agents, while approximately synchronous advantage estimation constrains policy changes so that current partner policies approximate future partner policies (Wan et al., 2020). Generalized Per-Agent Advantage Estimation retains the exponentially weighted residual structure of GAE but replaces the global value with

$a$9

Its per-agent residual is

$s$0

and the resulting trace is

$s$1

In the single-agent case, GPAE reduces to standard GAE. Its off-policy version introduces double-truncated importance sampling to balance sensitivity to an agent’s own policy change against non-stationarity caused by other agents (Kim et al., 3 Mar 2026).

Critic-free grouped estimators. GAGPO substitutes a non-parametric grouped value proxy for a learned critic. For state $s$2, it averages sampled returns across occurrences of that state in a rollout group: $s$3 It then computes

$s$4

and applies the standard GAE-style recursion. GAGPO therefore retains temporal bootstrapping without training a parametric value network, but depends on repeated state occurrences and exact textual state matching in its reported implementation (Zhu et al., 13 May 2026).

Trajectory-level LLM estimators. BASIS addresses a different problem from GAE: estimating a prompt-level terminal-reward baseline for LLM reasoning when only one rollout is sampled per prompt. It uses offline prompt-level value estimates and rewards from other prompts in the batch to form a leave-one-out baseline. BASIS does not use temporal-difference residuals, tokenwise value functions, $s$5, $s$6, or temporal bootstrapping. It is therefore a critic-free trajectory-level baseline estimator rather than a GAE approximation (Zhu et al., 13 May 2026).

Long-context reasoning. Segmental Advantage Estimation modifies where GAE applies trace decay in sparse-reward language-model reasoning. It identifies low-probability tokens as heuristic segment boundaries, applies no decay within a segment, and applies $s$7 only when the trace crosses a boundary. Its recursion is

$s$8

where $s$9 is $\lambda$00 within a segment and $\lambda$01 at a segment boundary. Experiments on mathematical, code, and STEM reasoning reported improvements over token-level GAE baselines, although the semantic-boundary assumption remains heuristic (Gong et al., 12 Jan 2026).

7. Implementation and computational considerations

The standard implementation computes TD residuals and then scans backward: $\lambda$02

$\lambda$03

where $\lambda$04 masks true terminal transitions. The distinction between true termination and artificial rollout truncation is essential. At termination, the bootstrap term is disabled; at a nonterminal rollout boundary, the final value estimate can be retained.

The rewards-to-go or value target is often formed as

$\lambda$05

and the critic is trained toward this target. GAE itself is not the complete PPO algorithm: it is the intermediate computation connecting rollout collection and critic inference to actor and critic losses.

The backward dependency in the recursion has also motivated hardware implementations. HEPPO uses an FPGA-based parallel, pipelined architecture for the GAE stage of PPO. It employs trajectory-aligned memory, first-in-last-out access, multiple processing elements, and $\lambda$06-step lookahead to mitigate the loop-carried dependency in

$\lambda$07

The reported design combines dynamic reward standardization, block standardization for values, and 8-bit uniform quantization. Its architectural claims include a fourfold memory-usage reduction, an approximately $\lambda$08 PPO speed increase, and throughput improvements based on hardware estimates; the reported speedups are not all end-to-end measurements (Taha et al., 22 Jan 2025).

GAE remains sensitive to value-function error, reward scale, terminal-state handling, rollout length, $\lambda$09, and $\lambda$10. Quantization errors, delayed rewards, and inaccurate intermediate values can accumulate through the factor $\lambda$11. Extensions such as partial, segmental, distributional, per-agent, grouped, and non-exponential estimators address specific failure modes, but none establishes that standard GAE is universally suboptimal. Its central role is the controlled aggregation of multi-step temporal-difference information into an advantage signal suitable for policy optimization.