---
title: Tail-Likelihood Reinforcement Learning (TailRL)
url: https://www.emergentmind.com/topics/tail-likelihood-reinforcement-learning-tailrl
type: topic
---

# Tail-Likelihood Reinforcement Learning (TailRL)

Tail-Likelihood Reinforcement Learning (TailRL) is a reinforcement-learning objective for stochastic generative policies that optimizes the probability of exceeding reward thresholds across the entire upper reward distribution, rather than optimizing expected reward alone. For an input $x$, rollout $z\sim\pi_\theta(\cdot\mid x)$, and normalized reward $r(x,z)\in[0,1]$, TailRL defines the objective as the expected log tail probability
$$
J_{\mathrm{TailRL}}(\theta;x)
=
\int_0^1 \log p_\theta(x,\tau)\,d\tau,
$$
where
$$
p_\theta(x,\tau)
=
\Pr_{z\sim\pi_\theta(\cdot\mid x)}
\left(r(x,z)>\tau\right).
$$
The objective gives greater marginal weight to rare threshold exceedances and is equivalent to a harmonic mixture of Best-of-$k$ objectives over all sampling budgets. Its practical estimator requires only group rollouts, reward sorting, and replacement of the conventional advantage function. TailRL is introduced in “Tail-Likelihood Reinforcement Learning” [2609.02987].

## 1. Motivation and conceptual basis

Standard policy-gradient reinforcement learning optimizes the expected reward
$$
J_{\mathrm{RL}}(\theta;x)
=
\mathbb{E}_{z\sim\pi_\theta(\cdot\mid x)}
[r(x,z)].
$$
Its gradient is
$$
\nabla_\theta J_{\mathrm{RL}}(\theta;x)
=
\mathbb{E}_{z\sim\pi_\theta}
\left[
r(x,z)\nabla_\theta\log\pi_\theta(z\mid x)
\right].
$$

Expected reward is appropriate when a single rollout is used or when the mean is the desired statistic. It is less suitable when deployment can generate multiple samples and select the best one. Two policies can have similar means while differing substantially in the probability of producing rare, high-reward outcomes. A policy that usually produces moderately good results may therefore be inferior under repeated sampling to a policy that occasionally produces an exceptionally high-reward result.

For binary rewards, the relevant repeated-sampling metric is
$$
\mathrm{Pass@}k(\theta;x)
=
1-
\left(
\Pr_{z\sim\pi_\theta}[r(x,z)=0]
\right)^k.
$$
For continuous rewards, the corresponding quantity is
$$
\mathrm{Best\text{-}of\text{-}k}(\theta;x)
=
\mathbb{E}_{z_1,\ldots,z_k\sim\pi_\theta}
\left[
\max_{1\leq i\leq k}r(x,z_i)
\right].
$$
At $k=1$, both reduce to mean reward. As $k$ increases, their values depend increasingly on the upper tail of the reward distribution.

TailRL addresses the possibility that expected-reward optimization removes probability mass from rare but valuable behaviors. Once such trajectories become sufficiently unlikely, they are sampled infrequently, reducing the data available for further improvement. TailRL instead seeks to increase the likelihood of exceeding every reward threshold, with particular emphasis on thresholds that are currently cleared rarely.

The method is distinct from several other uses of “tail” in reinforcement learning. Heavy-tailed reward methods treat extreme observations as a statistical nuisance while continuing to optimize expected reward [2102.12769] [2306.06836]. Heavy-tailed policy search uses heavy-tailed action distributions to improve nonlocal exploration and basin escape, while also optimizing expected return [2106.08414]. Distributional RL models the return distribution and may derive risk measures from it, but does not necessarily optimize tail probabilities directly [2507.12657]. TailRL specifically changes the policy objective toward upper-tail coverage.

## 2. Tail probabilities and the logarithmic objective

For each threshold $\tau\in[0,1)$, define the binary event
$$
\{r(x,z)>\tau\}
$$
and its probability
$$
p_\theta(x,\tau)
=
\Pr_{z\sim\pi_\theta(\cdot\mid x)}
\left(r(x,z)>\tau\right).
$$

The expected reward admits the layer-cake representation
$$
\mathbb{E}[r(x,z)]
=
\int_0^1p_\theta(x,\tau)\,d\tau.
$$
Expected-reward RL therefore aggregates tail probabilities arithmetically. TailRL instead aggregates their logarithms:
$$
\boxed{
J_{\mathrm{TailRL}}(\theta;x)
=
\int_0^1\log p_\theta(x,\tau)\,d\tau
=
\mathbb{E}_{\tau\sim\mathrm{Unif}[0,1]}
[\log p_\theta(x,\tau)].
}
$$

The logarithm produces inverse-probability weighting. Since
$$
\frac{d}{dp}\log p=\frac1p,
$$
an absolute increase in a rare tail probability has greater marginal value than the same increase in a commonly achieved event. The general family
$$
J_\phi(\theta;x)
=
\int_0^1\phi(p_\theta(x,\tau))\,d\tau
$$
has gradient
$$
\nabla_\theta J_\phi(\theta;x)
=
\int_0^1
\phi'(p_\theta(x,\tau))
\nabla_\theta p_\theta(x,\tau)\,d\tau.
$$
TailRL corresponds to $\phi(p)=\log p$.

Using the score-function identity,
$$
\nabla_\theta p_\theta(x,\tau)
=
\mathbb{E}_{z\sim\pi_\theta}
\left[
\mathbf{1}_{\{r(x,z)>\tau\}}
\nabla_\theta\log\pi_\theta(z\mid x)
\right],
$$
the TailRL gradient becomes
$$
\boxed{
\nabla_\theta J_{\mathrm{TailRL}}(\theta;x)
=
\mathbb{E}_{z\sim\pi_\theta}
\left[
\left(
\int_0^{r(x,z)}
\frac{1}{p_\theta(x,\tau)}\,d\tau
\right)
\nabla_\theta\log\pi_\theta(z\mid x)
\right].
}
$$

A rollout with reward $r(x,z)$ exceeds every threshold below $r(x,z)$. The integral therefore accumulates inverse-tail-probability weights over the thresholds cleared by that rollout. High-reward rollouts receive more credit when the thresholds they clear are rarely achieved by the current policy.

For a general bounded reward $r\in[r_{\min},r_{\max}]$, TailRL applies to the affine-normalized reward:
$$
J_{\mathrm{TailRL}}^{[r_{\min},r_{\max}]}(\theta;x)
=
\frac{1}{r_{\max}-r_{\min}}
\int_{r_{\min}}^{r_{\max}}
\log p_\theta(x,\tau)\,d\tau.
$$

## 3. Likelihood interpretation and relation to MaxRL

TailRL has an audit interpretation based on independent threshold assignments. Select $L$ thresholds
$$
\tau_\ell=\frac{\ell-1}{L},
\qquad
\ell=1,\ldots,L,
$$
and independently sample rollouts
$$
z_1,\ldots,z_L
\overset{\mathrm{i.i.d.}}{\sim}
\pi_\theta(\cdot\mid x).
$$
Define
$$
E_L
=
\bigcap_{\ell=1}^L
\{r(x,z_\ell)>\tau_\ell\}.
$$
Independence gives
$$
\Pr_\theta(E_L\mid x)
=
\prod_{\ell=1}^Lp_\theta(x,\tau_\ell).
$$
Consequently,
$$
\frac1L\log\Pr_\theta(E_L\mid x)
=
\frac1L\sum_{\ell=1}^L\log p_\theta(x,\tau_\ell).
$$
As $L$ tends to infinity, the Riemann sum converges to
$$
J_{\mathrm{TailRL}}(\theta;x)
=
\lim_{L\to\infty}
\frac1L\log\Pr_\theta(E_L\mid x)
=
\int_0^1\log p_\theta(x,\tau)\,d\tau.
$$

This interpretation requires independent rollouts. Reusing a single rollout for every threshold would collapse the nested events to the hardest threshold rather than produce a product of tail probabilities.

For binary rewards, let
$$
q_\theta(x)
=
\Pr_{z\sim\pi_\theta}[r(x,z)=1].
$$
For every $\tau\in[0,1)$,
$$
p_\theta(x,\tau)=q_\theta(x),
$$
and therefore
$$
J_{\mathrm{TailRL}}(\theta;x)
=
\log q_\theta(x).
$$
This recovers the binary MaxRL objective:
$$
J_{\mathrm{MaxRL}}(\theta;x)=\log q_\theta(x),
$$
with gradient
$$
\nabla_\theta J_{\mathrm{MaxRL}}(\theta;x)
=
\frac1{q_\theta(x)}
\mathbb{E}_{z\sim\pi_\theta}
\left[
\mathbf{1}_{\{r(x,z)=1\}}
\nabla_\theta\log\pi_\theta(z\mid x)
\right].
$$

The factor $1/q_\theta(x)$ emphasizes rare successful rollouts. TailRL generalizes this binary principle to the complete continuum of reward thresholds.

TailRL should not be confused with direct tail-likelihood estimation in the statistical-density sense. It does not require a parametric reward density, a learned conditional density, or a likelihood-ratio model. Its “likelihood” is the probability of threshold exceedance under the policy, and the logarithm of that probability is used as the optimization objective.

## 4. Best-of-\(k\) decomposition

Let $z_1,\ldots,z_k$ be independent rollouts and define
$$
M_k=\max_{1\leq i\leq k}r(x,z_i).
$$
The layer-cake identity gives
$$
\mathrm{Best\text{-}of\text{-}k(\theta;x)
=
\int_0^1\Pr(M_k>\tau)\,d\tau.
$$
Because all $k$ rollouts fail to exceed $\tau$ with probability $(1-p_\theta(x,\tau))^k$,
$$
\mathrm{Best\text{-}of\text{-}k(\theta;x)
=
\int_0^1
\left[
1-(1-p_\theta(x,\tau))^k
\right]d\tau.
$$

Using
$$
-\log p
=
\sum_{k=1}^{\infty}\frac{(1-p)^k}{k},
$$
TailRL admits the decomposition
$$
\boxed{
J_{\mathrm{TailRL}}(\theta;x)
=
\sum_{k=1}^{\infty}
\frac{\mathrm{Best\text{-}of\text{-}k(\theta;x)-1}{k}.
}
$$
Differentiation yields
$$
\boxed{
\nabla_\theta J_{\mathrm{TailRL}}(\theta;x)
=
\sum_{k=1}^{\infty}
\frac1k
\nabla_\theta\mathrm{Best\text{-}of\text{-}k(\theta;x).
}
$$

TailRL is therefore a harmonic mixture of Best-of-$k$ gradients over all sampling budgets. It is not tuned to a single deployment value of $k$. The harmonic weighting gives nonzero importance to every finite sampling budget while emphasizing lower orders more strongly.

A finite-order TailRL objective is
$$
J_{\mathrm{TailRL}}^{(T)}(\theta;x)
=
\sum_{k=1}^{T}
\frac{\mathrm{Best\text{-}of\text{-}k(\theta;x)-1}{k}.
$$
Its gradient can be written as
$$
\nabla_\theta J_{\mathrm{TailRL}}^{(T)}(\theta;x)
=
\int_0^1
\frac{
1-(1-p_\theta(x,\tau))^T
}{
p_\theta(x,\tau)
}
\nabla_\theta p_\theta(x,\tau)\,d\tau.
$$
The threshold multiplier is
$$
w_T(p)
=
\frac{1-(1-p)^T}{p}
=
\sum_{j=0}^{T-1}(1-p)^j.
$$

At $T=1$,
$$
w_1(p)=1,
$$
so the finite-order objective reduces to Best-of-$1$ and therefore to expected reward up to an additive constant. As $T$ increases, the objective incorporates increasingly higher-order Best-of-$k$ signals. In the limit,
$$
w_T(p)\to\frac1p,
$$
recovering population TailRL.

The rollout group size is thus not merely a variance-reduction parameter. It selects the order of the finite-sample objective being estimated:

- $N=1$: expected reward;
- larger $N$: increasingly higher-order Best-of-$k$ signals;
- $N\to\infty$: population TailRL.

These identities require independent rollouts, bounded rewards in the stated range, fixed policy support, differentiable policy probabilities, summable parameter gradients, and sufficient integrability to exchange differentiation, integration, and expectation.

## 5. Finite-rollout estimator and algorithm

In practice, $p_\theta(x,\tau)$ is unknown. TailRL draws a group of $N$ independent rollouts
$$
z_1,\ldots,z_N\sim\pi_\theta(\cdot\mid x)
$$
with rewards
$$
r_i=r(x,z_i).
$$
For rollout $i$, define
$$
\boxed{
\omega(r_i)
=
\int_0^{r_i}
\frac{d\tau}{
\sum_{j=1}^N\mathbf{1}_{\{r_j>\tau\}}
}.
}
$$

At threshold $\tau$, the credit is divided equally among the rollouts that exceed it. A threshold exceeded by only one rollout gives that rollout all the threshold credit; a threshold exceeded by all $N$ rollouts gives each rollout credit $1/N$.

The uncentered estimator is
$$
\boxed{
\widehat g_{\mathrm{TailRL}}(x)
=
\sum_{i=1}^N
\omega(r_i)
\nabla_\theta\log\pi_\theta(z_i\mid x).
}
$$
The paper establishes
$$
\mathbb{E}
\left[
\widehat g_{\mathrm{TailRL}}(x)
\right]
=
\nabla_\theta
J_{\mathrm{TailRL}}^{(N)}(\theta;x).
$$

The weights are computed by sorting rewards. Let
$$
r_{(1)}\leq\cdots\leq r_{(N)},
\qquad
r_{(0)}=0.
$$
Then
$$
\omega(r_{(i)})
=
\omega(r_{(i-1)})
+
\frac{r_{(i)}-r_{(i-1)}}{N-i+1},
$$
or equivalently
$$
\omega(r_{(i)})
=
\sum_{k=1}^{i}
\frac{r_{(k)}-r_{(k-1)}}{N-k+1}.
$$
The computation requires sorting and a cumulative calculation. A useful identity is
$$
\sum_{i=1}^N\omega(r_i)
=
\max_{1\leq i\leq N}r_i.
$$

For variance reduction, the practical implementation uses the group baseline
$$
\bar\omega
=
\frac1N\sum_{j=1}^N\omega(r_j)
$$
and the centered TailRL advantage
$$
\boxed{
A_i^{(\mathrm{TailRL})}
=
\omega(r_i)-\bar\omega.
}
$$
The policy-gradient update then has the standard form
$$
\sum_{i=1}^N
A_i^{(\mathrm{TailRL})}
\nabla_\theta\log\pi_\theta(z_i\mid x).
$$

Centering introduces a finite-order subtlety: the centered estimator targets the order-$(N-1)$ objective rather than the order-$N$ gradient. Mean-reduced implementations also require attention to whether coefficients are multiplied by $N$.

For comparison, RLOO uses
$$
A_i^{(\mathrm{RLOO})}
=
r_i-\frac1{N-1}\sum_{j\neq i}r_j,
$$
while GRPO uses
$$
A_i^{(\mathrm{GRPO})}
=
\frac{r_i-\bar r}{\sigma(r)+\epsilon}.
$$
Both retain the expected-reward target. TailRL changes the relative weighting of reward levels: high thresholds cleared by few rollouts receive disproportionately greater credit.

For binary rewards, if $M$ of the $N$ rollouts succeed,
$$
\omega(1)=\frac1M,
\qquad
\omega(0)=0,
$$
and hence
$$
\widehat g
=
\frac1M
\sum_{i:r_i=1}
\nabla_\theta\log\pi_\theta(z_i\mid x),
$$
which is the MaxRL estimator.

A practical TailRL iteration consists of:

1. Sampling a group of $N$ rollouts for each input.
2. Computing the reward of every rollout.
3. Sorting the rewards.
4. Computing the $\omega(r_i)$ values.
5. Optionally centering them to obtain advantages.
6. Substituting the resulting advantages into an existing policy-gradient loss.
7. Performing an on-policy update.

TailRL is compatible with group-based systems such as GRPO and RLOO. It does not require a learned critic, an explicit numerical integral over thresholds, or changes to the policy architecture, reward function, rollout engine, or token-level log-probability calculation.

## 6. Empirical results, scope, and limitations

TailRL was evaluated on object localization, text-maze navigation, GUI grounding, and code runtime optimization [2609.02987]. Across these settings, the reported pattern is that TailRL is most useful when high-reward outcomes are attainable but rare, and when expected-reward methods tend to collapse onto reliable moderate-reward behaviors.

### Object localization

The ImageNet object-localization task assigns an IoU reward in $[0,1]$ to sampled bounding boxes. A pretrained ResNet-50 backbone and four categorical policy heads produce $50$ bins per coordinate, yielding $50^4=6{,}250{,}000$ possible boxes. Because the output space is finite, population tail probabilities can be computed exactly.

TailRL was compared with GRPO, RLOO, PKPO, and supervised objectives including MSE, $L1$, GIoU, and $L1+\mathrm{GIoU}$. Rollout groups had sizes $N\in\{16,64,256,1024\}$. The population TailRL objective matched or exceeded direct-coordinate supervision on CorLoc and mean IoU while improving Best-of-$1024$ IoU. At matched $N=1024$, TailRL outperformed GRPO and RLOO on CorLoc, mean IoU, and Best-of-$1024$ IoU. At $N=16$, it exceeded GRPO and RLOO trained with $N=1024$ on the reported metrics.

The advantage was larger on difficult images. The reported TailRL improvement over the stronger expected-reward baseline was approximately $0.09$ IoU on easy images, $0.16$ on medium images, and $0.18$ on hard images. On hard images, expected-reward baselines were near $0.30$ IoU, while TailRL reached approximately $0.49$.

### Text-maze navigation

In the text-maze task, a decoder-only transformer generates movement sequences in a $17\times17$ maze. Rewards combine progress toward the goal and path efficiency, with reward $1$ corresponding to a shortest path.

The experiment used $N=16$ rollouts per prompt, with additional sweeps over $N\in\{4,16,64,256\}$. Initial shortest-path success rates ranged from approximately $0.012\%$ to $0.83\%$.

When initial success was relatively common, TailRL, GRPO, and RLOO behaved similarly. When successful paths were rare, expected-reward methods failed to improve reliably, whereas TailRL continued to improve. With $N=4$, TailRL remained near the baseline floor; with $N=16$, it improved sharply because successful paths appeared often enough to be exploited. Larger groups produced smaller additional gains.

At inference time, TailRL’s Pass@$k$ and Best-of-$k$ reward increased more rapidly with $k$. The advantage was not always large at $k=1$, but became increasingly visible with additional samples. Training dynamics showed that GRPO and RLOO lost entropy rapidly, while TailRL retained higher entropy and generated progressively longer paths.

### GUI grounding

The GUI-grounding task asks a vision-language model to generate click coordinates from screenshots and instructions. Qwen2.5-VL-3B and Qwen2.5-VL-7B models were evaluated on ScreenSpot-Pro.

At $k=1$, TailRL and RLOO had similar Pass@1. As $k$ increased, TailRL’s performance continued to improve while RLOO plateaued earlier. TailRL matched RLOO’s Pass@1024 using eight samples for the 3B model and four samples for the 7B model. Category-wise evaluation showed TailRL leading the stronger baseline in 11 of 12 model-scale/category combinations at $k=128$, with gains as large as 12.0 percentage points on Dev at 3B and 11.5 points on CAD at 7B.

### Code runtime optimization

The code-optimization task rewards functionally equivalent C++ programs according to execution speed. Copying the source program is always correct and receives reward exactly $1$, creating a reliable but suboptimal mode. Initially, $74.4\%$ of rollouts were incorrect, $23.5\%$ were correct but no faster than the input, and $2.1\%$ were correct and faster.

GRPO and RLOO rapidly collapsed toward copying, with correctness above $98\%$, mean reward below $1.0$, and Best-of-$1024$ speedups of $0.98\times$ and $0.96\times$, respectively. TailRL maintained higher entropy and continued producing risky rewrites. Its mean training reward reached $2.92$, nearly three times the copy baseline. On the held-out test set, TailRL achieved a Best-of-$1024$ speedup of $7.7\times$, compared with $0.98\times$ for GRPO and $0.96\times$ for RLOO.

### Theoretical and practical limitations

TailRL does not guarantee discovery of an outcome whose policy probability is zero. Rare-event learning still requires at least occasional successful or high-reward rollouts. The maze experiment demonstrates that a group size of $N=4$ can be insufficient when successful trajectories are extremely rare.

Inverse-tail weighting can produce large gradients when only one rollout clears a high threshold. Gradient clipping or careful optimization may therefore be needed. The method also depends on the reward function: if the reward is misspecified, aggressively optimizing rare high rewards can amplify reward hacking or other undesirable behaviors.

The population objective is generally unavailable in practice. Finite groups estimate an order-$N$ objective, and centering changes the exact finite-order target. Larger groups expose rarer thresholds but increase sampling cost. TailRL also assumes reliable reward evaluation and independent rollouts for its Best-of-$k$ interpretation.

The method does not establish global convergence or global optimality for neural-network policies. Its theoretical results concern gradient identities, finite-order estimators, and Best-of-$k$ decompositions under support, smoothness, boundedness, and integrability assumptions. The empirical results are limited to the reported localization, navigation, GUI, and code-optimization settings.

TailRL is also distinct from risk-averse tail optimization. It emphasizes the upper tail of reward outcomes and seeks rare high-quality successes. It does not optimize lower-tail safety criteria such as CVaR or expected shortfall. In contrast, IQN–CVaR methods estimate a return distribution and optimize lower-tail loss severity [2510.04555], while SteinGate compares rollout-cost distributions with a safe reference distribution rather than directly optimizing a reward upper tail [2607.13175].

The central distinction is therefore:

$$
\text{expected-reward RL}
\quad\longrightarrow\quad
\text{improve the average rollout},
$$

whereas

$$
\text{TailRL}
\quad\longrightarrow\quad
\text{increase the likelihood of exceeding every attainable reward threshold}.
$$

Its finite-sample implementation is a sorted, tail-aware replacement for the ordinary group advantage, while its population interpretation is a harmonic combination of Best-of-$k$ objectives aligned with inference-time sampling.

Source: https://www.emergentmind.com/topics/tail-likelihood-reinforcement-learning-tailrl