---
title: Reward-Augmented Maximum Likelihood (RAML)
url: https://www.emergentmind.com/topics/reward-augmented-maximum-likelihood-raml
type: topic
---

# Reward-Augmented Maximum Likelihood (RAML)

Reward-Augmented Maximum Likelihood (RAML) is a learning paradigm for structured prediction in which task-specific rewards are directly integrated into the statistical estimation process. RAML generalizes maximum likelihood estimation by defining a reward-smoothed target distribution over outputs, encouraging the model to place probability mass not only on the ground-truth label but also on other high-reward candidates. This technique is computationally efficient and empirically effective across structured prediction tasks—including neural sequence generation—where the end-goal is to maximize a task-defined reward rather than mere likelihood of reference structures. Theoretical advances have linked RAML to softmax Q-distribution estimation and Bayesian decision theory, providing foundations for its empirical success and connections to entropy regularized reinforcement learning [1705.07136][1609.00150][1804.10974].

## 1. RAML Objective and Formal Definition

Standard maximum likelihood (ML) for structured prediction minimizes:
\[
\mathcal{L}_{\rm ML}(\theta) = -\sum_{i=1}^n \log P_\theta(y_i \mid x_i) = \sum_{i=1}^n \mathrm{KL}(\tilde P(\cdot\mid x_i) \| P_\theta(\cdot\mid x_i))
\]
where $\tilde P$ is the empirical distribution concentrated on $y_i$. RAML, in contrast, disperses this mass according to the task reward $r(y, y^*)$, defining the exponentiated payoff distribution:
\[
q_\tau(y\mid y^*) = \frac{\exp(r(y, y^*)/\tau)}{\sum_{y' \in \mathcal Y} \exp(r(y', y^*)/\tau)}
\]
The RAML loss is then
\[
\mathcal{L}_{\rm RAML}(\theta) = -\sum_{i=1}^n \sum_{y\in\mathcal Y} q_\tau(y\mid y_i) \log P_\theta(y\mid x_i) = \sum_{i=1}^n \mathrm{KL}(q_\tau(\cdot\mid y_i) \| P_\theta(\cdot\mid x_i))
\]
Here, $\tau > 0$ is a temperature parameter governing the spread of mass from $y^*$ to other outputs according to their reward. As $\tau \rightarrow 0$, $q_\tau$ collapses to a point mass on $y^*$, recovering classical ML; larger $\tau$ distributes probability more widely across high-reward alternatives [1609.00150][1705.07136][1804.10974].

## 2. Theoretical Foundations: Bayes Decision Theory and Softmax Q-Distributions

Bayes-optimal structured prediction seeks a mapping $h^*(x)$ that maximizes the expected reward:
\[
h^*(x) = \arg\max_{y \in \mathcal Y} \mathbb{E}_{P(Y\mid X=x)} [ r(y, Y) ]
\]
The softmax Q-distribution provides a smooth approximation to the Bayes boundary:
\[
Q_\tau(Y=y\mid X=x) = \frac{\exp\left(\mathbb{E}_{P(Y\mid X=x)}[r(y,Y)]/\tau\right)}{\sum_{y'} \exp\left(\mathbb{E}_{P(Y\mid X=x)} [ r(y', Y) ]/\tau \right)}
\]
Decoding from $Q_\tau(y\mid x)$ recovers the Bayes decision rule for the supervised reward maximization problem. RAML is equivalent to maximizing the likelihood of samples from an empirical softmax $Q$-distribution, which smooths out label noise and bridges the gap between ML and Bayes-optimal risk minimization [1705.07136].

## 3. RAML as Approximate Softmax Q-Distribution Estimation

Directly estimating $Q_\tau$ is intractable due to the expectation in the exponent. RAML uses a tractable surrogate:
\[
Q'_\tau(y\mid x) = \mathbb{E}_{P(Y \mid x)} \left[ \frac{\exp(r(y, Y)/\tau)}{\sum_{y'} \exp(r(y', Y)/\tau)} \right]
\]
The empirical version, with the true distribution replaced by the empirical data distribution, returns to the RAML payoff $q_\tau(y\mid y^*)$. The minimizer of the KL between $Q_\tau$ and the model distribution $P_\theta$ coincides with the RAML estimator up to controlled approximation error. Specifically, the KL-divergence between $Q_\tau$ and $Q'_\tau$ can be bounded in terms of $R_{\max}$ and $\tau$, and additional conditions allow this bound to shrink for small $\tau$ [1705.07136].

## 4. Relation to Entropy-Regularized Reinforcement Learning

RAML shares a formal connection to entropy-regularized RL, particularly in the case of sequence prediction. The constructed exponentiated payoff distribution is analogous to the policy in a maximum-entropy Markov Decision Process (MDP), where the entropy penalty encourages distributional smoothness over possible outputs. Recent work has established that the token-level RAML target can be equivalently derived from the optimal soft Bellman recursion for a reward-MDP with entropy regularization, linking RAML to the broader RL landscape and suggesting improved variants such as Value-Augmented Maximum Likelihood (VAML) [1804.10974].

## 5. Algorithmic Implementation and Practical Aspects

A typical RAML optimization loop proceeds as follows:

1. For each training pair $(x, y^*)$, sample candidate outputs $y \sim q_\tau(\cdot \mid y^*)$.
2. Compute cross-entropy loss $-\log P_\theta(y\mid x)$ at the sampled $y$.
3. Update parameters $\theta$ with the gradient averaged over the batch.

Sampling from $q_\tau$ may use edit-distance-based proposals (for speech or translation), perturbing $y^*$ at various distances according to their exponentiated reward. The process can be efficiently implemented, with low variance Monte Carlo estimates for the gradient. The temperature $\tau$ is a crucial hyperparameter; best performance in practice is usually achieved with $\tau \in [0.7,1.0]$ [1609.00150]. RAML preserves fast training characteristics of ML, with minimal overhead and no need for online inference as in policy-gradient RL.

## 6. Empirical Performance and Comparative Analysis

Empirically, RAML demonstrates consistent improvements over standard ML across a range of structured prediction tasks. On speech recognition (TIMIT), RAML with $\tau=0.9$ reduced phone error rate from 22.18% (ML) to 19.89%. On WMT'14 En$\rightarrow$Fr translation, BLEU improved from 36.87 (ML) to 37.23 (RAML, $\tau=0.85$). In multi-reference captioning (MSCOCO), both RAML and softmax Q-distribution methods outperformed ML on BLEU metrics. Across named entity recognition, dependency parsing, and machine translation, RAML improves performance on relevant reward metrics while retaining ML's computational advantages [1609.00150][1705.07136][1804.10974].

A summary of empirical BLEU scores on machine translation and image captioning (test set, ± std) from [1804.10974]:

| System | WMT'14 DE→EN (input-feeding) | COCO Captioning |
|--------|-------------------------------|-----------------|
| MLE    | 28.06 ± 0.15                  | 29.54 ± 0.21    |
| RAML   | 28.56 ± 0.15                  | 29.84 ± 0.21    |
| VAML   | 28.84 ± 0.10                  | 29.93 ± 0.22    |
| ERAC   | 29.31 ± 0.04                  | 31.44 ± 0.22    |

*Note: VAML and ERAC are successor algorithms leveraging further RL-inspired constructs.*

## 7. Extensions, Limitations, and Future Directions

RAML has inspired a family of related algorithms including token-level RAML, Value-Augmented Maximum Likelihood (VAML), and entropy-regularized actor-critic (ERAC). While RAML uses sequence-level reward, VAML propagates reward to token-level supervision using a learned Q-function, improving sample efficiency. ERAC introduces systematic entropy regularization into the actor-critic framework for sequence modeling, consistently outperforming previous baselines [1804.10974].

A limitation of the RAML framework is the approximation error arising from empirical surrogates to the ideal softmax Q-distribution. The effectiveness of downstream algorithms may depend on the quality of value estimation and proposal distributions used for candidate sampling. Future directions may include leveraging stronger off-policy RL methods and improved value function approximators to better align learning with the true Bayes-optimal rule, as well as generalizing RAML to additional structure classes and reward functions [1705.07136][1804.10974].

Source: https://www.emergentmind.com/topics/reward-augmented-maximum-likelihood-raml