---
title: Reverse AIS Estimator (RAISE)
url: https://www.emergentmind.com/topics/reverse-ais-estimator-raise
type: topic
---

# Reverse AIS Estimator (RAISE)

The Reverse AIS Estimator (RAISE) is a stochastic estimator designed to provide conservative (lower bound) estimates of log-likelihoods for undirected graphical models such as Restricted Boltzmann Machines (RBMs), Deep Boltzmann Machines (DBMs), and Deep Belief Networks (DBNs). Its primary utility lies in reliably evaluating models for which the partition function is intractable, where standard Annealed Importance Sampling (AIS) may yield over-optimistic model evaluations due to a tendency to underestimate the partition function. RAISE blends ideas from AIS and Conservative Sampling-based Likelihood (CSL) to produce test log-likelihood lower bounds that are both practical and accurate for complex generative models [1510.02255], [1412.8566].

## 1. Problem Setting and Motivation

Undirected probabilistic graphical models like RBMs define densities of the form $p(x) = f(x) / Z$, where $f(x)$ is typically straightforward to compute, but the partition function $Z = \sum_{x} f(x)$ is intractable for high-dimensional $x$. Assessing model fit on held-out test data requires estimating the average test log-likelihood $\mathcal{L} = \frac{1}{N}\sum_{i=1}^N \log p(x^{(i)})$. Since $Z$ is unknown, one relies on stochastic estimators.

AIS is widely used for partition function estimation. By Jensen’s inequality, $E[\log \hat Z_\text{AIS}] \leq \log Z$, causing AIS to typically underestimate $\log Z$, and hence the estimated log-likelihood $\log p(x) = \log f(x) - \log \hat Z$ becomes an overestimate—making AIS a non-conservative, potentially misleading estimator in practice [1412.8566].

## 2. RAISE: Formulation and Theoretical Properties

RAISE reframes the estimation of $p(x)$ as the computation of a partition function of an augmented joint distribution, enabling the use of AIS in a reverse mode to yield a stochastic lower bound on $\log p(x)$. For RBMs, the marginalized likelihood is
\[
p(v) = \sum_h p(h)\,p(v\mid h),
\]
where $v$ are the visible units and $h$ the hidden. Identifying the summand as $f(h) = p(h)p(v|h)$, its sum over $h$ gives the partition function for this "augmented" distribution [1510.02255].

A sequence of $K+1$ intermediate distributions is specified as
\[
f_k(x) \propto f_0(x)^{1-\beta_k} f_K(x)^{\beta_k}, \quad 0 = \beta_0 < \cdots < \beta_K = 1,
\]
where $f_0$ is a tractable proposal (e.g., uniform or base-rate), and $f_K$ is the target [1510.02255], [1412.8566].

The RAISE estimator for $p(v)$ after a single reverse chain is
\[
\hat p(v) = \frac{f_K(v)}{Z_0} \prod_{k=1}^{K} \frac{f_{k-1}(x_k)}{f_k(x_k)},
\]
where $Z_0$ is the known normalizer for $f_0$, and the sequence $\{x_k\}$ is constructed by running a Markov chain *backwards* from the observed $v$ through the annealing path [1510.02255].

The estimator satisfies $E[\hat p(v)] = p(v)$, and by Jensen’s inequality $E[\log \hat p(v)] \leq \log p(v)$, ensuring a conservative (lower bound) estimate [1412.8566].

## 3. Algorithmic Description

A single-chain version of RAISE for RBMs proceeds as follows [1510.02255]:

```plaintext
Input:  test point v, model parameters θ, proposal f₀, β-schedule {β₀,…,β_K}, steps K
Output: estimate  p(v)

1. Precompute: Z₀ ← partition function of proposal f₀ (analytic)
2. Initialize:  x_K ← v; w ← f_K(v)/Z₀
3. For k = K downto 1:
     a) Sample x_{k−1} ∼ T_k(· | x_k) (transitions keep f_k invariant)
     b) Compute r ← f_{k-1}(x_{k-1}) / f_k(x_{k-1})
     c) Update w ← w × r
     d) Set x_k ← x_{k-1}
4. Return: p̂(v) = w
```
Running this estimator for each test datum and averaging yields an estimate of the average log-likelihood [1510.02255].

For tractable posteriors (e.g., RBMs), the reverse chain can sample directly from $p_K(h|v)$ at initialization. For intractable cases (e.g., DBMs), additional “heating” transitions are necessary as described in [1412.8566].

## 4. Hyperparameters and Estimation Quality

The principal hyperparameters include:

- **Number of intermediate distributions ($K$):** Increasing $K$ reduces bias and variance at linear computational cost. Large $K$ is critical for ensuring conservativeness, especially with a uniform proposal.
- **$\beta$-schedule:** Linear spacing is standard, but denser schedules in regions of rapid distributional change may reduce variance.
- **Proposal $f_0$:** A uniform proposal is safest for conservativeness but requires large $K$. A base-rate RBM proposal can yield smaller bias but may slightly overestimate unless $K$ is very large.
- **Reverse chain runs per datum:** While a single reverse chain can suffice for large $K$, multiple chains further reduce variance [1510.02255].
- **Variance-reduction:** Subtracting $\log f(x)$ as a control variate substantially reduces estimator variance over the test set [1412.8566].

## 5. Empirical Performance and Computational Cost

RAISE, AIS, and CSL were compared by [1510.02255] on MNIST using RBMs with 20, 200, and 500 hidden units. Representative log-likelihood results (average over 500 test points) are shown below:

| n hidden | AIS         | CSL         | RAISE (uniform) | RAISE (base-rate) |
|----------|-------------|-------------|-----------------|-------------------|
| 20       | –142.38     | –143.58     | –145.99         | –144.14           |
| 200      | –112.96     | –142.64     | –112.46         | –109.01           |
| 500      | –116.46     | –154.76     | –118.02         | –112.04           |

Interpretation:

- **AIS** generally provides the tightest (but potentially over-optimistic) estimates due to its tendency to underestimate $\log Z$.
- **CSL** is highly conservative with significant downward bias unless very large numbers of Gibbs samples are used.
- **RAISE** is a lower-bound estimator; for $K \geq 10\,000$, RAISE's log-likelihood estimates closely approach AIS and the (known for small $n$) ground truth, outperforming CSL by a significant margin. A base-rate proposal may slightly overestimate unless $K$ is high, while uniform is strictly conservative (but slower).
- **Computational cost:** RAISE, requiring one full reverse chain per test datum, is two to three orders of magnitude more expensive on MNIST than AIS (which only needs multiple chains per model), and more costly than CSL [1510.02255], [1412.8566].

More extensive experiments on RBMs, DBMs, and DBNs (including Omniglot and larger models, [1412.8566]) confirm that RAISE typically underestimates test log-likelihood by less than 1 nat relative to AIS, demonstrating that AIS and RAISE tightly bracket the true value. Empirical plots show AIS leveling off to a possibly optimistic value, while RAISE's lower bound rises monotonically to converge just below AIS [1412.8566].

## 6. Relationships to CSL and AIS

RAISE synthesizes elements of both CSL and AIS:

- **CSL:** Estimates $p(v)$ by Monte Carlo, but suffers strong downward (conservative) bias due to Jensen's inequality.
- **AIS:** Provides unbiased $\hat Z$, yielding stochastic lower bounds on $\log Z$, but upper bounds on $\mathcal{L}$.
- **RAISE:** By casting $\sum_h p(h)p(v|h)$ as a partition function and applying AIS “in reverse,” RAISE produces an unbiased estimator of $p(v)$. Jensen’s inequality assures conservativeness: $E[\log \hat p(v)] \leq \log p(v)$, e.g., a true stochastic lower bound [1510.02255], [1412.8566].

This lower-bound property holds for any finite $K$; as $K \to \infty$, the bound converges to the true log-likelihood of the model.

## 7. Practical Recommendations

RAISE requires only the MCMC transition kernels used in standard AIS, making implementation straightforward for those with an existing AIS codebase. Gibbs sampling is commonly used for these transitions. Using a data base-rate proposal improves accuracy and reduces error for both AIS and RAISE. Subtracting $\log f(x)$ as a control variate dramatically reduces variance in empirical settings [1412.8566].

For RBMs with moderate numbers of hidden units, RAISE is feasible on modern hardware, but for datasets with many test examples or expensive forward/reverse chains, the cost may be substantial. In models with intractable posteriors, additional “diagnostic” heating chains are required for correctness [1412.8566].

RAISE thus provides a robust, conservative methodology for evaluating generative undirected models, complementing standard AIS with rigorous lower-bound guarantees on log-likelihood estimation [1510.02255], [1412.8566].

Source: https://www.emergentmind.com/topics/reverse-ais-estimator-raise