---
title: Stochastic Expectation Propagation (SEP)
url: https://www.emergentmind.com/topics/stochastic-expectation-propagation-sep
type: topic
---

# Stochastic Expectation Propagation (SEP)

Stochastic Expectation Propagation (SEP) is an approximate Bayesian inference algorithm that merges the local refinement advantages of classical Expectation Propagation (EP) with the memory efficiency of variational methods. It achieves this by replacing the set of $N$ per-datapoint factors used in EP with a single, shared global factor—enabling constant-memory scalable inference, while largely retaining the quality of posterior uncertainty quantification characteristic of EP. SEP was first comprehensively described by Li, Hernández-Lobato, and Turner (2015) [1506.04132] and has since seen numerous theoretical and practical extensions [1511.03249], [1511.03405], [2111.13219].

## 1. Principles and Motivation

Expectation Propagation (EP) approximates an intractable posterior density
\[
p(\theta|\mathcal{D}) \propto p_0(\theta)\prod_{n=1}^N p(x_n|\theta)
\]
with a tractable distribution
\[
q_\mathrm{EP}(\theta) \propto p_0(\theta)\prod_{n=1}^N f_n(\theta)
\]
where each $f_n$ is an exponential-family site parameter refined by local moment-matching. This yields accurate posterior variances but incurs $O(N)$ memory, as all site factors must be stored.

Variational Inference (VI), by contrast, posits a single global approximate posterior and fits it to minimize KL divergence. VI’s memory cost is $O(1)$ in $N$, but it often underestimates posterior uncertainty due to factorized (mean-field) approximations.

SEP enforces a tied parameterization:
\[
q_\mathrm{SEP}(\theta) \propto p_0(\theta)\,[f(\theta)]^N
\]
maintaining a single average site factor $f(\theta)$, which undergoes stochastic, EP-like moment-matching updates. This achieves the $O(1)$ memory footprint of VI while preserving the local update mechanism of EP, and empirically delivers posterior calibrations competitive with full EP [1506.04132], [1511.03405], [2111.13219].

## 2. Algorithmic Structure and Updates

SEP operates by iteratively sampling datapoints (or minibatches) and performing update steps analogous to EP but aggregated into the global site. The canonical single-data SEP update proceeds as follows:

1. **Cavity Distribution**: Remove one copy of the global factor,
   \[
   q_{-1}(\theta) \propto p_0(\theta)\,[f(\theta)]^{N-1}
   \]
2. **Tilted Distribution**: Form the local posterior for datapoint $x_n$,
   \[
   \tilde{p}_n(\theta) \propto p(x_n|\theta)\,q_{-1}(\theta)
   \]
3. **Moment Matching**: Project $\tilde{p}_n$ back to the exponential family, yielding an intermediate site $f_n$ by matching moments:
   \[
   f_n(\theta) \leftarrow \operatorname{proj}[\tilde{p}_n(\theta)] / q_{-1}(\theta)
   \]
4. **Damped Global Update**: Update the shared factor by moving towards the intermediate site,
   \[
   f(\theta) \leftarrow f(\theta)^{1-\epsilon}\,f_n(\theta)^\epsilon
   \]
   where $\epsilon = 1/N$ is typical, but can be adapted or annealed.

In practice, minibatch updates generalize this scheme, with $M$ intermediate sites $f_{n_j}$ being aggregated:
\[
f(\theta) \leftarrow f(\theta)^{1-M/N}\prod_{j=1}^M [f_{n_j}(\theta)]^{1/N}
\]
This framework allows for Robbins–Monro step-size schedules and stochastic optimization techniques, supporting scalable learning on large datasets [1506.04132], [1511.03405].

## 3. Theoretical Characteristics and Convergence

SEP is formally a stochastic approximation to the fixed-point equations of Averaged-EP (AEP), which itself shares fixed points with full EP in the large-$N$ regime under mild regularity conditions (e.g., log-concave sites, slow parameter evolution). Under diminishing step sizes $\epsilon_t$ satisfying standard Robbins–Monro conditions, SEP converges almost surely to a stationary point of the EP energy function [1506.04132].

The use of a single shared factor means SEP approximates the geometric mean of all likelihood terms, so in the limit of infinitesimal learning rate, SEP's global approximation minimizes the KL divergence between $p_0(\theta) f(\theta)^N$ and the true posterior.

A notable limitation is that the tied global factor introduces a slight loss in modeling flexibility relative to per-datapoint factors, which can be observed in pathological cases (e.g., highly heterogeneous likelihoods). However, empirical results show that SEP retains near-EP calibration for posterior uncertainty on a wide spectrum of canonical models (probit regression, mixture models, Bayesian neural nets) [1506.04132], [1511.03249].

## 4. Computational Benefits and Scaling

SEP’s major computational advantage is its $O(1)$ memory cost with respect to the number of datapoints, in contrast to EP's $O(N)$. For Gaussian factors with parameter dimension $d$, SEP stores only the parameters of a single global factor (e.g., mean and covariance for Gaussian $f(\theta)$), while EP must maintain $N$ such factors.

Per-iteration computational complexity for online updates is $O(M \cdot C_{\text{upd}})$ for minibatch size $M$, which matches that of EP amortized over the dataset but without the memory overhead. For large-scale latent Gaussian models, such as sparse GP classification, this translates to $O(m^2)$ memory and $O(m^3)$-dominant computational step (inverting a covariance of dimension $m$) per iteration—enabling application to datasets with $n \gg 10^5$ [1511.03249], [1511.03405].

## 5. Extensions and Practical Variants

Significant algorithmic and practical refinements have built upon SEP’s foundation:

- **Differentially Private SEP (DP-SEP)**: By injecting Gaussian noise into the global factor's parameter updates and clipping the norm of intermediate sites, SEP can be made $(\epsilon,\delta)$-differentially private. The analysis demonstrates that with increasing $N$, privacy-induced approximation error vanishes as $N^{-1}$, allowing for strong privacy guarantees with minimal loss in posterior accuracy [2111.13219].
- **Deep Gaussian Processes and Probabilistic Backpropagation**: SEP has been used to construct scalable Bayesian learning algorithms for deep GP models. Here, the site updates are integrated with probabilistic backpropagation, exploiting SEP's memory efficiency and stochastic updates for layer-wise uncertainty propagation [1511.03405].
- **Minibatched/Distributed Implementations**: SEP’s update structure supports efficient parallelization via minibatches. Its global factor update admits distributed architectures, fitting streaming or cluster settings [1511.03405].

Empirically, SEP closely tracks EP in test log-likelihoods, KL to NUTS ground truth, and posterior coverage metrics, across synthetic and real datasets (e.g., Bayesian probit regression, mixture models, Bayesian neural networks, deep GPs, sparse GP classification tasks on large UCI and MNIST datasets) [1506.04132], [1511.03249], [1511.03405].

## 6. Best Practices and Limitations

Numerical stability is critical, requiring damping of site updates, addition of jitter to covariance matrices, and normalization of natural parameters to ensure positive-definiteness. The global factor update should use an appropriately annealed step-size or batch-damped inclusion, especially for high-dimensional or ill-conditioned factors.

In deep or hierarchical Gaussian models, per-iteration overhead can be dominated by $O(d^3)$ operations (for full-covariance transformations). Practical implementations often utilize low-rank approximations or parallel aggregation.

SEP is less flexible in modeling highly heterogeneous data than full EP due to the tied global factor. However, in regimes where the per-data likelihoods are relatively homogeneous, or in high-data settings where averaging is effective, SEP achieves strong uncertainty quantification with minimal resources [1506.04132], [1511.03249].

## 7. Relationship with Other Stochastic EP Variants

SEP is complementary to stochastic natural-gradient EP (SNEP) [1512.09327], which instead retains per-site parameters but employs natural-gradient updates with stochastic moment estimates and explicit convergence-theory under Monte Carlo noise. While SNEP offers theoretical guarantees and supports distributed architectures, SEP is preferred when memory is the limiting factor and per-site storage is prohibitive.

Recent advances in natural-gradient perspectives on EP further refine stochastic updates for site parameters, enhancing stability and tuning by directly leveraging natural-parameter or mean-parameter NGD, as in the EP-η and EP-μ schemes [2406.01801]. These methods enable stable single-sample Monte Carlo estimation for each site, trading off small bias versus noisy progress, and can be integrated in SEP-style frameworks for further memory reduction.

---

**Key References**

| Paper Title                                                    | arXiv ID      | Main Contribution                                       |
|:---------------------------------------------------------------|:--------------|:--------------------------------------------------------|
| Stochastic Expectation Propagation                             | 1506.04132    | Introduced SEP, theory, and empirical validation        |
| Stochastic EP for Large Scale Gaussian Process Classification  | 1511.03249    | Scalable SEP in GP settings, memory/performance analysis|
| DP-SEP                                                         | 2111.13219    | Differentially private SEP, theory and experiments      |
| Training DGPs with SEP and Probabilistic Backpropagation       | 1511.03405    | SEP-based deep GP learning, implementation insights     |
| Fearless Stochasticity in Expectation Propagation              | 2406.01801    | Natural-gradient EP, robust single-sample updates       |
| Distributed Bayesian Learning with SNEP                        | 1512.09327    | Stochastic natural-gradient EP, theory, distributed arch|

Source: https://www.emergentmind.com/topics/stochastic-expectation-propagation-sep