---
title: Bayesian Pseudo-Coresets
url: https://www.emergentmind.com/topics/bayesian-pseudo-coresets
type: topic
---

# Bayesian Pseudo-Coresets

A Bayesian pseudo-coreset is a compact, learned synthetic dataset—typically orders of magnitude smaller than the original dataset—which, when used for Bayesian inference, induces a posterior closely approximating that of the full data. Pseudo-coresets can be weighted subsets of real data (traditional coresets), learned pseudo-examples, or a combination, with the central goal of drastically reducing computational costs for posterior inference and Bayesian model averaging, particularly in large-scale or high-dimensional regimes [2210.06205, 2310.17852, 2502.21143]. The development of Bayesian pseudo-coreset methods draws on advances in information geometry, variational inference, dataset distillation, energy-based models, and scalable optimization.

## 1. Formal Definition and Conceptual Framework

Let $D = \{(x_i, y_i)\}_{i=1}^N$ denote a dataset, $\theta \in \Theta$ model parameters, and $\pi_0(\theta)$ a prior. The true posterior is
\[
\pi_D(\theta) \propto \pi_0(\theta) \prod_{i=1}^N p(y_i|x_i, \theta)
\]
Bayesian pseudo-coreset methods aim to find a much smaller "synthetic dataset" $S = \{(\hat x_j, \hat y_j)\}_{j=1}^M$ ($M \ll N$) and associated weights $\mathbf{w}$, such that the "pseudo-posterior"
\[
\pi_S(\theta) \propto \pi_0(\theta) \prod_{j=1}^M p(\hat y_j|\hat x_j, \theta)^{w_j}
\]
approximates $\pi_D(\theta)$ in a task-suitable sense.

Construction proceeds by minimizing a divergence $D(\pi_S\|\pi_D)$ over the pseudo-coreset parameters (synthetic inputs, labels, and weights). Choices for $D$ include reverse KL, forward KL, Wasserstein distance, contrastive divergence, and function-space divergences [2210.06205, 2310.17852, 2303.11278]. Pseudo-coresets generalize classical Bayesian coresets (which subsample and reweight actual data points) to learned pseudo-datapoints with greater expressive power.

## 2. Divergence Objectives, Algorithms, and Methodological Taxonomy

Construction algorithms are differentiated by the divergence minimized:

- **Reverse KL (BPC-rKL):** $\mathrm{KL}[\pi_S\|\pi_D]$ is mode-seeking and penalizes synthetic distributions that place mass where the true posterior does not. Practically, this can correspond to gradient-matching and is closely related to certain dataset distillation techniques [2210.06205].

- **Forward KL (BPC-fKL):** $\mathrm{KL}[\pi_D\|\pi_S]$ is mass-covering; it heavily penalizes under-coverage and generally produces pseudo-datasets that better span the posterior support. This objective is more robust in high-dimensional and multimodal settings [2210.06205].

- **Wasserstein (BPC-W):** Minimizes the expected squared distance between sampled parameters from each posterior, closely tied to training-trajectory matching [2210.06205].

- **Contrastive Divergence (CD-BPC):** Leverages a two-term structure analogous to training energy-based models, where the divergence to minimize is $\mathrm{KL}(p\|q^*) - \mathrm{KL}(\Pi_E^M p\|q^*)$, with $q^*$ the pseudo-coreset posterior, and $\Pi_E$ an MCMC kernel targeting $q^*$ [2303.11278].

- **Function-Space Divergences (FBPC):** Operate directly on the predictive function distribution (rather than weights), which can circumvent weight-space multimodality and improve scalability in deep Bayesian networks [2310.17852].

- **Beta-Divergence and Robustification:** By modifying the traditional likelihood with a $\beta$-divergence, methods such as $\beta$-Cores mitigate the impact of outliers, yielding robust pseudo-posteriors by reducing the influence function [2008.13600].

- **Variational Pseudo-Coresets (VBPC, BB-PSVI):** Recent methods formulate the coreset problem as a variational inference task, often in a bi-level or black-box manner, typically optimizing an ELBO on the full-data evidence via closed-form inner-loop solutions or importance-weighted outer-objectives [2502.21143, 2211.02377].

A generalized construction loop may be summarized as:
- Initialize synthetic data (either drawn from real data or random noise).
- Alternate between sampling/simulating from current pseudo-posterior and taking divergence-based gradient steps to adjust pseudo-core parameters.
- Terminate upon convergence of divergence or a fixed epoch budget.

## 3. Theoretical Guarantees and Robustness

Approximation guarantees underpinning Bayesian pseudo-coresets derive from properties of the selected divergence and the structure of the summarized posterior:

- For exponential-family posteriors and Hilbert-space coreset constructions, projection theorems ensure that closeness in divergence implies closeness of posterior means in function norm, with $O(1/\sqrt{M})$ worst-case bounds under mild regularity [2008.13600].
- Under Gaussian models, Hamiltonian flow coreset methods achieve exponential compression: it suffices to select $M = O(d \log N)$ points—where $d$ is the model dimension—to achieve zero KL-divergence between posterior and coreset posterior with high probability [2203.05723].
- $\beta$-divergence–based robustness ensures the influence function of outliers is universally bounded for $\beta>0$ and guarantees that an $O(1)$ fraction of outliers can be ignored as $N \to \infty$ [2008.13600].

These results are complemented by descent and convergence properties for contrastive divergence and variational coreset optimization [2303.11278, 2211.02377]. Most guarantees are empirical or finite-sample, with theoretical optimality proven primarily in tractable Gaussian/linear models.

## 4. Empirical Performance, Scalability, and Applications

Bayesian pseudo-coresets demonstrate significant empirical benefits across synthetic, regression, and high-dimensional deep learning applications:

- **Classification and Regression:** On datasets such as MNIST, CIFAR-10/100, Tiny-ImageNet, coreset-induced posteriors match or exceed the accuracy and NLL of full-batch or data-distillation baselines, especially at low coreset size (e.g., $M=100$ on CIFAR-10) [2310.17852, 2502.21143, 2303.11278].
- **Computational Costs:** State-of-the-art variational approaches (e.g., VBPC) offer $\sim50\%$ lower memory usage and $5-10\times$ faster training/inference compared to prior approaches, owing to closed-form inner solvers and efficient marginalizations [2502.21143].
- **Robustness:** $\beta$-Cores maintain near-clean posterior estimates and test accuracies under $10$–$30\%$ structured outlier contamination, outperforming both classical coresets and alternatives under corruption [2008.13600].
- **Function-Space and Architecture Generalization:** FBPC constructions transfer effectively across model architectures, achieving robust uncertainties and superior out-of-distribution generalization [2310.17852].
- **Continual and Large-Scale Learning:** Only variational and function-space methods scale to large images ($128\times128$), ImageNet1k, and continual learning tasks while retaining computational tractability and information retention [2502.21143].

Empirical comparisons are typically reported in terms of test accuracy, negative log-likelihood, calibration (ECE/Brier), OOD robustness, and training/inference time.

## 5. Practical Algorithmic Schemes

A selection of prominent algorithmic and architectural designs includes:

| Algorithm         | Objective           | Notable Features                                     |
|-------------------|--------------------|------------------------------------------------------|
| $\beta$-Cores     | $\beta$-divergence | Outlier-robust, Riemannian Hilbert coreset, black-box|
| BPC-fKL           | Forward KL         | Mass-covering, memory/scaling-efficient, HMC/SGHMC   |
| BPC-CD            | Contrastive Div.   | Short-run MCMC, no variational approx needed         |
| FBPC              | Func-space KL      | Low-dim marginals, cross-architecture transfer       |
| VBPC              | Var. ELBO          | Closed-form last-layer, single-pass BMA, low memory  |
| BB-PSVI           | Var. ELBO          | Black-box, IS-corrected, pseudo/real data mix        |

Each method provides variants for optimization (e.g., projected stochastic gradient, Adam), and data selection (uniform, noise, learned inputs/labels). Several methods employ IS weighting or Monte Carlo (SGHMC, HMC) for tractable expectations.

## 6. Limitations and Open Challenges

Key open challenges and limitations include:

- Importance-sampling degradation in high dimensions for black-box VI approaches [2211.02377].
- Under-representation of uncertainty and potential multimodality gaps for last-layer or function-space surrogates [2502.21143, 2310.17852].
- Lack of formal global error bounds for most divergences outside Gaussian models [2310.17852].
- Computational cost of repeated inner-loop (e.g., MAP) solves for current function-space methods, and limitations of Gaussian/mean-field surrogates in fully non-linear settings [2310.17852].
- Practical trade-off between coreset compactness and posterior fidelity; guidelines for the optimal $M$ for arbitrary tasks remain empirical.

Active research directions involve hierarchical pseudo-coresets, amortized/automated MAC optimizers, novel divergence design (e.g., relaxations or new functionals), and applications to extremely large-scale models such as vision transformers and LLMs [2502.21143, 2210.06205].

## 7. Summary and Significance

Bayesian pseudo-coresets represent a highly active intersection of scalable Bayesian inference, probabilistic deep learning, and data summarization. By minimizing suitable divergences between synthetic-coreset and full-data posteriors, these methods compress large datasets to compact pseudo-representations without sacrificing statistical fidelity or uncertainty quantification. Developments such as function-space matching, robust $\beta$-divergence objectives, and scalable variational/black-box optimization, have enabled application to high-dimensional settings such as Bayesian neural networks, continual learning, and large-scale image classification.

Synthetic pseudo-coresets are highly practical for settings where privacy, computation, or distributed inference precludes access to the original data, and are amenable to further research on theoretical guarantees, robustness, and multi-task/utilization in emerging machine learning architectures [2310.17852, 2210.06205, 2502.21143, 2008.13600].

Source: https://www.emergentmind.com/topics/bayesian-pseudo-coresets