Bayesian Pseudo-Coresets

• Bayesian pseudo-coresets are compact, synthetic datasets that approximate full-data Bayesian posteriors while drastically reducing computational costs.
• They employ divergence minimization techniques including reverse KL, forward KL, and Wasserstein distance to optimize synthetic data construction.
• These methods enhance scalability and robustness in high-dimensional settings, benefiting applications like deep learning and continual inference.

A Bayesian pseudo-coreset is a compact, learned synthetic dataset—typically orders of magnitude smaller than the original dataset—which, when used for Bayesian inference, induces a posterior closely approximating that of the full data. Pseudo-coresets can be weighted subsets of real data (traditional coresets), learned pseudo-examples, or a combination, with the central goal of drastically reducing computational costs for posterior inference and Bayesian model averaging, particularly in large-scale or high-dimensional regimes (Kim et al., 2022, Kim et al., 2023, Lee et al., 28 Feb 2025). The development of Bayesian pseudo-coreset methods draws on advances in information geometry, variational inference, dataset distillation, energy-based models, and scalable optimization.

1. Formal Definition and Conceptual Framework

Let $D = \{(x_i, y_i)\}_{i=1}^N$ denote a dataset, $\theta \in \Theta$ model parameters, and $\pi_0(\theta)$ a prior. The true posterior is

$$\pi_D(\theta) \propto \pi_0(\theta) \prod_{i=1}^N p(y_i|x_i, \theta)$$

Bayesian pseudo-coreset methods aim to find a much smaller "synthetic dataset" $S = \{(\hat x_j, \hat y_j)\}_{j=1}^M$ ($M \ll N$) and associated weights $\mathbf{w}$, such that the "pseudo-posterior"

$$\pi_S(\theta) \propto \pi_0(\theta) \prod_{j=1}^M p(\hat y_j|\hat x_j, \theta)^{w_j}$$

approximates $\pi_D(\theta)$ in a task-suitable sense.

Construction proceeds by minimizing a divergence $D(\pi_S\|\pi_D)$ over the pseudo-coreset parameters (synthetic inputs, labels, and weights). Choices for $\theta \in \Theta$0 include reverse KL, forward KL, Wasserstein distance, contrastive divergence, and function-space divergences (Kim et al., 2022, Kim et al., 2023, Tiwary et al., 2023). Pseudo-coresets generalize classical Bayesian coresets (which subsample and reweight actual data points) to learned pseudo-datapoints with greater expressive power.

2. Divergence Objectives, Algorithms, and Methodological Taxonomy

Construction algorithms are differentiated by the divergence minimized:

• Reverse KL (BPC-rKL): $\theta \in \Theta$1 is mode-seeking and penalizes synthetic distributions that place mass where the true posterior does not. Practically, this can correspond to gradient-matching and is closely related to certain dataset distillation techniques (Kim et al., 2022).
• Forward KL (BPC-fKL): $\theta \in \Theta$2 is mass-covering; it heavily penalizes under-coverage and generally produces pseudo-datasets that better span the posterior support. This objective is more robust in high-dimensional and multimodal settings (Kim et al., 2022).
• Wasserstein (BPC-W): Minimizes the expected squared distance between sampled parameters from each posterior, closely tied to training-trajectory matching (Kim et al., 2022).
• Contrastive Divergence (CD-BPC): Leverages a two-term structure analogous to training energy-based models, where the divergence to minimize is $\theta \in \Theta$3, with $\theta \in \Theta$4 the pseudo-coreset posterior, and $\theta \in \Theta$5 an MCMC kernel targeting $\theta \in \Theta$6 (Tiwary et al., 2023).
• Function-Space Divergences (FBPC): Operate directly on the predictive function distribution (rather than weights), which can circumvent weight-space multimodality and improve scalability in deep Bayesian networks (Kim et al., 2023).
• Beta-Divergence and Robustification: By modifying the traditional likelihood with a $\theta \in \Theta$7-divergence, methods such as $\theta \in \Theta$8-Cores mitigate the impact of outliers, yielding robust pseudo-posteriors by reducing the influence function (Manousakas et al., 2020).
• Variational Pseudo-Coresets (VBPC, BB-PSVI): Recent methods formulate the coreset problem as a variational inference task, often in a bi-level or black-box manner, typically optimizing an ELBO on the full-data evidence via closed-form inner-loop solutions or importance-weighted outer-objectives (Lee et al., 28 Feb 2025, Manousakas et al., 2022).

A generalized construction loop may be summarized as:

• Initialize synthetic data (either drawn from real data or random noise).
• Alternate between sampling/simulating from current pseudo-posterior and taking divergence-based gradient steps to adjust pseudo-core parameters.
• Terminate upon convergence of divergence or a fixed epoch budget.

3. Theoretical Guarantees and Robustness

Approximation guarantees underpinning Bayesian pseudo-coresets derive from properties of the selected divergence and the structure of the summarized posterior:

• For exponential-family posteriors and Hilbert-space coreset constructions, projection theorems ensure that closeness in divergence implies closeness of posterior means in function norm, with $\theta \in \Theta$9 worst-case bounds under mild regularity (Manousakas et al., 2020).
• Under Gaussian models, Hamiltonian flow coreset methods achieve exponential compression: it suffices to select $\pi_0(\theta)$0 points—where $\pi_0(\theta)$1 is the model dimension—to achieve zero KL-divergence between posterior and coreset posterior with high probability (Chen et al., 2022).
• $\pi_0(\theta)$2-divergence–based robustness ensures the influence function of outliers is universally bounded for $\pi_0(\theta)$3 and guarantees that an $\pi_0(\theta)$4 fraction of outliers can be ignored as $\pi_0(\theta)$5 (Manousakas et al., 2020).

These results are complemented by descent and convergence properties for contrastive divergence and variational coreset optimization (Tiwary et al., 2023, Manousakas et al., 2022). Most guarantees are empirical or finite-sample, with theoretical optimality proven primarily in tractable Gaussian/linear models.

4. Empirical Performance, Scalability, and Applications

Bayesian pseudo-coresets demonstrate significant empirical benefits across synthetic, regression, and high-dimensional deep learning applications:

• Classification and Regression: On datasets such as MNIST, CIFAR-10/100, Tiny-ImageNet, coreset-induced posteriors match or exceed the accuracy and NLL of full-batch or data-distillation baselines, especially at low coreset size (e.g., $\pi_0(\theta)$6 on CIFAR-10) (Kim et al., 2023, Lee et al., 28 Feb 2025, Tiwary et al., 2023).
• Computational Costs: State-of-the-art variational approaches (e.g., VBPC) offer $\pi_0(\theta)$7 lower memory usage and $\pi_0(\theta)$8 faster training/inference compared to prior approaches, owing to closed-form inner solvers and efficient marginalizations (Lee et al., 28 Feb 2025).
• Robustness: $\pi_0(\theta)$9-Cores maintain near-clean posterior estimates and test accuracies under $$\pi_D(\theta) \propto \pi_0(\theta) \prod_{i=1}^N p(y_i|x_i, \theta)$$0–$$\pi_D(\theta) \propto \pi_0(\theta) \prod_{i=1}^N p(y_i|x_i, \theta)$$1 structured outlier contamination, outperforming both classical coresets and alternatives under corruption (Manousakas et al., 2020).
• Function-Space and Architecture Generalization: FBPC constructions transfer effectively across model architectures, achieving robust uncertainties and superior out-of-distribution generalization (Kim et al., 2023).
• Continual and Large-Scale Learning: Only variational and function-space methods scale to large images ($$\pi_D(\theta) \propto \pi_0(\theta) \prod_{i=1}^N p(y_i|x_i, \theta)$$2), ImageNet1k, and continual learning tasks while retaining computational tractability and information retention (Lee et al., 28 Feb 2025).

Empirical comparisons are typically reported in terms of test accuracy, negative log-likelihood, calibration (ECE/Brier), OOD robustness, and training/inference time.

5. Practical Algorithmic Schemes

A selection of prominent algorithmic and architectural designs includes:

Algorithm	Objective	Notable Features
$$\pi_D(\theta) \propto \pi_0(\theta) \prod_{i=1}^N p(y_i|x_i, \theta)$$3-Cores	$$\pi_D(\theta) \propto \pi_0(\theta) \prod_{i=1}^N p(y_i|x_i, \theta)$$4-divergence	Outlier-robust, Riemannian Hilbert coreset, black-box
BPC-fKL	Forward KL	Mass-covering, memory/scaling-efficient, HMC/SGHMC
BPC-CD	Contrastive Div.	Short-run MCMC, no variational approx needed
FBPC	Func-space KL	Low-dim marginals, cross-architecture transfer
VBPC	Var. ELBO	Closed-form last-layer, single-pass BMA, low memory
BB-PSVI	Var. ELBO	Black-box, IS-corrected, pseudo/real data mix

Each method provides variants for optimization (e.g., projected stochastic gradient, Adam), and data selection (uniform, noise, learned inputs/labels). Several methods employ IS weighting or Monte Carlo (SGHMC, HMC) for tractable expectations.

6. Limitations and Open Challenges

Key open challenges and limitations include:

• Importance-sampling degradation in high dimensions for black-box VI approaches (Manousakas et al., 2022).
• Under-representation of uncertainty and potential multimodality gaps for last-layer or function-space surrogates (Lee et al., 28 Feb 2025, Kim et al., 2023).
• Lack of formal global error bounds for most divergences outside Gaussian models (Kim et al., 2023).
• Computational cost of repeated inner-loop (e.g., MAP) solves for current function-space methods, and limitations of Gaussian/mean-field surrogates in fully non-linear settings (Kim et al., 2023).
• Practical trade-off between coreset compactness and posterior fidelity; guidelines for the optimal $$\pi_D(\theta) \propto \pi_0(\theta) \prod_{i=1}^N p(y_i|x_i, \theta)$$5 for arbitrary tasks remain empirical.

Active research directions involve hierarchical pseudo-coresets, amortized/automated MAC optimizers, novel divergence design (e.g., relaxations or new functionals), and applications to extremely large-scale models such as vision transformers and LLMs (Lee et al., 28 Feb 2025, Kim et al., 2022).

7. Summary and Significance

Bayesian pseudo-coresets represent a highly active intersection of scalable Bayesian inference, probabilistic deep learning, and data summarization. By minimizing suitable divergences between synthetic-coreset and full-data posteriors, these methods compress large datasets to compact pseudo-representations without sacrificing statistical fidelity or uncertainty quantification. Developments such as function-space matching, robust $$\pi_D(\theta) \propto \pi_0(\theta) \prod_{i=1}^N p(y_i|x_i, \theta)$$6-divergence objectives, and scalable variational/black-box optimization, have enabled application to high-dimensional settings such as Bayesian neural networks, continual learning, and large-scale image classification.

Synthetic pseudo-coresets are highly practical for settings where privacy, computation, or distributed inference precludes access to the original data, and are amenable to further research on theoretical guarantees, robustness, and multi-task/utilization in emerging machine learning architectures (Kim et al., 2023, Kim et al., 2022, Lee et al., 28 Feb 2025, Manousakas et al., 2020).