Efficient and Fast Sampling from Arbitrary Probability Kernels using Sliced Gibbs Sampler

Abstract: An Automated Sliced Gibbs framework is proposed for fully automated Markov chain Monte Carlo sampling from arbitrary finite dimensional probability kernels. The method targets unnormalized, non-smooth, heavy tailed, and highly multimodal densities. A Cauchy transformation based effective support estimator is combined with slice driven Gibbs updates. This construction removes the need for user specified truncation bounds, proposal scales, step-size tuning, or conditional optimization. Unlike existing slice samplers, ASG does not require manually chosen bracket widths or geometric insight into the support. All calibration is performed automatically within each Gibbs cycle. The resulting Markov chain preserves invariance and ergodicity. Automated support detection allows efficient movement across disconnected high density regions. The sampler adapts to sharp curvature and irregular geometry without gradient information. Extensive numerical experiments evaluate performance on complex kernels, including univariate Beta mixtures, multivariate Rosenbrock and Ackley benchmarks, and non-smooth kernels derived from generalized LASSO type loss functions. Across these challenging settings, ASG consistently achieves higher effective sample size per second and faster decorrelation than Random Walk Metropolis Hastings, adaptive Gibbs variants, and some recently proposed slice based methods. The framework provides a scalable and general-purpose solution for sampling from complicated probability kernels where existing algorithms require substantial tuning or exhibit slow mixing.

• The paper presents the ASG framework, which automatically estimates effective support and uses slice-driven Gibbs updates for robust sampling from complex probability kernels.
• It eliminates manual tuning by integrating a Cauchy transformation with adaptive integration, enabling efficient exploration of non-smooth, multimodal, and heavy-tailed distributions.
• Empirical results show that ASG achieves substantially higher effective sample sizes and rapid convergence with theoretical guarantees like uniform geometric ergodicity.

Efficient and Fast Sampling from Arbitrary Probability Kernels using Sliced Gibbs Sampler

Introduction and Motivation

This paper introduces the Automated Sliced Gibbs (ASG) framework for Markov chain Monte Carlo (MCMC) sampling from arbitrary, potentially unnormalized, non-smooth, heavy-tailed, or multimodal probability kernels. Classical MCMC approaches such as Metropolis-Hastings (MH), Hamiltonian Monte Carlo (HMC), and standard Gibbs samplers are typically limited by their need for manual proposal scale tuning, explicit support bounds, conditioning, or gradient information. Even state-of-the-art adaptive methods often require significant user intervention, are not robust to irregular or highly multimodal targets, or cannot handle non-differentiable densities.

ASG fundamentally departs from the need for such tuning and domain-specific knowledge by integrating three innovations:

1. Automatic effective support estimation through a Cauchy transformation and adaptive integration,
2. Slice-driven Gibbs updates that operate coordinate-wise without geometric insights or conditioning optimization,
3. Integrated diagnostics based on effective sample size (ESS) and invariant properties to empirically assure convergence and mixing efficiency.

The ASG Framework: Algorithmic Foundations

The theoretical foundation of ASG lies in augmenting the state space with slice variables, and then efficiently sampling from the induced conditional distributions using automatically detected effective support.

Given an unnormalized kernel $K(x)$, ASG introduces an auxiliary variable $u$ such that the joint kernel $H(u, x) = \mathbb{I}\{K(x) > u\}$ marginalizes back to the original target. Slice sampling proceeds by alternately updating $u$ and each coordinate $x_j$ in a Gibbs cycle. The key computational challenge—uniformly sampling within the coordinate slice set— is addressed by automated calculation of the effective support via a Cauchy reparameterization and adaptive quadrature on $(0,1)$.

The method eliminates manual bracket, step-size, or bound selection required by traditional slice samplers. Instead, the Cauchy transformation ensures that, even for heavy-tailed or sharply peaked kernels, the domain of practical sampling is automatically determined with a pre-specified mass tolerance $\varepsilon$. This support is then used within each Gibbs cycle to sample uniformly from the conditional slice, always preserving the correct target distribution.

The following plot visualizes the empirical efficacy by comparing the histogram of samples generated by ASG with the true density for a challenging multimodal case.

Figure 1: The ASG sampler (histogram) rapidly and comprehensively explores all modes of a multimodal target, whereas standard MCMC methods may fail to do so.

Theoretical Properties: Invariance and Ergodicity

The Markov chain generated by the ASG framework possesses the correct stationary distribution, as demonstrated by direct verification of detailed balance for the slice-augmented target. More specifically, the ASG chain with transition kernel $T$ and joint target $H$ satisfies the invariance equation

$$\int T((u,x) \to (\tilde{u},\tilde{x}))\, H(u, x)\, du\, dx = H(\tilde{u}, \tilde{x}),$$

where all conditional densities are exactly normalized due to exact slice sampling over the automatically detected support intervals.

Moreover, under standard mild regularity conditions—boundedness of $u$0 and uniform positivity on conditional slice sets—the ASG sampler is uniformly geometrically ergodic. The Doeblin minorization condition is satisfied, guaranteeing exponential decay of autocorrelations and ensuring that the effective sample size grows linearly with the number of iterations. This strong form of ergodicity is uncommon among general-purpose MCMC algorithms, especially for non-smooth or irregular targets.

Numerical Performance and Diagnostics

Empirical benchmarks are conducted across a spectrum of pathological targets, including multimodal beta mixtures, strongly curved Rosenbrock and Ackley functions, high-dimensional norms, and non-smooth kernels derived from LASSO and bridge regression objectives.

Multimodal and Non-Gaussian Kernels

The ASG sampler demonstrates superior exploration and mixing properties on challenging univariate and bivariate targets, quickly discovering and traversing disconnected or sharply curved high-density regions.

Figure 2: Early-stage ASG iterations show rapid transitions across modes, facilitated by effective support estimation.

Notably, the conditional effective support adapts automatically to the geometry and multimodality of each coordinate's slice. For example, in the Rosenbrock "banana" kernel, ASG samples accurately and efficiently explore narrow, strongly nonlinear ridges that are notorious traps for RW-MH and even HMC.

Figure 3: ASG samples (red dots) overlayed on Rosenbrock contours; trajectory covers all high-density regions without manual tuning.

Effective Sample Size and Decorrelation

Across all problem settings, ASG yields orders of magnitude higher effective sample size (ESS) for fixed wall-clock time compared to RW-MH, adaptive Gibbs, or gradient-based methods. ESS per second (ESS/s) remains robust as dimension or target irregularity increases; in many cases, negative higher-lag autocorrelations induced by ASG further increase statistical efficiency, surpassing the actual draw count ("super-efficient" sampling).

Figure 4: The autocorrelation function for the 10-dimensional norm kernel decays rapidly, with negative values at moderate lags, indicating near-independence.

Figure 5: ASG achieves significantly higher ESS/s than RW-MH, especially as sample size increases.

Non-Smooth and High-Dimensional Targets

For loss-derived targets such as LASSO and bridge regression, ASG captures full posterior uncertainty, not just point estimates, producing dense, smooth posterior density estimates for all coefficients. Mixing remains excellent as measured by high-dimensional ESS—even in settings with heavy regularization, sharp ridges, or $u$1 penalties.

Figure 6: Marginal posterior densities of LASSO coefficients, accurately captured by ASG despite heavy penalty-induced sparsity and non-smoothness.

Figure 7: Multimodal and non-convex coefficient posteriors for bridge regression, robustly estimated by ASG.

Implications and Discussion

Practical Impact

By rendering autotuning transparent and trivial, ASG enables fully black-box MCMC sampling for arbitrary probability kernels, regardless of differentiability, support, or modality. The computational cost of automated support estimation is offset by a dramatic increase in effective yield (ESS/s), with negligible effect on per-iteration runtime for moderate dimensions. Crucially, ASG is less sensitive to initial conditions or target pathologies and empirically robust to highly anisotropic or constrained supports.

Theoretical Consequences

Uniform geometric ergodicity positions ASG as a uniquely theoretically solid general-purpose sampler, with CLT guarantees for all functionals of interest and transparent error quantification. The slice-augmentation framework also enables simple, low-dimensional convergence diagnostics via the log-kernel time series, obviating the need for elaborate or ad hoc MCMC diagnostic protocols.

ASG generalizes slice, Gibbs, and even random-direction/hybrid samplers, and is amenable to further parallelization or block updating schemes for higher-dimensional settings. By decoupling ergodic correctness and practical mixing from gradient or Hessian calculations, ASG is highly suitable for irregular Bayesian/Bayesian-like machine learning posteriors.

Future Directions

Potential for extension includes block or partial updates for extremely high-dimensional posteriors, hierarchical coupling with block or Hamiltonian substructures, or hybridization with proposal-based methods for further acceleration. Integration with modern probabilistic programming languages will make the approach even more widely accessible for high-throughput Bayesian computation. Further theoretical work may investigate spectral gap properties or optimal scan strategies for categorical and discontinuous latent variable models.

Conclusion

The Automated Sliced Gibbs sampler provides a robust, automated, and highly efficient MCMC framework for arbitrary finite-dimensional probability kernels, setting a new standard for general-purpose MCMC methodology. It offers both practical and theoretical advances over conventional and adaptive alternatives without requiring manual tuning, geometrical insight, or differentiability—a critical development for posterior inference in modern statistical and machine learning applications.

Figures

Figure 1: The plot shows the empirical histogram (in light blue) based on samples from the ASG sampler with true density (red line) overlaid.

Figure 2: First 10 steps of the ASG algorithm based on the effective support over the beta kernel.

Figure 3: Contour Plot of Rosenbrock Kernel where the red dots are the samples generated by ASG.

Figure 8: Ackley Function plot after ASG showing exploration of multimodal non-convex regions.

Figure 4: Autocorrelation function (ACF) of the sampled radii $u$2 for the 10-dimensional kernel, demonstrating rapid decay and negative correlations that boost ESS.

Figure 5: Comparison of ASG and RW-MH in terms of ESS/s for fixed computational time.

Figure 6: Posterior Densities of LASSO Coefficient after sampling from ASG Sampler.

Figure 7: Posterior Densities of Bridge regression Coefficient after sampling from the proposed ASG Sampler.

• (2603.29096) (the current paper)
• For slice sampling: Neal, "Slice sampling" [math/0307077]
• For related recent developments: Williams et al., "Geodesic Slice Sampler for Multimodal Distributions with Strong Curvature" (Williams et al., 28 Feb 2025)