A Separation in Heavy-Tailed Sampling: Gaussian vs. Stable Oracles for Proximal Samplers (2405.16736v1)

Abstract: We study the complexity of heavy-tailed sampling and present a separation result in terms of obtaining high-accuracy versus low-accuracy guarantees i.e., samplers that require only $O(\log(1/\varepsilon))$ versus $\Omega(\text{poly}(1/\varepsilon))$ iterations to output a sample which is $\varepsilon$-close to the target in $\chi^2$-divergence. Our results are presented for proximal samplers that are based on Gaussian versus stable oracles. We show that proximal samplers based on the Gaussian oracle have a fundamental barrier in that they necessarily achieve only low-accuracy guarantees when sampling from a class of heavy-tailed targets. In contrast, proximal samplers based on the stable oracle exhibit high-accuracy guarantees, thereby overcoming the aforementioned limitation. We also prove lower bounds for samplers under the stable oracle and show that our upper bounds cannot be fundamentally improved.

• The paper establishes that Gaussian proximal samplers incur polynomial iteration complexity, limiting high-accuracy sampling in heavy-tailed settings.
• It introduces stable oracles that employ fractional heat flows to achieve logarithmic iteration complexity under fractional Poincaré conditions.
• The study provides rigorous theoretical bounds and practical algorithms, setting a new direction for high-accuracy sampling in statistical computing.

A Separation in Heavy-Tailed Sampling: Gaussian vs. Stable Oracles for Proximal Samplers

The paper "A Separation in Heavy-Tailed Sampling: Gaussian vs. Stable Oracles for Proximal Samplers" by Ye He, Alireza Mousavi-Hosseini, Krishnakumar Balasubramanian, and Murat A. Erdogdu explores the complexity distinctions between Gaussian and stable oracles within proximal samplers when applied to heavy-tailed distributions. The paper proposes that while Gaussian-based samplers face fundamental limitations, stable-based samplers can achieve higher accuracy under certain conditions.

Introduction and Motivation

Sampling from heavy-tailed distributions is a significant challenge across various domains such as Bayesian statistics, machine learning, and robust statistics. This complexity arises because gradient-based MCMC methods such as Langevin Monte Carlo (LMC) often perform poorly due to the slow decay of gradients in heavy-tailed densities. The paper emphasizes that there's a lack of theoretical results proving the efficacy of high-accuracy samplers for such distributions.

Research Questions

The investigation is centered around two primary questions:

1. Q1: What are the fundamental limits of Gaussian-based proximal samplers for heavy-tailed distributions?
2. Q2: Can we design high-accuracy proximal samplers using stable oracles for heavy-tailed distributions?

Key Contributions

Lower Bounds for Gaussian Oracle

The paper establishes that proximal samplers using Gaussian oracles exhibit a fundamental barrier when applied to heavy-tailed distributions. Specifically:

• Langevin Diffusion Analysis: The authors show that Langevin diffusion (LD) suffers from poor scaling, with total variation distances converging at a rate that is polynomial in $1/\varepsilon$, where $\varepsilon$ represents the accuracy of the sample.
• Gaussian Proximal Sampler: Extending the results to discrete-time proximal samplers, the paper demonstrates similar limitations. For the generalized Cauchy densities, the Gaussian proximal sampler requires $\Omega(d^{3/2}\varepsilon^{-2/\nu})$ iterations, establishing that Gaussian-based methods are fundamentally limited to low-accuracy guarantees.

High-Accuracy Samplers via Stable Oracles

The paper introduces proximal samplers based on stable oracles, leveraging fractional heat flows and stable-driven stochastic processes. This construction overcomes the limitations identified for Gaussian oracles:

• Stable Proximal Sampler: Using stable oracles, these samplers achieve $\mathcal{O}(\log(1/\varepsilon))$ complexity for heavy-tailed distributions satisfying a fractional Poincaré inequality (FPI).
• Fractional Poincaré Inequality: The FPI serves as a weaker condition than the usual Poincaré inequality, accommodating a broader class of heavy-tailed distributions. The authors show that these stable-based samplers provide high-accuracy guarantees when the target density satisfies the FPI.

Practical Implementation and Bounds

An important aspect of the research is the practical implementation of the Restricted $\alpha$-Stable Oracle (R$\alpha$SO):

• Rejection Sampling Method: For the case $\alpha = 1$, the paper provides a sampling algorithm using rejection sampling, which relies on the fractional heat flows of stable processes to maintain the accuracy guarantees established theoretically.
• Complexity Analysis: The paper presents a detailed analysis, proving that with suitable assumptions, the stable proximal sampler can achieve significant performance improvements over Gaussian-based methods even under practical constraints.

Numerical and Theoretical Implications

Numerical Results

For generalized Cauchy densities, the paper shows:

• For Gaussian Proximal Sampler: The lower bound on the number of iterations indicates a polynomial dependency on $1/\varepsilon$.
• For Stable Proximal Sampler: When applied with $\alpha \le 1$, high accuracy is maintained with $\mathcal{O}(\log(1/\varepsilon))$ complexity, a stark contrast to Gaussian-based results.

Theoretical Contributions

The separation between the Gaussian and stable proximal samplers established by the authors is significant. It conclusively shows that stable oracles can be designed to overcome the limitations faced by Gaussian oracles in heavy-tailed settings. This suggests that adopting stable-driven methods could be a fruitful direction for future algorithmic developments in sampling theory.

Future Directions

The results outlined in the paper open several pathways for future research:

• Broader Applicability: Extending the stable proximal samplers to other classes of non-log-concave distributions.
• General $\alpha$ Implementation: Exploring efficient implementations of R$\alpha$SO for varying values of $\alpha$ beyond 1.
• Complexity Bounds Tightening: Further refining the bounds to better understand the separation between different oracle-driven samplers.

Conclusion

The paper provides a comprehensive analysis of the limitations of Gaussian-based proximal samplers and highlights the advantages of stable oracles for high-accuracy sampling from heavy-tailed distributions. By offering both theoretical insights and practical algorithms, it lays a robust foundation for future developments in this critical area of statistical computing and machine learning.