Posterior Contraction for Sparse Neural Networks in Besov Spaces with Intrinsic Dimensionality (2506.19144v1)

Abstract: This work establishes that sparse Bayesian neural networks achieve optimal posterior contraction rates over anisotropic Besov spaces and their hierarchical compositions. These structures reflect the intrinsic dimensionality of the underlying function, thereby mitigating the curse of dimensionality. Our analysis shows that Bayesian neural networks equipped with either sparse or continuous shrinkage priors attain the optimal rates which are dependent on the intrinsic dimension of the true structures. Moreover, we show that these priors enable rate adaptation, allowing the posterior to contract at the optimal rate even when the smoothness level of the true function is unknown. The proposed framework accommodates a broad class of functions, including additive and multiplicative Besov functions as special cases. These results advance the theoretical foundations of Bayesian neural networks and provide rigorous justification for their practical effectiveness in high-dimensional, structured estimation problems.

• The paper establishes that Bayesian neural networks adaptively achieve optimal posterior contraction rates over anisotropic Besov spaces.
• It demonstrates that sparse architectures using both spike-and-slab and continuous shrinkage priors can mitigate the curse of dimensionality.
• The analysis provides practical guidelines for selecting network architectures to efficiently model complex hierarchical functions.

Posterior Contraction for Sparse Neural Networks in Besov Spaces

This paper establishes theoretical guarantees for Bayesian neural networks (BNNs) with sparse architectures, focusing on their performance in high-dimensional function estimation problems. The principal findings are that BNNs exhibit optimal posterior contraction rates when applied to functions in anisotropic Besov spaces and their hierarchical compositions. These spaces are significant in capturing intrinsic dimensional structures, allowing BNNs to circumvent the traditional curse of dimensionality.

Key Findings

1. Optimal Posterior Contraction Rates: The paper rigorously quantifies how BNNs achieve near-minimax optimal posterior contraction rates over anisotropic Besov spaces. Specifically, the paper demonstrates that BNNs can adapt their estimation procedures based on the intrinsic dimension of the target function, which often leads to improved rates compared with treating the problem in a fully high-dimensional context.
2. Bayesian Advantages: A notable contribution is the framework's capability to perform rate adaptation. Unlike many frequentist methods requiring careful tuning based on prior knowledge of the function's smoothness, the BNNs discussed here achieve optimal rates without predefined smoothness levels due to the inherent adaptability of Bayesian inference.
3. Spike-and-Slab vs. Shrinkage Priors: The analysis includes both traditional spike-and-slab priors and more computationally efficient continuous shrinkage priors. It is shown that both types of priors support optimal rate contraction given certain conditions, with continuous priors offering a pragmatic edge due to reduced computational demands.
4. Composite Function Spaces: Extending beyond simple Besov spaces, the paper considers composite spaces indicative of hierarchical compositions, providing a realistic model for complex functional relationships captured by deep neural network architectures.
5. Empirical Validation and Adaptive Networks: The paper underscores the importance of selecting appropriate network architectures (depth, width, sparsity) for estimation, enhancing practical applicability without losing theoretical rigor. This approach has been validated through various theoretical experiments.

Implications and Future Directions

The theoretical advances outlined in the paper have notable implications for both theoretical and practical applications of neural networks:

• Theoretical Understanding: By providing a robust framework for the convergence rates of BNNs in complex functional settings, the paper enriches the existing understanding of Bayesian nonparametrics. It highlights the potential for BNNs to match or exceed the performance of frequentist counterparts in high-dimensional scenarios.
• Practical Applications: The findings on posterior contraction provide practical guidelines for deploying BNNs in environments where functions have intrinsic lower-dimensional structures, such as image or language processing where spatial or sequential dependencies can reduce effective dimensionality.
• Scalable Bayesian Inference: The insights into efficient prior selection, especially with continuous shrinkage priors, pave the way for scalable Bayesian inference in neural networks, potentially leading to more widespread adoption in large-scale systems where computational resources are a constraint.

Looking forward, expanding these results to architectures beyond simple feedforward networks and incorporating contemporary architectures like transformers or convolutional layers remains pivotal. Additionally, developing more computationally efficient sampling methods and variational approximations to handle the inferential complexity of BNNs could further bridge theory with practice.