Minimaxity and Admissibility of Bayesian Neural Networks

Abstract: Bayesian neural networks (BNNs) offer a natural probabilistic formulation for inference in deep learning models. Despite their popularity, their optimality has received limited attention through the lens of statistical decision theory. In this paper, we study decision rules induced by deep, fully connected feedforward ReLU BNNs in the normal location model under quadratic loss. We show that, for fixed prior scales, the induced Bayes decision rule is not minimax. We then propose a hyperprior on the effective output variance of the BNN prior that yields a superharmonic square-root marginal density, establishing that the resulting decision rule is simultaneously admissible and minimax. We further extend these results from the quadratic loss setting to the predictive density estimation problem with Kullback--Leibler loss. Finally, we validate our theoretical findings numerically through simulation.

• The paper demonstrates that fixed-scale BNN priors yield non-minimax Bayes rules due to insufficient heavy-tailed behavior in the normal location model.
• It introduces a BetaPrime hyperprior on the effective output variance to transform the prior into a heavy-tailed Gaussian scale mixture, ensuring minimaxity and admissibility.
• The study extends these results to predictive density estimation under KL loss and supports the findings with extensive simulations across varying dimensions and sparsity regimes.

Minimaxity and Admissibility in Bayesian Neural Network Priors

Introduction

This paper provides a decision-theoretic analysis of Bayesian Neural Networks (BNNs), focusing on minimaxity and admissibility of the induced Bayes rules under quadratic and Kullback–Leibler (KL) loss functions. The core technical context is the normal location model, with parameter $\boldsymbol{\theta} \in \mathbb{R}^p$ and observation $\boldsymbol{Y} \sim N_p(\boldsymbol{\theta}, I_p)$. The main contributions are twofold: (1) a demonstration that Bayes rules induced by deep, fixed-scale ReLU BNN priors are not minimax, and (2) the construction of a scale hyperprior (specifically, a BetaPrime prior on the effective output variance) under which the induced Bayes rule becomes both minimax and admissible. The results are extended to the predictive density estimation context, and supported by extensive simulation.

Non-Minimaxity of Fixed-Scale BNNs in the Normal Location Model

The analysis shows that, despite the universal function approximation capacity and hierarchical structure of BNNs, the standard fixed-scale Gaussian weight priors yield prior predictive distributions for $\boldsymbol{\theta}$ that are too light-tailed to produce minimax estimators under quadratic loss. Specifically, the marginal (prior predictive) for the BNN output has, asymptotically, stretched-exponential tails of the form $\exp(-c r^{2/d})$, where $r = \|\boldsymbol{y}\|$, rather than the heavier tails required for minimaxity. This is formalized through an analysis of the superharmonicity of the square root marginal, which (following classical results) is necessary for minimaxity in this estimation setting.

The central finding is that the square root of the BNN prior predictive is not superharmonic in dimension $p \geq 3$, and thus, the induced Bayes rule is not minimax. More precisely, as signal strength $\|\boldsymbol{\theta}\|$ increases, the risk of the fixed-scale BNN estimator exceeds the minimax bound, formalized via Stein's unbiased risk estimate (SURE) and a detailed study of the Bayes estimator's shrinkage profile. The asymptotic regime demonstrates that the BNN's shrinkage does not diminish quickly enough for large signals due to insufficient prior predictive mass in the tails. Theoretical results and simulations show this phenomenon clearly for moderate to large $p$.

Figure 1: Estimated risk for several decision rules in dimension $p=5$ as a function of $\|\boldsymbol{\theta}\|$.

Figure 2: Estimated risk for several decision rules in dimension $\boldsymbol{Y} \sim N_p(\boldsymbol{\theta}, I_p)$0 as a function of $\boldsymbol{Y} \sim N_p(\boldsymbol{\theta}, I_p)$1.

Figure 3: Estimated risk for several decision rules in dimension $\boldsymbol{Y} \sim N_p(\boldsymbol{\theta}, I_p)$2 as a function of $\boldsymbol{Y} \sim N_p(\boldsymbol{\theta}, I_p)$3.

These empirical plots confirm that, for moderate and high dimension, the risk of BNN estimators under fixed-scale priors exceeds $\boldsymbol{Y} \sim N_p(\boldsymbol{\theta}, I_p)$4 at high signal strength, confirming non-minimaxity.

Construction of a Minimax and Admissible BNN prior via Scale Hyperpriors

The authors address the outlined deficiency by introducing a scale hyperprior on the effective output variance of the BNN prior, specifically a $\boldsymbol{Y} \sim N_p(\boldsymbol{\theta}, I_p)$5 prior. This construction induces a Gaussian scale mixture prior with heavy tails. The choice of BetaPrime is motivated by classical minimaxity theory (e.g., the Strawderman prior for the normal means problem), as it ensures the superharmonicity of the square root marginal predictive and suffices to yield a minimax Bayes rule for all $\boldsymbol{Y} \sim N_p(\boldsymbol{\theta}, I_p)$6.

The induced Bayes rule under this hyperprior is shown to be both minimax and admissible. The explicit shrinkage rule has the form

$\boldsymbol{Y} \sim N_p(\boldsymbol{\theta}, I_p)$7

where the posterior mean of $\boldsymbol{Y} \sim N_p(\boldsymbol{\theta}, I_p)$8 controls the adaptive shrinkage. The minimax property is demonstrated via a precise calculation involving the Laplacian of the square root marginal predictive, which reduces to analyzing incomplete gamma integrals. Admissibility follows since the induced prior is proper and heavy enough in the tails, ensuring the procedure cannot be uniformly improved.

Predictive Density Minimaxity and Admissibility

The minimaxity and admissibility results are extended to predictive density estimation under KL loss (see, e.g., George et al. (2006)), an important setting in both Bayesian inference and modern meta-learning. The analysis exploits the invariance of the superharmonic condition under variance rescaling, showing that the same BetaPrime hyperprior ensures minimax and admissible Bayes predictive densities simultaneously in point estimation and predictive risk.

Numerical Results and Comparative Simulations

A detailed simulation study explores risk behavior across varying $\boldsymbol{Y} \sim N_p(\boldsymbol{\theta}, I_p)$9 and $\boldsymbol{\theta}$0, comparing:

• MLE (trivial minimax rule with constant risk $\boldsymbol{\theta}$1)
• BNN with fixed-scale prior
• BNN with minimax BetaPrime hyperprior
• Fixed-scale BNN augmented with dropout
• Horseshoe prior Bayes rule (considering sparsity dependence)

  Figure 4: Estimated risk for several decision rules in dimension $\boldsymbol{\theta}$2 as a function of $\boldsymbol{\theta}$3 under different sparsity regimes.

  

  Figure 5: Estimated risk for $\boldsymbol{\theta}$4 and varying sparsity.

  

  Figure 6: Estimated risk for $\boldsymbol{\theta}$5 and varying sparsity.

The simulations illustrate that the shrinkage rule induced by the BetaPrime hyperprior tracks the minimax benchmark ($\boldsymbol{\theta}$6) almost perfectly for all $\boldsymbol{\theta}$7. In contrast, the fixed-scale and dropout-augmented BNN rules exhibit clear risk inflation for large signal strength. The Horseshoe prior achieves lower risk in highly sparse regimes but substantially exceeds the minimax risk in the dense regime, demonstrating the uniform risk control advantage of the radial minimax BNN rule for settings where sparsity cannot be reliably assumed.

Theoretical and Practical Implications

The findings have significant consequences for both empirical Bayes methodology and the design of modern meta-learning systems. The analysis demonstrates that prior specification is the fundamental determinant of the statistical optimality of Bayesian neural decision rules, not only in classical estimation tasks but also for task-adaptive meta-learned predictors such as Prior-Data Fitted Networks (PFNs), where the choice of prior is effectively encoded in the training distribution.

From a theoretical perspective, the work tightens the connections between Bayesian deep learning, classical decision theory, and the structure of Gaussian scale mixture priors, and motivates further exploration into alternative heavy-tailed priors (e.g., involving Fox-H or Meijer-G hyperpriors). Practically, it suggests that optimal uncertainty quantification and robust inference with BNNs require going beyond default Gaussian priors in functional-space, and it provides a precise hyperprior construction ensuring optimal worst-case risk.

Conclusion

This work establishes that deep, fixed-scale ReLU BNNs do not satisfy statistical optimality criteria (minimaxity, admissibility) in the normal location model due to insufficient prior predictive tail mass. This deficiency is not intrinsic to BNNs but rather to the choice of scale hyperprior. By equipping the BNN with a BetaPrime hyperprior on effective output variance, minimaxity and admissibility are restored for both point estimation and predictive density estimation. This result is directly relevant for scientific, medical, and financial applications requiring decision-theoretically robust Bayesian uncertainty quantification, as well as for the meta-learning paradigm where the task prior governs generalization guarantees. These findings motivate renewed attention to principled prior design and hyperprior specification in Bayesian neural architectures and suggest robust future directions through the systematic exploration of heavier-tailed and more flexible scale mixture priors.

---

References

"Minimaxity and Admissibility of Bayesian Neural Networks" (2604.04673)