Quasi-Bayes empirical Bayes estimation of sums of random variables

Abstract: The estimation of sums of functions of observable and unobservable variables is a long-standing problem in statistics with applications across many domains. Empirical Bayes methods provide a natural framework for this task under mixture models, but existing approaches often rely on restrictive parametric assumptions or apply only to limited classes of functionals in nonparametric settings. We propose a nonparametric methodology, referred to as quasi-Bayes empirical Bayes, that addresses these limitations through a recursive estimation of the mixing distribution based on Newton's algorithm. The resulting plug-in estimate of the target sum is computationally efficient, scalable, and applicable to a broad class of utility functions, while enabling uncertainty quantification via asymptotic credible intervals derived from a Gaussian central limit theorem. We establish large sample asymptotic theoretical guarantees by proving a merging between the quasi-Bayes and Bayes estimates and by showing consistency under a correctly specified frequentist model. Synthetic-data and real-data analyses demonstrate the practical accuracy and stability of the method, with performance comparable to, and in some cases better than, existing empirical Bayes procedures.

• The paper introduces a quasi-Bayes empirical Bayes estimator that employs Newton's recursive algorithm to nonparametrically estimate sums of random variables.
• It demonstrates theoretical guarantees including asymptotic consistency, merging with traditional Bayesian posteriors, and accurate uncertainty quantification.
• Empirical evaluations reveal its computational scalability and robustness, outperforming classical estimators on both synthetic and real datasets.

Quasi-Bayes Empirical Bayes Estimation Of Sums Of Random Variables: A Technical Essay

Problem Formulation and Statistical Context

The estimation of sums of functionals of both observable and latent random variables is a core statistical problem spanning multiple domains, including clinical trials, species sampling, network analysis, and data confidentiality. The sum $S_n = \sum_{i=1}^n u(X_i,\theta_i)$ encapsulates diverse inferential targets, where $u(x,\theta)$ is an arbitrary utility function and $(X_i,\theta_i)$ are realizations from an unknown joint distribution with $X_i$ observed and $\theta_i$ either latent or constant. The prevailing empirical Bayes (EB) methodology is naturally suited for mixture models of the form $X_i|\theta_i \sim k(\cdot|\theta_i)$, $\theta_i \sim G$, but classical EB estimators are constrained by parametric assumptions or are limited in scope for generalized nonparametric target functionals.

Methodological Innovation: Quasi-Bayes Empirical Bayes and Newton's Recursive Algorithm

This paper departs from classical EB and proposes a nonparametric EB estimator, termed quasi-Bayes empirical Bayes (QB-EB). It hinges on recursive estimation of the mixing distribution $G$ via Newton's algorithm, a stochastic approximation procedure originally introduced by Smith and Makov (1978) and developed further by Newton et al. (1998). The recursion is specified by

$$G_{n+1}(d\theta) = (1 - a_{n+1}) G_n(d\theta) + a_{n+1} \frac{k(X_{n+1}|\theta) G_n(d\theta)}{\int k(X_{n+1}|\theta) G_n(d\theta)}$$

with a learning rate sequence $(a_n)$ satisfying $u(x,\theta)$0, $u(x,\theta)$1. The updated $u(x,\theta)$2 induces a plug-in QB-EB estimator for $u(x,\theta)$3: $u(x,\theta)$4.

This recursive learning process is interpreted as quasi-Bayesian in the sense of Fortini and Petrone, where inference proceeds not from a generative model but via sequential predictive learning. This approach endows computational scalability, constant per-observation cost, and flexibility in accommodating arbitrary utilities $u(x,\theta)$5. In contrast to classical f-modeling and g-modeling, the QB-EB paradigm operates nonparametrically, sidestepping model misspecification risks.

Theoretical Guarantees: Asymptotics and Frequentist Properties

The QB-EB estimator $u(x,\theta)$6 is shown to possess strong probabilistic guarantees. Under minimal identifiability and smoothness assumptions (kernel positivity, boundedness, continuity), the recursive estimate $u(x,\theta)$7 converges weakly to the true mixing distribution $u(x,\theta)$8 as $u(x,\theta)$9 (Martin and Tokdar, 2009). Specifically:

• Asymptotic Bayesian Merging: As $(X_i,\theta_i)$0, the quasi-Bayesian induced conditional distributions merge in Prohorov distance with the standard Bayesian posterior distributions. Thus, $(X_i,\theta_i)$1 approximates the Bayes estimator in large samples.
• Consistency and Rates: If $(X_i,\theta_i)$2 is the oracle mixing distribution, then $(X_i,\theta_i)$3 almost surely, and in the case of finite support for $(X_i,\theta_i)$4, explicit rates $(X_i,\theta_i)$5 for any $(X_i,\theta_i)$6 (where $(X_i,\theta_i)$7, $(X_i,\theta_i)$8) are established.
• Uncertainty Quantification: Central limit theorems yield asymptotic normality for $(X_i,\theta_i)$9, providing nonparametric, model-based credible intervals whose coverage converges to the nominal level.

These results are robust to the choice of utility, with sufficient conditions for boundedness and envelope integrability, and extend to multidimensional settings under coordinate-wise independence.

Empirical Evaluation: Synthetic and Real Data

Comprehensive numerical experiments validate the QB-EB estimator against parametric EB, Bayes EB, and, when available, nonparametric "u, v" estimators. In Poisson and Gaussian mixture settings, the QB-EB estimator is competitive or superior in mean absolute deviation, particularly under non-standard mixing distributions not captured by parametric forms. Computational efficiency is demonstrated, with per-observation CPU times scaling linearly with grid resolution but remaining low (e.g., 0.001s for $X_i$0). The QB-EB method exhibits robustness to grid discretization and model specification.

On real datasets—NHL goal statistics, European automobile insurance, and Twitter virality—the QB-EB estimator offers stable, accurate estimates even in out-of-sample prediction scenarios and across stratified subpopulations. Notably, the QB-EB estimator provides valid inference for aggregates where "u, v" methods are unavailable due to lack of explicit integral equation solutions.

Practical and Theoretical Implications

The QB-EB approach fundamentally broadens the scope of EB estimation:

• Nonparametric Flexibility: Accommodates arbitrary utility functions and complex latent structures without restrictive parametric specification.
• Computational Scalability: Constant per-observation update cost enables real-time inference and online learning on large datasets.
• Uncertainty Quantification: Asymptotic credible intervals are available without recourse to full Bayesian posterior computation (e.g., MCMC).
• Versatility: Adapts naturally to multidimensional settings and extends to species sampling and coverage estimation in biology and privacy risk in data science.

Future directions include development of convergence rate analysis in infinite-dimensional parameter spaces, extension to generalized mixture models (beyond exponential families), and investigation of frequentist coverage properties of QB-EB credible intervals. The methodology offers promising avenues in empirical Bayes estimation where compound decision or aggregate prediction is required.

Conclusion

The quasi-Bayes empirical Bayes estimator developed in this paper (2606.21707) constitutes a robust, scalable, and theoretically justified framework for the estimation of sums of random variables under general mixture models. The recursive Newton's algorithm enables nonparametric plug-in estimation and uncertainty quantification, addressing longstanding limitations of classical (parametric and nonparametric) EB procedures. Empirical evidence corroborates its accuracy and stability across synthetic and real datasets, with theoretical results supporting large-sample merging with Bayes and frequentist consistency. The approach has broad applicability across domains involving latent mixture structures and collective statistical measures, and opens several directions for rigorous statistical research and practical deployment.