Theory-Agnostic Hierarchical Bayesian Models

• Theory-Agnostic Hierarchical Bayesian Models are a statistical paradigm that decouples data-driven inference from domain-specific assumptions through multi-level latent structures and flexible priors.
• It integrates informed and weakly informative priors to incorporate auxiliary domain knowledge, enabling unified application across fields like sports analytics, AI evaluation, and physics.
• The framework employs versatile inference methods such as HMC, NUTS, and variational inference along with rigorous diagnostics to robustly quantify uncertainty and validate model adequacy.

A theory-agnostic hierarchical Bayesian framework is a statistical modeling paradigm that decouples data-driven inference from strong domain-specific modeling assumptions by explicitly structuring multiple levels of latent quantities and priors, with a formalism that remains agnostic to the internal specifics of any particular scientific theory, model family, or task context. This approach enables unified treatment of complex phenomena—such as quantifying uncertainty, partial pooling, and incorporating domain expertise—across disparate application domains ranging from the physical sciences to sports analytics, AI evaluation, and beyond. The defining property is that inferential machinery and hyperparameterization are specified in a modular, “black-box” fashion: only well-formed likelihoods and prior choices are needed for each model component, while the theory-agnostic Bayesian update enables robust and principled inference regardless of internal model content (Alamino, 2010, Mahmudlu et al., 28 Nov 2025, Luettgau et al., 8 May 2025, Guo et al., 3 Dec 2025, Shahmoradi, 2017, Wu et al., 2013, 2002.01129).

1. Foundational Principles and Mathematical Structure

Theory-agnostic hierarchical Bayesian models are characterized by the formal hierarchy of latent quantities, structural parameters, and priors/hyperpriors without explicit dependence on the internal workings of the underlying theory or domain. At minimum, the hierarchy consists of:

• Observed data $D$ (which may be indexed at multiple grouping levels, e.g., by player, event, or task).
• Structural or theory parameters (e.g., model coefficients, physical constants).
• Group- or unit-specific latent variables capturing heterogeneity (e.g., subject- or player-specific effects).
• Priors—possibly informed by auxiliary domain knowledge—placed over parameters and hyperparameters.

A domain-independent archetype is:

$$P(\theta,\,\phi\,|\,D) \propto P(D\,|\,\theta)\,P(\theta\,|\,\phi)\,P(\phi)$$

where $\theta$ denotes model-level or group-specific parameters, $\phi$ denotes hyperparameters, $P(D\,|\,\theta)$ is the likelihood, $P(\theta\,|\,\phi)$ is the hierarchical prior, and $P(\phi)$ is the hyperprior. No particular form for $P(D\,|\,\theta)$ or $P(\theta\,|\,\phi)$ is enforced, other than normalizability and tractability of inference (Alamino, 2010, Shahmoradi, 2017, Guo et al., 3 Dec 2025).

For theory selection or model comparison, the hierarchy may further extend:

$$P(\alpha,\,\pi\,|\,D) = \frac{P(D\,|\,\alpha,\,\pi)\,P(\pi\,|\,\alpha)\,P(\alpha)}{P(D)}$$

where $\alpha$ indexes model classes and $\pi$ are the associated free parameters (Alamino, 2010).

2. Informed and Weakly Informative Priors: Encoding Domain Knowledge

A theory-agnostic framework incorporates auxiliary domain information through priors or hyperpriors—without constraining the likelihood or requiring theory-specific generative models. Such priors may take the following forms:

• Empirical expert ratings: For example, in sports analytics, player-level covariate effects (e.g., shot distance, one-on-one finishing) are assigned informed prior means derived from expert databases (such as Football Manager attributes), then z-scored and mapped to feature-specific coefficients in the hierarchical model (Mahmudlu et al., 28 Nov 2025).
• Group-specific variance estimation via Empirical Bayes: In context-rich GLMs, meta-prior variances are empirically estimated by pooling by feature group, enabling learning rate decoupling and partial pooling without hand-crafted regularization (2002.01129).
• Noninformative or weakly informative priors: In evaluation settings involving novel agents (e.g., LLMs), weakly informative hyperpriors (e.g., $N(0,1)$, HalfNormal) are placed to favor data-dominant learning, allowing the evidence to determine the degree of regularization (Luettgau et al., 8 May 2025, Guo et al., 3 Dec 2025).
• Distributional constraints or truncations: Theory-induced domain validity can be enforced via smooth truncation functions in the prior or likelihood (as in soft-truncated priors for remnant spins in black-hole spectroscopy), which introduces no model-specific structure beyond the spectral mapping to observational parameters (Guo et al., 3 Dec 2025).

The crucial design principle is that the choice or mapping of priors is modifiable without affecting the inferential infrastructure, preserving the agnosticism to underlying mechanistic models.

3. Inference Procedures and Diagnostics

All such frameworks are Bayesian in the full sense, with inference conducted over the entire hierarchy. Computational posterior sampling is dominated by generic, model-agnostic methods:

• Hamiltonian Monte Carlo (HMC) and No-U-Turn Sampler (NUTS): For models of moderate-to-large dimension, NUTS is employed (e.g., via PyMC, Stan, NumPyro), with robust non-centered parameterizations ensuring convergence (Mahmudlu et al., 28 Nov 2025, Luettgau et al., 8 May 2025).
• Nested sampling: Employed for marginal likelihood estimation and model comparison in high-dimensional or non-conjugate problems (Guo et al., 3 Dec 2025).
• Variational Inference and Gibbs–Metropolis: Used for nonparametric and robust extensions such as the Hellinger distance model (Wu et al., 2013).
• Empirical Bayes estimators: For meta-prior variance learning, estimators are derived via variance decomposition, yielding unbiased and strongly consistent solutions for group-level uncertainty (2002.01129).

Standard diagnostics encompass:

• Gelman–Rubin $\hat{R}$ (target $<1.1$ or $=1.00$).
• Effective sample size (ESS) $>100$ or $n_\text{eff}\geq$ number of draws.
• Bayesian fraction of missing information (BFMI) $>0.2$.
• Trace plots, posterior predictive checks, and coverage metrics (Mahmudlu et al., 28 Nov 2025, Luettgau et al., 8 May 2025, Guo et al., 3 Dec 2025, Shahmoradi, 2017).

4. Counterfactual, Predictive, and Robust Estimation

Theory-agnostic hierarchical Bayes enables explicit estimation of counterfactual outcomes by exchanging unit-level or parameter-level effects, predictive performance intervals, and robust inference against model misspecification and outliers:

• Counterfactuals: In sports, posterior draws of global and unit-specific parameters enable construction of "what-if" expected goal (xG) estimates for one player acting in another's shot contexts, with full uncertainty quantification (Mahmudlu et al., 28 Nov 2025).
• Posterior predictive distributions: For new or future data $D'$, model-averaged predictions are given by

$$P(D' \mid D) = \sum_\alpha \int_\pi P(D'\mid\alpha, \pi) P(\alpha, \pi \mid D)\,d\pi$$

(Alamino, 2010), or the analogous forms in GLMs and physics hierarchies.

• Model inadequacy and noise: The multilevel Bayesian paradigm explicitly models measurement error and structural discrepancy, propagating uncertainty throughout the hierarchy and allowing for validation and sensitivity analysis (Shahmoradi, 2017).
• Robustness to contamination: Hierarchical Hellinger models automatically discount the effect of outliers by weighting prior/posterior density by exponential Hellinger distance, maintaining efficiency and robustness (Wu et al., 2013).

5. Generalization and Portability Across Domains

The definitionally theory-agnostic property allows immediate generalization:

• Only the definition of covariates or grouping structure, mapping of auxiliary prior means (if any), and the likelihood model are domain-specific; the hierarchical modeling, inference, and counterfactual or predictive estimation remain unaltered (Mahmudlu et al., 28 Nov 2025, Guo et al., 3 Dec 2025, Alamino, 2010, 2002.01129).
• In black-hole spectroscopy, the modular two-stage pipeline (DS parameter inference $\to$ spectral-matching hierarchy) is implementable for any QNM spectrum or physical scenario, with the same inference and diagnostics (Guo et al., 3 Dec 2025).
• For bandit, GLM, or AI benchmarking, only the structure of features, outcomes, and desired parameter grouping is toggled, preserving theory agnosticism (Luettgau et al., 8 May 2025, 2002.01129).

6. Empirical Performance, Validity, and Diagnostic Approaches

The frameworks have demonstrated:

• Reduction of posterior uncertainty and better calibration in player- or agent-level performance estimation versus conventional non-hierarchical or weak-prior models (Mahmudlu et al., 28 Nov 2025, Luettgau et al., 8 May 2025).
• External validity measured through $R^2$ correlations between hierarchical and baseline (non-hierarchical) models (Mahmudlu et al., 28 Nov 2025).
• Ability to recover latent generative parameters under theory mismatch or in presence of misspecification, as tested by injection/recovery studies and model comparison via Bayes factors or Kullback–Leibler divergence (Guo et al., 3 Dec 2025).
• Robustness of inference to prior choices and outlier contamination (Wu et al., 2013).
• Posterior predictive checks and formal model-comparison metrics (e.g., WAIC, LOO) for model adequacy (Luettgau et al., 8 May 2025, Shahmoradi, 2017).

These frameworks have also established best practices for inference workflow, including prior specification, model fitting, diagnostics, posterior summarization, and predictive validation.

7. Illustrative Applications and Extensible Recipes

Table: Core Applications of Theory-Agnostic Hierarchical Bayes

Domain	Hierarchical Levels	Key Functional Role
Black-hole spectroscopy (Guo et al., 3 Dec 2025)	DS parameters, (M, χ, ζ)	Spectral matching, theory inference, soft truncation
Football expected goals (Mahmudlu et al., 28 Nov 2025)	Player, shot, context	Specialization profiles, counterfactual xG
AI evaluation (HiBayES) (Luettgau et al., 8 May 2025)	Model, domain, subdomain	Agent/task uncertainty, model comparison
Physics theory selection (Alamino, 2010)	Data, theory/model, constants	Model evidence, posterior over theory classes
Bayesian meta-learning (Kim et al., 2018)	Task, meta-prior	Fast adaptation, Chaser loss, model-agnostication
Robust regression (Wu et al., 2013)	Parametric/nonparametric	Robustness to outliers, efficiency

The core recipe for instantiating a theory-agnostic hierarchical Bayesian inference system is:

1. Specify data and desired grouping structure.
2. Define likelihoods for observed units or groups.
3. Assign priors/hyperpriors—optionally informed by external sources or shiftable via empirical Bayes.
4. Fit via generic Bayesian inference engine (MCMC, VI, nested sampling).
5. Diagnose and validate model fit and predictive coverage.
6. For domain transfer, only redefine the observable structure and auxiliary prior mapping—the hierarchy and inference remain constant (Mahmudlu et al., 28 Nov 2025, Guo et al., 3 Dec 2025, Alamino, 2010, 2002.01129, Luettgau et al., 8 May 2025).

These properties collectively establish the theory-agnostic hierarchical Bayesian framework as a universally extensible approach to scientific and data-driven inference, unifying principled statistical rigor with modular adaptivity.