---
title: Hyvärinen Score
url: https://www.emergentmind.com/topics/hyvarinen-score
type: topic
---

# Hyvärinen Score

The Hyvärinen score is a strictly proper, local, homogeneous scoring rule for continuous probability densities, designed to enable parameter inference and model selection in contexts where the normalization constant of the model is intractable or ill-defined. It underpins the "score matching" estimation principle, provides a consistent foundation for bandwidth and hyperparameter selection, and allows for robust model comparison in both parametric and nonparametric settings, including unnormalized and pseudo-likelihood models.

## 1. Definition and Fundamental Properties

Let $X \in \mathbb{R}^d$ be a random variable with twice-differentiable density $p(x)$. The Hyvärinen score, $S_H(p, x)$, is given by

\[
S_H(p, x) = \Delta_x \log p(x) + \frac{1}{2} \|\nabla_x \log p(x)\|^2,
\]

where $\nabla_x$ denotes the gradient and $\Delta_x = \sum_{k=1}^d \partial^2/\partial x_k^2$ is the Laplacian. The score depends only on local properties of the (log-)density at $x$ up to second derivatives. 

Key structural properties include:
- **Strict Properness**: Expected score is uniquely minimized when $p$ equals the data-generating density $p_\star$.
- **2-Locality**: Only the derivatives of $\log p$ at the data point $x$ are required.
- **Homogeneity**: Multiplying $p$ by any positive constant leaves $S_H(p, x)$ invariant.

Properness follows from integration by parts; for any $p, p_\star$ sufficiently smooth and decaying at infinity,

\[
E_{p_\star}[S_H(p, X)] - E_{p_\star}[S_H(p_\star, X)] 
= \frac{1}{2} \int \|\nabla \log p(x) - \nabla \log p_\star(x)\|^2 p_\star(x) dx \ge 0,
\]

with equality iff $p \propto p_\star$ [1409.3690, 1711.00136].

## 2. Score Matching and Estimation Procedures

The Hyvärinen score forms the basis of the "score matching" estimator for parametric models $p(x; \theta)$. The empirical Hyvärinen score is minimized in place of log-likelihood:

\[
J_n(\theta) = \frac{1}{n}\sum_{i=1}^n \left[ \frac{1}{2} \| \nabla_x \log p(x^{(i)}; \theta) \|^2 + \Delta_x \log p(x^{(i)}; \theta) \right].
\]

This estimator requires only derivatives of the log-density and never involves the normalizing constant. For exponential family densities, $p(x; \theta) = \exp\left(\theta^\top T(x) - A(\theta) + b(x)\right)$, the score matching objective is quadratic in $\theta$ and often yields linear, closed-form estimating equations [2501.05105]. 

Robust score matching is achieved by partitioning the data into blocks, computing blockwise estimates, and aggregating via a geometric median-of-means, yielding estimators resilient to contamination and heavy tails while still relying exclusively on derivatives of $\log p$ [2501.05105].

## 3. Applications in Model Selection and Bandwidth Tuning

The Hyvärinen score is widely employed for model comparison, density estimation, and hyperparameter selection in settings where likelihood-based procedures are infeasible. For model selection, the cumulative prequential Hyvärinen score for a sequence of predictions is

\[
H_T(M) = \sum_{t=1}^T S_H \left( p_M(\cdot | y_{1:t-1}), y_t \right).
\]

Unlike the log-score, $S_H$ is invariant to normalizing constants and does not suffer from issues like Bartlett’s paradox or ill-defined Bayes factors with vague priors. Asymptotically, under regularity, for non-nested parametric models $M_1$ and $M_2$,

\[
\frac{1}{T} [H_T(M_2) - H_T(M_1)] \rightarrow D_H(p_\star, M_2) - D_H(p_\star, M_1),
\]
where $D_H$ is a Fisher-information–type divergence. The Hyvärinen score thus selects, in the limit, the model minimizing this divergence to the true process, a criterion distinct from Kullback–Leibler optimality [1711.00136].

For bandwidth selection, as in BART-based causal inference ("Direct Bayesian Additive Regression Trees for Conditional Average Treatment Effects in Regression Discontinuity Designs" [2603.03819]), the empirical Hyvärinen criterion over a grid of bandwidths $h$ is computed:

\[
\widehat H(h) = \sum_{i\in N_s} \left\{ 
2 \left[ \frac{1}{S} \sum_s \left( \ell_{i,h}^{(2),(s)} + (\ell_{i,h}^{(1),(s)})^2 \right) \right]
- \left[ \frac{1}{S} \sum_s \ell_{i,h}^{(1),(s)} \right]^2 
\right\},
\]
where $\ell^{(1)}$ and $\ell^{(2)}$ are first and second derivatives with respect to model predictions. The optimal $h$ is chosen by minimizing $\widehat H(h)$, sidestepping normalization issues and directly targeting predictive accuracy [2603.03819].

## 4. Time Series, Graphical Models, and Kernel Estimation

In time series, the Hyvärinen estimator operates on sequences or their sufficient statistics, producing estimators for AR, MA, and long-memory ARFIMA models that avoid the need to compute the likelihood normalization or marginalization over latent states [1904.12513, 1409.3690]. The efficiency of Hyvärinen-based estimators varies with the process: they are highly competitive in MA and ARFIMA, but less so in highly persistent AR models, where pairwise likelihood often dominates [1409.3690, 1904.12513].

For undirected graphical models and unnormalized exponential families, score matching based on the Hyvärinen score offers closed-form estimators and robustification via the geometric median-of-means [2501.05105].

In nonparametric kernel density estimation, the Hyvärinen score enables fully data-driven tuning of both bandwidth ($h$) and exponentiation parameter ($w$) in exponentiated KDEs. The Hyvärinen-based objective bypasses the intractable normalization constant inherent to exponentiated forms, yielding consistent and optimally convergent estimators for multi-modal densities and densities with outliers [2212.00984].

## 5. Computation for Intractable and State-space Models

For models with intractable marginal likelihoods or complex latent-variable structures, the Hyvärinen score admits efficient estimation by Monte Carlo. In the prequential framework for Bayesian model comparison, the required derivatives can be estimated from Sequential Monte Carlo (SMC) or SMC^2 schemes, even in non-linear and non-Gaussian state-space models. For discrete outputs, finite difference analogues of the score preserve strict propriety and homogeneity [1711.00136].

This makes the Hyvärinen score suitable for high-dimensional or otherwise complex models, including stochastic volatility driven by Lévy processes and SDE-based population models, where its robustness to prior vagueness and invariance to normalization are essential [1711.00136].

## 6. Theoretical Guarantees and Practical Guidance

Consistency, efficiency, and robustness results for Hyvärinen-score–based estimators are well-established:
- **Consistency:** Minimum-score estimators are consistent and asymptotically normal, with sandwich (Godambe) variance [1904.12513, 1409.3690].
- **Asymptotic Model Selection:** In both i.i.d. and state-space regimes, the prequential Hyvärinen score selects the asymptotically Fisher-information–optimal model [1711.00136].
- **Bandwidth and Hyperparameter Rates:** In kernel-based methods, tuning via the Hyvärinen score achieves established minimax rates for density estimation [2212.00984].
- **Robustness:** Median-of-means aggregation ensures stability in the presence of outliers or heavy-tailed noise [2501.05105].

Empirically, the Hyvärinen score exhibits superior performance in multi-modal, contaminated, or non-likelihood-amenable contexts, though higher Monte Carlo variance relative to likelihood-based criteria may be observed in finite-sample or highly non-Gaussian scenarios [2212.00984, 1711.00136].

## 7. Summary Table: Core Formulae and Properties

| Context                    | Hyvärinen/Score Matching Formula                                      | Key Features                                    |
|----------------------------|-----------------------------------------------------------------------|-------------------------------------------------|
| General (continuous)       | $S_H(p, x) = \Delta \log p(x) + \frac{1}{2}\|\nabla\log p(x)\|^2$    | Proper, local, homogeneous                      |
| Exponential family         | $J_n(\theta) = \frac{1}{n} \sum_i [\frac{1}{2}\|\nabla \log p(x^{(i)}; \theta)\|^2 + \Delta \log p(x^{(i)}; \theta)]$ | No normalizing constant needed                  |
| Time series, Gaussian      | $H(\theta; Y) = -\text{tr}(\Sigma^{-1}) + \frac{1}{2} (Y-\mu)^T \Sigma^{-2} (Y-\mu)$               | Avoids high-dimensional determinants            |
| Pseudo/BART bandwidth      | $\widehat H(h) = \sum_{i\in N_s} \{ 2\langle \ell^{(2)} + (\ell^{(1)})^2 \rangle - (\langle \ell^{(1)} \rangle )^2\}$ | Posterior MC, no normalization                  |
| Exponentiated KDE          | $\mathcal H(w,h) = \frac{1}{n} \sum_i 2w\partial_x^2\log\hat f_h(x_i) + (w^2-2w)(\partial_x\log\hat f_h(x_i))^2$ | Joint tuning of $w$, $h$; IHS-LOO consistency   |

Minimization is always with respect to the arguments of interest, be they parameters, hyperparameters, or kernel bandwidths.

---

*References:*  
[2603.03819], [2501.05105], [1409.3690], [1904.12513], [1711.00136], [2212.00984]

Source: https://www.emergentmind.com/topics/hyvarinen-score