Sliced Score Matching: Scalable Density Estimation

• Sliced Score Matching (SSM) is a method for density and score estimation that projects the score function onto random directions, avoiding full Hessian computations.
• It leverages the Hutchinson estimator to provide unbiased trace estimates, making it computationally efficient for high-dimensional and deep models.
• Generalizations like GSSM introduce nonlinear projections to further reduce bias, though at the cost of increased variance and sample complexity.

Sliced score matching (SSM) is a scalable method for density and score estimation in unnormalized statistical models. It generalizes Hyvärinen's score matching by projecting the score function onto random directions, avoiding the need to compute a full Hessian trace and enabling efficient estimation in high-dimensional and deep models. SSM is widely applicable across probabilistic modeling, implicit generative models, and high-dimensional stochastic differential equations.

1. Mathematical Formulation of Sliced Score Matching

Let $p_d(x)$ be the data distribution over $\mathbb{R}^d$, and $p(x; \theta)$ an unnormalized model with score $s_\theta(x) = \nabla_x \log p(x; \theta)$. The original score matching loss of Hyvärinen can be written (up to an additive constant) as: $$J_{\mathrm{SM}}(\theta) = \mathbb{E}_{x \sim p_d}\left[ \mathrm{Tr}(\nabla_x s_\theta(x)) + \frac{1}{2}\|s_\theta(x)\|^2 \right].$$ Direct computation of the trace $\mathrm{Tr}(\nabla_x s_\theta(x))$ is computationally expensive in high dimensions.

SSM replaces the trace with an expectation over random projections using a vector $v \sim p_v$ with $\mathbb{E}[v v^\top] = I$: $$J_{\mathrm{SSM}}(\theta) = \mathbb{E}_{x \sim p_d} \, \mathbb{E}_{v \sim p_v} \left[ v^\top \nabla_x s_\theta(x) v + \frac{1}{2} (v^\top s_\theta(x))^2 \right] + \text{const},$$ with $$\mathbb{E}_v [v^\top \nabla_x s_\theta(x) v] = \mathrm{Tr}(\nabla_x s_\theta(x))$$ by the Hutchinson estimator. The empirical estimator uses i.i.d. data $\mathbb{R}^d$0 and projections $\mathbb{R}^d$1: $\mathbb{R}^d$2 A variance-reduced version (SSM-VR) substitutes the quadratic term by its expectation $\mathbb{R}^d$3 for appropriate $\mathbb{R}^d$4 (Song et al., 2019).

2. Theoretical Guarantees and Statistical Properties

Under standard regularity assumptions (positivity of $\mathbb{R}^d$5, smoothness of $\mathbb{R}^d$6, compact parameter set, etc.), SSM has the following properties (Song et al., 2019):

• Consistency: The minimizer of $\mathbb{R}^d$7 converges in probability to the population minimizer as $\mathbb{R}^d$8 for fixed number of projections $\mathbb{R}^d$9.
• Asymptotic Normality: For sufficiently smooth models,

$p(x; \theta)$0

where $p(x; \theta)$1 is the variance of the gradient of the SSM loss.

• As $p(x; \theta)$2, variance matches exact score matching.

These results situate SSM within classical empirical risk minimization, ensuring reliability for large-scale learning tasks.

3. Computational Implementation and Projection Choices

SSM is amenable to efficient algorithmic implementation, primarily relying on Hessian-vector products that can be evaluated by reverse-mode automatic differentiation. In frameworks like PyTorch or TensorFlow, one computes:

• $p(x; \theta)$3
• $p(x; \theta)$4
• Then $p(x; \theta)$5 This requires two backward passes per projection, and the complexity is $p(x; \theta)$6 reverse-mode calls, independent of the ambient dimension $p(x; \theta)$7 as long as $p(x; \theta)$8.

Common projection distributions include:

• Isotropic Gaussian ($p(x; \theta)$9): straightforward to sample, higher variance due to $s_\theta(x) = \nabla_x \log p(x; \theta)$0
• Uniform on sphere ($s_\theta(x) = \nabla_x \log p(x; \theta)$1): reduced fourth moments, lowers estimator variance at a slight computational overhead

Any distribution with $s_\theta(x) = \nabla_x \log p(x; \theta)$2 can be used (Song et al., 2019).

4. Extensions: Generalized Sliced Score Matching

Recent work extends SSM to arbitrary smooth “slices” ($s_\theta(x) = \nabla_x \log p(x; \theta)$3), not just linear projections (Robbins, 2024). The generalized SSM (GSSM) objective,

$s_\theta(x) = \nabla_x \log p(x; \theta)$4

includes Hessian and Laplacian terms arising from nonlinear $s_\theta(x) = \nabla_x \log p(x; \theta)$5. For linear $s_\theta(x) = \nabla_x \log p(x; \theta)$6, one recovers standard SSM.

GSSM allows the use of nonlinear projections, resulting in greater flexibility and potential for bias reduction, at the cost of increased variance and sample complexity. Empirical studies demonstrate that, on certain high-dimensional problems, GSSM and its variance-reduced version outperform standard SSM in score-matching and test log-likelihood (Robbins, 2024).

5. Applications in Modern Machine Learning

SSM and its generalizations have been deployed in several advanced contexts:

• Deep Energy-Based Models: SSM enables training deep kernel exponential families, outperforming denoising score matching and other Hessian-free approximations on UCI benchmarks. It scales to high-dimensional flows (e.g., NICE on MNIST, 784D) where exact score matching is prohibitively slow (Song et al., 2019).
• Implicit Likelihood Models: SSM provides superior or competitive scores compared to Stein and spectral kernel methods in variational auto-encoding with implicit encoders, achieving improved negative test log-likelihood and FID metrics (Song et al., 2019).
• Wasserstein Auto-Encoders: Tighter divergence matching between posterior and prior is achieved using SSM, yielding higher synthetic sample quality (Song et al., 2019).
• High-Dimensional SDEs and Fokker–Planck Equations: SSM serves as a core loss in score-based solvers for high-dimensional Fokker–Planck PDEs, maintaining accuracy and scaling linearly with dimension. Coupled with ODE-based log-likelihood inference, it enables tractable evaluation and sampling up to hundreds of dimensions (Hu et al., 2024).

The following table summarizes key application domains and their main SSM-driven advances:

Domain	Model/Context	SSM Impact
Deep EBMs	Kernel Exp. Family	Efficient, scalable learning
Implicit VAEs	Score Estimation	Outperforms kernel/Stein methods
WAE	Aggregated posterior	Tighter KL, improved samples
SDEs/Fokker–Planck	High-dimensional SDEs	Robust, linear scaling in dim.

6. Limitations and Practical Considerations

Principal limitations and operational factors include:

• Trace estimation variance: For very high $s_\theta(x) = \nabla_x \log p(x; \theta)$7, stochastic (Hutchinson-type) trace estimators introduce variance that may slow convergence or degrade final accuracy (Hu et al., 2024).
• Boundary and Heavy-Tailed Failures: In SDEs with heavy-tailed or otherwise pathological distributions, the SSM loss can diverge, typically due to ill-posed conditional scores at domain boundaries. In such cases, PDE-based regularization (e.g., Score-PINN) is more robust (Hu et al., 2024).
• Comparison with Standard Score Matching: While SSM is slightly less efficient per iteration than direct SM (due to higher-order differentiation), it applies in cases where conditional densities are unknown, and SM is not available.
• Projection Distribution Trade-offs: Uniform sphere projections reduce variance but require normalization, while Gaussian projections are computationally simpler (Song et al., 2019).

A plausible implication is that, in practice, selecting the projection distribution and the number of projections is task-dependent, balancing computational budget and estimator variance.

7. Outlook and Recent Developments

The extension from linear projections in SSM to arbitrary smooth “slicing” functions in GSSM expands the methodology’s adaptability (Robbins, 2024). This generalization leverages change-of-variable identities for the score, supporting richer classes of projections that can reduce bias at some increase in estimator variance and sample requirements.

Empirical investigations demonstrate that variance-reduced versions of GSSM can both stabilize training and outperform linear SSM in certain real-data scenarios (e.g., deep kernel exponential families on UCI datasets). These findings suggest that leveraging non-linear, data-adaptive projections may become increasingly important for high-dimensional or structured data distributions (Robbins, 2024).

Together, these results establish SSM as a core tool for score-based estimation in modern unnormalized modeling and provide a methodological foundation for its further extension to complex, high-dimensional, and implicit learning problems.