The total variation distance between high-dimensional Gaussians with the same mean

Abstract: Given two high-dimensional Gaussians with the same mean, we prove a lower and an upper bound for their total variation distance, which are within a constant factor of one another.

• The paper establishes tight lower and upper bounds for the total variation distance between high-dimensional Gaussians sharing the same mean.
• It employs eigenvalue analysis of covariance matrix differences to derive computable bounds in terms of matrix norms.
• The findings have practical implications for hypothesis testing and machine learning by quantifying distribution divergence in complex data.

Analysis of Total Variation Distance Between High-Dimensional Gaussians with Identical Means

The subject of this paper is the total variation distance between two high-dimensional Gaussian distributions with the same mean. The authors establish lower and upper bounds for this metric, showing that these bounds are within a constant factor of each other. The study primarily focuses on Gaussian distributions due to their fundamental role in probability theory—a result of their properties as described by the central limit theorem.

Key Contributions

• Total Variation Distance: The paper examines the scenario where two Gaussian distributions, $N(\mu, \Sigma_1)$ and $N(\mu, \Sigma_2)$, share the same mean $\mu$ but have different covariance matrices, $\Sigma_1$ and $\Sigma_2$. This setup is paramount in understanding the closeness or divergence of distributions through the total variation distance, defined as:

$TV{P}{Q} = \sup_{A\subseteq R^d} |P(A)-Q(A)|$

For distributions with known densities, this can be expressed as half the $L^1$ distance between the densities.

• Bounds for Distribution with Identical Means: The authors derive both lower and upper bounds for the total variation distance in the context of Gaussian distributions sharing the same mean.
  • The pivotal result for positive definite covariance matrices ($\Sigma_1$ and $\Sigma_2$) is:

  $$\frac{1}{100} \leq TV{N(\mu,\Sigma_1)}{N(\mu,\Sigma_2)} \leq \frac{3}{2} \min \left\{1, \sqrt{\sum_{i=1}^{d} \lambda_i^2} \right\}$$

  Here, $\lambda_i$ are the eigenvalues of $\Sigma_1^{-1}\Sigma_2-I_d$.

• Extension to Positive Semi-Definite Case: They extend this result to the case where the covariance matrices are positive semi-definite. By leveraging the range of these matrices and projecting onto a suitable subspace, similar tight bounds on the total variation distance are derived.
• Case of Different Means: For Gaussian distributions with differing means, a lower bound for the total variation distance is established. The primary focus remains on expressing these bounds in easily computable terms involving matrix eigenvalues and norms.

Implications

The derivations of these bounds are not only beneficial in theoretical analysis but also in practical applications where quantification of divergence between Gaussian distributions is critical—such as in hypothesis testing and information theory. Researchers and practitioners can leverage these insights for areas like high-dimensional statistics, machine learning, and signal processing.

These results might spur further research into algorithms that utilize Gaussian distributions or into approximating distributional divergence efficiently in multi-dimensional spaces.

Future Directions

The results obtained, particularly the tight bounds in the same-mean case and the extension to distributions with different means, open the door for further research on:

• Developing algorithms that compute these metrics efficiently in high-dimensional data analysis scenarios.
• Extending the framework to incorporate non-Gaussian distributions or handling covariance matrices more general than positive definite ones.
• Addressing and solving the open problem of finding closed-form lower and upper bounds for distributions with identical means, which could enhance understanding and manipulation of total variation distances in more complex stochastic settings.

The insights gained from this work contribute substantially to the toolbox of methods for measuring distributional divergence, underpinning both theoretical advancements and practical implementations in a range of disciplines.