Robust Shrinkage Estimation of High-dimensional Covariance Matrices (1009.5331v1)

Abstract: We address high dimensional covariance estimation for elliptical distributed samples, which are also known as spherically invariant random vectors (SIRV) or compound-Gaussian processes. Specifically we consider shrinkage methods that are suitable for high dimensional problems with a small number of samples (large $p$ small $n$). We start from a classical robust covariance estimator [Tyler(1987)], which is distribution-free within the family of elliptical distribution but inapplicable when $n<p$. Using a shrinkage coefficient, we regularize Tyler's fixed point iterations. We prove that, for all $n$ and $p$, the proposed fixed point iterations converge to a unique limit regardless of the initial condition. Next, we propose a simple, closed-form and data dependent choice for the shrinkage coefficient, which is based on a minimum mean squared error framework. Simulations demonstrate that the proposed method achieves low estimation error and is robust to heavy-tailed samples. Finally, as a real world application we demonstrate the performance of the proposed technique in the context of activity/intrusion detection using a wireless sensor network.

• The paper introduces a shrinkage estimator that extends Tyler’s method to high-dimensional settings, ensuring convergence even when n is less than p.
• It derives a closed-form, data-dependent shrinkage coefficient using an MMSE framework to optimally balance empirical and target covariance matrices.
• Simulations demonstrate that the proposed method outperforms classical estimators, achieving robust accuracy for heavy-tailed and Gaussian data in real-world applications.

Robust Shrinkage Estimation of High-dimensional Covariance Matrices

The addressed paper proposes an innovative methodology for the estimation of high-dimensional covariance matrices. Specifically, the authors target spherically invariant random vectors (SIRV) or compound-Gaussian processes, which are relevant to numerous fields such as radar detection, wireless communication, and financial engineering. In transfer from low to high dimensional settings, classical covariance matrix estimation becomes impractical, mainly because the small sample size $n$ relative to the large number of dimensions $p$ results in ill-conditioned matrices. The authors tackle this issue by building upon robust estimation techniques and introducing shrinkage methods aimed at high-dimensional problems characterized by large $p$ and small $n$.

The paper begins with an examination of Tyler's robust covariance estimator, originally designed under the assumption of an elliptical distribution. This estimator, however, becomes ineffective when $n < p$ as it does not satisfy the necessary conditions for existence and convergence. To resolve this, the authors propose a regularization technique via a shrinkage estimator. By incorporating a shrinkage coefficient into Tyler's fixed point iterations, the authors ensure convergence to a unique solution for any initial positive definite matrix, regardless of the relationship between $n$ and $p$.

A significant advancement in the paper is the derivation of a closed-form, data-dependent shrinkage coefficient using a minimum mean-squared-error (MMSE) framework. Through simulations, this coefficient selects the optimal balance between the empirical sample covariance and the target matrix, achieving a robust estimation for heavy-tailed distributions such as the multivariate Student-T distribution.

The paper moves beyond theoretical exposition to practical application, demonstrating the effectiveness of the proposed methodology in real-world scenarios like activity or intrusion detection, particularly in wireless sensor networks. The authors highlight the non-Gaussian nature of most real-world data, making their proposed method especially potent. The method's performance is validated through simulations against well-established estimators such as Ledoit-Wolf and the classical sample covariance, showcasing superior performance in both heavy-tailed and Gaussian scenarios.

Key numerical results indicate the proposed method's enhanced performance with multivariate Student-T distributed samples. The authors demonstrate substantial robustness and improved accuracy compared to existing methods, particularly in challenging settings where $p > n$. Additionally, while the method is derived for elliptical distributions, the simulations demonstrate minimal performance loss when the distribution of the samples is purely Gaussian, thereby pointing to the method's versatility.

The paper's implications for practical and theoretical development in statistical signal processing are significant. The robustness to heavy-tailed data makes it applicable in diverse fields, including finance and communication, where such data types are commonplace. The convergence guarantees and closed-form solutions facilitate practical implementation and computational efficiency.

Future research might extend these techniques to other types of M-estimators, potentially broadening their application scope. Additionally, the geometric insights gained from alternative proofs of convergence, such as those using manifolds, could be exploited to further refine the method.

Overall, this research contributes significantly to robust covariance matrix estimation in high-dimensional scenarios where traditional techniques fail, setting the stage for further advances in signal processing and related domains.