The Optimal Hard Threshold for Singular Values is 4/sqrt(3) (1305.5870v3)

Abstract: We consider recovery of low-rank matrices from noisy data by hard thresholding of singular values, where singular values below a prescribed threshold $\lambda$ are set to 0. We study the asymptotic MSE in a framework where the matrix size is large compared to the rank of the matrix to be recovered, and the signal-to-noise ratio of the low-rank piece stays constant. The AMSE-optimal choice of hard threshold, in the case of n-by-n matrix in noise level \sigma, is simply $(4/\sqrt{3}) \sqrt{n}\sigma \approx 2.309 \sqrt{n}\sigma$ when $\sigma$ is known, or simply $2.858\cdot y_{med}$ when $\sigma$ is unknown, where $y_{med}$ is the median empirical singular value. For nonsquare $m$ by $n$ matrices with $m \neq n$, these thresholding coefficients are replaced with different provided constants. In our asymptotic framework, this thresholding rule adapts to unknown rank and to unknown noise level in an optimal manner: it is always better than hard thresholding at any other value, no matter what the matrix is that we are trying to recover, and is always better than ideal Truncated SVD (TSVD), which truncates at the true rank of the low-rank matrix we are trying to recover. Hard thresholding at the recommended value to recover an n-by-n matrix of rank r guarantees an AMSE at most $3nr\sigma^2$. In comparison, the guarantee provided by TSVD is $5nr\sigma^2$, the guarantee provided by optimally tuned singular value soft thresholding is $6nr\sigma^2$, and the best guarantee achievable by any shrinkage of the data singular values is $2nr\sigma^2$. Empirical evidence shows that these AMSE properties of the $4/\sqrt{3}$ thresholding rule remain valid even for relatively small n, and that performance improvement over TSVD and other shrinkage rules is substantial, turning it into the practical hard threshold of choice.

• The paper identifies the AMSE-optimal hard threshold for singular values as 4/sqrt(3), corresponding to approximately 2.309√nσ for n×n matrices with known noise levels.
• It introduces a data-driven threshold of 2.858 times the median singular value for cases when the noise level is unknown, outperforming traditional TSVD.
• The findings offer both theoretical insights and practical improvements for low-rank matrix denoising, with significant applications in signal processing and machine learning.

Optimal Hard Threshold for Singular Values

The paper by Matan Gavish and David L. Donoho explores low-rank matrix recovery from noisy data using singular value hard thresholding. This method involves setting singular values below a certain threshold to zero, aiming to minimize the asymptotic mean squared error (AMSE). The primary finding of this work is the identification of an AMSE-optimal threshold for hard thresholding of singular values, specifically $\frac{4}{\sqrt{3}}$.

Summary of Findings

The research establishes that the AMSE-optimal hard threshold for an $n \times n$ matrix in noise with a known level $\sigma$ is approximately $2.309 \sqrt{n} \sigma$. For cases where $\sigma$ is unknown, a data-driven threshold of $2.858 \times \text{median singular value}$ is recommended. These thresholds adapt optimally even when the rank and noise levels are unknown, outperforming Truncated SVD (TSVD) in AMSE, which truncates at the true rank of the matrix.

For non-square matrices, the optimal coefficients are adjusted based on the matrix's dimensions ratio. The hard thresholding method proposed is always superior or at least equal to any other threshold or shrinkage method, including the ideal TSVD. This paper also delivers strong numerical results, indicating that the recommended value grants substantial AMSE improvement over other techniques such as TSVD, soft thresholding, and various shrinkage rules.

Implications and Applications

This work has significant implications for practical and theoretical data recovery scenarios particularly concerning the denoising and recovery of low-rank matrices corrupted by noise.

1. Practical Applications:
   • The findings are crucial for data analysis tasks where high-dimensional data matrices are decomposed into low-rank approximations, such as in signal processing and machine learning.
   • The optimal thresholding method provides a simple yet effective way to perform matrix approximation and noise reduction without prior knowledge of the rank or exact noise levels.
2. Theoretical Contributions:
   • The theoretically derived threshold gives insight into the behavior of singular values and their optimal handling under the influence of noise.
   • This work challenges the widely used thresholding near the bulk edge, proving that a higher threshold is generally more effective when considering the AMSE.

Future Directions

The implications of the optimal threshold go beyond the specific setup of low-rank matrices in white noise. Future research could extend this methodology to different matrix forms and noise models, such as colored noise scenarios or matrices with correlated entries. Additionally, exploring the impact of different data distributions on the threshold parameter could further enhance the robustness of matrix approximation and reconstruction strategies.

To summarize, this paper provides a rigorous examination of singular value thresholding techniques, proposing an optimal threshold strategy that outperforms traditional methods. This finding not only informs practical implementations but also enriches the theoretical understanding of matrix recovery in noisy environments.