Ratio-consistent estimation for long range dependent Toeplitz covariance with application to matrix data whitening

Abstract: We consider a data matrix $X:=C_N^{1/2}ZR_M^{1/2}$ from a multivariate stationary process with a separable covariance function, where $C_N$ is a $N\times N$ positive semi-definite matrix, $Z$ a $N\times M$ random matrix of uncorrelated standardized white noise, and $R_M$ a $M\times M$ Toeplitz matrix. Under the assumption of long range dependence (LRD), we re-examine the consistency of two toeplitzifized estimators $\hat R_M$ (unbiased) and $\hat R_M^b$ (biased) for $R_M$, which are known to be norm consistent with $R_M$ when the process is short range dependent (SRD). However in the LRD case, some simulations suggest that the norm consistency does not hold in general for both estimators. Instead, a weaker {\it ratio consistency} is established for the unbiased estimator $\hat R_M$, and a further weaker {\it ratio LSD consistency} is established for the biased estimator $\hat R_M^b$. The main result leads to a consistent whitening procedure on the original data matrix $X$, which is further applied to two real world questions, one is a signal detection problem, and the other is PCA on the space covariance $C_N$ to achieve a noise reduction and data compression.