Asymptotic Joint Distribution of Extreme Eigenvalues of the Sample Covariance Matrix in the Spiked Population Model

Abstract: In this paper, we consider a data matrix $X\in\mathbb{C}^{N\times M}$ where all the columns are i.i.d. samples being $N$ dimensional complex Gaussian of mean zero and covariance $\Sigma\in\mathbb{C}^{N\times N}$. Here the population matrix $\Sigma$ is of finite rank perturbation of the identity matrix. This is the "spiked population model" first proposed by Johnstone in \cite{21}. As $M, N\to\infty$ but $M/N = \gamma\in(1, \infty)$, we first establish in this paper the asymptotic distribution of the smallest eigenvalue of the sample covariance matrix $S:= XX^*/M$. It also exhibits a phase transition phenomenon proposed in \cite{1} --- the local fluctuation will be the generalized Tracy-Widom or the generalized Gaussian to be defined in the paper. Moreover we prove that the largest and the smallest eigenvalue are asymptotically independent as $M, N\to\infty$.