Singular values for products of two coupled random matrices: hard edge phase transition

Abstract: Consider the product $GX$ of two rectangular complex random matrices coupled by a constant matrix $\Omega$, where $G$ can be thought to be a Gaussian matrix and $X$ is a bi-invariant polynomial ensemble. We prove that the squared singular values form a biorthogonal ensemble in Borodin's sense, and further that for $X$ being Gaussian the correlation kernel can be expressed as a double contour integral. When all but finitely many eigenvalues of $\Omega^{} \Omega^{*}$ are equal, the corresponding correlation kernel is shown to admit a phase transition phenomenon at the hard edge in four different regimes as the coupling matrix changes. Specifically, the four limiting kernels in turn are the Meijer G-kernel for products of two independent Gaussian matrices, a new critical and interpolating kernel, the perturbed Bessel kernel and the finite coupled product kernel associated with $GX$. In the special case that $X$ is also a Gaussian matrix and $\Omega$ is scalar, such a product has been recently investigated by Akemann and Strahov. We also propose a Jacobi-type product and prove the same transition.