On some limitations of probabilistic models for dimension-reduction: Illustration in the case of probabilistic formulations of partial least squares

Abstract: Partial Least Squares (PLS) refer to a class of dimension-reduction techniques aiming at the identification of two sets of components with maximal covariance, to model the relationship between two sets of observed variables $x\in\mathbb{R}^p$ and $y\in\mathbb{R}^q$, with $p\geq 1, q\geq 1$. Probabilistic formulations have recently been proposed for several versions of the PLS. Focusing first on the probabilistic formulation of the PLS-SVD proposed by el Bouhaddani et al., we establish that the constraints on their model parameters are too restrictive and define particular distributions for $(x,y)$, under which components with maximal covariance (solutions of PLS-SVD) are also necessarily of respective maximal variances (solutions of principal components analyses of $x$ and $y$, respectively). We propose an alternative probabilistic formulation of PLS-SVD, no longer restricted to these particular distributions. We then present numerical illustrations of the limitation of the original model of el Bouhaddani et al. We also briefly discuss similar limitations in another latent variable model for dimension-reduction.