MVPP: Multi-View Predictive Partitioning

• MVPP is a clustering method that partitions high-dimensional, paired multivariate data by maximizing within-cluster predictive agreement.
• It employs two-block partial least squares (TB-PLS) regression to perform joint dimensionality reduction and predictive modeling on each cluster.
• The algorithm optimizes cluster assignments using a novel, analytic PRESS-derived predictive influence measure to ensure robust clustering.

Multi-View Predictive Partitioning (MVPP) is a clustering methodology for high-dimensional data in which each observation is represented by paired multivariate vectorial “views.” It is designed to discover clusters that exhibit strong within-cluster predictive agreement between these paired representations, rather than simple geometric proximity. MVPP utilizes the two-block partial least squares (TB-PLS) regression model within each cluster, with a novel objective based on predictive influence—a measure derived from an analytic leave-one-out cross-validation (LOOCV) statistic. As a result, the algorithm directly partitions data such that predictive relationships between views are maximized within clusters, enabling the identification of structure ignored by methods that rely solely on geometric similarity (McWilliams et al., 2012).

1. Two-Block Partial Least Squares in Cluster Modeling

MVPP models the dependency structure between paired high-dimensional views $\mathbf x_i \in \mathbb R^p$ and $\mathbf y_i \in \mathbb R^q$ observed for each data point $i$. Within each cluster of $n$ i.i.d. observations, TB-PLS introduces $R$ latent factors, decomposing the data as

$$\mathbf X = \sum_{r=1}^R \mathbf t^{(r)}\mathbf p^{(r)T} + \mathbf E_x, \quad \mathbf Y = \sum_{r=1}^R \mathbf s^{(r)}\mathbf q^{(r)T} + \mathbf E_y,$$

where $\mathbf t^{(r)} = \mathbf X\,\mathbf u^{(r)}$, $\mathbf s^{(r)} = \mathbf Y\,\mathbf v^{(r)}$, and each $(\mathbf t^{(r)}, \mathbf s^{(r)})$ pair maximizes covariance. The optimal directions are obtained via singular value decomposition (SVD) of $$\mathbf X^\top\mathbf Y = \mathbf U\,\boldsymbol\Lambda\,\mathbf V^\top$$, identifying $\mathbf y_i \in \mathbb R^q$0 as the $\mathbf y_i \in \mathbb R^q$1th singular vectors. Final regression of $\mathbf y_i \in \mathbb R^q$2 on $\mathbf y_i \in \mathbb R^q$3 proceeds via

$\mathbf y_i \in \mathbb R^q$4

with $\mathbf y_i \in \mathbb R^q$5 extracted from the inner regression.

The TB-PLS framework performs joint dimensionality reduction and predictive modeling, which is advantageous in high-dimensional settings frequently encountered in domains such as web mining and genomics.

2. Predictive Influence and the PRESS-Derived Objective

MVPP replaces geometric similarity with a predictive influence criterion rooted in a cluster-wise LOOCV PRESS statistic:

$\mathbf y_i \in \mathbb R^q$6

A closed-form approximation avoids costly leave-one-out retraining:

$\mathbf y_i \in \mathbb R^q$7

with $\mathbf y_i \in \mathbb R^q$8, $\mathbf y_i \in \mathbb R^q$9, $i$0 the projected scores, $i$1 the row residuals in $i$2, $i$3, and $i$4.

The predictive influence of each point $i$5 is given by the total derivative:

$i$6

where $i$7 quantifies how much $i$8 would change if $i$9 were perturbed. Points with high influence are poorly predicted by the TB-PLS model; this property anchors the MVPP clustering objective.

3. MVPP Algorithm and Optimization

The MVPP algorithm simultaneously estimates cluster assignments and cluster-specific TB-PLS models by minimizing the sum of predictive influences within each cluster. The main steps are:

1. Randomly assign each data point to one of $n$0 clusters (initial $n$1).
2. E–step: For each cluster $n$2, fit a TB-PLS model with $n$3 factors and compute $n$4 for all $n$5.
3. P–step: For each $n$6, re-assign to cluster $n$7 that minimizes $n$8.
4. Iterate E– and P–steps until assignments stabilize or a maximum number of iterations is reached.

Each E–step reduces within-cluster influence, and each P–step assigns each point to its minimal-influence cluster, which guarantees convergence to a local minimum.

4. Model Selection and Computational Complexity

Selecting the number of clusters $n$9 is performed by running MVPP for $R$0 and choosing $R$1 that minimizes the average PRESS across clusters, which typically identifies the ground-truth $R$2, as PRESS penalizes overfitting. The number of PLS latent components $R$3 per cluster can similarly be determined by PRESS minimization, though a default of $R$4 per cluster is often sufficient unless further reduction in PRESS is achieved by more components.

The algorithm’s per-iteration complexity comprises:

• $R$5 TB-PLS fits, each requiring SVD of a $R$6 matrix: $R$7 operations per fit.
• For each of $R$8 points and $R$9 clusters, computing and comparing influence norms: $$\mathbf X = \sum_{r=1}^R \mathbf t^{(r)}\mathbf p^{(r)T} + \mathbf E_x, \quad \mathbf Y = \sum_{r=1}^R \mathbf s^{(r)}\mathbf q^{(r)T} + \mathbf E_y,$$0. Total per-iteration: $$\mathbf X = \sum_{r=1}^R \mathbf t^{(r)}\mathbf p^{(r)T} + \mathbf E_x, \quad \mathbf Y = \sum_{r=1}^R \mathbf s^{(r)}\mathbf q^{(r)T} + \mathbf E_y,$$1. For $$\mathbf X = \sum_{r=1}^R \mathbf t^{(r)}\mathbf p^{(r)T} + \mathbf E_x, \quad \mathbf Y = \sum_{r=1}^R \mathbf s^{(r)}\mathbf q^{(r)T} + \mathbf E_y,$$2, the SVD cost dominates; the analytic PRESS/influence calculation circumvents the need for $$\mathbf X = \sum_{r=1}^R \mathbf t^{(r)}\mathbf p^{(r)T} + \mathbf E_x, \quad \mathbf Y = \sum_{r=1}^R \mathbf s^{(r)}\mathbf q^{(r)T} + \mathbf E_y,$$3 separate fits.

5. Empirical Performance and Comparison to Other Methods

MVPP was assessed on both synthetic and real-world high-dimensional data with paired-view structure. In synthetic “Scenario A” (geometric clusters), MVPP demonstrated robustness to noise and superior clustering accuracy and LOOCV prediction error relative to alternatives such as WCC, MV-CCA, and multi-view kernel approaches. In “Scenario B” (confounding geometry), where clusters are geometrically intermixed but have distinct predictive maps, conventional geometric methods failed to recover clusters, whereas MVPP succeeded.

On web-mining datasets (WebKB-2, WebKB-4) and the Citeseer citation network—a setting with dual high-dimensional views (e.g., text and link features)—MVPP reached approximately 100% (WebKB-2) and 80% (WebKB-4) clustering accuracy, outperforming geometric baselines (which scored 50–70%), and consistently provided the lowest LOOCV mean squared error.

6. Algorithmic Distinctions and Practical Significance

MVPP is distinct in its choice of within-cluster predictive modeling as the basis for clustering, as opposed to classical Euclidean or geometric proximity. By leveraging TB-PLS, it enables explicit modeling of inter-view relationships even in high-dimensional settings, where geometric distances are less informative or dominated by noise. The focus on minimizing predictive influence through an analytic PRESS extension allows for efficient evaluation of point-wise cluster fit, fostering the discovery of clusterings that are invisible to methods relying solely on geometric structure. This framework is especially important in application areas such as web mining and genomics, where multiview, high-dimensional data are common and predictive relationships underpin the relevant structure (McWilliams et al., 2012).