Second Principal Component Analysis

• Second Principal Component Analysis (2ndPCA) is an umbrella term for PCA extensions that analyze non-leading directions or higher dimensional data structures.
• It encompasses classical second PCs, matrix/tensor formulations, extended PCA methods with auxiliary terms, and time-series based segmentation.
• 2ndPCA preserves orthogonality and variance structure while adapting PCA to manage complex data types and support secondary analysis.

Searching arXiv for papers directly relevant to "Second Principal Component Analysis (2ndPCA)" and closely related formulations. First, locating the tutorial and extension papers already implicated by the topic. Searching arXiv for: (Shlens, 2014) Principal Component Analysis Shlens

Second Principal Component Analysis (2ndPCA) denotes a cluster of PCA-related ideas rather than a single standardized method. In the cited literature, it can mean the ordinary second principal component in classical PCA; a two-dimensional, second-order, or multilinear PCA acting on matrices or tensors; an extended PCA with an additional optimized term; a PCA for second-order stationary vector time series; or a second-level analysis of the relations between variables and principal components. This plurality of meanings reflects a common theme: moving beyond the first principal direction or beyond the first-stage covariance analysis while retaining PCA’s emphasis on orthogonality, variance structure, and linear transforms (Shlens, 2014, Zhao et al., 2019, Soto-Quiros et al., 2021, Hung et al., 2011, Chang et al., 2014).

1. Terminological scope and competing usages

The sources do not present a single canonical usage of the label “2ndPCA.” In Shlens’ tutorial, there is no separate algorithm beyond the standard second principal component. In the review of two-dimensional PCA, the term is not used, but the paper states that it “could reasonably refer” to higher-order or supervised PCA-type extensions. The extended PCA paper likewise does not explicitly use the phrase, but interprets its three-term construction as a “second-level PCA” extension. The order-two tensor paper states that, in the image-analysis community, closely related methods are called 2D-PCA, $(2D)^2$PCA, or “second principal component analysis (2ndPCA)” (Shlens, 2014, Zhao et al., 2019, Soto-Quiros et al., 2021, Hung et al., 2011).

Usage	Characteristic formulation	Representative source
Classical second PC	$\max_{\|w\|=1,\; w\perp w_1} w^T \Sigma w$	(Shlens, 2014)
Matrix-valued 2DPCA	$Y_i = U^T (X_i-\Psi) V$	(Zhao et al., 2019)
Order-two tensor MPCA	$Y = A^T X B$	(Hung et al., 2011)
Extended PCA	$\tilde{x} = T_0 y + T_1 \mathcal{f}(y,w)$	(Soto-Quiros et al., 2021)
Time-series second-order PCA	$\mathbf{X}_t = \mathbf{A}\mathbf{Z}_t$ with lag-based eigenanalysis	(Chang et al., 2014)
Law discovery from non-leading PCs	$z_k = X v_k$ with small $\lambda_k$	(Marzban et al., 2024)

This suggests that “2ndPCA” is best treated as an umbrella expression for second-component, second-order, two-dimensional, or second-level extensions of PCA. A common source of confusion is the assumption that it names one fixed algorithm. The literature instead ties it to several technically distinct constructions.

2. The classical meaning: the second principal component in ordinary PCA

In the standard setting, PCA starts from a centered data matrix $\mathbf{X}$ and covariance matrix

$$\mathbf{C_X} \equiv \frac{1}{n}\mathbf{X}\mathbf{X}^T.$$

PCA seeks an orthonormal matrix $\max_{\|w\|=1,\; w\perp w_1} w^T \Sigma w$0 such that $\max_{\|w\|=1,\; w\perp w_1} w^T \Sigma w$1 has diagonal covariance,

$\max_{\|w\|=1,\; w\perp w_1} w^T \Sigma w$2

If $\max_{\|w\|=1,\; w\perp w_1} w^T \Sigma w$3, choosing $\max_{\|w\|=1,\; w\perp w_1} w^T \Sigma w$4 yields $\max_{\|w\|=1,\; w\perp w_1} w^T \Sigma w$5, so the principal directions are the eigenvectors of the covariance matrix (Shlens, 2014).

Within this framework, the first principal component direction is

$\max_{\|w\|=1,\; w\perp w_1} w^T \Sigma w$6

and the second principal component direction is

$\max_{\|w\|=1,\; w\perp w_1} w^T \Sigma w$7

The solution is the eigenvector $\max_{\|w\|=1,\; w\perp w_1} w^T \Sigma w$8 associated with the second-largest eigenvalue $\max_{\|w\|=1,\; w\perp w_1} w^T \Sigma w$9. In this strict sense, “Second Principal Component Analysis” is simply analysis along the second orthogonal direction of maximal remaining variance; it is not a different algorithm (Shlens, 2014).

Geometrically, the second component is the direction of maximum remaining variance after the first axis has been fixed. Statistically, it explains variance $Y_i = U^T (X_i-\Psi) V$0, with proportion

$Y_i = U^T (X_i-\Psi) V$1

Together, $Y_i = U^T (X_i-\Psi) V$2 and $Y_i = U^T (X_i-\Psi) V$3 span the best two-dimensional linear subspace in the mean-squared error sense, and the rank-2 approximation

$Y_i = U^T (X_i-\Psi) V$4

reduces reconstruction error from $Y_i = U^T (X_i-\Psi) V$5 to $Y_i = U^T (X_i-\Psi) V$6 (Shlens, 2014).

Orthogonality is central here for two reasons. It diagonalizes the covariance in the transformed basis and avoids redundancy by preventing the second component from re-encoding the same variation already captured by the first. That interpretation remains the baseline against which other uses of 2ndPCA are defined.

3. Matrix-valued and tensor-valued formulations

A second major meaning of 2ndPCA arises when PCA is transferred from vectors to matrices or order-two tensors. In two-dimensional PCA, samples are images $Y_i = U^T (X_i-\Psi) V$7 rather than vectorized observations. The basic covariance objects are

$Y_i = U^T (X_i-\Psi) V$8

and projection uses orthonormal matrices

$Y_i = U^T (X_i-\Psi) V$9

with projected sample

$Y = A^T X B$0

The objective

$Y = A^T X B$1

separates into row and column directions, so $Y = A^T X B$2 and $Y = A^T X B$3 are formed from the top eigenvectors of $Y = A^T X B$4 and $Y = A^T X B$5 (Zhao et al., 2019).

The same paper places several advanced variants under a general ridge-type regression viewpoint. Improved 2DPCA, 2DPCA-$Y = A^T X B$6, 2DPCA$Y = A^T X B$7-S, and G2DPCA modify the projection objective and the norm constraints, while R2DPCA introduces label information through a weighting vector

$Y = A^T X B$8

and a relaxed criterion

$Y = A^T X B$9

For multiple components, orthogonality constraints $\tilde{x} = T_0 y + T_1 \mathcal{f}(y,w)$0 and $\tilde{x} = T_0 y + T_1 \mathcal{f}(y,w)$1 are imposed, and extracted components are themselves weighted by $\tilde{x} = T_0 y + T_1 \mathcal{f}(y,w)$2 during classification and reconstruction (Zhao et al., 2019).

Order-two tensor MPCA provides a closely related but statistically explicit formulation. For $\tilde{x} = T_0 y + T_1 \mathcal{f}(y,w)$3, MPCA uses the bilinear projection

$\tilde{x} = T_0 y + T_1 \mathcal{f}(y,w)$4

with $\tilde{x} = T_0 y + T_1 \mathcal{f}(y,w)$5, $\tilde{x} = T_0 y + T_1 \mathcal{f}(y,w)$6, and tensor principal components $\tilde{x} = T_0 y + T_1 \mathcal{f}(y,w)$7. Its population objective is

$\tilde{x} = T_0 y + T_1 \mathcal{f}(y,w)$8

which makes MPCA a constrained PCA with Kronecker-structured subspace $\tilde{x} = T_0 y + T_1 \mathcal{f}(y,w)$9. The order-two tensor study states that MPCA and $\mathbf{X}_t = \mathbf{A}\mathbf{Z}_t$0PCA target the same underlying subspace under a tensor factor model, but MPCA is asymptotically more statistically efficient because its kernel matrices are less noise contaminated (Hung et al., 2011).

This family of matrix and tensor formulations gives one of the most common technical interpretations of 2ndPCA: PCA that preserves order-two structure instead of flattening data into vectors.

4. Extended and second-order generalizations beyond matrix PCA

Another line of work interprets 2ndPCA as a PCA extension with an additional optimized term. “Extended PCA” or “three-term PCA” reconstructs $\mathbf{X}_t = \mathbf{A}\mathbf{Z}_t$1 from observed data $\mathbf{X}_t = \mathbf{A}\mathbf{Z}_t$2 and an auxiliary vector $\mathbf{X}_t = \mathbf{A}\mathbf{Z}_t$3 through

$\mathbf{X}_t = \mathbf{A}\mathbf{Z}_t$4

A key construction is

$\mathbf{X}_t = \mathbf{A}\mathbf{Z}_t$5

which enforces $\mathbf{X}_t = \mathbf{A}\mathbf{Z}_t$6. The resulting covariance structure

$\mathbf{X}_t = \mathbf{A}\mathbf{Z}_t$7

permits a PCA-like truncated SVD on

$\mathbf{X}_t = \mathbf{A}\mathbf{Z}_t$8

The paper shows that the method always exists, including singular cases, because it is built from pseudo-inverses and SVD. It also states that the three-term PCA error is no worse than the GBT1/PCA error, and strictly smaller when $\mathbf{X}_t = \mathbf{A}\mathbf{Z}_t$9 is positive definite (Soto-Quiros et al., 2021).

The same paper quantifies numerical load. For $z_k = X v_k$0,

$z_k = X v_k$1

Hence three-term PCA costs approximately $z_k = X v_k$2 of GBT2 and approximately $z_k = X v_k$3 of GKLT for large dimensions (Soto-Quiros et al., 2021).

A different second-order generalization appears in principal component analysis for second-order stationary vector time series. There the goal is a contemporaneous transformation

$z_k = X v_k$4

such that $z_k = X v_k$5 is segmented into lower-dimensional subseries that are uncorrelated both contemporaneously and serially. The central matrices are

$z_k = X v_k$6

with

$z_k = X v_k$7

The procedure reduces to eigenanalysis of a positive definite matrix followed by a permutation step based on maximum cross-correlations or FDR-based multiple testing. Unlike PCA for independent data, the required linear transformation may not exist; when it does not, the method provides an approximate segmentation (Chang et al., 2014).

These two strands—three-term reconstruction and lag-structured segmentation—extend 2ndPCA away from ordinary variance maximization toward richer second-stage linear estimation problems.

5. Interpretation, component selection, and second-level analysis

A further meaning of 2ndPCA concerns how non-leading components are interpreted and selected. In equation discovery, PCA is used not primarily for the first few principal components but for the last or one of the last components, since a small eigenvalue implies that the corresponding linear combination

$z_k = X v_k$8

is nearly constant across cases. The paper argues that such constant-combinations may reflect underlying laws of nature. In the Kepler example, the law appears in a zero-eigenvalue component rather than in a leading PC; in the meteorology example, the relevant law is associated with eigenvector $z_k = X v_k$9, chosen by combining a relatively small eigenvalue with low spatial variability of the loadings for $\lambda_k$0 and $\lambda_k$1 (Marzban et al., 2024).

This use of PCA broadens the meaning of “second” in 2ndPCA. It may denote PC2, but it may also denote any non-leading component whose scores are nearly constant and whose loadings are physically interpretable. The paper explicitly states that the law may not correspond to the last principal component, but rather to one of the last, so a selection criterion is required (Marzban et al., 2024).

The “new interpretation” paper proposes a different second-level analysis. It treats the correlation between a variable and a principal component as a cosine,

$\lambda_k$2

and the determination coefficient as

$\lambda_k$3

With standardized variables, the squared correlations $\lambda_k$4 quantify how much of the variance of variable $\lambda_k$5 is explained by principal component $\lambda_k$6. The paper proposes selecting PCs not only by global explained variance but also by ensuring that, for each variable $\lambda_k$7,

$\lambda_k$8

exceeds an acceptable threshold. It also proposes clustering primary variables by similarity to principal components using these determination coefficients (Gniazdowski, 2017).

This suggests a secondary analytic layer over PCA itself: one examines not only the component directions but also the geometry of variable–component relations, variable-wise reconstruction quality, and vertical clustering of variables. In that sense, 2ndPCA can mean a second analysis of PCA structure rather than a second component alone.

6. Applications, reported results, and limitations

The practical uses associated with 2ndPCA depend on which formulation is intended. In face recognition and image reconstruction, R2DPCA and related 2DPCA variants operate directly on image matrices. The reported recognition rates are as follows (Zhao et al., 2019):

Dataset	Method	Reported rate
Faces95	2DPCA	0.8729
Faces95	G2DPCA	0.9451
Faces95	R2DPCA	0.9493
Color FERET	2DPCA	0.5982
Color FERET	G2DPCA	0.6918
Color FERET	R2DPCA	0.7085

The same study states that, in MNIST digit recognition, R2DPCA outperforms CNNs, DBNs, and DNNs under the specific architectures and training regimes used in the experiments, and that for $\lambda_k$9 R2DPCA coincides with G2DPCA (Zhao et al., 2019).

For order-two MPCA on the Olivetti Faces dataset, the reported mean test reconstruction error is $\mathbf{X}$0 for MPCA versus $\mathbf{X}$1 for conventional PCA, with standard deviations $\mathbf{X}$2 and $\mathbf{X}$3, respectively. The paper attributes this to the more parsimonious, module-oriented basis induced by bilinear structure (Hung et al., 2011).

In time-series analysis, TS-PCA serves as an initial step for segmentation, dimension reduction, and forecasting of high-dimensional linear dynamical structures. The paper reports lower mean and standard deviation of MSEs than VAR and restricted VAR in several real-data examples, and extends the idea to segment multiple volatility processes after GARCH modeling (Chang et al., 2014).

Several limitations recur across the literature. Classical PCA assumes linearity, orthogonality, and that large variances reflect important structure; the tutorial explicitly notes that the assumption that “large variances have important structure” is strong and sometimes incorrect. Equation discovery from PCA requires appropriate transformations, such as log transforms for multiplicative laws, and may be confounded by degeneracy of small eigenvalues, noise, or high dimensionality. Matrix and tensor formulations preserve structure but rely on row/column or multilinear assumptions. TS-PCA does not guarantee that an exact segmentation exists. Extended PCA requires auxiliary injection vectors and additional covariance estimation. These limitations explain why 2ndPCA remains a family resemblance term rather than a single settled method (Shlens, 2014, Marzban et al., 2024, Chang et al., 2014, Soto-Quiros et al., 2021).

Taken together, the literature presents 2ndPCA as a technically plural concept. In its narrowest meaning it is the second principal component of classical PCA. In broader usage it denotes two-dimensional or second-order PCA on matrices and tensors, extended PCA with an added transform term, lag-structured PCA for stationary vector time series, or second-level analyses of variable–component relations and low-variance law-bearing directions. The continuity across these meanings lies in the attempt to preserve PCA’s eigen-analytic core while extending what counts as the relevant “second” structure.