---
title: Second Principal Component Analysis
url: https://www.emergentmind.com/topics/second-principal-component-analysis-2ndpca
type: topic
---

# Second Principal Component Analysis

Searching arXiv for papers directly relevant to "Second Principal Component Analysis (2ndPCA)" and closely related formulations.
First, locating the tutorial and extension papers already implicated by the topic.
Searching arXiv for: 1404.1100 Principal Component Analysis Shlens

Second Principal Component Analysis (2ndPCA) denotes a cluster of PCA-related ideas rather than a single standardized method. In the cited literature, it can mean the ordinary second principal component in classical PCA; a two-dimensional, second-order, or multilinear PCA acting on matrices or tensors; an extended PCA with an additional optimized term; a PCA for second-order stationary vector time series; or a second-level analysis of the relations between variables and principal components. This plurality of meanings reflects a common theme: moving beyond the first principal direction or beyond the first-stage covariance analysis while retaining PCA’s emphasis on orthogonality, variance structure, and linear transforms [1404.1100] [1912.09970] [2111.03040] [1104.5281] [1410.2323].

## 1. Terminological scope and competing usages

The sources do not present a single canonical usage of the label “2ndPCA.” In Shlens’ tutorial, there is no separate algorithm beyond the standard second principal component. In the review of two-dimensional PCA, the term is not used, but the paper states that it “could reasonably refer” to higher-order or supervised PCA-type extensions. The extended PCA paper likewise does not explicitly use the phrase, but interprets its three-term construction as a “second-level PCA” extension. The order-two tensor paper states that, in the image-analysis community, closely related methods are called 2D-PCA, \((2D)^2\)PCA, or “second principal component analysis (2ndPCA)” [1404.1100] [1912.09970] [2111.03040] [1104.5281].

| Usage | Characteristic formulation | Representative source |
|---|---|---|
| Classical second PC | \(\max_{\|w\|=1,\; w\perp w_1} w^T \Sigma w\) | [1404.1100] |
| Matrix-valued 2DPCA | \(Y_i = U^T (X_i-\Psi) V\) | [1912.09970] |
| Order-two tensor MPCA | \(Y = A^T X B\) | [1104.5281] |
| Extended PCA | \(\tilde{x} = T_0 y + T_1 \mathcal{f}(y,w)\) | [2111.03040] |
| Time-series second-order PCA | \(\mathbf{X}_t = \mathbf{A}\mathbf{Z}_t\) with lag-based eigenanalysis | [1410.2323] |
| Law discovery from non-leading PCs | \(z_k = X v_k\) with small \(\lambda_k\) | [2401.04797] |

This suggests that “2ndPCA” is best treated as an umbrella expression for second-component, second-order, two-dimensional, or second-level extensions of PCA. A common source of confusion is the assumption that it names one fixed algorithm. The literature instead ties it to several technically distinct constructions.

## 2. The classical meaning: the second principal component in ordinary PCA

In the standard setting, PCA starts from a centered data matrix \(\mathbf{X}\) and covariance matrix
\[
\mathbf{C_X} \equiv \frac{1}{n}\mathbf{X}\mathbf{X}^T.
\]
PCA seeks an orthonormal matrix \(\mathbf{P}\) such that \(\mathbf{Y}=\mathbf{P}\mathbf{X}\) has diagonal covariance,
\[
\mathbf{C_Y}=\frac{1}{n}\mathbf{Y}\mathbf{Y}^T=\mathbf{P}\mathbf{C_X}\mathbf{P}^T.
\]
If \(\mathbf{C_X}=\mathbf{EDE}^T\), choosing \(\mathbf{P}=\mathbf{E}^T\) yields \(\mathbf{C_Y}=\mathbf{D}\), so the principal directions are the eigenvectors of the covariance matrix [1404.1100].

Within this framework, the first principal component direction is
\[
\mathbf{w}_1=\arg\max_{\|\mathbf{w}\|=1}\mathbf{w}^T\Sigma\mathbf{w},
\]
and the second principal component direction is
\[
\mathbf{w}_2=\arg\max_{\|\mathbf{w}\|=1,\;\mathbf{w}\perp\mathbf{w}_1}\mathbf{w}^T\Sigma\mathbf{w}.
\]
The solution is the eigenvector \(\mathbf{q}_2\) associated with the second-largest eigenvalue \(\lambda_2\). In this strict sense, “Second Principal Component Analysis” is simply analysis along the second orthogonal direction of maximal remaining variance; it is not a different algorithm [1404.1100].

Geometrically, the second component is the direction of maximum remaining variance after the first axis has been fixed. Statistically, it explains variance \(\lambda_2\), with proportion
\[
\mathrm{PVE}_2=\frac{\lambda_2}{\sum_{j=1}^m \lambda_j}.
\]
Together, \(\mathbf{w}_1\) and \(\mathbf{w}_2\) span the best two-dimensional linear subspace in the mean-squared error sense, and the rank-2 approximation
\[
\mathbf{X}\approx \mathbf{q}_1\mathbf{t}_1^T+\mathbf{q}_2\mathbf{t}_2^T
\]
reduces reconstruction error from \(\sum_{j=2}^m\lambda_j\) to \(\sum_{j=3}^m\lambda_j\) [1404.1100].

Orthogonality is central here for two reasons. It diagonalizes the covariance in the transformed basis and avoids redundancy by preventing the second component from re-encoding the same variation already captured by the first. That interpretation remains the baseline against which other uses of 2ndPCA are defined.

## 3. Matrix-valued and tensor-valued formulations

A second major meaning of 2ndPCA arises when PCA is transferred from vectors to matrices or order-two tensors. In two-dimensional PCA, samples are images \(X_i\in\mathbb{R}^{h\times w}\) rather than vectorized observations. The basic covariance objects are
\[
E_1=\frac{1}{N}\sum_{i=1}^N (X_i-\Psi)(X_i-\Psi)^T,\qquad
E_2=\frac{1}{N}\sum_{i=1}^N (X_i-\Psi)^T(X_i-\Psi),
\]
and projection uses orthonormal matrices
\[
U\in\mathbb{R}^{h\times k_1},\qquad V\in\mathbb{R}^{w\times k_2},
\]
with projected sample
\[
Y_i=U^T(X_i-\Psi)V.
\]
The objective
\[
J(U,V)=\operatorname{Tr}(U^T E_1 U)+\operatorname{Tr}(V^T E_2 V)
\]
separates into row and column directions, so \(\widehat U\) and \(\widehat V\) are formed from the top eigenvectors of \(E_1\) and \(E_2\) [1912.09970].

The same paper places several advanced variants under a general ridge-type regression viewpoint. Improved 2DPCA, 2DPCA-\(L_1\), 2DPCA\(L_1\)-S, and G2DPCA modify the projection objective and the norm constraints, while R2DPCA introduces label information through a weighting vector
\[
\omega_j=\frac{f(\lambda_{\max}(C_j))}{\sum_{i=1}^m f(\lambda_{\max}(C_i))}
\]
and a relaxed criterion
\[
J(u,v)=\gamma\,G(u,v)+(1-\gamma)\,\widetilde G(u,v),\qquad 0\le\gamma\le 1.
\]
For multiple components, orthogonality constraints \(u_i^T u_j=0\) and \(v_i^T v_j=0\) are imposed, and extracted components are themselves weighted by \(D=\operatorname{diag}(f_1,\dots,f_k)\) during classification and reconstruction [1912.09970].

Order-two tensor MPCA provides a closely related but statistically explicit formulation. For \(X\in\mathbb{R}^{p\times q}\), MPCA uses the bilinear projection
\[
Y=A^T X B,
\]
with \(A\in\mathcal{O}_{p,\tilde p}\), \(B\in\mathcal{O}_{q,\tilde q}\), and tensor principal components \(\{b_j\otimes a_i\}\). Its population objective is
\[
\arg\max_{A,B}\operatorname{tr}\big((B\otimes A)^T\Sigma(B\otimes A)\big),
\]
which makes MPCA a constrained PCA with Kronecker-structured subspace \(\Gamma=B\otimes A\). The order-two tensor study states that MPCA and \((2D)^2\)PCA target the same underlying subspace under a tensor factor model, but MPCA is asymptotically more statistically efficient because its kernel matrices are less noise contaminated [1104.5281].

This family of matrix and tensor formulations gives one of the most common technical interpretations of 2ndPCA: PCA that preserves order-two structure instead of flattening data into vectors.

## 4. Extended and second-order generalizations beyond matrix PCA

Another line of work interprets 2ndPCA as a PCA extension with an additional optimized term. “Extended PCA” or “three-term PCA” reconstructs \(\mathbf{x}\) from observed data \(\mathbf{y}\) and an auxiliary vector \(\mathbf{w}\) through
\[
\tilde{\mathbf{x}}=T_0\mathbf{y}+T_1\mathcal{f}(\mathbf{y},\mathbf{w}),
\qquad \operatorname{rank}[T_0\ T_1]\le k.
\]
A key construction is
\[
\mathbf{s}=\mathcal{f}(\mathbf{y},\mathbf{w})=\mathbf{w}-G_{wy}\mathbf{y},\qquad
G_{wy}=E_{wy}E_{yy}^\dag,
\]
which enforces \(E_{ys}=0\). The resulting covariance structure
\[
E_{zz}=
\begin{bmatrix}
E_{yy} & 0\\
0 & E_{ss}
\end{bmatrix}
\]
permits a PCA-like truncated SVD on
\[
G_z=G_y+G_s=E_{xz}E_{zz}^\dag E_{zx}.
\]
The paper shows that the method always exists, including singular cases, because it is built from pseudo-inverses and SVD. It also states that the three-term PCA error is no worse than the GBT1/PCA error, and strictly smaller when \(G_s\) is positive definite [2111.03040].

The same paper quantifies numerical load. For \(m=n=\ell\),
\[
C_{\text{PCA3}}=52m^3+2m^2(k+1),\quad
C_{\text{GBT2}}=140m^3+2m^2(k+2),\quad
C_{\text{GKLT}}=240m^3+4m^2(k+1)+mk.
\]
Hence three-term PCA costs approximately \(37\%\) of GBT2 and approximately \(22\%\) of GKLT for large dimensions [2111.03040].

A different second-order generalization appears in principal component analysis for second-order stationary vector time series. There the goal is a contemporaneous transformation
\[
\mathbf{X}_t=\mathbf{A}\mathbf{Z}_t
\]
such that \(\mathbf{Z}_t\) is segmented into lower-dimensional subseries that are uncorrelated both contemporaneously and serially. The central matrices are
\[
\mathbf{W}_Y=\mathbf{I}_p+\sum_{k=1}^{k_0}\boldsymbol{\Sigma}_Y(k)\boldsymbol{\Sigma}_Y(k)^T,\qquad
\mathbf{W}_X=\mathbf{I}_p+\sum_{k=1}^{k_0}\boldsymbol{\Sigma}_X(k)\boldsymbol{\Sigma}_X(k)^T,
\]
with
\[
\mathbf{W}_Y=\mathbf{A}^T\mathbf{W}_X\mathbf{A}.
\]
The procedure reduces to eigenanalysis of a positive definite matrix followed by a permutation step based on maximum cross-correlations or FDR-based multiple testing. Unlike PCA for independent data, the required linear transformation may not exist; when it does not, the method provides an approximate segmentation [1410.2323].

These two strands—three-term reconstruction and lag-structured segmentation—extend 2ndPCA away from ordinary variance maximization toward richer second-stage linear estimation problems.

## 5. Interpretation, component selection, and second-level analysis

A further meaning of 2ndPCA concerns how non-leading components are interpreted and selected. In equation discovery, PCA is used not primarily for the first few principal components but for the last or one of the last components, since a small eigenvalue implies that the corresponding linear combination
\[
z_k=Xv_k
\]
is nearly constant across cases. The paper argues that such constant-combinations may reflect underlying laws of nature. In the Kepler example, the law appears in a zero-eigenvalue component rather than in a leading PC; in the meteorology example, the relevant law is associated with eigenvector \(\#150\), chosen by combining a relatively small eigenvalue with low spatial variability of the loadings for \(T_v\) and \(H\) [2401.04797].

This use of PCA broadens the meaning of “second” in 2ndPCA. It may denote PC2, but it may also denote any non-leading component whose scores are nearly constant and whose loadings are physically interpretable. The paper explicitly states that the law may not correspond to the last principal component, but rather to one of the last, so a selection criterion is required [2401.04797].

The “new interpretation” paper proposes a different second-level analysis. It treats the correlation between a variable and a principal component as a cosine,
\[
\rho_{X,Y}=\cos(x,y),
\]
and the determination coefficient as
\[
R^2_{X,Y}=\cos^2(x,y).
\]
With standardized variables, the squared correlations \(R^2_{j,i}\) quantify how much of the variance of variable \(j\) is explained by principal component \(i\). The paper proposes selecting PCs not only by global explained variance but also by ensuring that, for each variable \(j\),
\[
\sum_{i=1}^k R^2_{j,i}
\]
exceeds an acceptable threshold. It also proposes clustering primary variables by similarity to principal components using these determination coefficients [1711.10420].

This suggests a secondary analytic layer over PCA itself: one examines not only the component directions but also the geometry of variable–component relations, variable-wise reconstruction quality, and vertical clustering of variables. In that sense, 2ndPCA can mean a second analysis of PCA structure rather than a second component alone.

## 6. Applications, reported results, and limitations

The practical uses associated with 2ndPCA depend on which formulation is intended. In face recognition and image reconstruction, R2DPCA and related 2DPCA variants operate directly on image matrices. The reported recognition rates are as follows [1912.09970]:

| Dataset | Method | Reported rate |
|---|---|---|
| Faces95 | 2DPCA | 0.8729 |
| Faces95 | G2DPCA | 0.9451 |
| Faces95 | R2DPCA | 0.9493 |
| Color FERET | 2DPCA | 0.5982 |
| Color FERET | G2DPCA | 0.6918 |
| Color FERET | R2DPCA | 0.7085 |

The same study states that, in MNIST digit recognition, R2DPCA outperforms CNNs, DBNs, and DNNs under the specific architectures and training regimes used in the experiments, and that for \(k=1\) R2DPCA coincides with G2DPCA [1912.09970].

For order-two MPCA on the Olivetti Faces dataset, the reported mean test reconstruction error is \(1.1346\times 10^5\) for MPCA versus \(8.6455\times 10^5\) for conventional PCA, with standard deviations \(9.6398\times 10^2\) and \(120.39\times 10^2\), respectively. The paper attributes this to the more parsimonious, module-oriented basis induced by bilinear structure [1104.5281].

In time-series analysis, TS-PCA serves as an initial step for segmentation, dimension reduction, and forecasting of high-dimensional linear dynamical structures. The paper reports lower mean and standard deviation of MSEs than VAR and restricted VAR in several real-data examples, and extends the idea to segment multiple volatility processes after GARCH modeling [1410.2323].

Several limitations recur across the literature. Classical PCA assumes linearity, orthogonality, and that large variances reflect important structure; the tutorial explicitly notes that the assumption that “large variances have important structure” is strong and sometimes incorrect. Equation discovery from PCA requires appropriate transformations, such as log transforms for multiplicative laws, and may be confounded by degeneracy of small eigenvalues, noise, or high dimensionality. Matrix and tensor formulations preserve structure but rely on row/column or multilinear assumptions. TS-PCA does not guarantee that an exact segmentation exists. Extended PCA requires auxiliary injection vectors and additional covariance estimation. These limitations explain why 2ndPCA remains a family resemblance term rather than a single settled method [1404.1100] [2401.04797] [1410.2323] [2111.03040].

Taken together, the literature presents 2ndPCA as a technically plural concept. In its narrowest meaning it is the second principal component of classical PCA. In broader usage it denotes two-dimensional or second-order PCA on matrices and tensors, extended PCA with an added transform term, lag-structured PCA for stationary vector time series, or second-level analyses of variable–component relations and low-variance law-bearing directions. The continuity across these meanings lies in the attempt to preserve PCA’s eigen-analytic core while extending what counts as the relevant “second” structure.

Source: https://www.emergentmind.com/topics/second-principal-component-analysis-2ndpca