Sparse Canonical Correlation Analysis

• Sparse Canonical Correlation Analysis is an extension of CCA that applies sparsity constraints, enabling interpretable feature selection in high-dimensional settings.
• It reformulates the problem into a combinatorial NP-hard subset selection, bridging closely related approaches like sparse PCA, sparse SVD, and regression.
• Advanced methods such as MISDP and MIQP provide near-optimal solutions with small suboptimality gaps and practical scalability for moderate dimensions.

Sparse Canonical Correlation Analysis (SCCA) is an extension of classical canonical correlation analysis that aims to identify maximally correlated linear projections between two sets of variables, while inducing exact or approximate sparsity in the canonical vectors for interpretable feature selection. SCCA addresses the principal limitations of CCA in high-dimensional settings—namely, the lack of interpretability due to dense canonical vectors and the non-invertibility of empirical covariance matrices when the number of variables exceeds the sample size. SCCA has become integral in genomics, neuroimaging, computational biology, and other domains requiring correlated structure discovery across large, heterogeneous data modalities (Li et al., 2023).

1. Classical CCA and Motivations for Sparse Extensions

Classical CCA, introduced by Hotelling (1935), seeks vectors $x\in\mathbb{R}^n$ and $y\in\mathbb{R}^m$ maximizing $x^\top A y$ under quadratic normalization $x^\top B x\leq 1$, $y^\top C y\leq 1$, with $B$ and $C$ the within-set covariance matrices and $A$ the cross-covariance block. The solution expresses $x^*,y^*$ via generalized inverses and the leading singular vectors of $M=\sqrt{B^\dagger}A\sqrt{C^\dagger}$, attaining optimal correlation $\sigma_{\max}(M)$ (Li et al., 2023).

In high-dimensional regimes ($n+m \gg$ sample size), $B$ or $C$ is singular and $x^*,y^*$ are typically dense, undermining interpretability and standard CCA's feasibility. SCCA imposes explicit cardinality or $\ell_1$-norm constraints to yield canonical vectors with interpretable, sparse support, making it possible to pinpoint the variables driving the cross-correlation in very large-scale, collinear, or $p, q > n$ scenarios (Li et al., 2023).

2. Exact Problem Formulation and NP-Hardness

The canonical form of SCCA is given by

$$\max_{x\in\mathbb{R}^n,\,y\in\mathbb{R}^m}\;\{\,x^\top A y : x^\top B x\le1,\,y^\top C y\le1,\,\|x\|_0\le s_1,\,\|y\|_0\le s_2\,\}$$

where $\|x\|_0$ counts nonzero entries and $s_1,s_2$ are sparsity levels. This admits an exact combinatorial formulation as a subset selection over supports $S_1\subseteq[n],\,|S_1|\le s_1$ and $S_2\subseteq[m],\,|S_2|\le s_2$: $$v^* = \max_{S_1,S_2} \sigma_{\max}\!\Bigl(\sqrt{(B_{S_1,S_1})^\dagger}A_{S_1,S_2}\sqrt{(C_{S_2,S_2})^\dagger}\Bigr)$$ This subset selection is NP-hard, generalizing three core problems:

• Sparse Principal Component Analysis (Sparse PCA): SCCA reduces to $\ell_0$-constrained PCA for $n=m$, $B=C=I$, and $A$ symmetric PSD.
• Sparse Singular Value Decomposition (Sparse SVD): SCCA becomes the sparse SVD for $B=C=I$.
• Subset selection in regression: When $A$ is rank one, SCCA splits into two separate sparse regression problems (Li et al., 2023).

Consequently, SCCA is a unifying, strictly harder generalization of these known NP-hard problems.

3. Mixed-Integer Semidefinite Programming (MISDP) Reformulation and Algorithms

A key methodological advance is the conversion of SCCA to a mixed-integer semidefinite program (MISDP). Introducing a lifted matrix $X$ with support indicators $z_i\in\{0,1\}$, SCCA can be recast as

$\max_{X,z}~\tr(\tilde A X) ~~\text{s.t.}~\tr(\tilde B X)\leq1,\;\tr(\tilde C X)\leq1,\;X_{ii}\leq M_{ii}z_i,\;\sum_{i=1}^n z_i\leq s_1,\;\sum_{i=n+1}^{n+m} z_i\leq s_2$

This enables a branch-and-cut algorithm leveraging analytical cuts derived from duality. For a fixed support, the dual minimizes over multipliers to yield closed-form Benders-style cuts

$$v \leq \sigma_{\max}(\cdots) + \lambda^*\sum_i M_{ii}z_i$$

which efficiently prune the search tree and enable the first exact branch-and-cut for SCCA (Li et al., 2023).

In low-rank regimes (rank$(B)=r$, rank$(C)=r$), supports can be enumerated in $O(n^{r-1}m^{r-1})$ time. For the rank-one cross-covariance case, the problem separates and becomes tractable for moderate dimensions via strong perspective relaxations (MIQP) (Li et al., 2023).

Branch-and-cut with analytical cuts scales exact solutions up to $n + m \leq 20$ variables, while continuous SDP relaxations provide tight upper bounds for larger problems ($n+m=240$). Greedy heuristics, local search, and SDP relaxations deliver solutions within 1% of the best-known upper bounds in under one second for moderate-sized problems (Li et al., 2023).

4. Statistical and Computational Complexities

The general SCCA problem is NP-hard by reduction to the classical subset selection, sparse PCA, and sparse SVD. Optimization-based formulations—especially those based on $\ell_0$ constraints—necessarily scale exponentially in sparsity level unless strong low-rank structure is present (Li et al., 2023).

In low-rank regimes:

• If $$s_1, s_2 \geq r = \operatorname{rank}(B), \operatorname{rank}(C)$$, sparsity constraints are inactive—standard CCA is polynomial-time solvable.
• For rank-one cross-covariance ($A=a\,b^\top$), the problem reduces to two independent, classical sparse-regression-type QP, which are themselves NP-hard.

Relaxations and approximation algorithms permit scalability to moderate (hundreds of variables) or rank-one cases (dimensions up to $500\times500$ via MIQP) at the cost of small suboptimality gaps ($\leq 11\%$) (Li et al., 2023).

5. Empirical Validation and Benchmarking

Numerical experiments reported for synthetic Gaussian data ($N=5000$, matrix sizes up to $120\times120$) show that:

• Greedy heuristics and local search recover SCCA solutions within $<1\%$ of SDP bounds in less than one second.
• SDP relaxations solve problems up to $n+m=240$ with up to $8\%$ duality gaps.
• The MISDP branch-and-cut solves exact SCCA up to $n+m=20$ (typically within minutes to hours).
• In the rank-one regime, perspective MIQP relaxations enable dimensions up to $500\times500$, with solution gaps not exceeding $11\%$.

These results validate that the developed formulations and algorithms deliver both tight bounds and practical scalability in key special cases (Li et al., 2023).

6. Structural Connections and Extensions

SCCA fully encompasses classical model classes:

• Sparse PCA is recovered as a diagonal SCCA with $A$ symmetric, $B=C=I$, $s_1=s_2$.
• Sparse SVD arises for $B=C=I$, with arbitrary rectangular $A$.
• Sparse regression when $A$ is rank one, leading to two independent best-subset selection problems.

Therefore, SCCA unifies diverse sparse matrix decomposition and selection tasks under a common convex-geometric and subset-selection framework (Li et al., 2023).

7. Practical Considerations and Guidance

SCCA's general nonconvexity necessitates careful algorithmic design. In practice:

• Tight continuous SDP relaxations provide valuable upper bounds for practical heuristics.
• Greedy and local search algorithms, when certified against SDP bounds, yield near-optimal solutions rapidly, allowing practical use for exploratory analysis and feature selection on moderate-scale problems.
• For rank-deficient and cross-covariance-rank-one situations commonly arising in genomics or imaging, strong MIQP relaxations offer both tractability and interpretability.

SCCA's interpretability comes at increased computational cost, especially for high target sparsities or lack of low-rank structure. Its effective use relies on leveraging problem structure, choosing relaxations or exact algorithms according to scale, and certifying heuristic solutions where feasible (Li et al., 2023).

---

Key References:

• Li, Bertsimas, Pauphilet, and Yi, "On Sparse Canonical Correlation Analysis" (Li et al., 2023).