Unpaired Canonical Correlation Analysis

Abstract: Canonical Correlation Analysis (CCA) is a fundamental method for multiview shared space learning. However, its strict reliance on paired data poses a significant limitation, as such data is often difficult to obtain or entirely unavailable. In this paper, we present Unpaired CCA (UCCA), a novel method that learns linear projections to maximize the correlation of the true underlying pairing without access to any paired samples during training. We first establish theoretical results connecting the Quadratic Assignment Problem (QAP) to CCA. Leveraging these theoretical insights, we derive a practical method to maximize correlation exclusively from unpaired data. To the best of our knowledge, UCCA is the first approach to learn maximally correlated projections in a strictly unpaired setting. We validate UCCA on real-world multi-modal datasets, demonstrating that it significantly outperforms recent unpaired alignment baselines in recovering the underlying true correlation. This work fills a critical gap between traditional statistical multiview learning and the growing field of unpaired data learning.

• UCCA introduces an anchor-based approximation to the quadratically assigned problem for recovering canonical correlations ok pairwise similarly random but independent vector-set pairing.
• UCCA's key result links an orthogonally invariant CCO to reconstructing pseudo pairview QAP.
• The method outperforms baselines across six real-world datasets, achieving close to paired PCA.

Canonical Correlation Analysis (CCA) is conventionally defined on paired observations: each row in one view must correspond to the same underlying instance in the other. “Unpaired Canonical Correlation Analysis” (2610.09530) studies the stricter setting in which two views are observed independently and no correspondence is available during training. The paper proposes Unpaired CCA (UCCA), a method that estimates a structural correspondence between the views and then applies ordinary CCA to the resulting pseudo-pairs. Its central technical claim is that the correspondence objective relevant to CCA can be reduced, under stated assumptions, to a linear-kernel Quadratic Assignment Problem (QAP).

Problem formulation and contribution

Let $X$ and $Y$ denote centered and whitened data matrices from two views. Under paired CCA, the objective is to find orthogonal projection matrices whose projected representations maximize cross-view total correlation. In the unpaired setting, the rows of $Y$ are related to those of $X$ through an unknown permutation $P^*$, but this permutation is not observed. The target is therefore not merely distributional alignment; it is recovery of projections that maximize correlation under the latent ground-truth pairing.

This distinction separates UCCA from many existing unpaired alignment methods. Domain adaptation and manifold-alignment methods can reduce discrepancies between marginal distributions, while shared-component methods can identify common structure, but neither objective necessarily selects the pairing that maximizes canonical correlation. UCCA instead introduces a permutation-dependent correlation criterion and uses it to construct pseudo-pairs before fitting CCA.

The paper’s contributions are threefold. First, it defines a maximum-correlation proxy pairing that evaluates candidate permutations jointly with orthogonal transformations. Second, it proves that a relaxed version of this proxy is equivalent to a QAP involving the within-view Gram matrices. Third, it implements the resulting method with a computationally bounded anchor-based procedure and evaluates it on six real-world view configurations spanning handwritten digits, single-cell multi-omics, and image-text data.

From unpaired CCA to maximum-correlation pairing

A naive strategy would select the permutation maximizing the raw total correlation,

$\operatorname{tr}(X^\top P Y).$

The paper shows why this is inadequate. This objective is sensitive to arbitrary rotations of either feature space, whereas CCA is explicitly invariant to orthogonal transformations. Consequently, a permutation that appears favorable in the original coordinates may be irrelevant after the view-specific coordinate systems are rotated.

To address this issue, the paper defines a generalized Maximum Correlation Pairing (MCP) that optimizes over both permutations and transformation classes. For the orthogonal group $O(d)$, the relevant proxy is $\operatorname{MCP}_{O(d)}$. The inner optimization over orthogonal transformations converts the cross-correlation matrix

$M_P = X^\top P Y$

into its nuclear norm:

$$\operatorname{MCP}_{O(d)}(X,Y) = \arg\max_{P}\|X^\top P Y\|_*.$$

Thus, the desired permutation is the one whose cross-view correlation matrix has the largest sum of singular values. This formulation directly reflects the fact that CCA aggregates canonical correlations rather than relying on a fixed coordinate-wise alignment.

The conceptual role of this construction is illustrated by the paper’s comparison between ordinary MCP and its orthogonally invariant variant. Ordinary MCP can fail after an input rotation, whereas $Y$0 evaluates each candidate pairing after optimizing the relevant orthogonal alignment.

Figure 1: Ordinary MCP is rotation-sensitive, whereas $Y$1 evaluates pairings jointly with orthogonal alignment.

The paper does not require exact recovery of the latent permutation as an independent objective. Instead, it assumes that a proxy permutation maximizing the CCA-relevant criterion is sufficient to recover the desired projection subspaces. This is a weaker requirement than reconstructing every individual correspondence, although it still presupposes that the shared geometry contains enough information to distinguish the correct alignment.

QAP equivalence

Directly optimizing over permutations and orthogonal matrices is computationally difficult. The paper therefore relaxes the orthogonal transformation class to a scaled Frobenius sphere. Under this relaxation, the objective becomes

$Y$2

Because the Frobenius norm is invariant to orthogonal transformations, its squared form can be written as

$Y$3

This is exactly the Koopmans–Beckmann QAP with similarity matrices $Y$4 and $Y$5:

$Y$6

This theorem is the paper’s main computational reduction. The unpaired CCA problem is not solved by directly learning a cross-view mapping. Rather, it is converted into structural graph matching between the two views’ linear kernels. The QAP remains NP-hard in general, but approximate solvers such as 2-Opt and FAQ can be applied after reducing the problem to a small set of representative anchors.

The paper further establishes a relationship between the nuclear-norm objective associated with $Y$7 and the Frobenius-norm objective associated with the relaxation. Let $Y$8 denote the singular-value vector of $Y$9. The nuclear objective sums the singular values, while the squared Frobenius objective sums their squares. The paper assumes that the singular-value vector under the optimal orthogonal-invariant pairing weakly majorizes that under every competing permutation. Under this weak-majorization assumption, monotonicity and convexity imply that the same permutation maximizes both objectives:

$Y$0

The assumption is established exactly when one view is an orthogonal image of the other, and is argued to hold asymptotically for random permutations under the paper’s whitening and fixed-dimensionality conditions. Empirical tests on the evaluated datasets show synchronized decay of both objectives as permutations are increasingly shuffled.

Figure 2: Partial sums of singular values support the weak-majorization assumption, while nuclear- and Frobenius-based objectives identify the same alignment.

This result is important but conditional. The QAP equivalence itself is exact for the Frobenius relaxation; equivalence with the original orthogonally invariant objective depends on the weak-majorization condition. The empirical evidence supports the condition on the selected benchmarks, but it does not establish that it holds for arbitrary multimodal datasets, especially when the views contain substantial view-specific variation, repeated structures, severe noise, or non-isometric geometry.

The UCCA algorithm

UCCA operationalizes the theoretical construction through anchor-based matching. First, each view is independently PCA-reduced and whitened. The default preprocessing uses ten PCA components. K-Means is then run separately on each view to obtain representative centroids. These centroids form reduced geometric skeletons on which the QAP is solved.

For each clustering repetition, UCCA constructs the anchor Gram matrices, solves a linear-kernel QAP to obtain a permutation of the anchors, and treats the matched centroids as pseudo-pairs. The process is repeated, with 50 QAP restarts per clustering instance and 500 clustering repetitions in the principal configuration. All matched anchors are concatenated, and standard paired CCA is finally fitted to this aggregate pseudo-paired dataset.

Figure 3: UCCA performs PCA whitening, independent anchor extraction, QAP matching, and final CCA on concatenated pseudo-pairs.

The anchor approximation is central to scalability. Solving QAP over all $Y$1 observations would be impractical, whereas matching $Y$2 anchors reduces the dominant QAP component to approximately $Y$3 for an $Y$4-iteration 2-Opt solver. The overall complexity is linear in the number of samples when $Y$5, the PCA dimension, the number of K-Means iterations, and the number of clustering repetitions are treated as fixed:

$Y$6

The method therefore trades exact point-level matching for repeated approximate matching of low-cardinality geometric summaries. Increasing the anchor count improves manifold resolution but increases QAP cost.

Experimental protocol

The experimental design withholds correspondence during training while retaining a paired test set for evaluation. For each benchmark, the paired observations are divided so that one disjoint subset contributes only the first view to training and another contributes only the second view. Consequently, no sample’s paired counterpart is available to the algorithm during training.

The evaluation includes three handwritten-digit view pairs—PC-KL, PC-PA, and KL-PA—along with SNARE ATAC-RNA, Flickr image-text, and COCO image-text. Total correlation is evaluated against the hidden test pairing, and cross-view classification measures whether labels inferred from one learned representation transfer to the other.

The comparison methods are not direct unpaired CCA competitors, because the paper argues that no previous method explicitly optimizes canonical correlation under a strictly unpaired regime. Instead, UCCA is compared with SCA, UCA, J-MDS, and SCOT variants. For methods that output point-cloud alignments but not functional projections, the authors fit CCA to their inferred pseudo-pairs.

Correlation recovery

UCCA is the only method that consistently obtains high total correlation across all six configurations. Its mean total-correlation scores are:

Dataset configuration	UCCA	Paired CCA
PC-KL	$Y$7	$Y$8
PC-PA	$Y$9	$X$0
KL-PA	$X$1	$X$2
SNARE	$X$3	$X$4
Flickr	$X$5	$X$6
COCO	$X$7	$X$8

Figure 4: UCCA substantially exceeds unpaired alignment baselines and approaches paired CCA on most configurations.

The results are strongest on KL-PA and SNARE, where UCCA reaches $X$9 and $P^*$0, respectively, compared with paired-CCA references of $P^*$1 and $P^*$2. The gap is larger on Flickr, where UCCA reaches $P^*$3 against the paired upper bound of $P^*$4. This contrast indicates that the method’s success depends on the extent to which the two views preserve a recoverable common geometry. Image-text data can contain substantial semantic ambiguity and modality-specific variation, making a purely geometric correspondence less reliable than in more tightly coupled views.

The particularly strong improvement over the adapted J-MDS-CCA and SCOT-CCA baselines isolates the contribution of the QAP objective. On PC-PA, for example, UCCA obtains $P^*$5, while the corresponding baselines achieve $P^*$6, $P^*$7, and $P^*$8 for J-MDS-CCA, SCOTv1-CCA, and SCOTv2-CCA, respectively. The comparison supports the paper’s claim that structural alignment alone is insufficient unless its objective is specifically related to the correlation criterion used by CCA.

The synthetic exhaustive experiment provides an additional diagnostic. For a ten-sample problem, the authors evaluate all approximately $P^*$9 million permutations and report a positive relationship between the QAP objective and the CCA total correlation. The QAP maximizer is nearly coincident with the permutation maximizing the CCA objective. This is a useful validation of the proposed surrogate, although it is necessarily limited to a small synthetic configuration.

Downstream cross-view classification

UCCA also improves transfer across views. A 5-nearest-neighbor classifier is trained on one learned embedding and evaluated on the other. The results are:

Dataset configuration	UCCA accuracy	Paired CCA accuracy
PC-KL	$\operatorname{tr}(X^\top P Y).$0	$\operatorname{tr}(X^\top P Y).$1
PC-PA	$\operatorname{tr}(X^\top P Y).$2	$\operatorname{tr}(X^\top P Y).$3
KL-PA	$\operatorname{tr}(X^\top P Y).$4	$\operatorname{tr}(X^\top P Y).$5
SNARE	$\operatorname{tr}(X^\top P Y).$6	$\operatorname{tr}(X^\top P Y).$7
COCO	$\operatorname{tr}(X^\top P Y).$8	$\operatorname{tr}(X^\top P Y).$9

UCCA is best among the unpaired methods on four of the five reported datasets and matches the paired CCA result on COCO in the primary transfer direction. Its SNARE accuracy of $O(d)$0 is especially close to the paired reference of $O(d)$1, while the PC-KL and PC-PA results remain substantially below their paired counterparts.

These results also clarify why downstream performance need not track total correlation exactly. UCCA can align class-level or manifold-level structure without reproducing every instance-level pairing. The paper’s embedding visualizations show that paired CCA produces tighter pointwise overlays, whereas UCCA preserves cross-view class clusters with greater within-cluster variation.

Figure 5: Paired CCA produces tighter instance-level overlap, while UCCA preserves cross-view cluster structure despite lacking paired supervision.

The nonlinear UCA baseline performs substantially worse despite its greater function-class capacity. UCCA obtains total correlations of $O(d)$2, $O(d)$3, $O(d)$4, $O(d)$5, $O(d)$6, and $O(d)$7 across the six configurations, whereas nonlinear UCA obtains $O(d)$8, $O(d)$9, $\operatorname{MCP}_{O(d)}$0, $\operatorname{MCP}_{O(d)}$1, $\operatorname{MCP}_{O(d)}$2, and $\operatorname{MCP}_{O(d)}$3. This result supports the paper’s specific methodological claim that explicit geometric matching can be more effective than increasing representation flexibility when the principal difficulty is unknown correspondence.

Ablations and computational behavior

The anchor studies show that K-Means centroids are substantially more stable than random samples. Across the six configurations, K-Means generally improves total correlation over random anchor selection. For example, on COCO, K-Means obtains $\operatorname{MCP}_{O(d)}$4, compared with $\operatorname{MCP}_{O(d)}$5 for random sampling. A paired K-Means oracle, which uses hidden correspondences only to match the centroids, approaches the paired CCA upper bound on most datasets. This indicates that anchor extraction is not the fundamental bottleneck; matching independently generated anchors is.

Performance increases sharply with the number of anchors and then stabilizes. The default choice of 20 anchors is presented as a compromise between geometric fidelity and QAP runtime. GMM-based anchors yield comparable results, suggesting that UCCA is not inherently tied to K-Means, although the experiments do not establish robustness across a broad range of clustering or density-estimation procedures.

Figure 6: Increasing the number of anchors improves structural alignment until computationally diminishing returns are reached.

UCCA is also robust to moderate data imbalance and benefits from additional unpaired samples. Its performance improves as the amount of available unpaired training data increases from 50% to 100%. This finding is significant because the method does not require even a small paired subset: additional independent samples provide more accurate estimates of each view’s geometry and therefore strengthen the QAP alignment signal.

Figure 7: UCCA remains effective under view imbalance and improves as the quantity of unpaired data increases.

Although QAP is NP-hard, the anchor restriction and approximate solvers yield practical runtimes. The paper reports execution times of several minutes per dataset, with UCCA often exceeding baseline performance even at low numbers of clustering repetitions. Increasing repetitions reduces variance and moves performance toward the paired-CCA reference.

The choice between 2-Opt and FAQ has little effect when both converge. However, FAQ fails to converge in two tested instances, producing near-zero outcomes, whereas 2-Opt executes consistently. The selection of 2-Opt is therefore motivated by robustness of execution rather than superior objective quality.

Figure 8: UCCA exceeds baseline performance early in the runtime budget and improves with additional clustering repetitions.

Direct alternating optimization of $\operatorname{MCP}_{O(d)}$6 performs markedly worse than the QAP relaxation. On PC-PA, UCCA with QAP obtains $\operatorname{MCP}_{O(d)}$7, compared with $\operatorname{MCP}_{O(d)}$8 for an EM-style UCCA variant, $\operatorname{MCP}_{O(d)}$9 for ACCA, and $M_P = X^\top P Y$0 for ACCA with anchors. This experiment suggests that the QAP formulation offers a more favorable optimization landscape than alternating between CCA and assignment updates.

Limitations and open questions

The paper’s principal theoretical limitation is the weak-majorization assumption needed to identify $M_P = X^\top P Y$1 with $M_P = X^\top P Y$2. The paper provides exact support for an isometric relationship between views and an asymptotic argument for random permutations, as well as empirical validation on six configurations. Nevertheless, the assumption is not derived from a general multimodal generative model, and its behavior under systematic domain shift, non-isometric transformations, multimodal clusters, or unequal sample supports remains open.

The QAP solver is approximate, so the implemented method does not generally solve the theoretical QAP exactly. Anchor matching introduces an additional approximation, and the final CCA is fitted to pseudo-pairs rather than observed correspondences. The empirical gains are therefore jointly attributable to the objective, anchor representation, restart strategy, and clustering repetitions. The paper’s ablations partially separate these factors but do not provide a formal error bound linking anchor approximation and QAP suboptimality to the final canonical correlations.

The evaluation also relies on benchmarks with known hidden pairings and paired test sets. This is appropriate for measuring recovery of true correlation, but it does not evaluate settings in which no ground-truth correspondence exists even for assessment. The benchmark scale is moderate, with the largest dataset containing 8,000 samples, and the method is restricted to two views. Extending the formulation to more than two modalities is nontrivial because pairwise QAP solutions may be mutually inconsistent and the combinatorial difficulty increases with the number of views.

Finally, UCCA remains below paired CCA on several configurations, most notably Flickr. The results therefore support recovery of useful shared structure rather than equivalence to supervised CCA in general. A specific unresolved question is whether stronger invariants, probabilistic correspondence models, or multi-view consistency constraints can reduce this gap without introducing paired supervision.

Conclusion

“Unpaired Canonical Correlation Analysis” (2610.09530) formulates strictly unpaired multiview representation learning as a correspondence problem governed by the CCA objective. Its main theoretical result reduces a Frobenius-relaxed maximum-correlation pairing to a linear-kernel QAP and, under a weak-majorization assumption, identifies this relaxation with the orthogonally invariant CCA proxy. UCCA implements the reduction through repeated anchor extraction, approximate QAP matching, and CCA on pseudo-pairs.

Across six real-world configurations, UCCA consistently recovers substantially more hidden correlation than existing unpaired alignment baselines, achieves total correlation close to paired CCA on several datasets, and produces strong cross-view classification transfer. The method’s claims are bounded by its geometric assumptions, approximate combinatorial optimization, anchor dependence, and restriction to two views. Within that scope, the paper provides a technically coherent connection between CCA and graph matching and demonstrates that unpaired data can support canonical-correlation objectives when the views retain sufficiently identifiable shared geometry.