HO-GSVD: Advanced Matrix Factorization

• HO-GSVD is a matrix factorization technique that generalizes GSVD to N≥2 matrices, identifying shared, isolated, and intermediate subspaces across heterogeneous datasets.
• It employs a unified basis V via generalized eigenproblems and introduces regularization to handle rank-deficient matrices for stable decomposition.
• HO-GSVD supports practical applications such as multi-task learning and model merging by quantifying subspace alignments and enabling robust expert selection.

The Higher-Order Generalized Singular Value Decomposition (HO-GSVD) is a matrix factorization technique that extends the classical Generalized SVD (GSVD) to $N\geq2$ data matrices, enabling the identification of shared, unique, and weighted subspaces across multiple large-scale datasets, including those with differing row dimensions and possible rank deficiency. In its standard form, HO-GSVD factors matrices $A_i\in\mathbb{R}^{m_i\times n}$ as $A_i=U_i\Sigma_i V^\text{T}$, with $V$ as a common basis, $U_i$ with orthonormal columns, and $\Sigma_i$ diagonal. Extensions of HO-GSVD have established robust algorithms and interpretations for rank-deficient matrices, facilitating applications in domains such as multi-task learning, bioinformatics, neuroscience, and model merging (Kempf et al., 2021, Skorobogat et al., 19 Jun 2025).

1. Mathematical Foundations and Standard HO-GSVD

The standard HO-GSVD generalizes the classical GSVD from two matrices to $N$ matrices $A_i\in\mathbb{R}^{m_i\times n}$ under the full column-rank condition for each $A_i$. The factorization is expressed as:

$$A_i = U_i\,\Sigma_i\,V^\text{T}, \quad i=1,\ldots,N,$$

where $V\in\mathbb{R}^{n\times n}$ (generally non-orthogonal) is shared across all decompositions, $U_i\in\mathbb{R}^{m_i\times n}$ have orthonormal columns, and $\Sigma_i$ are diagonal matrices of generalized singular values. The right singular vectors $V$ solve an eigenproblem based on the generalized arithmetic mean of the Gram matrices $D_{i,0}=A_i^\text{T}A_i$, resulting in the matrix $S^0$:

$$S^0 = \frac{1}{N(N-1)}\sum_{i<j}(D_{i,0}D_{j,0} + D_{j,0}D_{i,0}),$$

which is diagonalized as $S^0\,V = V\,\Lambda$ with $\Lambda$ diagonal (Kempf et al., 2021). This structure enables subspace intersections to be analyzed jointly across all matrices with a single global basis.

2. Extension to Rank-Deficient Matrices

When the rank condition $\text{rank}\,A_i=n$ fails for some $i$, the standard HO-GSVD construction is invalid due to singular $D_{i,0}$. The rank-deficient extension introduces a regularization term:

$$D_{i,\gamma} = A_i^\text{T}A_i + \gamma\,A^\text{T}A,$$

where $A$ is the vertically stacked matrix of all $A_i$ and $\gamma>0$ ensures invertibility of each $D_{i,\gamma}$. The generalized mean matrix becomes:

$$S_\gamma = \frac{1}{N(N-1)}\sum_{i<j}(D_{i,\gamma}D_{j,\gamma}^{-1} + D_{j,\gamma}D_{i,\gamma}^{-1}),$$

which can be diagonalized for stable factorization when the stacked $A$ is full rank, even if some $A_i$ are rank-deficient (Kempf et al., 2021).

This construction is essential in modern settings such as model merging for experts with varying support or rank, e.g., weight-differential matrices $\Delta_i$ in neural model ensembles (Skorobogat et al., 19 Jun 2025).

3. Subspace Structure: Common, Isolated, and Weighted Subspaces

HO-GSVD and its higher-order Cosine-Sine Decomposition (HO-CSD) counterpart enable a rigorous distinction between types of subspaces:

• Common subspaces are directions in $\mathbb{R}^n$ that are equally represented across all $A_i$, associated with minimal eigenvalues ($\tau_{\min}$ for $T_\gamma$, $\sigma_{\min}$ for $S_\gamma$).
• Isolated subspaces correspond to directions unique to a single $A_i$, associated with maximal eigenvalues ($\tau_{\max}$, $\sigma_{\max}$).
• Intermediate subspaces (weighted) are represented across a subset or variably weighted across all $A_i$.

For rank-deficient settings, HO-GSVD identifies these subspaces robustly, with $V$ partitioned so that columns associated with the common subspace can be isolated, and block structures in the factorization directly reflect the underlying subspace assignments (Kempf et al., 2021).

4. Algorithmic Workflow and Computational Complexity

The canonical algorithm for HO-GSVD proceeds as follows:

1. Stack and QR: Form $A=[A_1;\ldots;A_N]=Q\,R$, partition $Q$ into $Q_i$ blocks.
2. Regularization: For each $A_i$, build $D_{i,\gamma}=A_i^\text{T}A_i+\gamma A^\text{T}A$.
3. Generalized Mean: Construct $S_\gamma$ and diagonalize via eigendecomposition to obtain $V$.
4. Recovery: Calculate $U_i$ and $\Sigma_i$ for each $i$ using the obtained $V$, normalize by generalized Procrustes procedures if required.
5. Subspace Assignment: Identify indices corresponding to common and isolated subspaces by examining spectrum clustering ($\tau_k\approx\tau_{\min}$ for common; $\tau_k\approx\tau_{\max}$ for isolated).

The computational complexity is dominated by $O(Mn^2 + Nn^3)$, where $M=\sum_i m_i$, with eigendecomposition in $\mathbb{R}^{n\times n}$ and per-expert matrix operations (Kempf et al., 2021, Skorobogat et al., 19 Jun 2025).

5. Applications: Model Merging and Task Arithmetic

HO-GSVD has critical applications in subspace-boosted model merging, where $N$ task-vector matrices $\Delta_i$ are decomposed jointly. The unified $V$ basis captures global task directions, while per-expert $\Sigma_i$ characterize the "loading" of each task along those directions. Subspace structure enables:

• Quantification of task similarity: The alignment of subspaces is computed via the ratios $\sigma_{i,k}/\sigma_{j,k}$, or aggregated into an $N\times N$ alignment matrix:

$$\mathbf{A}_{ij} = \frac{1}{L}\sum_{l=1}^L\frac{1}{M_l}\sum_{p=1}^{M_l} \left|\log\left(\frac{\sigma^{(l)}_{i,p}+\epsilon}{\sigma^{(l)}_{j,p}+\epsilon}\right)\right|$$

Small entries in $\mathbf{A}$ indicate high interference (shared subspaces), large entries indicate subspace disjointness (less interference), guiding expert selection (Skorobogat et al., 19 Jun 2025).

• Subspace boosting: The method mitigates rank collapse during merging by detecting and augmenting collapsed (unique) subspaces, improving merged-model expressivity.
• Interpretability: Directions $v_k$ and their coefficients $\sigma_{i,k}$ precisely describe which features are shared or exclusive among tasks or data sources.

6. Connections to Related Decompositions

HO-GSVD collapses to standard GSVD and SVD in the $N=2$ case and full-rank setting; when $N>2$ and all $A_i$ are full rank, the result aligns with the original HO-GSVD from Ponnapalli et al. The HO-CSD provides an alternative characterization, especially when matrices are nearly orthogonal, via

$Q_i = U_i \Sigma_i Z^T,$

with $V = R^{-1}Z$ connecting HO-GSVD and HO-CSD representations (Kempf et al., 2021).

7. Advantages, Limitations, and Numerical Considerations

HO-GSVD robustly supports identification of both common and unique subspaces in heterogeneous, possibly rank-deficient data, and regularization (via $\gamma$ or $\pi$) ensures invertibility and numerical stability. Empirical studies (e.g., on CIFAR-10 subsets) confirm the ability to separate class-unique directions from shared ones (Kempf et al., 2021). In model merging, HO-GSVD stabilizes task-vector spectra and enables principled expert selection, where naïve GSVD or SVD approaches are inadequate for $N>2$ or rank-deficient cases (Skorobogat et al., 19 Jun 2025).

Limitations include the requirement that the stacked matrix $A$ be full rank and the numerical delicacy in tuning the regularization parameter $\gamma$ or $\pi$, especially as the separation between subspace spectra (e.g., $\tau_{\max}-\tau_{\min}$) may shrink for large regularization, complicating subspace assignment.

Table: HO-GSVD Key Concepts

Concept	Mathematical Object	Interpretation
Common subspace	$v_k$ with $\sigma_{i,k}\approx\sigma_{j,k}\ \forall i, j$	Shared direction, equally loaded by all $A_i$
Isolated subspace	$v_k$ with $\sigma_{j,k}=1$, $\sigma_{i\neq j,k}=0$	Unique direction, exclusive to one $A_i$
Intermediate subspaces	$v_k$ with variable $\sigma_{i,k}$	Shared but differentially loaded directions

The HO-GSVD is thus a principled generalization of the GSVD for $N>2$ matrices, enabling fine-grained analysis and application across rank-deficient and heterogeneous datasets, with increasing utility in modern data fusion, representation learning, and large-scale model merging methodologies (Kempf et al., 2021, Skorobogat et al., 19 Jun 2025).