---
title: 'HO-GSVD: Advanced Matrix Factorization'
url: https://www.emergentmind.com/topics/higher-order-generalized-svd-ho-gsvd
type: topic
---

# HO-GSVD: Advanced Matrix Factorization

The Higher-Order Generalized Singular Value Decomposition (HO-GSVD) is a matrix factorization technique that extends the classical Generalized SVD (GSVD) to $N\geq2$ data matrices, enabling the identification of shared, unique, and weighted subspaces across multiple large-scale datasets, including those with differing row dimensions and possible rank deficiency. In its standard form, HO-GSVD factors matrices $A_i\in\mathbb{R}^{m_i\times n}$ as $A_i=U_i\Sigma_i V^\text{T}$, with $V$ as a common basis, $U_i$ with orthonormal columns, and $\Sigma_i$ diagonal. Extensions of HO-GSVD have established robust algorithms and interpretations for rank-deficient matrices, facilitating applications in domains such as multi-task learning, bioinformatics, neuroscience, and model merging [2102.09822][2506.16506].

## 1. Mathematical Foundations and Standard HO-GSVD

The standard HO-GSVD generalizes the classical GSVD from two matrices to $N$ matrices $A_i\in\mathbb{R}^{m_i\times n}$ under the full column-rank condition for each $A_i$. The factorization is expressed as:
$$
A_i = U_i\,\Sigma_i\,V^\text{T}, \quad i=1,\ldots,N,
$$
where $V\in\mathbb{R}^{n\times n}$ (generally non-orthogonal) is shared across all decompositions, $U_i\in\mathbb{R}^{m_i\times n}$ have orthonormal columns, and $\Sigma_i$ are diagonal matrices of generalized singular values. The right singular vectors $V$ solve an eigenproblem based on the generalized arithmetic mean of the Gram matrices $D_{i,0}=A_i^\text{T}A_i$, resulting in the matrix $S^0$:
$$
S^0 = \frac{1}{N(N-1)}\sum_{i<j}(D_{i,0}D_{j,0} + D_{j,0}D_{i,0}),
$$
which is diagonalized as $S^0\,V = V\,\Lambda$ with $\Lambda$ diagonal [2102.09822]. This structure enables subspace intersections to be analyzed jointly across all matrices with a single global basis.

## 2. Extension to Rank-Deficient Matrices

When the rank condition $\text{rank}\,A_i=n$ fails for some $i$, the standard HO-GSVD construction is invalid due to singular $D_{i,0}$. The rank-deficient extension introduces a regularization term:
$$
D_{i,\gamma} = A_i^\text{T}A_i + \gamma\,A^\text{T}A,
$$
where $A$ is the vertically stacked matrix of all $A_i$ and $\gamma>0$ ensures invertibility of each $D_{i,\gamma}$. The generalized mean matrix becomes:
$$
S_\gamma = \frac{1}{N(N-1)}\sum_{i<j}(D_{i,\gamma}D_{j,\gamma}^{-1} + D_{j,\gamma}D_{i,\gamma}^{-1}),
$$
which can be diagonalized for stable factorization when the stacked $A$ is full rank, even if some $A_i$ are rank-deficient [2102.09822].  

This construction is essential in modern settings such as model merging for experts with varying support or rank, e.g., weight-differential matrices $\Delta_i$ in neural model ensembles [2506.16506].

## 3. Subspace Structure: Common, Isolated, and Weighted Subspaces

HO-GSVD and its higher-order Cosine-Sine Decomposition (HO-CSD) counterpart enable a rigorous distinction between types of subspaces:

- **Common subspaces** are directions in $\mathbb{R}^n$ that are equally represented across all $A_i$, associated with minimal eigenvalues ($\tau_{\min}$ for $T_\gamma$, $\sigma_{\min}$ for $S_\gamma$).
- **Isolated subspaces** correspond to directions unique to a single $A_i$, associated with maximal eigenvalues ($\tau_{\max}$, $\sigma_{\max}$).
- **Intermediate subspaces** (weighted) are represented across a subset or variably weighted across all $A_i$.

For rank-deficient settings, HO-GSVD identifies these subspaces robustly, with $V$ partitioned so that columns associated with the common subspace can be isolated, and block structures in the factorization directly reflect the underlying subspace assignments [2102.09822].

## 4. Algorithmic Workflow and Computational Complexity

The canonical algorithm for HO-GSVD proceeds as follows:
1. **Stack and QR:** Form $A=[A_1;\ldots;A_N]=Q\,R$, partition $Q$ into $Q_i$ blocks.
2. **Regularization:** For each $A_i$, build $D_{i,\gamma}=A_i^\text{T}A_i+\gamma A^\text{T}A$.
3. **Generalized Mean:** Construct $S_\gamma$ and diagonalize via eigendecomposition to obtain $V$.
4. **Recovery:** Calculate $U_i$ and $\Sigma_i$ for each $i$ using the obtained $V$, normalize by generalized Procrustes procedures if required.
5. **Subspace Assignment:** Identify indices corresponding to common and isolated subspaces by examining spectrum clustering ($\tau_k\approx\tau_{\min}$ for common; $\tau_k\approx\tau_{\max}$ for isolated).

The computational complexity is dominated by $O(Mn^2 + Nn^3)$, where $M=\sum_i m_i$, with eigendecomposition in $\mathbb{R}^{n\times n}$ and per-expert matrix operations [2102.09822][2506.16506].

## 5. Applications: Model Merging and Task Arithmetic

HO-GSVD has critical applications in subspace-boosted model merging, where $N$ task-vector matrices $\Delta_i$ are decomposed jointly. The unified $V$ basis captures global task directions, while per-expert $\Sigma_i$ characterize the "loading" of each task along those directions. Subspace structure enables:

- **Quantification of task similarity:** The alignment of subspaces is computed via the ratios $\sigma_{i,k}/\sigma_{j,k}$, or aggregated into an $N\times N$ alignment matrix:
  $$
  \mathbf{A}_{ij} = \frac{1}{L}\sum_{l=1}^L\frac{1}{M_l}\sum_{p=1}^{M_l} \left|\log\left(\frac{\sigma^{(l)}_{i,p}+\epsilon}{\sigma^{(l)}_{j,p}+\epsilon}\right)\right|
  $$
  Small entries in $\mathbf{A}$ indicate high interference (shared subspaces), large entries indicate subspace disjointness (less interference), guiding expert selection [2506.16506].
- **Subspace boosting:** The method mitigates rank collapse during merging by detecting and augmenting collapsed (unique) subspaces, improving merged-model expressivity.
- **Interpretability:** Directions $v_k$ and their coefficients $\sigma_{i,k}$ precisely describe which features are shared or exclusive among tasks or data sources.

## 6. Connections to Related Decompositions

HO-GSVD collapses to standard GSVD and SVD in the $N=2$ case and full-rank setting; when $N>2$ and all $A_i$ are full rank, the result aligns with the original HO-GSVD from Ponnapalli et al. The HO-CSD provides an alternative characterization, especially when matrices are nearly orthogonal, via
$$
Q_i = U_i \Sigma_i Z^T,
$$
with $V = R^{-1}Z$ connecting HO-GSVD and HO-CSD representations [2102.09822].

## 7. Advantages, Limitations, and Numerical Considerations

HO-GSVD robustly supports identification of both common and unique subspaces in heterogeneous, possibly rank-deficient data, and regularization (via $\gamma$ or $\pi$) ensures invertibility and numerical stability. Empirical studies (e.g., on CIFAR-10 subsets) confirm the ability to separate class-unique directions from shared ones [2102.09822]. In model merging, HO-GSVD stabilizes task-vector spectra and enables principled expert selection, where naïve GSVD or SVD approaches are inadequate for $N>2$ or rank-deficient cases [2506.16506].

Limitations include the requirement that the stacked matrix $A$ be full rank and the numerical delicacy in tuning the regularization parameter $\gamma$ or $\pi$, especially as the separation between subspace spectra (e.g., $\tau_{\max}-\tau_{\min}$) may shrink for large regularization, complicating subspace assignment.

## Table: HO-GSVD Key Concepts

| Concept                  | Mathematical Object                                   | Interpretation                                                        |
|--------------------------|------------------------------------------------------|-----------------------------------------------------------------------|
| Common subspace          | $v_k$ with $\sigma_{i,k}\approx\sigma_{j,k}\ \forall i, j$ | Shared direction, equally loaded by all $A_i$                         |
| Isolated subspace        | $v_k$ with $\sigma_{j,k}=1$, $\sigma_{i\neq j,k}=0$  | Unique direction, exclusive to one $A_i$                              |
| Intermediate subspaces   | $v_k$ with variable $\sigma_{i,k}$                   | Shared but differentially loaded directions                            |

The HO-GSVD is thus a principled generalization of the GSVD for $N>2$ matrices, enabling fine-grained analysis and application across rank-deficient and heterogeneous datasets, with increasing utility in modern data fusion, representation learning, and large-scale model merging methodologies [2102.09822][2506.16506].

Source: https://www.emergentmind.com/topics/higher-order-generalized-svd-ho-gsvd