---
title: Product of Projections in Multi-View Subspace Clustering
url: https://www.emergentmind.com/topics/product-of-projections-in-multi-view-subspace
type: topic
---

# Product of Projections in Multi-View Subspace Clustering

Deep Scalable Subspace Clustering (SDSNet) refers to a class of deep learning frameworks that address the scalability bottlenecks inherent in traditional and deep subspace clustering by replacing full affinity or self-expressive matrices with scalable, often low-rank, approximations within end-to-end architectures. Unlike classical approaches that require full-batch optimization of $n \times n$ affinity graphs for $n$ samples, SDSNet designs leverage mini-batch training, factorization schemes, and landmark-based approximations, enabling efficient clustering of tens of thousands of high-dimensional samples while retaining fidelity to subspace structure [1811.01045, 2512.21434].

## 1. Problem Formulation and Motivation

The objective of subspace clustering is to partition a dataset $X \in \mathbb{R}^{D \times n}$, whose $n$ columns are presumed to lie close to the union of $k$ (possibly non-linear) subspaces $\{\mathcal{S}_i\}_{i=1}^k$, into groups corresponding to those subspaces.  
Classical self-expressiveness models, notably Sparse Subspace Clustering (SSC) and Low-Rank Representation (LRR), express each data point as a linear combination of the others:
$$
X = X C + E,
$$
with $C \in \mathbb{R}^{n \times n}$ as the coefficient (affinity) matrix and $E$ as the residual. Deep Subspace Clustering Networks (DSC-Net) [1709.02508] extend this model into a non-linear latent space via an auto-encoder, then employ a self-expressive layer with $C$ as the learnable $n \times n$ matrix.

However, these approaches entail $\mathcal{O}(n^2)$ memory (for $C$) and $\mathcal{O}(n^3)$ or higher computational costs (for spectral clustering on $C$), making them impractical for large $n$ [1709.02508, 1811.01045, 2512.21434]. SDSNet was developed to eliminate this scaling bottleneck while preserving or improving clustering accuracy.

## 2. Architectural Innovations

SDSNet architectures share three critical design choices:

1. **Encoder-decoder backbone:** A convolutional auto-encoder parametrizes the mapping $f_{\theta_e} : x_j \mapsto z_j$ and $g_{\theta_d} : z_j \mapsto \widehat{x}_j$. The encoder produces a latent embedding $Z \in \mathbb{R}^{d \times n}$, with $d \ll D$.

2. **Scalable self-expressiveness module:**  
   - In [1811.01045], each point $z_j$ is assigned to one of $k$ learnable subspaces $S_i \in \mathbb{R}^{d \times p}$ (with $S_i^\top S_i = I_p$), avoiding the need for a global $n \times n$ affinity.
   - In [2512.21434], the coefficient matrix $C$ is replaced by a low-rank factorization $C \approx P P^\top$ with $P \in \mathbb{R}^{n \times m}$ and $m \ll n$. Instead of $Z \approx ZC$, SDSNet enforces $Z \approx L P^\top$ for a selected landmark matrix $L \in \mathbb{R}^{d \times m}$.

3. **Efficient clustering head:** Spectral clustering is performed in the low-dimensional space spanned by the factor $P$ or subspace assignment matrix $W$, with all operations costing $\mathcal{O}(n)$ when $m, d, k$ are fixed.

A comparison of key self-expressive strategies appears below:

| Model        | Self-Expressive Representation         | Parameter Count          |
|--------------|---------------------------------------|-------------------------|
| DSC-Net      | $C \in \mathbb{R}^{n \times n}$       | $\mathcal{O}(n^2)$      |
| SDSNet (2018)| $W \in \{0,1\}^{k \times n}$, $S_i$   | $\mathcal{O}(kd_zp)$    |
| SDSNet (2025)| $P \in \mathbb{R}^{n \times m}$, $L$  | $\mathcal{O}(nm + dm)$  |

## 3. Optimization Objectives and Training Algorithms

SDSNet jointly optimizes auto-encoder reconstruction and subspace-preserving self-expressiveness losses:

- **[1811.01045]:**  
  $$
  L_{\text{rec}}(\theta_e, \theta_d) = \sum_{j=1}^n \|x_j - g_{\theta_d}(f_{\theta_e}(x_j))\|_2^2
  $$
  $$
  L_{\text{ksc}}(S_1, \ldots, S_k, \theta_e) = \sum_{j=1}^n \sum_{i=1}^k w_{ij} \|f_{\theta_e}(x_j) - S_i S_i^\top f_{\theta_e}(x_j)\|_2^2
  $$
  $$
  L = L_{\text{rec}} + \lambda L_{\text{ksc}}
  $$
  Alternating minimization updates the assignment $W$, subspaces $S_i$ (via SVD or Grassmannian gradient), and network parameters by minibatch stochastic gradient descent (Adam).

- **[2512.21434]:**  
  $$
  \mathcal{L}(W, \widehat W, P, L) = \frac{1}{n}\|X - \widehat X\|_F^2 + \|Z - L P^\top\|_F^2, \quad P^\top P = I_m
  $$
  Optimizing over $P$ and $L$ is achieved via Procrustes SVD or least squares, alternating with back-propagation for encoder/decoder parameters. Affinity $C = P P^\top$ is block-diagonalized by optimizing representations during training.

## 4. Landmark-Based and Subspace-Assignment Approximations

A defining feature of SDSNet [2512.21434] is its replacement of the full self-expressiveness matrix $C$ with a landmark-based factorization. $m$ anchors are selected via $k$-means++ or random sampling in latent space, leading to $L \in \mathbb{R}^{d \times m}$. The resulting $P \in \mathbb{R}^{n \times m}$ encodes how each data point relates to the landmarks, with $C \approx P P^\top$. All affinity computations and subsequent spectral clustering occur in the $m$-dimensional anchor space, yielding $\mathcal{O}(nm^2)$ computation and $\mathcal{O}(nm)$ memory.

In [1811.01045], explicit assignment variables $W \in \{0,1\}^{k \times n}$ simplify clustering to an assignment to one of $k$ subspaces per point; subspaces are updated via SVD of assigned embeddings.

## 5. Computational Complexity and Scalability

SDSNet substantially alleviates the cubic and quadratic scaling issues of prior methods:

- **Classical affinity-based deep subspace clustering:** $\mathcal{O}(n^2)$ memory and $\mathcal{O}(n^3)$ time (building, storing, and decomposing $C$) [1709.02508].
- **SDSNet [1811.01045]:** Each epoch comprises encoding $n$ samples ($\mathcal{O}(n d_z)$), assignments ($\mathcal{O}(n k d_z)$), and SVD/Grassmann updates ($\mathcal{O}(k d_z^2 n/k)$), all linear when $k$ and $d_z$ are constant.
- **SDSNet [2512.21434]:** Encoder/decoder updates ($\mathcal{O}(n d)$), $P$/$L$ updates ($\mathcal{O}(n m^2 + n m d)$), and final spectral clustering ($\mathcal{O}(n m^2 + m^3)$).

Empirically, SDSNet clusters 70,000 MNIST images in under 8 minutes using $\approx$2 GB GPU memory; equivalent affinity-based approaches require $\approx$37 GB just for storing the affinity matrix [1811.01045]. On synthetic datasets with $n$ up to 10,000, computational time is observed to grow linearly [2512.21434].

## 6. Empirical Performance and Benchmarking

Extensive experiments demonstrate that SDSNet maintains or surpasses the clustering accuracy of both affinity-based and scalable state-of-the-art methods. On Fashion-MNIST (60,000 samples), SDSNet achieves ACC $\approx$ 63.8% and NMI $\approx$ 62.0% in under 10 minutes [1811.01045]. On YaleB, ORL, Coil100, UMIST, and Fashion-MNIST, [2512.21434] reports that SDSNet matches or exceeds alternatives (SSC-OMP, EnSC, SGL, S$^5$C, LMVSC, SSSC, A-DSSC) by margins of 5–35% in ACC and NMI, with subspace-preserving error (SPE) comparable to dense affinity approaches.

Convergence is achieved rapidly, with the relative change in $C$ or $P$ dropping below $10^{-3}$ in fewer than 10 outer iterations [2512.21434]; affinity matrices display clear block-diagonal structure early in training.

## 7. Implementation Strategies and Hyperparameters

Robust performance requires careful tuning and implementation choices:

- **Hyperparameters:** Number of subspaces $k$ is set to the true number of clusters (e.g., $k=10$ for MNIST), embedding dimension $d_z$ typically 20, subspace dimension $p$ in [7, 11], anchor number $m \ll n$ (proportional to $k$), loss weightings $\lambda \in [0.08, 0.11]$, batch size $b=256$, optimizer Adam ($10^{-3}$ lr), AE pre-training $200$ epochs, joint training $100$–$200$ epochs [1811.01045].
- **Subspace updates:** Either top-$p$ SVD on assignments (discarding 10% outliers) or Riemannian updates (Grassmann retraction).
- **Landmark selection:** $k$-means++ improves anchor robustness over random, especially as $m$ approaches $k$.
- **Initialization and batch scheduling:** Pre-training the auto-encoder facilitates stable convergence; initializing subspaces/anchors via $k$-means or shallow DSC improves accuracy and speed.

SDSNet’s design establishes a paradigm shift for deep subspace clustering, yielding highly scalable, accurate solutions across large-scale, high-dimensional datasets by circumventing the full affinity bottleneck and optimizing computational paths through architectural and algorithmic innovations [1811.01045, 2512.21434].

Source: https://www.emergentmind.com/topics/product-of-projections-in-multi-view-subspace