Sparsity-Driven Sampling in Union-of-Submanifolds Models

• Sparsity-driven latent sampling is a framework that represents high-dimensional data using unions of low-dimensional subspaces or manifolds for efficient and interpretable recovery.
• It integrates concepts from sparse coding, dictionary learning, and deep generative models to rigorously bound sample complexities in signal reconstruction tasks.
• Applications include compressive imaging, neural network interpolation, and generative modeling, offering improved expressivity and reduced computational costs.

Sparsity-driven latent sampling and union-of-submanifolds models constitute a foundational methodology for high-dimensional generative modeling, signal recovery, and compressed sensing. This framework posits that complex signals or data distributions are well-approximated by unions of low-dimensional structures—linear subspaces or nonlinear submanifolds—enabling compact, interpretable, and efficient representations. These models are underpinned by advances in sparse coding, dictionary learning, group-sparsity, and recent developments in deep generative modeling with structured latent spaces.

1. Core Principles: Sparse Coding and Unions of Subspaces

A central tenet is that data can be parsimoniously represented as residing in a union of a small number of low-dimensional subspaces within a high-dimensional ambient space. Classical sparse coding seeks to express latent variables as $z = D\alpha$, where $D \in \mathbb{R}^{d \times K}$ is a learned dictionary and $\alpha \in \mathbb{R}^K$ is a sparse coefficient vector, typically constrained via $\ell_0$ or $\ell_1$ penalties:

$$\min_\alpha \;\|z - D\alpha\|_2^2 + \lambda \|\alpha\|_1\quad \text{or} \quad \min_\alpha \;\|z - D\alpha\|_2^2 + \lambda \|\alpha\|_0,\ \|\alpha\|_0 \leq s.$$

In the context of deep generative models, sparsity-driven approaches replace the standard dense latent variable with structured, highly sparse representations, imposing a union-of-subspaces structure on the latent space (Li et al., 16 Sep 2024). The generative process integrates this model into the standard variational autoencoder (VAE) evidence lower bound (ELBO), enforcing sparsity through either regularization or explicit constraints.

Each choice of the support set (indices of nonzero coefficients) selects a specific $s$-dimensional subspace, making the overall latent space a union $$\mathcal{U} = \bigcup_{|S| = s} \operatorname{span}\{D_{:,j}: j \in S\}$$. For nonlinear generators $G_\theta$, each subspace in the latent domain maps to a distinct low-dimensional manifold in data space, yielding a union-of-submanifolds (Killedar et al., 2021).

2. Mathematical Formulations and Recovery Guarantees

The union-of-subspaces model rigorously underpins recovery guarantees and sampling complexity for both linear and nonlinear inverse problems. Given a signal $x \in \mathbb{R}^p$ lying in a union of $M$ subspaces $\{\operatorname{span}(K_i)\}$, with $K_i \in \mathbb{R}^{p \times d_i}$, signal recovery from $n$ measurements $y = \Phi x$ is formulated via atomic norm minimization: $$\hat{x} = \arg\min_{x \in \mathbb{R}^p} \|x\|_A \quad \text{s.t.} \ y = \Phi x,$$ where $$\|x\|_A = \inf\{\sum_i \|\alpha^i\| : x = \sum_i K_i \alpha^i\}$$ (Rao et al., 2012).

The precise sample complexity for exact recovery is governed by the Gaussian width of the tangent cone of the subspace union, yielding universal bounds such as

$$m \gtrsim k \left( \sqrt{2\log(M-k)} + \sqrt{B} \right)^2 + k B,$$

where $k$ is the number of active subspaces and $B = \max d_i$.

For deep generative latent models with $k$-dimensional latent variables and $s$-sparsity, the generator $G_\theta$ maps each of the $\binom{k}{s}$ coordinate subspaces to a distinct $s$-dimensional manifold. The relevant sample complexity for successful compressed recovery is

$m = \Omega\left( s d \log \frac{k h t}{s} \right),$

where $d$ is the network depth, $h$ the number of nodes per layer, $t$ the number of pieces per activation, and $s$ the number of active latent variables (Killedar et al., 2021). This scaling interpolates between traditional compressed sensing and deep generative sensing.

3. Model Implementations: From LASERS to SDLSS

LASERS (Latent Space Encoding for Representations with Sparsity) (Li et al., 16 Sep 2024) generalizes the vector quantization (VQ) approach in VAEs, which quantizes latent codes to a discrete codebook, to a continuous, sparse, dictionary-based latent model:

• The encoder infers a sparse code $\alpha$ and reconstruction $z = D\alpha$.
• The ELBO is augmented with a KL-divergence on $q_\phi(\alpha|x)$, a prior $p(\alpha)$ (Laplace or spike-and-slab), and a sparsity penalty $$\lambda \mathbb{E}_{q_\phi(\alpha|x)}[\|\alpha\|_1]$$.
• Dictionary $D$ is learned by gradient-based updates with periodic column normalization.
• Sampling proceeds by support selection for $\alpha$, sampling coefficients on the support, forming $z = D\alpha$, and generating $x$ via the decoder. This approach constructs a union-of-subspaces latent space that mitigates codebook collapse, offers increased expressivity over VQ-VAEs, and achieves empirically improved reconstruction at a modest computational overhead (10–30% slower per minibatch).

Sparsity-Driven Latent Sampling and Proximal Meta-Learning (SDLSS/PML) (Killedar et al., 2021) operates by enforcing latent $s$-sparsity via hard-thresholding within a meta-learning framework, where a generator $G_\theta$ plus (optionally) a learned measurement operator $A_\phi$ map a sparse latent code to the data domain. The union-of-submanifolds semantics arises as each distinct support in latent space maps to a disconnected submanifold; sample complexity and empirical results show superior performance in compressed recovery and representations.

Sparse Deep Latent Generative Models (SDLGM) (Xu et al., 2023) utilize discrete latent variables with a learned per-sample sparsity parameter $L_i$ (expressed via auxiliary Bernoulli variables) to generate sparse latent codes inhabiting a union of coordinate subspaces. Gumbel-Softmax relaxations provide a differentiable path for efficient training and variational inference, enabling unsupervised and supervised applications in settings requiring variable but controlled sparsity.

4. Union-of-Submanifolds: Nonlinear Generalizations

The union-of-subspaces paradigm extends naturally to nonlinear settings, where each local structure is a smooth manifold rather than a flat subspace. In high-dimensional nonlinear sampling frameworks (Zou et al., 2019):

• A surface $\mathcal{S}$ is modeled as the zero set of a bandlimited function: $\psi(\mathbf{x}) = 0$, where $$\psi(\mathbf{x}) = \sum_{\mathbf{k} \in \Lambda} c_{\mathbf{k}} e^{j 2\pi \mathbf{k}^T \mathbf{x}}$$.
• Each point is lifted via an exponential map $\Phi_\Lambda(\mathbf{x})$, so that features of points on the same irreducible surface lie in a linear subspace of feature space.
• For unions of irreducible surfaces, the feature set is a union of subspaces in the lifted domain.
• Sample complexity for exact recovery is sharply characterized by the cardinality $|\Lambda|$ of the basis used.
• Local neural-network-like representations arise through Dirichlet kernel interpolation in feature space, resulting in minimal parameterizations and strong data efficiency.

This framework yields both explicit reconstruction algorithms (via SVD-based nullspace computation) and theoretical sample bounds matching empirical recoverability in experiments.

5. Algorithms and Optimization: Greedy and Convex Methods

Recovery of signals in union-of-subspaces models leverages both greedy algorithms and convex-relaxation frameworks:

• Convex Atomic Norm Minimization: For known subspace collections, convex programs minimize the atomic (group) norm for exact signal recovery, with universal bounds holding regardless of subspace overlap or configuration (Rao et al., 2012).
• Generalized CoSaMP (GCoSaMP): Greedy pursuit algorithms are elevated from classical sparse-synthesis to general union-of-subspaces, where each iteration conducts subspace selection and projection rather than simple support thresholding. Convergence is established in terms of Gaussian mean width, with robust denoising and explicit error bounds under Gaussian measurements (Tirer et al., 2017).
• K-Subspaces and K-Submanifolds: Alternating minimization between subspace (or manifold) fitting and sample assignment achieves locally optimal decompositions, extending to infinite-dimensional and shift-invariant settings in Hilbert spaces (0707.2008). These methods underpin both classical dictionary learning and the more general union-of-manifolds clustering used in nonlinear sampling (Zou et al., 2019).

6. Applications and Empirical Results

Compressive Imaging and Signal Processing: Signal classes such as group-sparse or block-sparse signals, tree-structured wavelet coefficients, and low-rank matrix patches are naturally modeled as residing in unions of (possibly overlapping) subspaces (Rao et al., 2012). Sample complexity analyses and empirical simulations consistently demonstrate phase transitions matching theoretical bounds, and group-lasso or atomic-norm minimization achieves superior recovery compared to unstructured sparsity.

Deep Generative Models: Union-of-submanifold structures in the latent space (established via sparsity or structured discrete variables) lead to improved generative modeling—more expressive distributions, better code utilization, reduced codebook collapse, and higher-fidelity samples (Li et al., 16 Sep 2024, Xu et al., 2023). Sparsity provides both a powerful inductive prior and a practical mechanism for disentangled and interpretable representations.

Function Interpolation and Neural Networks: Nonlinear unions underpin approaches to learn low-rank function interpolation on sampled surfaces, yielding compact and sample-efficient local neural network architectures (Zou et al., 2019).

Learning Under Measurement Constraints: SDLSS and related methods, through control of latent sparsity and union-of-submanifolds modeling, improve the efficiency and generalization of compressive sensing in learned nonlinear settings, empirically outperforming previous deep learning approaches across objective metrics (PSNR, SSIM, RE) (Killedar et al., 2021).

7. Extensions and Theoretical Outlook

Recent work frames the union-of-subspaces and union-of-manifolds model as unifying paradigms spanning compressed sensing, dictionary learning, latent variable generative models, and nonlinear sampling theory. The Gaussian mean width provides an intrinsic measure of model complexity governing sample efficiency and recovery guarantees. Extensions actively explored include:

• Explicit modeling and learning of unions of nonlinear manifolds;
• Adaptive noise models and non-Gaussian measurements;
• Joint synthesis–analysis pursuits and combined structured sparsity models (Tirer et al., 2017);
• Algorithmic generalizations for efficient projection onto manifold unions.

A prevailing implication is that sparsity, when coupled with union structures (linear or nonlinear), enables a data-driven balance between model expressivity and sample efficiency, supporting both theoretical guarantees and practical advances in high-dimensional inference and generative modeling (Li et al., 16 Sep 2024, Zou et al., 2019, Killedar et al., 2021, Xu et al., 2023, Rao et al., 2012, 0707.2008, Tirer et al., 2017).