Dictionary Identification - Sparse Matrix-Factorisation via $\ell_1$-Minimisation (0904.4774v2)

Abstract: This article treats the problem of learning a dictionary providing sparse representations for a given signal class, via $\ell_1$-minimisation. The problem can also be seen as factorising a $\ddim \times \nsig$ matrix $Y=(y_1 >... y_\nsig), y_n\in \R^\ddim$ of training signals into a $\ddim \times \natoms$ dictionary matrix $\dico$ and a $\natoms \times \nsig$ coefficient matrix $\X=(x_1... x_\nsig), x_n \in \R^\natoms$, which is sparse. The exact question studied here is when a dictionary coefficient pair $(\dico,\X)$ can be recovered as local minimum of a (nonconvex) $\ell_1$-criterion with input $Y=\dico \X$. First, for general dictionaries and coefficient matrices, algebraic conditions ensuring local identifiability are derived, which are then specialised to the case when the dictionary is a basis. Finally, assuming a random Bernoulli-Gaussian sparse model on the coefficient matrix, it is shown that sufficiently incoherent bases are locally identifiable with high probability. The perhaps surprising result is that the typically sufficient number of training samples $\nsig$ grows up to a logarithmic factor only linearly with the signal dimension, i.e. $\nsig \approx C \natoms \log \natoms$, in contrast to previous approaches requiring combinatorially many samples.

• The paper establishes local identifiability conditions for dictionary and coefficient pairs using a non-convex ℓ1-minimization approach.
• It derives sufficient conditions ensuring that a basis achieves strict local minima, with sample complexity scaling as C · K log K under a Bernoulli-Gaussian model.
• The study employs geometric and probabilistic techniques to advance robust sparse coding, paving the way for efficient algorithmic developments in signal processing.

Sparse Matrix Factorization via $\ell_1$-Minimization for Dictionary Identification

This paper addresses the complex problem of dictionary learning, specifically focusing on identifying a dictionary that provides sparse representations of a given class of signals through $\ell_1$-minimization. The research problem is framed as a sparse matrix factorization task, where a $d \times N$ matrix $Y$ of training signals is factorized into a $d \times K$ dictionary matrix $\Phi$ and a $K \times N$ sparse coefficient matrix $X$.

Main Contributions

The paper offers several theoretical contributions to the field:

1. Local Identifiability Analysis: It provides algebraic conditions for local identifiability of the dictionary and coefficient pair $(\Phi, X)$ as a local minimum of a non-convex $\ell_1$-criterion. These conditions are rooted in the analysis of the tangent space to the constraint manifold and use concentration of measure phenomena.
2. Sufficient Conditions for a Basis: The work also derives sufficient conditions affirmed by Theorems ensuring that $(\Phi, X)$ is a strict local minimum when $\Phi$ is a basis, suggesting robustness in standard dictionary learning tasks.
3. Probability Bounds: For coefficients drawn from a Bernoulli-Gaussian distribution, the paper establishes that a basis matrix is locally identifiable with high probability, given a sufficient number of training samples $N$. The empirically derived relation suggests the sufficient number of training samples $N$ scales as $N \approx C K \log K$, where $C$ is a constant, deviating from previous requirements of combinatorial growth.

Technical Insights

• Convex Relaxation: The work uses the principle of convex relaxation by replacing the $\ell_0$-norm with $\ell_1$-norm to promote sparsity. The non-convex $\ell_1$-criterion poses challenges but enables a more computationally tractable problem than combinatorial alternatives.
• Random Sparse Model: Assuming a random Bernoulli-Gaussian model for coefficient matrix $X$, the research validates, through probabilistic analysis, that incoherent bases are identifiable if sample numbers grow logarithmically with signal dimensions. This model allows for non-sparse outliers, rendering the results more applicable to real-world tasks involving noisy or imperfect data.
• Geometric Interpretation: The geometric conditions are depicted via the projection of high-dimensional polyhedra, providing intuitive insights into the relationship between sparsity and dictionary identifiability.

Implications and Future Prospects

This paper not only provides a theoretical foundation for dictionary learning via $\ell_1$-minimization but also stimulates several avenues for future exploration in sparse coding and compressed sensing. Notably:

• Extension to Overcomplete Dictionaries: While the results explicitly cover the bases, extending these results to overcomplete dictionaries could open new frontiers in signal processing and machine learning applications.
• Algorithmic Development: The provided conditions serve as a basis for designing algorithms that might exploit $\ell_1$-based criteria for more efficient sparse representation learning, potentially replacing current combinatorial approaches.
• Robustness to Outliers: The paper hints at $\ell_1$-minimization's potential robustness to outliers, suggesting robustness enhancement strategies for practical dictionary learning algorithms.

In summation, this paper effectively lays the groundwork for more efficient and theoretically grounded approaches to sparse matrix factorization and dictionary learning, with significant implications for both theoretical research and practical algorithms in signal processing and machine learning.