Diverse Dictionary Learning

Abstract: Given only observational data $X = g(Z)$, where both the latent variables $Z$ and the generating process $g$ are unknown, recovering $Z$ is ill-posed without additional assumptions. Existing methods often assume linearity or rely on auxiliary supervision and functional constraints. However, such assumptions are rarely verifiable in practice, and most theoretical guarantees break down under even mild violations, leaving uncertainty about how to reliably understand the hidden world. To make identifiability actionable in the real-world scenarios, we take a complementary view: in the general settings where full identifiability is unattainable, what can still be recovered with guarantees, and what biases could be universally adopted? We introduce the problem of diverse dictionary learning to formalize this view. Specifically, we show that intersections, complements, and symmetric differences of latent variables linked to arbitrary observations, along with the latent-to-observed dependency structure, are still identifiable up to appropriate indeterminacies even without strong assumptions. These set-theoretic results can be composed using set algebra to construct structured and essential views of the hidden world, such as genus-differentia definitions. When sufficient structural diversity is present, they further imply full identifiability of all latent variables. Notably, all identifiability benefits follow from a simple inductive bias during estimation that can be readily integrated into most models. We validate the theory and demonstrate the benefits of the bias on both synthetic and real-world data.

• The paper presents a novel set-theoretic framework that recovers latent structure via intersections and complements of supports.
• It leverages a minimal inductive bias through dependency sparsity regularization, guaranteeing identifiability even in noisy, nonparametric settings.
• Empirical results on VAEs, GANs, and diffusion models demonstrate improved disentanglement and interpretability over previous methods.

Diverse Dictionary Learning: A Set-Theoretic Approach to Partial and Universal Identifiability

Introduction and Motivation

The "Diverse Dictionary Learning" framework addresses a fundamental limitation in latent variable modeling: identifiability under minimal assumptions. Traditional dictionary learning—recovering latent variables $Z$ from observations $X = g(Z)$ with unknown $g$—is severely ill-posed nonparametrically. Prevailing approaches achieve identifiability via strong model restrictions (e.g., linearity, post-nonlinear form), auxiliary variables, or interventional data. However, these assumptions seldom hold in practice, compromising theoretical guarantees and hampering model reliability in real-world applications.

This work reconceptualizes the identifiability question: rather than full recovery, what aspects of the latent structure remain consistently recoverable in highly unconstrained settings? This question is of particular importance for scientific discovery, mechanistic interpretability (notably in LLM analysis), causal inference, and robust representation learning, where guarantees must withstand assumption violations. The authors introduce the diverse dictionary learning paradigm—a set-theoretic approach that establishes which structural properties of latent space are universally identifiable, which depend only on minimal and verifiable inductive biases.

Theoretical Contributions

Generalized Identifiability via Set Algebra

The central conceptual advance is the definition of generalized identifiability based on set-theoretic decompositions of the dependency structure between $Z$ and $X$. Rather than focusing on global identifiability up to invertible transformations (as in nonlinear ICA or block identifiability), the framework quantifies what can be recovered about latent variables—even when model classes are unconstrained—through operations such as intersections, complements, and symmetric differences of supports.

For every subset of observed variables, the latent index set is the subset of latent variables that influence those observations (as defined by the support of the Jacobian $D_Z g$). The following are shown to be generically identifiable:

• Shared latent factors: The intersection of supports identifies generative features common to multiple observations.
• Unique factors: Symmetric differences/complements identify latent variables exclusive to certain observation groups.
• Dependency structure: The bipartite structure mapping $Z$ to $X$ is identifiable up to column permutations.

Crucially, these properties hold without parametric or independence assumptions, enabling recovery guarantees that are robust to realistic data-generating mechanisms.

Sufficient Inductive Bias: Dependency Sparsity

The theoretical results hinge on a single, minimal inductive bias: dependency sparsity regularization. Regularizing the support of the Jacobian matrix (i.e., the functional dependencies from each latent to each observed variable) is shown to be both necessary and sufficient for the identifiability guarantees to hold. Importantly, this is a regularization applied at estimation, not a property required of the true generative process. The framework is indifferent to whether the data-generating structure is dense or sparse; the key requirement is diversity in dependency patterns, not sparsity per se.

From Partial to Full Identifiability

The set-theoretic approach reveals that when the dependency structure between $Z$ and $X$ is sufficiently diverse (formally: every latent variable has a uniquely “exposed” atomic region in the Venn diagram defined by supports), the theoretical framework collapses to element-wise identifiability up to permutation and invertible reparametrization. This generalizes and subsumes prior block-identifiability results, while providing concrete structural conditions strictly weaker than previous “anchor”, “separability”, or “sparse connectivity” criteria.

Empirical Results

Synthetic Evaluations

The authors validate the theory on synthetic datasets generated from nonlinear functions, using VAEs with dependency sparsity regularization. Performance is measured via disentanglement metrics (e.g., $X = g(Z)$0, MCC between estimated and ground-truth latents):

• Set-theoretic disentanglement consistently holds: intersections and symmetric differences of supports yield low mutual predictability across structurally distinct components.
• Element-wise recovery (up to permutation) is only achieved when structural diversity holds, as predicted by theory.

Dependency sparsity shows significant robustness to additive noise, where competitors (e.g., latent sparsity, Hessian penalties) degrade.

Visual Disentanglement

Integration of dependency sparsity regularization into VAE-, GAN-, and diffusion-based disentanglement models (FactorVAE, DisCo, EncDiff) yields systematic improvements on both the FactorVAE score and DCI across benchmark datasets (Shapes3D, Cars3D, MPI3D). These improvements exceed those obtainable by latent sparsity regularization—especially in models with high-dimensional or strongly entangled representations. Latent traversals and latent swapping on complex images demonstrate controlled, semantically meaningful manipulation of independent factors, confirming interpretability and independent control downstream.

Practical and Theoretical Implications

Robustness and Universality

The set-theoretic approach decouples identifiability guarantees from unverifiable, brittle assumptions (linearity, post-nonlinearity, auxiliary views). This renders identifiability actionable—guarantees hold (locally and globally) provided a universal and easily implemented dependency sparsity regularizer is enforced, emphasizing its role as a domain-agnostic inductive bias. Theoretical results extend naturally to scenarios with additive noise and cover settings relevant to mechanistic interpretability (e.g., in LLMs), transfer learning, controllable generation, and multimodal alignment.

Limitations and Open Directions

While structurally flexible, the sufficiency of sparsity/diversity requires access to gradients with respect to latent variables; computational cost in large models is non-trivial but substantially mitigated via subspace Jacobian computations. Extension to non-invertible or actively stochastic generative processes demands further investigation. Asymptotic results remain relevant but do not offer finite-sample guarantees—future work could address the statistical-computational gap.

Prospects for AI and Disentangled Representation Learning

Diverse dictionary learning formalizes a new axis of identifiability theory, establishing a “local” (setwise, blockwise) recovery guarantee robust across domains. Adoption of dependency sparsity as a universal regularizer—rather than assumption-heavy model restrictions or access to interventions—is supported both empirically and theoretically. This has direct ramifications for mechanistic interpretability (e.g., in understanding and diagnosing LLMs), precise reward modeling, and scientific discovery, where only partial but reliable recovery of the latent structure is feasible or even desirable.

Future directions include scaling these inductive biases to foundation models, embedding them into training regimes for more faithful controllable generation, causal inference, and robust domain adaptation, and further unifying principles of identifiability with practice in real-world AI systems. As data and computational resources grow, focusing on structural identifiability will become essential for bridging prediction and explanation in machine learning.

Conclusion

Diverse dictionary learning reframes the identifiability problem in latent variable modeling. By leveraging set algebra and a minimal universal inductive bias—dependency sparsity—reliable partial and, when possible, full recovery of latent generative structure is attainable under minimal and verifiable conditions. The results generalize, complement, and strengthen previous work, offering robust theoretical backing for modern approaches to interpretability and representation learning, and guiding principled inductive bias design for future models.