Identifiability and Estimation in High-Dimensional Nonparametric Latent Structure Models

Abstract: This paper studies the problems of identifiability and estimation in high-dimensional nonparametric latent structure models. We introduce an identifiability theorem that generalizes existing conditions, establishing a unified framework applicable to diverse statistical settings. Our results rigorously demonstrate how increased dimensionality, coupled with diversity in variables, inherently facilitates identifiability. For the estimation problem, we establish near-optimal minimax rate bounds for the high-dimensional nonparametric density estimation under latent structures with smooth marginals. Contrary to the conventional curse of dimensionality, our sample complexity scales only polynomially with the dimension. Additionally, we develop a perturbation theory for component recovery and propose a recovery procedure based on simultaneous diagonalization.

• The paper introduces a unified identifiability theorem that generalizes conditions by leveraging higher dimensionality and variable diversity.
• It establishes quantitative convergence rates with polynomial dependence on dimension, mitigating the curse of dimensionality in density estimation.
• The proposed recovery algorithm efficiently recovers component densities under an incoherence condition, ensuring controlled estimation errors.

This paper addresses the identifiability and estimation problems in high-dimensional nonparametric latent structure models. These models represent a data distribution $\mu$ as a mixture of $m$ component distributions, where each component is a product of $d$ marginal measures:

$$\mu=\sum_{k=1}^m\pi_k (\mu_{k1}\times \mu_{k2}\times\cdots \times \mu_{kd})$$

The paper aims to overcome limitations in existing theoretical frameworks, particularly the reliance on strong linear independence conditions for the marginal measures $\mu_{kj}$.

Key Contributions

The main contributions of the paper are:

1. A Unified Identifiability Theorem: The paper introduces a new identifiability theorem that generalizes previous conditions. It explains how higher dimensionality, combined with diversity in variables, aids identifiability even when linear independence doesn't hold.
2. Quantitative Rates of Convergence: It establishes a perturbation theory for estimating the component densities under an incoherence condition. This theory shows how errors in estimating the joint density propagate to errors in estimating the components. Additionally, near-optimal minimax risk bounds for high-dimensional nonparametric density estimation are derived, demonstrating that sample complexity scales polynomially with dimension, thus circumventing the "curse of dimensionality."
3. A Recovery Algorithm: A practical algorithm is developed for recovering component densities from an estimator of the joint density. This algorithm relies on an incoherence condition rather than the stricter linear independence.

Identifiability without Linear Independence

Existing identifiability results, like Allman et al. (2009) [not cited with arXiv id in text, but common knowledge], often rely on the "linear independence condition," where for each dimension $j$, the marginals $\{\mu_{1j}, \dots, \mu_{mj}\}$ are linearly independent. This condition fails in important cases like conditional i.i.d. models (where $\mu_{k1} = \dots = \mu_{kd}$ for each component $k$) and Bernoulli mixture models (where $\mu_{kj}$ are Bernoulli distributions, and linear independence fails for $m \ge 3$).

To address this, the paper introduces the concept of $\ell$-independence:

• The $j$-th variable is $\ell$-independent if every subset of $\{\mu_{kj}\}_{k=1}^m$ of cardinality $\ell$ is linearly independent.
• $\textnormal{Ind}_\mu(j)$ is the maximum $\ell$ for the $j$-th variable.
• For a subset $S \subseteq [d]$, $$\tau_{\mu}(S) \triangleq \min\{m,\textnormal{Ind}_\mu(S)-|S|+1\}$$ denotes the total excess independence in $S$.

Theorem \ref{identi_thm} (Main Identifiability Result):

The model $\mu$ is identifiable if there exists a partition $S_1, S_2, S_3$ of the $d$ dimensions such that:

$$\tau_{\mu}(S_1)+\tau_{\mu}(S_2)+\tau_{\mu}(S_3)\geq 2m+2$$

Conversely, a non-identifiable $\mu$ exists if for every partition, this sum is $\le 2m+1$.

A key corollary relates to the separability condition (where $\mu_{kj} \neq \mu_{k'j}$ for $k \ne k'$): Corollary \ref{sepa_thm}: If the number of separable variables $N(\mu) \geq 2m-1$, then $\mu$ is identifiable. This unifies results from prior work showing $d=2m-1$ as a critical threshold.

The proof of Theorem \ref{identi_thm} uses Hilbert space embedding techniques. The measures $\mu_{kj}$ are represented as functions $f_{kj}$ in $L^2(\xi)$ for some dominating measure $\xi$. The joint density is mapped to a tensor in $L^2(\xi)^{\otimes d}$. The proof relies on:

1. Relating the Kruskal rank of the set of component functions to the Kruskal rank of their Gram matrices.
2. Lemma \ref{Hadamard}: A novel result showing that the Kruskal rank of a Hadamard product of Gram matrices $A \circ B$ is at least $\min\{n, k_A + k_B - 1\}$, where $k_A, k_B$ are the Kruskal ranks of $A, B$. This lemma is crucial for establishing the total Kruskal rank condition.
3. An extension of Kruskal's theorem for tensor decomposition in Hilbert spaces (Lemma \ref{Extension}).

Rate of Convergence under Incoherence

This section assumes each $\mu_{kj}$ has a density $f_{kj}$.

1. Recovering Component Density: Perturbation Analysis

The paper introduces an incoherence condition:

• A set of functions $\{f_k\}_{k=1}^m$ in a Hilbert space is $\mu$-incoherent if $$|\langle f_k, f_{k'} \rangle| \le \mu \|f_k\|_2 \|f_{k'}\|_2$$ for $k \ne k'$, with $\mu < 1$.

Assumption \ref{assume} (Estimable Condition):

1. $f_{kj}$ are square integrable. For each $j$, $\{f_{kj}\}_{k=1}^m$ is $\mu$-incoherent.
2. Mixing proportions $\pi_k \ge \zeta > 0$.

Theorem \ref{Angle_thm} (Robust Identifiability):

If $f$ is $(\mu, \zeta)$-estimable and an estimator $\tilde{f}$ (also a mixture of product densities) satisfies $\|f - \tilde{f}\|_2 \le \epsilon$ (for sufficiently small $\epsilon$), then there exists a permutation $\sigma$ such that the errors in component densities and mixing proportions are bounded by terms proportional to $\epsilon$:

$$\|f_{kj} - \tilde{f}_{\sigma(k)j}\|_2 \lesssim \epsilon$$

$\|\pi - \sigma(\tilde{\pi})\|_2 \lesssim \epsilon$

The constants depend on $C$ (uniform bound on $\|f_{kj}\|_\infty$), $m$, $\mu$, and $\zeta$. This shows that small errors in estimating the joint density translate to small errors in estimating the components, under incoherence.

The proof sketch involves:

• Considering marginal densities to reduce the problem to $d=2m-1$.
• Representing densities $f, \tilde{f}$ as tensors $T, \tilde{T}$ in $L^2([0,1])^{\otimes (2m-1)}$.
• Analyzing the mode-1 multiplication $T \times_1 w$ (where $w$ is a test function) and its matrix unfolding $T_w = A D_{\pi,w} B^*$.
• Using Weyl's inequality (Lemma \ref{Weyl}) for singular values: $$\sup_{\|w\|_2=1} \max_k |\sigma_k(T_w) - \sigma_k(\tilde{T}_w)| \le \epsilon$$.
• Using a probabilistic method (Lemma \ref{lemma_test}) to find test functions $w$ that can distinguish components or reveal contradictions if the theorem's conclusion doesn't hold. This involves showing that if a component $\tilde{f}_{kj}$ is far from all true components $f_{k'j}$, one can construct a $w_0$ such that $\sigma_m(\tilde{T}_{w_0}) = 0$ while $\sigma_m(T_{w_0}) > \epsilon$, a contradiction. Similar arguments establish the permutation's uniqueness and consistency across dimensions.

2. Estimation of the Joint Distribution under Hölder Smoothness

The paper considers the density class $\mathcal{G}_{\mathcal{F}^{(m,d)}}$ where component densities $f_{kj}$ belong to a Hölder smoothness class $\mathcal{F}_{L,q}$.

Theorem \ref{Holder_rate} (Minimax Rates):

For estimating $f \in \mathcal{G}_{\mathcal{F}_{L,q}^{(m,d)}}$ from $n$ i.i.d. samples:

• Under Hellinger distance ($H$): Minimax risk $$R^*_{H, \mathcal{F}_{L,q}^{(m,d)}} \asymp n^{-\frac{q}{q+1}}$$ (up to log factors and polynomial terms in $m,d$). For $n \ge md^{1+1/q}$:

  $$(n\log n)^{-\frac{q}{q+1}}d \lesssim R^*_{H} \lesssim n^{-\frac{q}{q+1}}m^{\frac{q}{q+1}}d$$

• Under Total Variation distance ($TV$): Minimax risk $$R^*_{TV, \mathcal{F}_{L,q}^{(m,d)}} \asymp n^{-\frac{2q}{2q+1}}$$ (similarly). For all $n \ge 1$:

  $$(n\log n)^{-\frac{2q}{2q+1}} \lesssim R^*_{TV} \lesssim n^{-\frac{2q}{2q+1}}m^{\frac{2q}{2q+1}}d^{\frac{2q+2}{2q+1}}$$

These rates show that the complexity depends polynomially on $d$, unlike standard nonparametric density estimation where it's exponential ($n^{-q/(q+d)}$ or $n^{-2q/(2q+d)}$). This demonstrates that the latent product structure mitigates the curse of dimensionality. The proof uses classical information-theoretic arguments based on metric entropy.

Algorithm for Recovery of Components

A practical algorithm based on simultaneous diagonalization (Leurgans et al., 1993) is proposed to recover $f_{kj}$ from an estimator $\hat{f}$ of the joint density $f$. It focuses on $d=2m-1$.

Algorithm 1:

1. Given $\hat{f}(x_1, \dots, x_{2m-1})$, compute $$\hat{T}_{+}(y,z) = \int \hat{f}(y,z,x_{2m-1}) dx_{2m-1}$$, where $y=(x_1,\dots,x_{m-1}), z=(x_m,\dots,x_{2m-2})$.
2. Find $\hat{T}_{+,m}$, the best rank-$m$ SVD approximation of $\hat{T}_{+}$: $$\hat{T}_{+,m} = \sum_{k=1}^m \hat{\lambda}_k \hat{\phi}_k(y) \hat{\psi}_k(z)$$.
3. Choose a subset $A \subset [0,1]$ (support of $x_{2m-1}$).
4. Compute a matrix $\hat{\eta}_A$ with entries $$\hat{\eta}_{lt} = \frac{1}{\hat{\lambda}_t} \int_A \hat{\phi}_l(y) \hat{f}(y,z,x_{2m-1}) \hat{\psi}_t(z) dy dz dx_{2m-1}$$.
5. Find eigenvectors $\hat{w}_k$ of $\hat{\eta}_A$.
6. Recover estimates for $f_{k1}$ by first forming $\hat{g}_k(y) = \sum_h \hat{w}_{kh} \hat{\phi}_h(y)$, normalizing to $\hat{h}_k = \hat{g}_k / \|\hat{g}_k\|_1$, and then marginalizing $\hat{f}_{k1} = \int \hat{h}_k dx_2 \dots dx_{m-1}$.

Theorem \ref{algo_error} (Correctness of Algorithm 1):

If $f$ is $(\mu,\zeta)$-estimable, $\|f_{kj}\|_\infty \le C$, and the set $A$ is chosen such that $\int_A f_{k(2m-1)}(x)dx$ are well-separated and bounded away from zero, then if $\|\hat{f}-f\|_2 \le \epsilon$ (for small $\epsilon$), Algorithm 1 outputs $\hat{f}_{k1}$ such that:

$$\|\hat{f}_{k1} - f_{\sigma(k)1}\|_2 \lesssim \epsilon$$

The error depends on $\epsilon$, constants $C,m$, incoherence $\mu$, proportion bound $\zeta$, and properties of set $A$ (measure $\mu_0$ and separation $\delta$). This algorithm relies only on incoherence, not linear independence. For $d > 2m-1$, it can be applied repeatedly to submodels.

Simulations:

The algorithm was tested on:

1. Conditional i.i.d. model ($m=3, d=5$, support size 4 for each $f_{kj}$).
2. Bernoulli mixture model ($m=3, d=5$, with $\alpha_{kj} = 0.1j + 0.2(k-1)$). An empirical estimate $\hat{f}$ was used from $n$ samples. The error $e = \sum_k \|f_{k1} - \hat{f}_{k1}\|_2$ was reported. Results showed a linear decay of log error with increasing log sample size, confirming the theoretical linear relationship between joint density error and component density error. The algorithm performed well even without linear independence.

Discussion

The paper significantly advances the understanding of high-dimensional nonparametric latent structure models by:

• Providing a unified identifiability theory beyond linear independence.
• Establishing quantitative convergence rates that show polynomial dependence on dimension.
• Proposing a practical recovery algorithm with theoretical guarantees under incoherence.

Future work includes:

• Refining identifiability conditions (e.g., removing the 3-partition requirement).
• Developing methods to utilize information from more than $2m-1$ variables more effectively for estimation.

The appendices contain detailed proofs for the theorems and lemmas presented. For instance, Appendix A covers proofs for Section 2 (Identifiability), Appendix B for Theorem \ref{Angle_thm}, Appendix C for Theorem \ref{Holder_rate}, and Appendix D for Algorithm 1 and Theorem \ref{algo_error}.