Likelihood-Ratio Distortion Metric

• Likelihood-Ratio Distortion Metric is a measure quantifying the maximum deviation in log-likelihood ratios due to data embeddings, crucial for preserving inferential accuracy.
• It is defined as the supremum difference between true and surrogate log-likelihood ratios over parameter pairs, ensuring asymptotic equivalence of hypothesis tests, MLEs, and Bayes factors.
• Applications include privacy-preserving inference and model selection, with neural frameworks demonstrating phase transitions when embedding dimensions align with model parameters.

The Likelihood-Ratio Distortion metric, denoted $\Delta_n$, quantifies the maximal error in log-likelihood ratios introduced by an embedding or representation. This metric emerges as the fundamental hinge for preserving inferential integrity in classical likelihood-based workflows—hypothesis testing, confidence intervals, model selection, and Bayesian comparison—when high-dimensional data is compressed by learned representations or neural embeddings. $\Delta_n$ plays a central role in delineating when and how surrogate models or compressed representations can safely replace raw data without compromising statistical conclusions.

1. Formal Definition and Significance

Consider $n$ i.i.d. samples $X_1,\ldots,X_n$ from a parametric family $\{P_\theta : \theta \in \Theta\}$, with log-likelihood $L_n(\theta) = \sum_{i=1}^n \ell_\theta(X_i)$ and $\ell_\theta(x) = \log p(x | \theta)$. An embedding $T_\phi: X \to \mathbb{R}^m$ produces a dataset summary $S_\phi(X_{1:n}) = n^{-1}\sum_{i=1}^n T_\phi(X_i)$, decoded by $h_\psi$ to yield surrogate log-likelihood $$\tilde{L}_n(\theta) = n \cdot h_\psi(\theta, S_\phi(X_{1:n}))$$.

The Likelihood-Ratio Distortion is defined as:

$$\Delta_n = \sup_{\theta, \theta' \in \Theta} \left| \left[ L_n(\theta) - L_n(\theta') \right] - \left[ \tilde{L}_n(\theta) - \tilde{L}_n(\theta') \right] \right|.$$

This measures the worst-case discrepancy in log-likelihood ratios over all $\theta, \theta'$ pairs. Since classical inferential procedures—e.g., likelihood-ratio tests, confidence intervals, Bayes factors—depend solely on differences in log-likelihood, controlling $\Delta_n$ is necessary and sufficient for preserving inference (Akdemir, 27 Dec 2025).

2. Hinge Theorem and Asymptotic Equivalence

The Hinge Theorem establishes that if $\Delta_n = o_p(1)$, all likelihood-ratio–based tests, Bayes factors, and surrogate maximum likelihood estimators (MLEs) are asymptotically preserved.

Let $$\epsilon_n = \sup_{\theta \in \Theta}|n^{-1}L_n(\theta) - h_\psi(\theta, S_\phi)|$$ be the pointwise error; then $\Delta_n \leq 2n \epsilon_n$, showing that pointwise error bounds ratio distortion. Under regularity conditions (identifiability, smoothness, positive-definite Fisher information), the theorem proceeds as follows:

• Test Preservation: For likelihood-ratio statistics $$\Lambda_n = 2[\sup_\theta L_n(\theta) - L_n(\theta_0)]$$, the surrogate $\tilde{\Lambda}_n$ satisfies $$|\tilde{\Lambda}_n - \Lambda_n| \leq 4\Delta_n=o_p(1)$$. By Wilks’ theorem, the asymptotic distribution and sizes/powers are identical.
• MLE Equivalence: Let $\hat{\theta} = \arg\max L_n$, $\tilde{\theta} = \arg\max \tilde{L}_n$. The quadratic expansion yields $$L_n(\hat{\theta}) - L_n(\tilde{\theta}) \simeq (n/2)(\hat{\theta}-\tilde{\theta})^\top I(\theta_0)(\hat{\theta}-\tilde{\theta})$$. Therefore, $||\hat{\theta} - \tilde{\theta}|| = o_p(n^{-1/2})$.
• Model Selection and Bayes Factors: Metrics such as $\mathrm{AIC} = -2L_n(\hat{\theta}) + 2k$ and log-Bayes factors suffer at most $O(n\epsilon_n)=o_p(1)$ error.

If $\Delta_n \not\to 0$, likelihood ratio preservation fails, disrupting inferential validity. Therefore, $\Delta_n=o_p(1)$ is both necessary and sufficient (Akdemir, 27 Dec 2025).

3. Impossibility of Universal Preservation

Theorem 3.4 ("No Free Lunch") demonstrates that for universal likelihood preservation ($\Delta_n=0$ for all densities in a nonparametric class $\mathcal{F}$), $T_\phi$ must be $\mu$-almost-surely injective. That is, only invertible embeddings can guarantee zero distortion for arbitrary model classes. For $k$-dimensional exponential families, exact preservation demands $m\geq k$ and, at $m=k$, the embedding must recover the minimal sufficient statistic invertibly. This establishes a sharp lower bound on embedding dimensionality and affirms that model-class specificity is unavoidable—universal compression without distortion is generally infeasible (Akdemir, 27 Dec 2025).

4. Constructive Neural Frameworks

Likelihood-preserving embeddings can be constructed via neural approximate sufficiency:

• Encoder: $T_\phi: X \to \mathbb{R}^m$
• Summary: $S_\phi = n^{-1}\sum_i T_\phi(X_i)$
• Decoder: $h_\psi: \Theta \times \mathbb{R}^m \to \mathbb{R}$

Training minimizes $$\mathcal{L}_\text{point}(\phi,\psi) = \mathbb{E}_{\theta \sim \Pi}\mathbb{E}_{X_{1:n} \sim P_\theta}\left[ n^{-1}L_n(\theta) - h_\psi(\theta, S_\phi(X_{1:n})) \right]^2$$. By Jensen's inequality, $$\mathbb{E}[\Delta_n] \leq 2n \sqrt{\mathcal{L}_\text{point}}$$.

For a parameter grid of size $G$ with $L$-Lipschitz log-likelihood, $\Delta_n$ admits the bound: $$\Delta_n \leq \sqrt{\epsilon}G + 2nL \operatorname{diam}(\Theta) / G$$, with optimal scaling $G \approx \sqrt{n}$ yielding $$\Delta_n=O(n^{1/2}\epsilon^{1/2} + n^{1/2}L\,\operatorname{diam}(\Theta))$$. Sample complexity results show that $N = \tilde{O}((d_T+d_H)/\alpha^2 \log(1/\delta))$ synthetic datasets suffice for generalization within tolerance $\alpha$ with high probability (Akdemir, 27 Dec 2025).

5. Statistical and Information-Theoretic Connections

In the context of classical rate-distortion (RD) and information bottleneck (IB) theory, the Likelihood-Ratio Distortion metric emerges naturally. For two densities $\pi_0(z)$ and $\pi_1(z)$, the sufficient statistic $\varphi(z) = \log \pi_1(z) - \log \pi_0(z)$ yields a one-parameter exponential family:

$$\pi_\beta(z) = \pi_0(z) \exp\{ \beta \varphi(z) - \psi(\beta) \} = \pi_0(z)^{1-\beta} \pi_1(z)^\beta / Z_\beta,$$

where $\psi(\beta)$ is the log-partition function. The negative log-likelihood-ratio, $$d(x,z) = -\varphi(x,z) = -\log\left(\pi_1(x,z)/\pi_0(z)\right)$$, serves as the distortion measure for RD optimization:

$$q_\beta(z|x) \propto \pi_0(z) \exp\{-\beta d(x,z)\}.$$

Key quantities include $$D(\beta) = \mathbb{E}_{q_\beta}[d(x,z)] = -\psi'(\beta)$$, $$R(\beta) = D_{KL}[q_\beta \| \pi_0] = \beta D(\beta) - \psi(\beta)$$. This variational framework steers the trade-off between rate and distortion and connects to Neyman-Pearson hypothesis testing via size-power exponents (Brekelmans et al., 2020).

6. Empirical Validation and Phase Transitions

Experiments on Gaussian and Cauchy distributions illustrate distinct behaviors:

• Gaussian ($\mathcal{N}(\mu, \sigma^2)$): The family admits an exact 2-dimensional sufficient statistic $T=(\sum X_i, \sum X_i^2)$. For $n=100$:
  • At $m=1$, $\epsilon_n \approx 1.74$, $\Delta_n \approx 148.2$.
  • At $m=2$, both $\epsilon_n$ and $\Delta_n$ drop to machine precision ($\sim 10^{-13}$), exhibiting a sharp phase transition as predicted by Theorem 3.5.
• Cauchy $(\theta, 1)$: Lacking a finite sufficient statistic, increases in $m$ (empirical quantile embeddings) yield smooth decreases in both $\epsilon_n$ and $\Delta_n$ (e.g., $\Delta_n$ from 1.2 to 0.3 as $m$ increases from 1 to 8), but never reach zero, matching Pitman–Koopman–Darmois non-existence (Akdemir, 27 Dec 2025).

7. Applications: Privacy-Preserving Inference

In distributed clinical trials, $\Delta_n$ enables valid statistical inference without raw patient-level data sharing. For multi-site linear regression (five sites, $n=200$ per site, $p=4$ covariates):

• Exact sufficient summary (size 16) achieves $\Delta_n=0$, perfectly reproducing pooled-data power.
• Compressed embedding ($m=8$) with small $\Delta_n$ attains $\sim99\%$ efficiency.
• Meta-analysis (no cross-site covariances) yields large $\Delta_n$, leading to $\sim50\%$ power loss.

Guidelines for practical use include matching embedding dimension to parameter count, training on synthetic data from the assumed model, and validating $\Delta_n$ on held-out parameters to ensure $o_p(1)$ scaling. This suggests a direct practical protocol for likelihood-preserving federated inference in privacy-sensitive domains (Akdemir, 27 Dec 2025).

---

The Likelihood-Ratio Distortion metric $\Delta_n$ provides the rigorous basis for the design, analysis, and deployment of compressed representations in statistical inference workflows. Its tight theoretical characterization, operational bounds, and empirical validations position $\Delta_n$ as the pivotal quantity for bridging modern machine learning embeddings with classical likelihood theory in both parametric and nonparametric regimes (Akdemir, 27 Dec 2025, Brekelmans et al., 2020).