Logarithmic Norm Relative Entropy (LNRE)

• LNRE is a two-parameter generalization of the Kullback–Leibler divergence that defines robust estimation methods applicable to various statistical models.
• The MLNREE utilizes a generalized likelihood approach to produce fixed-dimensional sufficient statistics, ensuring robustness even in small samples or contaminated data.
• The framework extends classical theorems, such as Fisher–Darmois, Rao–Blackwell, and Cramér–Rao, to establish optimality and variance bounds under divergence-based inference.

The Logarithmic Norm Relative Entropy (LNRE) is a two-parameter generalization of the Kullback–Leibler divergence. Originally introduced for robustness in parametric estimation, LNRE encompasses, as special cases, the standard likelihood divergence and the Logarithmic Density-Power Divergence (LDPD). LNRE-based estimation procedures are fundamental in defining robust statistical methods that are less sensitive to outliers than traditional maximum likelihood estimation (MLE), while retaining explicit links to information theory and sufficiency. The LNRE estimator (termed MLNREE) can be cast within a generalized likelihood framework, and for a broad family of models—namely, the $\mathcal{M}^{(\alpha, \beta)}$-family—it admits a fixed-dimensional sufficient statistic irrespective of sample size. This generalization enables the extension of classical results, such as the Fisher–Darmois–Koopman–Pitman theorem, the Rao–Blackwell theorem, and the Cramér–Rao inequality, under the new divergence paradigm (Singh et al., 15 Oct 2025).

1. Definition and Properties of LNRE

The Logarithmic Norm Relative Entropy between probability density functions $g$ and $f$ on a common support $S$ is given by

$$RE_{\alpha,\beta}^{LN}(g\|f) = \frac{\alpha}{\beta(\beta-\alpha)}\log\int_S g(y)^\beta f(y)^{\alpha-\beta}\,dy - \frac{1}{\beta-\alpha}\log\int_S g(y)^\alpha\,dy + \frac{1}{\beta}\log\int_S f(y)^\alpha\,dy,$$

where $\alpha>0$ and $\beta\in\mathbb{R}$ with $\beta\neq\alpha$. In the limit $\beta\to\alpha$,

$$RE_{\alpha,\alpha}^{LN}(g\|f) = \frac{\int g^\alpha\log(g/f)}{\int g^\alpha} + \frac{1}{\alpha}\log\left(\frac{\int f^\alpha}{\int g^\alpha}\right).$$

As $g$0, LNRE recovers the Kullback–Leibler (KL) divergence, $g$1, while specializations yield the LDPD (for $g$2) and encompass the Density-Power and Tsallis/Rényi divergences. This flexibility allows the tuning of robustness via $g$3 and $g$4 (Singh et al., 15 Oct 2025).

2. Generalized Likelihood and MLNREE

Within the parametric estimation setup, observations $g$5 are modeled i.i.d. from an unknown density $g$6, and a parametric family $g$7 is considered. Substituting $g$8 by its empirical estimator $g$9 in LNRE, one maximizes the generalized log-likelihood

$f$0

The corresponding first-order condition yields an estimating equation whose solution $f$1 is termed the Minimum LNRE Estimator (MLNREE). This estimator extends the robustness characteristics of the density-power-based methods, with explicit likelihood-like optimality criteria (Singh et al., 15 Oct 2025).

3. $f$2-Family and Sufficiency

A central concept in the LNRE framework is the $f$3-power-law family, which generalizes the classical exponential family to LNRE's divergence context. Densities in this family are of the form

$f$4

where $f$5 normalizes the density, $f$6 and $f$7 are fixed basis functions. A fundamental result generalizes the Fisher–Darmois–Koopman–Pitman theorem: A parametric family admits a sufficient statistic of fixed (parameter) dimension for $f$8 if and only if it is an $f$9-family. For a regular model, the minimal sufficient statistic can be explicitly constructed as

$S$0

Importantly, the dimension of the sufficient statistic does not depend on sample size $S$1 (Singh et al., 15 Oct 2025).

4. Generalized Rao–Blackwell and Cramér–Rao Theorems

In the LNRE framework, the “deformed” density—arising from exponentiating the generalized log-likelihood—supports sufficiency and optimality analogously to the classical likelihood case. Under this density, the generalized Rao–Blackwell theorem ensures that conditional expectations with respect to the sufficient statistic reduce variance, with equality only for functions thereof. The generalized Cramér–Rao inequality provides a lower bound for the variance of unbiased estimators:

$S$2

where the generalized Fisher information $S$3 is defined with respect to the deformed density. In general, the minimal sufficient statistic does not attain this lower bound, except in the classical limit ($S$4), unlike in the exponential family with standard likelihood (Singh et al., 15 Oct 2025).

5. Limiting Case: Recovery of Classical Likelihood

The classical MLE theory and exponential family behavior are restored as $S$5. In this limit, LNRE reduces to KL divergence, the $S$6-family collapses to the exponential family, and the sufficient statistics, deformed densities, and variance bounds revert to their standard forms. This property unifies and extends the landscape of likelihood-based, density-power-based, and divergence-based statistical estimation (Singh et al., 15 Oct 2025).

6. MLNREE for Student’s Distributions: Closed Forms and Robustness

The Student’s $S$7 family, with known degrees of freedom $S$8, can be parametrized as an $S$9-model via a specific relationship between $$RE_{\alpha,\beta}^{LN}(g\|f) = \frac{\alpha}{\beta(\beta-\alpha)}\log\int_S g(y)^\beta f(y)^{\alpha-\beta}\,dy - \frac{1}{\beta-\alpha}\log\int_S g(y)^\alpha\,dy + \frac{1}{\beta}\log\int_S f(y)^\alpha\,dy,$$0 and $$RE_{\alpha,\beta}^{LN}(g\|f) = \frac{\alpha}{\beta(\beta-\alpha)}\log\int_S g(y)^\beta f(y)^{\alpha-\beta}\,dy - \frac{1}{\beta-\alpha}\log\int_S g(y)^\alpha\,dy + \frac{1}{\beta}\log\int_S f(y)^\alpha\,dy,$$1. The MLNREE yields closed-form estimators for location and scale,

$$RE_{\alpha,\beta}^{LN}(g\|f) = \frac{\alpha}{\beta(\beta-\alpha)}\log\int_S g(y)^\beta f(y)^{\alpha-\beta}\,dy - \frac{1}{\beta-\alpha}\log\int_S g(y)^\alpha\,dy + \frac{1}{\beta}\log\int_S f(y)^\alpha\,dy,$$2

with a similar explicit formula for $$RE_{\alpha,\beta}^{LN}(g\|f) = \frac{\alpha}{\beta(\beta-\alpha)}\log\int_S g(y)^\beta f(y)^{\alpha-\beta}\,dy - \frac{1}{\beta-\alpha}\log\int_S g(y)^\alpha\,dy + \frac{1}{\beta}\log\int_S f(y)^\alpha\,dy,$$3 involving integrals of the Student-$$RE_{\alpha,\beta}^{LN}(g\|f) = \frac{\alpha}{\beta(\beta-\alpha)}\log\int_S g(y)^\beta f(y)^{\alpha-\beta}\,dy - \frac{1}{\beta-\alpha}\log\int_S g(y)^\alpha\,dy + \frac{1}{\beta}\log\int_S f(y)^\alpha\,dy,$$4 kernel. When $$RE_{\alpha,\beta}^{LN}(g\|f) = \frac{\alpha}{\beta(\beta-\alpha)}\log\int_S g(y)^\beta f(y)^{\alpha-\beta}\,dy - \frac{1}{\beta-\alpha}\log\int_S g(y)^\alpha\,dy + \frac{1}{\beta}\log\int_S f(y)^\alpha\,dy,$$5, these expressions reduce to the MLE (and the MDPDE/MLDPDE). Simulation studies show that for data contaminated with outliers, the classical estimators' variance inflates, while the MLNREE remains stable for appropriately tuned $$RE_{\alpha,\beta}^{LN}(g\|f) = \frac{\alpha}{\beta(\beta-\alpha)}\log\int_S g(y)^\beta f(y)^{\alpha-\beta}\,dy - \frac{1}{\beta-\alpha}\log\int_S g(y)^\alpha\,dy + \frac{1}{\beta}\log\int_S f(y)^\alpha\,dy,$$6 (Singh et al., 15 Oct 2025).

Performance Comparison for Student-$$RE_{\alpha,\beta}^{LN}(g\|f) = \frac{\alpha}{\beta(\beta-\alpha)}\log\int_S g(y)^\beta f(y)^{\alpha-\beta}\,dy - \frac{1}{\beta-\alpha}\log\int_S g(y)^\alpha\,dy + \frac{1}{\beta}\log\int_S f(y)^\alpha\,dy,$$7 Model (Key Results):

Contamination	MDPDE/MLDPDE $$RE_{\alpha,\beta}^{LN}(g\|f) = \frac{\alpha}{\beta(\beta-\alpha)}\log\int_S g(y)^\beta f(y)^{\alpha-\beta}\,dy - \frac{1}{\beta-\alpha}\log\int_S g(y)^\alpha\,dy + \frac{1}{\beta}\log\int_S f(y)^\alpha\,dy,$$8	Best MLNREE $$RE_{\alpha,\beta}^{LN}(g\|f) = \frac{\alpha}{\beta(\beta-\alpha)}\log\int_S g(y)^\beta f(y)^{\alpha-\beta}\,dy - \frac{1}{\beta-\alpha}\log\int_S g(y)^\alpha\,dy + \frac{1}{\beta}\log\int_S f(y)^\alpha\,dy,$$9
0%	$\alpha>0$0	$\alpha>0$1
5%	$\alpha>0$2	$\alpha>0$3
...	...	...

For distributions lacking a closed-form MLNREE, e.g., Student-$\alpha>0$4 ($\alpha>0$5), one may maximize numerically in each parameter, and again the MLNREE provides significantly improved robustness.

7. Practical Guidance, Real Data, and Extensions

The MLNREE has demonstrated practical utility on empirical datasets with known outliers, such as Newcomb’s light-speed residuals, where model fit (as measured by Kolmogorov–Smirnov statistics) is optimized for non-classical values of the tuning parameter $\alpha>0$6 ($\alpha>0$7), indicating effective outlier down-weighting. Parameter selection for $\alpha>0$8 is best approached via cross-validation, e.g., by minimizing the KS distance or out-of-sample deviance. The LNRE thus unifies maximum likelihood and divergence-based robust inference, supporting explicit optimality, sufficiency, and robust parameter recovery in contaminated regimes. The development of a full asymptotic theory for continuous MLNREE estimation remains a subject of ongoing research (Singh et al., 15 Oct 2025).