---
title: Minimum Rényi Pseudodistance Estimators
url: https://www.emergentmind.com/topics/minimum-renyi-pseudodistance-estimators
type: topic
---

# Minimum Rényi Pseudodistance Estimators

Minimum Rényi pseudodistance estimators (MRPEs) constitute a one-parameter family of robust statistical estimators and associated Wald-type statistical tests, constructed by minimizing the Rényi pseudodistance—a divergence measure parametrized by an order $\alpha > 0$—between the assumed parametric model and the empirical distribution. MRPEs interpolate smoothly between maximum likelihood estimators (MLE, recovered as $\alpha\to0$) and highly robust alternatives with bounded influence functions. Their applications include linear and logistic regression, normal mean and scale inference, bivariate normal models, multiple regression with fixed design, and high-dimensional scenarios, especially in the presence of contamination or model misspecification. MRPEs have explicit score-type estimating equations, asymptotic normality with tractable variance structure, and can be used as the foundation for robust Wald-type tests. Robustness/efficiency trade-offs and the choice of the tuning parameter $\alpha$ are supported by extensive theoretical and simulation studies [2102.12282][2402.02867][2202.00982][1104.1541].

## 1. Rényi Pseudodistance: Definition and Properties

For two distributions $P$ and $Q$ on a common measurable space (with respective densities $p$ and $q$), the Rényi pseudodistance (also known as the Rényi pseudodivergence) of order $\alpha > 0$ is
\[
D_\alpha(P,Q) = \frac{1}{1+\alpha} \log \int p(x)^{1+\alpha}\,dx - \frac{1}{\alpha}\log\int p(x)^\alpha q(x)\,dx.
\]
For empirical distributions $Q = P_n$, the pseudodistance becomes
\[
D_\alpha(P_\theta,P_n) = \frac{1}{1+\alpha}\log \int p_\theta(x)^{1+\alpha}\,dx - \frac{1}{\alpha}\log\left( \frac{1}{n}\sum_{i=1}^n p_\theta(X_i)^\alpha \right).
\]
As $\alpha \downarrow 0$, $D_\alpha(P, Q)$ converges to the Kullback–Leibler divergence $D_{KL}(Q\,\|\,P)$. Positivity and decomposability properties ensure suitability as an estimation criterion [1104.1541][2402.02867].

For independent non-identically distributed (i.n.i.d.) data $Y_1,\dots,Y_n$ with corresponding model densities $f_i(y;\theta)$,
\[
R_\alpha(P_\theta,P_n) = \frac{1}{\alpha(1+\alpha)}\sum_{i=1}^n \left( \int f_i^{1+\alpha} - (1+\alpha) f_i(Y_i;\theta)^\alpha \right)
\]
serves as the objective function. The prefactor ensures the formal limiting connection to the log-likelihood as $\alpha\rightarrow 0$ [2102.12282].

## 2. Minimum Rényi Pseudodistance Estimation

The MRPE is defined as the minimizer of the Rényi pseudodistance with respect to the parameter,
\[
\hat\theta_\alpha = \arg\min_{\theta} D_\alpha(P_\theta, P_n).
\]
Alternatively, maximizing $H_n^{(\alpha)}(\theta)$ is also valid:
\[
H_n^{(\alpha)}(\theta) = \frac{1}{1+\alpha}\log \int p_\theta^{1+\alpha} - \frac{1}{\alpha}\log\left( \frac{1}{n}\sum_{i=1}^n p_\theta(X_i)^\alpha \right).
\]
For i.n.i.d. models,
\[
\hat\theta_\alpha = \arg\min_{\theta} R_\alpha(P_\theta, P_n)
\]
with the empirical/robustification features controlled by $\alpha$.

MRPEs can be explicitly formulated for normal location–scale inference, linear regression, polytomous logistic regression, and bivariate normal models [2102.12282][2402.02867][2202.00982][1104.1541]. Importantly, as $\alpha \to 0$, MRPEs coincide exactly with MLEs.

## 3. Estimating Equations and Algorithms

Differentiation of the objective produces “score-type” estimating equations. For i.i.d. data:
\[
0 = \frac{\partial}{\partial \theta} D_\alpha(P_\theta,P_n) = -\frac{1}{\alpha} \frac{\sum_{i=1}^n p_\theta(X_i)^{\alpha-1} \dot p_\theta(X_i)}{ \sum_{i=1}^n p_\theta(X_i)^\alpha } + \frac{1}{1+\alpha} \frac{\int p_\theta^\alpha \dot p_\theta }{ \int p_\theta^{1+\alpha} }
\]
with $\dot p_\theta$ the parameter gradient of the density.

For i.n.i.d. data, a compact expression for the estimating equation is:
\[
\frac{\partial H_n^\alpha(\theta)}{\partial \theta_j} = \frac{1}{n} \sum_{i=1}^n \left\{ \frac{ \int f_i^{\alpha+1} u_{i,j} }{ \int f_i^{\alpha+1} } - \frac{ f_i(Y_i;\theta)^\alpha u_{i,j}(Y_i;\theta) }{ \int f_i^{\alpha+1} } \right\} = 0
\]
where $u_{i,j}(y;\theta)=\partial \log f_i(y;\theta)/\partial \theta_j$ [2102.12282].

Iterative reweighting (IRLS) approaches are effective in models such as bivariate normals. The iterative weights and parameter updates are expressed directly in terms of current parameter estimates and evaluated density values [2202.00982].

## 4. Asymptotics and Influence Function

Under standard regularity, MRPEs are consistent and asymptotically normal. Specifically,
\[
\sqrt{n}(\hat\theta_\alpha - \theta_0) \xrightarrow{d} N(0, V_\alpha)
\]
with
\[
V_\alpha = J_\alpha^{-1} K_\alpha J_\alpha^{-1}
\]
where $J_\alpha = E_{P_{\theta_0}}[ -\nabla_\theta \psi_\alpha(X;\theta_0)]$, $K_\alpha = \operatorname{Var}_{P_{\theta_0}}[ \psi_\alpha(X;\theta_0) ]$, and $\psi_\alpha$ as in the estimating equation [2402.02867][2102.12282][1104.1541]. At $\alpha=0$, both $J_0$ and $K_0$ equal the Fisher information.

The influence function (IF) for MRPE at $P_{\theta_0}$ is
\[
\mathrm{IF}(x;T_\alpha,P_{\theta_0}) = J_\alpha^{-1} \psi_\alpha(x;\theta_0)
\]
which is bounded in $x$ for $\alpha>0$. For MLE ($\alpha=0$), the IF is unbounded, highlighting the robustifying effect of nonzero $\alpha$. In i.n.i.d. models and regression, IF formulae are available in closed form, ensuring B-robustness under mild contamination [2102.12282][1104.1541].

## 5. Efficiency, Robustness, and the Choice of $\alpha$

MRPEs furnish a continuum between the efficient but nonrobust MLE ($\alpha=0$) and estimators with maximal outlier resistance for larger $\alpha$. The loss in Fisher information (asymptotic efficiency) is typically minor for moderate $\alpha\leq 0.2$ but the gain in robustness—e.g., bounded influence, minuscule bias and mean squared error under contamination—is substantial in finite samples [1104.1541][2402.02867]. The asymptotic relative efficiency (ARE) with respect to MLE is, for example, $\approx 0.99$ for $\alpha=0.2$ and $\approx 0.96$ for $\alpha=0.4$ in the polytomous logistic regression model [2402.02867].

A practical selection for $\alpha$ lies in $[0.1,0.5]$, with specific choices often guided by empirical MSE minimization (e.g., Warwick–Jones method), though this could be computationally costly and a fixed value suffices in many scenarios [2202.00982]. Monte Carlo studies consistently validate the robustness–efficiency trade-off [1104.1541].

## 6. Applications: Regression and Multivariate Models

In multiple linear regression ($Y_{i}=X_{i}^{T}\beta+\epsilon_i$), the MRPE objective is:
\[
\max_{\beta,\sigma} \sum_{i=1}^n \sigma^{-\alpha/(\alpha+1)} \exp\left[ -\frac{\alpha}{2} \left( \frac{Y_i - X_i^T\beta}{\sigma} \right)^2 \right]
\]
with estimating equations:
\[
\begin{cases}
\sum_{i=1}^n e^{ -\frac{\alpha}{2} r_i^2 } r_i X_i = 0\\
\sum_{i=1}^n e^{ -\frac{\alpha}{2} r_i^2 } \left\{ r_i^2 - \frac{1}{1+\alpha} \right\} = 0
\end{cases}
\]
where $r_i = (Y_i - X_i^T\beta)/\sigma$ [2102.12282][1104.1541].

In the bivariate normal setting with $\theta = (\mu_1,\mu_2,\sigma_1,\sigma_2,\rho)$, IRLS algorithms allow efficient computation of MRPEs and operate by iteratively updating weighted means, variances, and covariances [2202.00982].

For polytomous logistic regression, MRPEs demonstrate high classification robustness and maintain low misclassification rates under response-label contamination for $\alpha\approx 0.3$–$0.5$ [2402.02867].

## 7. Robust Wald-Type Tests

For hypotheses $H_0: m(\theta) = m_0$, Wald-type test statistics based on MRPEs follow:
\[
W_{n,\alpha} = n\left(m(\hat\theta_\alpha) - m_0\right)^{\top} \left[ \operatorname{Var}\left(m(\hat\theta_\alpha)\right) \right]^{-1} \left(m(\hat\theta_\alpha) - m_0\right)
\]
Under $H_0$, $W_{n,\alpha}$ is asymptotically $\chi^2$, and for $\alpha>0$, test level remains near nominal under moderate contamination, while classical Wald tests with MLE break down [2402.02867][2102.12282][2202.00982]. MRPE-based Wald tests are thus directly robustified by the properties of the estimators themselves.

---

**References**
- [2102.12282] Estimation and testing on independent not identically distributed observations based on Rényi's pseudodistances
- [2402.02867] A new robust approach for the polytomous logistic regression model based on Rényi's pseudodistances
- [2202.00982] Robust approach for comparing two dependent normal populations through Wald-type tests based on Rényi's pseudodistance estimators
- [1104.1541] Decomposable Pseudodistances and Applications in Statistical Estimation

Source: https://www.emergentmind.com/topics/minimum-renyi-pseudodistance-estimators