Minimax Estimation for Nonsmooth Functionals

• Minimax estimation for functional is a framework for designing estimators that minimize worst-case risk for nonsmooth targets using composite hypotheses and moment-matching priors.
• The methodology employs polynomial approximations and Hermite polynomial lifting to accurately estimate functionals like the L1 norm, effectively balancing bias and variance.
• This approach achieves minimax optimal rates with logarithmic sample costs, significantly advancing the theory and practice of nonregular statistical estimation.

Minimax estimation for a functional concerns the construction of estimators that achieve the smallest possible maximum risk over a parameter space, specifically targeting functionals that depend, often nonsmoothly, on the underlying parameter vector or function. In situations where the functional of interest lacks smoothness—such as $T(\theta) = (1/n)\sum_{i=1}^n |\theta_i|$—the theoretical and methodological challenges diverge sharply from those found in classical parametric or regular nonparametric estimation. When observing $Y \sim N(\theta, I_n)$, the estimation of nonsmooth functionals, particularly $T(\theta)$, requires tools from approximation theory, precise composite hypothesis testing, and the realization that classical quadratic or tangent-space methods are not adequate. The complexity in minimax risk rates and estimator construction for these problems reflects the deep interaction between functional analysis, statistical decision theory, and optimal approximation.

1. General Principle and Lower Bound: Composite Hypothesis Testing

Minimax estimation of nonsmooth functionals necessitates a framework beyond classical Le Cam two-point arguments. The central idea is to replace simple-vs-simple hypothesis tests by composite hypotheses involving pairs of priors $\mu_0$ and $\mu_1$, each supported on separate but interleaved subsets of the parameter space, with the functional values $T$ close in distribution. The lower bound for minimax risk is then characterized by three key quantities:

• The mean contrast $|m_1 - m_0|$ between the two priors for $T$,
• The variance $v_0^2$ under $\mu_0$,
• The chi-square divergence $I$ between the induced mixture distributions.

The critical constrained risk inequality is: $$\sup_{\theta \in \Theta} \mathbb{E}_\theta(\hat T - T(\theta))^2 \geq \frac{(|m_1 - m_0| - v_0 I)^2}{(I + 2)^2}$$ This bound structurally differs from smooth functionals, where point masses as priors suffice and the testing problem demands less delicate moment-matching.

2. Construction of Least Favorable Priors and the Role of Approximation Theory

For functionals such as $T(\theta) = (1/n) \sum |\theta_i|$, the nonsmoothness at $0$ (lack of Gâteaux or Fréchet differentiability) implies that simple local linearization cannot capture the hardest instance for estimation. Instead, the least favorable priors $\nu_0, \nu_1$ are constructed as moment-matched discrete or continuous measures on $[-1, 1]$ that agree up to moment $k$ (i.e., $\int t^\ell \nu_0(dt) = \int t^\ell \nu_1(dt)$ for $\ell = 0,\ldots, k$) but achieve a critical separation in $$\int |t| \nu_1(dt) - \int |t| \nu_0(dt) = 2\delta_k$$, where $\delta_k$ is the best polynomial approximation error of $|x|$ on $[-1, 1]$ by polynomials of degree $2k$.

This structure ensures that the two mixture distributions (formed by independently sampling coordinates from the two priors) have highly similar low-order moments with respect to data $Y$, making them nearly indistinguishable based on observed data, yet the functional values differ by a magnitude governed by the uniform polynomial approximation error: $$\delta_{2k}(|x|) = \min_{p \in \mathcal{P}_{2k}} \max_{x \in [-1, 1]} ||x| - p(x)|$$ where $\mathcal{P}_{2k}$ is the set of polynomials of degree at most $2k$. The Bernstein constant $$\beta^* = \lim_{k \to \infty} 2k \delta_{2k}(|x|) \approx 0.28017$$ quantifies the asymptotic hardness of the problem.

3. Minimax Optimal Estimator: Polynomial Approximation and Hermite Polynomial Lifting

Matching the lower bound requires an estimator that leverages the best (even) polynomial approximation to $|x|$ and unbiased estimators for each monomial moment. Specifically, given $Y_i \sim N(\theta_i, 1)$ ($\|\theta\|_\infty \leq M$),

1. Compute the best even-degree $2k$ polynomial $G^*_k(x) = \sum_{j=0}^k g^*_{2j} x^{2j}$ approximating $|x|$ over $[-M, M]$.
2. For each $j$, use Hermite polynomials $H_{2j}(y)$, exploiting the moment property $\mathbb{E}_{\theta_i}[H_{2j}(Y_i)] = \theta_i^{2j}$, to construct unbiased estimators $B_{2j} = (1/n) \sum_{i=1}^n H_{2j}(Y_i)$.
3. Aggregate using the polynomial coefficients: $\widehat T = \sum_{j=0}^k g^*_{2j} B_{2j}$ The choice of cutoff $k$ balances bias and variance; optimally, $k \propto \frac{\log n}{\log\log n}$. The mean squared error decomposes into two sources, with the dominant contribution coming from polynomial approximation error as bias and sampling variability as variance.

4. Minimax Rate: Exact Expression and Implications

The main result is that the minimax risk, for the bounded parameter case $\|\theta\|_\infty \leq M$, is

$$\inf_{\widehat T} \sup_{\theta} \mathbb{E}_\theta(\widehat T - T(\theta))^2 = \beta_*^2 M^2 \left( \frac{\log\log n}{\log n} \right)^2 (1 + o(1))$$

Contrasting with the $O(1/n)$ rate for smooth functionals, this risk is only logarithmic in $n$, reflecting the fundamental information-theoretic hardness from the nondifferentiability of $|x|$. No estimator (regardless of computational complexity) can surpass this rate.

Furthermore, the constructed estimator matches this rate, establishing its asymptotic sharp minimaxity. The estimator is also robust to the constraint $|\theta_i| \leq M$ being replaced by $|\theta_i| \leq o(\sqrt{\log n / \log\log n})$, at the cost of only higher-order terms.

5. Generalization and Broader Relevance

The composite hypothesis testing methodology, moment-matching priors, and polynomial approximation strategy generalize to other settings:

• Nonsmooth functionals in nonparametric models (e.g., $L_1$ norms in density estimation, or "excess mass" functionals for clustering), where the lack of differentiability of the target functional produces similarly slow minimax rates and precludes efficient locally quadratic estimation.
• Sparse and high-dimensional Gaussian models, where only a fraction of entries of $\theta$ are nonzero, can be handled by combining polynomial approximation with thresholding and adaptive procedures (cf. hybrid methods in related literature).
• The approach is applicable whenever the key difficulty is induced by the nonregular nature of the estimand (e.g., absolute values, indicator functionals).

6. Mathematical Structure: Key Relations and Technical Ingredients

Essential technical statements include:

• The minimax risk lower bound derived via composite hypotheses:

$$\sup_{\theta \in \Theta} \mathbb{E}_\theta(\widehat T - T(\theta))^2 \geq \frac{(|m_1 - m_0| - v_0 I)^2}{(I + 2)^2}$$

• The construction of moment-matched priors: for polynomials of degree $k$, there exist $\nu_0, \nu_1$ with matched moments up to $k$, but separated by $2\delta_k$ in expected absolute value.
• The final risk:

$$\inf_{\widehat T} \sup_{\theta \in \Theta_n(M)} \mathbb{E}(\widehat T - T(\theta))^2 = \beta_*^2 M^2 \left( \frac{\log\log n}{\log n} \right)^2 (1 + o(1))$$

where $$\Theta_n(M) = \{\theta \in \mathbb{R}^n: |\theta_i| \leq M\}.$$

7. Scientific and Practical Impact

This theory resolves a longstanding problem for minimax estimation of the $\ell_1$ norm in the normal mean model, establishing both nonasymptotic lower and sharp upper bounds. Beyond its intrinsic value for high-dimensional analysis, it offers a construction blueprint for optimal estimation of nonsmooth functionals:

• When nonregularity precludes direct plug-in or bias-corrected estimation, best polynomial approximation paired with orthogonal polynomial expansions (e.g., Hermite polynomials) supplies a broad and effective procedure.
• The approach yields explicit estimators and risk guarantees in scenarios where standard parametric or semiparametric regularity fails, and guides practical tradeoffs in bias, variance, and sample complexity.

This analytic methodology is transferable to various applied domains where functional estimation under information constraints or model nonregularity is critical, including calibration in high-throughput experiments, structural inference in signal processing, and statistical learning with resource limitations.