Penalized Comparison to Overfitting (PCO)

• PCO is a nonparametric model selection principle that compares candidate estimators with an overfitted reference and corrects for variance inflation using an explicit penalty.
• It is applied in wavelet projection and kernel density estimation to select optimal resolution or bandwidth without requiring plug-in bias estimators.
• PCO achieves oracle inequalities and adaptive minimax rates, offering robustness and computational advantages over traditional selection methods.

Penalized Comparison to Overfitting (PCO) is a data-driven selection principle for nonparametric estimation and model selection in which each candidate estimator is compared with an intentionally overfitted reference estimator, and the comparison is corrected by an explicit penalty that compensates for the resulting variance or complexity inflation. In its canonical statistical form, PCO is used to choose smoothing or resolution parameters—such as a kernel bandwidth or a wavelet multiresolution level—without requiring explicit plug-in estimation of the unknown bias term. Recent work develops PCO for wavelet projection density estimation, multivariate kernel density estimation, and weighted $\ell_p$-loss model selection, with oracle inequalities and adaptive minimax guarantees over Besov-type classes (Hoang et al., 9 Sep 2025, Varet et al., 2019, Lacour et al., 15 Apr 2025).

1. Core principle and statistical intuition

PCO is motivated by the standard bias–variance decomposition. In the wavelet density-estimation setting, for each resolution level $N$,

$$\mathbb{E}\left[\|f-\widehat f_N\|^2\right]=B_N+V_N,$$

where $B_N=\|f-\mathbb E[\widehat f_N]\|^2$ is the squared bias and $$V_N=\mathbb E[\|\mathbb E[\widehat f_N]-\widehat f_N\|^2]$$ is the variance (Hoang et al., 9 Sep 2025). The difficulty is that $B_N$ is unknown, so direct minimization of $B_N+V_N$ is infeasible.

The characteristic PCO heuristic is to use an overfitted estimator—typically the estimator computed at the highest admissible resolution or the smallest admissible bandwidth—as a proxy reference. In the wavelet formulation, $\|\widehat f_{N_{\max}}-\widehat f_N\|^2$ is used as a proxy for the remaining bias, but this empirical difference is itself noisy because it contains extra variance. The penalty term is therefore not ancillary; it is the mechanism that corrects for the variance inflation induced by the comparison to the overfitted reference (Hoang et al., 9 Sep 2025).

A common misunderstanding is to read “comparison to overfitting” as a rule that prefers overfitted models. The method does the opposite: it uses the overfitted estimator as a calibration point and then penalizes candidates to prevent the selection from drifting toward excessive variance. This is why the approach is described as fully data-driven while still targeting the bias–variance tradeoff (Hoang et al., 9 Sep 2025).

2. Canonical formulations in density estimation

For wavelet projection density estimation, Hoang et al. study projection estimators built from $\mathcal S$-regular, compactly supported wavelet bases. If $V_N$ is the linear space spanned by dilated-translated scaling functions at level $N$0, the projection estimator is

$N$1

PCO selects the multiresolution level through

$N$2

with

$N$3

and

$N$4

The selected estimator is $N$5 (Hoang et al., 9 Sep 2025).

In kernel density estimation, PCO takes an analogous form but the selected object is the bandwidth. For univariate kernel density estimation,

$N$6

and $N$7. The multivariate extension replaces $N$8 by a symmetric positive-definite bandwidth matrix $N$9: $$\mathbb{E}\left[\|f-\widehat f_N\|^2\right]=B_N+V_N,$$0 with $$\mathbb{E}\left[\|f-\widehat f_N\|^2\right]=B_N+V_N,$$1 and penalty

$$\mathbb{E}\left[\|f-\widehat f_N\|^2\right]=B_N+V_N,$$2

This formulation extends to diagonal and full bandwidth matrices, including parameterizations based on eigendecomposition (Varet et al., 2019).

Across these density-estimation instantiations, the overfitted reference is either the highest reasonable wavelet resolution $$\mathbb{E}\left[\|f-\widehat f_N\|^2\right]=B_N+V_N,$$3 or the smallest allowed bandwidth $$\mathbb{E}\left[\|f-\widehat f_N\|^2\right]=B_N+V_N,$$4 or $$\mathbb{E}\left[\|f-\widehat f_N\|^2\right]=B_N+V_N,$$5. The repeated structure is the defining formal pattern of PCO.

3. Oracle inequalities and adaptive rates

A central theoretical feature of PCO is the derivation of oracle inequalities. In the wavelet setting, the main result is a high-probability oracle inequality for the $$\mathbb{E}\left[\|f-\widehat f_N\|^2\right]=B_N+V_N,$$6-risk of the PCO-selected estimator. In paraphrased form, for all $$\mathbb{E}\left[\|f-\widehat f_N\|^2\right]=B_N+V_N,$$7, with high probability at least $$\mathbb{E}\left[\|f-\widehat f_N\|^2\right]=B_N+V_N,$$8,

$$\mathbb{E}\left[\|f-\widehat f_N\|^2\right]=B_N+V_N,$$9

where $B_N=\|f-\mathbb E[\widehat f_N]\|^2$0 is the projection of $B_N=\|f-\mathbb E[\widehat f_N]\|^2$1 onto scale $B_N=\|f-\mathbb E[\widehat f_N]\|^2$2, and the constants depend only on parameters of the method, not on $B_N=\|f-\mathbb E[\widehat f_N]\|^2$3 (Hoang et al., 9 Sep 2025). The interpretation given there is that the selected estimator performs nearly as well as the oracle choice of $B_N=\|f-\mathbb E[\widehat f_N]\|^2$4.

The same paper establishes adaptive minimax behavior over Besov classes. If $B_N=\|f-\mathbb E[\widehat f_N]\|^2$5 lies in a Besov smoothness class $B_N=\|f-\mathbb E[\widehat f_N]\|^2$6, then

$B_N=\|f-\mathbb E[\widehat f_N]\|^2$7

up to logarithmic and negligible factors, which is stated to match the minimax optimal rate for the class (Hoang et al., 9 Sep 2025).

For multivariate kernel density estimation, the numerical study is paired with theoretical guarantees: PCO satisfies an oracle inequality, with the optimal constant achieved at $B_N=\|f-\mathbb E[\widehat f_N]\|^2$8, and attains adaptive minimax rates over Nikol’skii or Besov classes (Varet et al., 2019). In both wavelet and kernel formulations, the method is therefore positioned not merely as a heuristic selector but as an estimator-selection rule with non-asymptotic risk control.

4. Extension to weighted $B_N=\|f-\mathbb E[\widehat f_N]\|^2$9-loss and nonparametric regression

PCO has also been generalized beyond $$V_N=\mathbb E[\|\mathbb E[\widehat f_N]-\widehat f_N\|^2]$$0-based density estimation. In the sequence model

$$V_N=\mathbb E[\|\mathbb E[\widehat f_N]-\widehat f_N\|^2]$$1

with i.i.d. centered sub-Gaussian noise $$V_N=\mathbb E[\|\mathbb E[\widehat f_N]-\widehat f_N\|^2]$$2, the model-based estimator indexed by a subset $$V_N=\mathbb E[\|\mathbb E[\widehat f_N]-\widehat f_N\|^2]$$3 is

$$V_N=\mathbb E[\|\mathbb E[\widehat f_N]-\widehat f_N\|^2]$$4

For weighted $$V_N=\mathbb E[\|\mathbb E[\widehat f_N]-\widehat f_N\|^2]$$5-loss,

$$V_N=\mathbb E[\|\mathbb E[\widehat f_N]-\widehat f_N\|^2]$$6

PCO selects

$$V_N=\mathbb E[\|\mathbb E[\widehat f_N]-\widehat f_N\|^2]$$7

and sets $$V_N=\mathbb E[\|\mathbb E[\widehat f_N]-\widehat f_N\|^2]$$8 (Lacour et al., 15 Apr 2025).

The analysis requires concentration inequalities for sums of $$V_N=\mathbb E[\|\mathbb E[\widehat f_N]-\widehat f_N\|^2]$$9-th powers of sub-Gaussian variables, which become sub-Weibull. The paper states that there exists a constant $B_N$0 such that, for all $B_N$1, with probability at least $B_N$2,

$B_N$3

This concentration behavior determines the form of the penalty, including the blockwise penalty

$B_N$4

and explains the “elbow” at $B_N$5: for $B_N$6, extra logarithmic factors may appear (Lacour et al., 15 Apr 2025).

Using suitable model collections and penalties, the paper derives oracle inequalities and minimax rates over Besov bodies $B_N$7, and then transfers these results to the functional model of nonparametric regression by applying PCO to empirical wavelet coefficients. The procedure is described as optimal up to logarithmic factors in all regimes considered (Lacour et al., 15 Apr 2025). This suggests that PCO is not limited to bandwidth or scale selection under quadratic risk, but can serve as a broader model-selection template for non-Euclidean losses.

5. Computational profile and comparison with competing methods

A recurring motivation for PCO is that it avoids some of the practical drawbacks of classical selectors. In the wavelet framework, compact support of the scaling function ensures feasible computation, and the procedure requires comparing each candidate level $B_N$8 only with a single reference estimator at $B_N$9, rather than performing pairwise comparisons over all model pairs. The paper explicitly contrasts this with Goldenshluger–Lepski and Lepski-type procedures, which are said to incur high computational cost and possible instability because they require risk estimation across all pairs of models (Hoang et al., 9 Sep 2025).

In multivariate kernel density estimation, the computational complexity is reported as

$B_N+V_N$0

which matches cross-validation methods. For the Gaussian kernel, the relevant $B_N+V_N$1 norms and cross-terms have analytical forms, making the implementation practical for moderate dimension. The same study recommends $B_N+V_N$2, reports a broad risk plateau around that value, and emphasizes that PCO can be applied to diagonal and full bandwidth matrices without algorithmic additional cost relative to standard competitors (Varet et al., 2019).

The competing procedures discussed in these works include penalized likelihood, Lepski’s method, Goldenshluger–Lepski, cross-validation, rule-of-thumb methods, and plug-in selectors. The drawbacks attributed to them are varied: explicit bias estimation, computational inefficiency, instability, or sensitivity to pilot choices. Against this background, PCO is presented as simpler and more direct: it relies on a single overfitted reference, requires no plug-in bias estimator, avoids subsampling, and in simulations is reported as robust, stable, and competitive or superior, especially for moderate to large sample sizes and in multivariate scenarios where classical methods can falter (Hoang et al., 9 Sep 2025, Varet et al., 2019).

6. Later reinterpretations and terminological extensions

The PCO principle has also become a reference point in adjacent literatures. In the two-sample testing paper on Complexity-Penalized MMD, PCO is explicitly cited as a related penalized-selection idea. The comparison drawn there is that both methods apply a complexity penalty to prevent overfitting in parameter selection, but the penalty in CP-MMD is derived from a two-sample uniform concentration inequality and is intended to support direct, grid-free optimization over continuous parametric classes, including deep kernels. That paper characterizes classical PCO as designed for discrete or very low-dimensional parameter selection such as bandwidth grids in density estimation (Ni et al., 7 May 2026).

A different reuse of the acronym appears in fast adversarial training. “FGSM-PCO” denotes a method for preventing catastrophic overfitting by fusing current and historical adversarial examples within a bi-level optimization framework, using an adaptive fusion ratio

$B_N+V_N$3

and a tailored loss

$B_N+V_N$4

Here, “Penalized Comparison to Overfitting” refers to a mechanism for preventing and reversing collapse of the inner maximization in adversarial training, not to estimator selection in nonparametric inference (Wang et al., 2024).

This terminological spread can obscure the statistical meaning of PCO. In its original and most developed form, PCO is a nonparametric model-selection methodology built around a comparison with an overfitted reference and a penalty calibrated to restore statistical validity. Subsequent usages preserve the rhetorical core—penalizing a comparison to an overfitted or collapsing solution—but adapt it to very different optimization problems (Hoang et al., 9 Sep 2025, Wang et al., 2024).