---
title: Penalized Comparison to Overfitting (PCO)
url: https://www.emergentmind.com/topics/penalized-comparison-to-overfitting-pco
type: topic
---

# Penalized Comparison to Overfitting (PCO)

Penalized Comparison to Overfitting (PCO) is a data-driven selection principle for nonparametric estimation and model selection in which each candidate estimator is compared with an intentionally overfitted reference estimator, and the comparison is corrected by an explicit penalty that compensates for the resulting variance or complexity inflation. In its canonical statistical form, PCO is used to choose smoothing or resolution parameters—such as a kernel bandwidth or a wavelet multiresolution level—without requiring explicit plug-in estimation of the unknown bias term. Recent work develops PCO for wavelet projection density estimation, multivariate kernel density estimation, and weighted \(\ell_p\)-loss model selection, with oracle inequalities and adaptive minimax guarantees over Besov-type classes [2509.07800, 1902.01075, 2504.11217].

## 1. Core principle and statistical intuition

PCO is motivated by the standard bias–variance decomposition. In the wavelet density-estimation setting, for each resolution level \(N\),
\[
\mathbb{E}\left[\|f-\widehat f_N\|^2\right]=B_N+V_N,
\]
where \(B_N=\|f-\mathbb E[\widehat f_N]\|^2\) is the squared bias and \(V_N=\mathbb E[\|\mathbb E[\widehat f_N]-\widehat f_N\|^2]\) is the variance [2509.07800]. The difficulty is that \(B_N\) is unknown, so direct minimization of \(B_N+V_N\) is infeasible.

The characteristic PCO heuristic is to use an overfitted estimator—typically the estimator computed at the highest admissible resolution or the smallest admissible bandwidth—as a proxy reference. In the wavelet formulation, \(\|\widehat f_{N_{\max}}-\widehat f_N\|^2\) is used as a proxy for the remaining bias, but this empirical difference is itself noisy because it contains extra variance. The penalty term is therefore not ancillary; it is the mechanism that corrects for the variance inflation induced by the comparison to the overfitted reference [2509.07800].

A common misunderstanding is to read “comparison to overfitting” as a rule that prefers overfitted models. The method does the opposite: it uses the overfitted estimator as a calibration point and then penalizes candidates to prevent the selection from drifting toward excessive variance. This is why the approach is described as fully data-driven while still targeting the bias–variance tradeoff [2509.07800].

## 2. Canonical formulations in density estimation

For wavelet projection density estimation, Hoang et al. study projection estimators built from \(\mathcal S\)-regular, compactly supported wavelet bases. If \(V_N\) is the linear space spanned by dilated-translated scaling functions at level \(N\), the projection estimator is
\[
\widehat f_N(x)=\sum_{k\in\mathbb Z}\widehat\alpha_{N,k}\,\varphi_{N,k}(x),
\qquad
\widehat\alpha_{N,k}=\frac1n\sum_{i=1}^n \varphi_{N,k}(X_i).
\]
PCO selects the multiresolution level through
\[
\mathrm{Crit}(N):=\|\widehat f_{N_{\max}}-\widehat f_N\|^2+\mathrm{pen}_\lambda(N),
\]
with
\[
\mathrm{pen}_\lambda(N)
=
\frac{(2A+1)^2\Phi_0^2\left(\lambda 2^{N+1}-(N_{\max}-N)2^{N_{\max}}\right)}{n},
\]
and
\[
\widehat N=\underset{N\in\mathcal H}{\operatorname{argmin}}\ \mathrm{Crit}(N),
\qquad
\mathcal H=\{N\in\mathbb N: N<n/\log n\}.
\]
The selected estimator is \(\widehat f_{\widehat N}\) [2509.07800].

In kernel density estimation, PCO takes an analogous form but the selected object is the bandwidth. For univariate kernel density estimation,
\[
l_{\rm PCO}(h)
=
\|\widehat f_{h_{\min}}-\widehat f_h\|^2
-
\frac{\|K_{h_{\min}}-K_h\|^2}{n}
+
\lambda \frac{\|K_h\|^2}{n},
\]
and \(\widehat h_{\rm PCO}=\arg\min_{h\in\mathcal H} l_{\rm PCO}(h)\). The multivariate extension replaces \(h\) by a symmetric positive-definite bandwidth matrix \(H\):
\[
l_{\rm PCO}(H)
=
\|\widehat f_{H_{\min}}-\widehat f_H\|^2
-
\frac{\|K_{H_{\min}}-K_H\|^2}{n}
+
\lambda \frac{\|K_H\|^2}{n},
\]
with \(\widehat H_{\rm PCO}=\arg\min_{H\in\mathcal H} l_{\rm PCO}(H)\) and penalty
\[
\mathrm{pen}_\lambda(H)=\frac{\lambda\|K_H\|^2-\|K_{H_{\min}}-K_H\|^2}{n}.
\]
This formulation extends to diagonal and full bandwidth matrices, including parameterizations based on eigendecomposition [1902.01075].

Across these density-estimation instantiations, the overfitted reference is either the highest reasonable wavelet resolution \(N_{\max}\) or the smallest allowed bandwidth \(h_{\min}\) or \(H_{\min}\). The repeated structure is the defining formal pattern of PCO.

## 3. Oracle inequalities and adaptive rates

A central theoretical feature of PCO is the derivation of oracle inequalities. In the wavelet setting, the main result is a high-probability oracle inequality for the \(L^2\)-risk of the PCO-selected estimator. In paraphrased form, for all \(N\in\mathcal H\), with high probability at least \(1-O(n^{-p})\),
\[
\|\widehat f_{\widehat N}-f\|^2
\leq
C_1 \min_{N\in\mathcal H}\|\widehat f_N-f\|^2
+
C_2\|f_{N_{\max}}-f\|^2
+
C_3\frac{2^{N_{\max}}}{n}
+\text{lower order terms},
\]
where \(f_N\) is the projection of \(f\) onto scale \(N\), and the constants depend only on parameters of the method, not on \(f\) [2509.07800]. The interpretation given there is that the selected estimator performs nearly as well as the oracle choice of \(N\).

The same paper establishes adaptive minimax behavior over Besov classes. If \(f\) lies in a Besov smoothness class \(B^r_{2q}\), then
\[
\sup_{f\in B^r_{2q}}
\mathbb E\|\widehat f_{\widehat N}-f\|^2
\leq
C\left(\frac{\log n}{n}\right)^{\frac{2r}{2r+1}}
\]
up to logarithmic and negligible factors, which is stated to match the minimax optimal rate for the class [2509.07800].

For multivariate kernel density estimation, the numerical study is paired with theoretical guarantees: PCO satisfies an oracle inequality, with the optimal constant achieved at \(\lambda=1\), and attains adaptive minimax rates over Nikol’skii or Besov classes [1902.01075]. In both wavelet and kernel formulations, the method is therefore positioned not merely as a heuristic selector but as an estimator-selection rule with non-asymptotic risk control.

## 4. Extension to weighted \(\ell_p\)-loss and nonparametric regression

PCO has also been generalized beyond \(L^2\)-based density estimation. In the sequence model
\[
Y_\lambda=\theta_\lambda+\varepsilon \xi_\lambda,\qquad \lambda\in\Lambda,
\]
with i.i.d. centered sub-Gaussian noise \(\xi_\lambda\), the model-based estimator indexed by a subset \(m\subset \Lambda^{(N)}\) is
\[
\theta^{(m)}_\lambda=
\begin{cases}
Y_\lambda & \lambda\in m,\\
0 & \text{otherwise}.
\end{cases}
\]
For weighted \(\ell_p\)-loss,
\[
\|\vartheta\|_{\ell_p(w)}^p=\sum_{\lambda\in\Lambda} w_\lambda |\vartheta_\lambda|^p,
\]
PCO selects
\[
\widehat m
=
\underset{m\in\mathcal M}{\operatorname{argmin}}
\left\{
\|\theta^{(m)}-\theta^{(\Lambda^{(N)})}\|_{\ell_p(w)}^p+\mathrm{pen}(m)
\right\},
\]
and sets \(\widehat\theta=\theta^{(\widehat m)}\) [2504.11217].

The analysis requires concentration inequalities for sums of \(p\)-th powers of sub-Gaussian variables, which become sub-Weibull. The paper states that there exists a constant \(\kappa_p\) such that, for all \(x\ge 1\), with probability at least \(1-2e^{-x}\),
\[
\sum_{\lambda\in m} |\xi_\lambda|^p
\le
\frac32 \sigma_p^p |m|+\kappa_p |m|^{(1-p/2)_+}x^{p/2}.
\]
This concentration behavior determines the form of the penalty, including the blockwise penalty
\[
p_j(m_j)=\frac32 \sigma_p^p |m_j|+\kappa_p |m_j|^{(1-p/2)_+}x_{m_j}^{p/2},
\qquad
\mathrm{pen}(m)=2^p\sum_j \omega_j p_j(m_j),
\]
and explains the “elbow” at \(p=2\): for \(p>2\), extra logarithmic factors may appear [2504.11217].

Using suitable model collections and penalties, the paper derives oracle inequalities and minimax rates over Besov bodies \(\mathcal B_{r,\infty}^s\), and then transfers these results to the functional model of nonparametric regression by applying PCO to empirical wavelet coefficients. The procedure is described as optimal up to logarithmic factors in all regimes considered [2504.11217]. This suggests that PCO is not limited to bandwidth or scale selection under quadratic risk, but can serve as a broader model-selection template for non-Euclidean losses.

## 5. Computational profile and comparison with competing methods

A recurring motivation for PCO is that it avoids some of the practical drawbacks of classical selectors. In the wavelet framework, compact support of the scaling function ensures feasible computation, and the procedure requires comparing each candidate level \(N\) only with a single reference estimator at \(N_{\max}\), rather than performing pairwise comparisons over all model pairs. The paper explicitly contrasts this with Goldenshluger–Lepski and Lepski-type procedures, which are said to incur high computational cost and possible instability because they require risk estimation across all pairs of models [2509.07800].

In multivariate kernel density estimation, the computational complexity is reported as
\[
O(d^3 n^2 |\mathcal H|),
\]
which matches cross-validation methods. For the Gaussian kernel, the relevant \(L_2\) norms and cross-terms have analytical forms, making the implementation practical for moderate dimension. The same study recommends \(\lambda=1\), reports a broad risk plateau around that value, and emphasizes that PCO can be applied to diagonal and full bandwidth matrices without algorithmic additional cost relative to standard competitors [1902.01075].

The competing procedures discussed in these works include penalized likelihood, Lepski’s method, Goldenshluger–Lepski, cross-validation, rule-of-thumb methods, and plug-in selectors. The drawbacks attributed to them are varied: explicit bias estimation, computational inefficiency, instability, or sensitivity to pilot choices. Against this background, PCO is presented as simpler and more direct: it relies on a single overfitted reference, requires no plug-in bias estimator, avoids subsampling, and in simulations is reported as robust, stable, and competitive or superior, especially for moderate to large sample sizes and in multivariate scenarios where classical methods can falter [2509.07800, 1902.01075].

## 6. Later reinterpretations and terminological extensions

The PCO principle has also become a reference point in adjacent literatures. In the two-sample testing paper on Complexity-Penalized MMD, PCO is explicitly cited as a related penalized-selection idea. The comparison drawn there is that both methods apply a complexity penalty to prevent overfitting in parameter selection, but the penalty in CP-MMD is derived from a two-sample uniform concentration inequality and is intended to support direct, grid-free optimization over continuous parametric classes, including deep kernels. That paper characterizes classical PCO as designed for discrete or very low-dimensional parameter selection such as bandwidth grids in density estimation [2605.06883].

A different reuse of the acronym appears in fast adversarial training. “FGSM-PCO” denotes a method for preventing catastrophic overfitting by fusing current and historical adversarial examples within a bi-level optimization framework, using an adaptive fusion ratio
\[
\lambda_t = \boldsymbol f_\theta^i(\boldsymbol x_t^*),\qquad i=\operatorname{argmax}(\boldsymbol y),
\]
and a tailored loss
\[
\mathcal L_{\text{PCO}}(\boldsymbol x_{\text{train}},\boldsymbol y)
=
\mathcal L_{CE}\big(\boldsymbol f_\theta(\boldsymbol x_{\text{train}}),\boldsymbol y\big)
+
\beta\Big[
\mathcal L_1\big(\boldsymbol f_\theta(\boldsymbol x_t^*),\boldsymbol f_\theta(\boldsymbol x_{t-1}^*)\big)
-
\mathcal L_1\big(\boldsymbol f_\theta(\boldsymbol x_{\text{train}}),\boldsymbol f_\theta(\boldsymbol x_t^*)\big)
\Big].
\]
Here, “Penalized Comparison to Overfitting” refers to a mechanism for preventing and reversing collapse of the inner maximization in adversarial training, not to estimator selection in nonparametric inference [2407.12443].

This terminological spread can obscure the statistical meaning of PCO. In its original and most developed form, PCO is a nonparametric model-selection methodology built around a comparison with an overfitted reference and a penalty calibrated to restore statistical validity. Subsequent usages preserve the rhetorical core—penalizing a comparison to an overfitted or collapsing solution—but adapt it to very different optimization problems [2509.07800, 2407.12443].

Source: https://www.emergentmind.com/topics/penalized-comparison-to-overfitting-pco