Non-penalized Deep Neural Network (NPDNN)

• NPDNN is a deep learning framework defined by sparsity-constrained architectures that regulate effective model complexity without an explicit penalty term.
• It is applied in statistical estimation as an empirical risk minimizer for regression and classification and in system identification via fading-memory predictors.
• The method achieves minimax optimal convergence rates and robust performance under both independent and dependent data scenarios.

Searching arXiv for the cited NPDNN-related papers to ground the article in current records. Non-penalized Deep Neural Network (NPDNN) denotes, in the arXiv literature summarized here, a family of deep-learning estimators defined without an explicit additive penalty term in the empirical objective, or, in nonlinear system identification, a structured deep architecture whose effective complexity is controlled through a fading-memory mechanism. In the statistical-learning formulations, NPDNN is an empirical risk minimizer over a sparsity-constrained DNN hypothesis class, so “non-penalized” refers to the criterion rather than to the complete absence of regularization. In the system-identification formulation, the same label is associated with a blockwise, lag-structured predictor equipped with normalization and regularization designed to encode fading memory and improve generalization (Kengne et al., 11 Mar 2026, Kengne et al., 29 Dec 2025, Zancato et al., 2021).

1. Terminological scope and principal usages

The term NPDNN is not fully standardized across the current literature. In the papers considered here, it appears in two principal senses.

Usage	Core definition	Primary setting
Statistical NPDNN	Empirical risk minimizer over a sparsity-constrained DNN class, with no explicit penalty term in the objective	Nonparametric regression and classification under dependence
System-identification NPDNN	Block-structured DNN with fading-memory organization, batch normalization, and regularized training	Nonlinear system identification

In the statistical sense, NPDNN is defined by optimization over classes such as $\mathcal H_\sigma(L,N,B,F,S)$, where depth, width, parameter magnitudes, output magnitude, and sparsity are controlled by the hypothesis class itself. In this usage, the contrast is with a sparse-penalized DNN estimator (SPDNN), which optimizes over a larger class and adds an explicit penalty $J_n(h)$ to the objective.

In the system-identification sense, NPDNN denotes a neural predictor for one-step-ahead forecasting that decomposes the input history into lagged local windows processed by separate DNN blocks. The model is deliberately structured so that recent lags can dominate and distant lags can be suppressed automatically. This suggests a broader conceptual commonality: NPDNN methods aim to control effective complexity through architectural or class constraints rather than through a conventional additive penalty as the defining feature.

2. NPDNN as an empirical-risk estimator over sparse DNN classes

A central formulation appears in deep nonparametric regression from strongly mixing observations with the minimum error entropy (MEE) principle. The data form a stationary, ergodic process

$Z_t=(X_t,Y_t),\qquad t\in\mathbb Z,$

generated by

$Y_t=h_0(X_t)+\xi_t,$

where $h_0:\mathbb R^d\to\mathbb R$ is the unknown regression function, $(\xi_t)$ is an i.i.d. centered error sequence independent of $X_t$, and $f$ is the density of $\xi_0$. The MEE risk is

$R(h)=\mathbb E\big[-\log f(Y_0-h(X_0))\big],$

with loss

$J_n(h)$0

The excess risk is

$J_n(h)$1

where $J_n(h)$2 minimizes $J_n(h)$3 over measurable functions. In the common symmetric-noise setting, $J_n(h)$4 itself is the target function minimizing $J_n(h)$5 (Kengne et al., 11 Mar 2026).

The DNN classes are feedforward networks with activation $J_n(h)$6, depth $J_n(h)$7, widths $J_n(h)$8, and parameter vector $J_n(h)$9. Two classes are defined: $Z_t=(X_t,Y_t),\qquad t\in\mathbb Z,$0 consisting of DNNs with depth at most $Z_t=(X_t,Y_t),\qquad t\in\mathbb Z,$1, width at most $Z_t=(X_t,Y_t),\qquad t\in\mathbb Z,$2, bounded parameters $Z_t=(X_t,Y_t),\qquad t\in\mathbb Z,$3, and bounded output norm $Z_t=(X_t,Y_t),\qquad t\in\mathbb Z,$4; and

$Z_t=(X_t,Y_t),\qquad t\in\mathbb Z,$5

which adds the sparsity constraint $Z_t=(X_t,Y_t),\qquad t\in\mathbb Z,$6.

Within this framework, the NPDNN estimator is

$Z_t=(X_t,Y_t),\qquad t\in\mathbb Z,$7

Its defining feature is the absence of an explicit penalty term in the empirical objective. Regularization is instead structural: the network is selected from a sparsity-constrained class.

A more general formulation appears in a deep-learning framework covering regression and classification. There the risk is

$Z_t=(X_t,Y_t),\qquad t\in\mathbb Z,$8

and the NPDNN estimator is the empirical risk minimizer

$Z_t=(X_t,Y_t),\qquad t\in\mathbb Z,$9

In both formulations, “non-penalized” does not mean unconstrained; it means that sparsity is enforced through the admissible model class rather than by adding $Y_t=h_0(X_t)+\xi_t,$0 to the criterion (Kengne et al., 29 Dec 2025).

3. Dependence assumptions, loss structure, and admissible network classes

The statistical NPDNN literature is explicitly designed for dependent data. In the MEE-based regression analysis, the process $Y_t=h_0(X_t)+\xi_t,$1 is assumed to be strongly mixing, stationary, and ergodic, with exponentially decaying mixing coefficients

$Y_t=h_0(X_t)+\xi_t,$2

for some $Y_t=h_0(X_t)+\xi_t,$3. This dependence structure enters the concentration arguments through Bernstein-type inequalities for strongly mixing sequences.

The broader framework replaces a specific mixing assumption by a generalized Bernstein-type inequality. For any bounded measurable $Y_t=h_0(X_t)+\xi_t,$4 with $Y_t=h_0(X_t)+\xi_t,$5, $Y_t=h_0(X_t)+\xi_t,$6, and $Y_t=h_0(X_t)+\xi_t,$7, it assumes

$Y_t=h_0(X_t)+\xi_t,$8

This framework explicitly covers i.i.d. observations, $Y_t=h_0(X_t)+\xi_t,$9-mixing processes, strongly mixing ($h_0:\mathbb R^d\to\mathbb R$0-mixing) processes, and $h_0:\mathbb R^d\to\mathbb R$1-mixing processes (Kengne et al., 29 Dec 2025).

The activation assumptions are similarly explicit. The activation $h_0:\mathbb R^d\to\mathbb R$2 is required to be Lipschitz and either piecewise linear or locally quadratic, and it must fix a nonempty interval $h_0:\mathbb R^d\to\mathbb R$3. ReLU is the key example. The loss is assumed Lipschitz in its prediction argument in the general framework, while the MEE-based analysis imposes additional regularity on the error density $h_0:\mathbb R^d\to\mathbb R$4, including Lipschitz continuity, differentiability, and boundedness of $h_0:\mathbb R^d\to\mathbb R$5 on bounded sets.

The MEE analysis also isolates a local structure condition for the excess risk and a control of low-density regions through truncation of the density at level $h_0:\mathbb R^d\to\mathbb R$6. For Subbotin noise

$h_0:\mathbb R^d\to\mathbb R$7

it proves that if $h_0:\mathbb R^d\to\mathbb R$8, then the relevant local excess-risk condition holds with $h_0:\mathbb R^d\to\mathbb R$9, and the low-density control condition also holds for suitable $(\xi_t)$0. This includes Laplace noise ($(\xi_t)$1) and Gaussian noise ($(\xi_t)$2). A plausible implication is that the NPDNN framework is intended to cover both classical light-tailed models and heavier-tailed settings without reverting to least-squares analysis.

4. Fading-memory NPDNN for nonlinear system identification

A distinct NPDNN construction appears in nonlinear system identification. The underlying system is modeled as

$(\xi_t)$3

with innovation form

$(\xi_t)$4

where $(\xi_t)$5. The target predictor is the one-step-ahead map

$(\xi_t)$6

The general identification problem is posed as

$(\xi_t)$7

and the NPDNN is a specific parametric choice for $(\xi_t)$8. Its input is a delay embedding

$(\xi_t)$9

The predictor is expressed as a linear combination of learned DNN blocks,

$X_t$0

where each block $X_t$1 acts on a shifted local window of length $X_t$2: $X_t$3

The architecture is designed to encode fading memory: recent history should matter more, older history should matter less, and the decay should be learned from data. This is implemented through a prior on the coefficients $X_t$4,

$X_t$5

with diagonal covariance

$X_t$6

The paper notes a degeneracy if one naively optimizes the joint negative log-posterior over $X_t$7: the objective tends to $X_t$8 as $X_t$9. To address this, it derives a variational criterion involving

$f$0

and optimizes

$f$1

Additional structure is imposed through soft orthogonality regularization,

$f$2

and batch normalization of each block output. The normalized architecture is written as

$f$3

The intended effect is not merely optimization stability but preservation of the fading-memory interpretation across blocks. Training uses standard stochastic optimizers, specifically SGD and Adam, with minibatches of size $f$4. The architecture is intended to be large enough in $f$5 to cover the relevant past, while the learned decay parameter $f$6 and the regularized coefficients $f$7 determine the effective memory length automatically (Zancato et al., 2021).

5. Theoretical guarantees and convergence rates

The strongest theory for NPDNN concerns expected excess risk. In the MEE-based regression setting over Hölder classes, if

$f$8

and the architecture is chosen as

$f$9

$\xi_0$0

with $\xi_0$1, then for all $\xi_0$2,

$\xi_0$3

For composition Hölder classes $\xi_0$4, with

$\xi_0$5

and with ReLU, $\xi_0$6, and architecture choices

$\xi_0$7

the bound becomes

$\xi_0$8

The general framework replaces $\xi_0$9 by an effective sample size $R(h)=\mathbb E\big[-\log f(Y_0-h(X_0))\big],$0. For Hölder targets, the NPDNN rate is

$R(h)=\mathbb E\big[-\log f(Y_0-h(X_0))\big],$1

For composition Hölder targets, it proves

$R(h)=\mathbb E\big[-\log f(Y_0-h(X_0))\big],$2

where

$R(h)=\mathbb E\big[-\log f(Y_0-h(X_0))\big],$3

The dependence-specific consequences are explicit. For i.i.d. data, $R(h)=\mathbb E\big[-\log f(Y_0-h(X_0))\big],$4, yielding the classical rates up to logarithmic factors. For exponential $R(h)=\mathbb E\big[-\log f(Y_0-h(X_0))\big],$5-mixing,

$R(h)=\mathbb E\big[-\log f(Y_0-h(X_0))\big],$6

and for subexponential $R(h)=\mathbb E\big[-\log f(Y_0-h(X_0))\big],$7-mixing,

$R(h)=\mathbb E\big[-\log f(Y_0-h(X_0))\big],$8

For geometrically $R(h)=\mathbb E\big[-\log f(Y_0-h(X_0))\big],$9-mixing,

$J_n(h)$00

and for polynomially $J_n(h)$01-mixing with $J_n(h)$02,

$J_n(h)$03

A major conclusion concerns minimax optimality. In the MEE-based strongly mixing regression paper, when the error is Gaussian, the excess risk becomes equivalent to squared $J_n(h)$04-error, and the resulting NPDNN rates are $J_n(h)$05 on Hölder regression and $J_n(h)$06 on composition Hölder regression. These rates match, up to logarithmic factors, the lower bounds established in Schmidt-Hieber (2020). The broader framework likewise states that NPDNN is minimax optimal up to logarithmic factors in many classical settings, including i.i.d., $J_n(h)$07-mixing, exponential $J_n(h)$08-mixing, and geometrically $J_n(h)$09-mixing regimes (Kengne et al., 11 Mar 2026, Kengne et al., 29 Dec 2025).

6. Comparison with SPDNN, empirical behavior, and related paradigms

The primary formal comparator is the sparse-penalized deep neural network estimator. In the MEE formulation,

$J_n(h)$10

with

$J_n(h)$11

The penalty $J_n(h)$12 is nondecreasing, satisfies $J_n(h)$13, and becomes constant $J_n(h)$14 beyond threshold $J_n(h)$15; clipped-$J_n(h)$16, SCAD, MCP, and seamless-$J_n(h)$17 are cited as examples. The conceptual distinction is therefore precise: NPDNN uses hard sparsity through the model class, whereas SPDNN uses an explicit penalty over a larger class. In the studied settings, both estimators achieve the same principal rates. A common misconception is that “non-penalized” means “unregularized”; the literature summarized here does not support that interpretation.

In nonlinear system identification, the empirical evidence emphasizes structural rather than asymptotic advantages. The experiments are Monte Carlo studies on four nonlinear benchmark systems, with trajectories generated from rest, $J_n(h)$18 when possible, $J_n(h)$19, and dataset lengths up to $J_n(h)$20. Each block $J_n(h)$21 is an overparameterized DNN with 5 hidden layers, 100 hidden units per layer, Tanh activations, and about 41k parameters per block. Performance is evaluated through the estimated innovation variance

$J_n(h)$22

The reported findings are that a plain DNN with similar parameter count and the same horizon overfits badly, whereas the fading architecture reduces the train-test gap, generalizes better across all benchmark systems, and remains robust when $J_n(h)$23 grows provided that $J_n(h)$24 is reasonable. The comparison with a GP/RKHS method is nuanced: in small-data regimes ($J_n(h)$25) the GP method performs better or comparably, whereas in mid-to-large data regimes ($J_n(h)$26) the proposed architecture becomes competitive and eventually better scaled (Zancato et al., 2021).

A further adjacent paradigm is Deep pNML. The cover-letter excerpt for “Deep pNML: Predictive Normalized Maximum Likelihood for Deep Neural Networks” states that it establishes a new learning paradigm and demonstrates it on the deep neural network hypothesis class, but it does not define an NPDNN formulation, provide equations, or establish any exact equivalence between Deep pNML and NPDNN. The defensible relationship is therefore limited: both concern alternatives to standard empirical risk minimization in deep models, but the excerpt does not support a stronger identification (Bibas et al., 2019).