Multigrade Deep Learning

• MGDL is a deep learning framework that constructs networks by sequentially training shallow subproblems to progressively reduce residual errors.
• It offers rigorous convergence guarantees and effectively mitigates spectral bias, enabling stable, interpretable, and computationally efficient model construction.
• Applications span high-frequency regression, PDE solving, and image processing, demonstrating significant improvements over traditional single-grade deep learning.

Multigrade Deep Learning (MGDL) is a framework for constructing deep neural networks via explicit hierarchical error refinement, wherein the full network is trained iteratively as a sequence of shallow subproblems—grades—each responsible for reducing the residual of the approximation yielded by preceding grades. MGDL replaces the monolithic nonconvex optimization typical of traditional deep learning with a grade-by-grade training paradigm, yielding block-structured architectures that facilitate stable, interpretable, and efficient model construction. MGDL has been developed theoretically and demonstrated empirically to achieve superior stability, approximation accuracy, frequency fidelity (mitigation of spectral bias), and computational robustness over single-grade deep learning (SGDL), across classical function approximation, PDE and operator learning, high-frequency regression, and online learning contexts (Xu, 2023, Fang et al., 2024, Fang et al., 27 Jul 2025, Zhang et al., 23 Jan 2026, Jiang et al., 8 Jan 2026, Jiang et al., 2024, Xu et al., 2023).

1. Formal Definition and Grade-by-Grade Training

MGDL organizes the construction of a deep neural network into $K$ sequential grades, each constituting a shallow neural subnetwork (typically 2–4 layers of moderate width). Let $y:\mathbb{R}^s \rightarrow \mathbb{R}^t$ be a target function, or $\{(x_i, y_i)\}_{i=1}^m$ a dataset. At grade $k$, one trainable subnetwork

$$h^{(k)}(x; W^{(k)}): \mathbb{R}^s \rightarrow \mathbb{R}^t$$

is trained to minimize the residual error

$$\min_{W^{(k)}} \sum_{i=1}^m L\left(r^{(k)}(x_i), h^{(k)}(x_i; W^{(k)})\right)$$

where $r^{(k)}(x) = y(x) - f^{(k-1)}(x)$ and $$f^{(k-1)}(x) = \sum_{j=1}^{k-1} h^{(j)}(x; W^{(j)})$$. Typical choices for $L$ include squared error and cross-entropy.

After $K$ grades, the overall network is assembled as a "stair-shape" additive model:

$y:\mathbb{R}^s \rightarrow \mathbb{R}^t$0

Training is strictly sequential; after grade $y:\mathbb{R}^s \rightarrow \mathbb{R}^t$1 is trained, all parameters of earlier grades are frozen and act as fixed feature extractors for subsequent grades (Xu, 2023, Fang et al., 27 Jul 2025).

High-level pseudocode:

$$h^{(k)}(x; W^{(k)}): \mathbb{R}^s \rightarrow \mathbb{R}^t$$7 (Xu, 2023)

2. Theoretical Guarantees and Operator-Theoretic Foundations

MGDL admits rigorous convergence and error reduction guarantees. For any continuous target $y:\mathbb{R}^s \rightarrow \mathbb{R}^t$2, there exists a fixed-width MGDL $y:\mathbb{R}^s \rightarrow \mathbb{R}^t$3 architecture such that residuals $y:\mathbb{R}^s \rightarrow \mathbb{R}^t$4 (with $y:\mathbb{R}^s \rightarrow \mathbb{R}^t$5 the $y:\mathbb{R}^s \rightarrow \mathbb{R}^t$6-grade approximation) satisfy

• $y:\mathbb{R}^s \rightarrow \mathbb{R}^t$7 decreases strictly and uniformly to zero,
• $y:\mathbb{R}^s \rightarrow \mathbb{R}^t$8 decreases strictly to zero for any $y:\mathbb{R}^s \rightarrow \mathbb{R}^t$9, for suitable choices of per-grade shallow architectures and training (Zhang et al., 23 Jan 2026).

The contraction property is derived operator-theoretically. Let $\{(x_i, y_i)\}_{i=1}^m$0 map functions to their grade-wise corrections with the balanced contraction

$\{(x_i, y_i)\}_{i=1}^m$1

Iterating $\{(x_i, y_i)\}_{i=1}^m$2 with pointwise domination and strict $\{(x_i, y_i)\}_{i=1}^m$3 decay produces provable convergence to the target, as each grade reduces only the unfit residual of its predecessor (Zhang et al., 23 Jan 2026).

At the optimization level, the $\{(x_i, y_i)\}_{i=1}^m$4th grade solve is over a shallow, reduced-parameter hypothesis class, with prior grades fixed; thus, the landscape per grade is lower-dimensional and typically better conditioned than global end-to-end optimization.

3. Spectral Bias, Frequency Localization, and High-Frequency Approximation

MGDL systematically overcomes the spectral bias of traditional deep learning, wherein standard DNNs (SGDL) prioritize low-frequency components and struggle to fit high-frequency structure, especially when trained via gradient descent (Fang et al., 2024, Jiang et al., 2024). Each shallow grade in MGDL naturally specializes in learning the low-frequency features of its current residual. High frequencies are "deferred" to later grades, which learn residuals representing the higher-frequency gaps left by prior approximations.

Empirical frequency-domain analysis demonstrates that:

• Early grades reproduce the dominant low-frequency behavior.
• Intermediate grades sequentially fit intermediate frequencies.
• Final grades capture fine-scale and high-frequency components.

For synthetic 1D regression targets (sum of sinusoids up to 200 Hz), MGDL achieves up to $\{(x_i, y_i)\}_{i=1}^m$5-fold improvement in test error and fully recovers the entire target spectrum, whereas SGDL exhibits large errors at upper frequency bands (Fang et al., 2024, Jiang et al., 2024).

This composition-of-features paradigm, theoretically illustrated via Jacobi-Anger identities and function composition, facilitates representing complex, oscillatory solutions—including those of PDEs and integral equations—with provable accuracy when SGDL fails due to bias (Fang et al., 2024, Jiang et al., 2024, Xu et al., 2023, Jiang et al., 8 Jan 2026).

4. Algorithmic Robustness, Stability, and Computational Advantages

By decomposing the full optimization into sequential shallow problems, MGDL offers increased stability and algorithmic robustness relative to SGDL (Fang et al., 27 Jul 2025). The effective Hessian norm of each per-grade problem is much smaller than that of the full network, permitting a significantly wider range of admissible learning rates and consequently greater resilience to hyperparameter tuning. Empirical spectral analysis of the Jacobians during training reveals that the eigenvalue spectra of the per-grade loss Hessians always lie within the stability interval $\{(x_i, y_i)\}_{i=1}^m$6, ensuring smooth monotone loss decay and avoiding the oscillatory loss dynamics typical for SGDL (Fang et al., 27 Jul 2025).

Further, MGDL's modularity supports adaptive grade splitting and stopping criteria. Adaptive MGDL schemes use grades as needed to reach a prescribed tolerance, with grade addition halting when residual error ceases to reduce (up to quadrature and training error) (Jiang et al., 8 Jan 2026).

In practical performance benchmarks:

• Image regression: MGDL improves testing PSNR by up to 4 dB relative to SGDL, matching or exceeding accuracy with far fewer parameters and more stable loss dynamics.
• Online learning: hybrid MGDL/cascade schemes such as MODL combine fast statistical learners (grade 1) with deep residual modules (grade 3), achieving lower cumulative error and faster convergence than state-of-the-art online deep learning (Valkanas et al., 2024).
• PDE and operator learning: in the 1D/2D/3D Burgers equations, MGDL-based solvers achieve 4–60$\{(x_i, y_i)\}_{i=1}^m$7 reductions in relative $\{(x_i, y_i)\}_{i=1}^m$8 error relative to SGDL, with faster and more stable convergence (Xu et al., 2023).

5. Applications: PDEs, Operator Equations, and High-Dimensional Learning

MGDL has been successfully applied to:

• Oscillatory and singular Fredholm integral equations, where standard DNNs fail due to spectral bias and high-frequency content; MGDL achieves order-of-magnitude reductions in $\{(x_i, y_i)\}_{i=1}^m$9 error and recovers all frequency components (Jiang et al., 2024, Jiang et al., 8 Jan 2026).
• Parametric and nonlinear PDEs (e.g., Burgers, Poisson-Boltzmann), where two-stage MGDL (TS-MGDL) architectures with unfreezing/fine-tuning of output layers outperform deep monolithic PINNs, reducing errors by an order of magnitude or more over multiple physics-informed regression tasks (Xu et al., 2023).
• High-frequency image regression, denoising, and deblurring, where MGDL-based convolutional or multigrid-inspired architectures (MgNet, MGiaD) deliver improved PSNR with reduced parameter complexity, leveraging multigrid structures in both spatial and channel dimensions to control model capacity and regularize learning (Betteray et al., 2022, He et al., 2019).
• Online learning settings, where fast adaptation and deep representation are achieved in a cascade by combining fast shallow learners with deep residual refiners, satisfying both speed and representational requirements in data-stream scenarios (Valkanas et al., 2024).

6. Limitations, Theoretical and Practical Considerations, and Research Directions

Limitations and open directions include:

• Theoretical analysis of per-grade spectral decomposition remains immature; current results guarantee monotone residual decay, but do not quantify the frequency coverage per grade or provide explicit rates (Fang et al., 2024, Zhang et al., 23 Jan 2026).
• MGDL as presently formulated requires nonconvex optimization per grade; while shallower problems are better conditioned, convexification (as in the Successive Affine Learning model) is an active area (Xu, 2023).
• Grade partitioning and adaptive depth selection remain heuristic; automation and optimal scheduling are open topics.
• Best practices indicate shallower grades (1–2 layers) enhance stability, and per-grade validation losses mitigate overfitting and guide adaptivity (Jiang et al., 8 Jan 2026).

Research frontiers include extending MGDL principles to convolutional, transformer, and attention-based architectures; theoretical expressivity and generalization bounds; and application to operator learning and hierarchical structured modeling beyond canonical function approximation (Zhang et al., 23 Jan 2026, Fang et al., 2024, Betteray et al., 2022).

7. Quantitative Performance Summary

MGDL achieves pronounced improvements across diverse benchmarks relative to SGDL, as summarized in the following table (selected metrics):

Task	SGDL Error	MGDL Error	Improvement	Source
1D regression (MSE)	$k$0	$k$1	$k$2-fold	(Xu, 2023)
1D sum-of-sinusoids	$k$3	$k$4	$k$5-fold	(Fang et al., 2024)
2D Burgers (Rel $k$6)	$k$7	$k$8	$k$9–$$h^{(k)}(x; W^{(k)}): \mathbb{R}^s \rightarrow \mathbb{R}^t$$0	(Xu et al., 2023)
Image regression (PSNR)	$$h^{(k)}(x; W^{(k)}): \mathbb{R}^s \rightarrow \mathbb{R}^t$$1 dB	$$h^{(k)}(x; W^{(k)}): \mathbb{R}^s \rightarrow \mathbb{R}^t$$2 dB	$$h^{(k)}(x; W^{(k)}): \mathbb{R}^s \rightarrow \mathbb{R}^t$$3 dB	(Fang et al., 27 Jul 2025)
Fredholm equation	$$h^{(k)}(x; W^{(k)}): \mathbb{R}^s \rightarrow \mathbb{R}^t$$4	$$h^{(k)}(x; W^{(k)}): \mathbb{R}^s \rightarrow \mathbb{R}^t$$5	$$h^{(k)}(x; W^{(k)}): \mathbb{R}^s \rightarrow \mathbb{R}^t$$6-fold	(Jiang et al., 2024)

In all cases, MGDL offers substantially lower error, reduced training cost, and stability improvements.

---

MGDL is a general, theoretically justified, and practically robust paradigm for deep learning through hierarchical, grade-wise residual reduction. It combines proven convergence, frequency adaptivity, and computational advantages, positioning it as a foundational methodology for constructing deep architectures across scientific, numerical, and data-driven applications (Xu, 2023, Fang et al., 2024, Fang et al., 27 Jul 2025, Zhang et al., 23 Jan 2026, Jiang et al., 8 Jan 2026, Xu et al., 2023, Valkanas et al., 2024, Betteray et al., 2022, He et al., 2019).