HG-TNet: Hierarchical Gradient-Enhanced TNet

• HG-TNet is a deep learning framework that integrates physics-based Tikhonov regularization with hierarchical, multi-scale (multi-grid) network structures to solve inverse PDE problems.
• It employs a coarse-to-fine strategy that projects corrections across scales, enhancing model generalization and efficiency compared to traditional Tikhonov solvers.
• The incorporation of high-order gradient penalties (Jacobian and Hessian norms) enforces smoothness and stability, leading to more accurate parameter recovery in inverse problems.

HG-TNet refers to the “Hierarchical/Gradient-Enhanced TNet,” a conceptual extension of the TNet model-constrained deep learning framework for inverse problems. TNet uses physics-based Tikhonov regularization within a deep neural network (DNN) to enforce both data and mathematical model constraints. HG-TNet incorporates hierarchical (multi-grid) architectures and high-order gradient penalties to further improve generalization, accuracy, and solution smoothness for challenging inverse problems governed by partial differential equations (PDEs) (Nguyen et al., 2021).

1. Mathematical Foundations

HG-TNet builds on the TNet formulation for solving inverse problems of recovering a parameter vector $u \in \mathbb{R}^m$ from observed data $y \in \mathbb{R}^n$ under a forward (parameter-to-observable) map $G(u)$:

$y = G(u_{\text{true}}) + \epsilon.$

Classical Tikhonov regularization solves

$$u^\star = \arg\min_u \frac{1}{2}\|G(u) - y\|_\Gamma^{-2} + \frac{1}{2}\alpha\|u - u_0\|_\Sigma^{-2},$$

where $\Gamma, \Sigma$ are positive-definite weighting matrices, $\alpha > 0$ is a regularization parameter, and $u_0$ is a prior mean.

TNet replaces the iterative solver with a DNN $\Psi_\theta: \mathbb{R}^n \to \mathbb{R}^m$, trained under the loss

$$L_{\text{TNet}}(\theta) = \frac{1}{N} \sum_{i=1}^N \left[\|\Psi_\theta(y^i) - u_0\|^2 + \alpha \|G(\Psi_\theta(y^i)) - y^i\|^2\right],$$

directly embedding the model constraint into the optimization.

HG-TNet further augments this loss with multi-level structure and higher-order derivative penalties to enforce more stringent smoothness and multi-scale consistency: $$L_{\text{HG}}(\theta) = L_{\text{TNet}}(\theta) + \gamma \left\| \nabla_y \Psi_\theta(y) \right\|_F^2 + \delta \left\|\nabla^2_y (G \circ \Psi_\theta(y))\right\|_F^2$$ where $\gamma, \delta > 0$ are hyperparameters for penalizing the Jacobian and Hessian norms.

2. Hierarchical and Multi-Grid Network Architecture

HG-TNet introduces hierarchical (multi-grid) model components. For multiple levels $\ell = 0, \ldots, L$, a series of subnetworks $\Psi_\ell$ are constructed, each mapping observations $y_\ell$ at grid level $\ell$ to parameter estimates $u_\ell$. The final reconstruction is obtained by composing coarse-to-fine contributions: $$u = u_0 + \sum_{\ell=1}^L \mathrm{Proj}_{\ell \to 0}\big(\Psi_\ell(y_\ell) - u_0\big),$$ where $\mathrm{Proj}_{\ell \to 0}$ lifts updates from coarse levels to the finest grid. The training loss incorporates cross-level consistency with an additional penalty,

$$\beta \|u_{\text{fine}} - (u_0 + \sum_\ell \mathrm{Proj}(\Psi_\ell - u_0))\|^2,$$

to promote agreement among the hierarchy.

A plausible implication is that HG-TNet’s multi-resolution structure combines the efficiency and robustness of classical multi-grid solvers with the expressivity of deep networks.

3. High-Order Regularity and Gradient Penalties

Beyond hierarchical design, HG-TNet introduces explicit penalties on network derivatives:

• The term $\| \nabla_y \Psi_\theta(y) \|_F^2$ regularizes the Jacobian, encouraging Lipschitz and smooth inverse maps.
• The term $\| \nabla_y^2 (G \circ \Psi_\theta(y)) \|_F^2$ enforces higher-order (Sobolev $H^2$) regularity of the composed inverse mapping.

This approach is motivated by theoretical results (Theorem 3.5 in (Nguyen et al., 2021)) which show that randomizing training data induces implicit Sobolev-type penalties. Adding explicit higher-order derivatives further mimics classical Hermite interpolation behavior, encouraging smoothness and stability in the learned inverse mapping.

These derivative norms are efficiently implemented using automatic differentiation, obtaining the Jacobian $J_\Psi = \partial\Psi_\theta/\partial y$ and Hessian-vector products for inclusion in the minibatch loss.

4. Training Methodology

HG-TNet employs Adam optimization (learning rate $\approx 10^{-3}$, typically $2 \times 10^4$ steps) without decay. Weight initialization is Gaussian, and biases are initialized to zero. Hierarchical architectures may be trained either:

• Jointly, via multi-scale Adam optimizer over all $\ell$,
• Or with block-coordinate (coarse-to-fine) optimization, optionally with curriculum learning based on grid resolution.

In low-data regimes, data replication with added Gaussian noise ($\epsilon \sim \mathcal{N}(0, \lambda^2 I)$) is used to enhance effective dataset size and, via randomization, to promote solution smoothness (Nguyen et al., 2021).

5. Context: TNet Performance and Motivation for HG-TNet

TNet, the precursor to HG-TNet, demonstrates quantitative accuracy and acceleration over classical iterated Tikhonov solvers in numerical benchmarks:

• For 1D deconvolution, TNet attains $\sim5\%$ test error with $N=20$–$200$, matching Tikhonov while outperforming pure data-driven DNNs and mildly outperforming mcDNN.
• For the 2D heat conductivity problem ($m=256$, $n=10$), with $N=50$–$200$ or extensive baseline/replication ($N_b=20$, $M=2000$), TNet’s error remains near $45\%$, substantially better than nDNN ($50-88\%$ error).
• In 2D Burgers’ and Navier–Stokes inverse PDE settings, TNet approaches Tikhonov-level accuracy with orders-of-magnitude fewer samples than pure DNNs.
• Speed-up: Forward TNet prediction requires $3\times10^{-4}$s per solve versus $0.04$–$7$s for Tikhonov, a $10^2$–$10^4\times$ acceleration (NVIDIA A100 GPU).

A plausible implication is that the hierarchical and high-order features of HG-TNet could further reduce error and enhance generalization, especially in settings where spatial multi-scale structure and higher-order smoothness are essential.

6. Implementation Aspects and Pseudocode

HG-TNet’s loss is computed via mini-batch sampling, forward evaluation of subnetworks, projection of multi-scale corrections, and incorporation of automatic-differentiation outputs:

Compute u_hat^i = Psi_theta(y^i) Compute residual r^i = G(u_hat^i) - y^i Compute J_Psi^i = dPsi_theta/dy (y^i) Compute H_G^i ~ d^2(G o Psi)/dy^2 (y^i) Loss = (1/N) sum_i [|u_hat^i - u0|^2 + alpha * |r^i|^2] + gamma * (1/N) sum_i |J_Psi^i|_F^2 + delta * (1/N) sum_i |H_G^i|_F^2 theta = AdamStep(theta, grad_theta Loss)

This approach extends transparently to multi-level subnetworks and mesh-based projection routines, compatible with block-coordinate or joint multi-scale scheduling.

7. Connections and Outlook

HG-TNet represents a fusion of model-constrained deep learning, classical Tikhonov regularization, multi-grid solvers, and modern automatic differentiation techniques. Unlike pure data-driven DNNs, the approach leverages the physics and mathematics of the underlying inverse problem, making it suitable for data-constrained scientific and engineering scenarios. The hierarchical and gradient-regularized components position HG-TNet as a structured, theoretically motivated deep learning framework for large-scale PDE-governed inverse problems. Further study is warranted on scalability, multi-scale convergence, and automatic selection or annealing of regularization hyperparameters (Nguyen et al., 2021).