PI-LoRA-HyperDeepONet

• The paper introduces PI-LoRA-HyperDeepONet, a low-rank adaptive variant of HyperDeepONet that reduces computational burden while maintaining predictive accuracy.
• It applies Low-Rank Adaptation to the hypernetwork's output layer, achieving up to 70%-90% parameter savings and implicit regularization for improved generalization.
• Empirical results on benchmark ODE and PDE problems demonstrate that the model efficiently balances parameter reduction with high expressivity in physics-informed settings.

PI-LoRA-HyperDeepONet is a low-rank adaptive variant of the physics-informed HyperDeepONet architecture for operator learning, specifically designed to reduce the parameter count and computational burden while maintaining predictive accuracy. The model applies Low-Rank Adaptation (LoRA) to the output layer of the hypernetwork responsible for trunk weight generation within the HyperDeepONet framework. By enforcing a low-rank decomposition of this critical layer, PI-LoRA-HyperDeepONet achieves significant parameter savings and inherent regularization, yielding a compact, generalizable operator surrogate for solving both ordinary and partial differential equations in a physics-informed setting (Zeudong et al., 24 Jul 2025).

1. HyperDeepONet and Low-Rank Decomposition

The foundational architecture, HyperDeepONet, consists of a branch network (hypernetwork) and a trunk network. The branch net processes the discretized input function (initial/boundary data sampled at $n_{\rm sensor}$ points) and, through its output layer, generates all the weights and biases for the trunk net. Specifically, if the trunk net has $n_{\rm trunk}$ parameters, the branch net’s output is governed by a weight matrix $$W^{\rm branch}_{\rm out} \in \mathbb{R}^{n_{\rm trunk} \times n_{\rm hidden}}$$ and bias $$b^{\rm branch}_{\rm out} \in \mathbb{R}^{n_{\rm trunk}}$$. The trunk net is a standard fully connected network taking $(t, x)$ as input and producing a basis vector used for reconstructing the solution via a learned linear map.

PI-LoRA-HyperDeepONet introduces low-rank adaptation by decomposing the hypernetwork’s output layer matrix $W$ as

$$W = W_0 + AB, \quad A \in \mathbb{R}^{d \times r},\; B \in \mathbb{R}^{r \times k}$$

where $d = n_{\rm trunk}$, $k = n_{\rm hidden}$, and $r \ll \min(d, k)$. $n_{\rm trunk}$0 is a fixed base matrix (pre-trained or zero-initialized), while only $n_{\rm trunk}$1 and $n_{\rm trunk}$2 are trainable, confining updates to a rank-$n_{\rm trunk}$3 subspace.

2. Physics-Informed Training and Loss Construction

Training is governed by a physics-informed composite loss function that enforces the governing equation residual, as well as initial and boundary conditions. For a general (possibly system) PDE:

$n_{\rm trunk}$4

the full loss is

$n_{\rm trunk}$5

with

$n_{\rm trunk}$6

The hypernetwork parameters to be trained are $n_{\rm trunk}$7, where $n_{\rm trunk}$8 and $n_{\rm trunk}$9 define the low-rank update for the output layer, which in turn generates the trunk net weights.

3. Parameter Reduction and Regularization

The standard HyperDeepONet's output layer possesses $$W^{\rm branch}_{\rm out} \in \mathbb{R}^{n_{\rm trunk} \times n_{\rm hidden}}$$0 parameters, whereas the LoRA adaptation reduces this to $$W^{\rm branch}_{\rm out} \in \mathbb{R}^{n_{\rm trunk} \times n_{\rm hidden}}$$1. Empirical results demonstrate up to a 70%–90% reduction in trainable parameters for typical configurations ($$W^{\rm branch}_{\rm out} \in \mathbb{R}^{n_{\rm trunk} \times n_{\rm hidden}}$$2). By restricting parameter updates to a low-dimensional subspace, the model introduces an implicit regularization that reduces overfitting and simplifies the loss landscape. This coupling of trunk net weights thereby constrains the hypothesis space, discouraging high-frequency or spurious weight configurations.

A summary of parameter counts for illustrative cases:

Configuration	Params Full	Params LoRA ($$W^{\rm branch}_{\rm out} \in \mathbb{R}^{n_{\rm trunk} \times n_{\rm hidden}}$$3)	Fraction LoRA/Full
HyperDeepONet ($$W^{\rm branch}_{\rm out} \in \mathbb{R}^{n_{\rm trunk} \times n_{\rm hidden}}$$4)	$$W^{\rm branch}_{\rm out} \in \mathbb{R}^{n_{\rm trunk} \times n_{\rm hidden}}$$5	$$W^{\rm branch}_{\rm out} \in \mathbb{R}^{n_{\rm trunk} \times n_{\rm hidden}}$$6 ($$W^{\rm branch}_{\rm out} \in \mathbb{R}^{n_{\rm trunk} \times n_{\rm hidden}}$$7)	$$W^{\rm branch}_{\rm out} \in \mathbb{R}^{n_{\rm trunk} \times n_{\rm hidden}}$$8

4. Expressivity, Theoretical Implications, and Generalization

While a rank-$$W^{\rm branch}_{\rm out} \in \mathbb{R}^{n_{\rm trunk} \times n_{\rm hidden}}$$9 update cannot realize the full parameter space of $$b^{\rm branch}_{\rm out} \in \mathbb{R}^{n_{\rm trunk}}$$0, for many operator-learning tasks, the learnable operator Jacobians effectively reside on a low-dimensional manifold. The imposed low-rank constraint impedes memorization of training-specific patterns and biases learning toward more global, low-complexity solutions.

Current theoretical results for operator approximation bounds in this architecture remain incomplete, but it is posited that if the target operator Jacobian is approximately $$b^{\rm branch}_{\rm out} \in \mathbb{R}^{n_{\rm trunk}}$$1-dimensional, a rank-$$b^{\rm branch}_{\rm out} \in \mathbb{R}^{n_{\rm trunk}}$$2 factorization yields error comparable to the full model. Empirically, smoother loss landscapes and improved generalization follow from this architectural bias.

5. Empirical Performance on Benchmark Problems

Experiments on classical operator benchmarks, all in a physics-informed (unsupervised) regime, demonstrate the efficacy of PI-LoRA-HyperDeepONet. Models were trained with $$b^{\rm branch}_{\rm out} \in \mathbb{R}^{n_{\rm trunk}}$$3 initial functions, $$b^{\rm branch}_{\rm out} \in \mathbb{R}^{n_{\rm trunk}}$$4 collocation points, and evaluated on $$b^{\rm branch}_{\rm out} \in \mathbb{R}^{n_{\rm trunk}}$$5 with 10 random seeds per configuration.

Results are summarized below:

Problem	Model	#Params	Error (short)	Error (long, if applicable)
Harmonic oscillator	DeepONet	15,060	0.0099 ± 0.0062	0.076 ± 0.048
$$b^{\rm branch}_{\rm out} \in \mathbb{R}^{n_{\rm trunk}}$$6	HyperDONet	14,902	0.0119 ± 0.0203	0.075 ± 0.089
	LoRA ($$b^{\rm branch}_{\rm out} \in \mathbb{R}^{n_{\rm trunk}}$$7)	6,890	0.0067 ± 0.0060	0.045 ± 0.036
Rigid body	DeepONet	29,740	0.0052 ± 0.0037	0.0469 ± 0.0272
(Euler, $$b^{\rm branch}_{\rm out} \in \mathbb{R}^{n_{\rm trunk}}$$8)	HyperDONet	29,963	0.0012 ± 0.0006	0.0110 ± 0.0035
	LoRA ($$b^{\rm branch}_{\rm out} \in \mathbb{R}^{n_{\rm trunk}}$$9)	8,235	0.0010 ± 0.0003	0.0107 ± 0.0034
1D advection	DeepONet	297,216	0.0800 ± 0.0348	—
$(t, x)$0	HyperDONet	279,682	0.0483 ± 0.0054	—
	LoRA ($(t, x)$1)	61,258	0.0284 ± 0.0051	—
Burgers’ equation	DeepONet	116,096	0.3161 ± 0.1051	—
($(t, x)$2)	HyperDONet	117,389	0.1130 ± 0.0044	—
	LoRA ($(t, x)$3)	73,227	0.1053 ± 0.0093	—
1D shallow-water	DeepONet	114,750	0.5918 ± 0.0285	—
	HyperDONet	114,802	0.0126 ± 0.0023	—
	LoRA ($(t, x)$4)	94,474	0.0110 ± 0.0022	—

Across benchmarks, PI-LoRA-HyperDeepONet achieves up to 70% fewer parameters than standard alternatives, with performance matching or surpassing full HyperDeepONet and DeepONet variants in both short- and long-time predictions. Optimal ranks are problem-dependent (e.g., $(t, x)$5–8 for ODEs, $(t, x)$6–32 for PDEs), but always $(t, x)$7.

6. Practical Significance and Limitations

The model delivers a favorable tradeoff between parameter efficiency and expressivity, enabling substantial reductions in computational resource requirements. The technique generalizes across ODE and PDE benchmarks, showing robust operator approximation even in stiff or shock-forming regimes. This suggests wide applicability for physics-informed operator learning in scientific machine learning domains.

A plausible implication is that for highly complex operator targets with intrinsically high-rank Jacobians, a fixed small $(t, x)$8 may limit performance. Conversely, for most practical problems, operator manifolds are sufficiently low-dimensional, and low-rank adaptation suffices.

7. Summary and Outlook

PI-LoRA-HyperDeepONet advances the state of operator learning by introducing low-rank adaptive decomposition into the hypernetwork-trunk mapping within the HyperDeepONet framework. This modification yields models that are lightweight, generalizable, and resistant to overfitting, while retaining or improving predictive accuracy and generalization on both ODE and PDE benchmarks. The architecture’s capacity to maintain high expressivity with reduced computational cost positions it as an efficient tool for physics-informed operator learning scenarios, with the potential for broader adoption as further theoretical insights and practical extensions are developed (Zeudong et al., 24 Jul 2025).