Sequential Deep Operator Network (S-DeepONet)

• S-DeepONet is a framework that extends DeepONet by incorporating sequential encoders (GRU/LSTM) and autoregressive methods to capture temporal causality and path dependence.
• It employs diverse architectural strategies—such as recurrent branches, vectorized multi-time prediction, and two-step sequential training—to improve prediction accuracy and stability in long-horizon rollouts.
• Applications in transient heat transfer, fluid dynamics, and multiphysics problems demonstrate significant speedups and enhanced accuracy compared to traditional simulation methods.

Sequential Deep Operator Network (S-DeepONet) denotes a set of DeepONet extensions designed to handle sequential structure in operator learning. In the cited literature, the label covers at least three closely related ideas: replacing the branch network with a sequential encoder such as a GRU or LSTM to process time-dependent loads or boundary histories; learning a one-step time-advance operator and applying it autoregressively to generate long trajectories; and training DeepONet in a sequential two-step manner by learning the trunk first and the branch second. Across these variants, the shared objective is to preserve the operator-learning character of DeepONet while improving its handling of temporal causality, path dependence, long-horizon rollout, or optimization stability (He et al., 2023, He et al., 2023, Bai et al., 2024, Lee et al., 2023, Park et al., 8 Jul 2025).

1. Scope and nomenclature

The earliest usage in the cited set appears in work that extends classical DeepONet by introducing sequential learning models such as the gated recurrent unit (GRU) and long short-term memory (LSTM) into the branch network, so that time-dependent or path-dependent loads can be mapped directly to full-field solutions (He et al., 2023). Later work generalizes the same idea to simultaneous prediction of multiple components and multiple output times in a single forward pass (He et al., 2023). A separate CFD-oriented formulation realizes the same underlying concept by learning a one-step operator from a short window of flow snapshots to the next snapshot and then applying that operator sequentially to roll out a long trajectory; the source paper states that the term “S-DeepONet” is not explicit there, but the methodology embodies the concept (Bai et al., 2024). Another paper uses “sequential” differently: it proposes a two-step training method in which the trunk is trained first, orthonormalized by QR or Gram–Schmidt, and only then is the branch trained (Lee et al., 2023).

Variant	Sequential mechanism	Prediction regime
Recurrent-branch S-DeepONet	GRU or LSTM in the branch	Final fields or directly queried spatio-temporal fields
Autoregressive DeepONet	Learned one-step operator applied repeatedly	Long-horizon rollout of future states
Two-step sequential training	Trunk-first, QR/Gram-Schmidt, then branch fitting	Same operator class with improved conditioning
Hybrid prior-refinement pipeline	S-DeepONet prior followed by residual video diffusion	Full spatio-temporal videos with refined detail

This diversity of usage is central to the topic. A recurrent-branch model that predicts final-time fields, a direct spatio-temporal operator that is not autoregressive in the state, and an autoregressive one-step propagator are all called, or are described as embodying, S-DeepONet in the cited literature (He et al., 2023, He et al., 2023, Bai et al., 2024, Park et al., 8 Jul 2025).

2. Operator formulations

The underlying mathematical core remains the standard DeepONet branch–trunk decomposition. In the notation used across the cited papers, the operator approximation takes the form

$$\hat{G}(m)(X)=\sum_{i=1}^{HD} b_i(m)\,t_i(X)+\beta,$$

or, equivalently, $B(u)^\top T(x)$ plus bias, where the branch encodes an input function or history and the trunk encodes the query location (He et al., 2023, Kushwaha et al., 2024).

For sequential inputs, one common extension is to let the branch encode a full history $a(t)$ while the trunk encodes space-time coordinates. In the spatio-temporal video formulation,

$$\hat{u}(x,y,t,\boldsymbol{a})=\sum_{i=1}^{HD} Br_i(\boldsymbol{a})\,Tr_i(x,y,t)+\beta,$$

with $HD=200$, a 4-layer GRU branch, and a 6-layer fully connected trunk. The branch summarizes the entire loading history, while the trunk localizes the response in $(x,y,t)$. In that formulation, the model is explicitly stated to be non-autoregressive in the predicted field $y$: it produces the field at any queried $t$ directly from the encoded input history (Park et al., 8 Jul 2025).

A second formulation targets multiple components and multiple output times simultaneously. There, the branch produces $B\in\mathbb{R}^{HD\times S}$, the trunk produces $T\in\mathbb{R}^{N\times HD\times C}$, and the output is obtained by tensor contraction,

$B(u)^\top T(x)$0

This yields a field tensor over spatial nodes $B(u)^\top T(x)$1, output times $B(u)^\top T(x)$2, and components $B(u)^\top T(x)$3 in a single forward pass. The interpretation given in the source is that the trunk learns $B(u)^\top T(x)$4 spatial basis shapes per component, while the branch supplies $B(u)^\top T(x)$5 weights per output time step (He et al., 2023).

A third formulation is explicitly time-advancing and autoregressive. For the unsteady cylinder-wake problem, DeepONet is trained on a one-step map

$B(u)^\top T(x)$6

where the branch ingests a window of $B(u)^\top T(x)$7 consecutive snapshots and the trunk ingests $B(u)^\top T(x)$8. At inference, the same learned operator is applied repeatedly: $B(u)^\top T(x)$9 This is the formulation in which error accumulation during rollout becomes a dominant issue (Bai et al., 2024).

The sequential training variant of DeepONet changes optimization rather than input encoding. It first solves

$a(t)$0

then applies a thin QR factorization $a(t)$1, orthonormalizes the trunk through $a(t)$2, and finally trains the branch by regressing the coefficient targets $a(t)$3: $a(t)$4 The cited paper proves that, under mild overparameterization of the branch, the achieved loss matches the monolithic optimum and derives a generalization bound in terms of training data, width, and sensor counts (Lee et al., 2023).

3. Architectural realizations

The recurrent-branch literature uses explicitly sequential encoders in the branch. In the 2023 formulation for final full-field prediction under time-dependent loads, the GRU branch is an encoder–decoder with four GRU layers: 256 units and 128 units in the encoder, a RepeatVector, then 128 units and 256 units in the decoder, followed by a TimeDistributed dense layer with linear activation. The LSTM version is analogous. The trunk is a seven-layer FNN with layer sizes $a(t)$5, and the branch latent dimension is aligned with the 101-step input sequence (He et al., 2023).

The vector, multi-time extension retains a four-layer GRU branch but modifies the information-combination mechanism. The branch output is time-indexed, the trunk output is component-indexed, and the final contraction is implemented as

$a(t)$6

This design allows simultaneous prediction of multiple fields across multiple time steps while sharing trunk and branch representations across outputs (He et al., 2023).

The autoregressive flow-forecasting variant uses a substantially different branch architecture. Its branch is a CNN with an inception-style multi-encoder design: the input window of $a(t)$7 snapshots is stacked along channels and passed through parallel $a(t)$8, $a(t)$9, and $$\hat{u}(x,y,t,\boldsymbol{a})=\sum_{i=1}^{HD} Br_i(\boldsymbol{a})\,Tr_i(x,y,t)+\beta,$$0 encoders, with the latter two preceded by $$\hat{u}(x,y,t,\boldsymbol{a})=\sum_{i=1}^{HD} Br_i(\boldsymbol{a})\,Tr_i(x,y,t)+\beta,$$1 convolutions for parameter reduction. Their outputs are concatenated and fused by a $$\hat{u}(x,y,t,\boldsymbol{a})=\sum_{i=1}^{HD} Br_i(\boldsymbol{a})\,Tr_i(x,y,t)+\beta,$$2 convolution. A non-local or multi-head non-local block is inserted in deeper branch layers to capture long-range dependencies in the wake. The trunk is a feed-forward network over $$\hat{u}(x,y,t,\boldsymbol{a})=\sum_{i=1}^{HD} Br_i(\boldsymbol{a})\,Tr_i(x,y,t)+\beta,$$3, and the branch–trunk fusion remains a dot product over latent features (Bai et al., 2024).

That same flow paper also introduces two optimization-oriented stabilizers. The first is branch-side non-local or multi-head non-local attention, with $$\hat{u}(x,y,t,\boldsymbol{a})=\sum_{i=1}^{HD} Br_i(\boldsymbol{a})\,Tr_i(x,y,t)+\beta,$$4 independent heads and substantial gains reported for $$\hat{u}(x,y,t,\boldsymbol{a})=\sum_{i=1}^{HD} Br_i(\boldsymbol{a})\,Tr_i(x,y,t)+\beta,$$5. The second is local smooth optimization, implemented through SAM and GSAM. The reported best hyperparameters are $$\hat{u}(x,y,t,\boldsymbol{a})=\sum_{i=1}^{HD} Br_i(\boldsymbol{a})\,Tr_i(x,y,t)+\beta,$$6 for SAM and $$\hat{u}(x,y,t,\boldsymbol{a})=\sum_{i=1}^{HD} Br_i(\boldsymbol{a})\,Tr_i(x,y,t)+\beta,$$7 for GSAM (Bai et al., 2024).

The hybrid spatio-temporal PDE surrogate uses S-DeepONet as a coarse prior generator. There, the branch is a 4-layer GRU encoder producing a 200-dimensional latent, the trunk is a 6-layer ReLU FNN producing a 200-dimensional latent, and the output is an Einstein-summation inner product plus a trainable bias. This prior video is then refined by a depth-4 encoder–decoder 3D U-Net diffusion model conditioned by FiLM on the input history and by direct concatenation of the S-DeepONet prior video; for the dogbone problem, a binary geometry mask is also concatenated (Park et al., 8 Jul 2025).

4. Temporal regimes, rollout behavior, and stability

The cited literature separates into three temporal regimes. The first predicts only the final state of a process from its full load history. This is the regime used for transient heat transfer, path-dependent plastic loading, steel continuous casting, and directed energy deposition. In those formulations, the recurrent branch absorbs the entire history and the operator returns only the final full-field temperature or stress solution; there is no autoregressive state rollout and no prediction of the intermediate time history (He et al., 2023, Kushwaha et al., 2024).

The second regime predicts multiple times directly. The vector S-DeepONet predicts $$\hat{u}(x,y,t,\boldsymbol{a})=\sum_{i=1}^{HD} Br_i(\boldsymbol{a})\,Tr_i(x,y,t)+\beta,$$8 output time steps and $$\hat{u}(x,y,t,\boldsymbol{a})=\sum_{i=1}^{HD} Br_i(\boldsymbol{a})\,Tr_i(x,y,t)+\beta,$$9 field components in one pass through the tensor contraction $HD=200$0 (He et al., 2023). The video-prior formulation likewise treats time $HD=200$1 as an explicit trunk input and states that the model is not autoregressive in $HD=200$2; the branch encodes the full loading history, and the field can be evaluated at any queried $HD=200$3 (Park et al., 8 Jul 2025).

The third regime is explicitly autoregressive. In the cylinder-wake study, DeepONet is trained as a one-step propagator on 2D flow around a circular cylinder at $HD=200$4, with branch inputs consisting of $HD=200$5 consecutive snapshots and trunk inputs $HD=200$6. The paper reports that, for vanilla DeepONet at $HD=200$7, the MSE over 4000 extrapolated snapshots grows from $HD=200$8 initially to $HD=200$9. Adding a one-head non-local block improves the average MSE over snapshots 3500–4000 to $(x,y,t)$0; multi-head non-local blocks with 5, 7, or 9 heads improve it to $(x,y,t)$1; SAM with $(x,y,t)$2 is comparable to the 9-head case; and GSAM with $(x,y,t)$3 yields the smallest long-horizon MSEs with negligible accumulation across 4000 steps. The same study also reports that increasing $(x,y,t)$4 from 6 to 11 reduces long-horizon error, with $(x,y,t)$5 particularly effective, while larger $(x,y,t)$6 increases fluctuations and sensitivity (Bai et al., 2024).

A related but distinct long-time strategy is the transfer-learning aided DeepONet for evolution equations. That work learns a short-step operator and then sequentially updates only the last linear layer of the branch during rollout, producing a sequence of locally adapted propagators $(x,y,t)$7. It reports large improvements in long-time stability and sample efficiency across reaction–diffusion, Allen–Cahn, Cahn–Hilliard, Navier–Stokes, and radiative transfer equations, while keeping the additional adaptation cost to a small least-squares solve at each step (Xu et al., 2022).

5. Applications and reported performance

The initial recurrent-branch S-DeepONet paper studies transient heat transfer with phase transformation and path-dependent plastic deformation. For heat transfer, the reported test performance at an 80–20 split is Relative $(x,y,t)$8 for FNN-DeepONet, $(x,y,t)$9 for GRU-DeepONet, and $y$0 for LSTM-DeepONet, with $y$1 values of $y$2, $y$3, and $y$4, respectively. For plasticity, the corresponding Relative $y$5 errors are $y$6, $y$7, and $y$8, with $y$9, $t$0, and $t$1. The same work reports inference speedups of at least two orders of magnitude over finite element simulation, and states that once trained the proposed DeepONets are at least two orders of magnitude faster than direct finite element simulations (He et al., 2023).

The vector, multi-time S-DeepONet extends the framework to transient fluid flow and path-dependent plastic loading. It reports that in almost all cases the trained model achieved an $t$2 value of above 0.99 and a relative $t$3 error of less than 10% with only 3200 training data points. The vector S-DeepONet has only 0.4% more parameters than a scalar model, can predict two output components simultaneously at an accuracy similar to two independently trained scalar models, and requires a 20.8% faster training time. For the dog-bone plasticity problem, the vector model infers both outputs in $t$4, versus approximately $t$5 for two scalar models, while finite element runtime per case is about 48 s (He et al., 2023).

Sequential DeepONet has also been applied to sequentially coupled multiphysics. In steel continuous casting, a GRU-branch S-DeepONet predicts final full-field temperature and von Mises stress from time-dependent chilled-surface heat-flux and displacement histories. On a 1,099-case test set, the reported values are $t$6 for temperature and $t$7 for stress, with worst 90th-percentile MAEs of $t$8 and $t$9, respectively. Inference time is $B\in\mathbb{R}^{HD\times S}$0 for 1,099 samples, or $B\in\mathbb{R}^{HD\times S}$1, versus about $B\in\mathbb{R}^{HD\times S}$2 for FEA, corresponding to a reported speedup of approximately $B\in\mathbb{R}^{HD\times S}$3. The same paper uses a ResUNet-based DeepONet for additive manufacturing with variable geometries; that model reports $B\in\mathbb{R}^{HD\times S}$4 for temperature and $B\in\mathbb{R}^{HD\times S}$5 for stress, with a reported speedup of approximately $B\in\mathbb{R}^{HD\times S}$6 (Kushwaha et al., 2024).

For autoregressive CFD forecasting, the cylinder-wake study reports online forecasting wall times for 4000 snapshots on an NVIDIA RTX 4090 of 26 s for vanilla DeepONet, 45 s for DeepONet with a one-head non-local block, and 91/131/173/215 s for multi-head variants with 3/5/7/9 heads. The same work states that SAM and GSAM improve accuracy without increasing online inference cost because they affect training only, and that the operator rollout generates thousands of snapshots in seconds whereas DNS integration is orders of magnitude more expensive. It also reports that DeepONet+GSAM consistently achieves much lower MSEs than DMD across $B\in\mathbb{R}^{HD\times S}$7 (Bai et al., 2024).

The hybrid residual-refinement framework uses S-DeepONet as a fast, globally coherent prior and then refines only the residual with conditional video diffusion. On lid-driven cavity flow, the mean test Relative $B\in\mathbb{R}^{HD\times S}$8 error decreases from $B\in\mathbb{R}^{HD\times S}$9 for S-DeepONet to $T\in\mathbb{R}^{N\times HD\times C}$0 for prior-conditioned residual diffusion; on dogbone plasticity, it decreases from $T\in\mathbb{R}^{N\times HD\times C}$1 to $T\in\mathbb{R}^{N\times HD\times C}$2, and the MAE decreases from $T\in\mathbb{R}^{N\times HD\times C}$3 to $T\in\mathbb{R}^{N\times HD\times C}$4. Runtime is reported as approximately 1,140 s for CFD, 0.01 s for S-DeepONet, and 1.7 s for diffusion on the flow task; for dogbone plasticity it is about 20 s for FEA, 0.02 s for S-DeepONet, and 7 s for diffusion (Park et al., 8 Jul 2025).

6. Relations to adjacent operator learners, limitations, and future directions

Within the operator-learning landscape, DeepONet is distinguished by branch–trunk dot-product fusion. One cited comparison states that DeepONet uses universal operator approximation via branch–trunk dot products, FNO learns Fourier-space integral operators via spectral convolutions, and Transformer-operator learns attention-based nonlocal kernels. The same source emphasizes that its cylinder-wake model is purely supervised and not constrained by PDE residuals, in contrast to PINO or physics-informed DeepONet variants (Bai et al., 2024).

The dot-product fusion is also the basis of a major critique. In the comparison between S-DON and the Sequential Neural Operator Transformer (S-NOT), S-DON is described as using a single global sequence code $T\in\mathbb{R}^{N\times HD\times C}$5 and a query embedding $T\in\mathbb{R}^{N\times HD\times C}$6, with fusion

$T\in\mathbb{R}^{N\times HD\times C}$7

That paper argues that this produces a query-invariant, rank-1 coupling and can limit expressivity when different spatial regions must attend to different portions of the loading history. Across multiphysics steel solidification, a 3D lug, and a dog-bone specimen, S-NOT is reported to reduce mean stress errors from $T\in\mathbb{R}^{N\times HD\times C}$8 to $T\in\mathbb{R}^{N\times HD\times C}$9, $B(u)^\top T(x)$00 to $B(u)^\top T(x)$01, and $B(u)^\top T(x)$02 to $B(u)^\top T(x)$03, respectively, while maintaining comparable parameter counts and inference times (Liu et al., 4 Jul 2025).

The limitations reported for S-DeepONet are correspondingly variant-dependent. Final-state recurrent-branch models predict only the terminal field and do not learn the full temporal evolution (He et al., 2023, Kushwaha et al., 2024). Pure MSE training can underweight low-magnitude targets, leading to disproportionately poor performance in low-stress or early-time regimes (He et al., 2023). In autoregressive rollout, fine-scale wake structures are harder to predict because of spectral bias, and stability deteriorates as the snapshot interval $B(u)^\top T(x)$04 increases (Bai et al., 2024). In the hybrid prior-diffusion setting, no hard PDE constraints are enforced; physics-awareness arises from conditioning and from the operator prior, while diffusion-only baselines can fail badly when the mapping from loading history to field is complex (Park et al., 8 Jul 2025).

The future directions reported in the cited literature are similarly diverse. They include incorporating all simulation time steps rather than only the final state, extending to 3D geometries, adding physics-informed regularization, developing uncertainty-aware surrogates, using multi-resolution refinement, distilling diffusion samplers, and introducing mixture-of-experts for multi-regime generalization (Kushwaha et al., 2024, Park et al., 8 Jul 2025). Taken together, these proposals suggest that S-DeepONet is best understood not as a single fixed architecture but as a continuing line of operator-learning methods that adapt DeepONet to sequential inputs, sequential rollout, or sequential training while retaining the branch–trunk operator representation.