DISCO Operator Meta-Learning

• Operator meta-learning (DISCO) is a framework that uses hypernetworks to generate evolution operators from short observational sequences in systems governed by unknown PDEs.
• It decouples the discovery of underlying system dynamics from numerical integration, enabling robust multi-physics prediction with state-of-the-art accuracy.
• The approach leverages efficient meta-learning with a U-Net based operator network, reducing fine-tuning time and eliminating the need for inference-time retraining.

Operator meta-learning, instantiated in the DISCO (DISCover an evolution Operator) framework, is a paradigm for temporal prediction in dynamical systems governed by unknown partial differential equations (PDEs), targeting scenarios where each data instance corresponds to a distinct and potentially unrelated physics context. DISCO achieves multi-physics-agnostic state prediction by meta-learning to generate efficient evolution operators from short observational sequences, decoupling the discovery of underlying system dynamics from the integration of those dynamics for forecasting. This approach advances data-driven modeling of spatiotemporal processes by leveraging hypernetworks for operator parameterization and meta-learning principles to generalize across varied dynamical regimes (Morel et al., 28 Apr 2025).

1. Formal Problem Setup

Operator meta-learning in the DISCO framework addresses the following setting: Observations are provided as discretized short trajectories $S = \{u_{t-T+1},\ldots, u_t\}$, where each $S$ is generated by a time-homogeneous, but unknown, PDE of the form:

$$\partial_t u(t,x) = g(u, \nabla_x u, \nabla^2_x u, \ldots), \quad x \in \Omega \subset \mathbb{R}^n,$$

combined with associated boundary and initial conditions. For each $S$, the evolution law $g$, coefficients, and all physical details—such as boundary terms and initializations—can vary, so each context represents an effectively distinct PDE instance. The learning objective is to predict the next frame $u_{t+1}$ (or $u_{t+\Delta t}$) from $S$ in a manner that generalizes across the dataset $\mathcal{D}$ of such trajectories. The global objective is

$$\min_{\varphi} \frac{1}{|\mathcal{D}|} \sum_{S \in \mathcal{D}} \mathrm{Loss}(u_{t+1}, \hat{u}_{t+1}(S; \varphi)),$$

where the parameters $S$0 specify the meta-learned predictive model (Morel et al., 28 Apr 2025).

2. Model Architecture

DISCO factorizes meta-sequence prediction into two distinct modules:

• Hypernetwork ($S$1): A high-capacity video/spatiotemporal transformer that ingests the short context $S$2 and, via learned transformations and pooling, outputs the full set of parameters $S$3 for a compact operator network.
  • Architecture: $S$412 axial attention layers, each with 6 heads and hidden/token dimension 384.
  • Patchification: Small CNN encoders map each context frame to patch embeddings.
  • Parameter generation: A 3-layer MLP expands the pooled embedding (384-dim) to the full convolutional parameterization (approx. 200K parameters), with layer-wise normalization enforcing weight scale consistency:

  $S$5

  Where $S$6 is the PyTorch initialization norm, $S$7, and $S$8 is the sigmoid.

• Operator Network ($S$9): Receives operator parameters from the hypernetwork and implements a U-Net architecture, including:

  • Four down-sampling and four up-sampling blocks, with a multilayer perceptron bottleneck.
  • GeLU activations and GroupNorm (GN, 4 groups).
  • Reflection-padding and mask channels for nonperiodic boundaries.
  • Channel widths: input/output channels match physical PDE fields; base width 8, maximum 128; total $$\partial_t u(t,x) = g(u, \nabla_x u, \nabla^2_x u, \ldots), \quad x \in \Omega \subset \mathbb{R}^n,$$0200K parameters.

The prediction is made by time integration of $$\partial_t u(t,x) = g(u, \nabla_x u, \nabla^2_x u, \ldots), \quad x \in \Omega \subset \mathbb{R}^n,$$1 applied to the last observed state:

$$\partial_t u(t,x) = g(u, \nabla_x u, \nabla^2_x u, \ldots), \quad x \in \Omega \subset \mathbb{R}^n,$$2

3. Time Integration and Differentiation

Time evolution is implemented via a continuous-time numerical integrator:

$$\partial_t u(t,x) = g(u, \nabla_x u, \nabla^2_x u, \ldots), \quad x \in \Omega \subset \mathbb{R}^n,$$3

The integration uses a Bogacki–Shampine adaptive RK3 (Runge–Kutta) solver with at most 32 sub-steps. Differentiation through the temporal integration is performed using the adjoint method, ensuring gradients with respect to $$\partial_t u(t,x) = g(u, \nabla_x u, \nabla^2_x u, \ldots), \quad x \in \Omega \subset \mathbb{R}^n,$$4 flow back through both the operator and the transformer's meta-parameters (Morel et al., 28 Apr 2025).

4. Meta-learning Objective, Training, and Loss

The learning objective is the minimization of normalized root-mean-square error (NRMSE) per field $$\partial_t u(t,x) = g(u, \nabla_x u, \nabla^2_x u, \ldots), \quad x \in \Omega \subset \mathbb{R}^n,$$5:

$$\partial_t u(t,x) = g(u, \nabla_x u, \nabla^2_x u, \ldots), \quad x \in \Omega \subset \mathbb{R}^n,$$6

The operator meta-learning process comprises:

• Pretraining over multi-physics datasets (PDEBench, The Well), with context lengths (e.g., $$\partial_t u(t,x) = g(u, \nabla_x u, \nabla^2_x u, \ldots), \quad x \in \Omega \subset \mathbb{R}^n,$$7) and resolutions up to $$\partial_t u(t,x) = g(u, \nabla_x u, \nabla^2_x u, \ldots), \quad x \in \Omega \subset \mathbb{R}^n,$$8 or $$\partial_t u(t,x) = g(u, \nabla_x u, \nabla^2_x u, \ldots), \quad x \in \Omega \subset \mathbb{R}^n,$$9.
• Optimization using the ADAN optimizer (adaptive Nesterov) with cosine learning rate decay, weight decay $S$0, DropPath 0.1, batch sizes 8–32, and 2000 batches per epoch for 300 epochs (multi-dataset).
• Fine-tuning for downstream tasks is required only for the hypernetwork ($S$1), with no retraining of $S$2. For transfer to a new fixed PDE, $S$3 is further updated on the new context (Morel et al., 28 Apr 2025).

5. Empirical Performance and Baselines

Performance is consistently reported in NRMSE on benchmarks:

• Next-step prediction (PDEBench): After 300 epochs of joint pretraining, DISCO matches or exceeds the state-of-the-art MPP on diverse PDEs (e.g., Burgers, 2D Shallow Water, Diffusion–Reaction, Incompressible NS), with a temporary deficit on Compressible NS (CNS) in joint training (0.095 vs. 0.031) that is reversed with CNS-only training (0.041 vs. 0.072 in 50 epochs).
• Rollout prediction: DISCO maintains competitive accuracy at longer horizons ($S$4, $S$5, etc.), outperforming GEPS and closely tracking Poseidon—despite Poseidon’s explicit multi-horizon training.
• Large-scale rollouts (The Well): At 50 epochs, DISCO outperforms GEPS and MPP across 9 multi-physics datasets (2D and 3D). Notably, DISCO operates without a decoder and requires no inference-time retraining.
• Fine-tuning on unseen Euler PDEs: Starting from pretrained weights, 20 additional epochs on a new PDE (Euler with fixed $S$6) yield NRMSE 0.029 ($S$7), compared to 0.032 (MPP), 0.052 (Poseidon), 0.36 (GEPS), 0.050 (U-Net), and 0.067 (FNO).
• Operator parameter clustering: UMAP visualizations of $S$8 reveal clustering by PDE coefficient (e.g., viscosity $S$9), indicating that $g$0 identifies crucial PDE parameters and is invariant to initial conditions.

Baseline Comparison Table

Baseline	Next-step NRMSE (Euler, $g$1)	Decoder/Inference Retraining
DISCO	0.029	None
MPP	0.032	None
Poseidon	0.052	Yes (decoder)
GEPS	0.36	Environment-specific
U-Net	0.050	Retrained for each PDE
FNO	0.067	Retrained for each PDE

6. Ablation, Robustness, and Insights

Systematic ablations in DISCO demonstrate:

• Hypernetwork capacity: Increasing layers, heads, and hidden dimension improves NRMSE to a point (e.g., 4 layers/3 heads/dim 192: 2.11e−3 NRMSE; 12L/6H/dim 384: 1.33e−3; further increases can yield over-parameterization and worse NRMSE).
• Operator network width: More base channels reduce NRMSE (4 channels/41K $g$2: 3.30e−3; 8/158K: 1.33e−3; 12/353K: 0.97e−3).
• Translation equivariance: DISCO's architecture is stable under spatial shifts, in contrast to standard transformers, which degrade rapidly.
• Fine-tuning efficiency: DISCO requires approximately five times fewer epochs than MPP to achieve state-of-the-art results.
• Parameter space structure: Clustering in $g$3 space, conditioned on PDE coefficients and invariant to initial conditions, demonstrates that the hypernetwork captures underlying physics variation.

7. Key Equations, Pseudocode, and Hyperparameters

Central Equations

• PDE: $g$4
• Parameter mapping: $g$5
• Prediction: $g$6
• Loss: $g$7

Training Loop (pseudocode)

$u_{t+1}$3

Default Hyperparameters

Component	Setting	Scale/Notes
Context Length	$g$8
Hypernetwork	12L, 6H, dim 384	$g$9M
Operator U-Net	4 downsamples, base ch. 8	$u_{t+1}$0K
Integrator	RK3, $u_{t+1}$132 steps	Adaptive
Optimizer	ADAN, cosine decay, wd $u_{t+1}$2	DropPath 0.1
Batch Size	8 (multi), 32 (single)
Pretraining	300 epochs
Fine-tuning	20 epochs

DISCO's methodology and experimental results establish operator meta-learning with hypernetwork-parameterized spatiotemporal evolution as a robust framework for multi-physics PDE prediction, exhibiting both strong generalization and empirical efficiency across broad classes of dynamical systems (Morel et al., 28 Apr 2025).