Flow Marching Transformer for PDEs

• The paper introduces FMT as a generative PDE backbone that couples diffusion forcing with latent flow matching to reduce long-term rollout drift.
• FMT is built with a SiT-style transformer using AdaLN-Zero and leverages a multi-scale latent temporal pyramid from a physics-pretrained VAE for efficient temporal conditioning.
• Empirical results show FMT achieves up to 15× throughput gains and improved few-shot adaptation on turbulent PDE simulations compared to deterministic models.

Searching arXiv for the specified paper and closely related work to ground the article. arxiv_search: query="^^^^^^^^1(Chen et al., 23 Sep 2025) OR \1^^^^^^^^"Flow marching for a generative PDE foundation model\""

Flow Marching Transformer (FMT) is the core neural backbone introduced in “Flow marching for a generative PDE foundation model” for generative modeling of PDE-governed spatiotemporal dynamics (&&&^^^^1(Chen et al., 23 Sep 2025) OR \1^^^^&&&). Within the broader Flow Marching framework, FMT implements the conditional flow-marching velocity field PRESERVED_PLACEHOLDER_^^^^1(Chen et al., 23 Sep 2025) OR \1^^^^ that transports a noisy latent state toward its clean successor, thereby coupling neural operator learning with flow matching. The model operates on latent embeddings produced by a Physics-Pretrained Variational Autoencoder (P2VAE), combines a diffusion-forcing history filter with a multi-scale latent temporal pyramid, and is trained on a corpus of approximately $2.5\text{M}$ trajectories spanning 12 distinct PDE families. Its stated purpose is to reduce long-term rollout drift while enabling uncertainty-aware ensemble generation, in contrast to deterministic Transformer-based PDE foundation models that lack generative flexibility (&&&^^^^1(Chen et al., 23 Sep 2025) OR \1^^^^&&&).

1. Functional role in the Flow Marching framework

FMT occupies the predictive core of the Flow Marching pipeline. The P2VAE first maps each physical frame $x_s \in \mathbb{R}^{H \times W \times C}$ to a latent grid $y_s \in \mathbb{R}^{H' \times W' \times C'}$. For a block of four consecutive times $\{s,s+1,s+2,s+3\}$, noisy latent states are constructed, passed through a small GRU or RNN with parameters $\phi$, and summarized into history codes $h_i$ that serve as the “diffusion-forcing” state. FMT then processes the multi-scale latent tokens together with temporal conditioning and the history summary to predict a preconditioned residual of the form

$$(1-t)\,g_\theta(x_{s,t}^k,t,h_{s-1}) \approx x_{s+1} - x_{s,t}^k.$$

The model is therefore not a direct next-step regressor in the deterministic neural-operator sense. Instead, it learns a unified velocity field over jointly sampled physical time and noise level. In the formulation given for FMT, the transformer stack fuses noisy latent observations from several recent time points, while the RNN state provides a learned summary of prior dynamics. A plausible implication is that the model treats short-term state evolution and uncertainty transport as parts of a single conditional generative process rather than as separate forecasting and noise-injection stages (&&&^^^^1(Chen et al., 23 Sep 2025) OR \1^^^^&&&).

2. Flow-marching objective and diffusion-forcing formulation

For each physical step $s \rightarrow s+1$, Flow Marching samples

$$t \sim \mathrm{Uniform}(0,1), \qquad k \sim \mathrm{Uniform}(0,1), \qquad z \sim \mathcal{N}(0,I),$$

and constructs the noisy state

$2.5\text{M}$^^^^1(Chen et al., 23 Sep 2025) OR \1^^^^

After P2VAE encoding, the analogous expression is used in latent space for $2.5\text{M}$1.

The instantaneous ground-truth velocity is defined as

$2.5\text{M}$2

and the paper states the decomposition

$2.5\text{M}$3

To avoid feeding $2.5\text{M}$4 into the network, the training rule adopts the simpler frame-interpolation target

$2.5\text{M}$5

Preconditioning near $2.5\text{M}$6 yields the flow-marching objective

$2.5\text{M}$7

The diffusion-forcing filter updates the latent history according to

$2.5\text{M}$8

and the conditional objective becomes

$2.5\text{M}$9

A recurrent misconception would be to view $x_s \in \mathbb{R}^{H \times W \times C}$^^^^1(Chen et al., 23 Sep 2025) OR \1^^^^ as an explicit conditioning variable required by the backbone. In the formulation used for FMT, the model is trained with noisy states generated from jointly sampled $x_s \in \mathbb{R}^{H \times W \times C}$1, but the prediction target is chosen so that the network need not receive $x_s \in \mathbb{R}^{H \times W \times C}$2 directly (&&&^^^^1(Chen et al., 23 Sep 2025) OR \1^^^^&&&).

3. Transformer architecture and conditioning pathway

FMT is described as a SiT-style transformer with AdaLN-Zero conditioning. Its inputs are the flattened noisy latents drawn from four pyramid scales corresponding to four consecutive time points. Positional and time embeddings for each $x_s \in \mathbb{R}^{H \times W \times C}$3 are added, and at each transformer layer the history code is injected through AdaLN-Zero. The implementation uses multi-head self-attention with FlashAttention v2, SwiGLU feed-forwards, and RMSNorm throughout.

The high-level execution sequence is fixed. First, the P2VAE encoder $x_s \in \mathbb{R}^{H \times W \times C}$4 maps physical states to latent grids. Second, noisy latent versions of adjacent states are formed using the sampled noise levels and Gaussian perturbations. Third, a small GRU or RNN consumes each noisy latent together with its sampled time to produce history codes. Fourth, the stacked latent-token sequence is processed by the transformer. Finally, the output token sequence is reassembled into a grid of the same shape as the most recent latent and linearly projected to the velocity field $x_s \in \mathbb{R}^{H \times W \times C}$5 (&&&^^^^1(Chen et al., 23 Sep 2025) OR \1^^^^&&&).

The model therefore combines two distinct forms of temporal conditioning: explicit tokenized latent context from recent frames and an implicit recurrent state summarizing the past. The paper states that all architectural modules follow standard implementations, and on that basis FMT is said to scale from 6 M to 138 M parameters with linear compute scaling (&&&^^^^1(Chen et al., 23 Sep 2025) OR \1^^^^&&&).

4. Latent temporal pyramids and the P2VAE interface

The latent temporal pyramid is central to FMT’s efficiency. The rationale given is that physics is often near-Markovian: recent frames matter more, while older frames can be processed more coarsely. For four consecutive latent frames $x_s \in \mathbb{R}^{H \times W \times C}$6, the construction is:

Frame level	Spatial treatment	Role
Level ^^^^1(Chen et al., 23 Sep 2025) OR \1^^^^	Downsample $x_s \in \mathbb{R}^{H \times W \times C}$7 by factor 8	Oldest coarse context
Level 1	Downsample $x_s \in \mathbb{R}^{H \times W \times C}$8 by factor 4	Intermediate context
Level 2	Downsample $x_s \in \mathbb{R}^{H \times W \times C}$9 by factor 2	Recent context
Level 3	Use $y_s \in \mathbb{R}^{H' \times W' \times C'}$^^^^1(Chen et al., 23 Sep 2025) OR \1^^^^ at full latent resolution	Current detailed state

These token sets are stacked along the sequence dimension, augmented with time embeddings, and passed into the transformer. Self-attention then fuses information across all pyramid scales, allowing the model to route fine-scale information from the most recent state while retaining coarse summaries of earlier states. A plausible implication is that FMT reallocates attention budget toward the temporally local information most relevant for near-Markovian dynamics.

The P2VAE provides the latent space in which FMT operates. The encoder-decoder pair is

$y_s \in \mathbb{R}^{H' \times W' \times C'}$1

with a latent downscaling of $y_s \in \mathbb{R}^{H' \times W' \times C'}$2 and channel compression $y_s \in \mathbb{R}^{H' \times W' \times C'}$3. The VAE is pretrained using the ELBO-style loss

$y_s \in \mathbb{R}^{H' \times W' \times C'}$4

with $y_s \in \mathbb{R}^{H' \times W' \times C'}$5 during P2VAE pretraining. Once these weights are frozen, every physical state $y_s \in \mathbb{R}^{H' \times W' \times C'}$6 is replaced by its latent $y_s \in \mathbb{R}^{H' \times W' \times C'}$7, and the paper states that FMT then sees only sequences of latent states, reducing memory and compute by approximately $y_s \in \mathbb{R}^{H' \times W' \times C'}$8 (&&&^^^^1(Chen et al., 23 Sep 2025) OR \1^^^^&&&).

5. Training regime and sampling procedures

Training is organized in two stages. First, the P2VAE is pretrained for 1^^^^1(Chen et al., 23 Sep 2025) OR \1(Chen et al., 23 Sep 2025) OR \1^^^^k steps to minimize $y_s \in \mathbb{R}^{H' \times W' \times C'}$9. Second, the P2VAE is frozen and FMT together with the diffusion-forcing RNN is trained for 1^^^^1(Chen et al., 23 Sep 2025) OR \1(Chen et al., 23 Sep 2025) OR \1^^^^k steps to minimize $\{s,s+1,s+2,s+3\}$^^^^1(Chen et al., 23 Sep 2025) OR \1^^^^. For joint fine-tuning on a new system, the total loss is

$\{s,s+1,s+2,s+3\}$1

with a stop-gradient placed on $\{s,s+1,s+2,s+3\}$2 to preserve the VAE (&&&^^^^1(Chen et al., 23 Sep 2025) OR \1^^^^&&&).

For deterministic long-term rollouts, sampling follows a probability-flow ODE. Given a clean history $\{s,s+1,s+2,s+3\}$3 from $\{s,s+1,s+2,s+3\}$4, the current step sets $\{s,s+1,s+2,s+3\}$5 so that no initial-condition noise is injected. The procedure initializes $\{s,s+1,s+2,s+3\}$6 and $\{s,s+1,s+2,s+3\}$7, then iterates for $\{s,s+1,s+2,s+3\}$8 with $\{s,s+1,s+2,s+3\}$9:

$\phi$^^^^1(Chen et al., 23 Sep 2025) OR \1^^^^

The resulting terminal state approximates $\phi$1. The paper states that this procedure exactly integrates the continuity equation and is numerically stable.

For aleatoric uncertainty, the model uses the reverse-time SDE adapted from PF-ODE:

$\phi$2

discretized by Euler–Maruyama. Here $\phi$3 controls the noise amplitude. A separate initial-condition uncertainty control is provided by setting $\phi$4 only at the final step while keeping the history clean. The paper explicitly states that ensembles generated by varying $\phi$5 isolate initial-condition uncertainty from aleatoric noise (&&&^^^^1(Chen et al., 23 Sep 2025) OR \1^^^^&&&).

6. Computational profile, benchmarks, and interpretation

The efficiency claim for FMT is derived from its pyramid attention structure. A naive 4-frame video model with 256 tokens per frame incurs attention cost of order $\phi$6 per layer. FMT instead uses token counts corresponding to the four pyramid resolutions, with

$\phi$7

in place of

$\phi$8

The stated efficiency gain is

$\phi$9

Empirically, the paper reports that FMT attains up to $h_i$^^^^1(Chen et al., 23 Sep 2025) OR \1^^^^ throughput increase compared to a bi-directional self-attention video-diffusion baseline (&&&^^^^1(Chen et al., 23 Sep 2025) OR \1^^^^&&&).

The reported downstream results include few-shot adaptation on unseen Kolmogorov turbulence at $h_i$1. With only 5k finetune steps, the model yields $h_i$2 relative error $h_i$3 and $h_i$4, whereas training from scratch under the same budget gives $h_i$5. For long-term rollout on PDEArena–NS and PDEBench–CNS–Low/High, FMT-B with 42 M parameters is reported to outperform VICON-88 M uniformly; on PA-NS at step 1^^^^1(Chen et al., 23 Sep 2025) OR \1^^^^, the average L2RE is ^^^^1(Chen et al., 23 Sep 2025) OR \1^^^^.2234 versus ^^^^1(Chen et al., 23 Sep 2025) OR \1^^^^.3618. The uncertainty-stratified ensembles are described as showing coherent spreads as $h_i$6 or $h_i$7 vary, without spurious artifacts (&&&^^^^1(Chen et al., 23 Sep 2025) OR \1^^^^&&&).

These results frame FMT as a generative PDE foundation-model backbone rather than a conventional deterministic sequence predictor. A common misunderstanding would be to classify it as a full-length video diffusion model. The paper instead characterizes it as a flow-matching model in latent space, equipped with diffusion forcing and temporal pyramids, and designed specifically to support stable rollouts and uncertainty-aware ensemble generation in PDE settings.