ST-VFM: Spatio-Temporal Model Frameworks

• ST-VFM is a collection of spatio-temporal model families with distinct architectures for lossy data compression, raw-input forecasting, and feature-space prediction.
• The framework employs techniques such as variational autoencoders with hyper-priors, super-resolution modules, and prompt-based cross-branch coordination to enhance performance.
• Evaluations show improved compression ratios and forecasting accuracy measured by metrics like NRMSE and RMSE across multiple scientific and real-world benchmarks.

ST-VFM is a name used for distinct spatio-temporal model families in recent literature. In one usage, ST-VFM denotes a “Spatiotemporal Foundation Model” for lossy compression of scientific simulation data, built from a variational autoencoder with a hyper-prior and a super-resolution decoder. In another, ST-VFM denotes a framework for reprogramming frozen Vision Foundation Models for general-purpose spatio-temporal forecasting through temporal token adaptation and prompt-based cross-branch coordination. A related line on world modeling also uses the phrase “ST-VFM or VFMF” for forecasting Vision Foundation Model features with a VAE and autoregressive flow matching (Li et al., 2024, Chen et al., 14 Jul 2025, Boduljak et al., 12 Dec 2025).

1. Nomenclature and scope

The acronym ST-VFM does not identify a single canonical architecture. In the current literature, it appears in at least three closely related but technically distinct settings: lossy scientific data compression, raw-input spatio-temporal forecasting with frozen VFMs, and feature-space forecasting of VFM representations.

Usage	Problem setting	Core components
ST-VFM as “Spatiotemporal Foundation Model”	Lossy compression of scientific data blocks of shape $T\times H\times W$	VAE backbone with hyper-prior, alternating 2D/3D convolutions, SR decoder
ST-VFM for reprogrammed VFMs	General-purpose spatio-temporal forecasting	Dual-branch raw ST and ST-flow inputs, TA-Adapter, BCPC, frozen VFM backbone
“ST-VFM or VFMF”	World modeling by forecasting VFM features	Frozen VFM feature extraction, VAE latent encoding, autoregressive flow matching, downstream decoders

This multiplicity is significant because the shared acronym masks different modeling assumptions. One line operates directly on scientific fields and optimizes rate–distortion; another uses patch tokens and prompt-based interaction for forecasting; the third treats VFM features as the state space for stochastic world modeling (Li et al., 2024, Chen et al., 14 Jul 2025, Boduljak et al., 12 Dec 2025).

2. ST-VFM as a spatiotemporal foundation model for lossy compression

In "Foundation Model for Lossy Compression of Spatiotemporal Scientific Data" (Li et al., 2024), ST-VFM is specifically engineered to compress blocks of scientific simulation data of shape $T\times H\times W$. Its core compression engine is a VAE that maps an input block $x\in\mathbb{R}^{T\times H\times W}$ to a latent tensor $y$ via a probabilistic analysis transform $q(y|x)$, quantizes $y$ to $\hat y$, and reconstructs $\hat x$ via a probabilistic synthesis transform $p(x|\hat y)$. The encoder is implemented as a cascade of 2D then 3D convolutions, while the decoder uses transposed 3D convolutions followed by a super-resolution network. To capture dependencies in $\hat y$, the model embeds a hyper-prior $T\times H\times W$0: a hyper-analysis transform produces $T\times H\times W$1, quantized to $T\times H\times W$2, and a hyper-synthesis transform predicts per-element Gaussian parameters $T\times H\times W$3 for $T\times H\times W$4. Each quantized latent coefficient is modeled as

$T\times H\times W$5

and the total rate is the sum of the expected bit-rates for $T\times H\times W$6 and $T\times H\times W$7:

$T\times H\times W$8

Because there is no parametric prior on $T\times H\times W$9, $x\in\mathbb{R}^{T\times H\times W}$0 is modeled as a nonparametric, fully factorized density, and arithmetic coding is used to losslessly compress both $x\in\mathbb{R}^{T\times H\times W}$1 and $x\in\mathbb{R}^{T\times H\times W}$2.

A central architectural choice is the alternation between 2D and 3D convolutional layers. Rather than applying full-resolution 3D convolutions to a block of size $x\in\mathbb{R}^{T\times H\times W}$3, ST-VFM first applies several 2D convolutions with stride-2 on each of the $x\in\mathbb{R}^{T\times H\times W}$4 time-slices independently, reducing $x\in\mathbb{R}^{T\times H\times W}$5 and increasing feature channels to $x\in\mathbb{R}^{T\times H\times W}$6. It then merges the time dimension into the channel tensor and applies 3D convolutions with stride-2 on all four dimensions, reducing $x\in\mathbb{R}^{T\times H\times W}$7 and doubling channels to $x\in\mathbb{R}^{T\times H\times W}$8. In the decoder, 3D transposed convolutions invert this sequence before the SR module restores full resolution. The paper states that postponing heavy 3D processing until the spatial resolution has already been reduced by a factor of 16 in area cuts FLOPs and memory by nearly an order of magnitude versus full-resolution 3D. The SR decoder then refines each time-slice independently through a shallow 2D feature extractor, a cascade of BCB blocks, multi-block feature concatenation plus skip connections, and a channel shuffle layer that rearranges and upscales features by $x\in\mathbb{R}^{T\times H\times W}$9 in spatial dimensions (Li et al., 2024).

3. Rate–distortion objective, training, and empirical behavior in compression

The final reconstruction $y$0 is trained under a Lagrangian rate–distortion objective. Distortion is the mean-squared error

$y$1

and the overall loss is

$y$2

During foundation training, the schedule starts with $y$3, raises it to $y$4 at iteration $y$5, uses a learning rate of $y$6 halved every $y$7 iterations, and runs for $y$8 iterations. Fine-tuning on a new domain runs $y$9 more iterations at $q(y|x)$0 and learning rate $q(y|x)$1, halved every $q(y|x)$2.

The reported experimental setup uses S3D, Hurricane, JHTDB, ERA5, and Sunquake for foundation training, with E3SM held out for evaluation. Blocks are preprocessed with $q(y|x)$3, $q(y|x)$4, padded by reflection if needed, and normalized per field to $q(y|x)$5; smaller datasets are up-sampled in sample counts so that each domain contributes equally. Evaluation focuses on NRMSE, bits per value $q(y|x)$6, and compression ratio $q(y|x)$7, while noting that PSNR and SSIM may also be reported in imaging settings. On E3SM five variables—PSL, T200, T500, VBOT, and UBOT—the foundation model without any fine-tuning already matches or betters SZ3 and ZFP at the same NRMSE across a wide range of compression ratios up to $q(y|x)$8. After $q(y|x)$9 iterations of domain-specific fine-tuning, ST-VFM achieves up to a $y$0 higher compression ratio on PSL and $y$1 on temperature and wind fields at the identical NRMSE. At $y$2, “FM with SR” compresses by $y$3 versus $y$4 for “FM without SR.” The SR module improves attainable compression ratios by up to 30% at the same reconstruction error, and spatial block sizes of 64, 128, and 256 confirm flexible input support, with larger blocks yielding higher compression ratios by exploiting more spatial context (Li et al., 2024).

4. ST-VFM as a framework for reprogramming vision foundation models

In "Reprogramming Vision Foundation Models for Spatio-Temporal Forecasting" (Chen et al., 14 Jul 2025), ST-VFM is a general-purpose spatio-temporal forecasting framework built around off-the-shelf, frozen Vision Foundation Models such as ViT backbones pretrained by DINO, CLIP, and VideoMAE. The framework is motivated by two stated challenges when applying VFMs to spatio-temporal tasks: the lack of inherent temporal modeling capacity and the modality gap between visual and ST data. To address these, it uses a dual-branch architecture with raw ST inputs and auxiliary ST-flow inputs, together with two lightweight reprogramming stages: a pre-VFM Temporal-Aware Token Adapter and a post-VFM Bilateral Cross-Prompt Coordination block.

The raw ST branch takes $y$5 and, after patch embedding and flattening, produces token sequence $y$6, where $y$7. The auxiliary ST-flow branch computes a pseudo–“optical-flow” representation for consecutive frames, for example via classical Lucas–Kanade or a small FlowNet, and processes $y$8 identically. The TA-Adapter sits immediately after patch embedding and injects temporal context into each per-patch token:

$y$9

where $\hat y$0 is GELU and $\hat y$1 is a fixed or learnable time embedding. Both branches then feed into the same frozen VFM encoder, producing feature maps $\hat y$2.

Post-VFM reprogramming is handled by BCPC. The model appends two small sets of learnable prompt tokens, $\hat y$3 and $\hat y$4, and performs $\hat y$5 rounds of mutual cross-attention:

$\hat y$6

After $\hat y$7 iterations, each branch is decoded with a lightweight MLP head. The paper describes this design as preserving 100% of the original VFM weights frozen while adding only a few million extra parameters (Chen et al., 14 Jul 2025).

5. Forecasting performance, ablations, and efficiency

The forecasting ST-VFM is evaluated on Crowd, Cellular, TDrive, BikeNYC, TaxiNYC(2), BikeNYC2, and TrafficZZ/TJ/SH. The reported metrics are

$\hat y$8

Against the best prior UniST, ST-VFM reports, for example, Crowd RMSE $\hat y$9 versus $\hat x$0 and MAE $\hat x$1 versus $\hat x$2, Cellular RMSE $\hat x$3 versus $\hat x$4 and MAE $\hat x$5 versus $\hat x$6, and BikeNYC RMSE $\hat x$7 versus $\hat x$8 and MAE $\hat x$9 versus $p(x|\hat y)$0. The paper states that similar gains hold on all 10+ benchmarks.

The ablations isolate the two reprogramming stages and the auxiliary flow branch. Removing the ST-flow branch with $p(x|\hat y)$1 degrades RMSE by approximately 5–8% across datasets. Disabling the TA-Adapter yields an approximately 3% drop in RMSE. Replacing BCPC with simple concatenation of VFM features loses another approximately 4% RMSE. The balance weight $p(x|\hat y)$2 between ST and flow MSE losses is reported as robust in $p(x|\hat y)$3. A lightweight variant keeps only one cross-attention round and smaller heads yet still outperforms UniST by 10% with only $p(x|\hat y)$4 min inference overhead. On Crowd, training time is $p(x|\hat y)$5 min versus $p(x|\hat y)$6 min for UniST, and inference is $p(x|\hat y)$7 min versus $p(x|\hat y)$8 min, with substantially higher accuracy. A plausible implication is that the framework trades a modest increase in runtime for stronger spatio-temporal forecasting accuracy while avoiding any fine-tuning of the frozen VFM backbone (Chen et al., 14 Jul 2025).

6. Related VFM-feature forecasting and recurrent confusions

A separate but related usage appears in "VFMF: World Modeling by Forecasting Vision Foundation Model Features" (Boduljak et al., 12 Dec 2025), whose detailed summary explicitly describes “Spatio-temporal forecasting with Vision Foundation Model Features (ST-VFM or VFMF).” Here the pipeline has four stages: frozen VFM feature extraction, compact latent encoding via a VAE, stochastic autoregressive forecasting in latent space by flow matching, and decoding into output modalities. A frozen encoder such as DINOv2 extracts dense feature tensors $p(x|\hat y)$9, a $\hat y$0-VAE compresses $\hat y$1 channels down to $\hat y$2, and a 12-layer space–time transformer forecasts future latents with rectified-flow matching using context length $\hat y$3 and Euler integration with 10 NFEs. Predicted latents are decoded into semantic segmentation, depth, surface normals, and even RGB. On Cityscapes, with a single context frame forecasting 9 future frames, the reported segmentation mIoU (all) changes from $\hat y$4 for deterministic regression to $\hat y$5 under mean-of-$\hat y$6 evaluation, depth $\hat y$7 rises from $\hat y$8 to $\hat y$9, and mean angular error for normals decreases from $T\times H\times W$00 to $T\times H\times W$01 (Boduljak et al., 12 Dec 2025).

A common source of confusion is that this VFM-feature formulation is not the same as the raw-input, dual-branch ST-VFM of (Chen et al., 14 Jul 2025), and neither is the same as the compression-oriented “Spatiotemporal Foundation Model” of (Li et al., 2024). The acronym should also not be conflated with STFM, the “Spatio-Temporal Fusion Model” for echocardiographic video view classification, which is an efficient dual-stream CNN-LSTM with evidential uncertainty estimation (Gou et al., 16 Jun 2026), or with ST-MFNet, the “Spatio-Temporal Multi-Flow Network” for frame interpolation, which combines multi-interflows, bi-directional linear flow, a 3D CNN texture enhancement network, and ST-GAN fine-tuning (Danier et al., 2021). This suggests that task context—lossy scientific compression, spatio-temporal forecasting from raw fields, or forecasting in VFM feature space—is necessary to disambiguate the term ST-VFM in current usage.