Adaptive PFGNet for Spatiotemporal Forecasting

• PFGNet is a fully convolutional model for spatiotemporal predictive learning that dynamically modulates receptive fields using frequency-guided gating.
• It combines large-kernel peripheral convolutions, learnable center suppression, and local spectral descriptors in a recurrence-free encoder–translator–decoder framework.
• The architecture achieves state-of-the-art results on benchmarks like Moving MNIST and TaxiBJ while reducing parameters and FLOPs compared to recurrent and transformer models.

Searching arXiv for the exact PFGNet paper and a small set of closely related STPL baselines to ground the article. PFGNet is a fully convolutional architecture for spatiotemporal predictive learning (STPL) that forecasts future frames from past observations by dynamically modulating receptive fields through pixel-wise frequency-guided gating. It was introduced to address a specific limitation of efficient pure-convolutional video prediction systems: although such models offer full parallelism and favorable efficiency, their spatially uniform fixed receptive fields are poorly matched to scenes in which motion patterns vary across pixels and regions. PFGNet combines large-kernel peripheral convolutions, learnable center suppression, and lightweight local spectral descriptors in a recurrence-free and attention-free encoder–translator–decoder framework, and reports state-of-the-art or near-state-of-the-art results on Moving MNIST, TaxiBJ, KTH, and Human3.6M while using substantially fewer parameters and FLOPs than many recurrent or transformer-like baselines (Cai et al., 24 Feb 2026).

1. Problem formulation and motivation

Spatiotemporal predictive learning is posed as future-frame forecasting from an observed sequence. Given input frames

$\{\mathbf{I}_t\}_{t=1}^{T_{\mathrm{in}}},$

the objective is to predict

$\{\mathbf{O}_t\}_{t=1}^{T_{\mathrm{out}}}.$

The benchmarks emphasized for PFGNet span synthetic motion, articulated human motion, and urban traffic forecasting: Moving MNIST, TaxiBJ, KTH, and Human3.6M (Cai et al., 24 Feb 2026).

The method is positioned against four broad families of STPL models. Recurrent models such as ConvLSTM, PredRNN, PredRNN++, MIM, E3D-LSTM, SwinLSTM, and VMRNN provide explicit temporal state updates but remain sequential. Hybrid recurrent-attention and recurrent-SSM designs attempt to augment recurrent cells with global context. Attention-based recurrent-free approaches improve long-range interaction modeling but often incur higher memory and computational cost. Standard pure-convolutional models such as SimVP, TAU, and STLight are efficient and parallelizable, but apply the same receptive field everywhere. PFGNet targets this last limitation directly: its premise is that STPL requires spatially adaptive context because some regions need broad support for motion estimation while others are largely static or low-information (Cai et al., 24 Feb 2026).

The conceptual motivation is drawn from biological center–surround organization and its signal-processing analogue. A large peripheral response and a smaller central response are contrasted through

$(H_L - \beta H_S) \ast x,$

which behaves as peripheral response minus a scaled central response. In the paper’s interpretation, this forms a spatially adaptive band-pass operator that suppresses redundant low-frequency content while preserving motion-relevant mid-frequency structure such as edges, boundaries, and textured transitions (Cai et al., 24 Feb 2026).

2. Architectural organization

PFGNet adopts a SimVP-style encoder–translator–decoder topology. A shared spatial encoder maps each observed frame to a latent feature map,

$$\mathbf{F}_t = \mathrm{Enc}(\mathbf{I}_t) \in \mathbb{R}^{C \times H' \times W'}.$$

Temporal information is then packed into channels,

$$\mathbf{Z} \in \mathbb{R}^{C' \times H' \times W'}, \qquad C' = T_{\mathrm{in}} \cdot C,$$

so that the middle stage operates on the full observed sequence without recurrence (Cai et al., 24 Feb 2026).

The middle translator contains two components. The first is MSInit, a lightweight multi-scale initialization module designed to create low-, mid-, and high-frequency multi-scale features before the main adaptive blocks. For each scale $m \in \{1,\dots,M\}$,

$$T_m(\mathbf{Z}) = v_m * (h_m * \mathbf{Z}) + d_m(\mathbf{Z}) + \mathbf{Z},$$

where $h_m \in \mathbb{R}^{1 \times k_m}$, $v_m \in \mathbb{R}^{k_m \times 1}$, $d_m(\cdot)$ is a $\{\mathbf{O}_t\}_{t=1}^{T_{\mathrm{out}}}.$0 depthwise convolution, and the branch outputs are projected and concatenated: $\{\mathbf{O}_t\}_{t=1}^{T_{\mathrm{out}}}.$1 In the reported configuration, $\{\mathbf{O}_t\}_{t=1}^{T_{\mathrm{out}}}.$2 (Cai et al., 24 Feb 2026).

The second component is a stack of $\{\mathbf{O}_t\}_{t=1}^{T_{\mathrm{out}}}.$3 Peripheral Frequency Gating blocks, which perform the main spatiotemporal modeling on the packed latent tensor. After the PFG stack produces

$\{\mathbf{O}_t\}_{t=1}^{T_{\mathrm{out}}}.$4

the representation is unpacked back into temporal slices and decoded framewise: $\{\mathbf{O}_t\}_{t=1}^{T_{\mathrm{out}}}.$5 The default setting is $\{\mathbf{O}_t\}_{t=1}^{T_{\mathrm{out}}}.$6. If $\{\mathbf{O}_t\}_{t=1}^{T_{\mathrm{out}}}.$7, the first $\{\mathbf{O}_t\}_{t=1}^{T_{\mathrm{out}}}.$8 frames are sliced. If $\{\mathbf{O}_t\}_{t=1}^{T_{\mathrm{out}}}.$9, prediction proceeds autoregressively by feeding predictions back as inputs (Cai et al., 24 Feb 2026).

Dataset-specific depth settings are part of the design. The reported values are $(H_L - \beta H_S) \ast x,$0 for Moving MNIST, $(H_L - \beta H_S) \ast x,$1 for TaxiBJ, $(H_L - \beta H_S) \ast x,$2 for KTH, and $(H_L - \beta H_S) \ast x,$3 for Human3.6M (Cai et al., 24 Feb 2026).

3. Peripheral Frequency Gating block

The PFG block is the defining mechanism of the architecture. It receives

$(H_L - \beta H_S) \ast x,$4

and computes a pixel-wise adaptive fusion of center-suppressed large-kernel peripheral responses (Cai et al., 24 Feb 2026).

The first step is extraction of localized spectral cues through fixed spatial filters: $(H_L - \beta H_S) \ast x,$5 Here $(H_L - \beta H_S) \ast x,$6 and $(H_L - \beta H_S) \ast x,$7 are Sobel filters, $(H_L - \beta H_S) \ast x,$8 is a Laplacian filter, and $(H_L - \beta H_S) \ast x,$9 is local variance via $$\mathbf{F}_t = \mathrm{Enc}(\mathbf{I}_t) \in \mathbb{R}^{C \times H' \times W'}.$$0 average pooling. These depthwise-filter responses are channel-averaged and concatenated into a three-channel descriptor,

$$\mathbf{F}_t = \mathrm{Enc}(\mathbf{I}_t) \in \mathbb{R}^{C \times H' \times W'}.$$1

A $$\mathbf{F}_t = \mathrm{Enc}(\mathbf{I}_t) \in \mathbb{R}^{C \times H' \times W'}.$$2 convolution generates per-scale gate logits,

$$\mathbf{F}_t = \mathrm{Enc}(\mathbf{I}_t) \in \mathbb{R}^{C \times H' \times W'}.$$3

and softmax over the scale dimension yields pixel-wise mixing weights

$$\mathbf{F}_t = \mathrm{Enc}(\mathbf{I}_t) \in \mathbb{R}^{C \times H' \times W'}.$$4

This is the source of the model’s spatially varying receptive-field selection (Cai et al., 24 Feb 2026).

The second step computes multi-scale peripheral responses using large separable depthwise kernels. For

$$\mathbf{F}_t = \mathrm{Enc}(\mathbf{I}_t) \in \mathbb{R}^{C \times H' \times W'}.$$5

the response is

$$\mathbf{F}_t = \mathrm{Enc}(\mathbf{I}_t) \in \mathbb{R}^{C \times H' \times W'}.$$6

A shared $$\mathbf{F}_t = \mathrm{Enc}(\mathbf{I}_t) \in \mathbb{R}^{C \times H' \times W'}.$$7 center branch then provides local context $$\mathbf{F}_t = \mathrm{Enc}(\mathbf{I}_t) \in \mathbb{R}^{C \times H' \times W'}.$$8, and each scale-specific response is center-suppressed via

$$\mathbf{F}_t = \mathrm{Enc}(\mathbf{I}_t) \in \mathbb{R}^{C \times H' \times W'}.$$9

where

$$\mathbf{Z} \in \mathbb{R}^{C' \times H' \times W'}, \qquad C' = T_{\mathrm{in}} \cdot C,$$0

is a channel-wise learnable parameter broadcast spatially. The use of $$\mathbf{Z} \in \mathbb{R}^{C' \times H' \times W'}, \qquad C' = T_{\mathrm{in}} \cdot C,$$1 constrains the center term to $$\mathbf{Z} \in \mathbb{R}^{C' \times H' \times W'}, \qquad C' = T_{\mathrm{in}} \cdot C,$$2, allowing either positive suppression or negative compensation depending on channel and scale (Cai et al., 24 Feb 2026).

The final spatial output of the block is

$$\mathbf{Z} \in \mathbb{R}^{C' \times H' \times W'}, \qquad C' = T_{\mathrm{in}} \cdot C,$$3

This is followed by channel mixing with a GLU-style path: $$\mathbf{Z} \in \mathbb{R}^{C' \times H' \times W'}, \qquad C' = T_{\mathrm{in}} \cdot C,$$4 with expansion ratio $$\mathbf{Z} \in \mathbb{R}^{C' \times H' \times W'}, \qquad C' = T_{\mathrm{in}} \cdot C,$$5, $$\mathbf{Z} \in \mathbb{R}^{C' \times H' \times W'}, \qquad C' = T_{\mathrm{in}} \cdot C,$$6, plus GRN normalization and LayerScale for training stability (Cai et al., 24 Feb 2026).

The paper gives a frequency-domain interpretation of the center–surround mechanism. Defining

$$\mathbf{Z} \in \mathbb{R}^{C' \times H' \times W'}, \qquad C' = T_{\mathrm{in}} \cdot C,$$7

the learned operator is treated as a ring-shaped band-pass response over a region

$$\mathbf{Z} \in \mathbb{R}^{C' \times H' \times W'}, \qquad C' = T_{\mathrm{in}} \cdot C,$$8

The supplementary analysis states that if

$$\mathbf{Z} \in \mathbb{R}^{C' \times H' \times W'}, \qquad C' = T_{\mathrm{in}} \cdot C,$$9

and there exist

$m \in \{1,\dots,M\}$0

such that

$m \in \{1,\dots,M\}$1

then there exist

$m \in \{1,\dots,M\}$2

with

$m \in \{1,\dots,M\}$3

The same supplement analyzes

$m \in \{1,\dots,M\}$4

and states that there exists at least one finite stationary point $m \in \{1,\dots,M\}$5, together with a lemma asserting the existence of $m \in \{1,\dots,M\}$6 such that

$m \in \{1,\dots,M\}$7

supporting the use of learned center suppression rather than large-kernel responses alone (Cai et al., 24 Feb 2026).

4. Efficiency and optimization

A central design constraint is computational efficiency. Each large $m \in \{1,\dots,M\}$8 depthwise convolution is factorized into a $m \in \{1,\dots,M\}$9 horizontal convolution followed by a $$T_m(\mathbf{Z}) = v_m * (h_m * \mathbf{Z}) + d_m(\mathbf{Z}) + \mathbf{Z},$$0 vertical convolution, reducing per-channel computational cost from

$$T_m(\mathbf{Z}) = v_m * (h_m * \mathbf{Z}) + d_m(\mathbf{Z}) + \mathbf{Z},$$1

to

$$T_m(\mathbf{Z}) = v_m * (h_m * \mathbf{Z}) + d_m(\mathbf{Z}) + \mathbf{Z},$$2

For $$T_m(\mathbf{Z}) = v_m * (h_m * \mathbf{Z}) + d_m(\mathbf{Z}) + \mathbf{Z},$$3, the paper states that this gives roughly a $$T_m(\mathbf{Z}) = v_m * (h_m * \mathbf{Z}) + d_m(\mathbf{Z}) + \mathbf{Z},$$4 reduction in kernel parameters and multiply-adds per layer (Cai et al., 24 Feb 2026).

The training and evaluation protocol is reported through OpenSTL. The paper states that spatially normalized MSE is used during training and validation, while non-normalized MSE is reported at test time. The normalized objective is

$$T_m(\mathbf{Z}) = v_m * (h_m * \mathbf{Z}) + d_m(\mathbf{Z}) + \mathbf{Z},$$5

where $$T_m(\mathbf{Z}) = v_m * (h_m * \mathbf{Z}) + d_m(\mathbf{Z}) + \mathbf{Z},$$6. The reported metrics are MSE, MAE, SSIM, and PSNR, together with parameter count and FLOPs. The supplementary definitions include

$$T_m(\mathbf{Z}) = v_m * (h_m * \mathbf{Z}) + d_m(\mathbf{Z}) + \mathbf{Z},$$7

and

$$T_m(\mathbf{Z}) = v_m * (h_m * \mathbf{Z}) + d_m(\mathbf{Z}) + \mathbf{Z},$$8

SSIM is computed per frame and averaged over batch and time (Cai et al., 24 Feb 2026).

The training schedule varies by dataset.

Dataset	Sequence setting	Reported training setup
Moving MNIST	$$T_m(\mathbf{Z}) = v_m * (h_m * \mathbf{Z}) + d_m(\mathbf{Z}) + \mathbf{Z},$$9, $h_m \in \mathbb{R}^{1 \times k_m}$0	2000 epochs, LR $h_m \in \mathbb{R}^{1 \times k_m}$1, batch size 16, 1× RTX 4090
TaxiBJ	$h_m \in \mathbb{R}^{1 \times k_m}$2, $h_m \in \mathbb{R}^{1 \times k_m}$3	50 epochs, LR $h_m \in \mathbb{R}^{1 \times k_m}$4, batch size 16, 1× RTX 3090
KTH	$h_m \in \mathbb{R}^{1 \times k_m}$5, $h_m \in \mathbb{R}^{1 \times k_m}$6	100 epochs, LR $h_m \in \mathbb{R}^{1 \times k_m}$7 / $h_m \in \mathbb{R}^{1 \times k_m}$8, batch size 2 / 1, 4× RTX 4090
Human3.6M	$h_m \in \mathbb{R}^{1 \times k_m}$9, $v_m \in \mathbb{R}^{k_m \times 1}$0	50 epochs, LR $v_m \in \mathbb{R}^{k_m \times 1}$1, batch size 2, 5× RTX 3090

These settings reinforce the paper’s emphasis on a recurrence-free, attention-free model whose main accuracy gains arise from adaptive convolutional design rather than heavier sequence mechanisms (Cai et al., 24 Feb 2026).

5. Empirical results and ablation evidence

The reported benchmark results establish two recurrent themes: PFGNet is usually strongest on the accuracy–efficiency tradeoff, and its advantages are especially visible on datasets where spatially heterogeneous motion is important (Cai et al., 24 Feb 2026).

Dataset	Representative PFGNet result	Comparative position
Moving MNIST	MSE 15.2, SSIM 0.967	Best reported in the table
TaxiBJ	1.9M params, 0.6G FLOPs, MSE 0.2881, MAE 14.75, SSIM 0.9857	Best MSE with smallest compute footprint in the table
KTH $v_m \in \mathbb{R}^{k_m \times 1}$2	SSIM 0.911, PSNR 34.10	Highest SSIM; PSNR top or tied
KTH $v_m \in \mathbb{R}^{k_m \times 1}$3	SSIM 0.891, PSNR 32.64	Highest SSIM; PSNR competitive
Human3.6M	7.3M params, 58.3G FLOPs, MSE 111.3, MAE 1392.4, SSIM 0.9838	Near-best, but not absolute best on every metric

On Moving MNIST, PFGNet improves upon recurrent and recurrent-free baselines listed in the paper, including ConvLSTM, PredRNN, MAU, SwinLSTM, VMRNN, SimVP, and TAU, reaching MSE $v_m \in \mathbb{R}^{k_m \times 1}$4 and SSIM $v_m \in \mathbb{R}^{k_m \times 1}$5. On TaxiBJ, it reports 1.9M parameters and 0.6G FLOPs with MSE $v_m \in \mathbb{R}^{k_m \times 1}$6, outperforming VMRNN’s MSE $v_m \in \mathbb{R}^{k_m \times 1}$7 while using fewer parameters and FLOPs, and substantially improving over SimVP and TAU. On KTH, it attains the highest SSIM for both $v_m \in \mathbb{R}^{k_m \times 1}$8 and $v_m \in \mathbb{R}^{k_m \times 1}$9 forecasting. On Human3.6M, it remains close to the strongest models while being markedly cheaper than TAU, which the paper notes uses $d_m(\cdot)$0 more parameters and $d_m(\cdot)$1 more FLOPs (Cai et al., 24 Feb 2026).

The ablation studies isolate the function of each major design choice. Removing MSInit raises TaxiBJ MSE from $d_m(\cdot)$2 to $d_m(\cdot)$3. Replacing softmax fusion with uniform mean fusion degrades performance, indicating that pixel-wise scale selection rather than fixed averaging is important. A $d_m(\cdot)$4 center kernel performs better than $d_m(\cdot)$5 in the presented configuration. Multi-scale kernels

$d_m(\cdot)$6

outperform any single kernel, although the single $d_m(\cdot)$7 branch is the strongest single-scale variant on TaxiBJ (Cai et al., 24 Feb 2026).

The center-suppression parameterization is particularly diagnostic. The reported TaxiBJ MSE values are $d_m(\cdot)$8 for fixed $d_m(\cdot)$9, $\{\mathbf{O}_t\}_{t=1}^{T_{\mathrm{out}}}.$00 for fixed $\{\mathbf{O}_t\}_{t=1}^{T_{\mathrm{out}}}.$01, $\{\mathbf{O}_t\}_{t=1}^{T_{\mathrm{out}}}.$02 for fixed $\{\mathbf{O}_t\}_{t=1}^{T_{\mathrm{out}}}.$03, $\{\mathbf{O}_t\}_{t=1}^{T_{\mathrm{out}}}.$04 for learnable $\{\mathbf{O}_t\}_{t=1}^{T_{\mathrm{out}}}.$05 with sigmoid, and $\{\mathbf{O}_t\}_{t=1}^{T_{\mathrm{out}}}.$06 for learnable $\{\mathbf{O}_t\}_{t=1}^{T_{\mathrm{out}}}.$07 with tanh. This provides direct evidence that both learnability and sign-symmetric modulation matter. Further ablations show that all three spectral cues—gradient magnitude, Laplacian response, and local variance—are complementary, and that increasing the number of PFG blocks improves performance up to a point: on TaxiBJ, $\{\mathbf{O}_t\}_{t=1}^{T_{\mathrm{out}}}.$08 gives the best MSE $\{\mathbf{O}_t\}_{t=1}^{T_{\mathrm{out}}}.$09, while $\{\mathbf{O}_t\}_{t=1}^{T_{\mathrm{out}}}.$10 slightly worsens it to $\{\mathbf{O}_t\}_{t=1}^{T_{\mathrm{out}}}.$11 (Cai et al., 24 Feb 2026).

The qualitative analysis described in the paper is consistent with these quantitative trends. On Moving MNIST, PFGNet is said to reduce blur and ghosting around trajectories and occlusions. On TaxiBJ, it better preserves traffic-flow structure. On KTH and Human3.6M, it is reported to preserve limb contours, joint trajectories, and coherent human motion more effectively than less adaptive convolutional baselines (Cai et al., 24 Feb 2026).

6. Interpretation, scope, and nomenclature

The paper’s mechanistic interpretation is that PFGNet learns a bank of center–surround operators whose effective receptive field varies across space. Gate-scale visualizations reportedly show larger kernels selected around dynamic regions and motion boundaries, with smaller kernels favored in smoother or more static areas. The learned $\{\mathbf{O}_t\}_{t=1}^{T_{\mathrm{out}}}.$12 distributions differ by dataset: on Moving MNIST they are roughly symmetric around zero with substantial mass near $\{\mathbf{O}_t\}_{t=1}^{T_{\mathrm{out}}}.$13, whereas on TaxiBJ they are centered closer to zero, suggesting milder center modulation. On Human3.6M, the effective kernel in the $\{\mathbf{O}_t\}_{t=1}^{T_{\mathrm{out}}}.$14 branch is described as showing center–surround sign contrast and approximate ring-shaped structure, together with spectral energy reallocation from low toward mid-to-high frequencies (Cai et al., 24 Feb 2026).

A plausible implication is that PFGNet’s improvements are concentrated in regimes where prediction depends strongly on mid-frequency spatial structure rather than solely on global context or purely local texture. The paper itself frames these regimes in terms of motion boundaries, textured dynamic regions, articulated limbs, and traffic-flow transitions (Cai et al., 24 Feb 2026).

Its limitations are mostly implicit rather than formalized as a dedicated limitations section. The model uses no explicit temporal recurrence or attention, which is central to its efficiency but may constrain very long-range temporal reasoning. Temporal modeling is performed by channel packing rather than by an explicitly sequence-aware state mechanism. Performance is not uniformly best on every dataset and metric; Human3.6M is the clearest example, where PFGNet is near-best rather than dominant. The band-pass interpretation is theoretically supported in simplified form, but real learned convolutional responses remain more complex than ideal radial filters. High-resolution, long-horizon scaling beyond the reported settings remains open (Cai et al., 24 Feb 2026).

The name “PFGNet” also requires disambiguation. It refers specifically to the “Fully Convolutional Frequency-Guided Peripheral Gating Network” for STPL (Cai et al., 24 Feb 2026). It is distinct from “Global Feature Pyramid Network” (Xiao et al., 2023), “Poisson Flow Generative Models” (Xu et al., 2022), “General Proximal Flow Networks” (Strunk et al., 28 Feb 2026), and “FGNet” for 3D EM neuron segmentation (Li et al., 17 Nov 2025), all of which belong to different methodological lineages and application domains.