Spatiotemporal In-Context Learning

• Spatiotemporal in-context learning is a paradigm that adapts models using both spatial configurations and temporal dynamics without changing parameters.
• Architectures like PredRNN and transformer-based models integrate local and global cues, using memory decoupling and fusion strategies to enhance prediction accuracy.
• Empirical studies demonstrate improved performance in video forecasting, anomaly detection, and few-shot classification through context-aware, rapid adaptation techniques.

Spatiotemporal in-context learning (ST-ICL) encompasses a class of learning algorithms and architectures that adapt instantaneously to new task instances or data streams by conditioning on the recent spatiotemporal context, without explicit parameter updates. The paradigm extends in-context learning—widely studied in language and vision—by leveraging both spatial (e.g., co-occurring cues, pixel regions, visual environments) and temporal (e.g., sequence history, dynamic latent environments) dependencies to perform rapid, context-adaptive prediction or control. Approaches span recurrent neural dynamics with explicit spatiotemporal memory (Wang et al., 2021), contextual prototypical memory models for few-shot classification in changing environments (Ren et al., 2020), and transformer-based diffusion models for video generation via latent fusion and prompt-based adaptation (Luo et al., 29 Jan 2026).

1. Key Principles and Formal Definitions

Spatiotemporal in-context learning generalizes classic in-context learning by considering both the spatial configuration and temporal evolution of the data stream or environment:

• Context is defined as the set of recent observations—in both time and space—used by the model to adapt its output distribution for the current prediction, without weight updates.
• Spatiotemporal adaptation is achieved by explicitly modeling temporal dependencies (e.g., via RNNs, LSTMs, sequence processing in transformers) and cross-spatial correlations (e.g., pixel neighborhoods, object co-occurrence, spatial attention).
• No parameter updates: The model leverages internal memories (hidden states, context tokens, prototypical memories) so that all adaptation to novel situations occurs via stateful computation and context aggregation.

This paradigm is instantiated in diverse settings, including video prediction, online few-shot classification, and cross-domain image animation.

2. Architectures for Spatiotemporal In-Context Learning

2.1 Spatiotemporal LSTM and Memory Flows

PredRNN (Wang et al., 2021) introduces a Spatiotemporal LSTM (ST-LSTM) cell at the core of its architecture. Each cell contains two distinct memory pathways:

• Horizontal (Temporal) Memory $\mathcal{C}_t^l$: Standard LSTM memory progressing across time for each layer $l$.
• Vertical (Spatiotemporal) Memory $\mathcal{M}_t^l$: A novel memory flow that propagates across both layers and time, enabling information at all abstraction levels to interact via a "zigzag" mechanism.

The update rules are as follows: $$\begin{aligned} \mathcal{C}_t^l &= f_t \odot \mathcal{C}_{t-1}^l + i_t \odot g_t \ \mathcal{M}_t^l &= f'_t \odot \mathcal{M}_t^{l-1} + i'_t \odot g'_t \ \mathcal{H}_t^l &= o_t \odot \tanh \bigl( W_{1 \times 1} * [\mathcal{C}_t^l, \mathcal{M}_t^l] \bigr) \end{aligned}$$ where "*" denotes convolution, and $[\cdot, \cdot]$ is channel concatenation.

The zigzag memory flow ensures top-down and bottom-up information routing, critical for retaining and integrating multi-scale spatiotemporal context across long prediction horizons (Wang et al., 2021).

2.2 Contextual Prototypical Memory

In online contextualized few-shot learning (OC-FSL), a Contextual Prototypical Memory (CPM) couples a fast-adapting task representation with an RNN-based context tracker (Ren et al., 2020). The key elements are:

• An RNN tracks latent environments via a context embedding $h_t^{\mathrm{RNN}}$.
• Per-class prototypes $p_t[k]$ are updated through online moving averages or gated averaging units, explicitly conditioned on the current context.
• Adaptive metric scaling and context-dependent novelty detection decision thresholds are computed by the RNN, allowing the system to reason about non-stationary distributions and rapidly incorporate new knowledge.

2.3 Spatiotemporal Prompt Fusion for Video Generation

DreamActor-M2 reframes motion transfer in character image animation as a spatiotemporal in-context prompting problem for a transformer-based video diffusion model (Luo et al., 29 Jan 2026):

• A reference image and a driving video are encoded into spatial tokens through a shared 3D-VAE.
• Tokens are fused via transformer blocks with interleaved spatial self-attention and temporal cross-attention, forming a unified context latent $z_{\mathrm{ctx}}$.
• During diffusion decoding, $z_{\mathrm{ctx}}$ acts as a learned prompt, conditioning each denoising block via attention on both identity and motion, with no explicit use of pose priors in the bootstrapped, end-to-end model.

3. Training Protocols and In-Context Adaptation Mechanisms

3.1 Memory Decoupling and Specialized State Learning

PredRNN employs a memory-decoupling loss to enforce orthogonality of updates to $\mathcal{C}$ and $\mathcal{M}$, ensuring that one pathway captures long-term slowly varying context and the other responds to fast, local temporal deformations: $$\mathcal{L}_\mathrm{decouple} = \sum_{t,l}\sum_c \frac{|\langle \Delta \mathcal{C}_t^l, \Delta \mathcal{M}_t^l \rangle_c|}{\|\Delta \mathcal{C}_t^l\|_c \|\Delta \mathcal{M}_t^l\|_c}$$ This encourages the network to represent context at multiple timescales, a hallmark of in-context adaptation (Wang et al., 2021).

3.2 Curriculum Learning and Reverse Scheduled Sampling

In spatiotemporal sequence prediction, scheduled sampling mitigates the train/test input discrepancy. PredRNN further introduces Reverse Scheduled Sampling (RSS) during the encoding phase: early in training, the model is fed its own predictions rather than ground truth, compelling it to learn long-term dependencies in the context frames (Wang et al., 2021).

3.3 Online Contextual Update Protocols

For CPM (Ren et al., 2020), adaptation proceeds online: as new labeled or unlabeled instances arrive, the contextualized prototypes are updated via gated averaging, and read/write gates mediate the risk of prototype corruption in the semi-supervised regime. The RNN's hidden state modulates distances and thresholds for both retrieval and novelty detection at each time $t$.

3.4 Self-Bootstrapped Curriculum in Video Generation

DreamActor-M2's two-stage paradigm uses a pose-based model to synthesize pseudo cross-identity video/image pairs. These are then used to train a pose-free, RGB-driven model using triplets (generated video, reference image, source driving video), enabling end-to-end spatiotemporal in-context conditioning without human annotation or explicit skeletons (Luo et al., 29 Jan 2026).

4. Empirical Evaluations and Benchmarks

Spatiotemporal in-context learners are evaluated across diverse domains:

Model / Domain	Key Benchmark(s)	Notable Metrics	Results
PredRNN (Wang et al., 2021)	Moving MNIST, KTH, Radar	MSE, SSIM, PSNR, LPIPS, CSI	Up to 53% lower MSE vs. ConvLSTM; robust long-term forecasts
CPM (Ren et al., 2020)	RoamingRooms, Omniglot	AP, 1-shot accuracy, forgetting curves	AP=89.1% (Rooms), 94.2% (Omniglot); superior recall dynamics
DreamActor-M2 (Luo et al., 29 Jan 2026)	AWBench, human eval	Imaging Q, Motion S, Temporal C, App. C	Top scores: Imaging Q = 4.72, Appearance C = 4.35

Ablation studies confirm the significance of both spatiotemporal context (e.g., RNN context, transformer fusion) and memory specialization (decoupling, gating) for robust, generalizable in-context adaptation.

5. Comparative Significance and Advantages

Spatiotemporal in-context learning confers several advantages over conventional episodic or frame-sequential paradigms:

• Integrated multi-scale context: Mechanisms like zigzag memory or fusion transformers enable per-step adaptation by propagating global and local information.
• Robustness to distributional drift: Context-adaptive gating and metric scaling manage non-stationarity, e.g., when environments or object classes change unpredictably.
• Rapid assimilation of new tasks: In OC-FSL, models detect and encode new classes online, yielding improved short-term recall and less forgetting even under sparse labeling (Ren et al., 2020).
• Generalization across domains and modalities: Spatiotemporal prompts in DreamActor-M2 allow seamless transfer of motion and appearance cues across arbitrary visual domains, eliminating dependency on object- or pose-specific priors (Luo et al., 29 Jan 2026).

A plausible implication is that the unification of spatial and temporal adaptation within the in-context learning framework considerably enhances both data efficiency and robustness in non-stationary, real-world scenarios.

6. Limitations and Prospects for Enhancement

Noted constraints and directions from empirical work include:

• Memory and computation: Transformer-based fusion and deep stacking of memory cells can be resource-intensive for long sequences or large spatial grids.
• Complex multi-agent dynamics: Current approaches (e.g., DreamActor-M2) demonstrate decreased fidelity and coherence for intricate multi-character interactions or extremely long output sequences (Luo et al., 29 Jan 2026).
• Prototype corruption: In semi-supervised or highly non-stationary settings, inaccurate read/write thresholding can cause catastrophic forgetting or error propagation. Tuned gating strategies (e.g., separate $\beta^r$, $\beta^w$) alleviate but do not eliminate this risk (Ren et al., 2020).
• Data diversity: Sparse or biased training data limits generalization to novel spatial layouts or motion regimes. Synthetic augmentation and trajectory forecasting heads are suggested as mitigations (Luo et al., 29 Jan 2026).

Potential improvements include hierarchical or continuous latent context models, explicit 3D/semantic spatial reasoning, and prompt-based specialization via lightweight LoRA-like adapters.

7. Broader Relevance and Applicability

Spatiotemporal in-context learning, as formalized in the referenced works, is impacting multiple areas:

• Video forecasting and anomaly detection: Adaptive sequence models with zigzag or spatiotemporal memory maintain high-fidelity predictions over long horizons (Wang et al., 2021).
• Embodied and continual learning: Online contextual memory models enable few-shot learning, agent localization, and rapid adaptation in non-i.i.d. environments (Ren et al., 2020).
• Cross-domain content generation: Prompt-based spatiotemporal conditioning extends animation, simulation, and avatar generation beyond explicit pose-driven or skeletal representations (Luo et al., 29 Jan 2026).

The paradigm supports a unified theoretical and empirical foundation for context-driven learning systems, with ongoing research into applicability for active perception, reinforcement learning, and embodied intelligence in continuously evolving environments.