StreamPhy: Streaming Inference of High-Dimensional Physical Dynamics via State Space Models

Abstract: Inferring the evolution of high-dimensional and multi-modal (e.g., spatio-temporal) physical fields from irregular sparse measurements in real time is a fundamental challenge in science and engineering. Existing approaches, including diffusion-based generative models and functional tensor methods, typically operate in offline settings, depend on full temporal observations, or incur substantial inference cost. We propose StreamPhy, an end-to-end framework that enables efficient and accurate streaming inference of full-field physical dynamics from incoming irregular sparse measurements. The framework integrates a data-adaptive observation encoder that is robust to arbitrary observation patterns, a structured state-space model that supports memory-efficient online updates across irregular time intervals, and an expressive Functional Tensor Feature-wise Linear Modulation (FT-FiLM) decoder for continuous-field generation. We prove that FT-FiLM is more expressive than the functional Tucker model, admitting a richer function class for handling complex dynamics. Experiments on three representative physical systems under challenging sampling patterns show that StreamPhy consistently outperforms state-of-the-art baselines, with at least 48\% improvement in accuracy and up to 20--100X faster inference than diffusion-based methods.

• The paper introduces a novel framework that leverages a HiPPO-based state space model for efficient sequential memory in streaming inference.
• The paper demonstrates that the FT-FiLM decoder significantly enhances expressivity and accuracy, achieving up to 48% VRMSE improvement and 20–100× faster inference.
• The paper validates its approach across various physical systems, achieving robust reconstruction even with extremely low (as low as 1%) and irregular observation ratios.

Streaming Inference for High-Dimensional Physical Dynamics via State Space Models

Motivation and Problem Formulation

The inference of high-dimensional spatiotemporal physical fields from sparse, irregular, and streaming observations is central to many scientific and engineering disciplines, ranging from environmental sensing to turbulent flow modeling. Traditional paradigms—including diffusion-based generative frameworks and functional tensor decomposition methods—often operate in batch or offline regimes, require complete temporal sequences, or incur considerable computational overhead. Their ability to assimilate irregular observation streams in real time, while providing high-fidelity reconstructions across continuous domains, remains restricted. These constraints motivate the need for principled, efficient, and memory-adaptive architectures capable of leveraging the intrinsic structure in spatiotemporal data while coping with practical observation deficiencies.

StreamPhy Framework Overview

StreamPhy addresses the streaming inference challenge by integrating a HiPPO-based structured State Space Model (SSM), a robust attention-driven observation encoder, and an expressive Functional Tensor Feature-wise Linear Modulation (FT-FiLM) decoder. The architecture is designed to ingest incoming batches of irregular observations, efficiently update latent states, and reconstruct the full field at arbitrary locations in the continuous spatial domain. A semantic illustration of StreamPhy's architecture is depicted below.

Figure 1: Semantic illustration of the StreamPhy framework.

The observation encoder maps variable-length, time-varying sets of measurements into structured latent representations, utilizing concatenated functional tensor embeddings and attention-based aggregation. The HiPPO SSM models temporal dependencies via online compression over orthogonal polynomial bases, naturally accommodating irregular temporal intervals through bilinear discretization. Critically, the FT-FiLM readout module enables expressive and flexible full-field generation by modulating functional tensor representations with state-dependent feature-wise affine transformations, significantly enhancing its approximation capacity.

Advances in Representation and Modulation

StreamPhy's decoder, FT-FiLM, extends the functional Tucker model (FTM) by integrating feature-wise linear modulation conditioned jointly on latent states and observational context. This mechanism decouples modulation parameters from latent factor ranks, eliminating rank-induced expressivity barriers inherent in FTM. The theoretical result in the paper proposes that FT-FiLM's closure is strictly larger, encompassing all continuous functions, as opposed to the limited class captured by FTM. The architectural difference is shown below.

Figure 2: FTM versus FT-FiLM for $K=3$ input modes, highlighting the greater expressive power of FT-FiLM through state-dependent modulation.

Uniform approximation capability is formally proven, providing a rigorous foundation for modeling complex and highly nonlinear spatiotemporal dynamics. This enhancement is especially critical for reconstructing physics fields under severe sampling deficiencies or complex multimodal patterns.

Empirical Evaluation and Results

StreamPhy is evaluated across three representative physical settings: Turbulent Flow, Ocean Sound Speed, and Active Matter, under both uniform and slab sampling regimes with observation ratios as low as 1%. Reconstruction accuracy is measured with variance-scaled root mean squared error (VRMSE), and both qualitative and quantitative results demonstrate strong advantages. For example, StreamPhy achieves at least 48% improvement in VRMSE compared to state-of-the-art baselines, and inference speedups of 20–100$\times$ over diffusion-based approaches, due in part to the SSM's sequential linear complexity and FT-FiLM's compact representation. Key qualitative results for turbulent flow and active matter are depicted below.

Figure 3: StreamPhy reconstructs turbulent flow dynamics under uniform sampling ($\rho = 3\%$), achieving high visual and numerical fidelity.

Figure 4: Reconstruction accuracy for active matter dynamics under slab sampling ($\rho = 3\%$), demonstrating StreamPhy’s robustness to challenging observation patterns.

These results validate that the combination of HiPPO SSM, adaptive attention encoder, and FT-FiLM yields robust, accurate, and scalable streaming inference even in the presence of strong spatial or temporal observation sparsity. StreamPhy also consistently outperforms tensor-based and diffusion-based alternatives under more challenging conditions, such as slab sampling or random spatial observation masks.

Component Analysis and Ablation

Ablation studies indicate the essential contribution of each architectural component:

• Removal of the SSM drastically impairs streaming performance, confirming its necessity for long-range sequential memory.
• Disabling masking in the observation encoder reduces accuracy and robustness, particularly under non-uniform sampling.
• Substituting FT-FiLM with FTM leads to marked expressivity and accuracy reduction, confirming FT-FiLM's theoretical and empirical advantage.

These findings highlight the synergistic effect of StreamPhy’s architecture: sequential memory, robust encoding, and expressive modulation are collectively indispensable for real-time, high-fidelity streaming field inference.

Practical and Theoretical Implications

StreamPhy’s construction directly addresses bottlenecks in streaming inference for scientific and engineering applications that require precise modeling of physical systems with limited observation budgets and operational constraints. Its unified, end-to-end design is applicable to real-time monitoring systems, adaptive prediction pipelines, and experimental data assimilation scenarios. The flexibility and theoretical universality of FT-FiLM position it as a general-purpose decoder for continuous field approximation, with implications for operator learning and surrogate modeling of PDEs.

The theoretical foundation provided by the expressivity results for FT-FiLM suggests directions for future representer architectures, decoupling latent state dimension from representational capacity. Additionally, the framework’s adaptability to irregular sampling and nonuniform spatial-temporal observation patterns offers a template for generative modeling in sparse sensing, climate modeling, and large-scale simulations.

Conclusion

StreamPhy presents an efficient and principled solution for streaming inference of high-dimensional physical dynamics. By combining adaptive encoding, HiPPO-based SSM sequence modeling, and the FT-FiLM modulation mechanism, it achieves state-of-the-art accuracy and efficiency across multiple real-world physical systems and sampling regimes. The architecture’s design and theoretical guarantees underscore its robustness and capacity, with broad implications for sequential modeling and operator approximation tasks in AI-driven physical modeling.