StreamPhy: Streaming Inference for Physical Dynamics

• StreamPhy is an end-to-end framework for streaming inference of high-dimensional physical dynamics from irregular sparse measurements, enabling continuous-field reconstruction in real time.
• It integrates a data-adaptive observation encoder, a structured state-space model with memory-efficient updates, and a FT-FiLM decoder to overcome limitations of diffusion-based and fixed-rank functional methods.
• Empirical evaluations on turbulent flow, ocean sound speed, and active matter show at least a 48% improvement in accuracy and 20–100X faster inference compared to state-of-the-art approaches.

Searching arXiv for the primary paper and closely related context papers to ground the article in current arXiv records. {"query":"StreamPhy Streaming Inference of High-Dimensional Physical Dynamics via State Space Models arXiv", "max_results": 5} {"query":"Senseiver arXiv sparse sensor field reconstruction", "max_results": 5} StreamPhy is an end-to-end framework for streaming inference of high-dimensional physical dynamics from incoming irregular sparse measurements. It is designed for settings in which a spatio-temporal physical field is only partially observed at each time step, the observation pattern can vary arbitrarily over time, and full-field reconstruction must be produced online without revisiting the past. The framework combines a data-adaptive observation encoder, a structured state-space model with memory-efficient online updates across irregular time intervals, and a Functional Tensor Feature-wise Linear Modulation decoder for continuous-field generation. In the reported experiments, StreamPhy is evaluated on turbulent flow, ocean sound speed, and active matter, and is described as achieving at least 48% improvement in accuracy and up to 20–100X faster inference than diffusion-based methods (Chen et al., 8 May 2026).

1. Problem formulation and scope

StreamPhy addresses the problem of inferring the evolution of high-dimensional and multi-modal physical fields from irregular sparse measurements in real time (Chen et al., 8 May 2026). The target object is a continuous field

$y_t : \mathbb{R}^K \to \mathbb{R},$

where $t$ denotes time and $\mathbf{i} = (i_1,\dots,i_K)$ denotes a continuous spatial or spatio-temporal coordinate. At each observation time $t_m$, the available data are a sparse set of entries

$$\mathcal{O}_{t_m} = \{(\mathbf{i}_{n_m}, y_{t_m,\mathbf{i}_{n_m}})\}_{n_m=1}^{N_m},$$

with $N_m$ varying over time and typically much smaller than the total number of grid points (Chen et al., 8 May 2026).

The central objective is to process the data stream

$\mathcal{O} = \{\mathcal{O}_{t_m}\}_{m=1}^M$

in temporal order, update a compact latent state online, and reconstruct the full continuous field at arbitrary query coordinates without storing or reprocessing the entire history (Chen et al., 8 May 2026). This formulation targets settings where measurements are sparse, spatially irregular, and temporally non-uniform.

The paper positions StreamPhy against three broad classes of prior methods. Diffusion-based generative models are described as typically offline, dependent on full temporal observations, and expensive at inference because of iterative reverse-time sampling. Functional tensor methods are described as powerful but often tied to single-sequence unsupervised decomposition, high rank, or memory replay. Standard sequence models, including RNNs, Transformers, and structured SSMs, are described as efficient for regular sequences but not natively adapted to variable-size sets of observations per time step and continuous spatial decoding of high-dimensional fields (Chen et al., 8 May 2026).

2. End-to-end architecture

The architecture has three components: an observation encoder, a structured state-space model, and an FT-FiLM decoder (Chen et al., 8 May 2026). The computation proceeds in three stages.

First, the sparse observations at time $t$ are encoded into a fixed-dimensional latent vector

$$\mathbf{z}_t = \mathrm{ObsEncoder}(\mathcal{O}_t) \in \mathbb{R}^P.$$

Second, a structured SSM updates the latent state

$$\mathbf{X}_t = \bar{\mathbf{A}} \mathbf{X}_{t-1} + \bar{\mathbf{b}} \circ \mathbf{z}_t,$$

where $t$0, $t$1 is the discretized transition matrix, $t$2, and $t$3 denotes the outer product (Chen et al., 8 May 2026). This update is the core streaming mechanism: the model receives the current observation summary $t$4, updates $t$5, and does not need to revisit earlier observations.

Third, the decoder reconstructs the field at arbitrary query coordinates. It computes modulation parameters

$t$6

and then predicts the field value at coordinate $t$7 via

$t$8

where $t$9 is the coordinate embedding (Chen et al., 8 May 2026).

This decomposition separates three functions that are often entangled in earlier approaches: encoding arbitrary observation sets, maintaining a compact temporal memory, and generating a continuous field. A plausible implication is that this division of labor is what allows the framework to remain online while still supporting arbitrary spatial queries.

3. Observation encoder for irregular sparse measurements

The observation encoder is designed for arbitrary observation patterns and variable cardinality per time step (Chen et al., 8 May 2026). Each coordinate $\mathbf{i} = (i_1,\dots,i_K)$0 is first embedded by mode-wise latent functions

$\mathbf{i} = (i_1,\dots,i_K)$1

implemented as an INR with Fourier features:

$\mathbf{i} = (i_1,\dots,i_K)$2

The mode-wise embeddings are then concatenated:

$\mathbf{i} = (i_1,\dots,i_K)$3

This produces a continuous coordinate representation rather than a grid-index representation (Chen et al., 8 May 2026).

For each observed point, the model concatenates $\mathbf{i} = (i_1,\dots,i_K)$4 with the scalar measurement $\mathbf{i} = (i_1,\dots,i_K)$5, maps the result through a linear layer, and aggregates the resulting set via single-query cross-attention from a learnable global query $\mathbf{i} = (i_1,\dots,i_K)$6:

$\mathbf{i} = (i_1,\dots,i_K)$7

An MLP then maps the attended representation to $\mathbf{i} = (i_1,\dots,i_K)$8 (Chen et al., 8 May 2026). A multi-head variant uses $\mathbf{i} = (i_1,\dots,i_K)$9 smaller queries and separate projections, producing the StreamPhy (MH) model used in the strongest reported results.

This encoder is permutation-invariant in the input set and naturally accommodates varying $t_m$0 across time steps (Chen et al., 8 May 2026). During training, the paper applies stochastic masking by sampling a masking rate $t_m$1 and randomly dropping a fraction of observations. The stated purpose is robustness to varying sparsity and observation geometry, including both uniform and slab sampling patterns (Chen et al., 8 May 2026).

A common misunderstanding would be to treat the encoder as merely a preprocessing step for irregular inputs. In the reported ablations, removing masking or removing the SSM-conditioned temporal structure substantially degrades performance, indicating that the encoder is part of a coupled temporal reconstruction system rather than an interchangeable front end (Chen et al., 8 May 2026).

4. Structured state-space model and online updates

The temporal core of StreamPhy is a HiPPO-based structured state-space model inspired by S4 and LSSL (Chen et al., 8 May 2026). The starting point is the continuous-time linear SSM

$t_m$2

The transition is not generic: StreamPhy adopts the HiPPO LegS structure

$t_m$3

which is used to compress input history in an orthogonal polynomial basis (Chen et al., 8 May 2026).

Discretization is performed with the bilinear transform. For time step $t_m$4,

$t_m$5

This explicitly permits irregular time intervals, because the discrete operators depend on $t_m$6 (Chen et al., 8 May 2026).

StreamPhy generalizes the scalar-input SSM to vector inputs by maintaining an augmented state matrix $t_m$7 and injecting the encoder output through an outer product, as in the recurrence above (Chen et al., 8 May 2026). The interpretation given in the paper is that this can be viewed as $t_m$8 parallel SSM channels sharing the same transition and input structure.

The model is deterministic rather than Kalman-based or variational (Chen et al., 8 May 2026). Its streaming behavior derives from the fact that the latent state is the only temporal memory required at inference. The paper states that the SSM update is $t_m$9 in sequence length, in contrast to SDIFT, whose MPDPS module is described as having per-frame complexity $$\mathcal{O}_{t_m} = \{(\mathbf{i}_{n_m}, y_{t_m,\mathbf{i}_{n_m}})\}_{n_m=1}^{N_m},$$0 and overall $$\mathcal{O}_{t_m} = \{(\mathbf{i}_{n_m}, y_{t_m,\mathbf{i}_{n_m}})\}_{n_m=1}^{N_m},$$1 in the number of frames $$\mathcal{O}_{t_m} = \{(\mathbf{i}_{n_m}, y_{t_m,\mathbf{i}_{n_m}})\}_{n_m=1}^{N_m},$$2 (Chen et al., 8 May 2026).

5. FT-FiLM decoder and expressivity

The decoder is the principal representational innovation. StreamPhy introduces Functional Tensor Feature-wise Linear Modulation, or FT-FiLM, as a conditional continuous-field generator (Chen et al., 8 May 2026). Functional Tucker models represent a continuous field as

$$\mathcal{O}_{t_m} = \{(\mathbf{i}_{n_m}, y_{t_m,\mathbf{i}_{n_m}})\}_{n_m=1}^{N_m},$$3

with a fixed core tensor and mode-wise latent functions (Chen et al., 8 May 2026). The paper argues that such models face a rank barrier at fixed rank.

FT-FiLM replaces the fixed multilinear core with state-dependent modulation. Given the coordinate embedding $$\mathcal{O}_{t_m} = \{(\mathbf{i}_{n_m}, y_{t_m,\mathbf{i}_{n_m}})\}_{n_m=1}^{N_m},$$4, with $$\mathcal{O}_{t_m} = \{(\mathbf{i}_{n_m}, y_{t_m,\mathbf{i}_{n_m}})\}_{n_m=1}^{N_m},$$5, the decoder forms

$$\mathcal{O}_{t_m} = \{(\mathbf{i}_{n_m}, y_{t_m,\mathbf{i}_{n_m}})\}_{n_m=1}^{N_m},$$6

The modulation parameters are generated from the current SSM state and observation embedding:

$$\mathcal{O}_{t_m} = \{(\mathbf{i}_{n_m}, y_{t_m,\mathbf{i}_{n_m}})\}_{n_m=1}^{N_m},$$7

In the description given by the paper, $$\mathcal{O}_{t_m} = \{(\mathbf{i}_{n_m}, y_{t_m,\mathbf{i}_{n_m}})\}_{n_m=1}^{N_m},$$8 functions as a dynamic weight matrix and $$\mathcal{O}_{t_m} = \{(\mathbf{i}_{n_m}, y_{t_m,\mathbf{i}_{n_m}})\}_{n_m=1}^{N_m},$$9 as a dynamic bias (Chen et al., 8 May 2026).

The formal expressivity claim is stated as Theorem 1. Let $N_m$0 denote fixed-rank functional Tucker models and $N_m$1 denote FT-FiLM with width $N_m$2 and a universal-approximator readout family. Then, on a compact domain $N_m$3,

$N_m$4

The stated meaning is that FT-FiLM is dense in $N_m$5, whereas fixed-rank functional Tucker is not (Chen et al., 8 May 2026). The paper gives $N_m$6 as an example of a function outside the closure of the fixed-rank functional Tucker family but inside the closure of FT-FiLM.

This matters computationally as well as theoretically. The paper states that FT-FiLM allows StreamPhy to use much smaller latent states than SDIFT’s functional Tucker latent space; for the active matter dataset, SDIFT’s core latent dimension is reported as $N_m$7, whereas StreamPhy uses a latent dimension $N_m$8 (Chen et al., 8 May 2026).

6. Empirical performance, ablations, and limitations

The evaluation covers three physical systems: turbulent flow, ocean sound speed, and active matter (Chen et al., 8 May 2026). The datasets are reported as follows.

For Turbulent Flow, the benchmark uses a 2D velocity field with 600 trajectories, each of size $N_m$9; 500 are used for training and 100 for test (Chen et al., 8 May 2026). For Ocean Sound Speed, the benchmark uses 1000 records of size $\mathcal{O} = \{\mathcal{O}_{t_m}\}_{m=1}^M$0; 950 are used for training and 50 for test, with only 10% of spatial points per frame used during training (Chen et al., 8 May 2026). For Active Matter, the benchmark uses 928 records of size $\mathcal{O} = \{\mathcal{O}_{t_m}\}_{m=1}^M$1; 900 are used for training and 28 for test, again with 10% of spatial points used at training time (Chen et al., 8 May 2026).

Two sampling regimes are used: uniform sampling and slab sampling, each with observation ratio $\mathcal{O} = \{\mathcal{O}_{t_m}\}_{m=1}^M$2 (Chen et al., 8 May 2026). Performance is measured by variance-scaled RMSE,

$\mathcal{O} = \{\mathcal{O}_{t_m}\}_{m=1}^M$3

The strongest quantitative results are obtained by StreamPhy (MH). Representative values reported in the paper are listed below.

Setting	Best reported baseline VRMSE	StreamPhy (MH) VRMSE
Turbulent Flow, uniform, $\mathcal{O} = \{\mathcal{O}_{t_m}\}_{m=1}^M$4	0.2779	0.0935
Ocean Sound Speed, uniform, $\mathcal{O} = \{\mathcal{O}_{t_m}\}_{m=1}^M$5	0.0941	0.0628
Active Matter, slab, $\mathcal{O} = \{\mathcal{O}_{t_m}\}_{m=1}^M$6	0.1960	0.0905

These figures are drawn from comparisons against LRTFR, OFTD, CATTE, Senseiver, SDIFT, and StreamPhy (SH) (Chen et al., 8 May 2026). The paper summarizes the aggregate outcome as consistent superiority across all datasets and sampling patterns, with at least 48% improvement in accuracy and 20–100X faster inference than diffusion-based methods (Chen et al., 8 May 2026).

The reported inference-time comparison is especially central to the framework’s identity as a streaming model. On Turbulent Flow at $\mathcal{O} = \{\mathcal{O}_{t_m}\}_{m=1}^M$7, StreamPhy (MH) is reported at approximately 0.04 s per record versus 5.14 s for SDIFT; on Ocean Sound Speed, 0.03–0.04 s versus 0.84–0.89 s; on Active Matter, 0.05 s versus 1.3–1.4 s (Chen et al., 8 May 2026). The paper attributes this to linear-in-$\mathcal{O} = \{\mathcal{O}_{t_m}\}_{m=1}^M$8 sequential updates, the absence of iterative denoising, and lower-dimensional latent representations.

Ablations indicate that the three main design choices are all consequential. Removing the SSM causes VRMSE to collapse to approximately 0.85–0.91 across tasks; removing stochastic masking significantly worsens performance, particularly under slab sampling; replacing FT-FiLM with FTM yields consistently worse performance (Chen et al., 8 May 2026). These ablations support the claim that StreamPhy is not merely an SSM wrapped around an implicit decoder, but a tightly integrated architecture.

The paper also identifies several limitations. StreamPhy is purely data-driven and does not encode PDEs, conservation laws, or boundary conditions explicitly. The transition matrix $\mathcal{O} = \{\mathcal{O}_{t_m}\}_{m=1}^M$9 is fixed by HiPPO-LegS rather than learned. The experiments cover three representative systems, but not strongly discontinuous PDEs, shocks, or more extreme long-horizon chaotic settings. Full-field decoding still scales with the number of query points, which may become expensive for very high-resolution 3D domains (Chen et al., 8 May 2026). A plausible implication is that future variants may combine the present streaming architecture with physics-informed constraints, uncertainty quantification, or control-oriented latent-state design.

7. Position within streaming physical inference

Within the recent literature on reconstruction of physical fields from partial observations, StreamPhy occupies the intersection of online sequence modeling, continuous coordinate representation, and functional tensor methods (Chen et al., 8 May 2026). Its distinctive claim is not only improved reconstruction accuracy, but a specific operational mode: incoming sparse observations are processed once, compressed into a latent state, and made immediately available for continuous-field queries.

This positioning also clarifies what StreamPhy is not. It is not a diffusion posterior sampler, because inference does not proceed through iterative reverse-time steps. It is not a classical functional Tucker decomposition, because its decoder is conditioned dynamically on the latent state through FT-FiLM. It is not a standard regular-grid SSM, because both the observation encoder and the decoder are designed for arbitrary spatial sampling patterns and continuous coordinate queries (Chen et al., 8 May 2026).

For applications in sensing, monitoring, and online scientific computing, the main significance of StreamPhy lies in this combination of properties: data-adaptive encoding of irregular sparse measurements, structured state-space memory across irregular time intervals, and expressive continuous reconstruction with formally stronger approximation guarantees than fixed-rank functional Tucker models (Chen et al., 8 May 2026).