MotionStream: Streaming Motion Forecasting

• MotionStream is a framework for streaming motion forecasting that redefines trajectory prediction as a continuous, temporally consistent process in autonomous driving.
• It incorporates multi-modal occlusion reasoning to predict trajectories for both visible and occluded agents, enhancing robustness over traditional snapshot methods.
• A differentiable filtering module ensures temporal coherence by smoothing predictions, reducing endpoint error and forecast fluctuation.

MotionStream designates a family of streaming methodologies and benchmarks for real-time, temporally consistent, and occlusion-aware motion forecasting in autonomous driving scenarios. As introduced in "Streaming Motion Forecasting for Autonomous Driving" (Pang et al., 2023), it formalizes the trajectory prediction problem on continuous data streams, as opposed to the traditional snapshot-based (per-frame, independent) paradigm. This task formulation and associated algorithmic solutions address the core practical requirements of deployed autonomous agents—namely, forecasting under agent occlusions, ensuring temporal coherence across predictions, and achieving plug-and-play compatibility with existing forecasters.

1. Streaming Forecasting Problem Formulation

MotionStream redefines motion forecasting for autonomous agents as a temporally continuous process in which, at each timestep, the system predicts the future trajectories of all agents whose existence has been observed up to that point, regardless of their current visibility. The canonical setup discards the i.i.d. snapshot assumption prevailing in existing benchmarks (e.g., Waymo, nuScenes, Argoverse), where each trajectory query is independent and only considers currently visible agents furnished with a fixed-length historical window.

In streaming forecasting, the model must:

• Handle causality and online operation: predictions are requested at every frame in the stream.
• Generate trajectories for static, moving, visible, and occluded agents, requiring temporal linkage across agent disappearances and re-appearances.
• Ensure that predictions for temporally overlapping windows are consistent (low fluctuation), which is necessary for downstream modules in perception and control stacks.

This formulation better reflects real-world perception systems, as shown by the new streaming forecasting benchmark Argoverse-SF constructed by repurposing the Argoverse tracking split to include occlusion events and temporally dense query schedules (Pang et al., 2023).

2. Emergent Challenges: Occlusion Reasoning and Temporal Coherence

Streaming forecasting surfaces algorithmic demands largely absent in snapshot evaluation:

• Occlusion Reasoning: Agents naturally disappear from sensor range due to occlusions, leaving their future states indeterminate via direct observation. Models must extrapolate plausible positions and behaviors through occlusion and resume valid forecasting upon re-emergence, including static agents.
• Temporal Coherence: Since future predictions from overlapping current windows should coincide for the intersecting time intervals, independent per-frame forecasts inevitably create temporal jitter and inconsistency. This increases the risk of erratic control decisions in autonomous stacks.

Metrics in the streaming regime are computed over all agents and timeframes, partitioned by visibility (visible/occluded) and dynamicity (moving/static). Standard displacement errors (minADE, minFDE, MR) are visibility-aware, and a specialized fluctuation metric quantifies frame-to-frame prediction smoothness.

3. Predictive Streamer Meta-Algorithm

The Predictive Streamer is a meta-algorithm enabling snapshot-based trajectory predictors to function as streaming (MotionStream-compatible) forecasters without redesign of their architectural core (Pang et al., 2023). This is accomplished by introducing two critical modules:

a. Occlusion Reasoning with Multi-modal Propagation

For each occluded agent, the module propagates agent state using the most confident trajectory predicted by the multi-modal underlying model. For agent $a$: $c_t^a = c_{t-1}^a + \widehat{p}^{a}_{t-1:t}$ $\widehat{p}^{a}_{t-1:t}$ is the best trajectory from the model’s multi-modal outputs. Unlike legacy approaches that default to a single, linear extrapolation or Kalman filter, this approach leverages the model’s context-driven and multi-modal outputs to supply richer, more plausible future agent paths through occlusion intervals.

b. Temporal Consistency via Differentiable Filtering (DF)

To address fluctuation, the predictions for each agent are smoothed using a differentiable filter reminiscent of a neural Kalman filter. The hidden state $x_t$ is updated with the model's observation $z_t$: $x_t = A x_{t-1} + w,\quad z_t = C x_t + \delta$ The filter recursively updates its mean and covariance: $$\begin{align*} \widetilde{\mu}_{x_t} &= A \mu_{x_{t-1}} \ \widetilde{\Sigma}_{x_t} &= A \Sigma_{x_{t-1}} A^\top + Q_t \ K_t &= \widetilde{\Sigma}_{x_t} C^\top (C \widetilde{\Sigma}_{x_t} C^\top + R_t)^{-1} \ \mu_{x_t} &= \widetilde{\mu}_{x_t} + K_t (z_t - C \widetilde{\mu}_{x_t}) \ \Sigma_{x_t} &= (\mathbb{I} - K_t C)\widetilde{\Sigma}_{x_t} \end{align*}$$ $R_t$ is instance-specific observation noise, predicted online by an auxiliary neural network from agent features. The transition matrix $A$ is structured to preserve consistency for overlapping future frames and appropriately extrapolate new steps.

This framework is wholly compatible with any black-box snapshot forecaster, operating as a wrapper and requiring no internal modifications.

4. Streaming Forecasting Benchmark and Evaluation

The Argoverse-SF benchmark implements streaming forecasting, querying predictions on all agents (including occluded and static) at every frame, with temporal linkage accounted for agent disappearances. The evaluation is tailored to streaming-specific requirements:

• Metrics for each subgroup: visible/occluded, moving/static.
• Fluctuation metric: quantifies $\ell_2$ displacement between predictions for the same physical future from adjacent frames, sensitive to jitter.
• Occlusion metrics: minFDE and minADE for predictions through agent-absent intervals.

These measures expose the deficiencies of snapshot-trained models operating in online deployments, which otherwise appear masked by snapshot-based evaluation.

5. Empirical Performance and Significance

In extensive experiments on Argoverse-SF:

• Predictive Streamer equipped with multi-modal occlusion reasoning achieves up to 25% reduction in endpoint error (minFDE) for occluded agents versus snapshot with linear or Kalman propagation.
• Temporal fluctuations between overlapping forecasts drop 10-20% using the differentiable filter, outperforming LSTM smoothing and legacy Kalman filters. For example, employing multi-modal occlusion reasoning with VectorNet baseline (K=6) reduces minFDE for occluded agents from 4.05m (KF) to 3.22m; with the differentiable filter, further to 3.03m.
• The approach is agnostic to the backbone forecaster and demonstrates similar improvements with mmTransformer and under various hyperparameter scalings.
• The plug-and-play property ensures rapid adoption in deployed systems, with the code provided for community standardization.

6. Implications and Future Directions

The MotionStream paradigm, as realized through the Predictive Streamer architecture and the streaming benchmark, shifts the focus of motion forecasting research toward deployment-realistic, temporally coherent, and occlusion-aware systems. Its generality ensures compatibility with any state-of-the-art forecaster. This benchmark is likely to inform subsequent dataset construction, challenge settings, and downstream planning module integration.

A plausible implication is that widespread adoption of streaming forecasting will yield safer and more reliable motion prediction subsystems in autonomous vehicles, given the direct alignment with the causal, partially observed, and temporally continuous reality of real-world environments. Further, the modularity of the Predictive Streamer suggests pathways for incorporating richer context signals or tighter integration with differentiable planners in future research.