Future Dynamics Predictor Overview
- Future Dynamics Predictor (FDP) is a forecasting module that extrapolates future system states from present or historical observations across diverse fields.
- It employs varied methods—ranging from off-the-shelf video generators and conditional VAEs to NeuralSDEs and reduced-order models—to predict trajectories, frames, or latent representations.
- Empirical results indicate that integrating FDP enhances performance in long-horizon robotic manipulation, reinforcement learning evaluation, and multi-agent motion forecasting.
Future Dynamics Predictor (FDP) denotes a forecasting component that extrapolates a system’s future evolution from present observations, latent states, or historical trajectories. In recent arXiv literature, the label appears across robotics, reinforcement learning, video prediction, multi-agent motion prediction, dynamic scene graph forecasting, reduced-order modeling, autonomous driving, redirected walking, finance, and time-series forecasting, but it does not identify a single canonical architecture. Instead, the term refers to a role within a larger pipeline: the module that produces future states, future trajectories, future frames, or future latent representations needed by downstream planning, control, evaluation, or decoding stages (Ye et al., 26 Sep 2025, Tang et al., 2021, Yılmaz et al., 14 Mar 2025, Rowe et al., 2022, Yang et al., 2 Jun 2025, Andreuzzi et al., 2021, Wang et al., 2024, Jeon et al., 2023, Abdelhakmi et al., 15 Sep 2025, Brachet et al., 21 Jul 2025).
1. Conceptual scope and formalizations
Across the cited works, FDPs are defined by the type of future object they predict and by the interface through which they condition that prediction. In long-horizon robotic manipulation, the FDP maps the robot’s current RGB observation and a HIT-produced guide prompt to an imagined rollout,
where the output is a short video of subtask execution (Ye et al., 26 Sep 2025). In value estimation for reinforcement learning, the predictive component is the dynamics function
which is composed with a return model so that (Tang et al., 2021). In the PRO-DYN formulation for time-series forecasting, the FDP is the “DYN” block that explicitly crosses from the historical interval to the forecast interval , recursively applying a transition to generate (Brachet et al., 21 Jul 2025).
The diversity of outputs is central. Some FDPs emit pixel-space futures; others emit latent embeddings, graph-structured futures, or conditional distributions. This terminological breadth is visible in the following summary.
| Domain | FDP output | Representative mechanism |
|---|---|---|
| Robotics manipulation | Imagined subtask video | Pre-trained video generator conditioned on and 0 |
| Reinforcement learning | Future trajectory embedding 1 | Conditional VAE plus convex return model |
| Video prediction | Next frame 2 | Flow-guided deformable convolution |
| Multi-agent motion | Joint future trajectories | DAG factorization with DAGNN decoding |
| Scene graph forecasting | Future entities, boxes, and predicates | NeuralSDE plus temporal aggregation |
| Finance | Conditional factor process 3 | OU conditioning on noisy expert views |
A recurring misconception is that an FDP must be a freshly trained, explicitly physics-constrained predictor. The cited literature does not support that restriction. One FDP is an off-the-shelf video generator used without fine-tuning (Ye et al., 26 Sep 2025); another is a conditional VAE trained by an ELBO (Tang et al., 2021); another is a reduced-order DMD pipeline with parameter regression (Andreuzzi et al., 2021); another is a closed-form conditional Gaussian process over factors after incorporating expert views (Abdelhakmi et al., 15 Sep 2025).
2. Long-horizon robotic manipulation
In Super-Mimic, the FDP is the second stage of a hierarchical zero-shot imitation framework. After the Human Intent Translator parses unscripted human demonstration video into subtasks 4, the FDP conditions a pre-trained video generation model on the current onboard RGB observation 5 and the textual instruction 6, producing a short imagined clip
7
for the physical execution of the current subtask (Ye et al., 26 Sep 2025).
The paper specifies that FDP relies on Wan2.2-Lightning as an out-of-the-box generative backbone. No custom diffusion U-Net or transformer is trained from scratch, and the module undergoes no additional training or fine-tuning. Generated clip length is set to 8 frames, while diffusion steps and guidance scale follow the model’s default settings. The intended object interactions are injected by appending 9 to the conditioning stream. The paper further states that 0 has been trained to respect spatial coherence across frames, yielding realistic object trajectories, rigid-body motions, and contact events in practice.
The imagined rollout is not used directly as a policy. Instead, Super-Mimic extracts control-relevant geometry from it. The pipeline computes a 2D pixel track using Grounded SAM 2, simplifies it with Ramer–Douglas–Peucker, estimates per-frame depth via 1, and back-projects simplified 2D waypoints into 3D using
2
forming 3. Although the generator does not output explicit contact points, the paper argues that visual cues such as collision and contact are encoded in pixel space and are converted by the tracker and depth estimator into 3D waypoint sequences that implicitly carry contact timing and geometry.
The reported evaluation measures FDP through end-task performance rather than pixel-level video prediction error. On “Meal Preparation,” removing FDP and replacing it with a static 2D waypoint planner drops Task Success Rate from 4. Keeping only the final endpoint of the imagined video yields intermediate performance, with TSR 5. With the full FDP in place, Super-Mimic outperforms state-of-the-art text-driven baselines by over 6 absolute TSR on long-horizon tasks, and the paper attributes most of that gain to the dynamics foresight supplied by the imagined trajectory sequence (Ye et al., 26 Sep 2025).
3. Latent future prediction in reinforcement learning
In VDFP, the Future Dynamics Predictor is a formal component of value estimation rather than a planner or renderer. The paper decomposes policy evaluation into two steps: first predict a latent future trajectory representation, then evaluate it. Starting from a trajectory embedding 7 and a deterministic return model 8, it defines the predictive dynamics function
9
When 0 is convex, Jensen’s inequality yields
1
with equality if 2 is linear (Tang et al., 2021).
The implemented FDP is a latent conditional VAE. A 1D CNN first encodes sliding windows of state-action trajectories into fixed-size embeddings 3. The VAE then models the distribution of future embeddings conditioned on 4 by introducing a latent code 5 with prior 6. The encoder
7
and the decoder
8
are trained with the ELBO
9
where 0 yields a 1-VAE. At policy-learning time, clipped Gaussian noise 2, with 3, is passed through the decoder to obtain 4.
The predicted embedding is scored by a convex return model 5, or a one-layer ReLU or ICNN, trained against full discounted returns. The resulting surrogate critic is 6. The same decomposition is integrated into off-policy VD-DDPG and on-policy VD-PPO. In both cases, 7 is trained as a direct regression to Monte Carlo returns rather than through TD-target bootstrapping.
Empirically, the paper evaluates on LunarLander-v2, InvertedDoublePendulum-v1, HalfCheetah-v1, and Walker2d-v1 over 1 M timesteps and 5 seeds. VD-DDPG learns faster and reaches higher asymptotic reward than DDPG and DDSR, and VD-PPO outperforms PPO and A2C. Under delayed-reward variants with delay 8 up to 128 steps, the decomposition is especially robust: at 9 on HalfCheetah, VD-DDPG retains 0 of its no-delay performance, whereas DDPG retains 1 (Tang et al., 2021).
4. Visual forecasting and structured scene extrapolation
In computer vision, FDPs appear both as frame predictors and as predictors of structured semantic futures. FG-DFPN is a future-frame predictor built from four stages: Optical Flow Estimator, Multi-Scale Feature Extraction and Warping, Offset and Mask Prediction, and Reconstruction. Four past grayscale frames are concatenated channel-wise and processed by FlowNet to estimate coarse flow fields 2. Each frame is encoded into three feature scales 3, warped in feature space to 4, then fused by a flow-guided deformable convolution. Offset refinement is defined as
5
and the deformable convolution itself is
6
Training uses a robust Charbonnier loss,
7
with 8, on Vimeo-90K septuplets, Adam, batch size 8, 9 iterations, and learning rate 0 halved every 1 iterations (Yılmaz et al., 14 Mar 2025).
FG-DFPN reports average PSNR/SSIM of 2 dB / 3 on eight MPEG test sequences, compared with 4 dB / 5 for DFPN, a runtime of 6 ms/frame on a single GPU, and a model size of 7 M parameters. Removing flow guidance reduces PSNR by approximately 8 dB and destabilizes training; single-scale variants lose approximately 9 dB; disabling learned masks reduces sharpness in high-motion regions (Yılmaz et al., 14 Mar 2025).
FDSG operates at a different representational level. It forecasts future entity labels, bounding boxes, and relationships for unobserved video frames. The core forecasting module decomposes object queries into content and location parts, forms triplet representations
0
and evolves each triplet with a NeuralSDE,
1
where 2 and 3 are 3-layer MLPs with hidden size 4 and Tanh activations. The forecasted triplets are then split back into per-entity content vectors. For locations, the simplest Identity Mapping proved best, and predicted boxes are produced by combining updated content with the previous location query through an MLP and sigmoid transform (Yang et al., 2 Jun 2025).
FDSG also includes a Temporal Aggregation Module that fuses observed references and forecasted triplets through cascaded Transformer decoder layers with self-attention and cross-attention. Training uses a Hungarian matching objective with focal loss on subject, object, and predicate classification and weighted 5+GIoU losses on boxes. On Action Genome, the method reports, for DSGG in SGDET at IoU 6, R@50 (No-Constraint) 7 versus 8 for OED and mR@50 (No-Constraint) 9 versus 0. For scene graph forecasting at anticipation fraction 1, SGDET forecast R@50 (No-Constraint) is 2 versus 3 for SceneSayer+ baselines, and mR@50 (No-Constraint) is 4 versus 5 (Yang et al., 2 Jun 2025).
These two systems illustrate that an FDP in vision need not predict raw pixels only. It may instead predict latent semantic triplets, future boxes, or future predicates when the target task is structured scene understanding rather than frame synthesis.
5. Interactive motion, trajectory prediction, and human behavior
For multi-agent driving, FJMP casts future dynamics prediction as factorized joint trajectory generation over a learned interaction graph. From observed histories 6 and HD-map context 7, agents are encoded with a future-aware LaneGCN backbone into node features 8. Pairwise interaction edges are classified into no-interaction, 9, or 0, producing a sparse directed graph 1. Cycles are then removed by enumerating directed cycles with Johnson’s algorithm and dropping the lowest-confidence edge until a DAG is obtained. The joint future distribution is factorized as
2
Decoding proceeds in topological order with a DAGNN: source nodes are decoded marginally, non-source nodes aggregate parental messages with graph attention and a GRU update, and conditional futures are then decoded. Training combines focal loss on interaction labels, an auxiliary proposal loss, a winner-takes-all joint regression loss over 3 modalities, and teacher forcing for influencer trajectories (Rowe et al., 2022).
The reported results place FJMP first on the INTERACTION multi-agent test leaderboard metric CMR. The paper also states that it obtains the lowest minADE/minFDE of published methods on that benchmark while nearly eliminating inter-agent collisions, with CrossCol 4. On Argoverse 2, it reduces minADE/minFDE by 5 m relative to a non-factorized baseline and lowers SMR by 6. Ablations show consistent gains from the learned DAG factorization, sparse edges, the proposal decoder, and teacher forcing (Rowe et al., 2022).
DEMO addresses multi-horizon autonomous-vehicle prediction through a two-stage architecture. The first stage uses a Dynamic Bicycle Model together with a DynCVAE to infer latent controls and short-term motion under the physics constraint
7
where 8 and 9. The dynamics stage is trained with a KL term and a dynamics-informed loss 00. The second stage uses a State-Space/Mamba temporal encoder, cross-modal fusion over vehicle, map, and dynamic features, and a spatial-temporal encoder composed of GRU, GCN, and Transformer components. The final decoder outputs 01 trajectory hypotheses and probabilities, with losses 02, 03, maneuver-classification loss, and trajectory-accuracy loss (Wang et al., 2024).
On NGSIM, MoCAD, HighD, and nuScenes, DEMO reports short-term and long-term improvements over state-of-the-art baselines. At 2 s RMSE, it achieves 04 m on NGSIM versus 05 m, 06 m on MoCAD versus 07 m, and 08 m on HighD, where it is second place against 09 m. At 5 s RMSE, it reports 10 m on NGSIM versus 11 m, 12 m on MoCAD versus 13 m, and 14 m on HighD versus 15 m. On nuScenes at 6 s, 16 m versus 17 m and 18 m versus 19 m. Inference speed is 20 ms per sample on RTX 3090, compared with 21 ms for baselines (Wang et al., 2024).
In virtual-reality redirected walking, F-RDW uses a lighter-weight FDP. A single-layer LSTM with 64 hidden units consumes a 50-frame history of gaze direction, body orientation, neck orientation, and velocity, forming 22, and predicts either a future position 23 or a movement-direction distribution over forward, left, and right. Training uses mean-squared error for position prediction or categorical cross-entropy for direction prediction, with Adam, learning rate 24, batch size 64, 100 epochs, and 5-fold cross-validation over 25 frames (Jeon et al., 2023).
The reported single-horizon forecast at 26 s yields 27 direction-classification accuracy and mean displacement error 28 m with standard deviation 29 m. These predictions are fused into existing RDW controllers such as MPCRed, S2C, TAPF, and ARC through a weighting parameter 30, thereby altering steering, artificial-potential forces, or branching priors without replacing the underlying controller logic (Jeon et al., 2023).
6. Reduced-order systems, forward-looking conditioning, and generic forecasting architectures
In scientific computing, FDP can denote a reduced-order extrapolator rather than a neural predictor. The DMD extension for parametric dynamical systems collects full-state snapshots 31 over training parameters 32 and equispaced times 33, stacks them into 34, computes a truncated POD basis 35, and advances reduced states with DMD in either monolithic or partitioned form. Future reduced states satisfy
36
after eigendecomposition of the reduced operator, and predictions for unseen parameters are obtained by fitting a regressor 37 over parameter space, followed by reconstruction
38
The paper reports three case studies: a toy linear system with 39 and pointwise error 40, an unsteady nonlinear heat equation with 41 and mean relative error 42, and parametric Navier–Stokes past a cylinder with 43, HODMD stabilization, and mean relative velocity-field error 44 (Andreuzzi et al., 2021).
In finance, the same label applies to conditional stochastic-process forecasting after incorporating expert views. A baseline dynamic factor model assumes an OU factor process
45
and affine asset-price dynamics driven by 46. Noisy views are modeled as
47
Conditioning on 48 yields posterior factor dynamics
49
and conditional asset-price dynamics that remain affine with modified coefficients 50 and 51. The paper establishes a connection to a Mean-Reverting Bridge and derives closed-form FDP outputs
52
Implementation consists of estimating 53, 54, specifying 55 and 56, solving for the stationary covariance 57, and computing 58, 59, and 60 on each recalibration date (Abdelhakmi et al., 15 Sep 2025).
For generic time-series forecasting, PRO-DYN treats the FDP as the dedicated terminal dynamics block. Historical inputs 61 are processed by PRO-pre modules into latent states 62, after which a DYN block recursively applies
63
and a decoder maps 64 back to 65. The paper evaluates this design on the TFB benchmark across 25 real-world datasets and forecast horizons 66. On ETTm1 at 67, NLinear records MSE 68, Informer 69, Informer + FDP 70, FEDformer 71, FEDformer + FDP 72, FiLM 73, and FiLM + FDP 74. Capacity-matched ablations also favor a terminal dynamics block over a time-preserving feed-forward alternative: Informer with FDP-last reaches MSE 75 versus 76 for PRO-FFN-last, and FEDformer with FDP-last reaches 77 versus 78 (Brachet et al., 21 Jul 2025).
This collection suggests that FDP has become a functional designation rather than a single methodological lineage. A plausible implication is that the common denominator is not architecture class but extrapolative responsibility: the FDP is the component entrusted with crossing the observed–unobserved boundary, whether by imagined video rollout, latent stochastic prediction, graph factorization, reduced-order propagation, Bayesian conditioning, or recursive state transition.