Foresight Trajectory Generation

• Foresight Trajectory Generation is a design principle where explicit future states (latent simulations, goal-conditioned planning, and world-model rollouts) guide current decision-making.
• It employs stochastic RNNs, IRL frameworks, and curvature-aware modules to enhance prediction accuracy and achieve measurable success in adaptive control systems.
• Various paradigms integrate uncertainty-driven control, structural synthesis, and generative models to align predicted trajectories with real-world feasibility and scene constraints.

Foresight trajectory generation, in the literature represented here, denotes methods that anticipate future states, paths, scenes, or action consequences before committing to prediction or control. Rather than relying only on local extrapolation from present observations, these methods use internal simulation, goal- or reward-conditioned planning, multimodal proposal generation, or world-model rollouts to bias behavior toward futures that are informative, feasible, or goal-aligned. The term is not tied to a single formalism: it appears in minimum-time aircraft intent generation (Shahzad et al., 2013), uncertainty-driven robot motion control (Hiruma et al., 2024), scene-induced trajectory forecasting via planning (Deo et al., 2019), multimodal motion prediction with reward heuristics (Pei et al., 16 Jul 2025), trajectory- and world-model generation with conditional flow or diffusion (Ye et al., 2024), and anticipatory autonomous driving and navigation frameworks that imagine future scenes before acting (Zhang et al., 8 May 2026).

1. Conceptual scope and lineage

A recurring property across these works is that future hypotheses are elevated to first-class computational objects. Depending on the domain, those hypotheses may be latent hidden states, candidate goal paths, scene-specific trajectory sets, anchor trajectories, dense point trajectories, imagined future video states, or identity-tagged object tracklets. This suggests that “foresight” is less a particular model family than a design principle: current decisions are conditioned on explicit future structure rather than inferred only implicitly from reactive state encodings.

The historical span is broad. In “Trajectory generation and display for free flight” (Shahzad et al., 2013), foresight appears as intent-based, predictive aircraft trajectories for minimum-time convergence under turn-rate and separation constraints, with a neural approximator used online. In robot control, “Adaptive Motion Generation Using Uncertainty-Driven Foresight Prediction” (Hiruma et al., 2024) embeds foresight inside a stochastic recurrent controller through internal rollouts and uncertainty-based hidden-state replacement. In motion forecasting, “Scene Induced Multi-Modal Trajectory Forecasting via Planning” (Deo et al., 2019) and “Foresight in Motion: Reinforcing Trajectory Prediction with Reward Heuristics” (Pei et al., 16 Jul 2025) treat future trajectory generation as planning toward goals or intentions derived from learned reward structure. In modern world-model work, imagined future scenes themselves become the substrate for planning (Zhang et al., 8 May 2026).

Paradigm	Representative mechanism	Representative papers
Uncertainty-driven control	Latent rollouts and variance-based selection	(Hiruma et al., 2024, Hiruma et al., 11 Oct 2025)
Goal/reward planning	Goal prediction, IRL, policy rollouts	(Deo et al., 2019, Pei et al., 16 Jul 2025)
Structure-aware synthesis	Key steps, lane-graph sets, curvature or kinematics constraints	(Zhang et al., 2024, Vivekanandan et al., 2024, Sun et al., 2020, Jiao et al., 2023)
Generative/world-model foresight	Flow matching, diffusion, imagined future scenes	(Ye et al., 2024, Zhang et al., 22 May 2025, Zhang et al., 8 May 2026, Hu et al., 25 Dec 2025, Mei et al., 23 Jun 2026)

2. Uncertainty-driven internal simulation in adaptive control

In predictive robot control, foresight trajectory generation can be implemented as short-horizon internal simulation over latent dynamics. The formulation in “Adaptive Motion Generation Using Uncertainty-Driven Foresight Prediction” (Hiruma et al., 2024) augments a predictive-learning controller with a stochastic RNN that outputs a Gaussian distribution over the next observation, with mean $o_{t+1}^{mean}$ and variance $o_{t+1}^{var}$. The controller operates in closed loop: the model predicts next-step sensor values, those predictions are applied to the robot controller, and the resulting real observation updates the recurrent state. The foresight module perturbs the hidden state, rolls each perturbation forward over horizon $T$ under closed-loop input substitution, and replaces the current hidden state with the candidate that yields the lowest predicted variance at the horizon:

$$H_{t+1} = f_\theta(H_t, i_t), \qquad p(o_{t+1}\mid H_{t+1}) = \mathcal{N}(o_{t+1}^{mean}, o_{t+1}^{var}),$$

$$\epsilon^{(j)} \sim \mathcal{N}(0, f(o_t^{var}) I), \qquad H_t^{(j)} = H_t + \epsilon^{(j)},$$

$\hat H_t = \arg\min_j\, o_{t+T}^{(j),var}.$

This design shifts uncertainty from being merely an output statistic to being an operational signal for latent-state selection. The paper explicitly interprets the controller as a stochastic policy over next observations rather than an explicit $\pi$ over actions, and argues that the foresight module embeds uncertainty in the latent policy rather than only in resultant observations (Hiruma et al., 2024). On a door-opening task with visually indistinguishable push, pull, and slide modes, trained from five demonstrations per mode, the model adaptively diverged its motion through interaction with the door, whereas conventional methods failed to stably diverge. The reported results state that the proposed model achieved over 80% success rates at the early stage of training and that conventional models typically succeeded at only two out of three motion types (Hiruma et al., 2024).

The dynamical interpretation is unusually explicit. The model was analyzed with local Lyapunov exponents defined by $J_t=\partial H_{t+1}/\partial H_t$ and $\lambda_t=\ln \sigma_{\max}(J_t)$. The foresight model showed clear peaks in $\lambda_t$ at critical interaction times such as grasping the handle and the onset of door movement, whereas a random-noise baseline produced peaks aligned more with resultant observation changes (Hiruma et al., 2024). This suggests that useful divergence was being organized around decision-relevant latent causes rather than downstream effects.

UF-RNN generalizes this pattern into a Stochastic Hierarchical LSTM with a pre-trained convolutional autoencoder for vision, adaptive noise scaling, and a variance-reduction objective over multimodal sensory predictions (Hiruma et al., 11 Oct 2025). It formalizes selection by maximizing

$o_{t+1}^{var}$0

runs at 10 Hz, uses $o_{t+1}^{var}$1 rollouts per timestep with horizon $o_{t+1}^{var}$2, and reports real-time performance in both simulation and a UR5e door-opening setting (Hiruma et al., 11 Oct 2025). Across both papers, the computational footprint scales as $o_{t+1}^{var}$3 per control cycle, and both explicitly note sensitivity to variance calibration and closed-loop model bias.

3. Planning, reward inference, and intention reasoning

Another major interpretation of foresight trajectory generation treats trajectory prediction as planning toward latent goals or intentions. In “Scene Induced Multi-Modal Trajectory Forecasting via Planning” (Deo et al., 2019), the pipeline consists of a goal prediction model, a Maximum Entropy Deep IRL reward model, and a GRU-based trajectory generator. A Fully Convolutional Network processes a top-down scene image to produce a goal heatmap, a GRU encodes the past trajectory, and their heatmaps are fused by addition and softmax to obtain $o_{t+1}^{var}$4. For each sampled goal, forward reinforcement learning yields an optimal policy and scene-conforming path, and a GRU decoder with soft attention over planned way-points generates continuous future trajectories. On Stanford Drone Dataset, the method reports test-set results of $o_{t+1}^{var}$5 and $o_{t+1}^{var}$6, improving over SocialGAN, DESIRE, MATF GAN, and SoPhie under the stated evaluation protocol (Deo et al., 2019).

The more recent “Foresight in Motion: Reinforcing Trajectory Prediction with Reward Heuristics” (Pei et al., 16 Jul 2025) moves the reasoning stage closer to explicit intention modeling. It introduces a query-centric MaxEnt IRL pipeline on a BEV grid, where learnable grid-shaped queries aggregate vectorized agent and map context, a stack of $o_{t+1}^{var}$7 CNN layers maps the resulting grid tokens to a scalar reward per cell, and soft-optimal policy rollouts sample multiple intention sequences called Grid-based Reasoning Traversals. Those intention-informed tokens guide a hierarchical DETR-like decoder enhanced by bidirectional selective state space models. The final trajectory output is refined by

$o_{t+1}^{var}$8

and the training objective is

$o_{t+1}^{var}$9

The paper reports, on Argoverse 2 custom validation, that FiM with longer-term intention supervision achieves $T$0, $T$1, and $T$2, and on the nuScenes leaderboard it reports $T$3, $T$4, $T$5, and $T$6 (Pei et al., 16 Jul 2025).

These planning-based systems share an architectural separation between high-level reasoning and low-level motion synthesis. Goals, rewards, or intention traversals provide spatial priors; a downstream decoder then produces temporally coherent trajectories. A plausible implication is that the main value of foresight here is not only multimodality, but also scene compliance: sidewalks, crosswalks, lane-following, turns, and overtaking maneuvers are represented before precise coordinates are generated.

A useful boundary case is “Hindsight Foresight Relabeling for Meta-Reinforcement Learning” (Wan et al., 2021). In that work, foresight does not synthesize new rollout trajectories. Instead, it evaluates how useful a pre-adaptation trajectory $T$7 will be for a task $T$8 after adaptation, through the utility

$T$9

with a practical Q-based estimator. The resulting relabeling posterior

$$H_{t+1} = f_\theta(H_t, i_t), \qquad p(o_{t+1}\mid H_{t+1}) = \mathcal{N}(o_{t+1}^{mean}, o_{t+1}^{var}),$$0

is explicitly described as foresight-guided trajectory relabeling rather than explicit rollout generation (Wan et al., 2021). This usage shows that “foresight trajectory generation” can extend beyond motor prediction into data assignment and meta-training.

4. Structure-preserving synthesis and feasibility-aware trajectory design

Several lines of work impose foresight through explicit structural constraints rather than through latent simulation or reward rollouts. “G2LTraj: A Global-to-Local Generation Approach for Trajectory Prediction” (Zhang et al., 2024) first generates global key steps uniformly covering the full horizon and then recursively fills the intermediate steps between adjacent key steps. Spatial constraints are enforced by a regression loss on relative displacements between adjacent key steps, temporal constraints are strengthened through positional embeddings and agent features, and a selectable granularity mechanism predicts which key-step spacing is best for each trajectory. The total objective is

$$H_{t+1} = f_\theta(H_t, i_t), \qquad p(o_{t+1}\mid H_{t+1}) = \mathcal{N}(o_{t+1}^{mean}, o_{t+1}^{var}),$$1

Across ETH/UCY and nuScenes, the method improves seven existing predictors, for example reducing AutoBot $$H_{t+1} = f_\theta(H_t, i_t), \qquad p(o_{t+1}\mid H_{t+1}) = \mathcal{N}(o_{t+1}^{mean}, o_{t+1}^{var}),$$2 from $$H_{t+1} = f_\theta(H_t, i_t), \qquad p(o_{t+1}\mid H_{t+1}) = \mathcal{N}(o_{t+1}^{mean}, o_{t+1}^{var}),$$3 to $$H_{t+1} = f_\theta(H_t, i_t), \qquad p(o_{t+1}\mid H_{t+1}) = \mathcal{N}(o_{t+1}^{mean}, o_{t+1}^{var}),$$4 and $$H_{t+1} = f_\theta(H_t, i_t), \qquad p(o_{t+1}\mid H_{t+1}) = \mathcal{N}(o_{t+1}^{mean}, o_{t+1}^{var}),$$5 from $$H_{t+1} = f_\theta(H_t, i_t), \qquad p(o_{t+1}\mid H_{t+1}) = \mathcal{N}(o_{t+1}^{mean}, o_{t+1}^{var}),$$6 to $$H_{t+1} = f_\theta(H_t, i_t), \qquad p(o_{t+1}\mid H_{t+1}) = \mathcal{N}(o_{t+1}^{mean}, o_{t+1}^{var}),$$7 (Zhang et al., 2024).

A complementary design appears in “Efficient Data Representation for Motion Forecasting: A Scene-Specific Trajectory Set Approach” (Vivekanandan et al., 2024). Here foresight trajectory generation is the construction of compact, scene-specific sets of plausible futures. Deterministic goal sampling uses a vector-lane graph, directional filtering, source-node selection, and DFS to separate intersection and non-intersection scenarios, while Recursive In-Distribution Subsampling removes redundant trajectories by an $$H_{t+1} = f_\theta(H_t, i_t), \qquad p(o_{t+1}\mid H_{t+1}) = \mathcal{N}(o_{t+1}^{mean}, o_{t+1}^{var}),$$8 m proximity threshold and recursively replenishes from the training distribution. Plausibility is operationalized by Driving Area Compliance,

$$H_{t+1} = f_\theta(H_t, i_t), \qquad p(o_{t+1}\mid H_{t+1}) = \mathcal{N}(o_{t+1}^{mean}, o_{t+1}^{var}),$$9

and the abstract states improvements of up to a 10% improvement in DAC on Argoverse 2, while the detailed tables report even larger gains in some scene-specific settings (Vivekanandan et al., 2024).

Geometric feasibility is the core issue in “An Efficient Generation Method based on Dynamic Curvature of the Reference Curve for Robust Trajectory Planning” (Sun et al., 2020). The paper identifies projection discontinuities, unmapped reference-curve segments, and self-intersections in classic Frenet planning, and proposes a curvature-aware Cartesian-to-Frenet transformation governed by the condition

$$\epsilon^{(j)} \sim \mathcal{N}(0, f(o_t^{var}) I), \qquad H_t^{(j)} = H_t + \epsilon^{(j)},$$0

This condition is used to prevent the singularity induced by $$\epsilon^{(j)} \sim \mathcal{N}(0, f(o_t^{var}) I), \qquad H_t^{(j)} = H_t + \epsilon^{(j)},$$1, while the new projection operator resolves multiple nearest-point ambiguities by selecting the largest $$\epsilon^{(j)} \sim \mathcal{N}(0, f(o_t^{var}) I), \qquad H_t^{(j)} = H_t + \epsilon^{(j)},$$2 and using bisector-based partitioning. The contribution is qualitative but foundational: foresight is tied to maintaining a well-posed trajectory representation across the full planning horizon.

Kinematic priors play an analogous role in “Kinematics-aware Trajectory Generation and Prediction with Latent Stochastic Differential Modeling” (Jiao et al., 2023). The model couples a latent neural SDE

$$\epsilon^{(j)} \sim \mathcal{N}(0, f(o_t^{var}) I), \qquad H_t^{(j)} = H_t + \epsilon^{(j)},$$3

to a bicycle-model latent state

$$\epsilon^{(j)} \sim \mathcal{N}(0, f(o_t^{var}) I), \qquad H_t^{(j)} = H_t + \epsilon^{(j)},$$4

$$\epsilon^{(j)} \sim \mathcal{N}(0, f(o_t^{var}) I), \qquad H_t^{(j)} = H_t + \epsilon^{(j)},$$5

with a kinematics-guidance loss aligning the two latent rollouts. On Argoverse, the method reports average jerk $$\epsilon^{(j)} \sim \mathcal{N}(0, f(o_t^{var}) I), \qquad H_t^{(j)} = H_t + \epsilon^{(j)},$$6, jerk violation $$\epsilon^{(j)} \sim \mathcal{N}(0, f(o_t^{var}) I), \qquad H_t^{(j)} = H_t + \epsilon^{(j)},$$7, acceleration Wasserstein distance $$\epsilon^{(j)} \sim \mathcal{N}(0, f(o_t^{var}) I), \qquad H_t^{(j)} = H_t + \epsilon^{(j)},$$8, $$\epsilon^{(j)} \sim \mathcal{N}(0, f(o_t^{var}) I), \qquad H_t^{(j)} = H_t + \epsilon^{(j)},$$9, and $\hat H_t = \arg\min_j\, o_{t+T}^{(j),var}.$0, outperforming both a DL-based generator and a purely model-based baseline on realism metrics (Jiao et al., 2023).

The free-flight formulation in (Shahzad et al., 2013) shows that this concern with structure predates modern generative modeling. There, optimal relative convergence trajectories are characterized by maximum-bank turns and straight segments, parameterized by turn directions, turn angles, and straight lengths. The neural approximator is used adaptively at 1 Hz, and when leader intent is known the reported convergence time decreases from approximately $\hat H_t = \arg\min_j\, o_{t+T}^{(j),var}.$1 s to approximately $\hat H_t = \arg\min_j\, o_{t+T}^{(j),var}.$2 s. This suggests that intent-aware foresight and feasibility-aware decomposition were already central long before diffusion, flow matching, or transformer-based world models.

5. Generative models, world models, and multimodal reasoning

Recent work often casts foresight trajectory generation as conditional generation over future motion or future world states. “Efficient Trajectory Forecasting and Generation with Conditional Flow Matching” (Ye et al., 2024) learns a time-varying vector field

$\hat H_t = \arg\min_j\, o_{t+T}^{(j),var}.$3

with a Gaussian bridge

$\hat H_t = \arg\min_j\, o_{t+T}^{(j),var}.$4

Inference integrates an ODE with as few as one Euler step. The paper reports up to $\hat H_t = \arg\min_j\, o_{t+T}^{(j),var}.$5 speed-up over diffusion, an average improvement of approximately $\hat H_t = \arg\min_j\, o_{t+T}^{(j),var}.$6 on aircraft forecasting metrics, and a $\hat H_t = \arg\min_j\, o_{t+T}^{(j),var}.$7 increase in Maze2D planning score at $\hat H_t = \arg\min_j\, o_{t+T}^{(j),var}.$8 relative to Diffuser (Ye et al., 2024). “Consistent World Models via Foresight Diffusion” (Zhang et al., 22 May 2025) addresses a different axis: sample consistency. It decouples deterministic condition understanding from stochastic denoising, freezes the predictive stream during stage two, and reports markedly lower sample STD together with better PSNR, SSIM, LPIPS, and Relative $\hat H_t = \arg\min_j\, o_{t+T}^{(j),var}.$9 on robot video prediction and PDE forecasting (Zhang et al., 22 May 2025).

A more radical shift is to generate motion directly rather than pixels. “What Happens Next? Anticipating Future Motion by Generating Point Trajectories” (Boduljak et al., 25 Sep 2025) formulates

$\pi$0

over quasi-dense trajectory grids from a single image, using a trajectory $\pi$1-VAE and latent rectified flow. The trajectory tensor is

$\pi$2

and the flow objective is

$\pi$3

On Kubric, LIBERO, and Physics101, the paper reports stronger motion-distribution and rigidity metrics than image-to-video baselines, and a user study in which the method ranked first 52% of the time (Boduljak et al., 25 Sep 2025). The authors explicitly argue that video generators struggle because they expend capacity on pixels rather than directly modeling motion.

World-model-centric planning pushes foresight one level higher, from trajectory proposal to imagined future scene generation. In autonomous driving, “See Tomorrow, Act Today: Foresight-Driven Autonomous Driving” (Zhang et al., 8 May 2026) samples future visual features $\pi$4 from a pretrained world model, compresses them with WM-QFormer, and conditions a state-based action decoder on both current and future representations. The training objective is

$\pi$5

and on NAVSIM the method reports $\pi$6, $\pi$7, $\pi$8, $\pi$9, $J_t=\partial H_{t+1}/\partial H_t$0, and $J_t=\partial H_{t+1}/\partial H_t$1 (Zhang et al., 8 May 2026). “AstraNav-World: World Model for Foresight Control and Consistency” (Hu et al., 25 Dec 2025) instead jointly models future visual states and action sequences within a unified probabilistic framework, coupling a diffusion-based video generator and a vision-language policy through shared planner tokens and bidirectional cross-attention. With Sparse Foresight Scheduling, the video generator is activated only at fixed intervals, yielding up to $J_t=\partial H_{t+1}/\partial H_t$2 speedup with minimal success-rate drop (Hu et al., 25 Dec 2025). “NavWM: A Unified Navigation World Model for Foresight-Driven Planning” (Mei et al., 23 Jun 2026) combines latent world tokens, anchor-based multimodal action prediction, and controllable visual generation; in rollout it reports success rate $J_t=\partial H_{t+1}/\partial H_t$3 on seen scenes and $J_t=\partial H_{t+1}/\partial H_t$4 zero-shot on unseen scenes (Mei et al., 23 Jun 2026).

Perception stacks can also internalize foresight. “ForeSight: Multi-View Streaming Joint Object Detection and Trajectory Forecasting” (Papais et al., 9 Aug 2025) couples detection and forecasting through a shared bidirectional query memory, using forecast-aware detection and streaming forecast transformers. It is explicitly tracking-free and reports $J_t=\partial H_{t+1}/\partial H_t$5, surpassing previous methods by $J_t=\partial H_{t+1}/\partial H_t$6, while also attaining the best mAP and minADE among multi-view detection and forecasting models on nuScenes (Papais et al., 9 Aug 2025). In multimodal reasoning, “Merlin: Empowering Multimodal LLMs with Foresight Minds” (Yu et al., 2023) uses Foresight Pre-Training and Foresight Instruction-Tuning so that the model first generates identity-tagged subject trajectories as normalized bounding boxes and then reasons about future events conditioned on those trajectories. The same article reports average future-reasoning scores of $J_t=\partial H_{t+1}/\partial H_t$7 on MMBench dev and $J_t=\partial H_{t+1}/\partial H_t$8 on test for Merlin-7B, together with tracking results on LaSOT and GOT10K (Yu et al., 2023).

6. Evaluation regimes, conceptual boundaries, and limitations

Because the term spans multiple subfields, evaluation is heterogeneous. Some works evaluate motor adaptation and latent dynamics, others forecasting accuracy, others planning success or visual consistency. This diversity is not incidental; it reflects the fact that different communities operationalize “foresight” at different interfaces of the perception–prediction–planning stack.

Setting	Metrics explicitly used	Representative papers
Adaptive control	Success rate, Lyapunov exponents	(Hiruma et al., 2024, Hiruma et al., 11 Oct 2025)
Motion forecasting	mADE, mFDE, minADE, minFDE, MR, Brier	(Deo et al., 2019, Pei et al., 16 Jul 2025)
Driving/navigation	EPA, mAP, PDMS, SR, SPL, ATE, RPE	(Papais et al., 9 Aug 2025, Zhang et al., 8 May 2026, Hu et al., 25 Dec 2025, Mei et al., 23 Jun 2026)
Representation/plausibility	DAC, jerk, Wasserstein, LRTL, MHD	(Vivekanandan et al., 2024, Jiao et al., 2023, Boduljak et al., 25 Sep 2025, Ogawa et al., 2019)

A common misconception is that foresight trajectory generation is synonymous with diffusion-based trajectory synthesis. The cited literature contradicts that view. Important formulations use stochastic RNNs with variance-guided hidden-state replacement (Hiruma et al., 2024), MaxEnt IRL with GRU or DETR-like decoders (Deo et al., 2019), curvature-aware Frenet transformations (Sun et al., 2020), anchor sets and RIDS (Vivekanandan et al., 2024), bicycle-model-guided latent SDEs (Jiao et al., 2023), or direct point-trajectory generation from a single image (Boduljak et al., 25 Sep 2025). Another misconception is that foresight always means explicit rollout generation; HFR instead computes post-adaptation trajectory utility and uses it to relabel existing trajectories across tasks (Wan et al., 2021).

The limitations are likewise domain-dependent but structurally similar. Uncertainty-driven controllers rely on calibrated variance estimates, short-horizon closed-loop predictions, and practical $J_t=\partial H_{t+1}/\partial H_t$9 settings; miscalibration or model bias can misguide hidden-state selection (Hiruma et al., 2024). Reward- and map-based systems depend on accurate scene parsing, discretization resolution, and goal or reward inference; ambiguous interactions, occlusions, and map inaccuracies can distort reward fields or scene tags (Pei et al., 16 Jul 2025). Flow- and diffusion-based world models trade multimodality against consistency, can suffer under long horizons or distribution shift, and may incur substantial runtime or memory costs despite techniques such as short-horizon rollouts, frozen predictive streams, or sparse foresight scheduling (Zhang et al., 22 May 2025, Hu et al., 25 Dec 2025). Structure-aware methods improve plausibility, but they can underrepresent rare maneuvers, interaction-driven deviations, or highly nonlocal dynamics (Vivekanandan et al., 2024, Sun et al., 2020).

Taken together, these works suggest that the central question of foresight trajectory generation is not whether a system can produce a future path, but where in the pipeline future structure is represented and how that representation constrains present decisions. In some systems, the crucial object is a low-variance latent rollout; in others, a reward-conditioned intention path, a set of scene-specific anchors, a dense motion field, or an imagined future world. The contemporary trend is toward tighter coupling among these representations: structural priors, multimodal proposals, and visual or latent world models are increasingly trained so that future imagination is not auxiliary to control, but part of the mechanism by which control is formed.