Goal-GAN for Trajectory Prediction

• Goal-GAN is a goal-conditioned, interpretable model that decomposes pedestrian trajectory forecasting into destination estimation and routing.
• It leverages a discrete goal module with Gumbel-Softmax sampling to capture multimodal destination distributions and encode scene semantics.
• The routing module employs attention over local visual patches to generate plausible, physically compliant trajectories reacting to real-world constraints.

Goal-GAN is an interpretable and end-to-end trainable model for human trajectory prediction that treats forecasting as a goal-conditioned generative process. Rather than predicting future motion as a monolithic sequence problem, it separates intention inference—where a pedestrian wants to go—from path planning—how that pedestrian reaches the destination through the scene. The model therefore decomposes trajectory prediction into goal estimation and routing, learns an explicit multimodal distribution over end-goal positions, and generates feasible trajectories conditioned on sampled goals while reacting to physical constraints in the visual scene (Dendorfer et al., 2020).

1. Problem formulation and probabilistic structure

Goal-GAN operates on pedestrian motion observed in ground-plane coordinates. For each agent $i$, the past trajectory is

$$X_i = \{(x_i^t, y_i^t) \in \mathbb{R}^2 \mid t = 1, \ldots, t_{\mathrm{obs}}\},$$

the future trajectory is

$$Y_i = \{(x_i^t, y_i^t) \in \mathbb{R}^2 \mid t = t_{\mathrm{obs}}+1, \ldots, t_{\mathrm{pred}}\},$$

and the end goal is a 2D coordinate $G_i = g \in \mathbb{R}^2$. The visual context is a top-down image or semantic map $I$ of size $H \times W$, rectified with homography to real-world coordinates.

The central modeling assumption is a two-stage probabilistic decomposition:

$$p(Y \mid X, I) = \sum_G p_\theta(G \mid X, I)\, p_\phi(Y \mid X, I, G).$$

Here, $p_\theta(G \mid X, I)$ is a multimodal goal estimator over possible end positions, and $p_\phi(Y \mid X, I, G)$ is a goal-conditioned routing distribution that generates plausible trajectories toward $G$.

This factorization makes the model explicitly interpretable. The first stage estimates a distribution over destinations, and the second stage generates trajectories that are directed toward sampled destinations. The paper characterizes this as an operationalization of prediction in two intuitive steps: first estimate the goal distribution, then condition the trajectory generator on a sampled goal. This suggests that multimodality is modeled primarily at the destination level rather than being delegated entirely to trajectory noise.

2. Goal estimation and multimodal spatial support

The Goal Module models destinations on a discrete 2D grid aligned with the bird’s-eye image. It outputs a score map

$$X_i = \{(x_i^t, y_i^t) \in \mathbb{R}^2 \mid t = 1, \ldots, t_{\mathrm{obs}}\},$$0

where $$X_i = \{(x_i^t, y_i^t) \in \mathbb{R}^2 \mid t = 1, \ldots, t_{\mathrm{obs}}\},$$1, and normalizes it with a softmax:

$$X_i = \{(x_i^t, y_i^t) \in \mathbb{R}^2 \mid t = 1, \ldots, t_{\mathrm{obs}}\},$$2

Because support is spatially discrete, distinct scene-driven modes appear as spatially separated peaks. The paper gives examples such as sidewalk branches at a crossroads and broad support over an open plaza.

The module combines motion history and scene context. Past motion is encoded by a Motion Encoder, where relative displacements $$X_i = \{(x_i^t, y_i^t) \in \mathbb{R}^2 \mid t = 1, \ldots, t_{\mathrm{obs}}\},$$3 are embedded by an MLP and processed by an LSTM:

$$X_i = \{(x_i^t, y_i^t) \in \mathbb{R}^2 \mid t = 1, \ldots, t_{\mathrm{obs}}\},$$4

The scene image $$X_i = \{(x_i^t, y_i^t) \in \mathbb{R}^2 \mid t = 1, \ldots, t_{\mathrm{obs}}\},$$5 is processed by a U-Net-like encoder-decoder CNN with skip connections. In the bottleneck, scene features are fused with $$X_i = \{(x_i^t, y_i^t) \in \mathbb{R}^2 \mid t = 1, \ldots, t_{\mathrm{obs}}\},$$6, and decoding yields the score map:

$$X_i = \{(x_i^t, y_i^t) \in \mathbb{R}^2 \mid t = 1, \ldots, t_{\mathrm{obs}}\},$$7

$$X_i = \{(x_i^t, y_i^t) \in \mathbb{R}^2 \mid t = 1, \ldots, t_{\mathrm{obs}}\},$$8

Training requires sampling goals while retaining differentiability. Goal-GAN uses the Gumbel-Softmax reparameterization:

$$X_i = \{(x_i^t, y_i^t) \in \mathbb{R}^2 \mid t = 1, \ldots, t_{\mathrm{obs}}\},$$9

where $$Y_i = \{(x_i^t, y_i^t) \in \mathbb{R}^2 \mid t = t_{\mathrm{obs}}+1, \ldots, t_{\mathrm{pred}}\},$$0, $$Y_i = \{(x_i^t, y_i^t) \in \mathbb{R}^2 \mid t = t_{\mathrm{obs}}+1, \ldots, t_{\mathrm{pred}}\},$$1 is a temperature, and $$Y_i = \{(x_i^t, y_i^t) \in \mathbb{R}^2 \mid t = t_{\mathrm{obs}}+1, \ldots, t_{\mathrm{pred}}\},$$2 is a relaxed one-hot over grid cells. The relaxed sample is mapped to a 2D coordinate $$Y_i = \{(x_i^t, y_i^t) \in \mathbb{R}^2 \mid t = t_{\mathrm{obs}}+1, \ldots, t_{\mathrm{pred}}\},$$3, for example the cell center in world coordinates.

At inference time, the learned goal distribution is sampled directly:

$$Y_i = \{(x_i^t, y_i^t) \in \mathbb{R}^2 \mid t = t_{\mathrm{obs}}+1, \ldots, t_{\mathrm{pred}}\},$$4

The model then generates $$Y_i = \{(x_i^t, y_i^t) \in \mathbb{R}^2 \mid t = t_{\mathrm{obs}}+1, \ldots, t_{\mathrm{pred}}\},$$5 trajectories conditioned on the sampled goals. The explicit goal distribution is therefore the primary source of multimodal forecasting.

3. Goal-conditioned routing and physical compliance

The Routing Module generates trajectories toward a sampled goal while reacting to local scene constraints. Its hidden state is initialized from motion and goal embeddings:

$$Y_i = \{(x_i^t, y_i^t) \in \mathbb{R}^2 \mid t = t_{\mathrm{obs}}+1, \ldots, t_{\mathrm{pred}}\},$$6

where $$Y_i = \{(x_i^t, y_i^t) \in \mathbb{R}^2 \mid t = t_{\mathrm{obs}}+1, \ldots, t_{\mathrm{pred}}\},$$7 is an MLP embedding of the sampled goal.

At each future step, routing is driven by the current predicted position, the remaining displacement to the goal, and the timestep:

$$Y_i = \{(x_i^t, y_i^t) \in \mathbb{R}^2 \mid t = t_{\mathrm{obs}}+1, \ldots, t_{\mathrm{pred}}\},$$8

$$Y_i = \{(x_i^t, y_i^t) \in \mathbb{R}^2 \mid t = t_{\mathrm{obs}}+1, \ldots, t_{\mathrm{pred}}\},$$9

Local traversability is estimated through soft attention over an image patch centered at the current predicted position:

$G_i = g \in \mathbb{R}^2$0

$G_i = g \in \mathbb{R}^2$1

The architecture distinguishes between global and local physical feasibility. Global feasibility is handled by the Goal Module, which concentrates probability mass on semantically plausible destinations and suppresses obstacles such as roads with cars or trees and grass. Local feasibility is handled by the Routing Module’s attention over patches around $G_i = g \in \mathbb{R}^2$2, steering predicted steps toward traversable regions and around obstacles.

The paper emphasizes that both mechanisms are differentiable and that there is no hard constraint or explicit cost map term. Physical compliance is learned implicitly from visual features and is further encouraged by adversarial training. This suggests that the model does not impose symbolic navigability constraints; instead, scene semantics are absorbed into the learned goal and routing distributions.

4. Objectives, adversarial learning, and inference protocol

Goal-GAN is trained end-to-end with trajectory, goal, and adversarial objectives. The trajectory accuracy term is a best-of-many loss over $G_i = g \in \mathbb{R}^2$3 samples:

$G_i = g \in \mathbb{R}^2$4

This is paired with a goal achievement loss that aligns the last predicted point with the sampled goal:

$G_i = g \in \mathbb{R}^2$5

and a goal cross-entropy loss over the discrete goal grid:

$G_i = g \in \mathbb{R}^2$6

Realism and scene compliance are promoted with Least-Squares GAN losses. The discriminator receives the observed prefix $G_i = g \in \mathbb{R}^2$7, the full trajectory $G_i = g \in \mathbb{R}^2$8 or $G_i = g \in \mathbb{R}^2$9, and visual features from the scene around the evolving position. Its losses are

$I$0

$I$1

The total generator loss is

$I$2

During inference, the procedure is explicitly two-stage. First, sample $I$3 goals from $I$4. Second, for each sampled goal, initialize the routing LSTM and roll out a trajectory using the remaining-goal vector $I$5 together with local visual attention.

Evaluation follows the standard variety protocol:

$I$6

$I$7

The training and evaluation regimen is fixed at observing 8 steps, corresponding to $I$8, and predicting 12 steps, corresponding to $I$9. The paper also notes that Least Squares GAN is used to avoid vanishing gradients present in sigmoid cross-entropy GANs.

5. Benchmarks, metrics, and reported performance

The model is evaluated on ETH and UCY, using 5 sequences across 4 scenarios with leave-one-out training and testing and positions at $H \times W$0 intervals, and on the Stanford Drone Dataset, using 20 outdoor scenes and the standard TrajNet split with a focus on pedestrians (Dendorfer et al., 2020). Predictions are reported in world coordinates on ETH/UCY and in pixels on SDD.

The reported ETH/UCY results use $H \times W$1 samples and ADE/FDE in meters.

Dataset	ADE	FDE
ETH	0.59	1.18
HOTEL	0.19	0.35
UNIV	0.60	1.19
ZARA1	0.43	0.87
ZARA2	0.32	0.65

The paper states that Goal-GAN achieves state-of-the-art on these benchmarks, with roughly 15% error reduction compared to Social-BiGAT and CGNS, and a $H \times W$2 FDE improvement over the prior best.

On SDD, with $H \times W$3, the reported ADE/FDE in pixels are as follows.

Method	ADE	FDE
Goal-GAN	12.2	22.1
SoPhie	16.3	29.4
CF-VAE	12.6	22.3
P2TIRL	12.6	22.1

The paper attributes these gains to explicit goals, stating that they substantially improve over non-goal conditioned baselines and that Goal-GAN matches or extends other goal-conditioned approaches without costly IRL training.

A synthetic multimodality study further isolates the behavior of the full system. Using Social Force Model trajectories and a 2-class semantic map, the full model attains ADE/FDE $H \times W$4, feasibility $H \times W$5, mode coverage $H \times W$6, and $H \times W$7, outperforming vanilla GAN baselines and Goal-GAN variants missing components.

6. Interpretability, ablations, limitations, and implementation

Interpretability is centered on the Goal Module’s heatmap. Because the model produces a spatial probability map over destinations, its belief over plausible goals can be inspected directly. The paper describes peaks at sidewalk exits and low mass on obstacles such as roads with parked cars and trees. Routing toward sampled peaks produces human-understandable plans.

The ablation results on the synthetic dataset clarify the role of each component. Removing the $H \times W$8 trajectory loss raises ADE to $H \times W$9, although mode coverage remains high at $$p(Y \mid X, I) = \sum_G p_\theta(G \mid X, I)\, p_\phi(Y \mid X, I, G).$$0, which the paper interprets as evidence that $$p(Y \mid X, I) = \sum_G p_\theta(G \mid X, I)\, p_\phi(Y \mid X, I, G).$$1 is crucial for accurate paths. Removing Gumbel-Softmax leads to degraded performance, with ADE $$p(Y \mid X, I) = \sum_G p_\theta(G \mid X, I)\, p_\phi(Y \mid X, I, G).$$2, FDE $$p(Y \mid X, I) = \sum_G p_\theta(G \mid X, I)\, p_\phi(Y \mid X, I, G).$$3, feasibility $$p(Y \mid X, I) = \sum_G p_\theta(G \mid X, I)\, p_\phi(Y \mid X, I, G).$$4, and mode coverage $$p(Y \mid X, I) = \sum_G p_\theta(G \mid X, I)\, p_\phi(Y \mid X, I, G).$$5, indicating that differentiable goal sampling is essential for stable training and effective use of the goal distribution. Adding visual attention improves feasibility and mode coverage versus GAN baselines that rely only on noise and attention.

The paper identifies several failure modes. Ambiguous scenes with many equally plausible destinations can yield broad goal distributions and scattered samples; routing may then produce diverse speeds and routes that challenge precise ADE and FDE. Rapid intention changes after $$p(Y \mid X, I) = \sum_G p_\theta(G \mid X, I)\, p_\phi(Y \mid X, I, G).$$6, such as sudden U-turns, and occlusions not reflected in the static top-down image can reduce goal estimation accuracy. Proposed mitigations include sampling more goals with larger $$p(Y \mid X, I) = \sum_G p_\theta(G \mid X, I)\, p_\phi(Y \mid X, I, G).$$7, leveraging broader distributions to hedge uncertainty, and incorporating additional context such as dynamic scene cues or social interactions in future extensions.

Implementation is fully end-to-end. The Motion Encoder, Goal Module, Routing Module, and GAN discriminator are optimized jointly. Visual input is a single top-down image, and coordinates are transformed to world space via homography. The model uses a CNN encoder-decoder fused with $$p(Y \mid X, I) = \sum_G p_\theta(G \mid X, I)\, p_\phi(Y \mid X, I, G).$$8, a goal-conditioned LSTM with soft attention, Gumbel-Softmax for differentiable goal sampling, and an LSGAN discriminator. The authors publicly released code at https://github.com/dendorferpatrick/GoalGAN.

Taken together, these components make the explicit learned goal distribution the linchpin of Goal-GAN’s multimodal and interpretable forecasts. Goals are modeled discretely over the scene, trained with cross-entropy against ground-truth end positions, sampled differentiably during training, and injected into a recurrent routing policy that reacts to local obstacles through attention. The result is a model that combines destination-level multimodality, scene-aware physical compliance, and competitive benchmark accuracy within a single probabilistic framework (Dendorfer et al., 2020).