Class-Conditioned Trajectory Prediction

• Class-conditioned trajectory prediction is a method that leverages discrete variables, such as agent types or intention states, to guide motion forecasting.
• It employs various conditioning strategies—including semantic, behavioral, and latent cues—in modules like encoders, decoders, and attention mechanisms.
• Empirical studies show that integrating class conditioning improves multimodal forecasting, heterogeneous-agent reasoning, and real-time prediction efficiency.

Class-conditioned trajectory prediction denotes trajectory forecasting in which the predictive distribution is conditioned on a discrete or discretized variable that encodes motion-relevant structure, such as an agent class, an action label, an intention state, a route command, a topological class, a latent pseudo-class, or a finite trajectory class. In the recent literature, the conditioning variable may enter as an explicit input to encoders, decoders, attention modules, CRFs, routers, or class-specific reference banks; in other settings it is used only during training to shape the representation. The resulting models target different desiderata, including multimodal forecasting, heterogeneous-agent reasoning, planner-predictor coupling, long-tail robustness, and physically plausible prediction (Schmidt et al., 2023, Mercurius et al., 2024, Halawa et al., 2022, Pathiraja et al., 2022).

1. Problem formulations and the scope of “class”

A common formalization is to predict a future sequence under a conditioning variable, as in $p(P_{t_0+1:t_0+T_p} \mid X, c, \mathcal{E})$ or equivalently $p(Y_{t_0+1:t_0+T_p} \mid X, c, \mathcal{E})$, where $X$ is the observation history, $c$ is a class label, and $\mathcal{E}$ denotes context features (Almeida et al., 4 Oct 2025). However, the literature uses “class” in several technically distinct senses. In heterogeneous-agent forecasting, it may mean a semantic category such as pedestrian, bicyclist, car, bus, or rider. In behavior-conditioned models, it may denote action or maneuver categories, route commands, or intention states. In set-based predictors, trajectories themselves become classes. In latent-variable approaches, the classes are implicit clusters learned from encoder features rather than externally annotated labels.

This breadth is explicit in recent formulations. RESET models conditional behavior prediction by discretizing trajectory space into a finite trajectory set $S=\{\tau_k\}_{k=1}^K$ and predicting $p(\tau_k \mid C, P_{\text{ego}})$, where the ego planned trajectory $P_{\text{ego}}$ is itself part of the conditioning context (Schmidt et al., 2023). DROGON introduces a discrete intention variable $g \in \mathcal{G}$ and factors prediction as

$$p(Y \mid X)=\sum_{g \in \mathcal{G}} p(Y \mid X,g)\,p(g \mid X),$$

so that goal regions serve as intention classes (Choi et al., 2019). In topological forecasting, homotopy classes indexed by reduced h-signatures play the role of sparse motion labels, and the low-level predictive distribution becomes

$p(Y_{t_0+1:t_0+T_p} \mid X, c, \mathcal{E})$0

where $p(Y_{t_0+1:t_0+T_p} \mid X, c, \mathcal{E})$1 is a partial h-signature extracted from the observed prefix (Wakulicz et al., 2023). S-T CRF uses discrete lateral and longitudinal intention states, with $p(Y_{t_0+1:t_0+T_p} \mid X, c, \mathcal{E})$2 for keep direction, turn left, and turn right, and $p(Y_{t_0+1:t_0+T_p} \mid X, c, \mathcal{E})$3 for keep speed, deceleration, and acceleration (Han et al., 2023).

Conditioning type	Representative class variable	Representative papers
Semantic or physical class	Agent type, class, dimensions	(Li et al., 2022, Pathiraja et al., 2022, He et al., 18 Apr 2025)
Behavioral or intentional class	Action, intention, route command, destination zone	(Halawa et al., 2022, Han et al., 2023, Buhet et al., 2019, Choi et al., 2019)
Structural or latent class	Trajectory class, homotopy class, latent cluster	(Schmidt et al., 2023, Wakulicz et al., 2023, Mercurius et al., 2024)

A persistent misconception is that class-conditioned prediction is equivalent to semantic multi-class forecasting. The surveyed work contradicts that view. Classes may be externally annotated, derived from geometry, inferred from latent clusters, or defined directly in trajectory space.

2. Conditioning variables and how they are represented

Explicit semantic conditioning is typically implemented by embeddings or class-specific modules. In the multi-class GAN framework for the Stanford Drone Dataset, each agent’s class label is encoded as a one-hot vector $p(Y_{t_0+1:t_0+T_p} \mid X, c, \mathcal{E})$4, embedded by a linear layer, concatenated with trajectory embeddings, and provided to both generator and discriminator (Li et al., 2021). Multiclass-SGCN separates velocity and label features through

$p(Y_{t_0+1:t_0+T_p} \mid X, c, \mathcal{E})$5

with distinct learned projections for velocity and one-hot labels before attention and graph convolution (Li et al., 2022). Class-Aware Attention extends conditioning beyond semantics to physical dimensions, using class and object area inside a weighted interaction module and concatenating physical properties to the decoder context (Pathiraja et al., 2022). RTMCT conditions prediction on class in three places: class-specific linear encoders for the observed agent and its neighbors, per-class reference trajectory banks, and per-class output heads, while the transformer itself is shared across classes (He et al., 18 Apr 2025). RESET introduces class-specific trajectory sets, separating non-vulnerable road users from vulnerable road users to inject prior motion constraints (Schmidt et al., 2023).

Other systems derive classes from behavior or structure rather than metadata. Action-based contrastive learning uses pedestrian action labels—“walking” and “standing” for PIE and JAAD, and nine atomic actions for TITAN—to define positive and negative pairs in the embedding space; those labels are used only during training, and inference takes only the observed trajectory (Halawa et al., 2022). S-T CRF computes intention labels from trajectory geometry by thresholding lateral and longitudinal motion in a normalized coordinate frame, so the classes are algorithmically derived rather than manually annotated (Han et al., 2023). The CARLA conditional predictor branches the ego decoder by route intent classes $p(Y_{t_0+1:t_0+T_p} \mid X, c, \mathcal{E})$6, with only the matching branch active at test time (Buhet et al., 2019). Topological forecasting defines classes through h-signatures obtained by ray-crossing words around obstacles, and equal reduced words identify the same homotopy class (Wakulicz et al., 2023). AMEND replaces explicit class labels with latent “pseudo-classes” obtained by K-means clustering in encoder latent space, assigning one cluster to each expert (Mercurius et al., 2024).

This diversity suggests that “class” functions less as a fixed ontology than as a mechanism for introducing discrete structure into motion prediction. Depending on the application, the relevant structure may be semantic, kinematic, topological, intentional, or purely latent.

3. Architectural patterns

The simplest architectural pattern is concatenative conditioning. In the multi-class GAN model, trajectory embeddings and class embeddings are concatenated at every encoding step, and the same conditioning is applied to the discriminator (Li et al., 2021). The small-data study of heterogeneous prediction adopts the same pattern in cRED, cTF, cGAN, and cVAE, where a learned class embedding $p(Y_{t_0+1:t_0+T_p} \mid X, c, \mathcal{E})$7 is concatenated to encoded temporal features prior to decoding, explicitly favoring a single shared model conditioned by embeddings rather than per-class modules (Almeida et al., 4 Oct 2025). The CARLA end-to-mid architecture uses conditional branching rather than concatenation for the ego vehicle: context and ego-history features are shared, but the final ego decoder is selected by the navigation command (Buhet et al., 2019).

A second pattern modifies the interaction model itself. Multiclass-SGCN computes scaled dot-product self-attention on fused velocity-label features, converts these attention maps into high-level interaction features, and then sparsifies them through an Adaptive Interaction Mask. The resulting sparse spatial and temporal graphs are processed by GCN branches in two orders, spatial-to-temporal and temporal-to-spatial, before a TCN predicts a bivariate Gaussian future (Li et al., 2022). Class-Aware Attention replaces standard content-based interaction with structured scalar functions of pairwise distance and area ratio:

$p(Y_{t_0+1:t_0+T_p} \mid X, c, \mathcal{E})$8

and concatenates the two outputs before multimodal decoding (Pathiraja et al., 2022). S-T CRF fuses explicit intention representation with spatial-temporal trajectory features through

$p(Y_{t_0+1:t_0+T_p} \mid X, c, \mathcal{E})$9

so the decoder is directly conditioned on CRF-inferred intention sequences (Han et al., 2023). RTMCT uses a shared 2-layer transformer decoder in which self-attention acts across class-specific mode queries and cross-attention injects neighbor context encoded by class-specific linear layers (He et al., 18 Apr 2025).

A third pattern makes the decoder operate over structured classes rather than regressing coordinates directly. RESET predicts a categorical distribution over a predefined trajectory set, with logits produced from graph-encoded context features and optional ego-plan features (Schmidt et al., 2023). Topological prediction first estimates a homotopy-class posterior with a variable-order Markov process and then conditions a per-class GMM on that posterior (Wakulicz et al., 2023). DROGON predicts an intention distribution over goal zones and conditions a CVAE decoder on that intention, while relational features gate the latent variable via element-wise multiplication (Choi et al., 2019). In all three cases, conditional prediction is mediated by discrete structure, but the structure is defined at different abstraction levels: trajectory exemplars, topological classes, or destination goals.

These architectural families are not mutually exclusive. Several systems combine more than one: RTMCT combines class-specific heads with reference-based multimodality and shared attention; DROGON combines intention classification with latent generative decoding; RESET combines set classification with ego-plan conditioning and class-specific trajectory libraries.

4. Objectives, inference, and uncertainty

Training objectives differ according to how conditioning is operationalized. RESET uses nearest-set assignment by average displacement error and optimizes cross-entropy over trajectory classes:

$X$0

with no reported calibration losses such as NLL, Brier, or ECE (Schmidt et al., 2023). AMEND does not optimize a joint mixture likelihood; instead, each expert is trained with a cluster-weighted loss

$X$1

and the router is trained by cross-entropy on the best-expert target $X$2 (Mercurius et al., 2024). Action-based contrastive learning adds a supervised contrastive regularizer to the trajectory objective,

$X$3

using shared action class to define positives and different actions to define negatives (Halawa et al., 2022).

Multimodal decoders often adopt winner-takes-all supervision. Class-Aware Attention follows the MTP-style objective, selecting the best-matching mode $X$4 and combining classification and regression on that mode (Pathiraja et al., 2022). RTMCT similarly selects the reference mode closest to ground truth and supervises only that prediction with Smooth L1 plus a score loss:

$X$5

with no NLL over mixtures and no explicit negatives in the score loss (He et al., 18 Apr 2025). Multiclass-SGCN instead predicts a bivariate Gaussian per future step and trains by negative log-likelihood over all agents and timesteps (Li et al., 2022). S-T CRF augments trajectory regression with CRF negative log-likelihoods for temporal and spatial intention consistency,

$X$6

while DROGON combines a CVAE objective, intention cross-entropy, and penalties for penetration, inconsistency, and dispersion (Han et al., 2023, Choi et al., 2019).

Inference procedures also vary markedly. RESET outputs $X$7 or $X$8 and can select a flexible top-$X$9 subset directly; endpoint NMS improves diversity without changing runtime dependence on $c$0 except for a small overhead (Schmidt et al., 2023). AMEND uses hard top-1 routing at inference, so only one expert forward pass is executed (Mercurius et al., 2024). S-T CRF performs Viterbi decoding over intention sequences and then decodes a deterministic future conditioned on the inferred intention representation (Han et al., 2023). DROGON either conditions on the most likely intention class or samples intention classes proportionally to $c$1, then samples latent variables to realize multimodality (Choi et al., 2019). Action-based contrastive learning is notable because action labels are not required at inference despite being central to training (Halawa et al., 2022).

A second misconception is that class conditioning necessarily requires class labels at deployment. The literature includes several counterexamples: train-time-only action labels, latent cluster routing, CRF-derived intention states, and class posteriors inferred from the scene itself.

5. Reported empirical behavior

On planner-coupled forecasting, RESET establishes that classification over trajectory sets can remain competitive with regression while improving kinematic feasibility and retaining late-fusion efficiency. On Argoverse 2 validation, RESET with agent-class-specific sets (1000+1000) and NMS reports, for $c$2, minADE $c$3, minFDE $c$4, MR $c$5, and TRI $c$6, whereas the LaneGCN regression baseline reports minADE $c$7, minFDE $c$8, MR $c$9, and TRI $\mathcal{E}$0; the set-based model is therefore slightly worse on Euclidean metrics but better on TRI. Under conditioning on ground-truth $\mathcal{E}$1, Variant 3 reaches $\mathcal{E}$2 and Variant 4 reaches $\mathcal{E}$3 for minADE/minFDE/MR/TRI at $\mathcal{E}$4, while Variant 4 has much lower RCC than Variant 3 (Schmidt et al., 2023).

On long-tail and heterogeneous-agent settings, explicit or implicit class conditioning usually improves tail performance or mixed-class robustness. AMEND improves ETH-UCY tail metrics over a contrastive baseline without explicit class conditioning: on Top 1% difficult scenarios it reports $\mathcal{E}$5 versus $\mathcal{E}$6 for minADE$\mathcal{E}$7/minFDE$\mathcal{E}$8, and for VaR$\mathcal{E}$9 it reports $S=\{\tau_k\}_{k=1}^K$0 versus $S=\{\tau_k\}_{k=1}^K$1 (Mercurius et al., 2024). Multiclass-SGCN reports mADE $S=\{\tau_k\}_{k=1}^K$2 and mFDE $S=\{\tau_k\}_{k=1}^K$3 on SDD, outperforming Semantics-STGCNN at $S=\{\tau_k\}_{k=1}^K$4 and $S=\{\tau_k\}_{k=1}^K$5; removing class labels degrades the model to mADE $S=\{\tau_k\}_{k=1}^K$6 and mFDE $S=\{\tau_k\}_{k=1}^K$7 (Li et al., 2022). The small- and imbalanced-data study further reports that all evaluated methods improve accuracy in most settings when considering class labels, but also finds that deep learning methods perform better on balanced datasets, whereas pattern-based methods may be preferable in cold-start or imbalanced regimes (Almeida et al., 4 Oct 2025).

Behavior- and intention-conditioned systems show comparable gains. Action-based contrastive learning improves egocentric pedestrian prediction over BiTraP on PIE, JAAD, and TITAN; for example, on PIE at $S=\{\tau_k\}_{k=1}^K$8, ADE improves from $S=\{\tau_k\}_{k=1}^K$9 to $p(\tau_k \mid C, P_{\text{ego}})$0, C-ADE from $p(\tau_k \mid C, P_{\text{ego}})$1 to $p(\tau_k \mid C, P_{\text{ego}})$2, and C-FDE from $p(\tau_k \mid C, P_{\text{ego}})$3 to $p(\tau_k \mid C, P_{\text{ego}})$4, and the best reported $p(\tau_k \mid C, P_{\text{ego}})$5 is $p(\tau_k \mid C, P_{\text{ego}})$6 (Halawa et al., 2022). S-T CRF reports average ADE/FDE of $p(\tau_k \mid C, P_{\text{ego}})$7 on ETH/UCY, compared with $p(\tau_k \mid C, P_{\text{ego}})$8 for Social-Implicit, and $p(\tau_k \mid C, P_{\text{ego}})$9 on SDD, compared with $P_{\text{ego}}$0 (Han et al., 2023). DROGON-Prob-20 reaches best-of-20 ADE/FDE of $P_{\text{ego}}$1 at $P_{\text{ego}}$2 on HID, outperforming DROGON-Best-20 at $P_{\text{ego}}$3 and Gated-RN at $P_{\text{ego}}$4 (Choi et al., 2019).

Semantic and physical conditioning also improves multimodal autonomous-driving benchmarks. Class-Aware Attention reports on nuScenes minADE$P_{\text{ego}}$5, minFDE$P_{\text{ego}}$6, missRate$P_{\text{ego}}$7, and Off-Road $P_{\text{ego}}$8, with inference over $P_{\text{ego}}$9 FPS, and ablations show that adding learnable distance attention and area attention improves over simpler attention variants (Pathiraja et al., 2022). RTMCT reports on CODa, for labels-as-input, ADE values of $g \in \mathcal{G}$0 for Pedestrian, $g \in \mathcal{G}$1 for Car, and $g \in \mathcal{G}$2 for Cyclist at $g \in \mathcal{G}$3, and the full perception stack runs at $g \in \mathcal{G}$4 fps on an RTX 3060 with prediction taking $g \in \mathcal{G}$5 ms (He et al., 18 Apr 2025).

Not every conditioning strategy yields uniform gains. In the multi-class GAN study on SDD, adding class labels by concatenated embeddings does not improve ADE/FDE on average, although qualitative improvements appear in scenes where class-specific affordances strongly constrain motion, such as bicyclists in a roundabout (Li et al., 2021). This negative result is significant because it shows that conditioning quality depends on how the class signal interacts with scene semantics, data balance, and decoder capacity.

6. Limitations, controversies, and open directions

The dominant limitations differ by conditioning regime. Discrete trajectory-set methods suffer from discretization error and sensitivity to set quality; RESET explicitly notes that larger sets may improve coverage but can hurt TRI and single-mode accuracy, and that the reported encoder uses restricted inputs such as lane centerlines and displacement vectors rather than richer tracked state or lane attributes (Schmidt et al., 2023). Mixture-of-experts methods face misrouting, expert collapse, and sensitivity to clustering quality; AMEND also notes that hard top-1 routing is brittle when router confidence is miscalibrated (Mercurius et al., 2024). Action-conditioned contrastive learning requires action annotations at training time, even though inference is label-free, and it does not report explicit class-imbalance strategies (Halawa et al., 2022). Sparse graph methods rely on accurate class labels and still incur $g \in \mathcal{G}$6 attention before masking (Li et al., 2022). Topological conditioning assumes accurate obstacle maps and ray geometry; wrong h-signature inference can shift the downstream GMM onto the wrong mode (Wakulicz et al., 2023). Intention-conditioned CRF models derive labels from thresholds, making them sensitive to coordinate normalization and threshold selection (Han et al., 2023).

A separate issue concerns calibration and probability semantics. RESET reports no NLL, Brier, or ECE; RTMCT supervises only the positive mode and does not use an NLL over mixtures; AMEND does not optimize a joint MoE likelihood or load-balancing regularizer (Schmidt et al., 2023, He et al., 18 Apr 2025, Mercurius et al., 2024). This suggests that many class-conditioned systems provide useful ranking or mode selection without fully characterizing calibrated predictive uncertainty.

Efficiency results indicate that conditioning need not preclude real-time use, but the efficient mechanism varies by architecture. RESET reports an average prediction time of $g \in \mathcal{G}$7 ms plus $g \in \mathcal{G}$8 ms for NMS and exploits late fusion to reuse most encoder computation across many ego plans (Schmidt et al., 2023). DROGON reports approximately $g \in \mathcal{G}$9 ms for the first prediction and $$p(Y \mid X)=\sum_{g \in \mathcal{G}} p(Y \mid X,g)\,p(g \mid X),$$0 ms for subsequent predictions in the same scene (Choi et al., 2019). RTMCT uses a shared transformer with class-specific lightweight components and runs prediction in $$p(Y \mid X)=\sum_{g \in \mathcal{G}} p(Y \mid X,g)\,p(g \mid X),$$1 ms inside a $$p(Y \mid X)=\sum_{g \in \mathcal{G}} p(Y \mid X,g)\,p(g \mid X),$$2 fps end-to-end system (He et al., 18 Apr 2025). Class-Aware Attention reports over $$p(Y \mid X)=\sum_{g \in \mathcal{G}} p(Y \mid X,g)\,p(g \mid X),$$3 FPS, whereas the multi-class Social GAN variants require about $$p(Y \mid X)=\sum_{g \in \mathcal{G}} p(Y \mid X,g)\,p(g \mid X),$$4 to converge, and the no-GAN version trains in about $$p(Y \mid X)=\sum_{g \in \mathcal{G}} p(Y \mid X,g)\,p(g \mid X),$$5 (Pathiraja et al., 2022, Li et al., 2021).

Proposed future directions in the surveyed work are comparatively consistent: joint mixture-likelihood training and diversity or load-balancing regularizers for expert routing, richer map encoders and multi-agent transformers, semi-supervised scenario labeling to align latent clusters with actionable semantics, uncertainty calibration, and dynamics-based class discovery for cases where manual labels are coarse or noisy (Mercurius et al., 2024, Almeida et al., 4 Oct 2025). A plausible implication is that future systems will increasingly treat class-conditioned prediction as a design space rather than a single method family: explicit labels, learned pseudo-classes, topological signatures, and planner-induced trajectory classes are all viable conditioning interfaces, and their relative utility depends on the balance between data scale, scene structure, and deployment constraints.