Neural Keypoint Trajectories

• Neural keypoint trajectories are temporally ordered spatial sequences of annotated points (e.g., joints or pixels) that enable dynamic prediction and semantic inference.
• Advanced neural architectures like spatio-temporal graph networks, implicit neural fields, and recurrent units drive robust forecasting and occlusion-aware tracking.
• Empirical benchmarks confirm improved accuracy in action recognition, motion prediction, and dense scene reconstruction across diverse modalities.

Neural key point trajectories are temporally ordered sequences of spatial locations for labeled points, such as joints in human skeletons, annotated pixels, or learned landmarks, whose neuralized (learned) modeling enables direct prediction, dynamic analysis, and semantic inference in both natural and artificial systems. Modern approaches combine high-dimensional input features (e.g., 3D keypoints from sensor data or pixel coordinates in images) with expressive neural architectures, such as spatiotemporal graph neural networks, implicit neural fields, or iterative feature matchers, to not only forecast future locations but also infer latent structure, decision boundaries, and occlusion-aware embeddings. This article systematically examines the principal methodologies, loss functions, empirically validated benchmarks, and conceptual extensions underlying neural key point trajectory research.

1. Mathematical Foundations and Problem Setting

Neural key point trajectories generalize classical trajectory modeling by representing each tracked entity as a sequence of observations $\{\mathbf{k}_t^p\}_{t=1}^T$, with each $\mathbf{k}_t^p \in \mathbb{R}^D$, where $D$ is typically 2 (image) or 3 (world coordinates) and $p$ indexes the keypoints. Multiple papers formalize this setting with stacked tensors, such as $\mathbf{K} \in \mathbb{R}^{T \times P \times D}$, and employ structured encoders to process temporally and spatially correlated measurements (Li et al., 2023, Fan et al., 2019). The key foundational task involves learning a function $f_\theta$ (with parameters $\theta$) that can, given historical data and possibly context, predict future keypoint arrangements, recognize activities or actions, or infer the underlying dynamical laws governing these trajectories.

Table: Core Problem Formulations Across Papers

Paper (arXiv ID)	Input Structure	Output Target(s)
(Li et al., 2023)	$\mathbf{K}\in\mathbb{R}^{T_h\times P\times 3}$, context	Binary action + future trajectory
(Fan et al., 2019)	$(\boldsymbol{P}_t,\boldsymbol{X}_t)$ sets over $n$ points	Future point coordinates
(Zhang et al., 5 Jun 2024)	$(x\in\mathbb{R}^3, t)$ tuples	Displacement field $\Delta x$
(Harley et al., 2022)	$(x_1,y_1)$, video frames of $T$ steps	$(x_t, y_t)_{t=1}^T$, visibility

These setups support both supervised (given correspondence/labels) and semi/self-supervised learning with auxiliary or contrastive signals.

2. Neural Architectures for Keypoint Trajectories

The design of neural encoders and decoders for keypoint sequences centers on exploiting both the spatial structure (e.g., body skeleton graphs, pixel neighborhoods, point cloud topology) and temporal evolution.

• Spatio-Temporal Graph Convolutional Networks (ST-GCNs) encode pose/motion by modeling joints as nodes, bones as edges, and connecting frames with temporal links. A typical layer computes:

$$H^{\ell+1}_{t,\cdot,:} = \sigma\left(\sum_{k=1}^{K_s}\sum_{m=1}^{K_t} A_s^{(k)} H^{\ell}_{t-m+1,\cdot,:} W^{\ell}_{k,m}\right)$$

This architecture is used for pedestrian 3D keypoints, aggregating both spatial and temporal context before downstream heads for action recognition and path prediction (Li et al., 2023).

• Implicit Neural Fields (e.g., SIREN models) embed motion dynamics as continuous mappings $f_\theta(x, t)$, allowing interpolation and extrapolation across space and time. DOMA-Affinity, for example, produces smooth displacement fields via periodic activation MLPs, outputting affine transforms $A_\theta(x, t)$ and translation vectors $u_\theta(x, t)$ (Zhang et al., 5 Jun 2024).
• Orderless Point Recurrent Units (PointRNN/GRU/LSTM) adapt standard RNN update logic to unordered point clouds, aligning states across time via local $k$-NN or ball queries:

$$\boldsymbol S_t^i = \mathrm{pool}\left\{ \boldsymbol W [\boldsymbol X_t^i,\, \boldsymbol S_{t-1}^j,\, \boldsymbol P_t^i - \boldsymbol P_{t-1}^j] + \boldsymbol b \right\}_{j \in \mathcal{N}(\boldsymbol P_t^i)}$$

This order-invariant design suits generic motion prediction for both synthetic and LiDAR-derived data (Fan et al., 2019).

• Pixel Trajectory Iterative Propagation (PIPs) for video tracking leverages learned cost maps and Mixer-based update blocks to refine per-pixel trajectories even across occlusions. Each iteration iteratively updates position and appearance features (Harley et al., 2022).

3. Training Objectives and Auxiliary Tasks

Robust keypoint trajectory learning incorporates both direct regression/classification heads and strategically designed auxiliary tasks:

• Crossing Action and Multi-modal Trajectory Heads: Fused embeddings ($[\mathbf{h}_C;\mathbf{h}_T;\mathbf{h}_K]$) are used for both binary cross-entropy classification and MLP-based trajectory regression, the latter employing minimum-ADE among multiple hypotheses (Li et al., 2023).
• Auxiliary Supervision:
  • Jigsaw Puzzle: Predict permutation of temporally shuffled keypoint segments, optimizing:

  $$\mathcal{L}_{\rm KJP} = -\frac{1}{N} \sum_{i=1}^{N} \log \frac{\exp(\ell_i[y_i])}{\sum_{j=1}^{S!} \exp(\ell_i[j])}$$ - Next-frame Prediction: Direct regression of future keypoints from embeddings. - Contrastive Representation Learning: InfoNCE loss between augmented views of the sequence, based on cosine similarity. - Smoothness Regularization: In DOMA,

  $$R_{\rm smooth} = \mathbb{E}_{x, t}\left[\Psi\left(\|\nabla A(x, t)\|_F^2 + \|\nabla u(x, t)\|_F^2\right)\right]$$

  enforces piece-wise regular motion fields.

• Loss Aggregation: Weighted sum of all component losses; keypoint-based auxiliaries typically receive lower weights to avoid distracting from main objectives (Li et al., 2023).

4. Empirical Benchmarks and Quantitative Results

Empirical validation leverages both synthetic datasets and large-scale driving or activity benchmarks:

• Pedestrian Crossing and Trajectory (Li et al., 2023):

  • Crossing accuracy: baseline TNT+AR (95.9%) vs neural keypoint model (96.3%); minADE$_6$: 0.418 m → 0.382 m.
  • Auxiliary task ablation: Jigsaw, prediction, and contrastive learning individually boost accuracy and reduce minADE.
• Point Cloud Motion Prediction (Fan et al., 2019):
  • Moving MNIST (synthetic): PointLSTM yields CD=1.16, EMD=1.78, outperforming voxel-based ConvLSTM, CubicLSTM, PointNet+++LSTM, and PointCNN+ConvLSTM.
  • Argoverse/nuScenes (real data): PointRNN variants deliver lower CD/EMD than point-wise copy-last or global-feature LSTM baselines.
• Occlusion-Aware Pixel Tracking (Harley et al., 2022):
  • FlyingThings++: PIPs achieves lower trajectory error in visible (15.5 px) and occluded (36.7 px) frames versus DINO and RAFT.
  • CROHD, KITTI, BADJA: Consistently improved keypoint propagation metrics, particularly under long occlusions.
• Dense 3D Motion Field Inference (Zhang et al., 5 Jun 2024):
  • DeformingThings4D: DOMA-Affinity EPE of $7.8\times 10^{-3}$ outperforms prior models.
  • Guided mesh alignment: DOMA achieves comparable Chamfer distance but significantly lower temporal irregularity.

5. Dynamical Interpretation and Theoretical Analysis

Some approaches seek not only accurate prediction but also interpretability of underlying dynamics via phase portrait extraction and fixed-point/bifurcation analysis (Zhao et al., 2016):

• Continuous-time Models for neural systems posit:

$\dot{x} = F(x, u) = F_x(x) + F_u(x)u$

with basis function parameterizations enforcing contraction to finite regions and enabling explicit analysis for attractors, saddles, slow points, and bifurcations.

• Fixed Point and Slow Point Extraction proceeds via Newton-Raphson and speed minimization, with stability classified by Jacobian eigenvalues:
  • Negative real parts indicate attractors.
  • Zero crossings or changes signal bifurcations (e.g., saddle-node or pitchfork).
• Empirical Phase Portraits: Applications include decision-making networks, oscillatory systems, continuous attractor codes (ring attractors), and chaotic flows.

This interpretable view connects trajectory modeling to broader computational principles, such as integration, categorical choice, memory, and chaos in both neural and artificial systems.

6. Applications and Extensions

Neural key point trajectories form foundational building blocks in diverse domains:

• Autonomous Driving: Pedestrian intent recognition and future motion forecasting; multi-agent context integration (Li et al., 2023).
• Scene Reconstruction and Tracking: Dense temporal correspondence for mesh alignment, novel point displacement prediction, and avatar creation (Zhang et al., 5 Jun 2024).
• Occlusion-Robust Video Analysis: Keypoint propagation through occlusions for object, animal, and crowd tracking (Harley et al., 2022).
• Neuroscience: Extraction of latent dynamical motifs underlying cognitive computations, motor control, and memory (Zhao et al., 2016).
• Generic Point Clouds: Spatially unordered moving point sets for robotics, sensors, and simulation (Fan et al., 2019).

Extensions discussed include uncertainty-aware dynamics, generative field models, fluid/cloth motion fields, and self-supervised pretraining for unlabelled sequences through contrastive and permutation-based objectives.

7. Limitations and Future Directions

Limitations include:

• Fixed neighborhood strategies for local association (PointRNN) may be improved via learned adaptive weights (Fan et al., 2019).
• Current models often deterministic; probabilistic or uncertainty quantification remains underexplored.
• Effective weighting of multi-task losses is non-trivial and typically dataset-dependent (Li et al., 2023).
• Reliance on labeled correspondences or guidance points (DOMA) can limit generalization in sparse or noisy scenarios (Zhang et al., 5 Jun 2024).
• Metrics often focus on geometric overlap rather than explicit identity-preserving error; there is scope for advanced evaluation (ADE/FDE) for persistent keypoints.

Potential future directions comprise spatiotemporal neural fields for continuous, uncertainty-aware motion modeling, advances in self-supervised representation learning, and domain adaptation for generalization across environments and modalities.

---

Neural key point trajectories unite spatial reasoning, temporal prediction, and semantic inference through principled neural architectures, rigorous loss construction, and interpretive dynamical analysis. The surveyed methodologies offer substantial accuracy and robustness gains while supporting both prediction and interpretative goals in major scientific and engineering domains.