LTMSformer: Lightweight Trajectory Transformer

• The paper introduces LTMSformer, which extends HiVT by incorporating Local Trend-Aware Attention, Motion State Encoding, and a Lightweight Proposal Refinement Module for accurate multi-agent trajectory prediction.
• It achieves a superior accuracy–efficiency tradeoff on Argoverse 1 by improving minADE, minFDE, and MR compared to HiVT-64 while reducing model size relative to HiVT-128.
• The model employs a two-stage training process with causal convolution, local-box attention, and MLP-based residual refinement, ensuring parameter efficiency and robust trajectory forecasts.

Searching arXiv for LTMSformer and closely related trajectory-prediction papers to ground the article. LTMSformer is a lightweight transformer-based framework for multi-agent, multi-modal trajectory prediction in autonomous driving. It is introduced in “LTMSformer: A Local Trend-Aware Attention and Motion State Encoding Transformer for Multi-Agent Trajectory Prediction” and is formulated as an extension of HiVT for forecasting multiple plausible future trajectories from observed agent histories and lane/map context (Yan et al., 7 Jul 2025). Its design centers on three additions to the HiVT family: Local Trend-Aware Attention (LTAA) for local temporal dependency, a Motion State Encoder for higher-order interaction cues, and a Lightweight Proposal Refinement Module (LPRM) for second-stage trajectory refinement. The reported empirical emphasis is an accuracy–efficiency tradeoff: LTMSformer improves over HiVT-64 on Argoverse 1 while remaining substantially smaller than HiVT-128.

1. Definition and problem setting

LTMSformer addresses multi-agent trajectory prediction under multimodal uncertainty. The input comprises observed trajectories of multiple agents together with lane segment embeddings or map features, and the output is a set of candidate future trajectories. The paper defines observed trajectories as a collection of agent coordinates over observed time steps and predicted future trajectories over future time steps. To obtain translation-invariant local representations, it uses vectorized relative displacements

$$\Delta \mathbf{p}_i^t = \mathbf{p}_i^t - \mathbf{p}_i^{t-1}, \qquad \Delta \mathbf{p}_{ij}^t = \Delta \mathbf{p}_j^t - \Delta \mathbf{p}_i^t,$$

followed by an agent-centric rotation

$$\mathbf{h}_i^t=\mathbf{R}_i^{T_o}\Delta \mathbf{p}_i^t,\qquad \mathbf{h}_{ij}^t=\mathbf{R}_i^{T_o}\Delta \mathbf{p}_{ij}^t.$$

The stated motivation is twofold. First, most prior methods are said to underemphasize local temporal dependency, even though each motion state is closely related to adjacent time steps. Second, prior interaction modeling often relies heavily on relative position while underusing higher-order motion attributes such as acceleration, jerk, and heading. LTMSformer is designed to encode both forms of structure: local temporal evolution within an agent’s history and richer spatial interaction cues across agents.

The benchmark used in the paper is Argoverse 1 motion prediction. The dataset statistics reported are 323,557 driving scenarios, each lasting 5 seconds, with 2D bird’s-eye-view trajectories sampled at 10 Hz and HD maps including lane centerlines, traffic directions, and intersection annotations. The split is 205,942 training scenes, 39,472 validation scenes, and 78,143 test scenes. Evaluation follows the Argoverse 1 convention of observing 2 seconds and predicting the next 3 seconds, with six candidate trajectories per agent and the metrics minADE, minFDE, and MR (Yan et al., 7 Jul 2025).

2. Architectural organization

LTMSformer is described as a two-stage lightweight trajectory prediction framework. Its top-level pipeline is organized as: a Local Temporal-Spatial Encoder, a Global Interaction module, a Multi-modal Decoder, and an LPRM second stage. Within the Local Temporal-Spatial Encoder, the paper lists four subcomponents: Agent-Agent Encoder, LTAA, Motion State Encoder, and Agent-Lane Encoder.

The Agent-Agent Encoder embeds center-agent and neighbor features and computes per-time-step spatial interaction with multi-head attention: $$\mathbf{s}_i^t = \phi_{\text{center}}(\mathbf{h}_i^t),\qquad \mathbf{s}_{ij}^t = \phi_{\text{nbr}}([\mathbf{h}_j^t,\mathbf{h}_{ij}^t]),$$

$$\mathbf{c}_i^t = \mathrm{MHA}(\mathbf{s}_i^t,\mathbf{s}_{ij}^t).$$

These features are stacked temporally into a sequence representation passed to LTAA.

The Agent-Lane Encoder integrates lane context after temporal and interaction encoding. Lane segments are embedded in the same local coordinate system, and local agent–lane interaction yields a local scene representation. A separate Global Interaction block then aggregates cross-agent social context using pairwise relative motion and heading-difference features. The paper writes the global interaction embedding as

$$\mathbf{g}_{ij}=\phi_{\text{global}}\left([\mathbf{h}_{ij}^{T_o},\cos(\Delta\theta_{ij}),\sin(\Delta\theta_{ij})]\right).$$

The Multi-modal Decoder produces, for each mode, future position parameters, scale parameters, and mode probability: $$\{(\hat{\mu}_{i,m},\hat{\beta}_{i,m}),\hat{\pi}_{i,m}\}_{m=1}^M = \phi_d([\psi_i^{\text{local}},\psi_i^{\text{global}}]).$$ The stage-one output is expressed as a mixture of Laplace distributions: $$\hat{\mathbf{Y}}_i^{1} = \sum_{m=1}^M \hat{\pi}_{i,m}\,\mathrm{Laplace}(\hat{\mu}_{i,m},\hat{\beta}_{i,m}).$$

This architecture is explicitly positioned as being built on top of HiVT rather than as a completely independent backbone. A plausible implication is that LTMSformer should be understood less as a replacement for the HiVT paradigm than as a targeted reconfiguration of its temporal encoding, interaction representation, and refinement stages.

3. Local Trend-Aware Attention

LTAA is the paper’s principal temporal module. Its purpose is to model local temporal dependency more explicitly than standard global self-attention. The method combines causal temporal convolution for query and key construction, local-box attention that restricts attention to nearby temporal windows, and hierarchical box sizes for multi-scale temporal context (Yan et al., 7 Jul 2025).

The paper computes queries and keys by causal convolution plus BatchNorm, and values by linear projection: $$\mathbf{Q}_i^{\text{LTAA}} = \mathrm{BatchNorm}(\mathrm{conv}(\mathbf{z}_i)),\qquad \mathbf{K}_i^{\text{LTAA}} = \mathrm{BatchNorm}(\mathrm{conv}(\mathbf{z}_i)),\qquad \mathbf{V}_i^{\text{LTAA}} = \mathbf{W}_V^{\text{LTAA}}\mathbf{z}_i.$$ Attention is then performed within non-overlapping local boxes rather than over the full temporal sequence: $$\mathbf{b}_i^{u_v} = \mathrm{LMHA}(\mathbf{Q}_i^{\text{LTAA}},\mathbf{K}_i^{\text{LTAA}},\mathbf{V}_i^{\text{LTAA}}).$$ Three cascaded layers with progressively larger box sizes are used, and for Argoverse 1 the implementation specifies

$B=\{3,7,21\}.$

The final representation is

$$\mathbf{h}_i^t=\mathbf{R}_i^{T_o}\Delta \mathbf{p}_i^t,\qquad \mathbf{h}_{ij}^t=\mathbf{R}_i^{T_o}\Delta \mathbf{p}_{ij}^t.$$0

Conceptually, LTAA differs from standard self-attention in three respects. It privileges adjacent-time correlations, it derives keys and queries from short-term motion trends rather than raw states, and it expands temporal receptive field hierarchically without reverting to unrestricted global attention. The paper further states that LTAA reduces parameter and computation complexity relative to standard MHA by restricting attention to localized matrices, although it does not provide a full complexity derivation for LTAA itself.

The reported implementation details are three LTAA layers, local box sizes $$\mathbf{h}_i^t=\mathbf{R}_i^{T_o}\Delta \mathbf{p}_i^t,\qquad \mathbf{h}_{ij}^t=\mathbf{R}_i^{T_o}\Delta \mathbf{p}_{ij}^t.$$1, GELU in the feedforward layer, 64 hidden units for the base model, and dropout 0.1. The paper does not specify the exact convolution kernel size, the exact number of attention heads inside LTAA, or the precise cascade operator.

4. Motion-state encoding and lightweight refinement

The Motion State Encoder introduces higher-order motion attributes into interaction modeling. The paper defines the motion-state vector as

$$\mathbf{h}_i^t=\mathbf{R}_i^{T_o}\Delta \mathbf{p}_i^t,\qquad \mathbf{h}_{ij}^t=\mathbf{R}_i^{T_o}\Delta \mathbf{p}_{ij}^t.$$2

where the explicitly identified attributes are relative positions, acceleration, jerk, and heading. These features are embedded and fused with the LTAA output through multi-head attention: $$\mathbf{h}_i^t=\mathbf{R}_i^{T_o}\Delta \mathbf{p}_i^t,\qquad \mathbf{h}_{ij}^t=\mathbf{R}_i^{T_o}\Delta \mathbf{p}_{ij}^t.$$3

The claimed role of this module is to distinguish interaction contexts that may be geometrically similar but dynamically different. Two neighboring vehicles can occupy comparable relative positions yet imply different future interactions if one is accelerating, one is braking, or their headings diverge. The paper argues that such high-order motion state attributes better expose intent and dynamic state than position-only interaction features.

Trajectory refinement is performed by the Lightweight Proposal Refinement Module. The LPRM is explicitly MLP-only: no recurrent refinement, no additional attention stack, and no iterative proposal update mechanism are introduced. The stage-one proposal is embedded as

$$\mathbf{h}_i^t=\mathbf{R}_i^{T_o}\Delta \mathbf{p}_i^t,\qquad \mathbf{h}_{ij}^t=\mathbf{R}_i^{T_o}\Delta \mathbf{p}_{ij}^t.$$4

and a consistency branch processes the concatenation of observed history and stage-one proposal. The final refined trajectory is produced residually: $$\mathbf{h}_i^t=\mathbf{R}_i^{T_o}\Delta \mathbf{p}_i^t,\qquad \mathbf{h}_{ij}^t=\mathbf{R}_i^{T_o}\Delta \mathbf{p}_{ij}^t.$$5

The paper attributes much of the parameter efficiency of LTMSformer to this refinement design. This suggests that the model’s lightweight character depends not only on local attention in LTAA but also on replacing heavier proposal-refinement machinery with shallow MLP-based residual correction.

5. Optimization, benchmark protocol, and empirical performance

Training uses two-stage supervision. The total loss is

$$\mathbf{h}_i^t=\mathbf{R}_i^{T_o}\Delta \mathbf{p}_i^t,\qquad \mathbf{h}_{ij}^t=\mathbf{R}_i^{T_o}\Delta \mathbf{p}_{ij}^t.$$6

with

$$\mathbf{h}_i^t=\mathbf{R}_i^{T_o}\Delta \mathbf{p}_i^t,\qquad \mathbf{h}_{ij}^t=\mathbf{R}_i^{T_o}\Delta \mathbf{p}_{ij}^t.$$7

and the second-stage refinement loss defined as

$$\mathbf{h}_i^t=\mathbf{R}_i^{T_o}\Delta \mathbf{p}_i^t,\qquad \mathbf{h}_{ij}^t=\mathbf{R}_i^{T_o}\Delta \mathbf{p}_{ij}^t.$$8

The sensitivity study reports best performance at $$\mathbf{h}_i^t=\mathbf{R}_i^{T_o}\Delta \mathbf{p}_i^t,\qquad \mathbf{h}_{ij}^t=\mathbf{R}_i^{T_o}\Delta \mathbf{p}_{ij}^t.$$9. Optimization settings are RTX 4090, 64 epochs, AdamW, batch size 32, initial learning rate $$\mathbf{s}_i^t = \phi_{\text{center}}(\mathbf{h}_i^t),\qquad \mathbf{s}_{ij}^t = \phi_{\text{nbr}}([\mathbf{h}_j^t,\mathbf{h}_{ij}^t]),$$0, weight decay $$\mathbf{s}_i^t = \phi_{\text{center}}(\mathbf{h}_i^t),\qquad \mathbf{s}_{ij}^t = \phi_{\text{nbr}}([\mathbf{h}_j^t,\mathbf{h}_{ij}^t]),$$1, dropout 0.1, and cosine annealing. The model uses $$\mathbf{s}_i^t = \phi_{\text{center}}(\mathbf{h}_i^t),\qquad \mathbf{s}_{ij}^t = \phi_{\text{nbr}}([\mathbf{h}_j^t,\mathbf{h}_{ij}^t]),$$2 predicted modes, a local region radius of 50 m, and explicitly reports no data augmentation, no ensembling, and no extra tricks (Yan et al., 7 Jul 2025).

On Argoverse 1 validation, the paper reports the following comparison: HiVT-64 achieves minADE 0.69, minFDE 1.03, MR 0.10 with 662k parameters; HiVT-128 achieves 0.66, 0.96, 0.09 with 2529k parameters; LTMSformer achieves 0.66, 0.94, 0.08 with 789k parameters. On the test set, HiVT-64 reports 0.8306, 1.3053, 0.1503; HiVT-128 reports 0.7993, 1.2320, 0.1369; LTMSformer reports 0.7993, 1.1932, 0.1254. The paper emphasizes two comparative statements: approximately 4.35%, 8.74%, and 20% improvement over HiVT-64 on validation for minADE, minFDE, and MR, respectively, and a 68% reduction in model size relative to HiVT-128 while tying its test minADE and improving its test minFDE and MR.

The ablation study presents an incremental path from base HiVT-64 to the full model. The base configuration $$\mathbf{s}_i^t = \phi_{\text{center}}(\mathbf{h}_i^t),\qquad \mathbf{s}_{ij}^t = \phi_{\text{nbr}}([\mathbf{h}_j^t,\mathbf{h}_{ij}^t]),$$3 reports 0.687 / 1.030 / 0.103. Adding the Motion State Encoder yields $$\mathbf{s}_i^t = \phi_{\text{center}}(\mathbf{h}_i^t),\qquad \mathbf{s}_{ij}^t = \phi_{\text{nbr}}([\mathbf{h}_j^t,\mathbf{h}_{ij}^t]),$$4 at 0.673 / 1.001 / 0.098. Adding LTAA yields $$\mathbf{s}_i^t = \phi_{\text{center}}(\mathbf{h}_i^t),\qquad \mathbf{s}_{ij}^t = \phi_{\text{nbr}}([\mathbf{h}_j^t,\mathbf{h}_{ij}^t]),$$5 at 0.672 / 1.000 / 0.096. The full LTMSformer $$\mathbf{s}_i^t = \phi_{\text{center}}(\mathbf{h}_i^t),\qquad \mathbf{s}_{ij}^t = \phi_{\text{nbr}}([\mathbf{h}_j^t,\mathbf{h}_{ij}^t]),$$6 reports 0.661 / 0.945 / 0.082. The paper interprets these results as showing that the Motion State Encoder provides the largest initial gain, LTAA adds further improvement, and LPRM yields the largest final jump in minFDE and MR.

6. Position in the literature and naming distinctions

LTMSformer belongs to the autonomous-driving trajectory-prediction literature rather than the time-series forecasting, lesion tracking, or latent-language-model literatures. This distinction matters because several recent arXiv papers have superficially similar acronyms while addressing different tasks. “LTM” in “A Time Series Multitask Framework Integrating a LLM, Pre-Trained Time Series Model, and Knowledge Graph” is a multitask time-series framework for forecasting, imputation, and anomaly detection, not LTMSformer (Hao et al., 10 Mar 2025). “LMFormer” is a lane-based motion prediction transformer with a different architecture and problem decomposition (Yadav et al., 14 Apr 2025). “Transformer Lesion Tracker” concerns longitudinal lesion correspondence in CT volumes (Tang et al., 2022). “Latent Thought Models” concerns latent-variable language modeling with variational Bayes inference-time computation (Kong et al., 3 Feb 2025).

Within motion forecasting specifically, LTMSformer is most directly positioned against HiVT, because the paper explicitly states that it is built on top of HiVT and evaluates primarily against HiVT-64 and HiVT-128. Its distinctive contribution is not a wholesale redesign of the forecasting pipeline, but a focused replacement of global temporal encoding by hierarchical local trend-aware attention, a richer motion-state representation for interaction modeling, and an MLP-only proposal-refinement stage. A plausible implication is that LTMSformer’s significance lies in showing that carefully chosen temporal inductive bias and lightweight refinement can improve the accuracy–parameter tradeoff without resorting to a larger transformer backbone.