Trajectory-Language Mapping Overview

• Trajectory-language mapping is a cross-modal design space that aligns motion trajectories with linguistic expressions through generation, retrieval, captioning, and localization.
• Systems leverage techniques like cross-attention, shared embedding spaces, and language as a latent intermediary to fuse spatiotemporal and semantic data.
• Applications span robotics, urban mobility, video analysis, and simulation, while challenges include real-time processing, data efficiency, and balancing flexibility with formal guarantees.

Searching arXiv for recent and foundational papers on trajectory–language mappings across video-language, robotics, prediction, and simulation. Trajectory-language mapping denotes the set of formulations in which trajectories and natural-language expressions are aligned, translated, or embedded into a shared representational space. In the cited literature, the trajectory side ranges from object tubes in video, robot waypoint sequences, multi-agent traffic rollouts, sparse GPS traces, human attention paths, and body-joint motion trajectories; the language side ranges from captions, retrieval queries, and low-level instructions to latent token sequences and spoken commands. Representative formulations include direct language-to-trajectory generation $g:L\rightarrow T$, instruction-conditioned prediction $\hat T=\arg\max_T P(T\mid L,(r_0,t_0))$, segment localization $f:(E_q,E_t)\rightarrow\{(\tau_k^s,\tau_k^e)\}$, and trajectory captioning $f_{\mathrm{cap}}:T\mapsto\hat L$ (Xia et al., 2024, Li et al., 11 May 2026, Raj et al., 2023).

1. Problem regimes and conceptual scope

The literature does not treat trajectory-language mapping as a single task. Instead, it instantiates several distinct but related regimes. Some works map language into trajectories or trajectory modifications, as in interactive traffic generation, robot trajectory reshaping, speech-to-trajectory control, and instruction-conditioned urban mobility generation (Xia et al., 2024, Bucker et al., 2022, Bamani et al., 7 Apr 2025, Li et al., 11 May 2026). Others map trajectories into language, either by generating captions of motion or by using language as an interpretable intermediate representation for prediction (Kuo et al., 2021, Li et al., 11 May 2026). A third group learns cross-modal alignment or localization, such as matching a word to an object tube in video, retrieving trajectories from text queries, or identifying a language-specified sub-segment of a long robot trajectory (Yang et al., 2023, Galoaa et al., 11 Dec 2025, Raj et al., 2023).

This diversity is reflected in explicit formalizations. InteractTraj defines a two-stage generator $g(L):=\mathcal D(\mathcal E(L))$ from language to interactive trajectories in $\mathbb R^{N\times T\times 2}$ (Xia et al., 2024). Raj et al. formulate language-conditioned change-point detection as prediction of start and end times $(\tau_k^s,\tau_k^e)$ for the sub-trajectory corresponding to each low-level instruction (Raj et al., 2023). TrajPrism separates instruction-conditioned trajectory generation, language-driven semantic trajectory retrieval, and trajectory captioning into three benchmark tasks with distinct objectives and metrics (Li et al., 11 May 2026).

A useful summary is that the field spans alignment, generation, retrieval, captioning, recovery, and control, rather than only “language-conditioned planning.” This suggests that “trajectory-language mapping” is best understood as a cross-modal design space whose unifying concern is verifiable correspondence between spatiotemporal structure and linguistic semantics.

Regime	Representative formulation	Example papers
Generation / adaptation	$g:L\rightarrow T$ or $$f_\theta(\xi_o,L,\{O_i\})\rightarrow \xi_{\mathrm{mod}}$$	(Xia et al., 2024, Bucker et al., 2022)
Localization / alignment	$f:(E_q,E_t)\rightarrow\{(\tau_k^s,\tau_k^e)\}$	(Raj et al., 2023)
Retrieval	$\hat T=\arg\max_T P(T\mid L,(r_0,t_0))$0	(Li et al., 11 May 2026)
Captioning	$\hat T=\arg\max_T P(T\mid L,(r_0,t_0))$1	(Li et al., 11 May 2026)
Promptized forecasting	trajectory as a text prompt	(Bae et al., 2024, Luo et al., 29 Jan 2025)

2. Trajectory representations and linguistic encodings

A central design choice is how a trajectory is represented before cross-modal fusion. In video-language grounding, TW-BERT replaces patch-centric aggregation with explicit object trajectories. For each query word embedding $\hat T=\arg\max_T P(T\mid L,(r_0,t_0))$2 and frame $\hat T=\arg\max_T P(T\mid L,(r_0,t_0))$3 with patch tokens $\hat T=\arg\max_T P(T\mid L,(r_0,t_0))$4, it computes the per-frame salient patch

$\hat T=\arg\max_T P(T\mid L,(r_0,t_0))$5

stacks the sequence

$\hat T=\arg\max_T P(T\mid L,(r_0,t_0))$6

and then performs trajectory fusion

$\hat T=\arg\max_T P(T\mid L,(r_0,t_0))$7

This representation treats an object as a “small spatiotemporal tube” rather than a single 2D patch (Yang et al., 2023).

TraceVision adopts a different notion of trajectory. Its input is a raw eye-gaze sequence $\hat T=\arg\max_T P(T\mid L,(r_0,t_0))$8 with $\hat T=\arg\max_T P(T\mid L,(r_0,t_0))$9, which is converted into semantically meaningful keypoints by a semantic-guided Douglas–Peucker procedure. Phrase-aligned segments receive importance scores $f:(E_q,E_t)\rightarrow\{(\tau_k^s,\tau_k^e)\}$0, normalized to $f:(E_q,E_t)\rightarrow\{(\tau_k^s,\tau_k^e)\}$1, and local tolerance is set to $f:(E_q,E_t)\rightarrow\{(\tau_k^s,\tau_k^e)\}$2 with $f:(E_q,E_t)\rightarrow\{(\tau_k^s,\tau_k^e)\}$3. The simplified trajectory $f:(E_q,E_t)\rightarrow\{(\tau_k^s,\tau_k^e)\}$4 retains $f:(E_q,E_t)\rightarrow\{(\tau_k^s,\tau_k^e)\}$5–$f:(E_q,E_t)\rightarrow\{(\tau_k^s,\tau_k^e)\}$6 of the original points, approximately $f:(E_q,E_t)\rightarrow\{(\tau_k^s,\tau_k^e)\}$7 versus $f:(E_q,E_t)\rightarrow\{(\tau_k^s,\tau_k^e)\}$8, while the Trajectory-aware Visual Perception module fuses visual tokens $f:(E_q,E_t)\rightarrow\{(\tau_k^s,\tau_k^e)\}$9 and trajectory embeddings $f_{\mathrm{cap}}:T\mapsto\hat L$0 by bidirectional cross-attention (Yang et al., 23 Feb 2026).

Other systems textualize trajectories directly. LMTraj rounds 2D coordinates to two decimal places, rewrites each trajectory as a string such as [(0.34,-1.27), ...], wraps the sequence in English sentences, and augments it with BLIP-2 scene captions and auxiliary Q&A templates for destination, direction, similar-pattern search, group membership, and collision risk (Bae et al., 2024). The flight-trajectory study similarly converts ADS-B waypoints $f_{\mathrm{cap}}:T\mapsto\hat L$1 into prompt text and fine-tunes LLaMA-3.1-8B with standard tokenizer splits over the numeric strings (Luo et al., 29 Jan 2025). PLMTrajRec combines explicit natural-language prompts about the sampling interval, start time, day of week, total time cost, and total space transfer distance with a uniformized sequence $f_{\mathrm{cap}}:T\mapsto\hat L$2 that inserts placeholder [m] tokens for missing points; observed points receive Fourier and road-network embeddings, whereas masked points use time-diffused area-flow features (Wei et al., 2024).

A third family uses discrete latent motion tokens rather than literal numeric text. TLControl partitions the skeleton into Head, Left arm, Right arm, Left leg, Right leg, and Root, learns separate codebooks, and quantizes per-group features into VQ-VAE tokens that are then predicted by a Masked Trajectories Transformer conditioned on CLIP text features and partial trajectory tokens (Wan et al., 2023). Lang2Motion aligns raw point trajectories, rendered overlay frames, and natural-language motion descriptions in a shared $f_{\mathrm{cap}}:T\mapsto\hat L$3 CLIP-based latent space, using a 4-layer Transformer encoder and an autoregressive decoder over frame-to-frame displacements (Galoaa et al., 11 Dec 2025).

3. Alignment mechanisms and training objectives

The mapping between trajectories and language is implemented by several recurrent architectural patterns. Cross-attention is the most explicit. TW-BERT is asymmetric because word-to-patch and trajectory-to-word use different $f_{\mathrm{cap}}:T\mapsto\hat L$4 orderings, so its cross-modal encoder interleaves a W2P block for vision-to-language and a T2W block for language-to-vision; it complements this with a Hierarchical Frame-Selector inserted at the 6th and 12th layers of a 12-layer TimeSformer to prune frames conditioned on text (Yang et al., 2023). TraceVision’s TVP block alternates Trajectory-Aware Visual Enhancement,

$f_{\mathrm{cap}}:T\mapsto\hat L$5

and Visually-Informed Trajectory Refinement,

$f_{\mathrm{cap}}:T\mapsto\hat L$6

thereby refining both modalities bidirectionally before decoding language and segmentation masks (Yang et al., 23 Feb 2026).

A second pattern is shared embedding and contrastive alignment. Lang2Motion uses frozen CLIP text and vision encoders to obtain $f_{\mathrm{cap}}:T\mapsto\hat L$7 and $f_{\mathrm{cap}}:T\mapsto\hat L$8, maps raw point trajectories into the same latent space as $f_{\mathrm{cap}}:T\mapsto\hat L$9, and trains with reconstruction, velocity consistency, range preservation, text alignment, image alignment, and text-to-trajectory reconstruction losses. Its ablations show that removing $g(L):=\mathcal D(\mathcal E(L))$0 reduces retrieval $g(L):=\mathcal D(\mathcal E(L))$1 from $g(L):=\mathcal D(\mathcal E(L))$2 to $g(L):=\mathcal D(\mathcal E(L))$3, while removing $g(L):=\mathcal D(\mathcal E(L))$4 yields $g(L):=\mathcal D(\mathcal E(L))$5 (Galoaa et al., 11 Dec 2025). TrajPrism’s retrieval model TrajFuse formalizes language-driven semantic trajectory retrieval through a dual-encoder with InfoNCE loss over text and fused trajectory embeddings (Li et al., 11 May 2026).

A third pattern treats language as an intermediate bottleneck for prediction. In “Trajectory Prediction with Linguistic Representations,” the encoder produces a latent $g(L):=\mathcal D(\mathcal E(L))$6, a language generator samples a discrete token sequence $g(L):=\mathcal D(\mathcal E(L))$7 with Gumbel–Softmax, and a trajectory decoder attends over those tokens while also grounding agent-specific tokens through agent attention. The generator loss is

$g(L):=\mathcal D(\mathcal E(L))$8

with $g(L):=\mathcal D(\mathcal E(L))$9 (Kuo et al., 2021).

A fourth pattern is full-sequence language modeling. LMTraj trains T5-small/medium/large on promptized histories, scene captions, and Q&A templates with per-token cross-entropy, and uses beam search for deterministic inference and temperature sampling for multimodal inference (Bae et al., 2024). The flight-trajectory study likewise frames single-step and multi-step prediction as next-token completion over promptized waypoint histories, optimized with

$\mathbb R^{N\times T\times 2}$0

It reports that self-attention implicitly models joint spatial and temporal dependencies without additional positional encoding beyond the base model’s default embeddings (Luo et al., 29 Jan 2025).

These mechanisms imply that “mapping” can be literal decoding, latent conditioning, contrastive alignment, or asymmetric grounding. A common misconception is that language enters only as an external instruction. In several models it also acts as an internal latent code, an auxiliary supervision source, or a prompt that normalizes heterogeneous temporal structures.

4. Robotics, manipulation, and embodied control

Robotics work makes the diversity of trajectory-language mapping especially clear. Bucker et al. formulate 2D trajectory reshaping as

$\mathbb R^{N\times T\times 2}$1

where the original curve $\mathbb R^{N\times T\times 2}$2 is discretized into $\mathbb R^{N\times T\times 2}$3 waypoints, BERT encodes the command into $\mathbb R^{N\times T\times 2}$4, CLIP computes command–object similarities $\mathbb R^{N\times T\times 2}$5, geometry tokens are encoded by a 2-layer Transformer encoder, and a 4-layer autoregressive Transformer decoder predicts the reshaped waypoints under a Huber imitation loss. On held-out simulation data, the multimodal transformer achieves average Huber loss $\mathbb R^{N\times T\times 2}$6 versus $\mathbb R^{N\times T\times 2}$7 for an FCN regression baseline and $\mathbb R^{N\times T\times 2}$8 for a naïve predictor (Bucker et al., 2022).

LATTE extends that formulation to 3D positions plus scalar speed,

$\mathbb R^{N\times T\times 2}$9

uses actual object images rather than only semantic labels, and predicts modified trajectories with a Transformer encoder/decoder conditioned on frozen BERT and CLIP encoders. Runtime post-processing steps from $(\tau_k^s,\tau_k^e)$0 toward $(\tau_k^s,\tau_k^e)$1 while enforcing a hard constraint set $(\tau_k^s,\tau_k^e)$2, after which inverse kinematics converts the Cartesian path into robot-specific commands (Bucker et al., 2022). This modularity is echoed in Maurya et al., who replace learned waypoint prediction with a zero-shot, code-centric mapping

$(\tau_k^s,\tau_k^e)$3

where GPT-4o first produces a high-level plan and then a Python function that edits trajectory waypoints via helper calls such as detect_objects() and get_trajectory() (Maurya et al., 17 Apr 2025).

Other systems move from open-loop reshaping toward closed-loop execution. “Language-Driven Closed-Loop Grasping with Model-Predictive Trajectory Replanning” extracts the noun phrase from a command such as “Grasp the orange drill,” uses OWL-ViT or OWLv2 to obtain a binary segmentation mask, initializes and refines 6D object pose with FoundationPose plus a Kalman filter, and then solves a receding-horizon trajectory optimization with a triple-integrator state and jerk input. The pose module runs at up to $(\tau_k^s,\tau_k^e)$4 and trajectory optimization at approximately $(\tau_k^s,\tau_k^e)$5 (Nguyen et al., 2024). “Speech-to-Trajectory” instead maps spoken commands through RNNoise, Whisper, BERT embeddings, and a Transformer-based diffusion policy over trajectories in $(\tau_k^s,\tau_k^e)$6; GPT-4 semantic augmentation expands the training set from roughly $(\tau_k^s,\tau_k^e)$7 to roughly $(\tau_k^s,\tau_k^e)$8 samples, and average end-to-end latency is approximately $(\tau_k^s,\tau_k^e)$9 (Bamani et al., 7 Apr 2025).

The robotics literature also includes direct LLM trajectory generation without task-specific training. “LLMs as Zero-Shot Trajectory Generators” uses a single fixed GPT-4 prompt, no in-context examples, no motion primitives, and no external trajectory optimizers, and asks the model to emit dense end-effector poses and gripper actions from language plus vision-model outputs such as 3D bounding boxes from LangSAM-style perception (Kwon et al., 2023). At the opposite end of the spectrum, ZLATTE is explicitly learning-free: GPT-4o, Grounding-DINO, SAM2, DBSCAN, and oriented bounding boxes produce geometric primitives; an LLM parses natural language into symbolic position or speed constraints; and a potential-field optimizer reshapes the path with attraction, repulsion, curvature regularization, self-adherence, and multi-agent conflict resolution (Huang et al., 7 Sep 2025).

A further embodied variant uses language not to generate motion directly but to segment demonstrations into reusable skills. Raj et al. adapt Moment-DETR to a long robot trajectory consisting of visual clips and discrete actions, learn a language-conditioned change-point detector with Hungarian matching, localization loss, classification loss, and saliency loss, and report a $g:L\rightarrow T$0 improvement over a baseline approach in changepoint detection (Raj et al., 2023). This indicates that trajectory-language mapping can support hierarchical planning even when it is not itself the execution policy.

5. Forecasting, simulation, and mobility-oriented generation

In prediction and simulation, language often functions as a compact semantic scaffold for long-horizon structure. The Argoverse study on linguistic representations learns a vocabulary of approximately $g:L\rightarrow T$1 synthetic tokens such as MoveFast, TurnLeft, LaneChangeLeft, and Yield Agent#k, without direct per-word supervision. On 3-second Argoverse forecasting it reports Minimum-of-6 ADE/FDE of $g:L\rightarrow T$2 for the full model versus $g:L\rightarrow T$3 for a multi-head social attention decoder, and sample entropy drops from $g:L\rightarrow T$4-style higher-entropy baselines to $g:L\rightarrow T$5 bits in the full configuration, indicating narrower and more plausible future sets (Kuo et al., 2021).

LMTraj pushes this idea further by translating pedestrian trajectories into natural-language prompts and framing prediction as question answering. With T5 plus a numerical tokenizer trained to separate integer and decimal parts, deterministic inference on ETH-UCY achieves average ADE/FDE of $g:L\rightarrow T$6, while stochastic inference with $g:L\rightarrow T$7 samples reports $g:L\rightarrow T$8; the numerical tokenizer contributes a reported $g:L\rightarrow T$9 relative ADE gain, and auxiliary QA tasks contribute a further $$f_\theta(\xi_o,L,\{O_i\})\rightarrow \xi_{\mathrm{mod}}$$0–$$f_\theta(\xi_o,L,\{O_i\})\rightarrow \xi_{\mathrm{mod}}$$1 gain (Bae et al., 2024). The flight-trajectory study likewise reports lower MAE than a Transformer baseline in single-step and multi-step prediction, but average inference latency remains $$f_\theta(\xi_o,L,\{O_i\})\rightarrow \xi_{\mathrm{mod}}$$2 for LLaMA-3.1 versus $$f_\theta(\xi_o,L,\{O_i\})\rightarrow \xi_{\mathrm{mod}}$$3 for the Transformer, making real-time deployment an explicit concern (Luo et al., 29 Jan 2025).

Urban and traffic simulation place stronger emphasis on interaction structure. InteractTraj uses GPT-4 as a language-to-code encoder that emits map codes, vehicle codes, and interaction codes, followed by a code-to-trajectory decoder with map-to-interaction and interaction-to-vehicle cross-attention. On WOMD entire-scene reconstruction, the full model reports $$f_\theta(\xi_o,L,\{O_i\})\rightarrow \xi_{\mathrm{mod}}$$4, $$f_\theta(\xi_o,L,\{O_i\})\rightarrow \xi_{\mathrm{mod}}$$5, $$f_\theta(\xi_o,L,\{O_i\})\rightarrow \xi_{\mathrm{mod}}$$6, and $$f_\theta(\xi_o,L,\{O_i\})\rightarrow \xi_{\mathrm{mod}}$$7, improving over TrafficGen and LCTGen and outperforming its own ablation without interaction codes (Xia et al., 2024). LangTraj instead uses a language-conditioned diffusion model over joint futures of $$f_\theta(\xi_o,L,\{O_i\})\rightarrow \xi_{\mathrm{mod}}$$8 agents, with denoising score matching, open-loop pretraining, distillation from $$f_\theta(\xi_o,L,\{O_i\})\rightarrow \xi_{\mathrm{mod}}$$9 to $f:(E_q,E_t)\rightarrow\{(\tau_k^s,\tau_k^e)\}$0 steps, and closed-loop refinement. Text conditioning reduces $f:(E_q,E_t)\rightarrow\{(\tau_k^s,\tau_k^e)\}$1 from $f:(E_q,E_t)\rightarrow\{(\tau_k^s,\tau_k^e)\}$2 to $f:(E_q,E_t)\rightarrow\{(\tau_k^s,\tau_k^e)\}$3 and yields a $f:(E_q,E_t)\rightarrow\{(\tau_k^s,\tau_k^e)\}$4 minADE reduction on InterDrive (Chang et al., 15 Apr 2025).

Trajectory recovery can also be cast as a language-compatible problem. PLMTrajRec uses explicit trajectory prompts and area-flow-guided implicit prompts, fine-tunes a BERT-small encoder with LoRA adapters, and decodes road segment plus moving ratio. On Chengdu, recovery from 4-minute sparse sampling to 15-second dense trajectories reaches $f:(E_q,E_t)\rightarrow\{(\tau_k^s,\tau_k^e)\}$5 road-segment accuracy and $f:(E_q,E_t)\rightarrow\{(\tau_k^s,\tau_k^e)\}$6 RMSE, compared with $f:(E_q,E_t)\rightarrow\{(\tau_k^s,\tau_k^e)\}$7 and $f:(E_q,E_t)\rightarrow\{(\tau_k^s,\tau_k^e)\}$8 for MM-STGED; at 1-minute sampling it reaches $f:(E_q,E_t)\rightarrow\{(\tau_k^s,\tau_k^e)\}$9 and $\hat T=\arg\max_T P(T\mid L,(r_0,t_0))$00 (Wei et al., 2024).

The mobility literature is therefore not merely “LLMs for route planning.” It includes code-mediated interactive generation, diffusion-based multi-agent simulation, promptized forecasting, and recovery of missing spatiotemporal detail, each with distinct assumptions about observability, map structure, and controllability.

6. Benchmarks, metrics, and unresolved issues

Evaluation protocols are correspondingly heterogeneous. Video-language grounding uses retrieval metrics such as $\hat T=\arg\max_T P(T\mid L,(r_0,t_0))$01 and $\hat T=\arg\max_T P(T\mid L,(r_0,t_0))$02, with TW-BERT improving zero-shot MSRVTT $\hat T=\arg\max_T P(T\mid L,(r_0,t_0))$03 from $\hat T=\arg\max_T P(T\mid L,(r_0,t_0))$04 to $\hat T=\arg\max_T P(T\mid L,(r_0,t_0))$05 through T2W alone and fine-tuned $\hat T=\arg\max_T P(T\mid L,(r_0,t_0))$06 to $\hat T=\arg\max_T P(T\mid L,(r_0,t_0))$07; its Hierarchical Frame-Selector configuration $\hat T=\arg\max_T P(T\mid L,(r_0,t_0))$08 improves retrieval and QA by $\hat T=\arg\max_T P(T\mid L,(r_0,t_0))$09 over uniform-8 selection (Yang et al., 2023). TraceVision evaluates trajectory-guided captioning with BLEU-4, METEOR, ROUGE-L, CIDEr, and SPICE; text-guided trajectory prediction with $\hat T=\arg\max_T P(T\mid L,(r_0,t_0))$10; segmentation with Box $\hat T=\arg\max_T P(T\mid L,(r_0,t_0))$11 and mask cIoU; and video scene understanding with METEOR, CIDEr, and multi-aspect scores. It reports BLEU-4 $\hat T=\arg\max_T P(T\mid L,(r_0,t_0))$12, $\hat T=\arg\max_T P(T\mid L,(r_0,t_0))$13, $\hat T=\arg\max_T P(T\mid L,(r_0,t_0))$14, and HC-STVG METEOR/CIDEr of $\hat T=\arg\max_T P(T\mid L,(r_0,t_0))$15 (Yang et al., 23 Feb 2026).

Retrieval and captioning metrics have become increasingly formalized in mobility benchmarks. TrajPrism defines destination hit rate, endpoint distance, H@K, Jaccard over H3 cells, DTW, Hausdorff distance, EDR, R@K, MRR, BERTScore F1, ROUGE-L, METEOR, POI Recall, and Named Location Count, over $\hat T=\arg\max_T P(T\mid L,(r_0,t_0))$16 selected trajectories across Porto, San Francisco, and Beijing, with $\hat T=\arg\max_T P(T\mid L,(r_0,t_0))$17 task instances derived from three instruction variants, three retrieval queries, and one caption per trajectory (Li et al., 11 May 2026). Lang2Motion evaluates retrieval with Recall@K, generation with ADE/FDE, Smoothness, CLIP similarity, and TRAJAN metrics, reporting $\hat T=\arg\max_T P(T\mid L,(r_0,t_0))$18, $\hat T=\arg\max_T P(T\mid L,(r_0,t_0))$19, $\hat T=\arg\max_T P(T\mid L,(r_0,t_0))$20, $\hat T=\arg\max_T P(T\mid L,(r_0,t_0))$21, $\hat T=\arg\max_T P(T\mid L,(r_0,t_0))$22, and $\hat T=\arg\max_T P(T\mid L,(r_0,t_0))$23 (Galoaa et al., 11 Dec 2025).

Several unresolved issues recur across domains. One is compute and latency. TW-BERT introduces HFS because sampling more frames makes pre-training infeasible and passing all frames through asymmetric cross-modal layers is too expensive (Yang et al., 2023). The flight-trajectory study states directly that high inference latency poses a challenge for real-time applications (Luo et al., 29 Jan 2025). Another is data efficiency: Raj et al. show a steep performance drop when training data shrinks, from $\hat T=\arg\max_T P(T\mid L,(r_0,t_0))$24 at full data to $\hat T=\arg\max_T P(T\mid L,(r_0,t_0))$25 at $\hat T=\arg\max_T P(T\mid L,(r_0,t_0))$26 of trajectories, which suggests that video-retrieval-style alignment methods may be difficult to scale to real-robot regimes without greater sample efficiency (Raj et al., 2023).

A second recurring issue is the trade-off between flexibility and guarantees. Maurya et al. emphasize that task-specific training is unnecessary because GPT-4o already encodes numeric reasoning, spatial language, and Python idioms, but they also note that generated code has no formal guarantees on smoothness or collision-free properties (Maurya et al., 17 Apr 2025). ZLATTE takes the opposite position by translating language into explicit geometric and kinematic constraints within a potential-field optimizer; the reported result is smoother and safer trajectory modification, with zero penetrations in the compared setup and $\hat T=\arg\max_T P(T\mid L,(r_0,t_0))$27 success on a Panda arm, but the framework depends on reliable object registration and constraint parsing (Huang et al., 7 Sep 2025). TLControl reports strong trajectory accuracy and runtime efficiency for human motion synthesis, yet explicitly notes the absence of a physics solver and scene-aware adaptation (Wan et al., 2023).

A final misconception is that trajectory-language mapping is inherently end-to-end neural. The cited record includes end-to-end Transformers, frozen-encoder plus lightweight-adapter systems, retrieval-augmented LLM prompting, code synthesis, modular perception–control stacks, and fully learning-free optimization. This suggests that the unifying criterion is not architecture class but whether the system establishes a usable, testable correspondence between linguistic intent or description and trajectory structure.