Trajectory-Guided Prompting

• Trajectory-guided prompting is a conditioning strategy where dynamic trajectory information serves as the prompt, guiding models through evolving intermediate states.
• It utilizes various forms—from control signals to spatial tokens—to dynamically condition generative models in domains like image editing, RL, and motion prediction.
• This approach improves controllability and performance, yielding enhanced reward-fidelity trade-offs and efficient prompt-tuning across diverse multimodal applications.

Trajectory-guided prompting is a family of conditioning strategies in which a trajectory—or a trajectory-derived object—serves as the primary vehicle of guidance. Depending on the setting, that object may be a time-varying control field over a generative path, a short trajectory segment of states, actions, and returns, a sequence of spatiotemporal cue tokens, a user-specified motion path attached to image regions, a complete observation–action history used as an exemplar, or a natural-language narrative derived from mobility traces. Across recent work, the common departure from static prompting is that guidance is distributed over a sequence or path, so the model is constrained not only by a target outcome but also by how intermediate states evolve over time (Chang et al., 30 Sep 2025, Hu et al., 2023, Saadatnejad et al., 2023, Zhang et al., 8 Sep 2025, Li et al., 19 May 2026, Zheng et al., 2023, Wang et al., 2024, Xie et al., 14 Oct 2025, Wei et al., 2024).

1. Conceptual scope and defining forms

In the literature, “prompt” is not restricted to text. In reward-guided image editing, the prompt is effectively a terminal reward whose adjoint backpropagation produces a time-varying control signal along the reverse generative trajectory. In Decision Transformer adaptation, the prompt is a short trajectory segment $\tau^*=\{(\hat r_t^*, s_t^*, a_t^*)\}_{t=1}^{K^*}$ that conditions future decisions. In promptable human trajectory prediction, the prompt can be trajectories, 2D or 3D poses, or 2D or 3D boxes. In mapless navigation, candidate trajectories themselves are rendered into an image and become the object over which a VLM reasons. In web control, complete successful trajectories become in-context exemplars. In mobility reasoning, trajectories are converted into activity chronicles and multi-scale summaries that guide hierarchical chain-of-thought inference (Chang et al., 30 Sep 2025, Hu et al., 2023, Saadatnejad et al., 2023, Song et al., 2024, Zheng et al., 2023, Gao et al., 16 Oct 2025, Xie et al., 14 Oct 2025).

Instantiation	Prompt carrier	Operational role
Reward-guided image editing (Chang et al., 30 Sep 2025)	Terminal reward and adjoint states	Steer the full reverse trajectory
Prompt-Tuning DT (Hu et al., 2023)	Trajectory segment $\tau^*$	Condition a frozen DT policy
Social-Transmotion (Saadatnejad et al., 2023)	Trajectories, poses, boxes	Condition future human motion
Zo3T I2V (Zhang et al., 8 Sep 2025)	Box plus user trajectory	Constrain region motion in video
SPOT (Li et al., 19 May 2026)	First-frame points or boxes	Specify object and target for EE motion
VL-TGS (Song et al., 2024)	Candidate paths overlaid on images	VLM trajectory selection
Synapse (Zheng et al., 2023)	Complete observation–action trajectories	In-context control exemplars
Open-vocabulary VidVRD (Wang et al., 2024)	Subject/object trajectories	Guide visual and language prompts
HiCoTraj (Xie et al., 14 Oct 2025)	Trajectory narratives and summaries	Hierarchical demographic reasoning
PTR (Wei et al., 2024)	Explicit text prompt plus implicit prompt embeddings	Condition PLM trajectory recovery

This breadth shows that trajectory-guided prompting is better understood as a structural principle than as a single algorithm. The shared principle is that trajectory information is elevated from passive data to active conditioning context. In some works it determines the control law; in others it narrows the feasible trajectory distribution; in others it provides exemplars, retrieval keys, or prompt tokens.

2. Formal mechanisms

One prominent formalization treats generation itself as a controllable dynamical system. In training-free reward-guided image editing, the reverse process is written as

$$d x_t^u = \left(b(x_t^u,t) + u(x_t^u,t)\right)\,dt + \sigma_t d\mathbf{B}_t,$$

with objective

$$\min_{u\in\mathcal{U}} \int_T^1 \frac{1}{2}\|u(x_t^u,t)\|^2\,dt \;-\; r(x_1^u).$$

Pontryagin’s Maximum Principle yields $u_t^*=-p_t^*$, so the adjoint state becomes a time-varying guidance signal defined over the entire path rather than a one-step reward gradient. The paper explicitly connects this to “trajectory-guided prompting”: the reward acts as a global prompt, and the adjoint field $\{u_t\}$ acts as dynamic, stepwise prompt vectors (Chang et al., 30 Sep 2025).

In offline RL, Prompt-Tuning DT formalizes the prompt as a short trajectory segment prepended to history: $\tau^{\text{input}} = (\tau_i^*, \tau_i).$ The prompt is flattened into $x\in\mathbb{R}^{d_x}$, perturbed by Gaussian noise,

$$x_i = x + \mu \xi_i,\quad \xi_i \sim \mathcal{N}(0, I_{d_x}),$$

and updated with the rank-based estimator

$$\tilde{g}(x) = \frac{1}{|\mathcal{E}|} \sum_{(i,j) \in \mathcal{E}} (\xi_j - \xi_i),\qquad x \leftarrow x - \eta \tilde{g}(x).$$

Here the trajectory prompt is not merely context; it is the optimization variable that adapts a frozen policy to new tasks or preferences (Hu et al., 2023).

In egocentric manipulation, SPOT defines a conditional generative model over future end-effector trajectory chunks,

$\tau^*$0

where first-frame spatial prompts specify the object and target once, while current observation and history determine the phase of execution. This gives a prompt-conditioned trajectory distribution rather than a direct action labeler (Li et al., 19 May 2026).

In sparse trajectory recovery, PTR combines explicit natural-language prompting and implicit embedding-space prompting. The explicit prompt embedding is

$\tau^*$1

where the components encode task, target, sampling interval, time, and movement summaries. Missing points are assigned implicit prompt vectors derived from road-condition diffusion,

$\tau^*$2

so the PLM attends jointly to textual task descriptions and trajectory-conditioned road-state embeddings (Wei et al., 2024).

Synapse gives a different but equally formal definition: a trajectory exemplar is a serialized alternating sequence of task, abstracted observations, and actions,

$\tau^*$3

The LLM is then queried with retrieved successful trajectories plus the current partial trajectory, making next-action generation an in-context trajectory-conditioned policy rather than isolated step prediction (Zheng et al., 2023).

3. Architectural realizations across domains

In multimodal motion prediction, Social-Transmotion operationalizes prompts as cue-specific token sequences. For each agent and cue $\tau^*$4,

$\tau^*$5

where $\tau^*$6 aggregates temporal, person-identity, and keypoint-type embeddings. A Cross-Modality Transformer fuses trajectories, poses, and boxes with latent future queries, and a Social Transformer models inter-agent interactions. Modality-masking and meta-masking train the same architecture to accept arbitrary cue subsets at inference, so “available cues at inference time” become the active prompt (Saadatnejad et al., 2023).

Open-vocabulary video visual relationship detection extends the idea by letting trajectories guide both visual and textual prompting of CLIP. Subject and object trajectories define the RoI features, spatial positions, and temporal indices used in spatio-temporal visual prompting, while those same trajectory features generate vision-guided language prompt tokens for object and relationship labels. The detector is made relationship-aware through a relationship query and auxiliary relationship loss, so trajectories are shaped by relationship context and, in turn, shape multi-modal prompts for classification (Wang et al., 2024).

In image-to-video generation, Zo3T interprets a user-specified object trajectory as a first-class control signal. A 3D-Aware Kinematic Projection converts a 2D path and an initial box into perspective-correct affine transforms and masks. These masks supervise region-consistent motion through Trajectory-Guided Test-Time LoRA and Guidance Field Rectification, so the motion prompt is embedded into both latent updates and denoising guidance (Zhang et al., 8 Sep 2025).

In mapless navigation, VL-TGS couples a CVAE trajectory generator with VLM-based selection. The CVAE proposes diverse candidate trajectories, these paths are projected into the camera image and numbered, and the VLM is prompted with the overlaid image plus textual rules such as “walk on pavements” and “cannot go over/under the curbs.” In this setting, the trajectories literally guide the prompt, and the prompt guides which trajectory is executed (Song et al., 2024).

Robotics and computer-control systems exhibit two further realizations. In zero-shot robot manipulation, GPT-4 is prompted to output dense end-effector trajectories directly from a language command and object detections, without in-context examples, motion primitives, or external trajectory optimizers. In Synapse, by contrast, the prompt consists of complete successful computer-control trajectories stored in memory and retrieved by similarity, enabling long-horizon web interaction through trajectory-as-exemplar prompting (Kwon et al., 2023, Zheng et al., 2023).

Mobility analytics provides yet another realization. HiCoTraj transforms trajectories into detailed activity chronicles and multi-scale visiting summaries, then constrains an LLM to proceed through factual extraction, behavioral pattern analysis, and demographic inference. The trajectory is thus converted into a staged reasoning scaffold rather than a control sequence (Xie et al., 14 Oct 2025).

4. Optimization and inference regimes

Trajectory-guided prompting does not imply a single training regime. Reward-guided image editing is entirely training-free: it initializes from an inverted source trajectory, computes discrete adjoints backward,

$\tau^*$7

updates controls by

$\tau^*$8

and re-simulates the trajectory iteratively. The prompt-like object is the control field itself, synthesized at inference time from the reward and the current path (Chang et al., 30 Sep 2025).

Zo3T is also zero-shot with respect to trajectory-labeled data, but it is not training-free in the narrower sense. It performs test-time training by jointly optimizing the latent and ephemeral LoRA parameters during denoising, with guidance applied at steps $\tau^*$9 to $$d x_t^u = \left(b(x_t^u,t) + u(x_t^u,t)\right)\,dt + \sigma_t d\mathbf{B}_t,$$0, latent learning rate $$d x_t^u = \left(b(x_t^u,t) + u(x_t^u,t)\right)\,dt + \sigma_t d\mathbf{B}_t,$$1, LoRA learning rate $$d x_t^u = \left(b(x_t^u,t) + u(x_t^u,t)\right)\,dt + \sigma_t d\mathbf{B}_t,$$2, guidance-field learning rate $$d x_t^u = \left(b(x_t^u,t) + u(x_t^u,t)\right)\,dt + \sigma_t d\mathbf{B}_t,$$3, and Fourier Orthogonal Recomposition cutoff $$d x_t^u = \left(b(x_t^u,t) + u(x_t^u,t)\right)\,dt + \sigma_t d\mathbf{B}_t,$$4. This is a prompt-conditioned co-adaptation regime rather than static conditioning (Zhang et al., 8 Sep 2025).

Prompt-Tuning DT freezes the pre-trained Decision Transformer and updates only the prompt vector through black-box preference ranking. The method has offline and online variants, depending on whether $$d x_t^u = \left(b(x_t^u,t) + u(x_t^u,t)\right)\,dt + \sigma_t d\mathbf{B}_t,$$5 is derived from imitation loss on limited data or negative episodic return from rollouts. The optimized prompt corresponds to about $$d x_t^u = \left(b(x_t^u,t) + u(x_t^u,t)\right)\,dt + \sigma_t d\mathbf{B}_t,$$6 of the full model parameters, so adaptation occurs almost entirely in prompt space (Hu et al., 2023).

SPOT trains a conditional trajectory generator and reports that a flow-matching head is the default, while a DDPM-like diffusion head is an alternative. PTR likewise uses a pre-trained encoder model but fine-tunes it with LoRA on joint datasets covering multiple sparse intervals $$d x_t^u = \left(b(x_t^u,t) + u(x_t^u,t)\right)\,dt + \sigma_t d\mathbf{B}_t,$$7, relying on explicit prompt text to tell the model which interval is currently being recovered (Li et al., 19 May 2026, Wei et al., 2024).

Other regimes are purely prompt-driven at inference. HiCoTraj is zero-shot and prompt-only, using hierarchical chain-of-thought without labeled training data. GPT-4 trajectory generation for robot control is also zero-shot, but augments prompting with a failure-detection loop in which object trajectories from XMem are fed back to the model for replanning. Synapse uses no parameter updates at test time; instead, it performs similarity-based exemplar retrieval from memory and conditions the LLM on complete successful trajectories (Xie et al., 14 Oct 2025, Kwon et al., 2023, Zheng et al., 2023).

A plausible implication is that trajectory-guided prompting is best viewed as orthogonal to the optimization regime. The defining feature is not whether parameters are updated, but whether guidance is trajectory-structured.

5. Empirical characteristics

In reward-guided image editing, the principal reported behavior is a better reward-versus-fidelity Pareto frontier than inversion-based training-free guidance baselines. The paper states that, for any given level of distortion measured by LPIPS, trajectory control achieves higher reward; it also reports that naive gradient ascent attains high reward with artifacts, whereas trajectory control improves reward, independent metrics, and human ratings without reward hacking. The method is more expensive than simple guided sampling, at roughly $$d x_t^u = \left(b(x_t^u,t) + u(x_t^u,t)\right)\,dt + \sigma_t d\mathbf{B}_t,$$8 in the timing table (Chang et al., 30 Sep 2025).

In RL adaptation, Prompt-Tuning DT reports that with only $$d x_t^u = \left(b(x_t^u,t) + u(x_t^u,t)\right)\,dt + \sigma_t d\mathbf{B}_t,$$9 of the parameters learned, prompt tuning achieves comparable or even better performance than full-model fine-tuning in low-data scenarios. On the meta-RL tasks, average scores are 445.3 for Prompt-DT, 450.0 for Prompt-DT-FT, 450.3 for PTDT-offline, and 449.9 for PTDT-online. On D4RL medium datasets, average normalized returns are 61.6 for DT, 62.0 for Prompt-DT, 63.2 for Prompt-DT-FT, 63.1 for PTDT-offline, and 62.8 for PTDT-online (Hu et al., 2023).

For human trajectory prediction, promptable cue conditioning yields consistent gains. On JTA, Social-Transmotion reports ADE/FDE of $$\min_{u\in\mathcal{U}} \int_T^1 \frac{1}{2}\|u(x_t^u,t)\|^2\,dt \;-\; r(x_1^u).$$0 with trajectories only, $$\min_{u\in\mathcal{U}} \int_T^1 \frac{1}{2}\|u(x_t^u,t)\|^2\,dt \;-\; r(x_1^u).$$1 with $$\min_{u\in\mathcal{U}} \int_T^1 \frac{1}{2}\|u(x_t^u,t)\|^2\,dt \;-\; r(x_1^u).$$2 pose, $$\min_{u\in\mathcal{U}} \int_T^1 \frac{1}{2}\|u(x_t^u,t)\|^2\,dt \;-\; r(x_1^u).$$3 with $$\min_{u\in\mathcal{U}} \int_T^1 \frac{1}{2}\|u(x_t^u,t)\|^2\,dt \;-\; r(x_1^u).$$4 pose, and $$\min_{u\in\mathcal{U}} \int_T^1 \frac{1}{2}\|u(x_t^u,t)\|^2\,dt \;-\; r(x_1^u).$$5 with all cues. On JRDB, adding 2D bounding boxes improves $$\min_{u\in\mathcal{U}} \int_T^1 \frac{1}{2}\|u(x_t^u,t)\|^2\,dt \;-\; r(x_1^u).$$6 to $$\min_{u\in\mathcal{U}} \int_T^1 \frac{1}{2}\|u(x_t^u,t)\|^2\,dt \;-\; r(x_1^u).$$7. Attention analyses indicate that later observed frames, and especially ankles, wrists, and knees, receive the highest weights, while head-only prompts are nearly equivalent to trajectories alone (Saadatnejad et al., 2023).

In egocentric manipulation, spatial prompting materially narrows the trajectory distribution. Across all scenes in EgoSPT, SPOT reports FDE $$\min_{u\in\mathcal{U}} \int_T^1 \frac{1}{2}\|u(x_t^u,t)\|^2\,dt \;-\; r(x_1^u).$$8 and Pos. L2 $$\min_{u\in\mathcal{U}} \int_T^1 \frac{1}{2}\|u(x_t^u,t)\|^2\,dt \;-\; r(x_1^u).$$9 for the no-prompt baseline, versus FDE $u_t^*=-p_t^*$0 and Pos. L2 $u_t^*=-p_t^*$1 for the default BBox+visual prompting configuration. The flow-matching head reports FDE $u_t^*=-p_t^*$2, Pos. L2 $u_t^*=-p_t^*$3, while the diffusion head reports FDE $u_t^*=-p_t^*$4, Pos. L2 $u_t^*=-p_t^*$5; frozen DINOv2 also outperforms a tuned backbone (Li et al., 19 May 2026).

In trajectory-controlled image-to-video generation, Zo3T reports FID $u_t^*=-p_t^*$6, FVD $u_t^*=-p_t^*$7, and ObjMC $u_t^*=-p_t^*$8 on VIPSeg, outperforming all zero-shot baselines and approaching or surpassing supervised baselines in quality with competitive motion accuracy. Ablations show degradation without 3D projection, TTT, GFR, or FOR; notably, removing TTT yields FID $u_t^*=-p_t^*$9, FVD $\{u_t\}$0, ObjMC $\{u_t\}$1, and removing GFR yields FID $\{u_t\}$2, FVD $\{u_t\}$3, ObjMC $\{u_t\}$4 (Zhang et al., 8 Sep 2025).

In navigation, VL-TGS reports an average improvement of $\{u_t\}$5 in satisfying traversability constraints and $\{u_t\}$6 in human-like navigation across four outdoor scenarios. The paper also reports scenario-wise gains in traversability and lower Fréchet distance to human teleoperation relative to MTG, ViNT, NoMaD, and CoNVOI, with the strongest margins in flower-bed, curb, and crosswalk settings (Song et al., 2024).

In open-vocabulary VidVRD, trajectory-guided multi-modal prompting improves both detection and relationship classification. On VidVRD SGDet, the novel split reports mAP$\{u_t\}$7 and mAP $\{u_t\}$8, compared with 14.37 and 12.15 for OV-MMP. Ablations further report that removing both visual and language prompting gives novel SGDet mAP $\{u_t\}$9, while using both gives $\tau^{\text{input}} = (\tau_i^*, \tau_i).$0; using only spatial visual prompting gives $\tau^{\text{input}} = (\tau_i^*, \tau_i).$1, only temporal visual prompting $\tau^{\text{input}} = (\tau_i^*, \tau_i).$2, and the full spatial-plus-temporal design $\tau^{\text{input}} = (\tau_i^*, \tau_i).$3 (Wang et al., 2024).

In computer and robot control, Synapse reports a $\tau^{\text{input}} = (\tau_i^*, \tau_i).$4 average success rate across 64 MiniWoB++ tasks using demonstrations from only 48 tasks, and is described as the first ICL method to solve the book-flight task. On Mind2Web, it reports a $\tau^{\text{input}} = (\tau_i^*, \tau_i).$5 relative improvement in average step success rate over the previous prompting scheme. The GPT-4 robot-trajectory work reports nontrivial success across 26 real-world tasks with a single task-agnostic prompt and shows that trajectory-based failure feedback can improve bowl-grasp success from $\tau^{\text{input}} = (\tau_i^*, \tau_i).$6 to $\tau^{\text{input}} = (\tau_i^*, \tau_i).$7 after replanning (Zheng et al., 2023, Kwon et al., 2023).

For zero-shot demographic reasoning, HiCoTraj reports, with Mistral-7B + CoT, age accuracy/F1 $\tau^{\text{input}} = (\tau_i^*, \tau_i).$8, income $\tau^{\text{input}} = (\tau_i^*, \tau_i).$9, and education $x\in\mathbb{R}^{d_x}$0; with Qwen3-8B + CoT, age $x\in\mathbb{R}^{d_x}$1, income $x\in\mathbb{R}^{d_x}$2, and education $x\in\mathbb{R}^{d_x}$3. The paper characterizes this as competitive in zero-shot scenarios and uses ablations to show slight degradation when removing Stage 1 or Stage 2 of the hierarchy (Xie et al., 14 Oct 2025).

6. Limitations, misconceptions, and prospective directions

A recurrent misconception is that prompting is necessarily static and linguistic. The surveyed work contradicts that view: prompts may be terminal rewards, adjoint-derived control laws, trajectory segments, spatial marks on a first frame, cue tokens from poses and boxes, candidate paths rendered into images, complete successful trajectories, or textual chronicles derived from mobility traces (Chang et al., 30 Sep 2025, Hu et al., 2023, Saadatnejad et al., 2023, Song et al., 2024, Li et al., 19 May 2026, Zheng et al., 2023, Xie et al., 14 Oct 2025). A second misconception is that trajectory-guided prompting is necessarily training-free. Some methods are training-free or zero-shot, but others use black-box prompt tuning, flow matching, LoRA fine-tuning, or joint end-to-end optimization (Hu et al., 2023, Zhang et al., 8 Sep 2025, Li et al., 19 May 2026, Wang et al., 2024, Wei et al., 2024).

The limitations are correspondingly heterogeneous. Reward-guided editing requires a differentiable reward, incurs multiple forward–backward sweeps, and remains local and non-convex; prompt-tuning in RL is sensitive to prompt initialization quality and depends on a reasonably strong pre-trained Prompt-DT; Social-Transmotion assumes physically plausible sensor cues and does not explore symbolic prompting; SPOT is evaluated open-loop and requires accurate first-frame prompts; VL-TGS inherits VLM robustness and bias issues; HiCoTraj raises privacy and fairness concerns because demographic inference from mobility is sensitive; Synapse depends on exemplar quality and can fail under poor retrieval; PTR depends on road-network information, reference-token transformation, and implicit road-condition prompting for best performance (Chang et al., 30 Sep 2025, Hu et al., 2023, Saadatnejad et al., 2023, Li et al., 19 May 2026, Song et al., 2024, Xie et al., 14 Oct 2025, Zheng et al., 2023, Wei et al., 2024).

Several extensions are explicit in the cited work. Reward-guided control papers suggest multi-objective rewards, human-in-the-loop feedback, and applications to text or multimodal generation. SPOT suggests combining spatial prompts with language and interactive re-prompting. Zo3T notes that the region-based machinery could use more complex spatial constraints than boxes. HiCoTraj suggests that the same staged narrative-to-reasoning design could transfer to other structured data domains. Synapse suggests distilling lighter agents from trajectory-conditioned prompting policies. Taken together, these directions suggest that trajectory-guided prompting is evolving toward a general framework in which prompts specify not only targets but admissible paths, intermediate structure, and control schedules across time (Chang et al., 30 Sep 2025, Zhang et al., 8 Sep 2025, Li et al., 19 May 2026, Xie et al., 14 Oct 2025, Zheng et al., 2023).