Trajectory-Token Interface

• Trajectory-token interface is a structured mapping that compresses high-level semantic intent into fixed tokens, uniting action semantics with spatio-temporal waypoints.
• It packages information as one semantic token and three waypoint tokens (START, CONTACT, END) that capture time, 3D position, and 6D rotation for precise motion prediction.
• The approach leverages conditional flow matching and joint optimization of reasoning and motion modules to deliver efficient and interpretable dynamic predictions.

A trajectory-token interface is a structured mapping designed to bridge semantic reasoning with motion or perception in learning systems, particularly where multimodal architectures must coherently unify high-level intent or object-centric information with dense, continuous motion or spatio-temporal dynamics. This approach, exemplified in "Flowing from Reasoning to Motion: Learning 3D Hand Trajectory Prediction from Egocentric Human Interaction Videos" (Chen et al., 18 Dec 2025), enables explicit and interpretable passage of critical waypoints and semantics between reasoning and motion modules, yielding a compact alternative to full trajectory curves and promoting stage-aware, generalizable prediction.

1. Formal Definition and Core Token Construction

The trajectory-token interface distills complex motion and intent into a small, fixed number of structured tokens, unlike approaches that transmit entire continuous curves or verbose language chains. Specifically, in EgoMAN (Chen et al., 18 Dec 2025):

• Semantic token (⟨ACT⟩): Encodes an action phrase's hidden state, e.g., "left hand grasps green cup."
• Three spatio-temporal waypoint tokens (⟨START⟩, ⟨CONTACT⟩, ⟨END⟩): Each contains a predicted timestamp, 3D wrist position $\hat p\in \mathbb{R}^3$, and 6D rotation parameter $\hat R\in\mathbb{R}^6$, respectively marking approach onset, manipulation onset, and completion.

Let $$r_{\text{act}}, r_{\text{start}}, r_{\text{contact}}, r_{\text{end}}$$ be hidden states; mapping functions $f_{\text{act}}, f_{\text{wp}}$ then project these to semantic and waypoint trajectory tokens:

• $$z_{\text{act}} = f_{\text{act}}(r_{\text{act}}) \in \mathbb{R}^d$$
• $z_{\text{start}} = f_{\text{wp}}(r_{\text{start}})$
• $$z_{\text{contact}} = f_{\text{wp}}(r_{\text{contact}})$$
• $z_{\text{end}} = f_{\text{wp}}(r_{\text{end}})$

Each waypoint token packages timestamp, position, and rotation.

2. Mathematical Mapping and Decoding

The interface vector $$Z = \{z_{\text{act}}, z_{\text{start}}, z_{\text{contact}}, z_{\text{end}}\}$$ becomes the conditioning segment for the motion generation module. Formally:

$$z_t = f_t(r_t) \qquad \forall t \in \{\text{act, start, contact, end}\}$$

Mappings:

• $$f_{\text{act}}(r) = W_{\text{act}} r + b_{\text{act}}$$
• $$f_{\text{wp}}(r) = [W_{\text{time}}\, r ; W_{\text{xyz}}\, r ; W_{\text{rot}}\, r] + b_{\text{wp}}$$

The motion expert decodes $Z$ via conditional flow matching. It models a velocity field $v(x,t;Z,\text{ctx})$ and deterministic transport:

$$x_{k+1} = x_k + \Delta t \cdot v(x_k, t_k; Z, \text{ctx})$$

Supervision is via mean squared error on predicted flow vectors:

$$L_{\text{FM}} = \mathbb{E}_{x_0, x_1, t} \| v(x_0, t; Z, \text{ctx}) - (x_1 - x_0) \|_2^2$$

Semantic and waypoint tokens serve as global and structural anchors for temporal transformer input.

3. Model Architecture and Operational Integration

A Qwen-VL-style vision-language reasoning module ingests an egocentric RGB frame, text intent, and wrist pose history, then produces structured output tokens via a special $\langle \text{HOI\_QUERY} \rangle$ prompt. Learned MLP heads map these to dense trajectory tokens.

The motion expert is a transformer encoder-decoder whose input sequence comprises:

• $H$ previous wrist motion tokens,
• three explicit waypoint tokens inserted at stage-anchored temporal positions,
• $T$ placeholders for future trajectory prediction,
• non-temporal context tokens for fused visual features and the semantic action token.

The decoder attends over this sequence and generates future 6DoF states.

4. Optimization Objectives and Training Strategies

Training is staged:

• Reasoning pretraining: Combines text modeling loss ($L_{\text{text}}$), contrastive semantic loss ($L_{\text{act}}$) aligning $z_{\text{act}}$ to CLIP embeddings, and waypoint regression ($L_{\text{wp}}$) comprising timestamp, 3D/2D position, rotation, and geodesic losses.

$$L_{\text{reason}} = L_{\text{text}} + \lambda_{\text{act}} L_{\text{act}} + \lambda_{\text{wp}} L_{\text{wp}}$$

• Motion expert pretraining: Uses ground truth waypoints and semantic tokens to learn the velocity field under flow matching objective $L_{\text{FM}}$.
• Joint fine-tuning: Reasoning and motion modules are end-to-end optimized, enforcing the physical accuracy of decoded trajectories as well as semantic answer quality.

$$L_{\text{joint}} = L_{\text{text}} + \lambda_{\text{FM}} L_{\text{FM}}$$

5. Comparative Ablations, Effectiveness, and Interpretability

Ablation analyses substantiated several design choices:

Ablation	Metric: ADE (m)	Effect
Omit waypoint tokens	0.215	Poor spatial anchoring, error ↑
Full token interface	0.151	Highest accuracy, explicit anchors
Implicit (no 6DoF waypoints)	Slightly worse	ADE/FDE and rotation error ↑
Waypoint-only (Omit semantics)	>50% ↓	Trajectory-warp distance, efficient

Training the reasoning and motion expert separately before joint optimization enhanced stability and accuracy over naïve end-to-end paradigms.

6. Broader Interface Roles in Related Systems

Trajectory-token interfaces appear in numerous domains:

• Video tokenization: Object trajectories mapped to semantic tokens ("TrajViT" (Zheng et al., 29 May 2025)), yielding efficient, scene-complexity-scaled representations.
• RL and multimodal reasoning: Token-level saliency and dependency steering policy optimization, favoring perceptually meaningful trajectory updates ("VPPO" (Huang et al., 10 Oct 2025)).
• Video generation: Explicit trajectory tokens drive controllable motion in DiT-style and VAE architectures ("TokenMotion" (Li et al., 11 Apr 2025), "DiTraj" (Lei et al., 26 Sep 2025), "InTraGen" (Liu et al., 25 Nov 2024)).
• Tracking and meta-learning: Dense temporal tokens encode and propagate object states for efficient tracking ("UM-ODTrack" (Zheng et al., 27 Jul 2025)) and action recognition ("Trokens" (Kumar et al., 5 Aug 2025)).
• Discrete planning: Rule- and data-driven trajectory tokenizers enable interpretable next-token behavior generation ("TrajTok" (Zhang et al., 23 Jun 2025)), autonomous driving MM-LLM planning ("TOKEN" (Tian et al., 1 Jul 2024)), test-time distribution shift correction ("T4P" (Park et al., 15 Mar 2024)), and autoregressive manipulation ("Chain-of-Action" (Zhang et al., 11 Jun 2025)).

7. Interpretability, Generalization, and Practical Impact

Trajectory-token interfaces facilitate interpretable, physically grounded induction over structured tasks, directly aligning high-level semantics with mid-level spatial-temporal anchors. They enable generalization to complex, previously unseen environments since they condense essential reasoning and control information into sparse, domain-meaningful tokens. Removal of explicit waypoints or reduction to implicit token structures measurably degrades error and task performance. These interfaces support scalable, flexible architectures—ranging from sequence-model decoders to video diffusion transformer blocks—while retaining computational and deployment efficiency.

In summary, trajectory-token interfaces provide a principled and effective framework for unifying reasoning and action, enabling efficient communication between semantic intent and dynamic prediction modules, and are empirically validated to yield interpretable, robust solutions across vision-language-motor domains (Chen et al., 18 Dec 2025).