Trajectory-Specific RQ-VAE

• Trajectory-Specific RQ-VAE is a discrete latent model that encodes sequential data by decomposing trajectory residuals through multiple additive quantization stages.
• It leverages trajectory-specific design such as CNN-Transformer encoders and variable-length correction to capture both coarse travel semantics and fine movement details.
• By converting micro-level GPS or motion data into hierarchical latent tokens, the model enhances downstream tasks like travel pattern generation, motion editing, and forecasting.

Searching arXiv for papers directly relevant to trajectory-specific residual quantization VAEs and closely related residual quantization sequence models. Trajectory-Specific Residual Quantization Variational Autoencoder (RQ-VAE) denotes a class of discrete latent autoencoding systems in which a trajectory is encoded as a temporal or segment-level latent sequence and then quantized by multiple residual codebooks in a coarse-to-fine manner. In the most explicit usage of the term, HTP introduces a “trajectory-specific residual quantization variational autoencoder (RQ-VAE)” for GPS trajectories that converts raw, variable-length point sequences into compact “travel pattern” tokens for downstream LLM generation (Zhou et al., 28 May 2026). Closely related work on human motion shows the same core principle in other sequential domains: residual quantization is used to decompose a motion latent into progressively finer temporal abstractions, later consumed by autoregressive or editing modules (Liu et al., 3 Nov 2025, Zargarbashi et al., 2 Feb 2026). Across these systems, the “VAE” label is conventional; the operative training objective is typically reconstruction plus VQ/RQ commitment terms rather than a classical ELBO with a KL regularizer (Zhou et al., 28 May 2026, Liu et al., 3 Nov 2025).

1. Conceptual scope and historical placement

A trajectory-specific RQ-VAE is best understood as the intersection of two ideas. The first is residual quantization, in which a latent vector is approximated by a sum of code vectors chosen stage by stage from multiple quantizers. The second is trajectory specialization, meaning that the encoder, decoder, losses, and token semantics are designed around sequential structure rather than around images or generic token streams. A representative early formulation of RQ-VAE in image generation quantized each latent position with a stack of discrete codes, using residual recursion and additive reconstruction to preserve fidelity at low latent resolution (Lee et al., 2022). Trajectory-oriented work retains the residual mechanism but changes what is being encoded: temporal slots, motion segments, or GPS segments rather than image patches.

The most direct trajectory-specific instantiation appears in HTP, where a trajectory is defined as

$$\tau = \langle p_i \rangle_{i=1}^{n}, \qquad p_i = (x_i, y_i),$$

with longitude and latitude only, and the model explicitly avoids direct GPS-point generation by the LLM (Zhou et al., 28 May 2026). The stated rationale is twofold: direct GPS generation is too fine-grained because tiny coordinate errors accumulate autoregressively and cause spatial drift, and raw GPS sequences are too long and expensive for an LLM to emit token by token. The RQ-VAE therefore functions as a bridge between micro-level GPS points and macro-level travel-pattern tokens (Zhou et al., 28 May 2026).

Related sequential models clarify the broader meaning of “trajectory-specific.” MoSa is framed as text-driven 3D human motion generation rather than GPS modeling, but it uses a hierarchical residual quantization VAE tailored to temporal motion sequences, with tokens corresponding to latent temporal slots and scales corresponding to increasingly fine temporal structure (Liu et al., 3 Nov 2025). VQ-Style similarly treats motion as a temporal signal and uses residual codebooks as a coarse-to-fine factorization in which early stages capture content and later stages capture style/detail (Zargarbashi et al., 2 Feb 2026). This suggests that “trajectory-specific RQ-VAE” is less a single canonical architecture than a design pattern for sequential data whose latent structure is inherently hierarchical.

Paper	Domain	Relevance
“From GPS Points to Travel Patterns: Flexible and Semantic Trajectory Generation with LLMs” (Zhou et al., 28 May 2026)	Urban GPS trajectories	Explicit trajectory-specific RQ-VAE
“MoSa: Motion Generation with Scalable Autoregressive Modeling” (Liu et al., 3 Nov 2025)	3D human motion	Temporal multi-scale residual quantization
“VQ-Style: Disentangling Style and Content in Motion with Residual Quantized Representations” (Zargarbashi et al., 2 Feb 2026)	Human motion	Coarse-to-fine residual codebook specialization
“Trajectory Forecasting through Low-Rank Adaptation of Discrete Latent Codes” (Benaglia et al., 2024)	Multi-agent forecasting	Trajectory-specific VQ, but not residual quantization
“HQ-VAE: Hierarchical Discrete Representation Learning with Variational Bayes” (Takida et al., 2023)	Hierarchical discrete learning	Bayesian generalization of RQ-style hierarchies

2. Residual quantization mechanism

The defining operation of an RQ-VAE is sequential quantization of residual error. In HTP, the encoder first maps the normalized GPS trajectory to a compressed sequence:

$$\mathbf{H} = \mathtt{Linear}(\tau) \in \mathbb{R}^{n \times d}, \qquad \mathbf{H}_e = \mathtt{Encoder}(\mathbf{H}) \in \mathbb{R}^{\frac{n}{8} \times d_e},$$

with implementation values $d=64$ and $d_e=256$ (Zhou et al., 28 May 2026). Quantization is applied after a linear projection

$$\mathbf{H}_q = \mathtt{Linear}(\mathbf{H}_e) \in \mathbb{R}^{\frac{n}{8} \times d_q},$$

where $d_q=64$ in the implementation (Zhou et al., 28 May 2026). Each segment embedding is quantized by nearest-neighbor lookup against a codebook, yielding a token sequence $K$ at one level. Residual quantization then applies multiple levels through

$$\mathbf{H}^{(l+1)}_q = \mathbf{H}^{(l)}_q - \mathbf{E}_q^{(l)},$$

starting from $\mathbf{H}^{(1)}_q = \mathbf{H}_q$, and reconstructs the latent by

$$\mathbf{H}_q^\prime = \mathtt{Linear}\left(\sum_{l=1}^{L}\mathbf{E}_q^{(l)}\right).$$

The model explicitly notes that $$\mathbf{H} = \mathtt{Linear}(\tau) \in \mathbb{R}^{n \times d}, \qquad \mathbf{H}_e = \mathtt{Encoder}(\mathbf{H}) \in \mathbb{R}^{\frac{n}{8} \times d_e},$$0 (Zhou et al., 28 May 2026).

MoSa states the same residual principle in motion form:

$$\mathbf{H} = \mathtt{Linear}(\tau) \in \mathbb{R}^{n \times d}, \qquad \mathbf{H}_e = \mathtt{Encoder}(\mathbf{H}) \in \mathbb{R}^{\frac{n}{8} \times d_e},$$1

$$\mathbf{H} = \mathtt{Linear}(\tau) \in \mathbb{R}^{n \times d}, \qquad \mathbf{H}_e = \mathtt{Encoder}(\mathbf{H}) \in \mathbb{R}^{\frac{n}{8} \times d_e},$$2

This is the explicit residual accumulation mechanism: each quantizer encodes the residual left by previous quantizers (Liu et al., 3 Nov 2025). VQ-Style expresses the same idea with residuals $$\mathbf{H} = \mathtt{Linear}(\tau) \in \mathbb{R}^{n \times d}, \qquad \mathbf{H}_e = \mathtt{Encoder}(\mathbf{H}) \in \mathbb{R}^{\frac{n}{8} \times d_e},$$3 and reconstruction from $$\mathbf{H} = \mathtt{Linear}(\tau) \in \mathbb{R}^{n \times d}, \qquad \mathbf{H}_e = \mathtt{Encoder}(\mathbf{H}) \in \mathbb{R}^{\frac{n}{8} \times d_e},$$4 (Zargarbashi et al., 2 Feb 2026).

The main representational consequence is compositional capacity. HTP makes this especially concrete by using $$\mathbf{H} = \mathtt{Linear}(\tau) \in \mathbb{R}^{n \times d}, \qquad \mathbf{H}_e = \mathtt{Encoder}(\mathbf{H}) \in \mathbb{R}^{\frac{n}{8} \times d_e},$$5 codebooks with sizes $$\mathbf{H} = \mathtt{Linear}(\tau) \in \mathbb{R}^{n \times d}, \qquad \mathbf{H}_e = \mathtt{Encoder}(\mathbf{H}) \in \mathbb{R}^{\frac{n}{8} \times d_e},$$6, deliberately setting $$\mathbf{H} = \mathtt{Linear}(\tau) \in \mathbb{R}^{n \times d}, \qquad \mathbf{H}_e = \mathtt{Encoder}(\mathbf{H}) \in \mathbb{R}^{\frac{n}{8} \times d_e},$$7 so that shallow layers encode coarse travel semantics and deeper layers refine them into finer movement details (Zhou et al., 28 May 2026). MoSa uses $$\mathbf{H} = \mathtt{Linear}(\tau) \in \mathbb{R}^{n \times d}, \qquad \mathbf{H}_e = \mathtt{Encoder}(\mathbf{H}) \in \mathbb{R}^{\frac{n}{8} \times d_e},$$8 quantizers and a finest latent length of 49 tokens, so the hierarchy is explicitly temporal as well as residual (Liu et al., 3 Nov 2025).

3. What makes the model trajectory-specific

Trajectory specificity does not arise from a new quantization rule; it arises from the surrounding representation, architecture, and loss design. In HTP, the encoder alternates CNN and Transformer layers, with CNNs capturing local movement patterns and Transformers capturing longer-range dependencies across the trajectory (Zhou et al., 28 May 2026). The encoder downsamples by $$\mathbf{H} = \mathtt{Linear}(\tau) \in \mathbb{R}^{n \times d}, \qquad \mathbf{H}_e = \mathtt{Encoder}(\mathbf{H}) \in \mathbb{R}^{\frac{n}{8} \times d_e},$$9 three times, following the dimensional path $d=64$0, thereby turning micro-level point sequences into macro-level segment embeddings (Zhou et al., 28 May 2026). The paper explicitly states that this is meant to “learn spatial irregularities from micro-level points to macro-level segment embeddings,” including point-density variation caused by traffic and movement changes.

A particularly trajectory-specific mechanism in HTP is variable-length correction under repeated $d=64$1 downsampling and upsampling. The model records the odd–even status at each downsampling stage as a 3-bit pattern. For a trajectory of length 26, the record is $d=64$2, which is then used to invert the process during decoding (Zhou et al., 28 May 2026). Later, these 3-bit records are also discretized into 8 length tokens for the LLM. This is not a generic VQ design choice; it is a sequence-length preservation device specific to hierarchical temporal compression.

The decoder in HTP is also trajectory-specific because it reconstructs road-relative signals rather than raw coordinates. An aligned road trajectory

$d=64$3

is used as external context, with road embeddings formed from Node2vec graph features and linestring geometry, then refined by a 4-layer Transformer (Zhou et al., 28 May 2026). The decoder predicts relative percent positions along the road trajectory and normalized offsets from road geometry, and these are combined with the road linestring to recover GPS coordinates. This design directly targets road adherence and point-density variation, rather than treating a trajectory as an unconstrained Euclidean sequence (Zhou et al., 28 May 2026).

MoSa shows a different but analogous specialization for motion trajectories. Its encoder processes the entire motion sequence rather than 64-frame windows because its Multi-scale Token Preservation Strategy requires perceiving the sequence across multiple scales (Liu et al., 3 Nov 2025). The finest motion latent length is 49 tokens, and the scale schedule

$d=64$4

implies a temporal abstraction ladder from three coarse tokens for the whole sequence to 49 fine-grained latent time steps (Liu et al., 3 Nov 2025). VQ-Style likewise uses a temporal sequence latent produced by a 1D convolutional encoder and reconstructs articulated motion with explicit velocity and acceleration losses (Zargarbashi et al., 2 Feb 2026). These motion results suggest that trajectory-specific RQ-VAE is fundamentally about preserving temporal hierarchy, not merely about discretizing any sequence.

4. Objectives, reconstruction targets, and the meaning of “VAE”

In HTP, the quantization loss is the standard VQ-style codebook plus commitment objective averaged over residual stages:

$d=64$5

with $d=64$6 in the appendix (Zhou et al., 28 May 2026). The reconstruction loss is not raw coordinate MSE over full map space. Instead, it is a “relative reconstruction loss” composed of relative percent loss and offset loss:

$d=64$7

and the final objective is

$d=64$8

The paper explicitly states that there is no KL term, adversarial term, or diffusion-style denoising objective (Zhou et al., 28 May 2026).

MoSa makes the same point from a different angle. Its preliminary VQ-VAE objective is

$d=64$9

and the hierarchical residual version uses

$d_e=256$0

$d_e=256$1

The paper explicitly notes that it does not give a formal probabilistic ELBO-style VAE derivation (Liu et al., 3 Nov 2025).

This is a recurring misconception in the area. In these models, “VAE” is largely inherited terminology rather than a reliable indicator of KL-regularized variational inference. HQ-VAE is the main exception in this set because it recasts hierarchical discrete representation learning in a variational-Bayes framework and introduces RSQ-VAE as a residual-top-down model with the same architectural structure as RQ-VAE but a different, Bayesian training scheme (Takida et al., 2023). Its final RSQ-VAE objective regularizes only the total accumulated approximation error rather than every intermediate prefix, precisely to avoid collapse in deeper layers (Takida et al., 2023). This suggests that the dominant practical question in trajectory-specific RQ-VAE is not whether the model is variational in a classical sense, but how the hierarchy is regularized and how codebooks remain usable.

5. Downstream interfaces: autoregression, LLMs, forecasting, and editing

The most visible use of trajectory-specific RQ-VAE is as a tokenizer for a more powerful sequence model. In HTP, the discrete travel-pattern tokens learned by the RQ-VAE are inserted into the LLM vocabulary as special tokens, together with road tokens, 8 length tokens, and boundary tokens (Zhou et al., 28 May 2026). During supervised fine-tuning, the LLM is trained to emit a length token and a travel-pattern token sequence rather than raw GPS coordinates. At inference time, the generated IDs are mapped back to codebook embeddings, summed across residual levels, and decoded into road-relative percent and offset signals, from which final GPS coordinates are reconstructed (Zhou et al., 28 May 2026). The paper reports that HTP outperforms the strongest baseline by an average of 29.78% in terms of generation quality on two real-world datasets (Zhou et al., 28 May 2026).

MoSa couples hierarchical residual quantization to Scalable Autoregressive modeling. Instead of standard token-by-token factorization,

$d_e=256$2

it uses scale-wise factorization,

$d_e=256$3

so one transformer step predicts all tokens at the next scale in parallel (Liu et al., 3 Nov 2025). Since $d_e=256$4, generation requires only 10 inference steps, matching the number of RQ-VAE quantization layers (Liu et al., 3 Nov 2025). This is a concrete demonstration that residual quantization can determine not only representation quality but also decoding complexity.

Other sequence domains use the same discrete interface differently. LRVQ trains a diffusion prior over trajectory token sequences, but it is explicitly not an RQ-VAE: it uses one trajectory-conditioned codebook per instance through low-rank adaptation rather than residual multistage quantization (Benaglia et al., 2024). VQ-Style uses residual codebooks for inference-time editing, swapping early codebooks from a content motion and later codebooks from a style motion to perform style transfer, style removal, or motion blending without fine-tuning (Zargarbashi et al., 2 Feb 2026). A plausible implication is that, for generic trajectories, residual codebooks can serve not only as compression devices but also as control axes for manipulating coarse path structure separately from fine local behavior.

6. Empirical properties, diagnostics, and open technical issues

The empirical record shows that residual depth, codebook allocation, and reconstruction target all matter. In HTP, increasing quantization layers from 1 to 4 consistently lowers validation reconstruction loss, with gains saturating after four layers (Zhou et al., 28 May 2026). The deployed configuration uses 4 residual codebooks of sizes $d_e=256$5, 100 epochs of Stage 1 training with AdamW, learning rate $d_e=256$6, cosine scheduling, and per-GPU batch size 256 on four A100 GPUs (Zhou et al., 28 May 2026). The ablation “w/o RQ-VAE,” which directly fine-tunes the LLM to generate raw GPS trajectories, degrades sharply on both Chengdu and Porto, and the “w/o R-Loss” ablation shows catastrophic degradation when the relative reconstruction loss is replaced with plain GPS-point MSE (Zhou et al., 28 May 2026). Those results isolate the discrete hierarchy and the trajectory-specific reconstruction target as critical components.

MoSa reports an analogous trade-off between deeper quantization and downstream modelability. Increasing $d_e=256$7 improves reconstruction, but too large $d_e=256$8 hurts generation because the token hierarchy becomes harder for the transformer to model; the paper finds $d_e=256$9 to be the best balance (Liu et al., 3 Nov 2025). VQ-Style shows the same diminishing-returns pattern in motion reconstruction: for RVQ-8-512, reconstruction error decreases from 0.100 with 1 codebook to 0.036 with 8 codebooks (Zargarbashi et al., 2 Feb 2026). These results suggest a general design constraint: deeper residual hierarchies increase expressiveness, but the resulting stacked token space may become harder for a prior, autoregressor, or LLM to learn.

A second technical issue is codebook collapse and boundary instability. HQ-VAE argues that deterministic hierarchical VQ training often relies on heuristics such as EMA, code reset, and stop-gradient, and that naive residual regularization can push too much information into early layers and collapse later ones (Takida et al., 2023). DRQ extends this discussion with explicit diagnostics for token robustness: expected codeword overlap

$$\mathbf{H}_q = \mathtt{Linear}(\mathbf{H}_e) \in \mathbb{R}^{\frac{n}{8} \times d_q},$$0

and effective codebook capacity

$$\mathbf{H}_q = \mathtt{Linear}(\mathbf{H}_e) \in \mathbb{R}^{\frac{n}{8} \times d_q},$$1

These quantities decompose symbolic failure into usage imbalance and geometry-sensitive overlap (Wang et al., 1 Jun 2026). Although DRQ is item-centric rather than trajectory-conditioned, its analysis is directly relevant: a trajectory tokenizer can fail because codes are underused, because their decision boundaries are unstable under perturbation, or because Euclidean residual codebooks distort a curved latent manifold (Wang et al., 1 Jun 2026).

A final misconception concerns what counts as “trajectory-specific.” LRVQ is trajectory-specific in the sense of per-instance codebook adaptation, but it is not an RQ-VAE because it performs only one quantization lookup per latent position and does not quantize residuals stage by stage (Benaglia et al., 2024). Conversely, motion models such as MoSa and VQ-Style are not generic GPS models, yet they are highly informative for trajectory-specific RQ-VAE because they show how residual codebooks can align with temporal abstraction, content/style separation, and scalable generation (Liu et al., 3 Nov 2025, Zargarbashi et al., 2 Feb 2026).

Taken together, the literature supports a precise characterization. A trajectory-specific RQ-VAE is a sequential discrete latent autoencoder in which full trajectories or compressed trajectory segments are encoded into temporally structured latent vectors, approximated by additive residual codebooks, and decoded with reconstruction objectives that respect trajectory geometry, temporal scale, or road/skeleton constraints. The strongest directly demonstrated instance is HTP’s GPS tokenizer (Zhou et al., 28 May 2026). The broader research picture, however, indicates that the central open problems are hierarchical token semantics, codebook utilization, geometry preservation, and the interface between residual tokens and downstream sequence models (Liu et al., 3 Nov 2025, Takida et al., 2023, Wang et al., 1 Jun 2026).