---
title: Trajectory-Specific RQ-VAE
url: https://www.emergentmind.com/topics/trajectory-specific-residual-quantization-variational-autoencoder-rq-vae
type: topic
---

# Trajectory-Specific RQ-VAE

Searching arXiv for recent papers directly relevant to trajectory-specific residual quantization VAEs and closely related residual quantization sequence models.
Trajectory-Specific Residual Quantization Variational Autoencoder (RQ-VAE) denotes a class of discrete latent autoencoding systems in which a trajectory is encoded as a temporal or segment-level latent sequence and then quantized by multiple residual codebooks in a coarse-to-fine manner. In the most explicit usage of the term, HTP introduces a “trajectory-specific residual quantization variational autoencoder (RQ-VAE)” for GPS trajectories that converts raw, variable-length point sequences into compact “travel pattern” tokens for downstream large language model generation [2605.30014]. Closely related work on human motion shows the same core principle in other sequential domains: residual quantization is used to decompose a motion latent into progressively finer temporal abstractions, later consumed by autoregressive or editing modules [2511.01200], [2602.02334]. Across these systems, the “VAE” label is conventional; the operative training objective is typically reconstruction plus VQ/RQ commitment terms rather than a classical ELBO with a KL regularizer [2605.30014], [2511.01200].

## 1. Conceptual scope and historical placement

A trajectory-specific RQ-VAE is best understood as the intersection of two ideas. The first is residual quantization, in which a latent vector is approximated by a sum of code vectors chosen stage by stage from multiple quantizers. The second is trajectory specialization, meaning that the encoder, decoder, losses, and token semantics are designed around sequential structure rather than around images or generic token streams. A representative early formulation of RQ-VAE in image generation quantized each latent position with a stack of discrete codes, using residual recursion and additive reconstruction to preserve fidelity at low latent resolution [2203.01941]. Trajectory-oriented work retains the residual mechanism but changes what is being encoded: temporal slots, motion segments, or GPS segments rather than image patches.

The most direct trajectory-specific instantiation appears in HTP, where a trajectory is defined as
$$
\tau = \langle p_i \rangle_{i=1}^{n}, \qquad p_i = (x_i, y_i),
$$
with longitude and latitude only, and the model explicitly avoids direct GPS-point generation by the language model [2605.30014]. The stated rationale is twofold: direct GPS generation is too fine-grained because tiny coordinate errors accumulate autoregressively and cause spatial drift, and raw GPS sequences are too long and expensive for an LLM to emit token by token. The RQ-VAE therefore functions as a bridge between micro-level GPS points and macro-level travel-pattern tokens [2605.30014].

Related sequential models clarify the broader meaning of “trajectory-specific.” MoSa is framed as text-driven 3D human motion generation rather than GPS modeling, but it uses a hierarchical residual quantization VAE tailored to temporal motion sequences, with tokens corresponding to latent temporal slots and scales corresponding to increasingly fine temporal structure [2511.01200]. VQ-Style similarly treats motion as a temporal signal and uses residual codebooks as a coarse-to-fine factorization in which early stages capture content and later stages capture style/detail [2602.02334]. This suggests that “trajectory-specific RQ-VAE” is less a single canonical architecture than a design pattern for sequential data whose latent structure is inherently hierarchical.

| Paper | Domain | Relevance |
|---|---|---|
| “From GPS Points to Travel Patterns: Flexible and Semantic Trajectory Generation with LLMs” [2605.30014] | Urban GPS trajectories | Explicit trajectory-specific RQ-VAE |
| “MoSa: Motion Generation with Scalable Autoregressive Modeling” [2511.01200] | 3D human motion | Temporal multi-scale residual quantization |
| “VQ-Style: Disentangling Style and Content in Motion with Residual Quantized Representations” [2602.02334] | Human motion | Coarse-to-fine residual codebook specialization |
| “Trajectory Forecasting through Low-Rank Adaptation of Discrete Latent Codes” [2405.20743] | Multi-agent forecasting | Trajectory-specific VQ, but not residual quantization |
| “HQ-VAE: Hierarchical Discrete Representation Learning with Variational Bayes” [2401.00365] | Hierarchical discrete learning | Bayesian generalization of RQ-style hierarchies |

## 2. Residual quantization mechanism

The defining operation of an RQ-VAE is sequential quantization of residual error. In HTP, the encoder first maps the normalized GPS trajectory to a compressed sequence:
$$
\mathbf{H} = \mathtt{Linear}(\tau) \in \mathbb{R}^{n \times d}, \qquad
\mathbf{H}_e = \mathtt{Encoder}(\mathbf{H}) \in \mathbb{R}^{\frac{n}{8} \times d_e},
$$
with implementation values \(d=64\) and \(d_e=256\) [2605.30014]. Quantization is applied after a linear projection
$$
\mathbf{H}_q = \mathtt{Linear}(\mathbf{H}_e) \in \mathbb{R}^{\frac{n}{8} \times d_q},
$$
where \(d_q=64\) in the implementation [2605.30014]. Each segment embedding is quantized by nearest-neighbor lookup against a codebook, yielding a token sequence \(K\) at one level. Residual quantization then applies multiple levels through
$$
\mathbf{H}^{(l+1)}_q = \mathbf{H}^{(l)}_q - \mathbf{E}_q^{(l)},
$$
starting from \(\mathbf{H}^{(1)}_q = \mathbf{H}_q\), and reconstructs the latent by
$$
\mathbf{H}_q^\prime = \mathtt{Linear}\left(\sum_{l=1}^{L}\mathbf{E}_q^{(l)}\right).
$$
The model explicitly notes that \(\mathbf{H}_q \approx \sum_{l=1}^{L}\mathbf{E}_q^{(l)}\) [2605.30014].

MoSa states the same residual principle in motion form:
$$
x^{(q)} = \mathrm{Quant}^{(q)}(z^{(q)}), \qquad
\hat{z}^{(q)} = \mathrm{Dequant}(Z^{(q)}, x^{(q)}),
$$
$$
z^{(q+1)} = z^{(q)} - \hat{z}^{(q)}, \qquad
\hat{z} = \sum_{q=1}^Q \hat{z}^{(q)}.
$$
This is the explicit residual accumulation mechanism: each quantizer encodes the residual left by previous quantizers [2511.01200]. VQ-Style expresses the same idea with residuals \(\mathbf{r}_i = \mathbf{r}_{i-1} - \mathbf{z}_{i-1}\) and reconstruction from \(\sum_{j=0}^{n-1}\mathbf{z}_j\) [2602.02334].

The main representational consequence is compositional capacity. HTP makes this especially concrete by using \(L=4\) codebooks with sizes \([32,64,128,256]\), deliberately setting \(C_1 < C_2 < \cdots < C_L\) so that shallow layers encode coarse travel semantics and deeper layers refine them into finer movement details [2605.30014]. MoSa uses \(Q=10\) quantizers and a finest latent length of 49 tokens, so the hierarchy is explicitly temporal as well as residual [2511.01200].

## 3. What makes the model trajectory-specific

Trajectory specificity does not arise from a new quantization rule; it arises from the surrounding representation, architecture, and loss design. In HTP, the encoder alternates CNN and Transformer layers, with CNNs capturing local movement patterns and Transformers capturing longer-range dependencies across the trajectory [2605.30014]. The encoder downsamples by \(2\times\) three times, following the dimensional path \(64 \rightarrow 128 \rightarrow 128 \rightarrow 256\), thereby turning micro-level point sequences into macro-level segment embeddings [2605.30014]. The paper explicitly states that this is meant to “learn spatial irregularities from micro-level points to macro-level segment embeddings,” including point-density variation caused by traffic and movement changes.

A particularly trajectory-specific mechanism in HTP is variable-length correction under repeated \(2\times\) downsampling and upsampling. The model records the odd–even status at each downsampling stage as a 3-bit pattern. For a trajectory of length 26, the record is \((0,1,1)\), which is then used to invert the process during decoding [2605.30014]. Later, these 3-bit records are also discretized into 8 length tokens for the LLM. This is not a generic VQ design choice; it is a sequence-length preservation device specific to hierarchical temporal compression.

The decoder in HTP is also trajectory-specific because it reconstructs road-relative signals rather than raw coordinates. An aligned road trajectory
$$
\tau_r = \langle v_i \rangle_{i=1}^{n_r}
$$
is used as external context, with road embeddings formed from Node2vec graph features and linestring geometry, then refined by a 4-layer Transformer [2605.30014]. The decoder predicts relative percent positions along the road trajectory and normalized offsets from road geometry, and these are combined with the road linestring to recover GPS coordinates. This design directly targets road adherence and point-density variation, rather than treating a trajectory as an unconstrained Euclidean sequence [2605.30014].

MoSa shows a different but analogous specialization for motion trajectories. Its encoder processes the entire motion sequence rather than 64-frame windows because its Multi-scale Token Preservation Strategy requires perceiving the sequence across multiple scales [2511.01200]. The finest motion latent length is 49 tokens, and the scale schedule
$$
S=(3,6,10,15,20,25,30,36,42,49)
$$
implies a temporal abstraction ladder from three coarse tokens for the whole sequence to 49 fine-grained latent time steps [2511.01200]. VQ-Style likewise uses a temporal sequence latent produced by a 1D convolutional encoder and reconstructs articulated motion with explicit velocity and acceleration losses [2602.02334]. These motion results suggest that trajectory-specific RQ-VAE is fundamentally about preserving temporal hierarchy, not merely about discretizing any sequence.

## 4. Objectives, reconstruction targets, and the meaning of “VAE”

In HTP, the quantization loss is the standard VQ-style codebook plus commitment objective averaged over residual stages:
$$
\mathcal{L}_{q} = \frac{1}{L} \sum_{l = 1}^{L} \left( \|\mathtt{sg}(\mathbf{H}_q^{(l)}) - \mathbf{E}_q^{(l)}\|^2_2 + \beta \|\mathbf{H}_q^{(l)}- \mathtt{sg}(\mathbf{E}_q^{(l)})\|^2_2 \right),
$$
with \(\beta=0.25\) in the appendix [2605.30014]. The reconstruction loss is not raw coordinate MSE over full map space. Instead, it is a “relative reconstruction loss” composed of relative percent loss and offset loss:
$$
\mathcal{L}_{r} = \mathcal{L}_\mathrm{percent} + \mathcal{L}_\mathrm{offset},
$$
and the final objective is
$$
\mathcal{L} = \mathcal{L}_{q} + \mathcal{L}_{r}.
$$
The paper explicitly states that there is no KL term, adversarial term, or diffusion-style denoising objective [2605.30014].

MoSa makes the same point from a different angle. Its preliminary VQ-VAE objective is
$$
\mathcal{L}_{\text{vq}} = \|m - \hat{m} \|_1 + \|sg[z] - \hat{z}\|_2 + \beta \|z - sg[\hat{z}]\|_2,
$$
and the hierarchical residual version uses
$$
\mathcal{L}_{\text{commit}}=\sum_{q=1}^Q \left( \|sg[z^{(q)}] - \hat{z}^{(q)}\|_2 + \beta \|z^{(q)} - sg[\hat{z}^{(q)}]\|_2 \right),
$$
$$
\mathcal{L}_{\text{rvq}}= \|m - \hat{m} \|_1 + \mathcal{L}_{\text{commit}}.
$$
The paper explicitly notes that it does not give a formal probabilistic ELBO-style VAE derivation [2511.01200].

This is a recurring misconception in the area. In these models, “VAE” is largely inherited terminology rather than a reliable indicator of KL-regularized variational inference. HQ-VAE is the main exception in this set because it recasts hierarchical discrete representation learning in a variational-Bayes framework and introduces RSQ-VAE as a residual-top-down model with the same architectural structure as RQ-VAE but a different, Bayesian training scheme [2401.00365]. Its final RSQ-VAE objective regularizes only the total accumulated approximation error rather than every intermediate prefix, precisely to avoid collapse in deeper layers [2401.00365]. This suggests that the dominant practical question in trajectory-specific RQ-VAE is not whether the model is variational in a classical sense, but how the hierarchy is regularized and how codebooks remain usable.

## 5. Downstream interfaces: autoregression, language models, forecasting, and editing

The most visible use of trajectory-specific RQ-VAE is as a tokenizer for a more powerful sequence model. In HTP, the discrete travel-pattern tokens learned by the RQ-VAE are inserted into the LLM vocabulary as special tokens, together with road tokens, 8 length tokens, and boundary tokens [2605.30014]. During supervised fine-tuning, the LLM is trained to emit a length token and a travel-pattern token sequence rather than raw GPS coordinates. At inference time, the generated IDs are mapped back to codebook embeddings, summed across residual levels, and decoded into road-relative percent and offset signals, from which final GPS coordinates are reconstructed [2605.30014]. The paper reports that HTP outperforms the strongest baseline by an average of 29.78% in terms of generation quality on two real-world datasets [2605.30014].

MoSa couples hierarchical residual quantization to Scalable Autoregressive modeling. Instead of standard token-by-token factorization,
$$
p(x_1, x_2, \dots, x_T \mid c) = \prod_{t=1}^{T} p(x_t \mid x_1, x_2, \dots, x_{t-1}, c),
$$
it uses scale-wise factorization,
$$
p(x^{(1)}, x^{(2)}, \dots, x^{(Q)} \mid c) = \prod_{q=1}^{Q} p(x^{(q)} \mid x^{(1)}, x^{(2)}, \dots, x^{(q-1)}, c),
$$
so one transformer step predicts all tokens at the next scale in parallel [2511.01200]. Since \(Q=10\), generation requires only 10 inference steps, matching the number of RQ-VAE quantization layers [2511.01200]. This is a concrete demonstration that residual quantization can determine not only representation quality but also decoding complexity.

Other sequence domains use the same discrete interface differently. LRVQ trains a diffusion prior over trajectory token sequences, but it is explicitly not an RQ-VAE: it uses one trajectory-conditioned codebook per instance through low-rank adaptation rather than residual multistage quantization [2405.20743]. VQ-Style uses residual codebooks for inference-time editing, swapping early codebooks from a content motion and later codebooks from a style motion to perform style transfer, style removal, or motion blending without fine-tuning [2602.02334]. A plausible implication is that, for generic trajectories, residual codebooks can serve not only as compression devices but also as control axes for manipulating coarse path structure separately from fine local behavior.

## 6. Empirical properties, diagnostics, and open technical issues

The empirical record shows that residual depth, codebook allocation, and reconstruction target all matter. In HTP, increasing quantization layers from 1 to 4 consistently lowers validation reconstruction loss, with gains saturating after four layers [2605.30014]. The deployed configuration uses 4 residual codebooks of sizes \([32,64,128,256]\), 100 epochs of Stage 1 training with AdamW, learning rate \(10^{-4}\), cosine scheduling, and per-GPU batch size 256 on four A100 GPUs [2605.30014]. The ablation “w/o RQ-VAE,” which directly fine-tunes the LLM to generate raw GPS trajectories, degrades sharply on both Chengdu and Porto, and the “w/o R-Loss” ablation shows catastrophic degradation when the relative reconstruction loss is replaced with plain GPS-point MSE [2605.30014]. Those results isolate the discrete hierarchy and the trajectory-specific reconstruction target as critical components.

MoSa reports an analogous trade-off between deeper quantization and downstream modelability. Increasing \(Q\) improves reconstruction, but too large \(Q\) hurts generation because the token hierarchy becomes harder for the transformer to model; the paper finds \(Q=10\) to be the best balance [2511.01200]. VQ-Style shows the same diminishing-returns pattern in motion reconstruction: for RVQ-8-512, reconstruction error decreases from 0.100 with 1 codebook to 0.036 with 8 codebooks [2602.02334]. These results suggest a general design constraint: deeper residual hierarchies increase expressiveness, but the resulting stacked token space may become harder for a prior, autoregressor, or LLM to learn.

A second technical issue is codebook collapse and boundary instability. HQ-VAE argues that deterministic hierarchical VQ training often relies on heuristics such as EMA, code reset, and stop-gradient, and that naive residual regularization can push too much information into early layers and collapse later ones [2401.00365]. DRQ extends this discussion with explicit diagnostics for token robustness: expected codeword overlap
$$
O_\pi = \sum_{i=1}^{K}\sum_{j=1}^{K} \pi_i\pi_j \exp\left(-\frac{\|c_i-c_j\|^2}{4\sigma^2}\right)
$$
and effective codebook capacity
$$
K_{\text{eff}} = \frac{1}{O_\pi}.
$$
These quantities decompose symbolic failure into usage imbalance and geometry-sensitive overlap [2606.01844]. Although DRQ is item-centric rather than trajectory-conditioned, its analysis is directly relevant: a trajectory tokenizer can fail because codes are underused, because their decision boundaries are unstable under perturbation, or because Euclidean residual codebooks distort a curved latent manifold [2606.01844].

A final misconception concerns what counts as “trajectory-specific.” LRVQ is trajectory-specific in the sense of per-instance codebook adaptation, but it is not an RQ-VAE because it performs only one quantization lookup per latent position and does not quantize residuals stage by stage [2405.20743]. Conversely, motion models such as MoSa and VQ-Style are not generic GPS models, yet they are highly informative for trajectory-specific RQ-VAE because they show how residual codebooks can align with temporal abstraction, content/style separation, and scalable generation [2511.01200], [2602.02334].

Taken together, the literature supports a precise characterization. A trajectory-specific RQ-VAE is a sequential discrete latent autoencoder in which full trajectories or compressed trajectory segments are encoded into temporally structured latent vectors, approximated by additive residual codebooks, and decoded with reconstruction objectives that respect trajectory geometry, temporal scale, or road/skeleton constraints. The strongest directly demonstrated instance is HTP’s GPS tokenizer [2605.30014]. The broader research picture, however, indicates that the central open problems are hierarchical token semantics, codebook utilization, geometry preservation, and the interface between residual tokens and downstream sequence models [2511.01200], [2401.00365], [2606.01844].

Source: https://www.emergentmind.com/topics/trajectory-specific-residual-quantization-variational-autoencoder-rq-vae