---
title: 'NOVA: Next-Step Open-Vocabulary 3D MOT'
url: https://www.emergentmind.com/topics/next-step-open-vocabulary-autoregression-nova
type: topic
---

# NOVA: Next-Step Open-Vocabulary 3D MOT

Searching arXiv for the specified NOVA paper and closely related tracking work.
arxiv_search(query="2603.06254 OR \"Next-step Open-Vocabulary Autoregression\" OR \"3D Multi-Object Tracking\" open-vocabulary autonomous driving", max_results=10, sort_by="submittedDate")
Refining the arXiv search to retrieve the target paper and nearby open-vocabulary 3D MOT literature.
arxiv_search(query="\"NOVA: Next-step Open-Vocabulary Autoregression for 3D Multi-Object Tracking in Autonomous Driving\"", max_results=5, sort_by="relevance")
Looking up the baseline Open3DTrack paper to support the historical positioning of NOVA.
arxiv_search(query="Open3DTrack open-vocabulary 3D MOT", max_results=10, sort_by="relevance")
Next-step Open-Vocabulary Autoregression (NOVA) is a paradigm for open-vocabulary 3D multi-object tracking in autonomous driving that reformulates association as autoregressive next-step sequence completion over serialized spatio-temporal semantic trajectories rather than as fragmented distance-based matching [2603.06254]. Its stated objective is to preserve identity consistency across both base and novel categories under semantic uncertainty, noisy 3D geometry, crowding, and occlusion by combining physical motion continuity with linguistic priors inside a compact autoregressive model. The acronym “NOVA” is also used by an unrelated training-free acceleration framework for Visual AutoRegressive image generation; in the present usage, NOVA denotes the tracking formulation introduced for 3D MOT [2602.01345].

## 1. Problem setting and conceptual scope

NOVA targets open-vocabulary 3D multi-object tracking for autonomous driving, specifically the online tracking of both base classes and novel classes when the upstream detector produces open-vocabulary labels [2603.06254]. The method is motivated by the claim that conventional 3D MOT pipelines remain constrained by closed-set assumptions and “semantic-blind” heuristics: they either suppress unseen categories or rely on a two-stage design in which 2D semantics are projected onto 3D proposals from a closed-set detector. Under such conditions, instability in open-vocabulary detections propagates downstream into fragmented trajectories, identity switches, class switches, and failures on novel objects.

The central conceptual shift is from hand-designed geometric association to generative spatio-temporal semantic modeling. Rather than treating tracking as the frame-wise optimization of Euclidean distance, IoU, or motion costs, NOVA treats association as trajectory-conditioned reasoning. The object history serves as a prefix, and the candidate continuation is evaluated as the next step in an autoregressive sequence. The paper characterizes this as a transition from traditional fragmented distance-based matching toward a language-model-style prediction process grounded in both semantics and geometry [2603.06254].

A common misunderstanding is that open-vocabulary tracking is only a detection problem. NOVA is premised on a different diagnosis: the principal challenge is association under open-world category shifts. The framework therefore concentrates on maintaining identity consistency when category names are uncertain or unfamiliar, the geometry is noisy or sparse, and the category space is effectively unbounded.

## 2. Trajectories as structured spatio-temporal semantic sequences

A defining element of NOVA is the representation of a 3D trajectory as a structured “spatio-semantic sentence” [2603.06254]. Each track is serialized as a temporal sequence of states, and each state contains the object’s 3D box geometry, its confidence or quality, and its semantic label or open-vocabulary placeholder. In the paper’s terminology, this yields a sequence that combines physical continuity—motion, position, size, yaw, and overlap over time—with linguistic structure via class names or masked semantic tokens.

This serialization is not merely a formatting choice. It is the mechanism by which an autoregressive language model can process tracking context as a sequence-completion problem. The object history becomes a context window analogous to a sentence prefix, and the association decision becomes the next token to be inferred. The paper’s interpretation is that association is not purely a metric-space problem but a contextual prediction problem in which the history of the trajectory supplies the relevant conditioning information [2603.06254].

The formal input at time \(t\) is the detector output
\[
D_t = \{b_t^j\}_{j=1}^{N_t},
\]
with each box represented as
\[
b_t^j = (x,y,z,l,w,h,\theta,s,c),
\]
where \((x,y,z)\) is the center, \((l,w,h)\) the size, \(\theta\) the yaw, \(s\) the confidence, and \(c \in \mathcal{C}_{base} \cup \mathcal{C}_{novel}\) the open-vocabulary class label. The tracker maintains active trajectories
\[
\mathcal{T}_{t-1} = \{T_i\}_{i=1}^{M_{t-1}},
\]
each storing a short history of previous associated boxes.

A plausible implication is that the representation expands the admissible evidence for association beyond local motion smoothness. Because the sequence jointly carries spatial, temporal, and semantic structure, the model can evaluate whether a candidate continuation is coherent at multiple levels of abstraction rather than only geometrically proximate.

## 3. Geometry encoding, prompting, and open-vocabulary conditioning

NOVA does not rely on plain text serialization of numeric coordinates. The paper argues that naive stringification of 3D box values is fragile because tokenization of numeric values is lossy and sensitive to small jitter [2603.06254]. To avoid this, it introduces a Geometry Encoder that maps continuous box features into the LLM embedding space. The raw feature vector is
\[
f_{\text{raw}} = [x,y,z,l,w,h,\text{vol},\theta,s] \in \mathbb{R}^9,
\]
where \(\text{vol}=lwh\). This vector is projected to a learned embedding \(E_{\text{geo}}\), which is injected into the LLM at special `<box>` token positions.

The prompting strategy is explicitly hybrid. For base classes, the real class name is retained in the prompt; for novel classes, the label is replaced with “Unknown.” The stated purpose is to force the model to learn class-agnostic association cues for novel instances instead of overfitting to base-class semantic names [2603.06254]. The paper presents this as a direct response to the open-vocabulary setting, where unstable or unfamiliar labels otherwise induce brittle downstream behavior.

The role of the LLM is correspondingly narrow and technical. It is not deployed as a generic text generator but as a reasoning engine for association. The paper attributes three advantages to this design: semantic priors derived from large-scale language regularities, commonsense reasoning about object persistence, and access to a hierarchical language space in which coarse and fine category labels can remain mutually informative. Categories such as vehicle \(\rightarrow\) car \(\rightarrow\) SUV illustrate the relevant granularity structure. The model can therefore fall back to broader semantic compatibility when detailed labels fluctuate [2603.06254].

A frequent misconception is that NOVA depends solely on semantics. The method is explicitly geometric as well as semantic: it injects continuous geometry embeddings, conditions on trajectory history, and uses semantic tokens as one component of a broader spatio-temporal representation.

## 4. Autoregressive association, objective functions, and online tracking pipeline

For each active trajectory \(T_i\) and each current detection \(b_t^j\), NOVA forms a prompt \(\mathcal{P}(T_i,b_t^j)\) by serializing recent trajectory history and appending the candidate detection as a query [2603.06254]. The model is trained to answer a binary question: whether the detection belongs to the track. The core association score is
\[
p_{i,j} = \Pr(Yes \mid \mathcal{P}(T_i,b_t^j)).
\]
In this formulation, association is reduced to next-step decision-token prediction.

The binary target is
\[
y_{i,j} \in \{0,1\}, \quad y_{i,j}=1 \Leftrightarrow b_t^j \text{ matches } T_i.
\]
At inference, match probabilities are converted to costs, typically
\[
C_{i,j} = 1 - p_{i,j},
\]
and standard Hungarian matching is used to obtain one-to-one track–detection correspondence. Matched detections update tracks, unmatched detections may initialize new tracks, and unmatched tracks are retained for up to \(K\) missed frames before termination [2603.06254]. NOVA therefore remains an online tracking-by-detection system in lifecycle management even though its association score is generative.

The training objective combines binary generation loss with IoU-quality regression:
\[
\mathcal{L} = \mathcal{L}_{\text{gen}} + \lambda_{\text{reg}} \mathcal{L}_{\text{iou}},
\]
where \(\mathcal{L}_{\text{gen}}\) is cross-entropy over the Yes/No decision token and \(\mathcal{L}_{\text{iou}}\) is an MSE loss between the predicted quality score and the IoU target in \([0,1]\) [2603.06254]. Training data include positive matched pairs and hard negatives, with negative sampling biased toward spatially proximate but identity-inconsistent detections. Mild geometric jitter is added to positives so that the model learns tolerance to localization noise and drift.

The resulting pipeline is concise: obtain open-vocabulary detections; serialize each track history and candidate detection; encode box geometry with the Geometry Encoder; apply hybrid prompting with either base labels or “Unknown”; compute the LLM probability of “Yes” for each pair; perform Hungarian assignment; and update tracks online [2603.06254].

## 5. Datasets, evaluation protocol, and reported empirical behavior

NOVA is evaluated on nuScenes, V2X-Seq-SPD, and KITTI, with classes split into Base and Novel partitions to explicitly measure open-vocabulary tracking performance [2603.06254]. The reported metrics are sAMOTA when available, otherwise AMOTA, together with AMOTP, MOTA, MOTP, and MT. A unified 3D IoU threshold of 0.25 is used for matching.

| Dataset | Base | Novel |
|---|---|---|
| nuScenes | Car, Trailer, Pedestrian, Bicycle | Truck, Bus, Motorcycle |
| V2X-Seq-SPD | Car, Van, Pedestrian, Motorcyclist | Bus, Truck, Cyclist, Tricyclist |
| KITTI | Car, Cyclist | Pedestrian |

The primary comparison baseline is Open3DTrack, and NOVA is tested with upstream open-vocabulary detectors including Find n’ Propagate, GroundingDINO, and YOLO-World [2603.06254]. The headline quantitative result is on nuScenes novel categories: Open3DTrack attains Novel AMOTA \(= 2.20\%\), whereas NOVA reaches Novel AMOTA \(= 22.41\%\), an absolute gain of \(20.21\) percentage points. On nuScenes base classes, Base AMOTA changes from \(52.30\) to \(48.87\) relative to Open3DTrack, which the paper interprets as a trade-off in favor of novel-category robustness. On V2X-Seq-SPD, with Find n’ Propagate + GroundingDINO, Base sAMOTA improves from \(26.50\) to \(68.17\), and Novel sAMOTA from \(11.07\) to \(22.95\). With Find n’ Propagate + YOLO-World, NOVA still improves Base performance and remains competitive on Novel classes. On KITTI, Open3DTrack reportedly degenerates, while NOVA achieves meaningful Base and Novel tracking performance [2603.06254].

The reported backbone is Qwen2.5-0.5B, fine-tuned with LoRA on RTX 3090 GPUs for a default 8 epochs, with default trajectory history length \(3\) [2603.06254]. The paper emphasizes that the compact \(0.5\)B autoregressive model attains the best overall and novel-category performance among the tested backbones, while running at about \(3.4\) FPS on the reported setup. It further states that larger \(3\)B+ models can be more conservative, reducing identity switches but increasing false negatives and slowing inference.

## 6. Ablations, interpretation, and limitations

The ablation studies are used to localize which design choices account for the reported performance [2603.06254]. Geometry encoding outperforms text-only coordinate serialization, indicating that continuous geometric representation is important for novel-category robustness. Adding the IoU auxiliary supervision yields a smaller but meaningful improvement and is described as reducing false negatives and improving MOTA. Hybrid prompting is one of the strongest ablations: if novel classes are ignored or discarded during training, novel sAMOTA is \(14.63\%\); if novel classes are labeled as “Unknown,” novel sAMOTA rises to \(22.95\%\). The paper interprets this as evidence that explicit training on ambiguous novel examples strengthens class-agnostic association cues. Hard negative mining outperforms random or merely local negative sampling, and positive jitter augmentation improves robustness to noisy detections. History-length ablation shows that no history causes collapse, one frame yields a large improvement, three frames gives the best trade-off, and five frames becomes overly conservative and reduces recall.

These findings motivate the paper’s broader explanation of why NOVA improves over distance-based or closed-set heuristics. Traditional trackers rely on frame-wise similarity terms; NOVA instead reasons over the sequence of past states and predicts whether a continuation “makes sense” in context [2603.06254]. This suggests that the gain comes not from replacing one matching cost with another, but from changing the representational unit of association from isolated detections to trajectory-conditioned semantic sequences.

Several limitations are explicit. NOVA still depends on the quality of upstream open-vocabulary detections; its inference procedure still uses gating plus Hungarian matching, so it is not a fully end-to-end detector-tracker; and the authors identify future work on lightweight appearance cues, heavier occlusion, and broader end-to-end autonomous driving reasoning [2603.06254]. Another common misconception is that the method is a general-purpose “LLM tracker.” In the reported formulation, it is more precise to describe NOVA as a tracking-by-detection system whose association module is recast as next-step open-vocabulary autoregression over geometry-grounded prompts.

In the literature represented here, NOVA’s principal significance lies in the formula “trajectory = spatio-temporal semantic sequence” and in the claim that open-vocabulary 3D MOT benefits from treating association as a language-like generative reasoning problem rather than as a purely geometric assignment problem [2603.06254].

Source: https://www.emergentmind.com/topics/next-step-open-vocabulary-autoregression-nova