NOVA: Next-Step Open-Vocabulary 3D MOT

• The paper introduces a paradigm that reformulates 3D MOT association as next-step autoregressive sequence completion to maintain identity consistency under semantic uncertainty.
• It integrates geometric and linguistic features by employing a Geometry Encoder and hybrid prompting to serialize spatio-temporal semantic trajectories.
• Empirical evaluations on nuScenes, V2X-Seq-SPD, and KITTI demonstrate significant gains in novel-category tracking performance compared to traditional distance-based methods.

Searching arXiv for the specified NOVA paper and closely related tracking work. arxiv_search(query="^^^^^^^^5(Luo et al., 6 Mar 2026) OR \5^^^^^^^^"Next-step Open-Vocabulary Autoregression^^^^^^^^5\5 OR \5^^^^^^^^"^^^^5submittedDate5 Multi-Object Tracking^^^^^^^^5\5 open-vocabulary autonomous driving5^^^^^^^^", max_results=^^^^5 OR \5(Luo et al., 6 Mar 2026) OR \5^^^^, sort_by="^^^^^^^^5submittedDate5 Refining the arXiv search to retrieve the target paper and nearby open-vocabulary ^^^^5submittedDate5^^^^D MOT literature. arxiv_search(query="^^^^^^^^5\5 Next-step Open-Vocabulary Autoregression for ^^^^5submittedDate5^^^^D Multi-Object Tracking in Autonomous Driving^^^^^^^^5\5 max_results=5, sort_by="relevance") Looking up the baseline Open^^^^5submittedDate5^^^^DTrack paper to support the historical positioning of NOVA. arxiv_search(query="Open^^^^5submittedDate5^^^^DTrack open-vocabulary ^^^^5submittedDate5^^^^D MOT", max_results=^^^^5 OR \5(Luo et al., 6 Mar 2026) OR \5^^^^, sort_by="relevance") Next-step Open-Vocabulary Autoregression (NOVA) is a paradigm for open-vocabulary ^^^^5submittedDate5^^^^D multi-object tracking in autonomous driving that reformulates association as autoregressive next-step sequence completion over serialized spatio-temporal semantic trajectories rather than as fragmented distance-based matching (&&&^^^^5(Luo et al., 6 Mar 2026) OR \5^^^^&&&). Its stated objective is to preserve identity consistency across both base and novel categories under semantic uncertainty, noisy ^^^^5submittedDate5^^^^D geometry, crowding, and occlusion by combining physical motion continuity with linguistic priors inside a compact autoregressive model. The acronym “NOVA” is also used by an unrelated training-free acceleration framework for Visual AutoRegressive image generation; in the present usage, NOVA denotes the tracking formulation introduced for ^^^^5submittedDate5^^^^D MOT (&&&^^^^5 open-vocabulary autonomous driving5^^^^&&&).

^^^^5 OR \5^^^^. Problem setting and conceptual scope

NOVA targets open-vocabulary ^^^^5submittedDate5^^^^D multi-object tracking for autonomous driving, specifically the online tracking of both base classes and novel classes when the upstream detector produces open-vocabulary labels (&&&^^^^5(Luo et al., 6 Mar 2026) OR \5^^^^&&&). The method is motivated by the claim that conventional ^^^^5submittedDate5^^^^D MOT pipelines remain constrained by closed-set assumptions and “semantic-blind” heuristics: they either suppress unseen categories or rely on a two-stage design in which ^^^^5 open-vocabulary autonomous driving5^^^^D semantics are projected onto ^^^^5submittedDate5^^^^D proposals from a closed-set detector. Under such conditions, instability in open-vocabulary detections propagates downstream into fragmented trajectories, identity switches, class switches, and failures on novel objects.

The central conceptual shift is from hand-designed geometric association to generative spatio-temporal semantic modeling. Rather than treating tracking as the frame-wise optimization of Euclidean distance, IoU, or motion costs, NOVA treats association as trajectory-conditioned reasoning. The object history serves as a prefix, and the candidate continuation is evaluated as the next step in an autoregressive sequence. The paper characterizes this as a transition from traditional fragmented distance-based matching toward a language-model-style prediction process grounded in both semantics and geometry (&&&^^^^5(Luo et al., 6 Mar 2026) OR \5^^^^&&&).

A common misunderstanding is that open-vocabulary tracking is only a detection problem. NOVA is premised on a different diagnosis: the principal challenge is association under open-world category shifts. The framework therefore concentrates on maintaining identity consistency when category names are uncertain or unfamiliar, the geometry is noisy or sparse, and the category space is effectively unbounded.

^^^^5 open-vocabulary autonomous driving5^^^^. Trajectories as structured spatio-temporal semantic sequences

A defining element of NOVA is the representation of a ^^^^5submittedDate5^^^^D trajectory as a structured “spatio-semantic sentence” (&&&^^^^5(Luo et al., 6 Mar 2026) OR \5^^^^&&&). Each track is serialized as a temporal sequence of states, and each state contains the object’s ^^^^5submittedDate5^^^^D box geometry, its confidence or quality, and its semantic label or open-vocabulary placeholder. In the paper’s terminology, this yields a sequence that combines physical continuity—motion, position, size, yaw, and overlap over time—with linguistic structure via class names or masked semantic tokens.

This serialization is not merely a formatting choice. It is the mechanism by which an autoregressive LLM can process tracking context as a sequence-completion problem. The object history becomes a context window analogous to a sentence prefix, and the association decision becomes the next token to be inferred. The paper’s interpretation is that association is not purely a metric-space problem but a contextual prediction problem in which the history of the trajectory supplies the relevant conditioning information (&&&^^^^5(Luo et al., 6 Mar 2026) OR \5^^^^&&&).

The formal input at time PRESERVED_PLACEHOLDER_^^^^5(Luo et al., 6 Mar 2026) OR \5^^^^ is the detector output

PRESERVED_PLACEHOLDER_^^^^5 OR \5^^^^

with each box represented as

PRESERVED_PLACEHOLDER_^^^^5 open-vocabulary autonomous driving5^^^^

where PRESERVED_PLACEHOLDER_^^^^5submittedDate5^^^^ is the center, PRESERVED_PLACEHOLDER_^^^^5\5^^^^ the size, $\theta$ the yaw, $s$ the confidence, and $c \in \mathcal{C}_{base} \cup \mathcal{C}_{novel}$ the open-vocabulary class label. The tracker maintains active trajectories

$\mathcal{T}_{t-1} = \{T_i\}_{i=1}^{M_{t-1}},$

each storing a short history of previous associated boxes.

A plausible implication is that the representation expands the admissible evidence for association beyond local motion smoothness. Because the sequence jointly carries spatial, temporal, and semantic structure, the model can evaluate whether a candidate continuation is coherent at multiple levels of abstraction rather than only geometrically proximate.

^^^^5submittedDate5^^^^. Geometry encoding, prompting, and open-vocabulary conditioning

NOVA does not rely on plain text serialization of numeric coordinates. The paper argues that naive stringification of ^^^^5submittedDate5^^^^D box values is fragile because tokenization of numeric values is lossy and sensitive to small jitter (&&&^^^^5(Luo et al., 6 Mar 2026) OR \5^^^^&&&). To avoid this, it introduces a Geometry Encoder that maps continuous box features into the LLM embedding space. The raw feature vector is

$$f_{\text{raw}} = [x,y,z,l,w,h,\text{vol},\theta,s] \in \mathbb{R}^9,$$

where PRESERVED_PLACEHOLDER_^^^^5 OR \5(Luo et al., 6 Mar 2026) OR \5^^^^. This vector is projected to a learned embedding PRESERVED_PLACEHOLDER_^^^^5 OR \5 OR \5^^^^, which is injected into the LLM at special <box> token positions.

The prompting strategy is explicitly hybrid. For base classes, the real class name is retained in the prompt; for novel classes, the label is replaced with “Unknown.” The stated purpose is to force the model to learn class-agnostic association cues for novel instances instead of overfitting to base-class semantic names (&&&^^^^5(Luo et al., 6 Mar 2026) OR \5^^^^&&&). The paper presents this as a direct response to the open-vocabulary setting, where unstable or unfamiliar labels otherwise induce brittle downstream behavior.

The role of the LLM is correspondingly narrow and technical. It is not deployed as a generic text generator but as a reasoning engine for association. The paper attributes three advantages to this design: semantic priors derived from large-scale language regularities, commonsense reasoning about object persistence, and access to a hierarchical language space in which coarse and fine category labels can remain mutually informative. Categories such as vehicle PRESERVED_PLACEHOLDER_^^^^5 OR \5 open-vocabulary autonomous driving5^^^^ car PRESERVED_PLACEHOLDER_^^^^5 OR \5submittedDate5^^^^ SUV illustrate the relevant granularity structure. The model can therefore fall back to broader semantic compatibility when detailed labels fluctuate (&&&^^^^5(Luo et al., 6 Mar 2026) OR \5^^^^&&&).

A frequent misconception is that NOVA depends solely on semantics. The method is explicitly geometric as well as semantic: it injects continuous geometry embeddings, conditions on trajectory history, and uses semantic tokens as one component of a broader spatio-temporal representation.

^^^^5\5^^^^. Autoregressive association, objective functions, and online tracking pipeline

For each active trajectory PRESERVED_PLACEHOLDER_^^^^5 OR \5\5^^^^ and each current detection PRESERVED_PLACEHOLDER_^^^^5 OR \55^^^^, NOVA forms a prompt PRESERVED_PLACEHOLDER_^^^^5 OR \5^^^^6 by serializing recent trajectory history and appending the candidate detection as a query (&&&^^^^5(Luo et al., 6 Mar 2026) OR \5^^^^&&&). The model is trained to answer a binary question: whether the detection belongs to the track. The core association score is

PRESERVED_PLACEHOLDER_^^^^5 OR \5^^^^7

In this formulation, association is reduced to next-step decision-token prediction.

The binary target is

PRESERVED_PLACEHOLDER_^^^^5 OR \5^^^^8

At inference, match probabilities are converted to costs, typically

PRESERVED_PLACEHOLDER_^^^^5 OR \5^^^^9

and standard Hungarian matching is used to obtain one-to-one track–detection correspondence. Matched detections update tracks, unmatched detections may initialize new tracks, and unmatched tracks are retained for up to PRESERVED_PLACEHOLDER_^^^^5 open-vocabulary autonomous driving5(Luo et al., 6 Mar 2026) OR \5^^^^ missed frames before termination (&&&^^^^5(Luo et al., 6 Mar 2026) OR \5^^^^&&&). NOVA therefore remains an online tracking-by-detection system in lifecycle management even though its association score is generative.

The training objective combines binary generation loss with IoU-quality regression: PRESERVED_PLACEHOLDER_^^^^5 open-vocabulary autonomous driving5 OR \5^^^^ where PRESERVED_PLACEHOLDER_^^^^5 open-vocabulary autonomous driving5 open-vocabulary autonomous driving5^^^^ is cross-entropy over the Yes/No decision token and PRESERVED_PLACEHOLDER_^^^^5 open-vocabulary autonomous driving5submittedDate5^^^^ is an MSE loss between the predicted quality score and the IoU target in PRESERVED_PLACEHOLDER_^^^^5 open-vocabulary autonomous driving5\5^^^^ (&&&^^^^5(Luo et al., 6 Mar 2026) OR \5^^^^&&&). Training data include positive matched pairs and hard negatives, with negative sampling biased toward spatially proximate but identity-inconsistent detections. Mild geometric jitter is added to positives so that the model learns tolerance to localization noise and drift.

The resulting pipeline is concise: obtain open-vocabulary detections; serialize each track history and candidate detection; encode box geometry with the Geometry Encoder; apply hybrid prompting with either base labels or “Unknown”; compute the LLM probability of “Yes” for each pair; perform Hungarian assignment; and update tracks online (&&&^^^^5(Luo et al., 6 Mar 2026) OR \5^^^^&&&).

5. Datasets, evaluation protocol, and reported empirical behavior

NOVA is evaluated on nuScenes, V^^^^5 open-vocabulary autonomous driving5^^^^X-Seq-SPD, and KITTI, with classes split into Base and Novel partitions to explicitly measure open-vocabulary tracking performance (&&&^^^^5(Luo et al., 6 Mar 2026) OR \5^^^^&&&). The reported metrics are sAMOTA when available, otherwise AMOTA, together with AMOTP, MOTA, MOTP, and MT. A unified ^^^^5submittedDate5^^^^D IoU threshold of ^^^^5(Luo et al., 6 Mar 2026) OR \5^^^^.^^^^5 open-vocabulary autonomous driving55^^^^ is used for matching.

Dataset	Base	Novel
nuScenes	Car, Trailer, Pedestrian, Bicycle	Truck, Bus, Motorcycle
V^^^^5 open-vocabulary autonomous driving5^^^^X-Seq-SPD	Car, Van, Pedestrian, Motorcyclist	Bus, Truck, Cyclist, Tricyclist
KITTI	Car, Cyclist	Pedestrian

The primary comparison baseline is Open^^^^5submittedDate5^^^^DTrack, and NOVA is tested with upstream open-vocabulary detectors including Find n’ Propagate, GroundingDINO, and YOLO-World (&&&^^^^5(Luo et al., 6 Mar 2026) OR \5^^^^&&&). The headline quantitative result is on nuScenes novel categories: Open^^^^5submittedDate5^^^^DTrack attains Novel AMOTA PRESERVED_PLACEHOLDER_^^^^5 open-vocabulary autonomous driving55^^^^, whereas NOVA reaches Novel AMOTA PRESERVED_PLACEHOLDER_^^^^5 open-vocabulary autonomous driving5^^^^6, an absolute gain of PRESERVED_PLACEHOLDER_^^^^5 open-vocabulary autonomous driving5^^^^7 percentage points. On nuScenes base classes, Base AMOTA changes from PRESERVED_PLACEHOLDER_^^^^5 open-vocabulary autonomous driving5^^^^8 to PRESERVED_PLACEHOLDER_^^^^5 open-vocabulary autonomous driving5^^^^9 relative to Open^^^^5submittedDate5^^^^DTrack, which the paper interprets as a trade-off in favor of novel-category robustness. On V^^^^5 open-vocabulary autonomous driving5^^^^X-Seq-SPD, with Find n’ Propagate + GroundingDINO, Base sAMOTA improves from PRESERVED_PLACEHOLDER_^^^^5submittedDate5(Luo et al., 6 Mar 2026) OR \5^^^^ to PRESERVED_PLACEHOLDER_^^^^5submittedDate5 OR \5^^^^, and Novel sAMOTA from PRESERVED_PLACEHOLDER_^^^^5submittedDate5 open-vocabulary autonomous driving5^^^^ to PRESERVED_PLACEHOLDER_^^^^5submittedDate5submittedDate5^^^^. With Find n’ Propagate + YOLO-World, NOVA still improves Base performance and remains competitive on Novel classes. On KITTI, Open^^^^5submittedDate5^^^^DTrack reportedly degenerates, while NOVA achieves meaningful Base and Novel tracking performance (&&&^^^^5(Luo et al., 6 Mar 2026) OR \5^^^^&&&).

The reported backbone is Qwen^^^^5 open-vocabulary autonomous driving5^^^^.5-^^^^5(Luo et al., 6 Mar 2026) OR \5^^^^.5B, fine-tuned with LoRA on RTX ^^^^5submittedDate5(Luo et al., 6 Mar 2026) OR \5^^^^9^^^^5(Luo et al., 6 Mar 2026) OR \5^^^^ GPUs for a default 8 epochs, with default trajectory history length PRESERVED_PLACEHOLDER_^^^^5submittedDate5\5^^^^ (&&&^^^^5(Luo et al., 6 Mar 2026) OR \5^^^^&&&). The paper emphasizes that the compact PRESERVED_PLACEHOLDER_^^^^5submittedDate55^^^^B autoregressive model attains the best overall and novel-category performance among the tested backbones, while running at about PRESERVED_PLACEHOLDER_^^^^5submittedDate5^^^^6 FPS on the reported setup. It further states that larger PRESERVED_PLACEHOLDER_^^^^5submittedDate5^^^^7B+ models can be more conservative, reducing identity switches but increasing false negatives and slowing inference.

6. Ablations, interpretation, and limitations

The ablation studies are used to localize which design choices account for the reported performance (&&&^^^^5(Luo et al., 6 Mar 2026) OR \5^^^^&&&). Geometry encoding outperforms text-only coordinate serialization, indicating that continuous geometric representation is important for novel-category robustness. Adding the IoU auxiliary supervision yields a smaller but meaningful improvement and is described as reducing false negatives and improving MOTA. Hybrid prompting is one of the strongest ablations: if novel classes are ignored or discarded during training, novel sAMOTA is PRESERVED_PLACEHOLDER_^^^^5submittedDate5^^^^8; if novel classes are labeled as “Unknown,” novel sAMOTA rises to PRESERVED_PLACEHOLDER_^^^^5submittedDate5^^^^9. The paper interprets this as evidence that explicit training on ambiguous novel examples strengthens class-agnostic association cues. Hard negative mining outperforms random or merely local negative sampling, and positive jitter augmentation improves robustness to noisy detections. History-length ablation shows that no history causes collapse, one frame yields a large improvement, three frames gives the best trade-off, and five frames becomes overly conservative and reduces recall.

These findings motivate the paper’s broader explanation of why NOVA improves over distance-based or closed-set heuristics. Traditional trackers rely on frame-wise similarity terms; NOVA instead reasons over the sequence of past states and predicts whether a continuation “makes sense” in context (&&&^^^^5(Luo et al., 6 Mar 2026) OR \5^^^^&&&). This suggests that the gain comes not from replacing one matching cost with another, but from changing the representational unit of association from isolated detections to trajectory-conditioned semantic sequences.

Several limitations are explicit. NOVA still depends on the quality of upstream open-vocabulary detections; its inference procedure still uses gating plus Hungarian matching, so it is not a fully end-to-end detector-tracker; and the authors identify future work on lightweight appearance cues, heavier occlusion, and broader end-to-end autonomous driving reasoning (&&&^^^^5(Luo et al., 6 Mar 2026) OR \5^^^^&&&). Another common misconception is that the method is a general-purpose “LLM tracker.” In the reported formulation, it is more precise to describe NOVA as a tracking-by-detection system whose association module is recast as next-step open-vocabulary autoregression over geometry-grounded prompts.

In the literature represented here, NOVA’s principal significance lies in the formula “trajectory = spatio-temporal semantic sequence” and in the claim that open-vocabulary ^^^^5submittedDate5^^^^D MOT benefits from treating association as a language-like generative reasoning problem rather than as a purely geometric assignment problem (&&&^^^^5(Luo et al., 6 Mar 2026) OR \5^^^^&&&).