- The paper introduces QueST, a novel framework that uses persistent semantic queries to actively monitor and suppress drift in long-horizon tracking.
- It combines transformer-based spatiotemporal encoding with 3D physical grounding to enforce geometric consistency and mitigate accumulated error.
- Experimental results demonstrate a 67.7% reduction in APE and significant improvements in identity accuracy compared to state-of-the-art models.
QueST: Persistent Semantic Queries for Drift Suppression in Long-Horizon Tracking
Overview and Motivation
The paper "QueST: Persistent Queries as Semantic Monitors for Drift Suppression in Long-Horizon Tracking" (2605.09513) addresses the fundamental challenge of semantic drift in long-horizon point tracking within video sequences, particularly under conditions of articulation, occlusion, and distribution shift. Existing Markovian trackers suffer from accumulated error, leading to catastrophic loss of semantic identity, which is especially hazardous for real-world embodied perception tasks. The authors propose QueST, a framework that redefines tracking as a persistent semantic monitoring task, combining global query-based representations with 3D physical grounding to actively detect and suppress drift.
Methodological Innovations
Monitoring-by-Design Paradigm
QueST departs from the conventional sequential correspondence paradigm by treating interaction-relevant entities as learnable, persistent queries. Each query acts as a semantic anchor, attending globally across the video feature space rather than being confined to local frame-to-frame propagation. This architecture fundamentally shifts the problem from passive tracking to active semantic monitoring at the representation level.
A transformer-based video encoder, inspired by ViT, extracts spatiotemporal features, with learnable spatial and temporal positional encodings. Query embeddings, initialized from a learned distribution, serve as persistent tokens representing semantic hypotheses (e.g., hinges, handles), and are refined at each time step via a low-capacity transformer decoder with global cross-attention.
Physical Grounding in 3D
Critical to QueST's efficacy is the imposition of geometric plausibility via 3D grounding. 2D predicted queries are lifted using depth back-projection and camera intrinsics to output world-space trajectories. The objective combines spatial and temporal loss terms, notably enforcing smoothness in velocity and acceleration as well as consistency with known kinematic manifolds. This acts as an implicit regularizer, penalizing out-of-manifold predictions that typically result from semantic drift.
Semantic Identity Versus Physical Plausibility
By constraining trajectories both semantically and physically, QueST ensures that the representation remains tied to an interaction-relevant entity rather than collapsing to visually similar but incorrect regions. The system is thus robust to occlusion and articulation, providing an explicit mechanism for detecting and correcting drift that standard flow- or correspondence-based models lack.
Experimental Results and Numerical Analysis
The evaluation leverages long-horizon, multi-joint articulated sequences from PartNet-Mobility rendered in SAPIEN, with rigorous protocols for drift analysis in the presence of occlusion and complex kinematics. The principal metrics are Absolute Point Error (APE), Drift@100 (terminal drift at long horizons), and Identity Accuracy.
A summary of the results is below:
| Method |
APE |
Drift@100 |
Identity Accuracy |
| RAFT-3D |
0.341 |
0.472 |
8.7% |
| CoTracker |
0.276 |
0.398 |
19.2% |
| TAP-Net |
0.251 |
0.372 |
21.4% |
| QueST |
0.081 |
0.155 |
86.5% |
QueST demonstrates a 67.7% reduction in APE over TAP-Net, with terminal drift remaining bounded rather than growing linearly. Ablation studies show that removing persistent queries or 3D grounding leads to pronounced identity switching and rapid drift, directly substantiating the central claims of the work.
Robustness experiments also highlight QueST's resilience to synthetic noise (maintaining >96% accuracy at non-trivial noise levels), while temporal window and query capacity ablations demonstrate that performance benefits saturate at moderate settings (T=4 and K=8).
Analysis of Failure Cases
Two principal failure modes are documented:
- Extreme Occlusion: When an entity is occluded for more than 30 frames (>80% occlusion), the model may drift to a semantically similar but incorrect region.
- Symmetric Ambiguity: In objects with indistinguishable parts (e.g., multiple identical handles), queries may jump between semantically equivalent targets, resulting in loss of strict correspondence, although overall tracking accuracy remains high.
These are significant, highlighting the current limitations of the persistent query mechanism and the need for additional context or disambiguation strategies in such scenarios.
Theoretical and Practical Implications
The persistent semantic query paradigm enforced by QueST implies a fundamental reframing of the tracking problem, with deep consequences for future embodied AI systems. By embedding semantic monitors directly into perception, agents gain an explicit mechanism for detecting both representation-level and geometric consistency failures, possibly serving as an early-warning system in critical applications.
This monitoring-by-design principle has relevance for robotics, autonomous systems, and industrial automation, where silent drift events can precipitate unsafe or catastrophic outcomes if undetected. Furthermore, the architectural compatibility with transformer-based perception models and real-time performance on current GPUs suggests QueST's practical viability for deployment in diverse active vision settings.
On the theoretical side, QueST's approach foregrounds the union of statistical feature consistency and explicit physical reasoning as a necessary condition for robust long-horizon perception, echoing emerging trends in representation learning that prioritize interpretability and active monitoring over black-box flow affinity.
Conclusion
QueST introduces a principled framework for drift-suppressed long-horizon tracking, substituting frame-local correspondence with global, persistent semantic queries constrained by 3D physical reasoning. Experimental results exhibit marked reductions in terminal drift and catastrophic identity loss under challenging articulated, occluded, and noise-rich conditions. The joint use of semantic monitoring and physical grounding distinguishes QueST from prior art and establishes a foundation for future research in self-monitoring perception systems. Limitations remain concerning extreme occlusions and symmetric ambiguity, indicating avenues for further advancement, including enhanced scene memory and context-aware query disambiguation.
