TAPNext++: What's Next for Tracking Any Point (TAP)?

Abstract: Tracking-Any-Point (TAP) models aim to track any point through a video which is a crucial task in AR/XR and robotics applications. The recently introduced TAPNext approach proposes an end-to-end, recurrent transformer architecture to track points frame-by-frame in a purely online fashion -- demonstrating competitive performance at minimal latency. However, we show that TAPNext struggles with longer video sequences and also frequently fails to re-detect query points that reappear after being occluded or leaving the frame. In this work, we present TAPNext++, a model that tracks points in sequences that are orders of magnitude longer while preserving the low memory and compute footprint of the architecture. We train the recurrent video transformer using several data-driven solutions, including training on long 1024-frame sequences enabled by sequence parallelism techniques. We highlight that re-detection performance is a blind spot in the current literature and introduce a new metric, Re-Detection Average Jaccard ($AJ_{RD}$), to explicitly evaluate tracking on re-appearing points. To improve re-detection of points, we introduce tailored geometric augmentations, such as periodic roll that simulates point re-entries, and supervising occluded points. We demonstrate that recurrent transformers can be substantially improved for point tracking and set a new state-of-the-art on multiple benchmarks. Model and code can be found at https://tap-next-plus-plus.github.io.

• The paper introduces TAPNext++ with long-sequence training and roll augmentation to achieve robust point re-detection and extended memory over thousands of frames.
• It leverages a linear recurrent SSM integrated with a Vision Transformer to enable efficient distributed multi-GPU training at high throughput (e.g., 348 FPS).
• Empirical results highlight improved re-detection metrics and stability across challenging tracking scenarios, setting a new benchmark for online point tracking.

TAPNext++: Advancing Online Point Tracking through Long-Sequence Training and Re-Detection Robustness

Introduction

TAPNext++ ("TAPNext++: What's Next for Tracking Any Point (TAP)?" (2604.10582)) presents a robust, efficient solution to sequential 2D point tracking in video, with particular focus on scenarios demanding long-term temporal consistency and the ability to re-detect points after occlusion or when points exit and re-enter the field of view. The work builds on the TAPNext framework, addressing its critical limitations in long-sequence tracking and re-detection, primarily via scalable training paradigms, architectural efficiency, and novel data augmentations. The evaluation is rigorous, introducing the Re-Detection Average Jaccard ($AJ$) metric to measure post-occlusion re-detection capabilities—an area overlooked in prior benchmarks.

Limitations of Prior TAP Models

Contemporary TAP architectures—windowed or frame-by-frame, explicit or implicit-memory based—suffer from performance degradation over long video horizons and struggle to robustly re-detect points following significant occlusion or departure/reentry from the frame. While window-based approaches (e.g., sliding attention contexts, memory banks) offer some mitigation, they incur substantial computational/memory overhead and systematically break down for extended durations. TAPNext [Zholus_2025_ICCV], with its linear recurrent SSM layers, introduced the potential for constant-memory, frame-by-frame point tracking, but was empirically shown to lose temporal fidelity past 150 frames due to limited training regime and out-of-domain evaluation.

Model and Training Scalability

TAPNext++ maintains the recurrent SSM/Vision Transformer architecture of TAPNext, leveraging the state-space efficiency for high-throughput (e.g., 348 FPS with 1024 points on H100). The key technical advancement is scaling sequence length during training to 1024 frames, made possible via a distributed parallel scan mechanism that exploits the algebraic structure of linear recurrence for associative parallelism. This enables end-to-end training across multiple GPUs with logarithmic communication complexity in the number of devices, surpassing the single-device memory bottleneck without architectural modification.

Figure 1: TAPNext++ parallelizes long-sequence training using distributed scan of SSM blocks across GPUs, decoupled from spatial-attention ViT modules.

Robustness to Re-Detection and Challenging Dynamics

The paper highlights the critical inadequacy of prior TAP methods: repeated point tracking failure when objects re-appear or are re-introduced to the frame at different positions. TAPNext++ directly addresses this by:

• Introducing "roll augmentation," which simulates periodic shifts (wrapping objects/points around the frame) to force the model to learn appearance-dependent re-detection rather than relying solely on spatio-temporal locality.
• Supervising occluded but visible-in-frame points with a down-weighted loss, encouraging the model to maintain plausible point location hypotheses through occlusion.

These modifications ensure meaningful state evolution in the recurrent memory even through extended absence of query points.

Figure 2: Roll augmentation applies periodic spatial translations, robustly simulating point re-entry and large displacements for data diversity.

Empirical evaluation demonstrates that only TAPNext++ with roll augmentation successfully tracks all objects through challenging re-entry scenarios involving multi-object juggling, whereas all other methods fail immediately or upon the object’s re-entry.

Figure 3: TAPNext++ uniquely tracks objects through multiple exits and entries, outperforming all baselines in scenarios with frequent point reappearance.

Long-Sequence Stability and Memory Analysis

Through static-image input experiments and long-sequence tracking on the synthetic PointOdyssey and Kubric-1024 datasets, TAPNext++ is shown to substantially outperform prior architectures in memory longevity, maintaining point fidelity for sequences exceeding several thousand frames. Distributed parallel scan training, together with augmentation and re-weighted supervision, ensures that the recurrent SSM memory state avoids catastrophic drift or oscillatory divergence—a phenomenon observed in the baseline BootsTAPNext.

Figure 4: Finetuning on 1024-frame videos significantly extends TAPNext++’s memory horizon, stabilizing mean pixel error over extremely long static sequences.

Benchmark Results and Metrics

TAPNext++ is evaluated on multiple benchmarks: PointOdyssey, DAVIS, RoboTAP, Kinetics, and RGB-Stacking. Key numerical outcomes include:

• Highest $\delta^{avg}$, survival rate, and AJ on PointOdyssey (mean $\delta^{avg}$ 52.6, Survival 67.9%, $AJ$ 23.3).
• Superior re-detection scores on RoboTAP, with AJ preserved after long occlusions, in contrast to window-based CoTracker3, which degrades after $d=256$ undetectable frames.

  Figure 5: TAPNext++ with roll augmentation maintains high re-detection AJ even after long periods of undetectability, exceeding window-based approaches.

TAPNext++ also achieves state-of-the-art results across standard and re-detection metrics on DAVIS, Kinetics, and synthetic datasets, all at significantly lower latency and higher throughput compared to prior art.

Qualitative Performance and Failure Modes

Qualitative analysis on challenging real-world and synthetic videos further validates the efficacy of TAPNext++. In dynamic tracking scenarios with fine, thin structures (e.g., clock hands), competitor models routinely fail (either dropping tracks or drifting irreversibly), whereas TAPNext++ maintains accurate tracking across all points to the end of the sequence.

Figure 6: Only TAPNext++ reliably tracks both clock hands throughout the full video, while other methods lose track accuracy or coverage.

Architectural Analysis and Generalization

Crucially, the improved long-term tracking capacity of TAPNext++ does not arise from trivial architectural modifications such as memory bank extension or window size increase, nor from hyperparameter tuning. Rather, it reflects principled system-algorithmic co-design, leveraging the SSM’s linear recurrence algebra for efficient multi-GPU training and the use of challenging, realistic data augmentations. Detailed analysis shows that the method generalizes robustly: models trained without direct exposure to a target test set (e.g., not trained on PointOdyssey) still outperform all prior baselines on out-of-domain sequences.

Implications and Future Directions

TAPNext++ establishes that the memory limitations of SSM-based TAP models are not architectural but arise primarily from myopic training regimes and inadequate data diversity. By scaling sequence length and augmenting for challenging motion and occlusion scenarios, substantial robustness and generalization can be achieved with constant-compute models—critical for AR/XR, robotics, and mobile vision applications. The introduction of $AJ$ as a metric sets a standard for evaluating re-detection, a core requirement for deployment in dynamic environments. Open research questions remain in scaling to hour-long or truly unbounded sequences, explicit occluder tracking, consistent ambiguity modeling, and further system-level optimization.

Conclusion

TAPNext++ fundamentally elevates the state of the art in online point tracking through scalable, efficient long-sequence training, robust data augmentation for re-detection, and rigorous evaluation. It demonstrates that properly trained, memory-efficient recurrent architectures can outperform heavy windowed or memory-bank approaches both in accuracy and in speed, even for complex, long-horizon video sequences. These results have significant implications for memory-constrained inference in real-time settings and solidify the importance of re-detection-aware metrics and training in future point tracking research.