KV-Tracker: Real-Time Pose Tracking with Transformers

Abstract: Multi-view 3D geometry networks offer a powerful prior but are prohibitively slow for real-time applications. We propose a novel way to adapt them for online use, enabling real-time 6-DoF pose tracking and online reconstruction of objects and scenes from monocular RGB videos. Our method rapidly selects and manages a set of images as keyframes to map a scene or object via $π^3$ with full bidirectional attention. We then cache the global self-attention block's key-value (KV) pairs and use them as the sole scene representation for online tracking. This allows for up to $15\times$ speedup during inference without the fear of drift or catastrophic forgetting. Our caching strategy is model-agnostic and can be applied to other off-the-shelf multi-view networks without retraining. We demonstrate KV-Tracker on both scene-level tracking and the more challenging task of on-the-fly object tracking and reconstruction without depth measurements or object priors. Experiments on the TUM RGB-D, 7-Scenes, Arctic and OnePose datasets show the strong performance of our system while maintaining high frame-rates up to ${\sim}27$ FPS.

• The paper introduces a transformer-based KV-caching mechanism that drastically accelerates real-time 6-DoF pose tracking.
• It decouples mapping and tracking by caching keyframes to compute lightweight cross-attention, significantly reducing computational complexity.
• Experimental results show state-of-the-art performance on benchmarks with up to 27 FPS tracking, robust accuracy in both scene-level and object-centric tracking.

KV-Tracker: Real-Time 6-DoF Pose Tracking with Transformer-Based KV-Caching

Introduction and Motivation

KV-Tracker introduces a model-agnostic, training-free paradigm for deploying multi-view transformer-based 3D geometry networks in real-time camera and object pose tracking. The framework is motivated by the growing need for robust metric scene understanding in online robotics and AR/VR, where existing multi-view 3D transformer models, though powerful, are computationally prohibitive for streaming scenarios due to their $O(N^2)$ scaling with the number of input images. The proposed solution leverages a key-value (KV) caching mechanism around the global self-attention modules within such networks to enable order-of-magnitude acceleration and robust online operation.

Methodology

Keyframe Selection and Mapping–Tracking Split

KV-Tracker builds on the $\pi^3$ architecture, a feed-forward, fully-attention-based model, to automatically select and maintain a keyframe set during streaming input. During mapping, full bidirectional global self-attention is computed among these keyframes, and the resultant KV pairs are cached, forming a persistent memory representation of the observed geometry and appearance. When processing a new live frame, the system reuses the cached keys and values, computing only the query for the novel image.

Figure 1: Real-time object tracking and online reconstruction using RGB images; red frustums indicate mapping keyframes, blue frustums tracking frames, with accumulated global geometry.

Figure 2: System overview showing the mapping (with keyframes and KV-cache generation) and tracking (with live queries against KV-cache) split.

This design decouples mapping (full multi-view geometry aggregation and cache update) from tracking (lightweight localization against the frozen cache), with the cache being re-computed only upon keyframe buffer updates. This yields highly efficient online inference.

KV-Cache as a Scene Representation

The cached KV pairs serve as a powerful learned implicit scene model, analogous to traditional SLAM map primitives (e.g., sparse 3D points or implicit MLPs), but with the advantage of directly encoding both geometry and appearance in a transformer latent space. This cache can be flexibly applied to both scene-level and object-centric tracking, with new frames localized against the cache through cross-attention.

Figure 3: Bi-directional (mapping) versus cross-attention with KV-cache (tracking); only mapping recomputes full keys/values.

Computational Optimizations

By replacing bidirectional frame-to-frame attention during tracking with cross-attention from the new query frame to the frozen keyframe cache, the computational complexity per new frame drops from $O(N^2 M^2)$ to $O((N+1) M^2)$ ($N$ keyframes, $M$ patches per image). This enables practical real-time performance.

Experimental Results

Scene-Level and Object-Centric Tracking

Experimental validation is performed on standard benchmarks (TUM RGB-D, 7-Scenes, ARCTIC, OnePose, OnePose Low Texture), demonstrating strong quantitative results at high frame rates.

• Scene-Level Camera Tracking: On TUM RGB-D, KV-Tracker achieves an average Absolute Trajectory Error (ATE) of 0.108m, improving over TTT3R by 18%. On 7-Scenes, KV-Tracker’s average ATE is 0.08m, offering a 44% reduction compared to TTT3R.
• Object Tracking: On ARCTIC, KV-Tracker achieves the lowest average ATE (0.228m), outperforming all streaming transformer baselines. On OnePose Low Texture, it attains 94.4% recall at 5cm/5° thresholds, surpassing all competitive offline approaches, especially at coarser regimes—without requiring CAD models or depth sensors.

  Figure 4: Real-time object tracking and reconstruction on a moving car, demonstrating robust performance under in-the-wild motion.

  

  Figure 5: Trajectory and online reconstruction results for object tracking benchmarks, illustrating accurate pose estimation and geometry fusion.

Runtime and Scalability Analysis

KV-Tracker’s tracking module operates at up to 27 FPS, with robust performance sustained for cache sizes up to 110 frames (20+ FPS). Evaluation demonstrates a $15\times$ speedup versus standard recomputation, with graceful degradation due to linear scaling with keyframes.

Figure 6: Throughput comparison: full bidirectional attention ($O(N^2)$) vs. KV-cache ($O(N)$) tracking, showing substantial FPS gains for large $N$.

Ablations and Benchmarking

Comparisons span not only streaming and offline geometry transformers, but also classical approaches (DPVO), with results highlighting competitive or state-of-the-art accuracy, especially given the substantially reduced requirements: no CAD, no depth, no retraining or fine-tuning, and fully online operation.

Implications and Future Directions

KV-Tracker establishes that off-the-shelf multi-view geometry transformers can be effectively repurposed for online tracking and mapping, given an appropriate KV-caching front-end. The method is agnostic to model details and applicable across recently proposed transformer architectures (e.g., Depth Anything 3), as demonstrated in supplementary benchmarks.

One current limitation is cache size, which bounds scalability to objects and small- to medium-sized environments. Opportunities for further research include cache pruning, compression, or hierarchical partitioning, as well as methods for continuous incremental update. The approach also suggests promising avenues for robust online tracking in robotics, AR/VR, and beyond, where existing transformer-based priors can be leveraged without retraining.

Conclusion

KV-Tracker presents a formal architecture and thorough empirical justification for real-time pose tracking and online reconstruction using transformer-based multi-view geometry networks. By introducing a frozen KV-cache memory mechanism decoupled from streaming tracking, the system achieves high-accuracy, scalable tracking without retraining or prior knowledge of the tracked object or scene. This paradigm both narrows the gap between offline and online visual geometry models and points to practical directions for future large-scale, real-time 3D understanding and SLAM systems.

Reference: "KV-Tracker: Real-Time Pose Tracking with Transformers" (2512.22581)