VOCA: Visual Odometry with Codec Awareness

Abstract: Camera pose estimation from image streams is a critical component of spatial world models that integrate perception into planning and decision-making. Nearly all Visual Odometry (VO) and Simultaneous Localization and Mapping (V-SLAM) systems have focused on datasets containing raw, uncompressed videos. Many working systems instead use ubiquitous hardware units to efficiently compress and decode video streams, saving orders of magnitude in storage and bandwidth. However, this lossy compression introduces visual artifacts that hinder the performance of traditional tracking systems. We present VOCA, a causal stereo visual-odometry method that exploits codec information to improve tracking performance. We achieve state-of-the-art performance on causal VO for relative trajectory error, efficiency, and absolute trajectory error on compressed streams. This work highlights the potential of leveraging widely available video codec information for vision tasks.

• The paper introduces VOCA, which initializes KLT tracking with H.264 motion vectors, combines prior-based and prior-free tracking, and bridges I-frames to recover accuracy on compressed video.
• VOCA improves relative trajectory error by about 15% on EuRoC, 62% on TUM-VI, and 40% on the Monado SLAM Dataset while operating at compression levels up to roughly 100×.
• The method shows that discarded codec metadata can support bandwidth-efficient visual odometry, although performance remains limited by noisy imagery, dynamic objects, B-frames, and the added cost of parallel tracking.

Motivation and problem setting

Nearly all visual odometry (VO) and visual SLAM systems are developed and benchmarked on raw, uncompressed image streams, yet practical deployments—mixed-reality headsets, drones, wearables, and edge-assisted pipelines—almost universally transmit camera data through lossy video codecs such as H.264/AVC under a compress-then-analyze design. The paper quantifies the stakes: a raw stereo monochrome stream at 640×480 and 30 fps exceeds one gigabyte per minute, while H.264 reduces sequence sizes by roughly 35–100×. The cost is that quantization artifacts—blurring, contrast reduction, jagged edges—violate the photometric-constancy assumption underlying both KLT-style sparse tracking and direct methods, degrading state-of-the-art trackers (2607.00189).

VOCA addresses this gap as a causal stereo VO method built on Basalt's KLT-tracking front-end that exploits codec-internal information—per-block motion vectors and I-frame structure—to recover tracking accuracy on compressed streams. To the authors' knowledge, only MoV-SLAM had previously attempted to actively use codec information for tracking; MoV-SLAM instead derives EXPRESS features from H.264 macroblocks for descriptor matching.

Method

The core observation is structural: Lucas-Kanade tracking assumes locally constant motion within a patch neighborhood, which mirrors block-based inter-frame prediction in hybrid codecs. For each macroblock partition $B_k$, the encoder selects a motion vector $\mathbf{d}_k$ such that $$I_t(\tilde{\mathbf{p}}) \approx I_{t-1}(\tilde{\mathbf{p}} + \mathbf{d}_k)$$, so $-\mathbf{d}_k$ acts as a coarse, block-wise approximation of forward optical flow. Because motion vectors are optimized for rate–distortion efficiency rather than motion fidelity—and are purely translational, lacking the SE(2) in-plane rotation component of Basalt's patch model—they cannot replace KLT refinement. Instead, VOCA uses them as initialization priors: the translation component $\mathbf{t}_{\text{2D}}$ is initialized with $-\mathbf{d}_k$ (identity rotation), and the KLT optimizer refines from there. Empirically, for successful tracks the majority of motion-vector priors lie within 1–2 pixels of the converged solution, substantially shrinking the distance to the minimum compared with the previous-pixel prior.

Two failure modes arise from unconstrained priors: motion vectors pointing to photometrically similar but incorrect regions, and correct vectors attached to dynamic objects. Both evade standard forward–backward consistency checks, since backward tracking initialized near its origin appears consistent. VOCA therefore tracks every point twice—with and without motion priors—in parallel, keeping single-mode successes and rejecting dual-mode tracks that disagree.

I-frames carry no inter-prediction and hence no motion vectors, and they tend to coincide with large scene changes where correspondences are hardest. VOCA bridges these boundaries by reusing motion vectors from the last P-frame under a constant-motion assumption. Two additional adaptations to the Basalt baseline—a relaxed forward–backward threshold and larger tracking patches—are applied symmetrically to the reported baseline numbers.

Experimental results

The evaluation covers EuRoC, TUM-VI room sequences, and the Monado SLAM dataset (MSD), all compressed per-camera with two-pass ffmpeg/libx264 encoding at 500 kbps (roughly 35× compression on MSD, ~70× on EuRoC/TUM-VI). All methods report causal pose estimates with loop closure disabled for ORB-SLAM3 and OKVIS2.

Dataset	Metric	Best baseline	VOCA	Improvement
EuRoC	RTE avg	1.947 cm (Basalt)	1.669 cm	≈15%
EuRoC	ATE avg	19.24 cm (Basalt)	17.35 cm	≈10%
TUM-VI	RTE avg	2.120 cm (OKVIS2)	0.815 cm	≈62%
TUM-VI	TR5 ATE	7.80 cm (ORB-SLAM3)	7.70 cm	Basalt: 54.50 cm
MSD	ATE median	124.70 cm (OKVIS2)	78.30 cm	≈37%
MSD	RTE median	8.692 cm (OKVIS2)	5.212 cm	≈40%
MSD	ATE success rate	34.4%	46.9%	+13 pts (36% rel.)

On TUM-VI, ORB-SLAM3 attains comparable global consistency (ATE) but exceeds the RTE divergence threshold on all sequences, indicating good global alignment with poor frame-to-frame tracking—precisely the regime where codec-aware priors help. On MSD, the hardest dataset with dynamic occlusions and aggressive motion, all methods fail frequently, but VOCA achieves the best aggregates across devices (Valve Index, Odyssey+, Reverb G2), with the exception that gains are less pronounced on the Odyssey+'s noisy VGA cameras, which the authors attribute to H.264's aggressive effect on grainy imagery.

The bitrate sweep shows VOCA is the most stable system across settings, maintaining performance up to 500 kbps (~100× compression), whereas OKVIS2 degrades severely and Basalt suffers particularly on TUM-VI. An ablation over three integration strategies finds that running prior-based and prior-free tracking in parallel with consistency filtering (strategy C) performs best overall, with I-frame bridging yielding the best median ATE on TUM-VI; fallback-only strategies (A, B) trail slightly. Supplementary results show DROID-SLAM outperforming VOCA on EuRoC ATE (9.92 vs. 17.00 cm average) while losing substantially on TUM-VI (45.58 vs. 9.20 cm), though this comparison is qualified by DROID-SLAM's non-causal backend and GPU requirements. A prototypical AV1 extension suggests the approach transfers beyond H.264.

Limitations and open questions

Several caveats bear directly on the results. First, MoV-SLAM could not be reproduced despite best efforts—the official implementation was run instead of the claimed numbers—and it is excluded from TUM-VI and MSD entirely because it requires stereo rectification, which fisheye cameras make disproportionately costly. Second, the method assumes causal encoding order with past-only references; bidirectional prediction (B-frames) is outside the evaluated scope. Third, I-frame bridging relies on a constant-motion assumption that is admittedly approximate, and the parallel dual-track strategy doubles per-point tracking work, an overhead not analyzed in detail. Fourth, the degraded performance on low-quality sensors (Odyssey+) indicates that codec priors do not compensate when source imagery is itself poor. Finally, the AV1 result is explicitly preliminary and unoptimized, leaving open whether motion-vector fidelity across modern codecs generalizes at scale.

Conclusion

VOCA demonstrates that a substantial fraction of compression-induced tracking degradation can be recovered by reusing decoded motion vectors as KLT initialization priors, combined with dual-mode outlier filtering and I-frame bridging. It achieves state-of-the-art causal stereo VO on compressed streams across three benchmarks, with the largest gains in relative trajectory error—the metric most relevant to mixed-reality user experience—and remains robust at approximately 100× compression where conventional trackers fail. The broader implication is that codec metadata, already computed and discarded by every decoder, is a free and widely available signal for vision tasks operating under bandwidth constraints.