VeloDepth: Real-Time Video Depth Estimation

• VeloDepth is an online video depth estimation system that converts a strong monocular metric model into a video-aware framework using feature propagation and keyframe resets.
• It employs a Propagation Module that warps previous depth and feature maps with optical flow and applies learned residual corrections to maintain efficiency and temporal consistency.
• The system achieves a trade-off by slightly reducing per-frame accuracy while significantly improving temporal consistency and throughput for real-time causal video inference.

VeloDepth is an online video depth estimation system introduced in “Video Depth Propagation” that turns an accurate but frame-wise monocular metric depth model into a temporally consistent, efficient video model by learning to propagate and correct depth features over time (Piccinelli et al., 11 Dec 2025). It addresses causal video inference, where at time $t$ the model uses only frames $\le t$, and outputs metric 3D point maps $\mathbf{P}_t\in\mathbb{R}^{3\times H\times W}$ and implicitly depth $\mathbf{D}_t$. The method is organized around a Propagation Module that warps previous depth and neck features with optical flow, applies learned residual corrections, and reuses a frozen base decoder, with the explicit goal of combining per-frame accuracy, temporal consistency, and real-time suitability.

1. Problem setting and design objectives

VeloDepth addresses online video depth estimation, also referred to as video depth propagation. Its input is a video stream $\{\mathbf{I}_t\}$, where $\mathbf{I}_t\in\mathbb{R}^{H\times W\times 3}$ is the RGB frame at time $t$, and its output is a sequence of metric 3D point maps or depth maps predicted causally (Piccinelli et al., 11 Dec 2025). The stated objectives are threefold: accurate per-frame prediction comparable to a strong single-image model, temporal consistency across consecutive frames, and efficiency suitable for real-time deployment.

The motivation is framed against two limitations of prior approaches. Frame-by-frame monocular depth estimation methods such as DPT, MiDaS, UniDepth, Metric3D, and DepthAnything treat each image independently, ignore temporal priors, suffer from temporal inconsistency, and are computationally heavy if run on every frame. Heavy temporal models for video depth often rely on recurrent networks, 3D CNNs, temporal transformers, or diffusion-based models, but these approaches are described as expensive in FLOPs and memory, frequently offline, and unsuitable for low-latency causal usage (Piccinelli et al., 11 Dec 2025).

VeloDepth’s central design move is to avoid recomputing the full RGB-to-depth pipeline on every frame. Instead, it runs the full base monocular metric depth model only on sparse keyframes; for intermediate “fast frames,” it propagates and corrects previously computed features and predictions. This makes temporal stability a property of the architecture rather than only a post hoc regularization target. A plausible implication is that VeloDepth should be understood not as a new image-depth backbone, but as a wrapper that converts a strong monocular metric model into a video-aware depth system.

2. Core architecture

The base model in VeloDepth is UniDepthV2 or UniK3D, with a frozen Base Encoder and a frozen Base Decoder (Piccinelli et al., 11 Dec 2025). On a keyframe, the model runs the full Base Encoder on RGB, obtains multi-scale neck features $\mathcal{F}_0=\{\mathbf{F}_{i,0}\}$, and then applies the Base Decoder to produce 3D points $\mathbf{P}_0$ and depth $\mathbf{D}_0$. On non-keyframes, the Base Encoder is not run again; instead, previously computed neck features are propagated and corrected before being passed to the same Base Decoder.

The Propagation Module takes four inputs at time $\le t$0: the current RGB frame $\le t$1, the previous frame depth prediction $\le t$2, the previous frame multi-scale neck features $\le t$3, and a backward optical flow estimate $\le t$4 from frame $\le t$5 to $\le t$6. Its goal is to produce corrected features $\le t$7 such that decoding them yields the current-frame depth.

The input optical flow is initially computed by a lightweight CPU-based DIS algorithm, although the method is described as agnostic to the specific flow source and able to use motion vectors or SEA-RAFT at test time. Flow is encoded as a 3-channel image in which red and green store normalized horizontal and vertical flow, and blue stores luma difference between frames. A small flow backbone produces flow features $\le t$8, and a two-layer convolutional network predicts a correction $\le t$9, yielding refined flow

$\mathbf{P}_t\in\mathbb{R}^{3\times H\times W}$0

Using the refined flow, VeloDepth warps previous depth and previous neck features into the current frame. With a generic warping operator $\mathbf{P}_t\in\mathbb{R}^{3\times H\times W}$1, any tensor $\mathbf{P}_t\in\mathbb{R}^{3\times H\times W}$2 at $\mathbf{P}_t\in\mathbb{R}^{3\times H\times W}$3 is mapped into frame $\mathbf{P}_t\in\mathbb{R}^{3\times H\times W}$4 as

$\mathbf{P}_t\in\mathbb{R}^{3\times H\times W}$5

A multi-modal encoder based on ConvNeXt-Tiny then fuses three branches: RGB features $\mathbf{P}_t\in\mathbb{R}^{3\times H\times W}$6, propagated depth features $\mathbf{P}_t\in\mathbb{R}^{3\times H\times W}$7, and flow features $\mathbf{P}_t\in\mathbb{R}^{3\times H\times W}$8. The initial fused correction tensor is

$\mathbf{P}_t\in\mathbb{R}^{3\times H\times W}$9

with

$\mathbf{D}_t$0

Here $\mathbf{D}_t$1 is a depth gate derived from flow features. The fused tensor is processed by shared ConvNeXt blocks to produce multi-scale correction features $\mathbf{D}_t$2.

The final step is residual correction of the warped neck features. For each scale $\mathbf{D}_t$3,

$\mathbf{D}_t$4

where

$\mathbf{D}_t$5

This residual form makes propagated features the default state and uses gated corrections where warping, depth propagation, or flow estimation are unreliable. The paper explicitly interprets this as an identity-biased propagation mechanism (Piccinelli et al., 11 Dec 2025).

3. Temporal propagation, recurrent state, and keyframes

VeloDepth is causal. At time $\mathbf{D}_t$6, it uses only the current RGB frame, the previous neck features $\mathbf{D}_t$7, the previous depth $\mathbf{D}_t$8, and the flow estimate between $\mathbf{D}_t$9 and $\{\mathbf{I}_t\}$0 (Piccinelli et al., 11 Dec 2025). The recurrent state consists of the neck feature pyramid and the previous depth prediction. Operationally, the system iterates through the following sequence: compute or read the flow, warp $\{\mathbf{I}_t\}$1 into frame $\{\mathbf{I}_t\}$2, use current RGB plus warped depth plus flow in the Propagation Module to obtain $\{\mathbf{I}_t\}$3, decode $\{\mathbf{I}_t\}$4 to $\{\mathbf{I}_t\}$5 and $\{\mathbf{I}_t\}$6, and store $\{\mathbf{I}_t\}$7 for the next step.

Temporal consistency is therefore structurally enforced. Because $\{\mathbf{I}_t\}$8 unless corrected, small apparent motion tends to produce small changes in neck features and hence small changes in decoded depth. This is distinct from approaches that impose temporal smoothness only through a loss or a post-processing stabilizer.

Propagation is periodically reset by keyframes. VeloDepth defines a keyframe heuristic based on flow magnitude and a warping-based difference criterion. A new keyframe is triggered if and only if

$\{\mathbf{I}_t\}$9

where $\mathbf{I}_t\in\mathbb{R}^{H\times W\times 3}$0 denotes the number of frames since the last keyframe. The first frame is always a keyframe, and the paper reports average keyframe intervals around 30 frames, adapted by dynamics and dataset (Piccinelli et al., 11 Dec 2025). Longer intervals improve consistency but can hurt accuracy because of over-smoothing; the adaptive heuristic is presented as a trade-off between these effects.

4. Training objectives and supervision

VeloDepth is trained with a mixture of supervised and pseudo-supervised signals. On depth datasets such as TartanAir, Wild-RGBD, HabitatMatterport3D, PointOdyssey, and Waymo, ground-truth depth is used; on in-the-wild video datasets such as Kinetics-700, Moments-in-Time, and SA-v2, supervision is derived from the frozen Base Model’s predictions (Piccinelli et al., 11 Dec 2025).

Depth accuracy is optimized with two losses: the scale-invariant log loss $\mathbf{I}_t\in\mathbb{R}^{H\times W\times 3}$1 and the per-video-clip scale- and shift-invariant $\mathbf{I}_t\in\mathbb{R}^{H\times W\times 3}$2 loss $\mathbf{I}_t\in\mathbb{R}^{H\times W\times 3}$3. The paper emphasizes that the latter is computed over the entire clip instead of per image.

To keep propagated features aligned with what the Base Model would have produced on the current frame, VeloDepth imposes a feature supervision loss:

$\mathbf{I}_t\in\mathbb{R}^{H\times W\times 3}$4

where $\mathbf{I}_t\in\mathbb{R}^{H\times W\times 3}$5 are Base Model neck features obtained by running the full encoder on frame $\mathbf{I}_t\in\mathbb{R}^{H\times W\times 3}$6. Refined flow is supervised with pseudo-ground-truth backward flow from SEA-RAFT via an $\mathbf{I}_t\in\mathbb{R}^{H\times W\times 3}$7 loss on flow vectors.

A central training term is the pose-agnostic, radial-distance consistency loss. Let $\mathbf{I}_t\in\mathbb{R}^{H\times W\times 3}$8 be the 3D point map at time $\mathbf{I}_t\in\mathbb{R}^{H\times W\times 3}$9, and define radial distance as $t$0. The loss is

$t$1

with a bidirectional total

$t$2

The paper describes this loss as rotation invariant, translation-corrected, and not requiring known camera extrinsics. It also reports that consistency on Euclidean radial distance performs better than consistency on raw depth (Piccinelli et al., 11 Dec 2025).

Temporal flip augmentation reverses frame order with 50% probability during training, specifically to avoid a forward-motion bias. The implementation uses PyTorch + CUDA, AdamW with $t$3, learning rate $t$4 with cosine annealing, 150k iterations, a batch of 256 images across sequences, and a curriculum sequence length from 2 to 20 frames between 50k and 150k iterations. The Base Encoder and Decoder are initialized with UniK3D or UniDepthV2 and frozen; the propagation encoder is ConvNeXt-Tiny; the default flow source is DIS, always refined via FlowRefine (Piccinelli et al., 11 Dec 2025).

5. Empirical performance and ablation findings

VeloDepth is evaluated zero-shot on ScanNet, Sintel, Bonn-RGBD, and TUM-RGBD. The reported metrics include depth accuracy $t$5 and $t$6, and temporal consistency $t$7 and $t$8 (Piccinelli et al., 11 Dec 2025). In aggregate affine-invariant evaluation over TUM, ScanNet, Sintel, and Bonn, UniDepthV2 achieves $t$9 and $\mathcal{F}_0=\{\mathbf{F}_{i,0}\}$0, whereas VeloDepth achieves $\mathcal{F}_0=\{\mathbf{F}_{i,0}\}$1 and $\mathcal{F}_0=\{\mathbf{F}_{i,0}\}$2, with FPS$\mathcal{F}_0=\{\mathbf{F}_{i,0}\}$3 and FPS$\mathcal{F}_0=\{\mathbf{F}_{i,0}\}$4. On metric evaluation, UniDepthV2 attains $\mathcal{F}_0=\{\mathbf{F}_{i,0}\}$5 and $\mathcal{F}_0=\{\mathbf{F}_{i,0}\}$6, while VeloDepth reports $\mathcal{F}_0=\{\mathbf{F}_{i,0}\}$7 and $\mathcal{F}_0=\{\mathbf{F}_{i,0}\}$8. The paper characterizes this as a slight reduction in per-frame accuracy but a large increase in temporal consistency and roughly doubled speed relative to the base model.

Against other online video methods, the reported aggregate affine-invariant results are: NVDS with $\mathcal{F}_0=\{\mathbf{F}_{i,0}\}$9, $\mathbf{P}_0$0, and 3.9 FPS; VideoDA with $\mathbf{P}_0$1, $\mathbf{P}_0$2, and 11.3 FPS; and VGG-T with $\mathbf{P}_0$3, $\mathbf{P}_0$4, and 2.3 FPS GPU (Piccinelli et al., 11 Dec 2025). VeloDepth is therefore positioned on the reported Pareto frontier, combining comparable accuracy with higher consistency and substantially higher throughput than these alternatives.

The ablation studies isolate several components. Adding propagated depth alone greatly improves temporal consistency but not accuracy, because the network mainly learns to output zero residual. Adding flow alone improves both accuracy and consistency. The best performance occurs when both propagated depth and flow are used. Gating propagated neck features is reported as crucial, while inverse gating on RGB hurts performance. Removing the consistency loss and temporal flip augmentation reduces both accuracy and consistency. Flow quality matters, but the gains show diminishing returns; refined DIS performs close to high-quality RAFT flow. Propagation plus keyframe initialization gives the best trade-off, whereas running the Base Encoder on every frame improves accuracy but not temporal consistency (Piccinelli et al., 11 Dec 2025).

Runtime and model-scale results underline the efficiency claim. On an RTX 3090 at 0.5 MP, VeloDepth runs at 26.2 FPS, compared with 13.4 FPS for the UniDepthV2 base model and 25.3 FPS for CS-LSTM; on Apple M1 Pro with MPS, it runs at 2.7 FPS. The full model has 409.4M parameters and 0.7T FLOPs, while VGG-T has 1261M parameters and 8.8T FLOPs, and DepthCrafter has 1524.6M parameters and 27.1T FLOPs (Piccinelli et al., 11 Dec 2025).

6. Position in the literature, limitations, and downstream relevance

Within the depth-estimation literature, VeloDepth is explicitly positioned as neither a frame-by-frame monocular depth estimator nor a heavy offline video model. Relative to monocular depth estimation systems such as DPT, MiDaS, UniDepth, Metric3D, and DepthAnything, it augments an existing metric depth model with temporal propagation. Relative to offline video systems such as CVD, R-CVD, DepthCrafter, ChronoDepth, VideoDA, and VGG-T, it remains online, metric, and lighter. Relative to online RNN-based video depth approaches such as CS-LSTM and NVDS, it replaces recurrent time modeling with geometric flow-based warping plus residual corrections, while reusing previous neck features and depth directly (Piccinelli et al., 11 Dec 2025).

The paper identifies four specific novelties: propagation at feature level plus metric 3D, flow-guided gating of residual corrections through $\mathbf{P}_0$5 and $\mathbf{P}_0$6, the pose-agnostic radial-distance consistency loss, and the keyframe-plus-propagation design with a frozen base model (Piccinelli et al., 11 Dec 2025). A plausible implication is that VeloDepth is best understood as a systems recipe: given a sufficiently strong monocular metric base model, one can obtain temporally stable online video depth by propagating deep features and enforcing motion-aware consistency, rather than by retraining an end-to-end video transformer.

The reported limitations are equally specific. If the base model’s keyframe prediction is wrong, those errors propagate across fast frames. High motion and large disocclusions degrade accuracy because flow and propagation assumptions break down. The method is robust to moderate flow errors but still depends on flow quality. The propagation module is lightweight and often matches but does not surpass the base model’s per-frame accuracy ceiling (Piccinelli et al., 11 Dec 2025).

In relation to adjacent arXiv work, MOVEDepth is a distinct earlier line that uses camera velocity to guide multi-frame depth hypothesis selection and cost-volume construction in self-supervised depth learning, rather than propagating deep features from previous predictions (Wang et al., 2022). DepthVLA, by contrast, uses a pretrained depth expert inside a vision-language-action policy to improve spatial reasoning in robot manipulation, showing a downstream control setting in which depth-centric representations are integrated at the policy level rather than used solely for perception (Yuan et al., 15 Oct 2025). These comparisons suggest a broader pattern: recent depth research has moved beyond isolated frame-wise prediction toward explicit integration of geometric priors, temporal structure, and reusable depth representations across perception and control.