ICDepth: Monocular Video Depth Estimation

• ICDepth is a monocular video depth estimation framework that repurposes a pre-trained text-to-video diffusion transformer using in-context conditioning to achieve temporal consistency, geometric accuracy, and cross-domain generalization.
• SAND-Attention and Semantic-Resolution Feature Modulation are two task-specific mechanisms that align representations and inject semantic and resolution priors for precise dense depth prediction.
• Empirical evaluations show ICDepth achieves state-of-the-art performance on datasets like Sintel, ScanNet, KITTI, and Bonn while demonstrating data efficiency and robustness under domain shifts.

Searching arXiv for ICDepth and closely related video depth estimation papers to ground the article in current literature. ICDepth is a monocular video depth estimation framework that repurposes a pre-trained text-to-video diffusion transformer, specifically Wan 2.1, through In-Context Conditioning (ICC) to estimate depth from RGB video while targeting temporal consistency, geometric accuracy, and cross-domain generalization (He et al., 2 Jul 2026). The method is positioned against a common trade-off in the literature: discriminative approaches provide strong per-frame accuracy but can exhibit temporal drift under limited context windows, whereas generative approaches improve consistency and generalization but typically require more extensive training data and may lack geometric precision. Within this setting, ICDepth introduces two task-specific mechanisms—SAND-Attention and Semantic-Resolution Feature Modulation (SRFM)—to adapt a video-generation backbone to dense geometric prediction (He et al., 2 Jul 2026).

1. Conceptual framing and problem setting

Monocular video depth estimation requires simultaneous control over temporal consistency, geometric fidelity, and robustness across diverse visual domains. ICDepth is designed for precisely this setting. Its central premise is that pre-trained text-to-video diffusion transformers have already internalized rich spatio-temporal priors from tens of millions of videos, and that these priors can be redirected toward depth estimation through conditioning rather than through a wholesale redesign of the backbone (He et al., 2 Jul 2026).

The framework adopts In-Context Conditioning as the mechanism for this redirection. In the formulation given for ICDepth, an RGB video $V_I\in\mathbb R^{T\times H\times W\times3}$ is encoded into conditioning latents $\mathbf{c}\in\mathbb R^{n\times d}$, while the corresponding depth video $V_D$ is encoded into clean depth latents $\mathbf{z}_0\in\mathbb R^{n\times d}$. At diffusion timestep $t$, the clean depth latents are corrupted into $\mathbf{z}_t$ and concatenated with the RGB latents along the token dimension:

$$\mathbf{s}_t = [\,\mathbf{z}_t;\,\mathbf{c}\,]\;\in\;\mathbb R^{2n\times d}.$$

The transformer $u_\Theta$ is then trained, via flow matching, to predict the instantaneous velocity field that recovers $\mathbf{z}_0$, and inference proceeds by solving the corresponding ODE from random $\mathbf{z}_T$ to produce a depth video conditioned on the input RGB (He et al., 2 Jul 2026).

A key feature of this design is that it is described as “non-invasive”: no new attention layers or convolutions need to be injected into the pre-trained backbone. Instead, ICDepth relies on the model’s native self-attention and cross-attention patterns, provided that attention is constrained so that alignment is preserved and noisy depth tokens do not corrupt the conditioning stream. This design requirement motivates the SAND-Attention mechanism.

2. In-Context Conditioning as adaptation strategy

ICDepth uses ICC to reinterpret a video-generation model as a dense predictor. In the original generative setting, the denoising backbone learns a vector field

$\mathbf{c}\in\mathbb R^{n\times d}$0

whose integration yields a video sample consistent with the conditioning input. ICDepth preserves this denoising formalism but changes the semantics of the latent stream: the target latent sequence corresponds to depth rather than RGB video, while the conditioning tokens are the encoded RGB frames (He et al., 2 Jul 2026).

The framework’s use of ICC distinguishes it from a simpler channel-concatenation strategy. The reported ablation on Sintel states that replacing ICC with standard channel concatenation degrades AbsRel from $\mathbf{c}\in\mathbb R^{n\times d}$1 to $\mathbf{c}\in\mathbb R^{n\times d}$2 (He et al., 2 Jul 2026). This suggests that the benefit is not merely the presence of RGB and depth information in a shared tensor, but the exploitation of a pre-trained attention architecture whose token interactions already encode useful spatial-temporal structure.

A plausible implication is that ICC functions as an adaptation interface between generative pretraining and geometric estimation. Rather than requiring domain-specific architectural augmentation at every stage, it enables the model to reuse internal priors while exposing failure modes—especially positional misalignment and contamination from noisy latents—that must be corrected explicitly.

3. SAND-Attention: alignment and noise decoupling

SAND-Attention is introduced to solve two problems in vanilla ICC: first, diffusion noise in the depth latent stream can leak into the clean RGB conditioning tokens; second, standard Rotary Positional Encoding can fail to respect the one-to-one spatial-temporal correspondence between depth and RGB tokens (He et al., 2 Jul 2026). The mechanism therefore combines shared positional indexing with decoupled attention flow.

The first component is shared RoPE. ICDepth assigns the same positional index $\mathbf{c}\in\mathbb R^{n\times d}$3 to the noisy depth token $\mathbf{c}\in\mathbb R^{n\times d}$4 and the corresponding RGB token $\mathbf{c}\in\mathbb R^{n\times d}$5. Queries and keys are split into depth and conditioning parts and then transformed with RoPE using identical index maps:

$\mathbf{c}\in\mathbb R^{n\times d}$6

$\mathbf{c}\in\mathbb R^{n\times d}$7

The stated purpose is to ensure that the attention score between a depth token and its RGB counterpart at the same space-time location is computed with identical positional phase, thereby preserving exact alignment (He et al., 2 Jul 2026).

The second component is decoupled, unidirectional attention. Clean RGB tokens attend only among themselves:

$\mathbf{c}\in\mathbb R^{n\times d}$8

whereas noisy depth queries attend to both depth and RGB keys and values:

$\mathbf{c}\in\mathbb R^{n\times d}$9

The two outputs are concatenated as

$V_D$0

after which computation proceeds through the usual MLP. The description emphasizes that no gradients or noise flow back into $V_D$1, preserving the integrity of the clean conditioning signal (He et al., 2 Jul 2026).

The paper also gives a multi-head formulation in which the same logic is applied per attention head. Empirically, the ablations identify SAND-Attention as structurally necessary: omitting it and reverting to full attention degrades Sintel AbsRel to $V_D$2, while dropping only RoPE alignment yields $V_D$3; removing decoupled attention alone results in a smaller but still measurable degradation to $V_D$4 (He et al., 2 Jul 2026). These results are presented as evidence that both precise alignment and noise decoupling are essential components rather than implementation details.

4. Semantic-Resolution Feature Modulation

Even with conditioning and attention control in place, a text-to-video diffusion transformer is not explicitly optimized for geometric precision. ICDepth addresses this by introducing Semantic-Resolution Feature Modulation, which injects two priors into every transformer block’s MLP output: a DINOv2 semantic prior and a resolution prior (He et al., 2 Jul 2026).

For the semantic prior, per-frame DINOv2 features $V_D$5 are extracted from the RGB frames, pooled, and reshaped into an embedding $V_D$6. A small MLP $V_D$7 produces scale and shift vectors:

$V_D$8

Given the layer-$V_D$9 MLP output $\mathbf{z}_0\in\mathbb R^{n\times d}$0, the depth portion $\mathbf{z}_0\in\mathbb R^{n\times d}$1 is modulated as

$\mathbf{z}_0\in\mathbb R^{n\times d}$2

This modulation is described as adding semantic awareness, particularly around edges and objects (He et al., 2 Jul 2026).

For the resolution prior, the spatial dimensions $\mathbf{z}_0\in\mathbb R^{n\times d}$3 are encoded into sinusoidal embeddings and processed by another MLP $\mathbf{z}_0\in\mathbb R^{n\times d}$4 to produce

$\mathbf{z}_0\in\mathbb R^{n\times d}$5

The semantically modulated depth tokens are then further transformed:

$\mathbf{z}_0\in\mathbb R^{n\times d}$6

The unchanged conditioning tokens are re-concatenated before the next residual block (He et al., 2 Jul 2026).

The corresponding ablation results indicate that SRFM is a major contributor to geometric quality. Without SRFM, Sintel AbsRel rises from $\mathbf{z}_0\in\mathbb R^{n\times d}$7 to $\mathbf{z}_0\in\mathbb R^{n\times d}$8. Removing only the DINOv2 insert yields $\mathbf{z}_0\in\mathbb R^{n\times d}$9, and removing only the resolution embedding yields $t$0 (He et al., 2 Jul 2026). The reported interpretation is that semantic and resolution modulation jointly improve geometric accuracy; a plausible implication is that they compensate for inductive biases that are strong for video generation but insufficiently constrained for dense depth boundaries and variable aspect ratios.

5. Training protocol, supervision, and inference regime

ICDepth is trained on a compact synthetic mixture comprising Virtual KITTI (VKITTI), a subset of TartanAir and TartanGround with single-direction cameras, and a synthetic subset of OmniWorld (He et al., 2 Jul 2026). The total training set contains approximately $t$1 million frames, which the paper describes as a $t$2–$t$3 reduction relative to competing generative methods. This data-efficiency claim is central to the method’s positioning.

The training regime uses multi-resolution clips containing between $t$4 and $t$5 frames, with temporal extent adjusted inversely to spatial resolution so that $t$6 tokens (He et al., 2 Jul 2026). Supervision is based on a flow matching objective masked to valid depth regions. Let $t$7 denote the downsampled binary mask of valid VAE-latent locations, corresponding to regions where true depth is less than $t$8. The loss is

$t$9

with $\mathbf{z}_t$0 and $\mathbf{z}_t$1 the target velocity field (He et al., 2 Jul 2026).

Optimization is performed for $\mathbf{z}_t$2 epochs on $\mathbf{z}_t$3 NVIDIA H800 GPUs with batch size $\mathbf{z}_t$4 per GPU, learning rate $\mathbf{z}_t$5, and gradient accumulation over $\mathbf{z}_t$6 steps. Inference uses $\mathbf{z}_t$7–$\mathbf{z}_t$8 DDIM sampling steps as the reported speed–accuracy trade-off (He et al., 2 Jul 2026).

A common misconception about diffusion-based depth estimation is that generative pretraining alone should suffice once conditioning is provided. The ICDepth formulation does not support that view. The architecture adds explicit attention constraints and feature modulation, and the ablations indicate that these additions materially affect performance. Another misconception is that data efficiency necessarily implies minimal temporal context; in this case, the model is explicitly trained with clips up to $\mathbf{z}_t$9 frames, suggesting that compact data usage and extended spatio-temporal context are treated as complementary rather than opposing design choices.

6. Empirical performance, generalization, and limitations

In zero-shot evaluation, ICDepth is reported to set new state of the art on Sintel, ScanNet, KITTI, and Bonn (He et al., 2 Jul 2026). On Sintel with $$\mathbf{s}_t = [\,\mathbf{z}_t;\,\mathbf{c}\,]\;\in\;\mathbb R^{2n\times d}.$$0 frames, it reduces AbsRel to $$\mathbf{s}_t = [\,\mathbf{z}_t;\,\mathbf{c}\,]\;\in\;\mathbb R^{2n\times d}.$$1 and raises $$\mathbf{s}_t = [\,\mathbf{z}_t;\,\mathbf{c}\,]\;\in\;\mathbb R^{2n\times d}.$$2 to $$\mathbf{s}_t = [\,\mathbf{z}_t;\,\mathbf{c}\,]\;\in\;\mathbb R^{2n\times d}.$$3. On ScanNet with $$\mathbf{s}_t = [\,\mathbf{z}_t;\,\mathbf{c}\,]\;\in\;\mathbb R^{2n\times d}.$$4 frames, it achieves AbsRel $$\mathbf{s}_t = [\,\mathbf{z}_t;\,\mathbf{c}\,]\;\in\;\mathbb R^{2n\times d}.$$5 and $$\mathbf{s}_t = [\,\mathbf{z}_t;\,\mathbf{c}\,]\;\in\;\mathbb R^{2n\times d}.$$6, described as nearly matching top specialized methods. On KITTI and Bonn with $$\mathbf{s}_t = [\,\mathbf{z}_t;\,\mathbf{c}\,]\;\in\;\mathbb R^{2n\times d}.$$7 frames, it reports AbsRel $$\mathbf{s}_t = [\,\mathbf{z}_t;\,\mathbf{c}\,]\;\in\;\mathbb R^{2n\times d}.$$8 and $$\mathbf{s}_t = [\,\mathbf{z}_t;\,\mathbf{c}\,]\;\in\;\mathbb R^{2n\times d}.$$9 $u_\Theta$0, respectively (He et al., 2 Jul 2026).

The paper also emphasizes robustness under domain shift. On unseen low-light Sintel, ICDepth’s relative $u_\Theta$1 drop is reported as only $u_\Theta$2, compared with $u_\Theta$3–$u_\Theta$4 for alternatives (He et al., 2 Jul 2026). This suggests that the pre-trained spatial-temporal priors inherited from the video diffusion backbone remain useful under appearance changes that commonly destabilize frame-centric discriminative models.

The reported limitations are equally explicit. Inference remains slow because the method is diffusion-based, at approximately $u_\Theta$5 seconds for $u_\Theta$6 frames at $u_\Theta$7 (He et al., 2 Jul 2026). The model is currently trained for relative depth, up to per-video scale-shift, rather than absolute metric depth. The discussion also notes remaining domain biases in extreme settings such as very high-speed motion or non-Lambertian materials, and states that static backgrounds and motion parallax cues are underused (He et al., 2 Jul 2026).

These limitations place ICDepth within a broader methodological tension. It demonstrates that a generative video transformer can be converted into a state-of-the-art depth estimator through careful conditioning, alignment control, and task-specific priors, but it does not eliminate the classical costs of diffusion inference or the challenge of recovering absolute scale. A plausible implication is that subsequent work may focus less on proving the viability of the paradigm and more on reducing its computational overhead and extending it toward metric and multimodal depth estimation.