Test-Time Depth Adaptation

• Test-Time Depth Adaptation is a framework that dynamically fine-tunes depth models on-the-fly using minimal parameters to address domain shifts.
• It employs techniques such as proxy-based supervision, low-rank decoder tuning, and energy minimization to optimize predictions without source data.
• Empirical evaluations reveal substantial improvements in RMSE and MAE across varied indoor and outdoor environments while ensuring computational efficiency.

Test-time depth adaptation encompasses a suite of methodologies for dynamically modifying depth prediction or completion models during deployment, optimizing their performance on each test instance or stream in the presence of distribution shifts. These approaches address the persistent domain gap problem in depth estimation, where models pretrained on source domains degrade on real-world or target environments. Test-time mechanisms span continuous parameter adaptation, dynamic architectural modification, energy minimization, plug-in adaptation modules, and efficient parameter-efficient fine-tuning. The scope includes dense and sparse depth completion, monocular and stereo estimation, and foundation model zero-shot settings.

1. Problem Domain and Motivation

Depth completion and estimation models typically experience significant performance drops under domain shift: the statistical distributions of images and depth samples in new environments differ from those seen in source training. Standard domain adaptation requires access to source data or computation over large batches of target data, which are often unavailable in robotic, embedded, or time-critical deployments. Test-time adaptation (TTA) aims to mitigate error for each test instance—single or streaming—by adaptively updating either the model parameters, its architectural pathway, or its output transformation, without access to ground-truth or revisiting historical data.

Key scenarios include:

• Inferring dense depth from synchronized RGB and sparse sensor measurements without source data at deployment (Park et al., 2024, Seo et al., 2 Mar 2026, Chung et al., 8 Aug 2025, Ke et al., 16 Feb 2026).
• Recovering metric depth from monocular predictions via sparse external cues without fine-tuning (Marsal et al., 2024).
• Adapting monocular or stereo models for dynamic environments, including robustness to moving objects, weather, or illumination variations (Sung et al., 7 Nov 2025, Ko et al., 13 Nov 2025).
• Accelerating or improving predictions in vision or LLMs by per-input architectural adaptation or iterative refinement (Li et al., 10 Jul 2025, Mathur et al., 17 Jul 2025).

2. Algorithmic Approaches to Test-Time Depth Adaptation

Contemporary techniques for test-time depth adaptation include proxy-supervision, parameter-efficient tuning, energy-based minimization, architectural morphing, and robust metric rescaling.

2.1 Proxy-based and Modality-alignment Methods

ProxyTTA (Park et al., 2024) introduces an embedding module trained in the source domain to map features from sparse depth (less shifted under domain changes) to joint RGB+depth features. At test time, sparse depth proxies regularize adaptation of a deep RGB-branch adaptation layer by aligning its features to the fixed-depth proxy via a cosine loss. This leverages the observation that sparse depth is less affected by covariate shift than RGB inputs. A single-pass adaptation suffices to bridge most of the domain performance gap.

2.2 Low-Rank Decoder and Parameter-Efficient Adaptation

Parameter-efficient strategies focus adaptation on a compact subspace:

• Low-rank decoder adaptation: Adapting only low-dimensional factors in decoder weights (LoRA-style) causes substantial error reduction with >90% fewer trainable parameters and drastically reduced inference latency, as depth-relevant information in foundation models is concentrated in the decoder (Seo et al., 2 Mar 2026).
• PEFT in ViT backbones: CAPA dynamically updates only LoRA adapters or prompt tokens at test time, freezing the ViT backbone and aligning model output to sparse depth via masked L1 loss. For video, sequence-level parameter sharing amortizes adaptation over frames, enforcing temporal consistency and multi-frame robustness (Ke et al., 16 Feb 2026).

2.3 Energy-Based and Adversarial Approaches

ETA (Chung et al., 8 Aug 2025) trains a local patchwise energy model to score in-distribution vs. out-of-distribution predictions, using adversarial perturbations generated on the source domain. At test-time, adaptation minimizes this energy, steering predictions to plausible in-distribution regions without source or pseudo-labels.

2.4 Architecture- and Pathway-Adaptive Depth

• CoLa for LLMs: Optimal per-input depth is achieved by selectively skipping or repeating pretrained layers, guided by rewards balancing correctness and inference cost, yielding architectural dynamism according to sample difficulty (Li et al., 10 Jul 2025).
• SELF-Transformers: For BERT or ViT-style models, each self-attention layer refines its alignment matrix iteratively at test-time until a fixed-point criterion is met, scaling computational depth adaptively to input complexity (Mathur et al., 17 Jul 2025).

2.5 Robust Non-parametric Test-Time Rescaling

Zero-shot monocular estimators can recover metric scale via RANSAC-based affine fitting against sparse sensor depths at test-time, providing competitive accuracy without model fine-tuning and preserving generalization (Marsal et al., 2024).

2.6 Instance-Aware, Pose-Agnostic, and Stereo TTA

• PITTA (Sung et al., 7 Nov 2025): Pose-agnostic, instance-aware adaptation masks dynamic objects using panoptic segmentation and reinforces dynamic-object depth accuracy with edge-alignment losses, updating only BatchNorm parameters.
• RobIA (Ko et al., 13 Nov 2025): Incorporates a parameter-efficient, row-wise attention Routed MoE and a robust AdaptBN teacher, supporting continual adaptation in streaming stereo, balancing efficiency and dense pseudo-label coverage.

3. Loss Functions, Adaptation Modules, and Efficiency

3.1 Loss Components

• Sparse depth/self-supervision: Masked L1 or L2 loss between predictions and sparse input depth at measurement locations (Park et al., 2024, Seo et al., 2 Mar 2026, Ke et al., 16 Feb 2026).
• Proxy consistency: Cosine distance between test RGB+depth embedding and hallucinated sparse-depth proxy (Park et al., 2024).
• Smoothness/edge alignment: Image-equivariant depth smoothing, or Laplacian-based edge matching between predicted depth and input intensity (Park et al., 2024, Sung et al., 7 Nov 2025).
• Energy-based: Patch-level cross-entropy between predicted and source-calibrated energy scores (Chung et al., 8 Aug 2025).
• Affine metric rescaling: Robust scale/shift fit via RANSAC and robust losses for metric alignment in monocular models (Marsal et al., 2024).

3.2 Adapted Parameter Sets

Methods universally restrict adaptation to a small number of parameters for stability and speed:

• A single conv layer (plus optional BN) in the RGB encoder branch (Park et al., 2024, Chung et al., 8 Aug 2025).
• LoRA factors or prompt tokens in ViT/Transformer blocks (Seo et al., 2 Mar 2026, Ke et al., 16 Feb 2026).
• Only BN scale/shift in the encoder (Sung et al., 7 Nov 2025).
• Row-wise MoE router weights in stereo models (Ko et al., 13 Nov 2025).

Empirical ablations confirm that such narrow adaptation produces better trade-offs and avoids catastrophic forgetting relative to full-network fine-tuning.

4. Experimental Evaluation and Quantitative Impact

Test-time depth adaptation consistently achieves significant error reductions and enhanced robustness on both indoor and outdoor scenes. Representative comparative results:

• ProxyTTA (Park et al., 2024): Average 21.1% RMSE reduction over pretrained, 15.7% over BN-Adapt+smoothness baseline.
• Low-rank decoder TTO (Seo et al., 2 Mar 2026): At rank 8, reduces MAE and RMSE by 30–50% compared to prior visual-prompt or diffusion-based TTA, at 2s/frame.
• CAPA (Ke et al., 16 Feb 2026): State-of-the-art AbsRel (%) and temporal consistency (OPW) across ScanNet, 7-Scenes, IBims, Metropolis, outperforming both vanilla and energy-learning TTA approaches.
• ETA (Chung et al., 8 Aug 2025): 6.94% MAE improvement outdoors, 10.23% indoors over previous state-of-the-art.
• PITTA (Sung et al., 7 Nov 2025): Absolute improvements in AbsRel and RMSE over non-adapted baselines and recent TTA competitors, particularly in inclement weather scenarios.
• RobIA (Ko et al., 13 Nov 2025): D1-all and EPE on DrivingStereo, DSEC, KITTI RAW—outperforming both AdaptBN-only and full fine-tuning with ∼50% fewer trainable parameters and 3× speedup.

Ablation studies universally demonstrate that omitting domain-stable modalities, loss regularization terms, or architectural dynamicity degrades both mean accuracy and robustness.

5. Practical Implementation, Limitations, and Future Directions

Test-time depth adaptation enables real-time, memory-efficient deployment of foundation models and task-specific predictors in robotics, autonomous navigation, and mobile devices:

• Maintenance of generalization properties by freezing all foundation weights and adapting minimal submodules (Ke et al., 16 Feb 2026, Marsal et al., 2024).
• Amortization of adaptation cost via shared PEFT parameters in video and streaming (Ke et al., 16 Feb 2026).
• Robustness to outliers and sensor noise through robust fitting and adversarial OOD coverage (Chung et al., 8 Aug 2025, Marsal et al., 2024).

Limitations include:

• Adaptation latency (typically 0.1–2s/image) restricts applicability in strict real-time contexts (Seo et al., 2 Mar 2026).
• Reliance on high-quality sparse cues (panoptic masks, sparse depth) may expose failure modes under sensor dropout or segmentation errors (Sung et al., 7 Nov 2025).
• Catastrophic forgetting and domain drift if more than minimal parameter subsets are updated (Park et al., 2024).

Future directions propose integrating spatially-varying adaptation, temporal priors across video, hybrid coarse/fine schemes, or online refinement of frozen experts to extend expressivity without inflating parameter count.

6. Connections Across Related Modalities and Architectures

Test-time depth adaptation unifies methodology paradigms across dense scene understanding:

• Per-image and streaming image depth completion (Park et al., 2024, Seo et al., 2 Mar 2026, Ke et al., 16 Feb 2026, Chung et al., 8 Aug 2025).
• Robust metric scale recovery in monocular foundation models (Marsal et al., 2024).
• Adaptive computational depth and architectural flexibility in vision and language transformers (Mathur et al., 17 Jul 2025, Li et al., 10 Jul 2025).
• Dense stereo estimation via instance-aware, dynamic MoE modules and unsupervised teachers (Ko et al., 13 Nov 2025).
• Pose-agnostic adaptation for monocular depth and dynamic scene composition (Sung et al., 7 Nov 2025).

The shared principle is adaptive exploitation of domain-stable features and compact per-instance optimization, enabling high-fidelity geometric prediction under non-stationary real-world conditions.