Deep Virtual Stereo Odometry (DVSO)

• Deep Virtual Stereo Odometry (DVSO) is a monocular visual odometry method that uses deep neural networks to predict metric depth and injects virtual stereo constraints into a direct sparse odometry pipeline.
• DVSO overcomes traditional scale ambiguity and drift issues by leveraging virtual stereo cues from learned disparity maps to produce scale-consistent 3D reconstructions.
• By integrating adversarial domain adaptation and mutual reinforcement strategies, DVSO achieves performance competitive with stereo systems on benchmarks like KITTI.

Deep Virtual Stereo Odometry (DVSO) is a paradigm for monocular visual odometry (VO) that integrates deep neural network-based metric depth prediction into optimization-based VO pipelines, thereby overcoming the inherent scale ambiguity and drift of classical monocular approaches. By leveraging virtual or learned stereo cues in the form of predicted disparities, DVSO provides metric depth supervision to direct visual odometry, yielding accuracy competitive with stereo VO, while requiring only monocular input. The methodology has evolved from semi-supervised learning on real stereo rigs (Yang et al., 2018) to domain-adaptive learning from virtual environments (Zhang et al., 2022), ensuring scale-consistent monocular VO without dependence on real-world stereo ground truth at training or inference time.

1. Background and Motivation

Traditional monocular VO estimates camera poses $T_i \in SE(3)$ and scene structure (depths $d_p$ for observed points $p$) using only the geometric information present in monocular image sequences $I_1, I_2, \ldots$. This process is inherently limited by the projective nature of monocular vision: depth and absolute scale cannot be directly observed, leading to physically unconstrained reconstructions, global scale ambiguity, and, over long trajectories, gradual scale drift—manifested as cumulative errors in translation and depth estimation. Classical monocular VO thus yields $T_i \sim s\cdot T_i^*$ and $d_p \sim d_p^*/s$ for some unknown scale $s>0$, with $s$ potentially drifting over time (Yang et al., 2018).

Prior approaches to mitigate scale drift relied on additional sensors (stereo, inertial, ground truth), but these solutions incur logistical, calibration, and dataset limitations. Recent methods instead learn metric depth from data. DVSO introduces the idea of injecting deep metric depth prediction into the VO backend as "virtual stereo," tying monocular estimation to metric constraints without hardware dependencies.

2. Core Methodology

The core mechanism of DVSO is the integration of deep-learned, metric-aligned disparity (inverse depth) maps into a direct sparse odometry pipeline via a virtual stereo term. For each monocular frame, a deep network predicts left and right disparities ($D^L$, $D^R$), trained to act as if every monocular image participates in a (virtual) stereo pair. At inference, only a single camera is present; right disparities represent the epipolar shift expected under a notional baseline.

2.1. Optimization Formulation

The direct sparse odometry backend maintains a set of active keyframes $\mathcal{F}$ and sparse points $\mathcal{P}_i$ each with inverse depth $d_p$. For observed points $p$ in host frame $i$ and observing frames $j$, the classical direct photometric error is

$$E_{ij}^p = \omega_p \cdot \left\| [I_j(p') - b_j] - \left(\frac{e^{a_j}}{e^{a_i}}\right)[I_i(p)-b_i] \right\|_\gamma$$

where $p'$ is the reprojected location of $p$ in $j$, $\omega_p$ robustifies against outliers, $a_i,b_i$ model affine intensity changes, and $\|\cdot\|_\gamma$ is the Huber norm.

DVSO adds a virtual stereo term by treating the network-predicted right disparity $D^R$ as defining a synthetic stereo partner:

$p^\dagger = \Pi_k ( \Pi_k^{-1}(p, d_p) + t_b )$

$$I_i^\dagger[p^\dagger] = I_i[ p^\dagger - (D^R(p^\dagger), 0)^\top ]$$

$$E_{vs}^p = \omega_p \cdot \| I_i^\dagger[ p^\dagger ] - I_i[ p ] \|_\gamma$$

The full energy minimized is

$E_{total} = E_{photo} + \lambda \cdot E_{vs}$

where $\lambda$ is a coupling weight calibrated per sequence.

2.2. Metric Depth Initialization

New keyframe points are initialized with inverse depth

$d_p = D^L(p) / (f_x \cdot b)$

where $f_x$ and $b$ are the focal length and baseline used during network training, respectively. This achieves metric depth alignment, avoiding random scale and drift.

Points are selected based on image gradients, with occlusion masked by a left-right consistency check:

$$e_{lr} = | D^L(p) - D^R(p') | > 1 \text{ contains occlusions,}$$

with $p' = p - (D^L(p), 0)^\top$.

3. Deep Disparity Prediction Networks

DVSO architectures for depth prediction have evolved from a semi-supervised StackNet (Yang et al., 2018) to a domain-adaptive architecture (Zhang et al., 2022):

3.1. StackNet (Yang et al., 2018)

• SimpleNet: Fully-convolutional ResNet-50 encoder-decoder predicting 4-scale disparities for left/right images.
• ResidualNet: Refines SimpleNet outputs by leveraging reconstructed stereo images and disparity maps, learning residual corrections.
• Training is semi-supervised:
  • Photometric loss (self-supervision): Matches original and reconstructed images via a mixture of SSIM and $L_1$ loss.
  • Sparse supervised loss: Matches predicted disparities to those from sparse Stereo DSO.
  • Left-right consistency, edge-aware smoothness, occlusion regularizer round out the objectives.
• Training schedule employs targeted fine-tuning on labeled and unlabeled splits, final post-processing via left-right flicker reduction.

3.2. Adversarial Domain-Aware Networks (Zhang et al., 2022)

• Shared Encoder ($M_S$): ResNet18-based encoder jointly processes real and virtual images, driving feature space domain alignment.
• Disparity Decoder ($M_D$): Up-convolutional module predicting per-pixel disparities, output at four scales.
• Auxiliary Pose Regressor ($M_P$): Small CNN predicting frame-to-frame $SE(3)$ transformations.
• Domain Discriminator ($M_{adv}$): 4-layer CNN in a WGAN-GP adversarial setting to align features from real and virtual domains.
• Losses:
  • Outer adversarial loss ($L_{adv}$), gradient penalty ($L_{gp}$), reconstruction loss ($L_{rec}$).
  • Task loss ($L_{task}$) comprises:
  • Photometric consistency ($L_{pc}$) for both real and virtual data, mixing SSIM and $L_1$.
  • Edge-aware disparity smoothness ($L_s$).
  • Supervised virtual disparity ($L^v_{gt}$) with perfect simulator ground truth.
  • Stereo photometric ($L^v_{sc}$), enforcing correct baseline in virtual data.

4. Mutual Reinforcement and Bidirectional Learning

Traditional pipelines like DVSO and earlier deep stereo-augmented VO methods unidirectionally inject learned depth as a prior into the optimization backend, but do not propagate bundle-adjusted geometric corrections back to the depth network. (Zhang et al., 2022) introduces mutual reinforcement: after the VO backend refines trajectory estimates via bundle adjustment, a "backward reinforcement" photometric loss re-renders current frames from optimized poses, incorporating this synthetic supervision into subsequent network fine-tuning.

Formally, for the optimized poses $T^{r,*}_{(i-1,i)}$ and predicted depth $D^r_L$, the backward photometric loss is

$$L^{r,*}_{pc} = \frac{1}{N} \sum_i \min_\delta L( I^r_L(p_i), I^r_\delta( \text{warp}(p_i; T^*_\delta, D^r_L ) ) )$$

This bidirectional loop measurably improves both depth and pose estimation, reducing training and validation VO error by 10–15% in few fine-tuning epochs on KITTI (Zhang et al., 2022).

5. Experimental Results and Comparative Analysis

5.1. Accuracy Results

On the KITTI benchmark (Yang et al., 2018, Zhang et al., 2022):

Method	Translational Error $t_{rel}$	Rotational Error $r_{rel}$	Comments
Mono DSO (Sim(3))	$65\%$ drift	$0.21^\circ$	Baseline monocular, no metric scale
DVSO ("in,vs,lr,tb")	$0.77\%$	$0.20^\circ$	Monocular, with virtual stereo pipeline
Stereo DSO	$0.84\%$	$0.20^\circ$	Stereo rig
ORB-SLAM2 (stereo)	$0.81\%$	$0.26^\circ$	Stereo rig

DVSO narrows the accuracy gap between monocular and stereo approaches, even slightly outperforming stereo baselines under the ablation setting of KITTI without global bundle adjustment or loop closure.

5.2. Depth Prediction

On KITTI depth evaluation (Eigen split):

Method	RMSE (0–80m)	$\delta<1.25$ Accuracy
StackNet (DVSO)	4.44 m	0.898
Kuznietsov et al.	4.62 m	0.875
Godard et al.	4.94 m	0.873

StackNet achieves stronger absolute error and accuracy, with particularly robust recovery of thin structures.

5.3. Mutual Reinforcement

With the mutual reinforcement strategy (Zhang et al., 2022), VRVO achieves on KITTI sequence 09:

• Without reinforcement: $1.81\%$ translational error, 5.96 m ATE.
• With reinforcement: $1.55\%$ translational error, 4.39 m ATE.
• VRVO closes 360 m loops with almost perfect alignment; DSO drifts tens of meters.

6. Practical and Methodological Insights

• DVSO and its virtual-reality variant VRVO demonstrate that unlimited virtual stereo data can be synthesized using contemporary simulation engines (e.g., Unity, Unreal, Habitat, TartanAir), sidestepping the expense of stereo rig collection and calibration.
• Adversarial feature-space domain adaptation is often more stable than image-level translation (e.g., CycleGAN). Feature discriminators operate on compact latent codes, rendering the network lightweight.
• The virtual stereo term in bundle adjustment preserves geometric rigor, functioning as a learned extension of stereo reprojection error based on deep predictions.
• Mutual reinforcement induces further robustness, with a handful of fine-tuning epochs improving both motion and depth.
• Residual domain gap between simulated and real-world images may affect disparities in out-of-distribution settings. The choice of virtual baseline $t^v_b$ should approximate real camera parameters; misalignment can degrade performance. The VO backend's additional energy term increases per-frame computation by $10$–$20\%$.
• Limitations persist in scenes with non-Lambertian surfaces, areas where rendered simulation diverges from photorealism, and for domains outside those represented in training.

7. Outlook and Future Directions

Potential extensions for DVSO, as suggested in the literature, include end-to-end fine-tuning of the depth network within the VO pipeline, online adaptation for new scenes or camera setups, and application to domains beyond driving (e.g., indoor, aerial) via retraining or expanded domain adaptation (Yang et al., 2018, Zhang et al., 2022). Continued advances in photorealistic simulation and unsupervised domain transfer are expected to further bridge the gap between virtual and real environments, facilitating scale-consistent, robust monocular VO across broad application scenarios.