Consistent Direct Time-of-Flight Video Depth Super-Resolution

Abstract: Direct time-of-flight (dToF) sensors are promising for next-generation on-device 3D sensing. However, limited by manufacturing capabilities in a compact module, the dToF data has a low spatial resolution (e.g., $\sim 20\times30$ for iPhone dToF), and it requires a super-resolution step before being passed to downstream tasks. In this paper, we solve this super-resolution problem by fusing the low-resolution dToF data with the corresponding high-resolution RGB guidance. Unlike the conventional RGB-guided depth enhancement approaches, which perform the fusion in a per-frame manner, we propose the first multi-frame fusion scheme to mitigate the spatial ambiguity resulting from the low-resolution dToF imaging. In addition, dToF sensors provide unique depth histogram information for each local patch, and we incorporate this dToF-specific feature in our network design to further alleviate spatial ambiguity. To evaluate our models on complex dynamic indoor environments and to provide a large-scale dToF sensor dataset, we introduce DyDToF, the first synthetic RGB-dToF video dataset that features dynamic objects and a realistic dToF simulator following the physical imaging process. We believe the methods and dataset are beneficial to a broad community as dToF depth sensing is becoming mainstream on mobile devices. Our code and data are publicly available: https://github.com/facebookresearch/DVSR/

• The paper proposes DVSR and HVSR methods that fuse low-res dToF depth data with high-res RGB frames to significantly boost depth accuracy.
• It utilizes multi-frame fusion and integrates histogram information to reduce spatial ambiguity and improve temporal coherence.
• Empirical evaluations show notable improvements, with HVSR reducing absolute error from 59.2 mm to 27.5 mm in dynamic indoor scenarios.

Consistent Direct Time-of-Flight Video Depth Super-Resolution

The paper, "Consistent Direct Time-of-Flight Video Depth Super-Resolution," addresses the challenges posed by the lower spatial resolution of direct time-of-flight (dToF) sensors compared to conventional sensors used in 3D sensing applications. dToF sensors are advantageous due to their high accuracy, compact form factor, and low power consumption. However, their low spatial resolution limits the utility of the raw depth data they produce, necessitating a super-resolution step to reconstruct high-resolution depth maps suitable for downstream tasks.

Technical Approach

The paper introduces two novel methods: Direct Time-of-Flight Depth Video Super-Resolution (DVSR) and Histogram Video Super-Resolution (HVSR), both aiming to enhance dToF depth data by leveraging high-resolution RGB frames in a multi-frame context. Unlike traditional RGB-guided depth enhancement approaches, which often operate on single frames, this work advocates for multi-frame processing to reduce spatial ambiguity and improve temporal stability.

1. Multi-frame Fusion: The authors propose a deep learning-based framework for fusing low-resolution dToF data with high-resolution RGB data over multiple frames. By leveraging temporal correlations, the DVSR approach enhances both geometric accuracy and temporal coherence, outperforming per-frame processing baselines.
2. Incorporation of Histogram Information: dToF sensors inherently generate a histogram of depth information for each pixel. The HVSR method exploits this characteristic by integrating histogram information into the network's processing pipeline, thereby further reducing spatial ambiguity and improving depth estimation in challenging scenes with complex geometries.

Empirical Evaluation

The authors evaluate their methods on a newly introduced synthetic dataset termed DyDToF, designed to mimic real-world scenarios more closely. This dataset includes dynamic indoor environments with animated objects, addressing the need for training data that reflects complexities encountered in practical applications. The paper reports significant improvements in prediction accuracy and temporal stability, with HVSR achieving superior performance by leveraging additional histogram-based insights.

Numerical Results

Notably, empirical results demonstrate that HVSR considerably outperforms both state-of-the-art per-frame methods and the proposed DVSR when evaluated on various datasets. For example, on the TarTanAir dataset, HVSR achieves an absolute error (AE) of 27.5 mm compared to 59.2 mm by a per-frame baseline. The temporal endpoint error (TEPE), a metric indicating temporal coherence, also shows substantial improvement, indicating the method's robustness in handling dynamic scenes.

Implications and Future Directions

This work has practical implications for enhancing real-time 3D sensing capabilities on mobile devices, where dToF sensors are increasingly prevalent. The multi-frame processing framework can be adapted to enhance other depth sensing technologies, such as stereo or indirect time-of-flight (iToF) sensors, suggesting a wide scope for application.

The introduction of the DyDToF dataset fills a critical gap by providing publicly available synthetic sequences that facilitate the training and evaluation of depth estimation models in varied and dynamic scenarios. Future research could focus on extending this approach to other sensing modalities and refining the framework to handle hardware imperfections and noise more effectively. Additionally, further exploration into integrating this method with real-world captured data would solidify its position in practical AR/VR applications.

The paper provides a meaningful advancement in the field of depth sensing and super-resolution, offering insights and tools that other researchers can leverage for advancing 3D sensing technologies on next-generation devices.