NAIMA: Semantics Aware RGB Guided Depth Super-Resolution

Abstract: Guided depth super-resolution (GDSR) is a multi-modal approach for depth map super-resolution that relies on a low-resolution depth map and a high-resolution RGB image to restore finer structural details. However, the misleading color and texture cues indicating depth discontinuities in RGB images often lead to artifacts and blurred depth boundaries in the generated depth map. We propose a solution that introduces global contextual semantic priors, generated from pretrained vision transformer token embeddings. Our approach to distilling semantic knowledge from pretrained token embeddings is motivated by their demonstrated effectiveness in related monocular depth estimation tasks. We introduce a Guided Token Attention (GTA) module, which iteratively aligns encoded RGB spatial features with depth encodings, using cross-attention for selectively injecting global semantic context extracted from different layers of a pretrained vision transformer. Additionally, we present an architecture called Neural Attention for Implicit Multi-token Alignment (NAIMA), which integrates DINOv2 with GTA blocks for a semantics-aware GDSR. Our proposed architecture, with its ability to distill semantic knowledge, achieves significant improvements over existing methods across multiple scaling factors and datasets.

• The paper presents NAIMA, a framework that leverages pretrained DINOv2 semantics via guided token attention to improve depth super-resolution.
• It uses a multi-level cross-attention mechanism to align RGB and depth features, resulting in sharper boundaries and reduced artifacts.
• Experimental results demonstrate competitive performance at high upscaling factors with improved RMSE and enhanced preservation of fine geometric details.

NAIMA: Semantics Aware RGB Guided Depth Super-Resolution

Introduction and Problem Statement

RGB-guided depth super-resolution (GDSR) aims to recover high-resolution depth maps by fusing a low-resolution depth input with a high-resolution RGB image. The challenge in GDSR is that RGB images often contain texture and color signals not aligned with geometric depth discontinuities, leading to noise propagation, blurred boundaries, and artifacts in the reconstructed depth maps. Previous works have attempted to mitigate this issue with architectures leveraging attention-driven fusion or auxiliary tasks (e.g., monocular depth estimation, gradient prediction, semantic segmentation). However, most approaches lack explicit semantic guidance, resulting in suboptimal cross-modal alignment and structural fidelity loss.

Figure 1: Blurred depth discontinuities caused by RGB noise when performing super-resolution without semantic guidance. In contrast, the semantics-aware approach leverages global contextual information to better preserve structural fidelity.

Methodology

NAIMA (Neural Attention for Implicit Multi-token Alignment) addresses the above limitations by introducing a framework that injects pretrained semantic priors—extracted from vision transformers (DINOv2)—into the GDSR pipeline. This explicit semantic guidance aligns RGB and depth features more robustly, enabling structure-aware fusion and reducing RGB-induced artifacts.

Network Architecture

The NAIMA pipeline operates as follows:

1. Semantic Encoding: High-level semantic representations are extracted from the RGB input via a pretrained DINOv2 backbone, yielding multi-level token embeddings that capture rich global context.
2. Depth Encoding: The low-resolution depth map is encoded across multiple levels using residual channel attention blocks to extract geometric features.
3. Guided Token Attention (GTA): The central contribution is the insertion of the Guided Token Attention module at each network level, which employs cross-attention to align depth features (queries) with the projected semantic tokens (keys/values), selectively injecting semantic priors into the depth representation.
4. RGB Feature Fusion: To recover high-frequency details, spatial RGB features (extracted via a residual encoder) are further fused—after the semantic injection—with the semantically enriched depth map features.
5. Upsampling: The final representation is upsampled via a dedicated upsampler composed of convolutional, deconvolutional, and residual channel attention layers, with the predicted depth output being combined with the bicubic-upsampled depth input for reconstruction refinement.

Figure 2: Overview of NAIMA’s architecture: semantic encoding from DINOv2, cross-attention-based GTA, and progressive upsampling for depth map reconstruction.

Figure 3: Internal structure of the Guided Token Attention module: depth features query projected DINOv2 semantics, resulting in a refined, context-aligned representation.

Loss Function

A gradient-aware pixel loss is utilized, defined as a combination of the $\mathcal{L}_1$ loss between prediction and ground truth and an additional $\mathcal{L}_1$ on the corresponding spatial gradients. The latter term, weighted by a hyperparameter, encourages the recovery of sharp depth discontinuities and structural consistency.

Experimental Results

Quantitative Performance

Evaluations across standard GDSR datasets (NYU_v2, RGBDD, Middlebury, Lu, and TOFDSR) and upscaling factors (4x, 8x, 16x) demonstrate that NAIMA achieves competitive or superior performance compared to state-of-the-art techniques. The highest gains are observed for large scaling factors (16x), where semantic guidance is critical due to the strong information bottleneck in the low-resolution depth input. For instance, on NYU_v2, NAIMA delivers an RMSE reduction of 0.16 $cm^2$ over competing methods at 16x.

Qualitative Analysis

Qualitative comparisons show that NAIMA substantially improves boundary sharpness and preserves fine object structures, while suppressing noise artifacts from RGB misalignment. Notable improvements are observed in cross-modal adaptivity and the integrity of complex scene geometry.

Figure 4: Depth super-resolution results (8x) illustrating that NAIMA yields superior boundary alignment and detail recovery across various benchmarks.

Figure 5: Error map: NAIMA demonstrates lowest error concentration along object boundaries in high-frequency regions.

Figure 6: Visualization of intermediate feature maps: semantic injection via GTA reduces RGB-driven noise, resulting in coherent and denoised edge representation across levels.

Additional results for the most challenging 16x and standard 4x upscaling are provided, consistently showcasing NAIMA’s ability to reconstruct sharp geometric details absent in the low-res depth input, especially in regions where RGB texture cues are misleading.

Figure 7: At 16x scale, NAIMA successfully exploits semantic priors to reconstruct sharp depth discontinuities and intricate shape features from extremely blurred inputs.

Figure 8: At 4x scale, the method enables accurate estimation of thin, fine structures and enforces boundary fidelity unmatched by baselines.

Convergence and Training Stability

NAIMA’s loss curves for all scaling factors converge without instability, indicating robust training dynamics enabled by the architectural and loss design.

Figure 9: Training loss convergence for NAIMA at different upscaling factors.

Ablation Studies

Ablation experiments removing the cross-attention-based semantic alignment (using feature addition only) or the gradient loss, reveal marked drops in performance. This indicates that semantic tokens require explicit attention-based fusion, and that preserving gradient information is essential for accurate boundary reconstruction.

Theoretical and Practical Implications

NAIMA bridges a critical gap between attention-based fusion and explicit semantic alignment for GDSR. By leveraging robust, pretrained DINOv2 embeddings, the architecture transcends the limitations of naively fusing RGB and depth features, mitigating the adverse effects of cross-modal misalignment. The introduction of the Guided Token Attention module demonstrates that explicit, multi-level semantic injection can significantly advance both the theoretical understanding and empirical performance envelope of GDSR tasks.

Practically, this approach informs the development of next-generation multimodal reconstruction systems for robotics, AR/VR, and autonomous systems, where compact, high-fidelity depth sensing is a key enabler. The design suggests strong avenues for extending semantics-aware cross-modal alignment to other multi-modal super-resolution and scene understanding applications.

Future Directions

Potential research extensions involve integrating task-specific semantic tokens (e.g., instance-level or panoptic guidance), adapting the framework for dynamic or video-based environments, and exploring self-adaptive semantic token selection. Additionally, replacing frozen pretrained transformers with architectures capable of on-the-fly semantic adaptation to new domains may further enhance cross-dataset generalization.

Conclusion

NAIMA presents a principled GDSR framework that injects semantic priors through pretrained transformer embeddings and explicit cross-attention, leading to consistent improvements over prior baselines. The design demonstrates that semantically aware depth reconstruction ameliorates the characteristic boundary blurring and artifact propagation afflicting RGB-guided methods, especially at high scaling factors. This approach underlines the necessity of leveraging pretrained semantic knowledge in multi-modal low-level vision tasks and provides a robust foundation for future development in semantics-enhanced image-to-depth super-resolution.