DINOcular: Self-Supervised Visuospatial Representations

Abstract: We introduce a self-supervised framework for learning joint visuospatial representations from RGB-D observations. While modern vision foundation models are trained almost exclusively on RGB images, many embodied systems have access to explicit depth sensing, which provides geometric information that monocular inputs cannot recover. Our method integrates depth-derived geometric priors with a visual backbone through inter-patch and intra-patch fusion, enabling the model to encode both appearance and spatial structure efficiently. The resulting representation shows promising improvements on 3D awareness while preserving semantic transfer: it outperforms prior methods of comparable scale on multiple 3D geometry benchmarks, and remains competitive when probed for standard RGB-D semantic segmentation tasks.

• The paper introduces DINOcular, a self-supervised RGB-D model using depth at inter- and intra-patch levels to improve geometric and semantic representations.
• DINOcular outperforms RGB-only and supervised models in tasks like semantic segmentation (up to 7.48% improvement), 3D correspondence, and depth reconstruction.
• Depth dropout in DINOcular acts as a regularizer to prevent over-specialization to spatial tasks, maintaining both geometric and semantic information.

Problem formulation and contribution

“DINOcular: Self-Supervised Visuospatial Representations” (2608.27226) addresses a specific limitation of contemporary vision foundation models: their representations are predominantly learned from RGB images, despite the widespread availability of depth measurements in robotic, automotive, augmented-reality, and mobile systems. RGB-only representations can encode semantic regularities but are intrinsically limited by monocular scale ambiguity. Increasing the quantity of RGB data or exposing models to more spatially structured tasks does not eliminate this information deficit when depth is unavailable at the input.

The paper proposes DINOcular, a self-supervised RGB-D representation learner that incorporates depth at two complementary spatial scales. At the inter-patch level, depth is embedded into a three-dimensional rotary positional encoding, treating image coordinates and mean patch depth as a joint position. At the intra-patch level, a lightweight depth encoder extracts local geometric structure from the depth values within each patch and fuses it with the RGB patch embedding. The resulting architecture is trained with a combination of DINO-style image-level distillation, iBOT-style masked patch prediction, and a multi-view contrastive objective based on 3D correspondences.

The central empirical claim is that explicit depth input and multi-view supervision produce representations that are substantially more geometrically consistent than RGB-only self-supervised features, while retaining useful semantic transfer. This claim is evaluated through linear probing on semantic segmentation, 3D correspondence estimation, one-shot object pose estimation, and depth reconstruction.

Architectural design

DINOcular builds on a hierarchical Swin/RMT-style vision transformer and avoids the computational cost of a fully independent depth encoder. This design choice is important because the paper does not treat depth as a second appearance stream. Instead, depth is introduced as a geometric prior that alters positional structure and local token content.

The inter-patch mechanism replaces DFormerv2’s proximity-based attention bias with 3D RoPE. For a patch at image-plane coordinates $(u,v)$ with mean depth $\bar{z}$, the token is positionally transformed using a rotation parameterized by the three coordinates $(u,v,\bar{z})$. Consequently, attention can represent relative spatial relationships in the observed 3D configuration without enforcing the heuristic that metrically close patches must attend more strongly. The paper explicitly argues that this is more expressive than adding a depth-dependent scalar bias to attention logits: depth affects the coordinate system in which feature interactions are computed rather than merely reweighting an otherwise two-dimensional affinity.

The intra-patch mechanism addresses information discarded by patch pooling. Mean depth provides coarse spatial placement, but it cannot distinguish local surface structure such as discontinuities, curvature, or depth variation within a patch. DINOcular therefore applies a lightweight MLP-based depth embedding to per-pixel depth values inside each patch and fuses the resulting representation with the RGB embedding. An attempted alternative based on surface normals, motivated by their scale-free character, performs poorly and collapses during training. This negative result is significant: in this self-supervised setting, normalized local geometry is not automatically a better invariant than metric depth.

Figure 1: DINOcular architecture combining 3D RoPE for inter-patch geometry with a lightweight intra-patch depth encoder.

The model uses dropout on the intra-patch depth pathway. This modification responds to an observed failure mode in which the combination of local depth features and multi-view training over-specializes the representation toward spatial tasks and suppresses visual information. Depth dropout therefore acts as a modality-bottleneck regularizer: the representation must remain predictive under partial removal of local geometric evidence rather than solving the multi-view objective exclusively through depth.

Self-supervised learning objectives

The semantic component follows the DINO and iBOT paradigm. Global student and teacher views are aligned through the DINO image-level objective, while masked student patch tokens are matched to teacher predictions through iBOT. In the RGB-D setting, masking removes both RGB content and intra-patch depth content, but the masked tokens retain their position based on average patch depth. This distinction allows the model to use global geometry while requiring it to infer missing local appearance and geometry.

The spatial component uses paired views of the same object or scene. Known point maps and depth establish patches that correspond to nearby 3D points across views. The model then aligns their feature embeddings. Three alternatives are examined: cosine similarity, multi-view iBOT, and a contrastive ranking loss. The contrastive formulation is selected because it is the only alternative that improves correspondence accuracy when used in isolation and remains compatible with the semantic distillation losses.

The ablation exposes a nontrivial interaction between objectives. Training solely with multi-view losses sharply reduces semantic segmentation performance. For example, the baseline DINO-trained model reaches 40.16 mIoU on NYU Depth V2, whereas multi-view cosine and multi-view iBOT fine-tuning reduce performance to 11.10 and 30.30 mIoU, respectively. Multi-view contrastive training performs better on correspondence, reaching 19.99 compared with 17.90 for the baseline, but still reduces segmentation to 28.54 mIoU. Combining DINO with multi-view contrastive learning recovers part of the semantic performance while improving correspondence to 20.58.

This result establishes that geometric invariance is not a free by-product of semantic self-distillation. It must be explicitly optimized, but an isolated invariance objective can erase discriminative appearance information. DINOcular’s final loss therefore combines DINO, iBOT, multi-view contrastive learning, and KoLeo regularization. The paper’s broader representation-learning result depends on this joint optimization rather than on multi-view supervision alone.

Training data and implementation

DINOcular is trained on 1.3 million ImageNet-1k images and 307,000 selected MVImgNet2.0 samples, with approximately eight views per retained sample. Depth for ImageNet images and multi-view point maps for MVImgNet2.0 are generated with MapAnything. Object masks from SAM3 constrain the multi-view objective to relevant regions. The use of estimated rather than uniformly measured depth is deliberate: it makes the training recipe applicable to larger collections, although it introduces dependencies on the quality and calibration of the depth estimator.

The models are trained for 150 epochs using AdamW and BF16 mixed precision. The projection dimension is 65,536, and the teacher is updated by exponential moving average. The reported DINOcular-S and DINOcular-L models contain approximately 27 million and 94 million parameters, respectively. In the efficiency comparison, the 26.7-million-parameter DINOcular configuration has an average latency of 51.07 ms and peak memory usage of 249.12 MB. DFormerv2 has the same parameter count but is slower at 53.14 ms and uses 289.35 MB, whereas the RGB-only model is faster at 47.14 ms and uses 245.73 MB. Thus, depth integration incurs only a modest computational cost relative to RGB-only processing and is more efficient than the compared depth-attention design.

Ablation results

The architectural ablation strongly favors 3D RoPE over proximity attention. Under a DINO objective on ImageNet-1k, the 2D RoPE configuration obtains 33.50 mIoU on NYU Depth V2, 20.71 correspondence recall on NAVI under the $90^\circ$–$120^\circ$ viewpoint range, and 11.35 on ScanNet under the $60^\circ$–$180^\circ$ range. Replacing 2D RoPE with 3D RoPE raises these values to 39.92, 22.74, and 12.00, respectively.

The strongest ablation configuration combines 3D RoPE, an MLP-based intra-patch depth encoder, DINO, iBOT, multi-view contrastive learning, and depth dropout. It obtains 40.27 mIoU on NYU Depth V2, 25.25 NAVI recall at large viewpoint changes, and 16.52 ScanNet recall at severe viewpoint changes. Without depth dropout, the corresponding results are 38.58, 25.29, and 13.19. The small NAVI difference but larger ScanNet improvement indicates that regularization is particularly useful when scenes are cluttered and geometric shortcuts are more available.

The results also contradict a plausible expectation about surface normals. With 3D RoPE and surface-normal intra-patch features under the DINO objective, NYU performance falls to 38.38 mIoU, NAVI recall falls to 12.80, and ScanNet recall falls to 5.80. The paper therefore provides empirical evidence that scale-free local geometry is not sufficient for this representation-learning setup and may be actively destabilizing.

Semantic transfer

DINOcular’s semantic performance is evaluated by freezing the backbone and fitting linear probes. At comparable model and data scales, DINOcular-L achieves 40.23 mIoU on ADE20k, 47.46 on NYU Depth V2, 42.25 on SUN RGB-D, and 59.19 on Cityscapes. These results exceed DINO ViT-B trained on ImageNet-1k, which obtains 31.80, 34.49, 37.13, and 56.90, respectively. They also exceed DFormerv2-L on ADE20k, NYU Depth V2, and SUN RGB-D, although DFormerv2-L is slightly better on Cityscapes with 60.25 mIoU.

DINOcular-S reaches 33.88 mIoU on ADE20k, 40.27 on NYU Depth V2, 39.99 on SUN RGB-D, and 55.35 on Cityscapes. It surpasses the three-times-larger DINO ViT-B trained on comparable data on ADE20k, NYU Depth V2, and SUN RGB-D, though not on Cityscapes. The comparison should not be interpreted as evidence that RGB-D input universally dominates scale: DINOv2 and DINOv3 models trained on substantially larger datasets remain stronger on several semantic metrics. The relevant claim is narrower and better supported: at moderate and approximately comparable training scales, DINOcular provides a favorable semantic–geometric trade-off.

The multi-view objective also produces a measurable semantic cost. DINOcular-S without multi-view training reaches 36.02 on ADE20k, 40.45 on NYU Depth V2, 39.19 on SUN RGB-D, and 56.31 on Cityscapes, compared with 33.88, 40.27, 39.99, and 55.35 for the full model. The difference is modest but consistent on several datasets. This establishes that multi-view supervision improves geometric consistency at the expense of some purely semantic separability.

Figure 2: Qualitative semantic segmentation comparison showing more contiguous regions and sharper boundaries for DINOcular than for DFormerv2 and MultiMAE.

The qualitative segmentation results support the linear-probe measurements. DINOcular produces less fragmented masks on indoor scenes and maintains coherent parsing in Cityscapes scenes with illumination changes and shadows. These visual results are consistent with, but do not independently establish, robustness: the quantitative evaluation remains based on frozen linear probes and does not test robustness under controlled perturbation suites.

3D correspondence and pose estimation

The most direct evidence for the paper’s geometric claim comes from 3D correspondence estimation. On NAVI, DINOcular-L obtains 23.2% recall for viewpoint changes between $90^\circ$ and $120^\circ$, compared with 21.0% for DINO and 17.2% for DUNE. On ScanNet, it reaches 13.1% recall for changes between $60^\circ$ and $\bar{z}$0, exceeding DINOv3 ViT-B at 15.7% only in the table’s broader reference comparison? More precisely, DINOcular-L’s ScanNet result is 13.1%, while DINOv3 ViT-B reaches 15.7%; therefore the paper’s claim that DINOcular-L outperforms every reference model on this metric is not supported by the supplied numerical table. DINOcular-S, however, obtains 16.5% under the same ScanNet condition, exceeding DINOv3 ViT-B and all listed baselines.

At lower viewpoint changes, DINOcular is competitive rather than uniformly superior. DINOv2 ViT-B trained or evaluated with ImageNet-only representations achieves 93.1% on the easiest NAVI range, compared with 88.2% for DINOcular-L. The advantage of DINOcular emerges primarily under difficult viewpoint changes and in the smaller model’s ScanNet results. This pattern is important: depth and multi-view supervision appear to improve extrapolation across substantial viewpoint variation, not necessarily local matching under near-identical views.

Figure 3: Qualitative cross-view correspondence comparison, with DINOcular producing fewer outliers under large viewpoint changes.

DINOcular also performs strongly in one-shot, CAD-model-free pose estimation on low-texture objects. DINOcular-L reaches recalls of 10%, 46%, and 63% at the reported $\bar{z}$1 cm/$\bar{z}$2, $\bar{z}$3 cm/$\bar{z}$4, and $\bar{z}$5 cm/$\bar{z}$6 thresholds, respectively. These values exceed DINOv3 ViT-B at the two coarser thresholds, where it obtains 32% and 54%, and also exceed the listed baselines at the strictest threshold except for the smaller DINOv2 reference in some comparisons. The smaller DINOcular-S model is weaker at the strictest threshold, reaching 6%, which the paper associates with spatial blurring visible in the feature maps. The result suggests that the representation supports coarse pose localization particularly well, while fine pose precision remains sensitive to feature resolution and spatial sharpness.

Depth reconstruction provides an additional probe of whether depth is represented rather than merely consumed as an auxiliary signal. DINOcular-L achieves 0.26 m RMSE on NYU Depth V2, compared with 0.72 m for DFormerv2-L and 0.63 m for the ImageNet-only DINOv2 ViT-B representation. DINOcular-S obtains 0.24 m, while its no-multi-view counterpart obtains 0.26 m. These results indicate substantial linear accessibility of depth information, although they do not show that the learned features recover metric geometry without downstream supervision.

Figure 4: PCA components of DINOcular features exhibit stronger semantic-part separation and cross-view structural consistency than DINO and DUNE.

Depth-source robustness

The paper evaluates several depth sources at inference time. On ADE20k, replacing MapAnything depth with DepthAnything3 changes DINOcular’s mIoU from 33.36 to 32.68. On NYU Depth V2, MapAnything yields 40.27 mIoU, nearest-neighbor-densified sensor depth yields 39.01, and sensor depth densified with MapAnything yields 41.05. The relatively small variation supports robustness to heterogeneous depth quality and sparsity.

This conclusion should nevertheless be qualified. The study does not provide a controlled comparison across equivalent stereo, structured-light, and time-of-flight measurements. Nor does it systematically vary scale calibration, missing-depth patterns, depth bias, or adversarial geometric corruption. The reported robustness therefore concerns the tested depth-generation and densification pipelines rather than depth sensing in general.

Limitations and open questions

The principal limitation is the moderate training scale. DINOcular uses 1.3 million ImageNet images and a selected 307,000-sample subset of MVImgNet2.0, whereas DINOv2 and DINOv3 rely on much larger datasets whose exact training data and procedures are not fully comparable. The paper accordingly cannot establish whether DINOcular’s geometric advantage persists under equivalent large-scale RGB-only pretraining.

The use of estimated depth creates a second limitation. MapAnything supplies both training depth and point maps, so errors or biases in the estimator may be inherited by the representation and by the multi-view correspondence labels. The method’s dependence on object masks from SAM3 further constrains the training formulation toward object-centric or maskable regions. How performance changes with raw sensor depth, sparse LiDAR, temporally inconsistent depth, or scenes without reliable object masks remains open.

Finally, the semantic–spatial trade-off is unresolved. Removing multi-view supervision improves semantic probing in several settings, whereas retaining it improves large-viewpoint correspondence and pose estimation. The paper demonstrates this trade-off but does not derive a principled method for selecting the loss weighting as a function of downstream requirements. It also leaves open whether a larger model or more diverse data can eliminate the trade-off, or merely shift its operating point.

Conclusion

DINOcular presents a coherent RGB-D self-supervised learning framework in which depth contributes both positional geometry through 3D RoPE and local surface information through intra-patch fusion. Its multi-view contrastive objective produces representations with substantially improved geometric consistency under large viewpoint changes, while DINO and iBOT objectives preserve broad semantic transfer. At comparable moderate data scales, DINOcular improves over RGB-only and supervised RGB-D baselines on many segmentation, correspondence, pose, and depth-probing metrics, with limited computational overhead.

The strongest conclusion supported by the experiments is not that depth universally improves visual representations, but that explicit depth input can be integrated efficiently and usefully when paired with an objective that directly enforces cross-view geometric consistency. The remaining technical question is whether this advantage survives scale-matched pretraining with substantially larger and more heterogeneous RGB and RGB-D corpora.