Zero-Shot Depth from Defocus

Abstract: Depth from Defocus (DfD) is the task of estimating a dense metric depth map from a focus stack. Unlike previous works overfitting to a certain dataset, this paper focuses on the challenging and practical setting of zero-shot generalization. We first propose a new real-world DfD benchmark ZEDD, which contains 8.3x more scenes and significantly higher quality images and ground-truth depth maps compared to previous benchmarks. We also design a novel network architecture named FOSSA. FOSSA is a Transformer-based architecture with novel designs tailored to the DfD task. The key contribution is a stack attention layer with a focus distance embedding, allowing efficient information exchange across the focus stack. Finally, we develop a new training data pipeline allowing us to utilize existing large-scale RGBD datasets to generate synthetic focus stacks. Experiment results on ZEDD and other benchmarks show a significant improvement over the baselines, reducing errors by up to 55.7%. The ZEDD benchmark is released at https://zedd.cs.princeton.edu. The code and checkpoints are released at https://github.com/princeton-vl/FOSSA.

• The paper introduces ZEDD, a high-fidelity real-world benchmark, and FOSSA, a ViT-based framework that utilizes defocus cues for zero-shot depth estimation.
• The paper leverages a stack attention module and focus distance embeddings to fuse multi-image information, achieving superior metric accuracy and robust cross-domain performance.
• The paper demonstrates significant quantitative improvements, reducing error metrics by over 50% compared to leading methods, setting a new standard for practical DfD applications.

Zero-Shot Depth from Defocus: Advancing Generalizable Metric Depth Estimation from Real Focus Stacks

Introduction and Motivation

Depth from Defocus (DfD) leverages a stack of RGB images captured at varying focus distances to estimate a dense, metrically accurate depth map. Unlike monocular depth estimation approaches, which suffer from scale ambiguity, DfD directly exploits the geometric cues offered by the defocus blur variations as the focal plane sweeps through the scene. However, despite significant historical progress on DfD, prior deep models lack generalization, commonly overfitting to narrow synthetic or domain-constrained datasets, limiting practical deployment.

This work introduces two primary contributions to resolve these challenges: (1) ZEDD—a high-fidelity, real-world benchmark dataset specifically designed for zero-shot DfD evaluation (Figure 1), and (2) FOSSA—a scalable, ViT-based architecture incorporating a stack attention mechanism with explicit focus distance parametrization for superior information fusion across focus stacks. The proposed approach demonstrates not only strong in-distribution performance but also robust zero-shot generalization across diverse indoor and outdoor domains (Figure 2).

Figure 2: Overview of the zero-shot DfD pipeline and the qualitative/quantitative impact of the proposed method on real-world and synthetic data.

ZEDD: Construction of a Real-World Defocus Benchmark

ZEDD (ZEro-shot Depth from Defocus) addresses critical limitations of previous DfD benchmarks by providing an order of magnitude more scene diversity, denser and higher resolution images, and metrically precise ground-truth depths (Figure 1). Each of the 100 scenes is captured at 9 focus distances and 6 aperture settings, utilizing a DSLR with remote-controlled lens and a high-performance Lidar sensor for ground-truth acquisition. Extensive calibration processes correct for lens breathing, and multi-frame accumulation with ICP-based registration ensures dense, artifact-free depth maps. Manual point cloud clean-up eliminates residual artifacts near occlusions and reflective surfaces (Figures 7, 8).

Figure 1: Representative samples from ZEDD, demonstrating geometric diversity and photometric quality.

Figure 3: Hardware configuration showing rigid camera–Lidar mounting for spatial calibration and accurate, consistent multi-modal data capture.

Figure 4: Manual clean-up to eliminate Lidar noise at occlusion boundaries and on reflective material.

ZEDD offers a wide depth range (0.3m–15m), enabling robust training and evaluation across scenarios (Figures 9, 10).

Figure 5: Distribution of scene types in ZEDD, emphasizing coverage of practical indoor and outdoor environments.

Figure 6: Depth histogram in ZEDD, illustrating challenging metric diversity.

FOSSA: A Transformer Architecture for DfD

FOSSA is constructed atop a Vision Transformer (ViT) backbone, and introduces a critical architectural element: the stack attention layer (Figure 7). This module facilitates efficient cross-image interaction, essential for extracting defocus signals across the stack, whilst remaining computationally tractable due to the typically small stack dimension.

Figure 7: FOSSA pipeline: focus stack feature extraction (blue) with stack attention and focus distance embedding, followed by global feature refinement blocks (gray).

Each focus image’s features are augmented by learned embeddings of their focus distance, promoting interpretability and robustness to photometric changes. After several focus stack extraction layers with shared weights, features are collapsed and fused for further global refinement via standard ViT blocks and a DPT decoder.

The training pipeline leverages existing RGBD datasets to synthesize focus stacks using physically-inspired, randomized PSF models (Figure 8), blur kernel variability, and diverse focus/aperture parameters.

Figure 8: Influence of PSF kernel shape on blur; variation across $p$ enables modeling the transition from diffraction-limited to geometric blur.

This domain randomization is critical in bridging the synthetic-to-real gap and achieving strong out-of-distribution generalization.

Evaluation and Strong Numerical Claims

FOSSA and ZEDD enable stringent zero-shot evaluations; all models are tested without additional fine-tuning on ZEDD and multiple unrelated datasets. The following quantitative findings are emphasized:

• On ZEDD, FOSSA reduces AbsRel by 55.7% relative to DepthPro, the strongest monocular baseline, and outperforms all DfD-specific competitors, none of which transfer effectively across domains.
• On Infinigen Defocus (synthetic benchmark), AbsRel is reduced by 51.7% compared to DepthPro, with qualitative fidelity on thin structures and geometric nuances (Figure 9).
• On iBims, DIODE, and HAMMER, FOSSA consistently achieves the highest $\delta_{1.25}$ and lowest AbsRel, outperforming both monocular and DfD alternatives.
• On DDFF (the only prior real DfD benchmark), after fine-tuning, FOSSA further decreases MSE by 40.4% compared to DualFocus, the prior state-of-the-art.

  Figure 10: Qualitative head-to-head on ZEDD, Infinigen, and DDFF, reflecting sharpness and metric accuracy.

These strong results are complemented by qualitative evidence of correct geometric reasoning and robustness to scale ambiguity (Figure 11).

Figure 11: Visual comparison of focus stacks and depth accuracy from ZEDD (left) and DDFF (right) showing the superiority of the new benchmark in blur realism and depth ground-truth.

Robustness analyses reveal insensitivity to aperture size, focus sampling distribution, and stack size down to two images (Figure 12), highlighting the practical flexibility of the approach.

Figure 12: FOSSA’s performance stability across different stack sizes, apertures, and focus distributions on ZEDD validation.

Ablations and Architectural Analysis

Ablation studies confirm key factors for generalization:

• Removal of stack attention or focus distance embedding drastically degrades performance, verifying their necessity.
• Delayed fusion (collapsing stack features at later model stages) increases accuracy but with computational tradeoff.
• Domain randomization on both blur kernels and focus distance distributions is essential for avoiding overfitting to specific defocus or focal arrangements; omitting either produces 21–40% drops in $\delta_{1.25}$.

Theoretical and Practical Implications

This work demonstrates that explicit modeling of defocus cues, in combination with scaled-up transformer architectures and domain-randomized synthetic training, is sufficient to overcome the traditional generalization barriers in DfD. By decoupling DfD from the idiosyncrasies of prior small-scale datasets, it opens up practical deployment scenarios: robust mobile depth from focus bracketing, precise depth for post-capture refocusing, 3D photography, and interactive graphics.

On the theoretical side, the deliberate fusion of geometry-based and learned priors through ViT, stack attention, and physical PSF models suggests a blueprint for further advances in hybrid, data-driven geometric perception.

Remaining challenges include adaptation to dynamic scenes—as current calibration assumes static environments—and further closing the sim-to-real domain gap in blur synthesis. Potential future directions include integrating flow-based alignment for dynamic focus stacks and employing generative models (e.g., neural rendering for physical-blur synthesis) to minimize bias between training and deployment domains.

Conclusion

Zero-Shot Depth from Defocus constitutes a significant advance for both DfD benchmarking and architectural modeling. ZEDD redefines the standard for real-world DfD evaluation and dataset fidelity. FOSSA’s ViT-based framework, stack attention, and focus embedding not only overcome the generalization limitations of prior DfD methods but also outperform leading monocular depth baselines in zero-shot settings. This establishes a robust, scalable foundation for future work at the intersection of physical optics and large-scale visual learning.