DP-SfM: Dual-Pixel Structure-from-Motion without Scale Ambiguity

Abstract: Multi-view 3D reconstruction, namely, structure-from-motion followed by multi-view stereo, is a fundamental component of 3D computer vision. In general, multi-view 3D reconstruction suffers from an unknown scale ambiguity unless a reference object of known size is present in the scene. In this article, we show that multi-view images captured using a dual-pixel (DP) sensor can automatically resolve the scale ambiguity, without requiring a reference object or prior calibration. Specifically, the defocus blur observed in DP images provides sufficient information to determine the absolute scale when paired with depth maps (up to scale) recovered from multi-view 3D reconstruction. Based on this observation, we develop a simple yet effective linear method to estimate the absolute scale, followed by the intensity-based optimization stage that aligns the left and right DP images by shifting them back toward each other using cross-view blur kernels. Experiments demonstrate the effectiveness of the proposed approach across diverse scenes captured with different cameras and lenses. Code and data are available at https://github.com/lilika-makabe/dp-sfm-tpami.git

• The paper introduces a dual-pixel imaging method that resolves monocular scale ambiguity by integrating SfM with defocus blur cues.
• It employs joint estimation of per-shot focus and global scale using an overdetermined linear system solved via IRLS on patch-wise blur sizes.
• Experimental results demonstrate a mean scale error of 0.153, significantly outperforming monocular depth estimators, with further gains via multi-aperture capture.

DP-SfM: Dual-Pixel Structure-from-Motion without Scale Ambiguity

Problem Statement and Motivation

Structure-from-Motion (SfM) and Multi-View Stereo (MVS) pipelines are foundational in 3D vision but inherently suffer from scale ambiguity: reconstructions from monocular, calibrated cameras yield geometry estimates up to an unknown global scale. Common approaches resolve this ambiguity using reference objects, fiducials, or measurements from external sensors—each imposing practical deployment barriers. The paper "DP-SfM: Dual-Pixel Structure-from-Motion without Scale Ambiguity" (2605.01852) presents a method to resolve monocular scale ambiguity by leveraging the unique imaging characteristics of dual-pixel (DP) sensors, requiring only multi-view DP images and known intrinsics.

Dual-Pixel Imaging and Defocus Blur Cues

DP sensors, available on many recent DSLRs and smartphones, split each pixel into two photodiodes, producing left and right sub-aperture views per image. Disparity between these views arises due to defocus: objects displaced from the focal plane induce opposite-signed disparities, directly related to blur size. For any scene point, the blur size $b$ is an affine function of the inverse true depth $z$ and the (unknown) per-shot focus distance $g$:

$$b \approx \frac{l f}{1-f/g} \left( \frac{1}{g} - \frac{1}{z} \right)$$

where $f$ is focal length and $l$ is aperture. Notably, both the true global scale $s$ (since $z$ is only determined up to $s$ post-SfM) and $g$ are unknown, inducing a pair of ambiguities.

Figure 1: The method leverages interrelated scale and focus ambiguities by fusing DP disparity (defocus) with scale-ambiguous depths for direct scale estimation; visualization of opposite disparity directions for near/far points is shown below.

DP sensors capture these sub-aperture disparities, which can be robustly used for patch-wise blur size estimation without calibration or supervision, by cross-aligning the left/right DP images via re-blurring (Equation 1 in the paper).

Methodology: Joint Estimation of Scale and Focus Distance

The proposed scale recovery method utilizes the affine coupling between the inverse scale-ambiguous depths from SfM and the patch-wise DP blur sizes.

1. Data Acquisition: Multi-view DP images are captured with known intrinsics (focal, aperture, pixel pitch), but unknown scene scale and per-shot focus.
2. Preprocessing:
   • SfM (and MVS) pipeline yields up-to-scale dense depth maps.
   • For each DP image, reliable blur size per patch is estimated using the method from [DefocusDisparity].
3. Formulation: For each patch, the relationship between estimated blur $z$0, scale-ambiguous depth $z$1, and unknowns is linear in the inverse focus and inverse scale. A linear regression system spanning all views/patches is assembled:

$z$2

where $z$3, $z$4, and $z$5 encodes per-patch terms.

1. Initial Solution: An overdetermined linear system, covering all selected views and high-confidence patches, is solved via Iteratively Reweighted Least Squares (IRLS) for robustness.
2. View/Patch Selection: Outlier views/patches—e.g., with insufficient defocus variation or unreliable estimates—are pruned.
3. Intensity-Based Refinement (Discrete Search): The initial scale is further refined by optimizing a cross-view DP patch alignment objective that explicitly accounts for the thin-lens PSF and image formation model:

$z$6

This discrete search, centered around the initial estimate, yields a refined absolute scale.

Figure 2: Schematic of the approach: DP disparity provides defocus-based scale cues; a linear system estimates per-shot focus and global scale by leveraging the affine depth-defocus relationship.

Empirical Evaluation

The method is rigorously evaluated using two types of cameras: a Canon EOS 5D Mark IV DSLR (varied lenses/apertures) and a Google Pixel 4, each capturing multi-view DP images across six scenes with ground-truth scales for validation.

Quantitative Analysis:

• Error metric: Average absolute error of scale ratio

$z$7

with $z$8 (scene-wise).

• Across all scenes/lenses/apertures, the mean scale error $z$9 is 0.153 (after optimization), improving sharply over the initial estimate.

Comparative Analysis:

• State-of-the-art monocular metric depth estimators (UniDepth v2, ZoeDepth, DepthAnything v2) applied to the same data yield mean errors of 0.416–0.565, substantially worse than the DP-based method.
• The DP-based approach better generalizes across scenes and camera settings, with error rates ~3x lower compared to monocular predictors.

  Figure 3: Qualitative results on real DSLR and smartphone data; ground-truth and estimated baseline lengths demonstrate accurate scale recovery through DP-SfM, unattainable by classical SfM pipelines.

Ablation: Multi-Aperture Capture:

• Using multiple apertures per scene (constant focal length), scale estimation accuracy increases. Multi-aperture data reduces error to as low as 0.098, highlighting the utility of intra-scene focus diversity and aperture variation.

  Figure 4: Visualization of estimated blur sizes across patches validates the quality of the crucial DP-based blur estimation inputs.

Observations, Limitations, and Failure Modes

Strengths:

• The DP-SfM method reliably recovers absolute scene scale across diverse hardware and capture settings.
• No priors, training data, or reference objects are required—just intrinsic calibration and DP images.
• Outperforms monocular and learned approaches by a large margin, particularly when multiple aperture settings are available.

Limitations:

• Effectiveness drops with pan-focus (large depth of field, small aperture) scenes, especially from smartphone cameras with limited blur; informative defocus cues are diminished.
• Failure cases arise from erroneous or unstable blur size estimation (e.g., textureless/repetitive areas), or when SfM itself fails (insufficient geometry/texture).
• Robust patch/view selection and multi-aperture image acquisition mitigate, but cannot eliminate, some error sources.

  Figure 5: DP artifact example—spatially varying, channel-dependent issues in certain lenses motivate patchwise normalization and careful artifact-aware processing.

Practical Implications:

• The algorithm enables single-camera, calibration-free scale estimation for general 3D capture—the missing component for geometry measurements in uncontrolled scenes without auxiliary scale cues.
• The approach is immediately applicable to consumer devices with DP sensors, enabling metrology, AR, and robotics applications unfeasible with classical SfM pipelines.

Theoretical and Future Directions

DP-SfM demonstrates that sensor-level imaging cues (defocus-induced disparity) provide sufficient scale information to resolve the foundational ambiguity in monocular 3D vision, contingent on accurate modeling and robust data selection. The method also underscores the value of fusing physical optics models with geometric vision algorithms.

Future directions should address:

• Improved blur size estimation under challenging texture, lighting, or noise conditions.
• Advanced, possibly learning-based patch selection or artifact correction exploiting DP statistics.
• Extension to dynamic or non-Lambertian scenes, and tuning for further robustness in challenging real-world conditions.
• Potential cross-fertilization with learned depth priors when DP cues are marginal.

  Figure 6: Comparison of recovered scales—DP-SfM (blue), DepthAnything v2 (red), ZoeDepth (orange)—with ground-truth; DP-SfM delivers accurate, consistent scale, while learned monocular estimators show substantial drift.

Conclusion

DP-SfM presents a principled, sensor-level solution for direct, automatic recovery of absolute scale in SfM/MVS pipelines via dual-pixel imaging. It shows robust numerical performance on diverse datasets, surpassing learned monocular estimators and prior calibration-dependent methods. The theoretical framework unifies geometric structure-from-motion and physical imaging models, yielding an elegant, practical method instantly applicable to many real-world vision problems.