Modeling Depth Ambiguity: A Mixture-Density Representation for Flying-Point-Free Depth Estimation

Abstract: Despite advances in depth estimation, flying points remain a persistent failure mode: near object boundaries, depth estimators often predict spurious 3D points in the empty space between foreground and background surfaces. We trace this artifact to a standard modeling choice: assigning each pixel a single depth hypothesis. At boundaries, a pixel can straddle a foreground and a background surface, so its true depth is ambiguous between the two. A model that predicts a single depth cannot keep both possibilities, so training instead pulls the prediction toward an intermediate depth that lies on neither surface. We address this with MDA, a mixture-density representation that lets the model predict multiple depth hypotheses and their associated probabilities for each pixel. Near boundaries, different hypotheses can align with different surfaces, and the decoded depth is selected from one of these hypotheses rather than placed in the empty space between them. Across different backbones, MDA substantially improves boundary reconstruction and largely removes flying-point artifacts even under severe input blur, while adding negligible runtime overhead. The same mixture-density framework naturally extends to transparent objects, where it predicts multiple depth layers at transparent pixels, and to sky regions, where a dedicated component separates the unbounded sky from finite-depth regions, producing flying-point-free skylines. Project Page: https://biansy000.github.io/mda-site/.

• The paper introduces a novel mixture-density architecture that assigns a K-component depth distribution to eliminate flying points at boundaries.
• It leverages probabilistic loss formulations to specialize mixture components for foreground, background, and transparent objects, reducing boundary errors.
• Empirical results on NRGBD and other datasets show significant improvements in boundary localization and robustness against input blur with minimal runtime overhead.

Mixture-Density Depth: A Probabilistic Approach to Resolving Depth Ambiguity and Eliminating Flying Points

Introduction and Context

Depth estimation from images is foundational for 3D perception, with impacts spanning AR/VR, robotics, autonomous driving, and graphics. Contemporary feed-forward monocular and multi-view depth estimators achieve high accuracy on smooth surfaces, yet a persistent artifact—the generation of "flying points" near depth discontinuities—compromises practical reliability. These flying points occur at object boundaries and regions where per-pixel evidence is fundamentally ambiguous, such as along occlusion boundaries, within transparent objects, or near sky regions. Recognizing the failure of existing unimodal representations to handle intrinsic scene ambiguity, "Modeling Depth Ambiguity: A Mixture-Density Representation for Flying-Point-Free Depth Estimation" (2606.02552) provides an explicit formulation and solution for ambiguous depth prediction.

Figure 1: Visual illustration of the flying-point artifact at occlusion boundaries—standard unimodal models predict depths that interpolate between foreground and background, yielding spurious 3D geometry.

Analysis of Flying Point Artifacts

The paper systematically traces flying points to depth estimators’ core design—the per-pixel unimodal (single-depth) prediction paradigm. In feature regions straddling foreground and background surfaces, these models are intrinsically forced to commit to a single depth hypothesis under $\ell_1$ or $\ell_2$ regression. This elicits averaged predictions that fall into the empty space between real surfaces, contaminating geometric boundaries. The situation worsens under low-contrast or blurred input, where the evidence for boundary localization further deteriorates, increasing ambiguity and thus flying point prevalence.

Figure 2: Schematic depiction of the depth ambiguity at pixel boundaries; boundary pixels capture image evidence for both foreground and background surfaces.

Prior attempts to address flying points—post-hoc diffusion refinement or mixture-of-experts ensembles—either incur severe runtime cost or fail to address the underlying representation mismatch. The former category (e.g., Pixel-Perfect-Depth, PPVD) is slow and vulnerable to input blur; the latter (e.g., MoE3D) lacks a principled multitarget loss formulation and merely redistributes boundary ambiguity.

Mixture-Density Representation: Probabilistic Formulation

The central contribution is the Mixture-Density Architecture (MDA), which endows each pixel with a $K$-component mixture distribution over candidate depth hypotheses and associated confidences. This design is motivated probabilistically: boundaries are naturally multimodal in depth, so the loss is derived as the negative log-likelihood of a mixture Laplacian or Gaussian density. Each head predicts $(d_k, b_k, \pi_k)$ for mixture component $k$, and the representation is fully compatible with standard pre-trained backbones (e.g., DA3 and VGGT).

Critically, this representation resolves ambiguity by letting individual mixture components explain distinct surfaces. In ambiguous pixels, the heads specialize—one captures the foreground, another the background, etc. At inference, the depth decoding rule simply selects the most likely component, returning a value locked to a true surface, never an interpolated "flying" point. This decoding gives negligible runtime overhead—$K$ density evaluations per pixel.

Empirical Evaluation

Boundary Localization and Flying Point Suppression

MDA is instantiated atop both DA3 and VGGT backbones. Quantitative and qualitative results across diverse datasets (NRGBD, 7Scenes, HiRoom, Sintel, etc.) show a consistent and substantial reduction in boundary error. For instance, on NRGBD, DA3+MDA achieves a boundary Acc of 25 mm (vs. 57 mm for baseline DA3), and similar margins hold in Chamfer Distance and accuracy at both frame and scene levels.

Figure 3: Qualitative comparison—DA3, VGGT, and PPD baselines produce extensive flying points at boundaries, while the MDA output confines predictions to proper surfaces, eliminating flying-point artifacts.

The model maintains standard depth estimation performance in non-boundary areas, with negligible overhead and speed almost identical to the backbone. Inference is $\sim$80× faster than diffusion-based refinement, enabling persistent deployment at real-time rates.

Robustness to Input Blur

Boundary prediction robustness persists under strong synthetic blur. As input downsampling scale increases, unimodal baselines generate progressively thicker bands of flying points; MDA preserves sharp boundary demarcations by maintaining multimodality unless the evidence for one hypothesis vanishes.

Figure 4: Under increasing input blur, only the mixture model maintains sharp foreground-background separation, while baselines accumulate smoothed flying bands.

Component Specialization

Per-component visualizations reveal spatial partitioning—different mixture heads dominate in distinct spatial regions, especially at occlusion boundaries, confirming the model's learned specializations.

Transparent Objects and Sky Regions

MDA naturally generalizes to other forms of depth ambiguity: (1) For transparent objects, the softmax over mixture logits is replaced by independent sigmoid weights, enabling simultaneous activation of multiple components. At inference, the sum of weights classifies transparency and yields multiple depth layers where needed—demonstrated on LayeredDepth, with improvements over recent baselines in AbsRel, $\delta_{<1.25}$, and multi-surface ordering metrics.

Figure 5: For a transparent object, MDA predicts both visible and occluded depths, as well as a segmentation distinguishing transparent from opaque.

(2) For sky regions, an extra fixed, large-mean component represents infinite depth. This permits threshold-free sky segmentation, eliminating flying points at skylines. Qualitative evidence shows that MDA produces clean sky boundaries, while baselines introduce extensive skyline artifacts.

Ablation and Analysis

Ablations establish that the boundary gains stem primarily from the probabilistic mixture representation, not merely from network capacity increases due to multiple heads. Increasing $K$ beyond 4 yields diminishing returns, confirming bimodal depth ambiguity as the dominant boundary pattern. Expectation decoding (using the mean of the mixture) fails—reintroducing flying points—further supporting the importance of selecting a surface-aligned mode.

Theoretical gradient analysis demonstrates that mixture posteriors gate per-component learning. Each component only receives gradients when it is responsible for the ground-truth label, so in ambiguous pixels, heads specializing to different surfaces stabilize, and "compromise" predictions are avoided. No multi-depth ground truth is needed: ordinary single-depth supervision suffices.

Implications and Prospects

Practically, this mixture-density representation is suited for deployment in any depth prediction pipeline encountering ambiguous pixels due to occlusions, transparency, or unbounded regions. The negligible overhead, modularity, and compatibility with pre-existing architectures (requiring only a final-layer modification) make it practical for widespread adoption in robotic perception, SLAM, scene reconstruction, and 3D content generation.

Theoretically, the approach offers a template for similar ambiguity-aware modeling in other dense prediction tasks (e.g., surface normals, optical flow, segmentation near class boundaries). Future directions include scaling to even higher ambiguity regions with adaptive $K$, extending to uncertainty quantification with richer posterior parameterizations, or integrating with temporal tracking for consistent hypothesis selection.

Conclusion

This work presents a rigorously principled and efficient mixture-density formulation for depth estimation that directly overcomes the flying-point artifact at occlusion and ambiguity boundaries. By providing per-pixel multimodality, the model significantly improves boundary localization, handles transparency and sky without ad hoc heuristics, and maintains state-of-the-art global depth accuracy at real-time speeds. The implications are substantial for both robust geometric perception and broader uncertainty-aware dense prediction in vision.