SeeGroup: Multi-Layer Depth Estimation of Transparent Surfaces via Self-Determined Grouping

Abstract: Transparent objects are common in daily life, and it is important to understand their multilayer depth, including the transparent surface and the objects behind it. Existing methods for multilayer depth typically extend single-layer prediction. They define layers by the front-to-back ordering of 3D points and predict the layers sequentially. However, as layered geometry can admit multiple valid groupings of 3D points into layers, a predefined grouping strategy is inherently restrictive. In this work, we propose SeeGroup, a multi-layer depth estimation method that avoids imposing a predefined grouping and allows the model itself to adaptively assign surfaces to depth maps. We formulate per-pixel multi-layer depth as a point process, treating depth layers as unordered events along each camera ray. This induces a permutation-invariant likelihood over the observed depth layers, yielding a loss that naturally supports arbitrary layer groupings. Experiments demonstrate that our method significantly advances the state of the art of multi-layer depth estimation, improving quadruplet relative depth accuracy on LayeredDepth benchmark from 61.34% to 70.09%. Code is available at https://github.com/princeton-vl/SeeGroup.

• The paper introduces a recurrent decomposition module and permutation-invariant point-process objective that let the network learn coherent, scene-dependent depth groupings instead of using fixed depth ordering.
• SeeGroup achieves 70.09% quadruplet relative-depth accuracy on the LayeredDepth real-world benchmark, a 14.26% relative improvement over the strongest baseline, while using fewer parameters and lower inference cost.
• The method combines max-mixture Laplace intensity functions with coverage and gradient-matching losses, but still faces over-prediction, limited out-of-distribution generalization, and uncertainty about adaptive layer counts.

SeeGroup addresses multi-layer depth estimation for transparent surfaces by removing a structural constraint that previous methods imposed: a fixed, predefined grouping of per-pixel depth values into depth maps. Instead of forcing the model to assign, say, the $i$-th smallest depth at every pixel to the $i$-th map, SeeGroup lets the network itself determine how depths across pixels are grouped into coherent layers. The central technical device is a point-process formulation of per-pixel multi-layer depth that yields a permutation-invariant training objective. On the LayeredDepth real-world benchmark, this design improves quadruplet relative depth accuracy from 61.34% to 70.09%, a 14.26% relative gain, while using fewer parameters than the strongest baseline (2605.28735).

Motivation: grouping is scene-dependent

In images containing transparent objects, a single camera ray can intersect several media transitions—air to glass, air to water—each defining a distinct depth layer, so one pixel corresponds to multiple depths. Prior work either predicts only the first (front) surface or only the background, both of which are inadequate for applications that require perceiving the transparent surface and what lies behind it simultaneously, such as a robot retrieving an object from a plastic container or navigating around glass walls.

The multi-layer formulation of LayeredDepth predicts, per pixel, an ordered sequence of depths of variable length. Transferring single-layer estimators to this setting requires deciding how depth values across pixels are grouped into depth maps. The depth-ordered strategy used previously works for simple cases but produces incoherent maps for partially overlapping transparent planes: the first map would merge the entire front plane with part of the rear plane, creating abrupt geometric and semantic discontinuities. An object-centric grouping would be preferable there. Since the appropriate grouping appears highly scene-dependent and even region-dependent, SeeGroup's authors argue it should be learned rather than fixed, with the model free to emit depth values in any order and sorting applied only at evaluation time.

Method

SeeGroup has two main components: a recurrent decomposition module and a point-process intensity parameterization trained with permutation-invariant losses.

The recurrent decomposition module treats the backbone feature map as a mixture of contributions from multiple surfaces. A decomposer $D$ extracts a component $\mathbf{C}_i$ capturing the dominant group of depth layers in the residual feature map; a remapper $R$ projects the component back to feature space, which is subtracted from the residual (with a norm-based rescaling factor) so subsequent iterations focus on unexplained content. The grouping of surfaces into components is thus fully learned and order-irrelevant.

Each component is mapped by a predictor to the center $d_i$ and scale $b_i$ of a Laplace component, and the per-pixel intensity function is the max-mixture $\boldsymbol{\Lambda} = \max_i \mathbf{L}_i$. The authors argue the max-mixture is better suited than a weighted mixture because at each depth only the dominant component contributes, encouraging components to specialize in separate depth ranges and preventing collapse into a single broad mode. At inference, multi-layer depths are obtained by peak detection along the depth axis with a suppression threshold of 0.02.

The training objective treats per-pixel depths as events of a point process on the depth axis. The likelihood of observing ground-truth depths $\{d_1, \dots, d_m\}$ is proportional to $\prod_i \boldsymbol{\Lambda}(d_i)$, which is commutative and therefore permutation invariant—this is the property that enables self-determined grouping. Because this likelihood is one-sided (it does not penalize components unsupported by any ground truth), a complementary component-coverage loss $i$0 penalizes predicted components that match no ground-truth depth, yielding a bidirectional matching objective. A gradient-matching loss is added for fine detail. Training is scale-invariant, on the procedurally generated LayeredDepth-Syn dataset (14,800 images), with the feature extractor initialized from Depth Anything V2 metric-depth checkpoints (DINOv2-ViT-L) and $i$1 decomposition iterations.

Experimental results

On the LayeredDepth real-world benchmark (1,500 images, 14.2M relative depth tuples, evaluated zero-shot via tuple-wise accuracy), SeeGroup outperforms all four baselines on 14 of 15 metrics. The headline result:

Method	Quadruplet (All)	Triplet (All)	Pair (All)
Multi-head (DA v2)	61.34	70.57	82.56
Multi-head (NeWCRFs)	23.77	40.70	25.32
Index Concat (NeWCRFs)	23.77	40.70	26.90
Recurrent (NeWCRFs)	23.77	40.70	23.77
SeeGroup	70.09	74.88	82.62

The NeWCRFs-based baselines perform dramatically worse on quadruplets than the DA v2-based ones (23.77% vs 61.34%), indicating the pretrained backbone is a dominant factor; SeeGroup's gain is measured against the strongest baseline sharing its backbone. The improvement holds across Mixed tuples and same-layer subsets for layers 1, 3, and 5, with the sole exception of Layer 3 pairs, where Multi-head (DA v2) retains the lead.

The ablation studies support each design choice. Among architectures, the recurrent decomposition module achieves the best overall tuple-wise accuracy with the fewest parameters (356M total, 51M excluding the encoder, versus 428M/124M for Multi-head and 731M/427M for GRU) and the lowest CPU and GPU inference times; the Multi-head variant, lacking cross-layer information exchange, degrades catastrophically on deeper layers. Among intensity parameterizations, the max-mixture substantially outperforms weighted and sorted mixtures on all but the first layer. Among losses, the intensity-plus-coverage-plus-gradient-matching combination is best overall; notably, the permutation-invariant intensity loss with gradient matching (68.36% quadruplet on the validation split) far exceeds L1-based objectives (42.49–57.00%), which supports the claim that L1 regression in ambiguous transparent regions pushes models toward dataset priors rather than visual evidence.

On the LayeredDepth-Syn validation set—an in-domain evaluation the authors themselves flag as less informative due to limited diversity—SeeGroup is slightly worse than Multi-head (DA v2) on most metrics, though still clearly better than NeWCRFs-based variants. The authors attribute this to the model's smaller parameter count and to the noisy ray-tracer ground-truth boundaries in the synthetic data: because SeeGroup decomposes features into coherent depth-range components, it learns smooth background layers rather than fitting the boundary noise, which is a defensible behavior but means the synthetic metric comparison is not decisive.

Limitations and open questions

The paper concedes several limitations. Over-prediction is a common failure mode: the back side of a glass bottle may be incorrectly split into two depth layers, and the coverage loss only partially mitigates this. Performance also degrades on highly out-of-distribution examples, reflecting the limited availability of multi-layer training data—LayeredDepth-Syn is the sole training source, and LayeredDepth is the only real-world benchmark with multi-layer annotations. The synthetic evaluation's ground-truth noise and the in-domain setting leave open whether the slight synthetic regression versus Multi-head (DA v2) is an inherent trade-off of the decomposition design or an artifact of the data. The fixed decomposition count $i$2 is another open question: how the method should handle pixels with more layers than components, or adapt $i$3 to scene complexity, is not addressed.

Conclusion

SeeGroup reformulates multi-layer depth estimation as point-process learning over per-pixel depth events, combining a recurrent decomposition module with max-mixture Laplace intensity functions and bidirectional permutation-invariant losses. The formulation removes the predefined depth-ordered grouping constraint of prior work and delivers the strongest reported results on the LayeredDepth benchmark, with a 14.26% relative gain in quadruplet accuracy over the best baseline at lower parameter count and inference cost. The main open problems the paper leaves are over-prediction control, generalization beyond the limited synthetic training distribution, and adaptive determination of the number of depth components.