SeeClear: Reliable Transparent Object Depth Estimation via Generative Opacification

Abstract: Monocular depth estimation remains challenging for transparent objects, where refraction and transmission are difficult to model and break the appearance assumptions used by depth networks. As a result, state-of-the-art estimators often produce unstable or incorrect depth predictions for transparent materials. We propose SeeClear, a novel framework that converts transparent objects into generative opaque images, enabling stable monocular depth estimation for transparent objects. Given an input image, we first localize transparent regions and transform their refractive appearance into geometrically consistent opaque shapes using a diffusion-based generative opacification module. The processed image is then fed into an off-the-shelf monocular depth estimator without retraining or architectural changes. To train the opacification model, we construct SeeClear-396k, a synthetic dataset containing 396k paired transparent-opaque renderings. Experiments on both synthetic and real-world datasets show that SeeClear significantly improves depth estimation for transparent objects. Project page: https://heyumeng.com/SeeClear-web/

• The paper introduces a two-stage pipeline that segments transparent objects, generates geometry-consistent opaque appearances with conditional diffusion, and feeds them to frozen depth estimators.
• SeeClear achieves strong zero-shot results, including 0.033 AbsRel and 21.45 mm RMSE on ClearGrasp Real with Depth Anything V3, while substantially improving weaker backbones such as MoGe-2.
• The method’s gains depend on precise masks, semantic CLIP conditioning, perceptual training, and learned alpha compositing, but synthetic-only training leaves robustness to new materials, scenes, and video as open challenges.

Motivation and problem statement

Monocular depth estimation degrades sharply on transparent objects because refraction and transmission violate the appearance–geometry correspondence that depth networks rely on. Existing remedies—fine-tuning depth backbones on transparent RGB–depth pairs such as ClearGrasp, TransCG, or ClearPose—require large-scale paired supervision and must be repeated whenever a new foundation backbone is released. SeeClear reframes the problem: rather than adapting the depth model to transparent appearance, it adapts the image to the depth model by converting transparent regions into geometry-consistent opaque appearances before feeding the result to an off-the-shelf estimator (2603.19547).

Method overview

The framework is a two-stage front-end. First, a coarse-to-fine segmentation pipeline (Trans4Trans providing prompts to SAM 3) localizes transparent objects, yielding a mask $M^{seg}$. Second, a conditional latent diffusion model—initialized from Paint-by-Example and fine-tuned on synthetic paired data—generates an opaque counterpart $I^{pred}$ conditioned on three inputs concatenated channel-wise: the noisy latent, the VAE-encoded transparent image $\mathcal{E}(I^{tr})$, and the downsampled mask. Semantic identity is preserved via a frozen CLIP ViT-L/14 class token projected through a lightweight transformer mapper into cross-attention space; the authors deliberately avoid DINOv2 tokens and patch-level conditioning because patch tokens encode background-contaminated local appearance.

Training combines the standard DDPM noise-prediction objective with a masked LPIPS perceptual loss applied only at low-noise timesteps ($t < 0.3T$), following the observation that perceptual supervision at high noise levels harms sample quality. A mask augmentation scheme—randomly adding or removing small geometric primitives near boundaries—trains the model to tolerate imperfect inference-time masks. Sampling uses UniPC with 10 steps.

A lightweight Mask Refinement Module (MRM), a four-layer fully convolutional head trained with BCE plus a mid-value penalty, predicts a soft blending mask that is binarized at inference for alpha compositing of the generated region with the original background. This addresses two failure modes: VAE-induced color drift outside object regions, and seam artifacts from hard pasting with inaccurate segmentation masks.

Dataset construction

SeeClear-396k contains 396,000 Blender-rendered images: 11 everyday transparent objects in 10 indoor scenes (110 object–scene units), each expanded over 10 viewpoints, two lighting setups, and six anisotropic deformation modes at five magnitudes. Each configuration yields paired transparent and opaque renderings sharing identical geometry, camera pose, and illumination—the opaque variant replaces the glass shader with a fixed-color Lambertian material—along with aligned depth, normals, and masks. Coverage is organized around Booster-style failure modes: non-Lambertian cue distortion, depth-definition ambiguity, and complex scene structure. The dataset is fully synthetic; this is both its strength (perfect alignment) and a potential generalization constraint discussed below.

Quantitative results

Evaluation covers real-world ClearGrasp (ToM/All/Other regions) and synthetic TransPhy3D, with affine-invariant alignment for relative-depth models; all SeeClear results are zero-shot with frozen Depth Anything V3 (DA3) or MoGe-2 backbones.

On ClearGrasp Real (ToM), SeeClear+DA3 achieves AbsRel 0.033, RMSE 21.45 mm, and MAE 17.94 mm, beating all baselines on those metrics—including DKT, which was fine-tuned on this dataset and only retains an advantage on $\delta_{1.025}$ (52.69 vs. 51.27) and $\delta_{1.05}$. The effect on weak backbones is striking: MoGe-2's ToM RMSE drops from 56.83 mm to 26.62 mm, and its $\delta_{1.025}$ rises from 5.63% to 42.75%. On the All and Other regions, SeeClear+MoGe-2 attains the best score on every metric (e.g., RMSE 17.87 mm and 17.12 mm respectively), indicating no degradation on non-transparent content. On TransPhy3D, SeeClear+MoGe-2 improves AbsRel over the raw backbone by 25% (0.016 → 0.012) and surpasses dataset-fine-tuned DKT on AbsRel while remaining competitive on RMSE. In the appendix, on unseen TDoF20, SeeClear+DA3 outperforms zero-shot DKT on seven of eight metrics (RMSE 42.00 vs. 51.39 mm).

A notable negative control: replacing generative opacification with solid-color inpainting (the Depth4ToM strategy) yields RMSE 33.09 mm—worse than the unmodified DA3 baseline (24.03 mm). This supports the paper's central claim that plausible generated shading cues, not merely mask-based occlusion of transparent pixels, drive the improvement.

Ablations

The ablation study isolates each design choice on ClearGrasp Real (ToM) with DA3:

• Localization granularity: pixel masks beat bounding boxes (22.96 mm) and point heatmaps (22.38 mm); coarse spatial guidance causes boundary over- or under-opacification.
• Conditioning encoder: CLIP class token (21.45 mm) outperforms CLIP full tokens, DINOv2 CLS, and DINOv2 full tokens; weaker conditioning produces translucent interiors.
• Compositing: removing MRM raises iRMSE; removing alpha blending entirely raises RMSE to 23.60 mm and drops $\delta_{1.025}$ to 45.45%, due to overlapping patches restoring transparent appearance.
• Auxiliary loss: LPIPS (21.45 mm) beats LDM-only (22.30 mm), gradient loss (22.21 mm)—which is confounded by caustics and specular highlights—and L1 (21.81 mm).

Limitations and open questions

The paper concedes several constraints implicitly through its evaluation setup. Training data are limited to 11 objects and 10 indoor scenes rendered with a single glass-like shader and a fixed diffuse color, so generalization to materials with substantially different transmissive behavior rests on the diversity of the sampled deformations and viewpoints rather than explicit material coverage. The pipeline depends on segmentation quality: although mask augmentation mitigates imperfect masks, the ablations show localization errors propagate directly into opacification artifacts. Per-instance patch processing also implies computational cost scaling with object count, which the paper does not quantify. Finally, the method is evaluated only on static images; temporal consistency for video, and extension to specular, translucent, and multi-layer non-Lambertian materials, remain explicitly open.

Conclusion

SeeClear demonstrates that transparent-object monocular depth estimation can be addressed as an input-side opacification problem, decoupled from the depth backbone. With frozen foundation estimators it matches or exceeds specialized methods that require dataset-specific fine-tuning, and it delivers large gains on backbones that otherwise fail severely on transparent surfaces. The main open questions concern robustness beyond the curated synthetic training distribution and extension to video and broader non-Lambertian material classes.