Diffusion Knows Transparency: Repurposing Video Diffusion for Transparent Object Depth and Normal Estimation

Abstract: Transparent objects remain notoriously hard for perception systems: refraction, reflection and transmission break the assumptions behind stereo, ToF and purely discriminative monocular depth, causing holes and temporally unstable estimates. Our key observation is that modern video diffusion models already synthesize convincing transparent phenomena, suggesting they have internalized the optical rules. We build TransPhy3D, a synthetic video corpus of transparent/reflective scenes: 11k sequences rendered with Blender/Cycles. Scenes are assembled from a curated bank of category-rich static assets and shape-rich procedural assets paired with glass/plastic/metal materials. We render RGB + depth + normals with physically based ray tracing and OptiX denoising. Starting from a large video diffusion model, we learn a video-to-video translator for depth (and normals) via lightweight LoRA adapters. During training we concatenate RGB and (noisy) depth latents in the DiT backbone and co-train on TransPhy3D and existing frame-wise synthetic datasets, yielding temporally consistent predictions for arbitrary-length input videos. The resulting model, DKT, achieves zero-shot SOTA on real and synthetic video benchmarks involving transparency: ClearPose, DREDS (CatKnown/CatNovel), and TransPhy3D-Test. It improves accuracy and temporal consistency over strong image/video baselines, and a normal variant sets the best video normal estimation results on ClearPose. A compact 1.3B version runs at ~0.17 s/frame. Integrated into a grasping stack, DKT's depth boosts success rates across translucent, reflective and diffuse surfaces, outperforming prior estimators. Together, these results support a broader claim: "Diffusion knows transparency." Generative video priors can be repurposed, efficiently and label-free, into robust, temporally coherent perception for challenging real-world manipulation.

• The paper leverages video diffusion models with LoRA adaptation to repurpose implicit optical priors for transparent object depth and normal estimation.
• It introduces TransPhy3D, a large-scale synthetic dataset generating over 1.3M frames for challenging transparent and reflective scenes using physics-based rendering.
• Empirical results show DKT achieves state-of-the-art temporal consistency and accuracy, significantly boosting robotic grasping success on transparent surfaces.

Diffusion Models for Transparent Object Depth and Normal Estimation: The DKT Paradigm

Problem Overview and Motivation

Depth and normal estimation for transparent or highly reflective objects remains an unsolved challenge in robotic perception and 3D computer vision. Standard approaches such as stereo matching, ToF sensors, and discriminative monocular networks systematically fail due to physical phenomena (refraction, reflection, transmission) that violate their underlying assumptions, leading to missing data and severe temporal instability. The paper "Diffusion Knows Transparency: Repurposing Video Diffusion for Transparent Object Depth and Normal Estimation" (2512.23705) departs from previous work by hypothesizing that modern video diffusion models (VDMs), trained on vast video data, already capture complex light transport associated with transparency and reflectance. Leveraging this implicit optical knowledge, the authors introduce DKT—a foundation model for temporally consistent depth and normal estimation in videos containing transparent objects, built through efficient adaptation of a generic VDM.

Dataset Construction: TransPhy3D

A critical contribution is the construction of TransPhy3D, the first large-scale, fully synthetic video dataset focused exclusively on transparent and reflective objects. It contains 11,000 scenes (totaling 1.32 million frames), generated by physically plausible rendering of category-rich static and procedurally generated parametric assets labeled via an aesthetic score for transparency or reflectivity. Asset placement leverages physics simulation (Blender), with camera trajectories sampled to ensure challenging, dynamic multiview coverage. Materials include diverse glasses, plastics, metals, and ceramics, rendered with cycles ray tracing and OptiX denoising for high fidelity RGB, depth, and normal sequences.

Figure 1: Rendering pipeline for generating synthetic video data with static and parametric 3D assets, each scene rendered from dynamic camera trajectories.

DKT Architecture and Training Paradigm

DKT reframes depth and normal estimation from a discriminative, regression-based task into a conditional video-to-video translation problem on top of a pretrained WAN diffusion backbone [wan2025]. A LoRA-based adaptation injects an additional RGB latent channel, enabling joint modeling of image and depth/normal signals. Training exploits both TransPhy3D sequences and frame-wise synthetic image datasets via a staged, mixed batching scheme for efficiency and generalization. All original VDM weights are frozen except for LoRA low-rank adapters in the DiT transformer blocks, preventing catastrophic forgetting and preserving strong video priors.

The model pipeline processes input video through a VAE encoder, combines RGB and noisy depth/normal latents, and predicts velocity fields via conditional diffusion (flow-matching loss). During inference, arbitrary video lengths are processed by windowed, overlapped sliding, fused in the latent domain for temporal consistency.

Figure 2: Overview of the DKT architecture, which adapts a pretrained video diffusion backbone via LoRA for RGB-conditioned depth and normal sequence generation.

Experimental Evaluation

Quantitative and Qualitative Benchmarks

DKT achieves zero-shot state-of-the-art (SOTA) results on multiple real and synthetic datasets comprising challenging transparent and reflective scenarios, including ClearPose [chen2022clearpose], DREDS-STD [dai2022domain], and a held-out TransPhy3D-Test. Performance is reported in strict alignment with global scaling, with key metrics: $\delta_{1.05}$, $\delta_{1.10}$, $\delta_{1.25}$, REL, and RMSE in centimeters. DKT surpasses strong image (Depth-Anything-v2 [yang2024depth]) and video (DepthCrafter [hu2025depthcrafter]) baselines, highlighting improvements in both per-frame accuracy and—critically—temporal consistency, especially for long sequences and complex transparent phenomena. Notably, DKT attains an 87.17% $\delta_{1.05}$ on TransPhy3D-Test (next best: 31.92%), and dominates ClearPose with substantial margins: 38.17% $\delta_{1.05}$ at 93.04% $\delta_{1.25}$.

Figure 3: In-the-wild qualitative DKT results—robust, temporally consistent depth maps from unconstrained videos featuring transparent objects.

Figure 4: Qualitative comparison on ClearPose with temporally aligned depth profiles, showing DKT’s stability and detail across challenging scenes.

For video normal estimation, DKT-Normal (the normal prediction variant) surpasses prior SOTA (NormalCrafter [bin2025normalcrafter], Marigold-E2E-FT) in both angular error and consistency, demonstrating the efficacy of the diffusion-driven, video-centric approach.

Figure 5: DKT-Normal yields temporally consistent and sharp normal estimates on ClearPose compared to state-of-the-art baseline methods.

Ablations and Efficiency

The adoption of LoRA for video model adaptation is validated: naive fine-tuning is both less effective and computationally more demanding. Scaling up model size offers further gains. Importantly, the compact 1.3B parameter DKT model operates at 0.17 sec/frame (832×480), with modest GPU memory requirements, outperforming prior baselines in speed while maintaining state-of-the-art accuracy.

Robotic Manipulation Deployment

DKT integration into a real-world robotic grasping pipeline marks a practical validation of temporal depth consistency for embodied visual tasks. Using a PiPER ARM and RealSense D435, DKT-informed grasp planning achieves significantly higher success rates for tasks spanning translucent, reflective, and diffuse surfaces (mean: 0.73 vs. 0.48 for DepthCrafter and 0.46 for DAv2).

Figure 6: Cobot Magic system used for DKT-powered robotic grasping employing PiPER Arm and Realsense sensors.

Figure 7: Experimental setup showing three canonical surface types (translucent, reflective, diffusive) evaluated for grasping.

Implications and Future Directions

This work empirically supports the strong claim: "Diffusion knows transparency." It demonstrates that general-purpose generative video models, pre-trained on internet-scale data, inherently encapsulate non-trivial physical priors (e.g., refraction/reflection) sufficient for robust, temporally coherent transparent-object perception through lightweight adaptation and synthetic data. Such an approach directly addresses dataset bias, generalization, and the temporal instability entrenched in previous discriminative frameworks. This paradigm holds potential for expansion to other challenging modalities (e.g., specular, translucent, or textureless surfaces) and may catalyze broader adoption of generative pretraining strategies in dense video perception across robotics and AR/VR domains.

Conclusion

DKT constitutes a significant advancement in video-based depth and normal estimation, particularly in the context of transparent and reflective object perception. Through innovative dataset generation, generative adaptation, and diffusion-based modeling, it achieves state-of-the-art generalization and practical impact in both academic and robotics settings. The paper substantiates the viability of leveraging generative video priors for robust, real-world perception without reliance on real-world labels—a blueprint for future development in foundation models for physically informed, dense video understanding.