ColonCrafter: Diffusion Depth Estimation

• ColonCrafter is a diffusion-based depth estimation model that produces temporally consistent dense depth maps from monocular colonoscopy videos.
• It leverages synthetic colonoscopy sequences and LoRA fine-tuning to impart geometric priors and enhance stability across video frames.
• The method integrates a training-free style transfer to adapt clinical imagery to the synthetic domain, improving reconstruction and metric performance.

to=arxiv_search.search 天天中彩票不中返json {"^^^^^^^^3query3 OR abs:\3^^^^^^^^"ColonCrafter\"","max_results":5,"sort_by":"submittedDate","sort_order":"descending"} to=arxiv_search.search 鲁夜夜啪json {"^^^^^^^^3query3 depth estimation diffusion DepthCrafter endoscopy C3VD","max_results":^^^^3ti:\3query3^^^^,"sort_by":"relevance","sort_order":"descending"} to=arxiv_search.search ՞նչjson {"^^^^^^^^3query3 video depth diffusion monocular arXiv","max_results":5,"sort_by":"relevance","sort_order":"descending"} to=arxiv_search.search 大发快三和值json {"^^^^^^^^3query3 &&&^^^^3ti:\3^^^^&&&, &&&^^^^3 OR abs:\3^^^^&&&, Shi et al., 2023, Ju et al., 2024, Lena et al., 9 Feb 2026, Biffi et al., 2024)","max_results":^^^^3ti:\3query3^^^^,"sort_by":"relevance","sort_order":"descending"} ColonCrafter is a diffusion-based depth estimation model for monocular colonoscopy videos that generates temporally consistent dense depth maps from RGB frame sequences. It was introduced to address a central limitation of prior endoscopic depth estimators: per-frame predictions may be locally plausible yet vary arbitrarily over time, which yields incoherent 3D structure and amplifies downstream pose-estimation drift. The method learns geometric priors from synthetic colonoscopy fly-throughs rendered from CT colonography, adapts a general video depth model through LoRA fine-tuning, and optionally applies a training-free style transfer that shifts clinical appearance toward the synthetic training domain while preserving geometric structure (&&&^^^^3query3^^^^&&&).

^^^^3ti:\3^^^^. Problem setting in monocular colonoscopy

Three-dimensional scene understanding in colonoscopy is difficult because the mucosal surface has few distinctive textures, strong non-Lambertian specular highlights, rapidly changing illumination from a point light source, and continuously deforming geometry due to peristalsis and insufflation. These properties break assumptions behind classical SLAM and many depth-estimation pipelines, notably feature richness, static scene structure, and diffuse reflectance. Monocular depth estimation is additionally ill-posed because many pixels are nearly textureless, surfaces are glossy, and the field of view is narrow and often occluded.

A further constraint is the scarcity of ground-truth RGB-depth pairs in clinical colonoscopy. This limits conventional supervised learning and makes domain transfer from general-purpose depth models unreliable. ColonCrafter addresses the problem at the sequence level rather than the frame level: the target is not only dense depth per frame but a mutually consistent depth sequence suitable for video-based reconstruction.

This positioning matters because temporal consistency is not a cosmetic property in endoscopy. When depth fluctuates across consecutive frames, the resulting backprojections become geometrically inconsistent, and even moderate pose error can accumulate into substantial map distortion. ColonCrafter therefore treats coherent depth generation across time as a primary modeling objective rather than a post hoc regularization target (&&&^^^^3query3^^^^&&&).

^^^^3 OR abs:\3^^^^. Generative model and latent diffusion formulation

ColonCrafter is a conditional video diffusion model based on the EDM formulation. It adapts the DepthCrafter architecture with a spatio-temporal U-Net denoiser PRESERVED_PLACEHOLDER_^^^^3query3^^^^ operating in latent space, together with a frozen VAE used to encode and decode both RGB video and depth volumes. For an RGB sequence PRESERVED_PLACEHOLDER_^^^^3ti:\3^^^^ and a depth sequence PRESERVED_PLACEHOLDER_^^^^3 OR abs:\3^^^^, the latent representations are

$z_0^{(x)} = E(x), \qquad z_0^{(d)} = E(d).$

Forward diffusion is performed in the latent space of the depth volume:

$$z_t^{(d)} = z_0^{(d)} + \sigma_t \epsilon, \qquad \epsilon \sim \mathcal{N}(0, I).$$

Conditional denoising then estimates the clean latent given the noisy latent, the noise level, and the RGB video latent:

$$\hat{z}_0^{(d)} = D_\theta\!\left(z_t^{(d)}; \sigma_t; z_0^{(x)}\right).$$

The predicted pixel-space depth sequence is obtained by decoding:

$\hat{d} = \mathcal{D}(\hat{z}_0^{(d)}).$

The denoising objective is the EDM-style regression loss

$$L_{\text{depth}} = \lambda_t \cdot \left\| D_\theta\!\left(z_t^{(d)}; \sigma_t; z_0^{(x)}\right) - z_0^{(d)} \right\|_2^2.$$

No optical flow, camera pose, or additional modalities are used. No photometric, SSIM, temporal warping, smoothness, or ordinal losses are specified. Temporal consistency is enforced implicitly by the video diffusion prior: the denoiser reconstructs the full sequence jointly through shared spatio-temporal convolutions or attention rather than through explicit temporal penalties.

The model is initialized from a public DepthCrafter checkpoint and fine-tuned via LoRA with rank ^^^^3ti:\3^^^^6 applied to attention modules inside the U-Net denoiser, while the VAE encoder and decoder remain frozen. This design specializes a general video depth prior to colonoscopy without retraining the full backbone and without changing the inference modality: input remains RGB video, output remains per-pixel depth for each frame (&&&^^^^3query3^^^^&&&).

3. Synthetic supervision and structure-preserving style transfer

The training signal comes from synthetic colonoscopy sequences rendered from 5 CT colonography scans with colonic segmentation and centerline extraction performed in 3D Slicer. Virtual fly-throughs along the centerline were rendered with PyTorch3D or similar tools to produce RGB frames and ground-truth depth, yielding ^^^^3ti:\3query3^^^^9,3^^^^3 OR abs:\3^^^^9 synthetic colonoscopy images. The underlying rationale is that synthetic sequences provide dense geometric supervision that is effectively unavailable at comparable scale in clinical monocular colonoscopy.

To reduce the domain gap at inference time, ColonCrafter introduces a training-free real-to-synthetic style transfer using a pre-trained Stable Diffusion model. For a real clinical frame $x^{(c)}$ and a synthetic frame $x^{(s)}$, both are encoded into VAE latents, inverted to noisy latents over PRESERVED_PLACEHOLDER_^^^^3ti:\3query3^^^^ steps, and the content queries PRESERVED_PLACEHOLDER_^^^^3ti:\3ti:\3^^^^ together with the style keys and values PRESERVED_PLACEHOLDER_^^^^3ti:\3 OR abs:\3^^^^ are recorded. Reverse diffusion begins from an AdaIN-modulated latent PRESERVED_PLACEHOLDER_^^^^3ti:\33^^^^. During each reverse step and each self-attention layer, PRESERVED_PLACEHOLDER_^^^^3ti:\3^^^^4 are replaced by PRESERVED_PLACEHOLDER_^^^^3ti:\3^^^^5 while PRESERVED_PLACEHOLDER_^^^^3ti:\3^^^^6 is kept unchanged, so the geometry-bearing content stream is preserved while appearance shifts toward the synthetic domain.

Three additional adjustments are specified. First, specularity masking and inpainting mask pixels whose intensities exceed 3 standard deviations within a PRESERVED_PLACEHOLDER_^^^^3ti:\3^^^^7 patch and let the diffusion model inpaint those areas during reverse diffusion. Second, local histogram matching aligns the intensities of the adapted frame to the original clinical frame to stabilize brightness and contrast. Third, truncated inversion stops at PRESERVED_PLACEHOLDER_^^^^3ti:\3^^^^8 with PRESERVED_PLACEHOLDER_^^^^3ti:\3^^^^9 to reduce content drift. No adversarial or cycle-consistency losses are used, because the transfer is explicitly training-free. The intended outcome is appearance adaptation that preserves geometric cues critical for depth estimation (&&&^^^^3query3^^^^&&&).

4. Training protocol, inference pipeline, and benchmark performance

Training uses AdamW with learning rate PRESERVED_PLACEHOLDER_^^^^3 OR abs:\3query3^^^^, a cosine schedule with ^^^^3ti:\3^^^^,^^^^3query3query3query3^^^^ warmup steps, 5^^^^3query3^^^^,^^^^3query3query3query3^^^^ total steps, sequence length ^^^^3ti:\3^^^^6, and batch size 8. Data augmentation consists of random sequence sampling to vary inter-frame translations, random flipping of sequence sections to reflect non-straight trajectories, random jittering of camera intrinsics to simulate device variability, and random brightness attenuation to emulate endoscopic lighting changes. Hardware and training duration are not reported. No joint training on real data is performed; style transfer is applied at inference.

Inference proceeds at the sequence level. The video is windowed into short frame sequences such as PRESERVED_PLACEHOLDER_^^^^3 OR abs:\3ti:\3^^^^, optionally style-transferred, encoded by the frozen VAE, denoised in latent space by the spatio-temporal U-Net under EDM sampling, and decoded back to pixel-space depth. For metric evaluation, a single global scale and shift per video are estimated in the model’s native training domain:

PRESERVED_PLACEHOLDER_^^^^3 OR abs:\3 OR abs:\3^^^^

If needed, predictions are then converted to the depth domain for final scoring.

Evaluation is performed on ^^^^3ti:\3query3^^^^ sequences from C3VD, a phantom colon dataset with paired depth from ^^^^3 OR abs:\3^^^^D-3D registration. Metrics are PRESERVED_PLACEHOLDER_^^^^3 OR abs:\33^^^^, PRESERVED_PLACEHOLDER_^^^^3 OR abs:\3^^^^4, PRESERVED_PLACEHOLDER_^^^^3 OR abs:\3^^^^5, and PRESERVED_PLACEHOLDER_^^^^3 OR abs:\3^^^^6, with alignment done once per video rather than per frame. The paper compares ColonCrafter against both general-purpose and endoscopy-specific baselines. EndoOmni is noted as not strictly zero-shot because it was partially trained on C3VD.

Method	PRESERVED_PLACEHOLDER_^^^^3 OR abs:\3^^^^7	AbsRel / SqRel / RMSE
Depth-Anything V^^^^3ti:\3^^^^	^^^^3query3^^^^.55	^^^^3query3^^^^.^^^^3 OR abs:\3^^^^8 / 3.58 / ^^^^3ti:\3query3^^^^.^^^^3query3 mm
Depth-Anything V^^^^3 OR abs:\3^^^^	^^^^3query3^^^^.6^^^^3ti:\3	^^^^3query3^^^^.^^^^3 OR abs:\3^^^^4 / ^^^^3 OR abs:\3^^^^.34 / 8.^^^^3 OR abs:\3query3^^^^ mm
DepthCrafter	^^^^3query3^^^^.59	^^^^3query3^^^^.^^^^3 OR abs:\3 OR abs:\3^^^^ / ^^^^3 OR abs:\3^^^^.65 / ^^^^3ti:\3query3^^^^.5^^^^3ti:\3 mm
EndoDAC	^^^^3query3^^^^.5^^^^3query3	^^^^3query3^^^^.^^^^3 OR abs:\3^^^^7 / 4.45 / ^^^^3ti:\33^^^^.9^^^^3ti:\3 mm
EndoSfM-Learner	^^^^3query3^^^^.56	^^^^3query3^^^^.^^^^3 OR abs:\3^^^^4 / 3.49 / ^^^^3ti:\3 OR abs:\3^^^^.34 mm
EndoOmni	^^^^3query3^^^^.77	^^^^3query3^^^^.^^^^3ti:\3 / ^^^^3ti:\3^^^^.^^^^3ti:\3 / 6.9^^^^3ti:\3^^^^ mm
ColonCrafter	^^^^3query3^^^^.77	^^^^3query3^^^^.^^^^3ti:\3 / ^^^^3ti:\3^^^^.^^^^3ti:\3 / 6.4^^^^3 OR abs:\3^^^^ mm
ColonCrafter + style transfer	^^^^3query3^^^^.79	^^^^3query3^^^^.^^^^3ti:\3 / ^^^^3ti:\3^^^^.^^^^3query3 / 6.^^^^3 OR abs:\3ti:\3^^^^ mm

These results support two specific conclusions. First, ColonCrafter achieves state-of-the-art zero-shot performance among strictly zero-shot models on C3VD. Second, the training-free style transfer improves all reported metrics. The paper’s ablation study is limited to this style-transfer component; it does not report separate ablations on temporal window length or internal diffusion modules (&&&^^^^3query3^^^^&&&).

5. Reconstruction outputs and clinically relevant uses

ColonCrafter is not presented as a complete full-trajectory reconstruction system, but it does provide depth sequences that are sufficiently coherent for downstream 3D operations. A depth pixel at image coordinate PRESERVED_PLACEHOLDER_^^^^3 OR abs:\3^^^^8 with depth PRESERVED_PLACEHOLDER_^^^^3 OR abs:\3^^^^9 is backprojected using camera intrinsics $z_0^{(x)} = E(x), \qquad z_0^{(d)} = E(d).$^^^^3query3^^^^ as

$z_0^{(x)} = E(x), \qquad z_0^{(d)} = E(d).$^^^^3ti:\3^^^^

Using these per-frame backprojections, the paper demonstrates dense 3D point cloud generation. Camera motion is estimated through feature tracking with SpaTracker followed by bundle adjustment minimizing reprojection error across tracked points, after which coherent 3D reconstructions can be formed from the predicted depths.

The paper also formulates pose optimization as

$z_0^{(x)} = E(x), \qquad z_0^{(d)} = E(d).$^^^^3 OR abs:\3^^^^

where $z_0^{(x)} = E(x), \qquad z_0^{(d)} = E(d).$3 is backprojection under intrinsics $z_0^{(x)} = E(x), \qquad z_0^{(d)} = E(d).$4 and $z_0^{(x)} = E(x), \qquad z_0^{(d)} = E(d).$5 is the pose of frame $z_0^{(x)} = E(x), \qquad z_0^{(d)} = E(d).$6. This indicates that ColonCrafter is intended to interface with explicit geometric estimation rather than to replace it.

A second application is surface coverage assessment. The reconstructed colon is unrolled around its estimated centerline into a two-dimensional map parameterized by distance along the centerline and circumferential angle. Coverage is then defined as

$z_0^{(x)} = E(x), \qquad z_0^{(d)} = E(d).$7

After morphological cleaning to remove noise and artifacts, unseen areas appear as black regions. The paper reports that ColonCrafter’s coverage closely matches coverage computed from ground-truth depth, enabling quantification of missed surfaces. The key takeaways additionally list lesion registration among the clinically relevant outputs enabled by coherent depth (&&&^^^^3query3^^^^&&&).

6. Position within the colonoscopy research landscape

Within monocular colonoscopy depth estimation, ColonCrafter is defined by three linked choices: video diffusion for temporal coherence, synthetic colonoscopy supervision with known depth, and training-free structure-preserving style transfer for sim-to-real adaptation. This differentiates it from general-purpose monocular models such as Depth-Anything and DepthCrafter, and from endoscopy-specific self-supervised or SfM-based methods whose view-synthesis assumptions can fail under specularities and deformation. In the broader literature, adjacent systems target complementary parts of the same pipeline: ColNav addresses real-time coverage guidance from depth and pose (&&&^^^^3 OR abs:\3^^^^&&&), ColonNeRF targets long-sequence neural reconstruction (Shi et al., 2023), SegCol provides fold-edge and tool labels that can support depth discontinuity reasoning (Ju et al., 2024), RealSynCol supplies synthetic RGB-depth-flow-pose supervision for reconstruction research (Lena et al., 9 Feb 2026), and REAL-Colon provides full-procedure clinical videos for CADe and CADx benchmarking (Biffi et al., 2024).

Several misconceptions are usefully avoided. ColonCrafter is not a general colonoscopy foundation model, not a segmentation system, and not a complete SLAM stack. It should also not be conflated with virtual colonoscopy systems based on CT colonography, immersive viewers, or crowd-assisted video annotation workflows, which operate in a different imaging regime from monocular optical colonoscopy (&&&^^^^3ti:\3^^^^8&&&, &&&^^^^3ti:\3^^^^9&&&). Nor is it identical to auxiliary segmentation or organoid-analysis methods such as CRIS or D-CryptO, which address different tasks (&&&^^^^3ti:\3^^^^&&&, &&&^^^^3 OR abs:\3ti:\3^^^^&&&).

The paper’s stated limitations are specific. Evaluation is confined to phantom data from C3VD, so clinical validation on diverse patient videos remains pending. The method is optimized for shorter segments because of computational and memory constraints, and full-trajectory reconstruction remains challenging. Severe specular highlights, fast rotations or thrusting motions, large deformations, occlusions, and textureless regions can still degrade depth quality and temporal stability. Future directions named in the paper include semi-supervised joint training on real and synthetic sequences, improved scalability to full-length procedures, and tighter integration with reconstruction and pose constraints. Reproducibility is partial: the paper is open access under CC BY-NC 4.^^^^3query3^^^^, but code release and detailed reproduction instructions are not provided (&&&^^^^3query3^^^^&&&).