Intraoperative 2D/3D Image Registration via Differentiable X-ray Rendering (2312.06358v2)

Abstract: Surgical decisions are informed by aligning rapid portable 2D intraoperative images (e.g., X-rays) to a high-fidelity 3D preoperative reference scan (e.g., CT). 2D/3D image registration often fails in practice: conventional optimization methods are prohibitively slow and susceptible to local minima, while neural networks trained on small datasets fail on new patients or require impractical landmark supervision. We present DiffPose, a self-supervised approach that leverages patient-specific simulation and differentiable physics-based rendering to achieve accurate 2D/3D registration without relying on manually labeled data. Preoperatively, a CNN is trained to regress the pose of a randomly oriented synthetic X-ray rendered from the preoperative CT. The CNN then initializes rapid intraoperative test-time optimization that uses the differentiable X-ray renderer to refine the solution. Our work further proposes several geometrically principled methods for sampling camera poses from $\mathbf{SE}(3)$, for sparse differentiable rendering, and for driving registration in the tangent space $\mathfrak{se}(3)$ with geodesic and multiscale locality-sensitive losses. DiffPose achieves sub-millimeter accuracy across surgical datasets at intraoperative speeds, improving upon existing unsupervised methods by an order of magnitude and even outperforming supervised baselines. Our code is available at https://github.com/eigenvivek/DiffPose.

• The paper presents DiffPose, a novel self-supervised framework achieving sub-millimeter intraoperative 2D/3D image registration.
• It applies differentiable X-ray rendering combined with Lie algebraic transformations to enhance accuracy and operational speed.
• Evaluations on clinical datasets show DiffPose outperforming traditional methods, indicating strong potential for real-time surgical guidance.

Intraoperative 2D/3D Image Registration via Differentiable X-ray Rendering: An Overview

The paper presents DiffPose, a novel framework for intraoperative 2D/3D image registration utilizing differentiable X-ray rendering. DiffPose innovatively applies principles of self-supervision and physics-based rendering, achieving landmark-free, sub-millimeter registration accuracy through a process grounded in differentiating synthetic X-ray images. The incorporation of geometrical nuances and the use of Lie algebraic structures underpin the framework's design, aiming for rapid, precise, and clinically viable outcomes.

Summary of Approach

The authors address the challenges inherent in conventional 2D/3D registration methods by employing a self-supervised approach that leverages synthetic data from preoperative CT scans. Unlike traditional methods that rely on supervised learning with annotated landmarks or small datasets insufficient for generalization, DiffPose circumvents such limitations through a patient-specific Convolutional Neural Network (CNN) trained on unlimited synthetic X-rays. The CNN engages in self-supervised pose regression, initializing subsequent test-time optimization that refines the camera pose estimation with high precision.

Crucially, DiffPose operates in the Lie algebra $\se3$, improving pose estimation via the fundamental geometry of transformations. The multi-scale, local-normalized cross-correlation (mNCC) underpins the image similarity computation, stabilized utilizing a sparse rendering technique that is both computationally efficient and robust to local minima.

Key Findings

DiffPose's performance evaluation on datasets like DeepFluoro and Ljubljana demonstrates its efficacy across diverse clinical scenarios. The approach outperforms conventional and even some supervised techniques, achieving a sub-millimeter success rate substantially higher than alternatives, as evident in both success metrics and qualitative assessments.

The methodological application highlights include:

• Demonstrating an unsupervised method surpassing baseline supervised methods in registration accuracy while maintaining operational speed conducive to surgical procedures.
• Incorporating Lie theory to parameterize transformations within $\se3$, providing a sophisticated approach to camera pose estimation not seen in traditional Euler or quaternion-based methods.
• Employing a sparse mNCC, demonstrating computational efficiency without sacrificing accuracy.

Implications and Future Directions

The implications of DiffPose are far-reaching, particularly in enhancing surgical precision through real-time, accurate image registration, fostering advancements in augmented reality and robotic surgery systems. The robustness across patient-specific anatomical variability indicates potential scalability and adaptability to other clinical imaging modalities and applications.

Future developments could explore extending DiffPose to handle deformable registrations or adapting its lengthy pretraining requirement for emergency scenarios using rapid pre-training techniques. Integrating direct applications in piecewise rigid transformations stands as a prospective avenue.

Despite its advancements, the possibility of integrating DiffPose within broader clinical workflows necessitates careful consideration of real-time constraints and diverse surgical environments. Additionally, exploring transfer learning mechanisms in initializing pose regressors may lead the methodologies to broader applicability without per-patient training requirements.

In conclusion, the paper presents a methodically sound approach to intraoperative 2D/3D image registration, offering a significant leap in surgical imaging technology with DiffPose, moving toward a future where real-time guidance systems become a surgical staple.