Towards Patient-Specific Deformable Registration in Laparoscopic Surgery

Abstract: Unsafe surgical care is a critical health concern, often linked to limitations in surgeon experience, skills, and situational awareness. Integrating patient-specific 3D models into the surgical field can enhance visualization, provide real-time anatomical guidance, and reduce intraoperative complications. However, reliably registering these models in general surgery remains challenging due to mismatches between preoperative and intraoperative organ surfaces, such as deformations and noise. To overcome these challenges, we introduce the first patient-specific non-rigid point cloud registration method, which leverages a novel data generation strategy to optimize outcomes for individual patients. Our approach combines a Transformer encoder-decoder architecture with overlap estimation and a dedicated matching module to predict dense correspondences, followed by a physics-based algorithm for registration. Experimental results on both synthetic and real data demonstrate that our patient-specific method significantly outperforms traditional agnostic approaches, achieving 45% Matching Score with 92% Inlier Ratio on synthetic data, highlighting its potential to improve surgical care.

• The paper presents a patient-tailored registration pipeline that dynamically generates synthetic training pairs to model realistic non-rigid deformations in laparoscopic procedures.
• It integrates KPConv for keypoint extraction with a Transformer encoder-decoder to enhance overlap prediction and establish dense, high-confidence correspondences.
• Empirical evaluations on IRCAD-Liver1 and DePoLL datasets demonstrate superior matching scores and lower registration errors compared to conventional and deep-learning methods.

Patient-Specific Deformable Registration for Laparoscopic Surgery

Motivation and Scope

This work addresses the critical challenge of registering preoperative anatomical 3D models to intraoperative observations during laparoscopic surgery, where patient safety and surgical accuracy depend on effective guidance. The standard pipeline of overlaying preoperative organ models onto intraoperative views is impeded by non-rigid deformations, noise, partial field of view, and rapid intraoperative tissue changes. Conventional and state-of-the-art learning-based point cloud registration algorithms typically adopt agnostic training paradigms using large, generalized datasets, which can result in sub-optimal and unreliable performance given the stringent requirements and patient-specific anatomical variability inherent in the surgical domain.

Methodological Contributions

Patient-Specific Data Generation and Registration Pipeline

The paper introduces a non-rigid point cloud registration pipeline explicitly designed for patient-specific adaptation. The core methodology integrates several components:

• Dynamic On-the-Fly Data Generation: The pipeline generates synthetic training pairs for each patient from preoperative CT-derived meshes, with rigid and non-rigid deformations using ARAP. This includes realistic partial views via geometry-projected cropping and varying deformation control, ensuring that training data closely simulate intraoperative conditions.
• KPConv-Based Keypoint Feature Extraction: The architecture employs KPConv to extract semantically relevant keypoints and corresponding features from both preoperative (dense, complete) and intraoperative (sparse, partial, deformed) point clouds. The resulting downsampled sets are matched in density for feature learning.

  Figure 1: The architecture integrates KPConv for keypoint and feature extraction, transformer-based conditioning, and dense correspondence prediction tailored to overlap regions.

• Transformer Encoder-Decoder with Overlap Prediction: The model utilizes multi-head self- and cross-attention (Transformer) layers for conditioning keypoint features and directly modeling local-to-global geometric correspondences between clouds. The architecture introduces an overlap prediction head to focus correspondence estimation on the relevant overlapping anatomical region, a necessity for typical low-overlap (<20%) intraoperative scenarios.
• Point-to-Node Decoder and Matching: Point features and overlap probabilities are upsampled and decoded for dense matching. Mutual nearest-neighbor criteria are applied to establish high-confidence correspondences.
• Physics-Based Non-Rigid Registration: Correspondences initialize a physics model where the displacement field is solved as a linear elasticity energy minimization problem. The system employs a finite element biomechanical model parameterized by soft-tissue physics (e.g., Young's modulus, Poisson ratio), encoded via a stiffness matrix, and optimized with conjugate gradients.

Optimization and Losses

The training objective is a weighted sum of:

• Matching loss (focal loss supervising confidence matrix versus ground truth),
• Chamfer loss weighted by overlap scores, prioritizing geometrically plausible registrations over strict one-to-one matching,
• Overlap classification loss, regularizing the network's region-of-interest focus.

Patient-Specific Dataset Synthesis

For each test subject, realistic deformations (simulating pneumoperitoneum, localized lobe movement) and view-dependent croppings are generated dynamically. This continually refreshed training data ensures that the network does not overfit to a static dataset and that intraoperative variations are thoroughly covered within each training epoch.

Empirical Evaluation

Dense Correspondence and Matching

On the IRCAD-Liver1 dataset, the proposed patient-specific method demonstrates robust matching performance under extreme partiality and deformation, achieving a 45% Matching Score with a 92% Inlier Ratio on synthetic test instances. This significantly outperforms prior methods, including classical hand-crafted feature baselines (e.g., FPFH) and recent deep-learning algorithms (LePard and LiverMatch), both in terms of match accuracy and the absolute number of correspondences established.

Figure 2: Qualitative analysis shows that the proposed method achieves substantially more exact matches (green lines) and fewer incorrect correspondences (red lines) compared to competitors.

Non-Rigid Registration and Real-World Performance

The registration pipeline is evaluated using synthetic and real anatomical surface datasets with known ground-truth transformations, as well as the DePoLL in vivo porcine liver dataset. On IRCAD-Liver1, the approach achieves a Target Registration Error (TRE) of 4.82 ± 3.33 mm and a Fiducial Registration Error (FRE) of 1.68 ± 1.11 mm, substantially surpassing generic approaches (e.g., LiverMatch, which records TREs >17 mm). Experiments on DePoLL yield stable performance with a mean Hausdorff Distance of 8.45 ± 3.60 mm and FRE of 15.90 ± 6.41 mm.

Figure 3: Visualization of non-rigid registration on IRCAD-Liver1, including intermediate and final stages, with correspondence and grid representations of deformation.

Qualitative results also demonstrate accurate surface correspondences (Figure 4), with the model robust to typical intraoperative artifacts, missing regions, and large biomechanical deformations.

Figure 4: Example DePoLL cases show accurate pointwise matching and robust surface registration, with consistent anatomical alignment and limited outlier correspondences.

Implications and Future Directions

By constructing a patient-specific, dynamically generated partial-deformation training set and optimizing the registration network for each case individually, the paper advances surgical computer vision towards the realization of safe, highly accurate, real-time augmented reality guidance. The focus on robust overlap estimation and biomechanical regularization directly addresses the constraints and uncertainties unique to surgical environments. This methodology enables enhanced intraoperative anatomical context for surgeons, potentially reducing error, improving surgical workflow, and paving the way for more adaptive AR systems in the OR.

Theoretically, this pipeline bridges strong learning-based feature correspondence with physical priors, demonstrating that classical biomechanical modeling can be tightly coupled with deep learning in medically critical applications. The approach’s adaptability suggests further utility in cases with patient-to-patient anatomical variation, as well as the integration of topological change handling and lifelong intraoperative adaptation.

Future work is anticipated to address cases with topological alterations (e.g., resections, tissue excisions) and continuous, real-time registration during ongoing surgical procedures by incorporating temporal models and further physics-informed learning mechanisms.

Conclusion

This study presents a rigorously validated, patient-specific deep learning and physics-based pipeline for non-rigid registration of preoperative and intraoperative point clouds under extreme constraints of partiality, noise, and deformation in laparoscopic surgery. Strong numerical benchmarks underscore superiority over agnostic alternatives, with practical implications for safer and more reliable navigated interventions. The integration of tailor-made data generation, Transformer-based matching, and biomechanical constraints marks substantive progress for real-time, intraoperative image-guided surgical systems.