Automatic Registration of SHG and H&E Images with Feature-based Initial Alignment and Intensity-based Instance Optimization: Contribution to the COMULIS Challenge (2409.15931v1)

Abstract: The automatic registration of noninvasive second-harmonic generation microscopy to hematoxylin and eosin slides is a highly desired, yet still unsolved problem. The task is challenging because the second-harmonic images contain only partial information, in contrast to the stained H&E slides that provide more information about the tissue morphology. Moreover, both imaging methods have different intensity distributions. Therefore, the task can be formulated as a multi-modal registration problem with missing data. In this work, we propose a method based on automatic keypoint matching followed by deformable registration based on instance optimization. The method does not require any training and is evaluated using the dataset provided in the Learn2Reg challenge by the COMULIS organization. The method achieved relatively good generalizability resulting in 88% of success rate in the initial alignment and average target registration error equal to 2.48 on the external validation set. We openly release the source code and incorporate it in the DeeperHistReg image registration framework.

• The paper presents a registration method that integrates preprocessing, feature-based initial alignment using SuperPoint/SuperGlue, and intensity-based instance optimization.
• It achieves an 88% success rate in initial alignments by addressing intensity disparities between SHG and H&E images through targeted preprocessing techniques.
• The study finds that deformable registration offers minimal additional benefits for cases with limited nonrigid deformations, emphasizing the effectiveness of the initial alignment.

Automatic Registration of SHG and H&E Images

Introduction to the Problem

The automatic registration of noninvasive second-harmonic generation (SHG) microscopy images and hematoxylin and eosin (H&E) slides addresses a notable challenge in multimodal image registration due to disparate intensity distributions and partial data representation. SHG microscopy provides insights into collagen fibers without the need for staining, unlike H&E slides, which offer comprehensive tissue morphological information. The differing data representations necessitate solution strategies that address both data modality mismatches and substantial initial misalignments.

Figure 1: Exemplary pair of H&E and SHG images. Note the significantly different intensity distributions and the amount of missing data in the SHG image.

Methodology Overview

The registration system proposed in this paper consists of three main components: preprocessing, feature-based initial alignment, and deformable registration via instance optimization. The preprocessing step aims to reconcile geometric features from both modalities through intensity normalization and filtering. Feature-based alignment is executed using SuperPoint and SuperGlue, which allows for recovery of large deformations and bypasses traditional scale and rotation invariance constraints.

Figure 2: Visualization of the registration pipeline presenting intermediate results. Best viewed zoomed in for detail examination.

Preprocessing Techniques

Preprocessing reduces intensity distribution disparities and enhances the similarity of geometric features between the SHG and H&E images. The H&E images are transformed to the HSV color space, and only the hue channel is utilized, followed by intensity normalization and global histogram equalization. SHG images undergo histogram equalization as well and are filtered using a median filter. This preparation addresses impulse noise and emphasizes pertinent structural features necessary for reliable registration.

Figure 3: Visualization of the proposed preprocessing.

Initial Alignment Strategy

Initial alignment leverages a learning-based approach with SuperPoint keypoint extraction and SuperGlue matching. SuperPoint, paired with SuperGlue in an exhaustive search framework across several resolutions and initial rotations, demonstrates significant scalability and generalizability when compared to traditional methods like SIFT combined with RANSAC.

Deformable Registration

Deformable registration implements an iterative optimization using local mutual information as a similarity metric alongside diffusive regularization. This stage optionally enhances fine registration, depending on the nonrigid transformations observed across the dataset. However, the impact of the deformable registration is notably less pronounced, as evidenced through quantitative measurements such as target registration error (TRE).

Figure 4: The TRE calculated for the validation pairs using the Grand-Challenge platform.

Experimental Results

The methodology is assessed using TRE on a validation set, successful initial alignment rates on both training and validation data, and qualitative visual assessments. The proposed approach achieves an 88% success rate in initial alignments. Visual overlays of H&E and SHG images reveal minimal deformable registration impact, suggesting its optional necessity for datasets with limited nonrigid deformations.

Figure 5: Qualitative registration results using several samples from the validation subset.

Discussion on Limitations and Observations

The primary limitation relates to the lengthy computational time for initial alignment due to lack of inherent scale and rotation invariance in SuperPoint/SuperGlue, necessitating exhaustive searches. Additionally, discrepancies in imaging modalities could result in missing data, challenging the applicability of dense deformable registration strategies across all image regions.

Notably, SuperPoint combined with SuperGlue outperforms OmniGlue, contrary to claims about OmniGlue's generalizability in foundational model keypoint matching. The success highlights distinct multimodal image characteristics that influence keypoint matching performance.

Figure 6: Examples of several incorrectly registered pairs, illustrating challenges in aligning certain combinations of patches.

Conclusion

The paper introduces a robust multimodal registration approach tailored for SHG and H&E images, highlighting significant initial alignment efficacy and optional deformable registration. The research underscores the need to refine methodologies accommodating vastly different imaging modalities, with further exploration suggested in assessing the utility of whole slide images versus cropped patches to enhance registration reliability.

This work contributes to advancing multimodal image registration techniques, fostering methodologies aligning diverse imaging modalities with limited geographic correlations and inconsistent structural presentations.