Image To Tree with Recursive Prompting (2301.00447v1)

Abstract: Extracting complex structures from grid-based data is a common key step in automated medical image analysis. The conventional solution to recovering tree-structured geometries typically involves computing the minimal cost path through intermediate representations derived from segmentation masks. However, this methodology has significant limitations in the context of projective imaging of tree-structured 3D anatomical data such as coronary arteries, since there are often overlapping branches in the 2D projection. In this work, we propose a novel approach to predicting tree connectivity structure which reformulates the task as an optimization problem over individual steps of a recursive process. We design and train a two-stage model which leverages the UNet and Transformer architectures and introduces an image-based prompting technique. Our proposed method achieves compelling results on a pair of synthetic datasets, and outperforms a shortest-path baseline.

• The paper introduces a two-stage neural network (I2TRP) that reformulates tree extraction as an optimization problem for 2D angiography images.
• It leverages UNet for keypoint detection and recursive decoding with Vision Transformer and ResNet to address overlapping branches.
• Evaluated using Chamfer and Hausdorff distances, the approach significantly outperforms classic methods in predicting coronary tree structures.

Image To Tree with Recursive Prompting: An Expert Overview

The paper "Image To Tree with Recursive Prompting" by Batten et al. addresses the challenge of extracting tree-structured geometries from 2D medical images, specifically from coronary X-ray angiography. In projective imaging, issues arise due to overlapping branches in these anatomical structures, complicating the extraction process. The authors introduce a novel methodology that reformulates the extraction task as an optimization problem, implementing a two-stage neural network model named I2TRP that leverages both UNet and Transformer architectures with an innovative image-based prompting technique.

Methodology

The two-stage approach of the I2TRP model begins with keypoint detection using a UNet model. This model predicts topologically significant keypoints (root, bifurcation, leaf nodes) by generating Gaussian "blobs" around their locations in the input image. The keypoints are extracted using non-maximum suppression during inference.

The second stage focuses on recursive tree extraction. It formulates tree decoding as a series of recursive steps that the model processes by attending to candidate keypoints. The model employs supervised learning by decomposing the tree extraction problem, which enables training on deterministically sampled recursive steps without the need for complex end-to-end optimization. The architecture combines a Vision Transformer (ViT) with ResNet encoders for processing global and local image information, respectively. A Fourier feature-based positional encoding enhances the model's capability to pinpoint node locations.

Data and Evaluation

The experiments utilize two synthetic datasets: Volumetrically Rendered Meshes (VRM) from real 3D coronary artery data, and Simple Synthetic Angiography (SSA). These datasets provide a controlled environment for evaluating the efficacy of the I2TRP model. The evaluation employs Chamfer and Hausdorff distance metrics to compare predicted and ground-truth tree structures.

The results on both datasets indicate that the I2TRP model outperforms classic minimum cost path approaches, particularly in contexts with overlapping branches. On the VRM dataset, the model demonstrates significant improvement in tree structure prediction, exhibiting better quantitative and qualitative performance than baseline models.

Implications and Future Work

This research offers a promising methodology for medical image analysis, specifically in the task of extracting tree-like structures from projective images. The potential implications include a reduction in the complexity of extracting full curvilinear centerline trees, thus paving the way for more automated and accurate analysis of coronary angiography.

Future directions involve bridging the gap between synthetic and real-world data, possibly integrating diffusion models to enhance realism in synthetic datasets. Additionally, scaling this approach to 3D imaging modalities, such as CT angiography, could offer further insights and applications in medical diagnostics and treatment planning.

Overall, the proposed I2TRP model presents a significant advancement in the extraction of tree-structured data from images, reaffirming the importance of combining novel architectures and optimization strategies in tackling complex image analysis problems in healthcare.