Structure-Aware Sparse-View X-ray 3D Reconstruction (2311.10959v3)

Abstract: X-ray, known for its ability to reveal internal structures of objects, is expected to provide richer information for 3D reconstruction than visible light. Yet, existing neural radiance fields (NeRF) algorithms overlook this important nature of X-ray, leading to their limitations in capturing structural contents of imaged objects. In this paper, we propose a framework, Structure-Aware X-ray Neural Radiodensity Fields (SAX-NeRF), for sparse-view X-ray 3D reconstruction. Firstly, we design a Line Segment-based Transformer (Lineformer) as the backbone of SAX-NeRF. Linefomer captures internal structures of objects in 3D space by modeling the dependencies within each line segment of an X-ray. Secondly, we present a Masked Local-Global (MLG) ray sampling strategy to extract contextual and geometric information in 2D projection. Plus, we collect a larger-scale dataset X3D covering wider X-ray applications. Experiments on X3D show that SAX-NeRF surpasses previous NeRF-based methods by 12.56 and 2.49 dB on novel view synthesis and CT reconstruction. Code, models, and data are released at https://github.com/caiyuanhao1998/SAX-NeRF

• The paper presents a novel SAX-NeRF framework that uses a Lineformer transformer to capture internal structural dependencies in X-ray imaging.
• The paper introduces a Masked Local-Global ray sampling strategy that selects informative rays to improve the efficiency of 3D reconstruction.
• The paper demonstrates significant improvements with a 12.56 dB boost in novel view synthesis and a 2.49 dB gain in CT reconstruction on the X3D dataset.

Structure-Aware Sparse-View X-ray 3D Reconstruction

This paper presents a framework named Structure-Aware X-ray Neural Radiodensity Fields (SAX-NeRF) for enhancing sparse-view X-ray 3D reconstruction through the innovative integration of neural radiodensity fields and transformer-based architectures. The research addresses the limitations of existing NeRF-based algorithms which often fail to capture the crucial structural details inherent in X-ray imaging.

Methodology

The SAX-NeRF framework introduces two key innovations: the Line Segment-based Transformer (Lineformer) and the Masked Local-Global (MLG) ray sampling strategy.

1. Line Segment-based Transformer (Lineformer): The Lineformer is designed to capture internal structural dependencies by modeling the interactions of points within each X-ray line segment. This approach diverges from traditional multilayer perceptrons (MLPs) used in prior NeRF methods that can inadequately model the 3D structural complexities of objects.
2. Masked Local-Global Ray Sampling Strategy: MLG improves the extraction of geometric and contextual information from 2D projections by utilizing a binary mask to identify and sample rays from informative foreground regions. This method enhances the efficiency and effectiveness of the data used for training by focusing computational resources on meaningful regions within the projections.

Experimental Evaluation

The framework was evaluated using the X3D dataset, which encompasses a broad range of applications from medicine to industry. SAX-NeRF demonstrated significant improvements over current state-of-the-art NeRF methods, with average enhancements of 12.56 dB in novel view synthesis and 2.49 dB in CT reconstruction.

Implications

The introduction of the Lineformer transformer harnesses the potential of self-attention mechanisms in X-ray imaging, representing a novel exploration in the field. The MLG strategy's emphasis on informative ray sampling could guide future methodologies in efficiently utilizing sparse data for 3D reconstruction.

Future Directions

The insights gained from SAX-NeRF's improvements in capturing structural nuances suggest several avenues for future work. Further exploration could involve refining transformer designs to achieve even greater efficiency and accuracy or extending the approach to other complex imaging modalities.

In summary, the paper provides a rigorous exploration into enhancing X-ray 3D reconstruction by leveraging the structural properties of X-rays in conjunction with advanced neural network architectures, paving the way for future advancements in the field.