Dual-Branch S-LRM for Semantic-Aware Reconstruction

• The paper introduces a dual-branch neural network that fuses sparse imaging data with high-quality priors to improve reconstruction accuracy and semantic consistency.
• It leverages dual encoder‐decoder architectures with an auxiliary segmentation head to reduce artifacts and minimize misregistration errors.
• Empirical results show significant gains in PSNR, SSIM, and RMSE while enhancing clinical detection of key interventional instruments.

A Dual-Branch Semantic-aware Large Reconstruction Model (Dual-Branch S-LRM) is a neural network framework that integrates parallel processing of both under-sampled (or noisy) imaging data and high-fidelity priors, with auxiliary semantic segmentation or detection heads. This architecture, exemplified in Dual-Branch Prior-SegNet for interventional cone-beam computed tomography (CBCT), simultaneously refines image reconstruction and encourages semantic consistency by guiding the network towards clinically critical features, such as interventional instruments. The model achieves enhanced accuracy and robustness, particularly in challenging scenarios with sparse or misaligned data, through architectural innovations, multi-task learning, and targeted data augmentations (Ernst et al., 2022).

1. Architectural Foundations

Dual-Branch S-LRM is based on a dual-encoder–decoder design, processing two complementary data streams: the primary input (e.g., sparse-view or degraded medical image) and a secondary prior (e.g., a high-quality planning scan or anatomical atlas). Each stream is encoded independently, and their feature representations are fused in the decoding stages to reconstruct the target output.

• Reconstruction Branch: In the context of CBCT, the branch receives a stack of 13 equiangular, sparse-view projections (pre-reconstructed via FDK or similar). The encoder consists of four downsampling blocks (each: two 3×3 convolution + ReLU, followed by 2×2 max-pooling), incrementally increasing feature channels from 32 to 256. The decoder applies four symmetric upsampling blocks (upsample, concatenate encoder features from both branches, two 3×3 conv + ReLU), culminating in a final 3×3 convolution with linear activation to predict the reconstructed volume.
• Prior Branch: The high-fidelity prior (e.g., a planning scan) undergoes an identical encoding process. At each decoding level, features from both branches are concatenated, promoting correction and enhancement guided by prior knowledge.
• Segmentation Head: An additional pathway operates on the final decoder feature map, applying a 3×3 convolution and sigmoid activation to predict per-voxel probability maps (e.g., for needle segmentation), thus embedding semantic awareness into the reconstruction.

The following pseudocode outlines the forward pass:

$g_r$6

2. Loss Formulation and Multi-Task Synergy

Dual-Branch S-LRM employs a composite loss designed to prioritize fidelity in reconstruction while simultaneously benefiting from semantic auxiliary tasks. For CBCT, the formulation comprises:

• Reconstruction Loss: Mean Squared Error (MSE) between prediction $p_r$ and ground truth $g_r$:

$$L_{\mathrm{recon}} = \frac{1}{N} \sum_{i=1}^N (p_r^{(i)} - g_r^{(i)})^2$$

• Segmentation Loss: Dice coefficient loss comparing predicted mask $p_s$ and binary ground truth $g_s$:

$$L_{\mathrm{seg}} = 1 - \frac{2\sum_i p_s^{(i)}g_s^{(i)} + \epsilon}{\sum_i p_s^{(i)} + \sum_i g_s^{(i)} + \epsilon}$$

• Combined Objective:

$$L_{\mathrm{total}} = L_{\mathrm{recon}} + \lambda L_{\mathrm{seg}}$$

with $\lambda = 10^{-3}$, allowing reconstruction to dominate while leveraging semantic guidance to reduce artifacts (notably streaking near interventional instruments).

This approach generalizes:

$$L_{\mathrm{total}} = L_{\mathrm{task1}(\mathrm{primary}\,\mathrm{data},\mathrm{prior})} + \sum_{k=2}^K \lambda_k L_{\mathrm{aux},k}$$

where each auxiliary loss $L_{\mathrm{aux},k}$ steers the model towards additional semantic or diagnostic goals.

3. Training Methodology and Robustness Strategies

Training a Dual-Branch S-LRM involves carefully curated datasets, label generation, and extensive augmentation to simulate realistic clinical scenarios:

• Dataset: For Dual-Branch Prior-SegNet, base volumes are derived from LungCT-Diagnosis, with in-house synthetic needle scans. Needle masks are obtained via thresholding at 900 HU.
• Projection Simulation: Sparse-view projections are synthesized (here, 13 projections along a circular trajectory), and all volumes are normalized by the 99th percentile of attenuation coefficients.
• Misalignment Augmentation: Priors are randomly rotated in-plane by angles $g_r$0 during training to simulate registration errors. Additional augmentations include random 3D rotations, scalings, and flips.
• Optimization: Adam optimizer, learning rate $g_r$1, batch size 32, for 150 epochs. Mixed-precision training enhances speed/memory efficiency.

These strategies systematically promote generalization, robustness to misaligned or imperfect priors, and adaptability to various domains or modalities.

4. Performance Evaluation and Empirical Outcomes

Quantitative and qualitative assessment of Dual-Branch S-LRM architectures demonstrates substantive improvements over prior methods. In Dual-Branch Prior-SegNet (Ernst et al., 2022):

Model	PSNR (dB)	SSIM (%)	RMSE (HU)
FDKConvNet	lower	lower	higher
Dual-Branch Prior-Net	41.09	96.71	28.7
Dual-Branch Prior-SegNet	43.97	97.15	23.2

• All improvements are statistically significant (Wilcoxon, $g_r$2).
• Robustness: Models trained with in-plane prior misalignment (up to $g_r$3) maintained PSNR stability up to $g_r$4 misalignment; models without such augmentations degraded beyond $g_r$5.
• Qualitatively, Prior-SegNet more sharply localizes needles with minimal streak artifacts, outperforming both FDKConvNet and Prior-Net.

5. Generalization of the Dual-Branch S-LRM Paradigm

The principles of Dual-Branch Prior-SegNet extend to a broad class of models (here termed "Dual-Branch S-LRM" Editor's term), applicable across imaging modalities and tasks:

• Dual Encoders: Integrate both observation and complementary priors/metadata (e.g., MR-PET fusion, historical scans, textual embeddings).
• Multi-Task Heads: Accommodate various semantic tasks—reconstruction, segmentation, detection, classification—adapting the model to application-specific requirements.
• Composite Losses: Harmonize main and auxiliary objectives for holistic task performance.
• Robustness Augmentations: Incorporate online transformations (rigid/non-rigid registration, intensity, noise) to simulate operational challenges.
• Scalability: Architectures may scale to multiple tasks, large fields-of-view, or adopt hybrid backbones (e.g., CNN-transformer hybrids) to extend representational capacity and context capture.

A plausible implication is that Dual-Branch S-LRM constitutes a modular and extensible blueprint for robust, high-fidelity, context-aware reconstruction in demanding clinical imaging settings, readily adaptable via plug-in priors and new semantic sub-tasks.

6. Contextual Significance and Research Impact

Dual-Branch S-LRM addresses enduring challenges in medical image reconstruction: undersampled acquisition, artifact suppression, instrument-localized streak reduction, and resilience to prior misregistration. By systematically leveraging semantic segmentation as an auxiliary supervision mechanism, this class of models demonstrably outperforms prior encoder–decoder approaches, yielding state-of-the-art quantitative performance and increased clinical relevance (Ernst et al., 2022). Its generalizability suggests wide applicability to multi-modal fusion, longitudinal monitoring, and other image-guided interventions, marking a substantive development in semantic-aware, prior-informed large-scale reconstruction methodologies.