US-X Complete: Multi-Modal 3D Spine Recovery

• US-X Complete is a multi-modal deep learning framework that completes occluded vertebral anatomy by fusing ultrasound with a lateral X-ray projection.
• It employs a two-stage VAE architecture with early and late fusion modules, using paired synthetic CT-derived observations for robust training.
• Evaluations on synthetic and phantom data demonstrate significant improvements in vertebral shape accuracy, reducing reconstruction error metrics.

to=arxiv_search.^^^^^^^^3query3^^^^^^^^ 天天彩票中大奖_json code content='{"^^^^^^^^3query3 Complete\"^^^^^^^^3 OR ti:\3^^^^^^^^"US-X Complete: A Multi-Modal Approach to Anatomical 3D Shape Recovery\"","max_results"^^^^11sort_by10^^^^"sort_by":"relevance","sort_order":"descending"}'{"result":&&&^^^^3query3 Complete: A Multi-Modal Approach to Anatomical 3D Shape Recovery","authors":"Miruna Gafencu, Joerg Traub, Jan Wendler, Michael Schär, Rafael Palomar","categories":"eess.IV cs.CV","published":"^^^^3 OR ti:\3query3 OR ti:\3^^^^5-^^^^3all:\3all:\3^^^^- OR ti:\3query3 OR ti:\3^^^^5-^^^^3all:\3all:\3^^^^- to=arxiv_search.^^^^^^^^3query3^^^^^^^^ 天天中彩票在_json code content='{"^^^^^^^^3query3 AND vertebral completion ultrasound","max_results":^^^^3all:\3query3^^^^,"sort_by":"relevance","sort_order":"descending"}'{"result":[{"arxiv_id":"(Irany et al., 2024)","version":"v^^^^3all:\3 A Dataset for 3D Shape Completion in Intraoperative Spinal Ultrasound","authors":"Miruna Gafencu, Jan Wendler, Michael Schär, Rafael Palomar","categories":"cs.CV eess.IV","published":"^^^^3 OR ti:\3query3 OR ti:\3^^^^4-^^^^3query3^^^^7- OR ti:\3query3 OR ti:\3^^^^4-^^^^3query3^^^^7- to=arxiv_search.^^^^^^^^3query3^^^^^^^^ 夫妻性生活影片_json code content='{"^^^^^^^^3query3 completion ultrasound spine shape completion VRCNet X-ray arXiv","max_results":^^^^3all:\3query3^^^^,"sort_by":"relevance","sort_order":"descending"}'{"result":&&&^^^^3query3 Complete: A Multi-Modal Approach to Anatomical 3D Shape Recovery","authors":"Miruna Gafencu, Joerg Traub, Jan Wendler, Michael Schär, Rafael Palomar","categories":"eess.IV cs.CV","published":"^^^^3 OR ti:\3query3 OR ti:\3^^^^5-^^^^3all:\3all:\3^^^^- OR ti:\3query3 OR ti:\3^^^^5-^^^^3all:\3all:\3^^^^- A Dataset for 3D Shape Completion in Intraoperative Spinal Ultrasound","authors":"Miruna Gafencu, Jan Wendler, Michael Schär, Rafael Palomar","categories":"cs.CV eess.IV","published":"^^^^3 OR ti:\3query3 OR ti:\3^^^^4-^^^^3query3^^^^7- OR ti:\3query3 OR ti:\3^^^^4-^^^^3query3^^^^7- US-X Complete is a multi-modal deep learning method for completing occluded anatomical structures in 3D ultrasound by leveraging complementary information from a single X-ray image. In spinal imaging, it addresses the fundamental under-constraint of ultrasound-only vertebral reconstruction by fusing the local, real-time 3D information from a tracked ultrasound sweep with the global, projection-based ^^^^3 OR ti:\3^^^^D cues of a single lateral X-ray. The method is trained from paired synthetic observations derived from VerSe^^^^3 OR ti:\3query3 OR ti:\3query3^^^^ lumbar CT segmentations and is evaluated on both synthetic and phantom data. In phantom studies, it produces a more accurate, complete volumetric lumbar spine visualization overlayed on the ultrasound scan without the need for registration with preoperative modalities such as computed tomography (&&&^^^^3all:\3^^^^&&&).

^^^^3all:\3^^^^. Problem setting and imaging rationale

Ultrasound offers a radiation-free, cost-effective solution for real-time visualization of spinal landmarks, paraspinal soft tissues and neurovascular structures, making it valuable for intraoperative guidance during spinal procedures. Its main limitation in this setting is incomplete visualization of vertebral anatomy, in particular vertebral bodies, due to acoustic shadowing effects caused by bone. US-X Complete is designed to mitigate that limitation by integrating a single lateral X-ray projection while preserving ultrasound as the primary imaging modality (&&&^^^^3all:\3^^^^&&&).

The central problem formulation is shape completion under severe partial observability. The ultrasound input supplies tracked 3D information from a sweep, but that information is local and affected by shadowing and limited field-of-view. The lateral X-ray supplies a global but projection-based cue. US-X Complete combines these modalities so that vertebral reconstruction is no longer conditioned on ultrasound alone.

The method is defined within the fragment of 3D spinal shape recovery rather than general medical image reconstruction. Its target output is a complete vertebral surface and, in the reported phantom setting, a full lumbar surface that can be directly overlaid on the live ultrasound volume. The reported motivation is explicitly intraoperative guidance, level confirmation, and CT-free volumetric visualization.

^^^^3 OR ti:\3^^^^. Synthetic paired data generation

Training requires paired “partial” ultrasound and X-ray observations of the same vertebra, together with the complete ground-truth shape. All of these are derived from the VerSe^^^^3 OR ti:\3query3 OR ti:\3query3^^^^ lumbar CT segmentations (&&&^^^^3all:\3^^^^&&&).

For ultrasound-consistent partial point clouds, acoustic shadowing and limited field-of-view are simulated by ray-casting each CT mesh under multiple virtual probe poses. For each vertex PRESERVED_PLACEHOLDER_^^^^3query3^^^^ with surface normal PRESERVED_PLACEHOLDER_^^^^3all:\3^^^^ and each probe ray direction PRESERVED_PLACEHOLDER_^^^^3 OR ti:\3^^^^, the vertex is kept only if

$n_s \cdot (-d) > 0$

thus mimicking specular reflection. To model incoherent returns and spatial shadowing, small lateral and anteroposterior perturbations $\{d_k\}_{k=1\ldots K}$ of the probe are applied, and only points visible across all poses are retained:

$P_{US} = \bigcap_k \mathrm{Visibility}(M, d_k).$

Vertebral levels are then isolated by fixed-size bounding boxes around each centroid, yielding noisy per-vertebra point clouds that improve robustness to segmentation error.

For lateral X-ray projections, ^^^^3 OR ti:\3^^^^D lateral views are generated by orthographically projecting each vertebral point onto the sagittal plane along the left–right axis. If $(x,y,z)$ is a 3D point and $e_x$ is the left–right unit vector, then its X-ray projection $p$ is

$p = (I - e_x e_x^\top)(x,y,z)^\top.$

The missing depth coordinate of PRESERVED_PLACEHOLDER_^^^^3all:\3query3^^^^ is assigned to the mid-slice PRESERVED_PLACEHOLDER_^^^^3all:\3all:\3^^^^ of the CT volume along PRESERVED_PLACEHOLDER_^^^^3all:\3 OR ti:\3^^^^. Embedding PRESERVED_PLACEHOLDER_^^^^3all:\33^^^^ back into 3D yields PRESERVED_PLACEHOLDER_^^^^3all:\3max_results10^^^^ which is inherently aligned with PRESERVED_PLACEHOLDER_^^^^3all:\3^^^^5 in the CT coordinate frame.

This synthetic pipeline is not merely a convenience for supervision. It is the mechanism by which paired partial observations and complete targets are made available for end-to-end learning under controlled geometric alignment. A plausible implication is that the paired generation procedure is itself a core contribution, because it creates modality-consistent training tuples without requiring manual correspondence annotation.

3. Two-stage multi-modal completion architecture

The network is a two-stage completion architecture in which both stages are variational autoencoders, trained end-to-end with a combined KL divergence and Chamfer reconstruction loss. The architecture augments the standard single-modality VAE with two fusion modules—Early Fusion at the coarse stage and Late Fusion at the refinement stage—to integrate ultrasound and X-ray information (&&&^^^^3all:\3^^^^&&&).

In the coarse completion stage, an MLP encoder PRESERVED_PLACEHOLDER_^^^^3all:\3^^^^6 learns a prior over full vertebral shapes:

PRESERVED_PLACEHOLDER_^^^^3all:\3^^^^7

In parallel, two modality-specific encoders PRESERVED_PLACEHOLDER_^^^^3all:\3^^^^8 and PRESERVED_PLACEHOLDER_^^^^3all:\3^^^^9 process PRESERVED_PLACEHOLDER_^^^^3 OR ti:\3query3^^^^ and PRESERVED_PLACEHOLDER_^^^^3 OR ti:\3all:\3^^^^ into feature vectors PRESERVED_PLACEHOLDER_^^^^3 OR ti:\3 OR ti:\3^^^^ and PRESERVED_PLACEHOLDER_^^^^3 OR ti:\33^^^^. Early Fusion concatenates these features,

PRESERVED_PLACEHOLDER_^^^^3 OR ti:\3^^^^4

and projects them via another MLP into posterior parameters PRESERVED_PLACEHOLDER_^^^^3 OR ti:\3^^^^5 defining PRESERVED_PLACEHOLDER_^^^^3 OR ti:\3^^^^6. Alignment to the prior is enforced with the standard VAE KL loss:

PRESERVED_PLACEHOLDER_^^^^3 OR ti:\3^^^^7

Sampling PRESERVED_PLACEHOLDER_^^^^3 OR ti:\3^^^^8 produces a coarse point cloud PRESERVED_PLACEHOLDER_^^^^3 OR ti:\3^^^^9 via a shared decoder $n_s \cdot (-d) > 0$^^^^3query3^^^^. Reconstruction from full shapes also passes through $n_s \cdot (-d) > 0$^^^^3all:\3^^^^, enabling $n_s \cdot (-d) > 0$^^^^3 OR ti:\3^^^^ to learn a lumbar-vertebra shape prior.

The refinement stage is introduced because $n_s \cdot (-d) > 0$3 captures global geometry, whereas fine anatomical details such as pedicles and processes benefit from direct conditioning on the original observations. The refinement input is a single point cloud

$n_s \cdot (-d) > 0$4

of size $n_s \cdot (-d) > 0$5, with a one-hot modality tag $n_s \cdot (-d) > 0$6 so that each point $n_s \cdot (-d) > 0$7 is augmented to $n_s \cdot (-d) > 0$8. A point-based encoder–decoder with self-attention, adapted from VRCNet, then outputs the final completed surface $n_s \cdot (-d) > 0$9.

The functional distinction between the fusion modules is explicit. Early Fusion acts at the latent-feature level during coarse completion, whereas Late Fusion allows the network to learn differential weighting of prior, US, and X-ray cues at the local feature level during refinement.

4. Objective function and training protocol

The total training loss is

$\{d_k\}_{k=1\ldots K}$^^^^3query3^^^^

where $\{d_k\}_{k=1\ldots K}$^^^^3all:\3^^^^ is the Chamfer distance between the predicted $\{d_k\}_{k=1\ldots K}$^^^^3 OR ti:\3^^^^ and the ground-truth $\{d_k\}_{k=1\ldots K}$3:

$\{d_k\}_{k=1\ldots K}$4

In practice, $\{d_k\}_{k=1\ldots K}$5. No adversarial terms were required (&&&^^^^3all:\3^^^^&&&).

The reported training protocol is summarized below.

Item	Setting
Dataset	^^^^3all:\3^^^^49 synthetic lumbar spines (VerSe^^^^3 OR ti:\3query3^^^^), split 6^^^^3query3^^^^/^^^^3 OR ti:\3query3^^^^/^^^^3 OR ti:\3query3^^^^% into training/validation/test
Optimizer	Adam with learning rate $\{d_k\}_{k=1\ldots K}$6
Batch size	4 vertebrae per batch (training); ^^^^3 OR ti:\3^^^^ at test time
Epochs	^^^^3all:\3query3query3^^^^
Data augmentation	probe perturbations in US simulation serve as on-the-fly augmentation; no additional noise or scaling was applied
Hardware	NVIDIA GeForce RTX 4^^^^3query3descending3query3^^^^
Inference	average inference per vertebra $\{d_k\}_{k=1\ldots K}$7 s

These choices define the reported computational regime. The absence of adversarial terms and the use of Chamfer reconstruction plus KL regularization indicate that the completion problem is treated as VAE-based geometric inference rather than adversarial point-cloud synthesis.

5. Quantitative and qualitative evaluation

Completion quality is assessed on both synthetic (in-silico) and physical phantom data using three standard metrics: Chamfer Distance (CD), Earth-Mover’s Distance (EMD), and F^^^^3all:\3^^^^-Score at a fixed distance threshold $\{d_k\}_{k=1\ldots K}$8 mm. Lower is better for CD and EMD; higher is better for F^^^^3all:\3^^^^-Score. The evaluation also separates vertebral arch and body accuracy by splitting points around the center of gravity $\{d_k\}_{k=1\ldots K}$9:

$P_{US} = \bigcap_k \mathrm{Visibility}(M, d_k).$^^^^3query3^^^^

On synthetic test data, the US-only baseline [Gafencu et al. ^^^^3 OR ti:\3query3 OR ti:\3^^^^4] achieves $P_{US} = \bigcap_k \mathrm{Visibility}(M, d_k).$^^^^3all:\3^^^^ and $P_{US} = \bigcap_k \mathrm{Visibility}(M, d_k).$^^^^3 OR ti:\3^^^^, while the full US+X-ray model reaches $P_{US} = \bigcap_k \mathrm{Visibility}(M, d_k).$3 and $P_{US} = \bigcap_k \mathrm{Visibility}(M, d_k).$4, with $P_{US} = \bigcap_k \mathrm{Visibility}(M, d_k).$5 under Wilcoxon signed-rank. On two lumbar phantoms under clinical-like imaging, baseline $P_{US} = \bigcap_k \mathrm{Visibility}(M, d_k).$6 mm is reduced to $P_{US} = \bigcap_k \mathrm{Visibility}(M, d_k).$7 mm, again with $P_{US} = \bigcap_k \mathrm{Visibility}(M, d_k).$8. EMD and F^^^^3all:\3^^^^-Score improve consistently, and the reported interpretation is that a single lateral X-ray projection substantially constrains the ill-posed ultrasound gap (&&&^^^^3all:\3^^^^&&&).

In qualitative phantom experiments, tracked 3D ultrasound volumes and loop-X CBCT/X-ray pairs are acquired. Ultrasound and X-ray are co-registered via the known CBCT-US and CBCT-Xray transforms, and the X-ray’s lateral placement is refined heuristically by centering it on the bounding box midpoint of the US spine segmentation. Feeding these into US-X Complete yields a full lumbar surface that can be directly overlaid on the live ultrasound volume, with the completed body in blue and the original arch in orange. The lateral renderings show that the network corrects both scale and curvature of the vertebral bodies, producing an immediately interpretable volumetric spine guide in the ultrasound field of view.

The evaluation therefore has two distinct roles. Quantitatively, it reports statistically significant improvements on synthetic and phantom data. Qualitatively, it demonstrates a CT-free overlay workflow in which the completed anatomy is used as an intraoperative visualization aid.

6. Clinical relevance, limitations, and research position

By leveraging one quick, low-dose lateral radiograph—already common for level confirmation—US-X Complete overcomes ultrasound’s acoustic-shadow “blind spot” on vertebral bodies while preserving real-time, radiation-free soft-tissue visualization. The reported claim is that this fusion obviates CT-to-US registration pitfalls, including patient posture shifts and repeated intraoperative alignment, and dispenses with preoperative imaging altogether (&&&^^^^3all:\3^^^^&&&).

The reported limitations are equally explicit. Robust intraoperative ultrasound/X-ray registration is currently heuristic and must be made both automatic and resilient to limited US field-of-view. Completion is per-vertebra, so modeling intervertebral spatial constraints could further improve anatomical consistency across the full spine. Clinical patient data are expected to present more variability in tissue contrast and probe pressure than phantoms. These limitations identify the present method as an initial step to future clinical translation rather than a completed clinical deployment.

Within the literature, US-X Complete is presented as the first multi-modal deep network for 3D spinal shape recovery that integrates tracked ultrasound and a single lateral X-ray. A nearby antecedent is “SpinePartial: A Dataset for 3D Shape Completion in Intraoperative Spinal Ultrasound” (Irany et al., 2024). This suggests a progression from ultrasound-only vertebral completion toward multi-modal completion in which ultrasound remains the primary imaging modality and a single lateral X-ray is introduced to constrain the missing vertebral-body geometry.

Taken together, the method is best understood as a vertebral completion framework with three defining characteristics: paired synthetic supervision derived from CT segmentations, two-stage VAE-based fusion of tracked 3D ultrasound and a single lateral X-ray, and CT-free volumetric overlay in phantom imaging. Its significance lies not in replacing ultrasound, but in mitigating ultrasound’s principal anatomical blind spot while preserving the modality’s real-time intraoperative strengths.