Cross3DReg: Differentiable 2D/3D Registration

• Cross3DReg is a fully differentiable, correlation-driven framework that aligns CT-derived digitally reconstructed radiographs with live X-ray images for precise rigid 2D/3D registration.
• It employs a dual-branch CNN-transformer architecture with global-local feature decomposition to enhance interpretability and preserve high-frequency details amid modality differences.
• The method uses a novel correlation-driven loss and convex geodesic similarity shaping to improve convergence and initialization compared to previous registration approaches.

Cross3DReg is a fully differentiable, correlation-driven method for rigid 2D/3D registration in X-ray-to-CT fusion, designed for fluoroscopic guidance in interventions such as spine surgery. Its objective is to estimate the 6-DoF pose $\theta \in SE(3)$ that aligns a digitally reconstructed radiograph (DRR) from a CT volume with a live X-ray image. In the medical-imaging literature, the name commonly refers to the 2024 framework introduced in “Fully Differentiable Correlation-driven 2D/3D Registration for X-ray to CT Image Fusion” (Chen et al., 2024). The same name was later reused for a distinct cross-source point cloud registration benchmark and framework, so the problem domain must be inferred from context (Xu et al., 8 Sep 2025).

1. Problem formulation and motivation

Cross3DReg addresses the standard rigid 2D/3D registration objective

$$\mathcal{F}(\theta)=\arg\min_{\theta}\mathcal{S}(I_x, I_m) =\arg\min_{\theta}\mathcal{S}(I_x,\mathcal{P}(\theta;V)),$$

where $I_x$ is the fixed intraoperative X-ray, $V$ is the CT volume, $I_m$ is the DRR generated by a projection operator $\mathcal{P}(\theta;V)$, and $\mathcal{S}$ is a similarity measure (Chen et al., 2024).

The method is positioned against two established classes of approaches. Classical optimization-based pipelines such as CMA-ES-based registration can be robust, but they are not naturally differentiable and can get stuck when the initial pose is far from the truth. Prior learning-based differentiable methods, such as ProST and SOPI, improve capture range by learning a latent similarity, but the paper argues that they still lack control and interpretability: CNN feature extraction is hard to inspect, the gradient flow is not easily shaped, and standard CNNs tend to over-focus on local patterns while missing globally consistent structure needed for precise registration (Chen et al., 2024).

Cross3DReg is therefore framed not only as an end-to-end learnable registration pipeline, but also as an attempt to make feature extraction and gradient flow more controllable and interpretable than prior fully differentiable deep registration methods. A central premise is that registration should emphasize high-frequency, detail-rich, correspondence-relevant structures while suppressing low-frequency nuisance variation caused by modality differences between DRRs and X-rays (Chen et al., 2024).

2. Dual-branch CNN-transformer architecture

The full pipeline takes an input CT volume $V$, a fixed X-ray $I_x$, and an initial pose $\theta_{\text{ini}}$. Following prior work, a 3D CNN first learns a residual representation from the CT volume, and a differentiable projection module generates the moving DRR $$\mathcal{F}(\theta)=\arg\min_{\theta}\mathcal{S}(I_x, I_m) =\arg\min_{\theta}\mathcal{S}(I_x,\mathcal{P}(\theta;V)),$$0. The pair $$\mathcal{F}(\theta)=\arg\min_{\theta}\mathcal{S}(I_x, I_m) =\arg\min_{\theta}\mathcal{S}(I_x,\mathcal{P}(\theta;V)),$$1 is then processed by a dual-branch CNN-transformer encoder composed of three parts: a shallow shared feature encoder (SFE), a global-local feature decomposition (GLD) layer, and a similarity evaluation (SE) layer (Chen et al., 2024).

The SFE uses a weight-shared Global Poolformer module to extract shallow features from both modalities. This choice is intended to capture shared spatial structure with global dependencies while avoiding the heavy computational cost of full self-attention. The GLD layer then separates the shared features into global and local branches. For global features, Cross3DReg uses residual Fast Fourier Convolution, motivated by its ability to capture long-range context and global frequency information. For local features, it uses an invertible neural network with affine coupling layers, chosen because invertibility helps preserve input information and thus retain local details (Chen et al., 2024).

The decomposition is written as

$$\mathcal{F}(\theta)=\arg\min_{\theta}\mathcal{S}(I_x, I_m) =\arg\min_{\theta}\mathcal{S}(I_x,\mathcal{P}(\theta;V)),$$2

where $$\mathcal{F}(\theta)=\arg\min_{\theta}\mathcal{S}(I_x, I_m) =\arg\min_{\theta}\mathcal{S}(I_x,\mathcal{P}(\theta;V)),$$3 and $$\mathcal{F}(\theta)=\arg\min_{\theta}\mathcal{S}(I_x, I_m) =\arg\min_{\theta}\mathcal{S}(I_x,\mathcal{P}(\theta;V)),$$4 denote the global and local feature extractors, and $$\mathcal{F}(\theta)=\arg\min_{\theta}\mathcal{S}(I_x, I_m) =\arg\min_{\theta}\mathcal{S}(I_x,\mathcal{P}(\theta;V)),$$5 are the shared shallow features for X-ray and moving DRR. This architectural split is the main structural mechanism by which Cross3DReg separates shallow shared representation learning from global-local decomposition (Chen et al., 2024).

3. Correlation-driven loss and similarity-landscape shaping

The correlation-driven mechanism gives Cross3DReg both its name and its main conceptual contribution. The key assumption is that conventional registration ultimately increases correlation in the information that matters for alignment—especially high-frequency structural content—while low-frequency components can be misleading because DRRs and X-rays differ in appearance. To encode that assumption directly, the method introduces a correlation-driven decomposition loss:

$$\mathcal{F}(\theta)=\arg\min_{\theta}\mathcal{S}(I_x, I_m) =\arg\min_{\theta}\mathcal{S}(I_x,\mathcal{P}(\theta;V)),$$6

where $$\mathcal{F}(\theta)=\arg\min_{\theta}\mathcal{S}(I_x, I_m) =\arg\min_{\theta}\mathcal{S}(I_x,\mathcal{P}(\theta;V)),$$7 is normalized cross-correlation and $$\mathcal{F}(\theta)=\arg\min_{\theta}\mathcal{S}(I_x, I_m) =\arg\min_{\theta}\mathcal{S}(I_x,\mathcal{P}(\theta;V)),$$8 is a small stabilizer (Chen et al., 2024).

Minimizing this ratio encourages the global branches to become less correlated while forcing the local branches to become more correlated, effectively decomposing embedded information into registration-relevant high-frequency detail and registration-irrelevant low-frequency context. The paper presents this as its main answer to the interpretability and control problem: gradient flow is guided by an explicit decomposition objective rather than being an opaque byproduct of black-box feature learning (Chen et al., 2024).

The SE layer estimates image similarity from both the global and local feature paths using two small multilayer perceptrons, each with five fully connected layers and ReLU activations. The network output is

$$\mathcal{F}(\theta)=\arg\min_{\theta}\mathcal{S}(I_x, I_m) =\arg\min_{\theta}\mathcal{S}(I_x,\mathcal{P}(\theta;V)),$$9

where $I_x$0 is a sigmoid function and $I_x$1 denotes inner product. This output acts as a learned similarity objective for the differentiable registration process (Chen et al., 2024).

Training further uses a “double backward” mechanism to approximate a convex-shape geodesic similarity landscape in $I_x$2. Instead of applying a naive $I_x$3 loss on pose parameters, the method trains the network so that its gradient approximates the gradient of the squared geodesic pose error $I_x$4. The approximation objective is

$I_x$5

and the total loss uses uncertainty-weighted multi-loss balancing:

$I_x$6

where $I_x$7 are learnable weights. The paper argues that the convex-shape approximation reduces the risk of poor local minima and makes gradient-based pose updates more stable and predictable (Chen et al., 2024).

4. End-to-end differentiable registration process

Cross3DReg is fully differentiable end-to-end. During training, the target X-ray is synthesized on the fly from a randomly sampled target pose $I_x$8 and CT volume $I_x$9, while the network input uses another randomly sampled pose $V$0. The differentiable projection module and the learned similarity module together allow gradients to flow from the image-level similarity output back through the projected DRR and into the pose parameters (Chen et al., 2024).

In this formulation, the method behaves like an iterative optimizer whose objective is learned rather than hand-designed. The paper explicitly states that, in inference, the pose update is implemented using SGD, enabling gradient-based alignment over $V$1 by backpropagating through the entire pipeline. A plausible implication is that Cross3DReg should be read not merely as a similarity network, but as a learned registration energy whose geometry is itself part of the training target (Chen et al., 2024).

This design differentiates Cross3DReg from approaches that only learn a latent similarity embedding. Its stated aim is to learn a similarity landscape whose geometry supports reliable convergence, rather than only producing a score that is informative in aggregate (Chen et al., 2024).

5. Dataset, evaluation protocol, and reported performance

The reported experiments use an in-house lumbar spine dataset of 465 CT scans. After automatic spine segmentation, CTs are resampled to isotropic 1.0 mm spacing and cropped or padded to $V$2. The data split uses 418 scans for training and validation and 47 for testing. Simulated X-rays are generated at $V$3 resolution with 0.798 mm pixel spacing, and test cases include 500 simulated X-rays with randomized pose perturbations: rotations drawn from $V$4 degrees in each axis and translations from $V$5 mm in-plane and $V$6 mm in depth (Chen et al., 2024).

Evaluation uses mean target registration error (mTRE) at the top 50%, 75%, and 95% of cases, and success rate (SR), defined as the percentage of cases with TRE below 10 mm. The paper reports that Cross3DReg outperforms the compared fully differentiable methods ProST and SOPI, and also improves over the initial pose and over the optimization-based baseline when used as a learned initializer before CMA-ES (Chen et al., 2024).

Without CMA-ES refinement, Cross3DReg achieves the best learned-registration numbers among the methods compared: top-95% mTRE of $V$7 mm, top-75% mTRE of $V$8 mm, top-50% mTRE of $V$9 mm, and the paper reports a stronger success rate than the competing fully differentiable methods. After CMA-ES refinement, performance further improves to $I_m$0 mm / $I_m$1 mm / $I_m$2 mm for the 95/75/50% mTREs, with SR of 61.0%, compared with 55.6% for ProST+CMA-ES and 58.4% for SOPI+CMA-ES. The authors interpret this as evidence that Cross3DReg provides a broader capture range and a better initialization for subsequent optimization. Qualitatively, overlay images show better alignment of DRR edges with fixed X-ray structures after registration (Chen et al., 2024).

The reported evidence is specific to a synthetic spine registration benchmark derived from an in-house dataset. This suggests that the paper’s claims are strongest for the stated experimental setting: rigid X-ray-to-CT registration for fluoroscopic guidance in lumbar spine imagery (Chen et al., 2024).

6. Relation to adjacent literature and naming ambiguity

Within cross-modal registration research, Cross3DReg occupies a specific position. It addresses image-based rigid 2D/3D registration in X-ray-to-CT fusion, whereas several adjacent methods in the provided literature operate on different modality pairs or different geometric primitives. PCR-CG is a cross-modal RGB-D registration method that augments Predator with explicitly projected deep image features for point-cloud registration (Zhang et al., 2023). CrossKEY is a fully 3D, patient-specific cross-modal keypoint descriptor and registration framework for preoperative MRI and intraoperative ultrasound that uses synthetic iUS generated from the patient’s own MRI to learn a shared descriptor space (Morozov et al., 24 Jul 2025). CMHANet and CrossI2P incorporate 2D image cues with 3D point-cloud features in coarse-to-fine multimodal registration pipelines (Zhang et al., 13 Mar 2026, Wang et al., 19 Sep 2025). These works share the broad goal of cross-modal registration, but they do not address the same X-ray-to-CT differentiable rendering problem as Cross3DReg (Chen et al., 2024).

The name “Cross3DReg” later acquired a second meaning in a different subfield. In “Cross3DReg: Towards a Large-scale Real-world Cross-source Point Cloud Registration Benchmark,” it denotes both a large-scale real-world benchmark and a registration framework for cross-source point cloud registration, collected by a rotating mechanical lidar and a hybrid semi-solid-state lidar and using unaligned images to predict overlapping regions between source and target point clouds (Xu et al., 8 Sep 2025). That later usage concerns point-cloud alignment under sensor heterogeneity rather than X-ray-to-CT 2D/3D fusion.

In the medical-imaging sense established in 2024, Cross3DReg is distinguished by three linked design commitments: explicit global-local decomposition, a correlation-driven loss that pushes those components in different correlation directions, and training that learns to approximate a convex-shaped geodesic similarity landscape. The paper presents these elements as the basis for improved controllability, interpretability, and optimization behavior relative to prior fully differentiable baselines (Chen et al., 2024).