2D/3D Registration Quality Verification

• 2D/3D registration quality verification is a process that assesses the accuracy of aligning 2D images with 3D models using metrics like mTRE, reprojection error, and multimodal consistency.
• It employs a combination of ground-truth benchmarks, internal consistency measures, and optimization diagnostics to classify registrations as acceptable or rejectable.
• The approach supports diverse applications—from surgical imaging to biometrics—by integrating automated quantitative metrics with human-centered explainable AI for robust decision-making.

2D/3D registration quality verification is the assessment of whether a completed alignment between a 2D observation and a 3D model, mesh, or volume should be accepted or rejected, using geometric error measures, image-based consistency, multimodal labels, optimizer diagnostics, or human/operator review. Across colonoscopy, neurovascular intervention, surgical fluoroscopy, and cross-modal biometrics, the verification problem is shaped by the same core difficulty: registration algorithms can produce plausible but incorrect alignments, while the available evidence ranges from full ground truth pose and depth to indirect similarity scores and sparse annotations (Bobrow et al., 2022, Homan et al., 2021, Cho et al., 23 Jul 2025, Guan et al., 15 May 2026).

1. Problem formulation and verification targets

In image-guided surgery, the verification task has been explicitly formulated as binary classification of a completed registration as Accept or Reject, with ground-truth quality defined by mean Target Registration Error (mTRE) computed from anatomical landmarks; in the pelvis study, a registration is labeled successful if $\mathrm{mTRE} < 2\ \mathrm{mm}$ and rejected otherwise (Cho et al., 23 Jul 2025). In neurovascular interventions, the operational target is rigid motion compensation after initial machine-based alignment from C-arm geometry sensors, and quality is judged by residual registration errors under induced rigid perturbations, with success criteria of residual translation $< 1\ \mathrm{mm}$ and residual rotation $< 3^\circ$ (Homan et al., 2021).

In colonoscopy, verification is tied to a multimodal 2D–3D registration pipeline in which optical video is transformed to depth with a conditional GAN, edge features are aligned with CMA-ES, and a single rigid transform $T_{\text{final}}$ aligns a static phantom model to a measured camera trajectory (Bobrow et al., 2022). The same dataset exposes per-frame depth, normals, optical flow, occlusion, and 6-DoF pose, so verification is not limited to a single residual but can be expressed as a family of consistency tests. In a different domain, cross-modal fingerprint registration treats verification as ridge-level agreement between 3D point clouds and 2D rolled-equivalent representations, using fusion accuracy, projection accuracy, and downstream matching compatibility as indicators of registration quality (Guan et al., 15 May 2026).

A recurring distinction is between verification with ground truth and verification without ground truth. The former supports direct pose, reprojection, or depth error measurement; the latter relies on internal consistency, confidence surrogates, or structured priors. Another recurring distinction is between single-frame and multi-frame verification. In C3VD, the loss is accumulated over $K$ keyframes, and using $K=5$ improved translation and rotation accuracy relative to single-frame registration, which directly links temporal consistency to verification robustness (Bobrow et al., 2022).

2. Geometric, photometric, and multimodal quality measures

Rigid 2D/3D registration verification is commonly organized around translation, rotation, and projection consistency. In C3VD, if $t_{\mathrm{est}}, t_{\mathrm{gt}} \in \mathbb{R}^3$, the translation error is

$e_t = \| t_{\mathrm{est}} - t_{\mathrm{gt}} \|_2,$

and if $R_{\mathrm{est}}, R_{\mathrm{gt}} \in SO(3)$, the rotation error can be expressed as the geodesic angle

$$e_r = \arccos\!\left(\frac{\mathrm{trace}(R_{\mathrm{est}} R_{\mathrm{gt}}^\top)-1}{2}\right),$$

or equivalently as

$< 1\ \mathrm{mm}$0

For 3D model points $< 1\ \mathrm{mm}$1 and a calibrated omnidirectional projection $< 1\ \mathrm{mm}$2, the reprojection error is

$< 1\ \mathrm{mm}$3

The same framework also defines a Chamfer distance between edge sets, silhouette IoU, depth consistency between GAN-predicted and rendered depth, temporal continuity measures $< 1\ \mathrm{mm}$4 and $< 1\ \mathrm{mm}$5, cumulative drift, and flow endpoint error $< 1\ \mathrm{mm}$6 using rendered optical flow and occlusion-aware masking (Bobrow et al., 2022).

In neurovascular fluoroscopy, quality is tied to a gradient-difference similarity between the live fluoroscopy image $< 1\ \mathrm{mm}$7 and the digitally reconstructed radiograph $< 1\ \mathrm{mm}$8:

$< 1\ \mathrm{mm}$9

Residual rotational error is measured from $< 3^\circ$0 via

$< 3^\circ$1

while the reported residual translational error is the Euclidean norm of the in-plane component

$< 3^\circ$2

The search space excludes translation along the source-detector axis because a single projection primarily manifests $< 3^\circ$3-motion as magnification changes and is poorly observable (Homan et al., 2021).

In surgical fluoroscopy with CT-derived DRRs, the paper on explainable AI for collaborative assessment uses

$< 3^\circ$4

for the rigid transform, standard pinhole projection,

$< 3^\circ$5

and TRE

$< 3^\circ$6

with mTRE used as the acceptability variable for labeling verification outcomes (Cho et al., 23 Jul 2025).

A notable feature of the literature is that verification is rarely a single metric. High-quality assessments typically combine pose-space residuals, image-plane residuals, contour or silhouette agreement, temporal smoothness, and modality-specific observables such as depth, flow, or ridge structure.

3. Ground-truth generation and benchmark construction

The most direct verification regime is built on datasets that expose paired 2D observations and trusted 3D-derived labels. C3VD is a canonical example: 22 short video sequences were registered to generate 10,015 total frames with paired ground-truth depth, surface normals, optical flow, occlusion, six degree-of-freedom pose, coverage maps, and 3D models; the dataset also includes simulated screening videos and screening videos acquired by a gastroenterologist with paired ground truth pose and 3D surface models (Bobrow et al., 2022). In simulation experiments where error-free ground truth is available, the proposed registration method achieved an average translation error of $< 3^\circ$7 millimeters and an average rotation error of $< 3^\circ$8 degrees, with video information improving registration accuracy by $< 3^\circ$9 for translation and $T_{\text{final}}$0 for rotation compared to single-frame registration (Bobrow et al., 2022).

A different strategy avoids fiducials entirely. In neurovascular intervention, the raw 2D projection frames and the reconstructed 3D rotational angiography volume share a nearly perfect geometric relationship determined solely by C-arm calibration and the assumption of no motion during the rotational scan. Artificial rigid offsets are then applied to the known pose, and the image-based algorithm is evaluated by its ability to recover the ground truth retrospectively, without changes to clinical workflow (Homan et al., 2021). Phantom experiments comprised 512 registrations; 97% of the clinically relevant subset passed the success criterion of residual translation below $T_{\text{final}}$1 and residual rotation below $T_{\text{final}}$2, while clinical data experiments yielded 87.1% of attempts with residual translation below $T_{\text{final}}$3 and 83.9% with residual rotation below $T_{\text{final}}$4 (Homan et al., 2021).

The pelvis verification study creates labels by repeated registration under randomized initializations. For each of 1,000 projections across 5 cadaveric specimens, 100 single-view 2D/3D registration results were generated, mTRE was computed from ground-truth 3D landmarks and transformed 3D points, and each result was labeled Accept if $T_{\text{final}}$5 and Reject otherwise (Cho et al., 23 Jul 2025). This turns verification itself into a supervised learning problem rather than a direct optimization residual.

Cross-modal fingerprints extend the benchmark idea beyond medical imaging. The multimodal database contains 150 fingers from 15 subjects; for each finger it includes two partial 3D scans, three contactless 2D fingerprints, and a rolling sequence of contact-based fingerprints, enabling stage-wise verification of 3D fusion, pose normalization, contactless 2D–3D alignment, and contact-based pose-aware re-unwrapping (Guan et al., 15 May 2026). The reported fusion errors are concentrated around $T_{\text{final}}$6, contactless 2D–3D registration reaches projection errors concentrated around 6 pixels on 96-dpi images, and pose normalization shows median angle errors around $T_{\text{final}}$7 with most errors below $T_{\text{final}}$8 (Guan et al., 15 May 2026).

These datasets illustrate three distinct benchmark logics: direct physical ground truth, retrospective geometric ground truth from calibrated acquisition, and label generation from trusted landmark-derived error measures.

4. Verification without explicit ground truth

When direct ground truth is unavailable, verification shifts toward internal consistency and optimization diagnostics. In C3VD, the edge-based similarity objective over $T_{\text{final}}$9 keyframes is bounded in $K$0, so the normalized final cost $K$1 serves as a confidence indicator; high similarity implies better alignment, and thresholds can be calibrated per dataset segment and texture (Bobrow et al., 2022). The same framework recommends edge inlier ratios, Chamfer tolerances, silhouette IoU, temporal smoothness, optical-flow consistency, coverage-map plausibility, and rejection of sequences with unstable optimizer convergence or anomalously high costs. It also provides a local covariance approximation

$K$2

with small covariance interpreted as high confidence, and suggests practical thresholds such as $K$3–$K$4 pixels and $K$5 under the omnidirectional camera model (Bobrow et al., 2022).

The neurovascular study uses different but structurally related surrogates. It recommends monitoring the gradient content, the trend of the similarity cost $K$6, and the imaging geometry itself: if image format is very small, if the overlay or 3DRA field of view is substantially smaller than the 2D image, or if large apparent magnification changes indicate residual $K$7-misalignment, image-based registration should be disabled and machine-based registration retained (Homan et al., 2021). This turns verification into a gatekeeping stage conditioned on observability and feature support, not only on final residual size.

A more optimization-theoretic notion of verification appears in the semidefinite programming framework for rigid 2D/3D point-set registration. There, solver-side quantities become certificates: a small primal–dual gap certifies global optimality of the convex relaxation; near-orthogonality of the rounded rotation, $K$8, near-integrality of the correspondence matrices, and satisfaction of the block-PSD cycle-consistency constraint indicate that the relaxation is tight and that pose and correspondence recovery are trustworthy (Khoo et al., 2015). Practical verification then combines duality gap, orthogonality error, determinant error, integrality checks, and reprojection residuals, followed if desired by local refinement initialized from the SDP solution (Khoo et al., 2015).

Taken together, these approaches show that ground-truth-free verification is not reducible to a single similarity score. It typically requires agreement between the objective landscape, motion continuity, projection geometry, and the algebraic structure of the optimizer’s solution.

5. Human-centered verification and explainable AI

Human review remains a central safeguard, but visualization alone has been found insufficient to enable humans to reliably detect 2D/3D registration misalignments in high-stakes surgical settings (Cho et al., 23 Jul 2025). The explainable-AI framework for collaborative assessment addresses this by training a dedicated accept/reject verifier on early-fused X-ray and DRR inputs. The architecture uses an initial double-channel convolutional block, repeated convolutional blocks with GELU, max pooling, and batch normalization, followed by channel splitting, bidirectional cross-attention, channel-wise averaging, dropout, and a final fully connected classifier (Cho et al., 23 Jul 2025).

The verification model outputs an Accept probability $K$9, and uncertainty is handled with split-conformal prediction. With calibration threshold $K=5$0 at confidence level $K=5$1, the method forms a prediction set $K=5$2; singleton sets indicate certain decisions, while $K=5$3 indicates uncertainty or abstention (Cho et al., 23 Jul 2025). Explanations are supplied by Grad-CAM heatmaps over the X-ray image, highlighting regions that most influence the accept/reject decision.

Algorithm-centric evaluation reported cross-validated performance of accuracy $K=5$4, precision $K=5$5, recall $K=5$6, F1 $K=5$7, and ROC-AUC $K=5$8 (Cho et al., 23 Jul 2025). In the user study, weighted real-world accuracy was 0.55 for Human-only, 0.71 for Human+AI, and 0.68 for Human+XAI. Explanations modestly improved trust and willingness to override AI errors, but they did not exceed the standalone AI in aggregate performance, and did not surpass the Human+AI condition in accuracy (Cho et al., 23 Jul 2025).

This line of work recasts verification as a collaborative decision problem. The output is not merely a metric but a structured recommendation: accept, reject, or abstain, accompanied by confidence and localized rationale.

6. Failure modes, interpretation limits, and broader verification principles

Verification performance is strongly conditioned by modality-specific failure modes. In colonoscopy, specular highlights, low texture, occlusions, tissue deformation, and wide field-of-view lens distortion alter the meaning of residuals; the recommended response is to use occlusion-aware masking, rely on depth discontinuity edges rather than color gradients, and preserve the calibrated omnidirectional camera model because mixing pinhole projections with fisheye data will bias metrics (Bobrow et al., 2022). In neurovascular imaging, low-gradient scenes, very small fields of view, mismatched collimation, and single-projection observability limits degrade reliability, particularly because translation along the source-detector axis is not optimized (Homan et al., 2021). In 3D–2D fingerprints, small overlap between partial scans, holes or weak ridge relief in the 3D acquisition, and poor minutiae quality in contactless 2D images weaken both coarse registration and quality assessment (Guan et al., 15 May 2026).

A common misconception is that larger similarity values always indicate better registration. Related evaluation work on serial-section registration shows the opposite possibility: over-registration becomes visible when registered metrics exceed the ground-truth distribution, such as GS maximal SSIM of 0.91995 compared with a ground-truth maximum of 0.90530 on CT, and analogous overshoots on light-sheet data (Lobachev et al., 2020). This suggests that in 2D/3D verification, similarity should be interpreted relative to a task-specific bound on achievable correspondence rather than as monotonic evidence of correctness.

Another limitation arises when landmark correspondences are themselves ambiguous. Uncertainty-aware annotation work models each correspondence as a Gaussian distribution $K=5$9 and evaluates a candidate transformation against the GP posterior mean and covariance induced by sparse uncertain annotations (Peter et al., 2021). A plausible implication is that 2D/3D verification can benefit from the same principle whenever “hard” point correspondences are unstable, for example in weakly textured or partially occluded views. In that setting, expected error, posterior uncertainty, and dense Mahalanobis-type incompatibility maps can complement direct TRE.

Across domains, several general principles recur. Core metrics such as $t_{\mathrm{est}}, t_{\mathrm{gt}} \in \mathbb{R}^3$0, $t_{\mathrm{est}}, t_{\mathrm{gt}} \in \mathbb{R}^3$1 on $t_{\mathrm{est}}, t_{\mathrm{gt}} \in \mathbb{R}^3$2, reprojection error, silhouette or edge agreement, depth consistency, and flow consistency transfer broadly beyond colonoscopy (Bobrow et al., 2022). Multi-frame or multi-view consistency improves robustness, whether through accumulated keyframe loss, bundle adjustment, pose-graph reasoning, or cycle-consistent correspondences (Bobrow et al., 2022, Khoo et al., 2015). Acceptance thresholds remain application-specific, but the literature consistently treats verification as a compound inference problem in which geometry, image evidence, uncertainty, and human oversight must agree before a registration is trusted.