SAMIR: Foundation-Model Assisted Registration

• SAMIR is a medical image registration framework that repurposes the Segment Anything Model as a feature extractor for robust, label-free anatomical alignment.
• It adapts a 2D pretrained SAM encoder to 3D volumes using a lightweight convolutional head and intelligent upsampling for volumetric consistency.
• The framework employs a coarse-to-fine pyramid deformation estimation and a hierarchical feature consistency loss to improve alignment accuracy.

Searching arXiv for the specified SAMIR paper and closely related registration/foundation-model context. SAMIR is a medical image registration framework that repurposes the Segment Anything Model (SAM) as a robust feature extractor for deformable registration. It is introduced as “Segment Anything Model for Image Registration” and is designed to improve anatomical alignment without depending on weak labels such as segmentation masks or landmarks at inference time. Rather than relying directly on raw intensity similarity between moving and fixed images, SAMIR uses SAM’s pretrained image encoder to produce structure-aware embeddings, refines them with a lightweight 3D head, and optimizes registration in a coarse-to-fine pyramid with a Hierarchical Feature Consistency Loss (He et al., 17 Sep 2025).

1. Problem setting and motivation

Medical image registration is the task of aligning a moving image $I_m$ to a fixed image $I_f$ through a deformation field. In SAMIR, the registration model predicts

$\phi(x) = x + u(x),$

where $x$ is the voxel coordinate and $u(x)$ is the displacement field (He et al., 17 Sep 2025).

The framework is motivated by a limitation of prior deep registration methods: many optimize intensity-based losses such as NCC or MSE on warped images. This is problematic because intensity patterns vary significantly across scanners, protocols, and operators; repeated scans can still differ in contrast or noise; raw-intensity similarity can miss semantic or anatomical consistency; and large deformations are difficult to handle with single-scale models (He et al., 17 Sep 2025).

Weakly supervised methods partly address these issues by incorporating anatomical priors through segmentation masks or landmarks, but such annotations are often unavailable. SAMIR treats SAM’s pretrained visual representations as a substitute source of anatomical prior information. This suggests a shift from task-specific representation learning toward foundation-model-assisted registration, with the frozen encoder supplying robust structural information learned from large-scale pretraining (He et al., 17 Sep 2025).

2. Architectural composition

SAMIR has two main parts: Structure-Aware Feature Embedding (SAFE) and Pyramid Deformation Field Prediction (PDFP) (He et al., 17 Sep 2025).

SAFE uses the frozen SAM image encoder to extract robust structural features from both the moving and fixed images. The encoder is applied to produce embeddings that are intended to preserve anatomical content under noise, contrast shifts, and acquisition differences. Since SAM is naturally a 2D model for natural images, this component requires explicit adaptation to 3D medical volumes (He et al., 17 Sep 2025).

PDFP then operates on these refined embeddings through a coarse-to-fine hierarchy. The refined feature embeddings are downsampled hierarchically to create multi-scale feature maps. At each pyramid level, moving and fixed features are passed into a small three-layer convolution block that predicts a velocity field $\mathbf{u}$, which is then integrated to obtain the deformation field $\phi^l_0$. Hierarchical refinement is expressed as

$$\tilde{\phi}^{l+1} = {up}(\phi^{l+1}), \nonumber \ \phi^{l} = \tilde{\phi}^{l+1} \circ \Delta \phi^{l}_0,$$

where ${up}(\cdot)$ is $2\times$ trilinear upsampling and scaling, $I_f$0 is the scaling-and-squaring function applied to the displacement field, and $I_f$1 denotes warping or composition. At the bottom pyramid level, $I_f$2 (He et al., 17 Sep 2025).

The architectural intent is to combine pretrained structure-aware features with coarse-to-fine deformation estimation. In the paper’s framing, this allows SAMIR to capture global, large-scale deformation at coarse levels while refining local anatomical details at finer levels (He et al., 17 Sep 2025).

3. Adaptation of SAM to volumetric registration

A central technical issue is that SAM is designed for 2D square natural-image inputs, whereas medical registration commonly involves 3D volumes with arbitrary aspect ratios, different spatial resolutions, and volumetric consistency requirements (He et al., 17 Sep 2025).

SAMIR addresses this with a slice-wise adaptation strategy. The input volume has shape $I_f$3. Each slice is extracted as $I_f$4, and slice features are extracted independently: $I_f$5 These slice embeddings are then stacked back into a 3D representation $I_f$6 (He et al., 17 Sep 2025).

Because SAM expects square inputs, medical slices are first padded from $I_f$7 to $I_f$8 and then upsampled to $I_f$9 before encoding. The paper describes the resolution mismatch as being bridged by “intelligent upsampling” to match SAM’s expected input size (He et al., 17 Sep 2025).

Spatial detail recovery is treated explicitly. SAM’s output features are at $\phi(x) = x + u(x),$0 of the input spatial resolution, which risks degrading deformation prediction. To restore spatial detail, SAMIR applies a two-stage upsampling strategy: first upsample the input by $\phi(x) = x + u(x),$1, then upsample the feature map by $\phi(x) = x + u(x),$2. This yields embeddings of size

$\phi(x) = x + u(x),$3

The paper’s algorithmic summary is: pad each slice, upsample to $\phi(x) = x + u(x),$4, pass through the SAM encoder, stack slice features, restore spatial dimensions, and apply a 3D head for enhancement (He et al., 17 Sep 2025).

This design indicates that SAMIR does not fine-tune SAM end-to-end for volumetric registration. Instead, it preserves the pretrained encoder and builds a task-specific bridge between 2D foundation-model features and 3D anatomical deformation modeling.

4. Lightweight 3D head and hierarchical feature loss

After SAFE produces structure-aware embeddings, SAMIR applies a lightweight 3D convolutional head. The paper gives two reasons for this component: SAM outputs high-dimensional embeddings, such as 256 channels, which can be expensive, and slice-wise 2D extraction alone does not enforce volumetric consistency (He et al., 17 Sep 2025).

The 3D head refines the SAM features in the embedding space, fuses information across adjacent slices, improves spatial continuity, and adapts generic visual features to medical anatomy and deformation patterns. It is characterized as a lightweight 3D convolutional module designed to “jointly reduce computational complexity through efficient feature fusion and improve spatial continuity by enhancing inter-slice correlations” (He et al., 17 Sep 2025).

A major contribution of the framework is the Hierarchical Feature Consistency Loss $\phi(x) = x + u(x),$5, which encourages alignment in feature space across multiple scales rather than relying only on image intensity similarity. The paper defines

$\phi(x) = x + u(x),$6

Here $\phi(x) = x + u(x),$7 and $\phi(x) = x + u(x),$8 are the moving and fixed feature embeddings at level $\phi(x) = x + u(x),$9, and the moving features are spatially transformed by the deformation field at that scale (He et al., 17 Sep 2025).

The total similarity loss is

$x$0

and the total training objective is

$x$1

with

$x$2

and

$x$3

In this formulation, $x$4 is image-level normalized cross-correlation, $x$5 is regional Dice loss for the weakly supervised setting, and $x$6 is diffusion-based smoothness regularization (He et al., 17 Sep 2025).

The role of HFC is to enforce consistency from coarse to fine levels in feature space. A plausible implication is that the loss acts as a multi-scale semantic regularizer, encouraging the deformation to respect anatomical structure even when intensity patterns are unstable across scans.

5. Training configuration, datasets, and baselines

SAMIR is evaluated on two benchmark datasets. The first is ACDC cardiac MRI, used for intra-subject registration between end-diastole and end-systole. The split is 80 training, 20 validation, and 50 test cases, and bidirectional registration gives 160 train pairs, 40 validation pairs, and 100 test pairs. The data are preprocessed to $x$7 with resolution $x$8 mm$x$9 (He et al., 17 Sep 2025).

The second dataset is abdomen CT for inter-subject registration with large deformations. It contains 30 CT scans divided into 20 train, 3 validation, and 7 test scans, and pairwise combinations yield 380 training pairs, 6 validation pairs, and 42 test pairs. These volumes are preprocessed to $u(x)$0 (He et al., 17 Sep 2025).

Optimization uses Adam with learning rate $u(x)$1. The implementation is in PyTorch and runs on a single NVIDIA RTX A6000. The paper states that SAM features were precomputed to avoid repeated extraction, which supports the efficiency claim that the pretrained backbone does not add runtime burden during registration inference (He et al., 17 Sep 2025).

The evaluation metrics are Dice, HD95, SDlogJ, MAs, PS, and Time. The listed baselines include VoxelMorph, TransMorph, LKU-Net, Fourier-Net, CorrMLP, MemWarp, and RDP. For abdomen CT, the comparison also includes LapIRN, ConvexAdam, and SAMConvex (He et al., 17 Sep 2025).

The positioning relative to prior work is explicit. VoxelMorph, TransMorph, LKU-Net, and Fourier-Net are described as learning features directly from medical images using standard CNN or Transformer backbones while still relying mainly on intensity-driven optimization and task-specific feature learning. LapIRN, RDP, CorrMLP, and MemWarp improve large-deformation handling through coarse-to-fine optimization but still learn features from registration data itself. Weakly supervised methods depend on masks or landmarks. SAMIR differs by using a frozen SAM encoder, adapting it to 3D with a lightweight head, matching features with HFC, and combining these with pyramid deformation field prediction (He et al., 17 Sep 2025).

6. Quantitative performance, ablations, and efficiency

On ACDC, the reported Dice scores are: Initial 58.14, VoxelMorph 75.26, TransMorph 74.97, LKU-Net 76.53, Fourier-Net 76.61, CorrMLP 77.31, MemWarp 76.74, RDP 78.06, and SAMIR-vith (Ours) 80.74. Other reported metrics for SAMIR on ACDC are HD95 8.22, SDlogJ 0.048, MAs 230.34 G, PS 7.15 MB, and Time 0.32 s. The paper states that SAMIR improves Dice by 2.68% over the sub-optimal method RDP on ACDC (He et al., 17 Sep 2025).

On abdomen CT, the reported Dice scores are: Initial 30.68, VoxelMorph 47.05, TransMorph 47.94, LKUNet 52.78, LapIRN 54.55, CorrMLP 56.11, RDP 58.77, MemWarp 60.24, FourierNet 42.80, ConvexAdam 51.10, SAMConvex 53.65, and SAMIR-vith (Ours) 66.68. Other metrics for SAMIR are HD95 13.45 and SDlogJ 0.17. The paper reports a 6.44% Dice improvement over the sub-optimal method MemWarp on abdomen CT (He et al., 17 Sep 2025).

The ablation results isolate the SAM encoder and HFC loss. On abdomen CT, only HFC loss gives Dice 64.90, HD95 15.86, SDlogJ 0.17; only SAM encoder gives Dice 66.19, HD95 13.93, SDlogJ 0.17; and SAM encoder plus HFC loss gives Dice 66.68, HD95 13.45, SDlogJ 0.17. This shows that both components contribute, with the encoder providing strong structural features and HFC further improving alignment, especially HD95 (He et al., 17 Sep 2025).

Backbone ablations on ACDC compare SAM-ViT B, L, H, and MedSAM-ViT B. The reported numbers are: SAM-ViT B Dice 80.49, HD95 8.39; SAM-ViT L Dice 80.74, HD95 8.27; SAM-ViT H Dice 80.74, HD95 8.22; MedSAM-ViT B Dice 80.97, HD95 7.90. The paper concludes that ViT-L and ViT-H offer only marginal gains over ViT-B, that ViT-L already captures sufficient structure information, and that larger models provide limited additional benefit (He et al., 17 Sep 2025).

Input-size ablation on abdomen CT gives Dice 65.36 for $u(x)$2, 66.68 for $u(x)$3, and 67.05 for $u(x)$4. Although $u(x)$5 is best, the gain is small and computation increases substantially, so the selected input size is $u(x)$6 (He et al., 17 Sep 2025).

The paper also evaluates robustness under gamma-based contrast changes and Gaussian noise of varying standard deviations, stating that SAMIR performs better than conventional image-only registration methods under these perturbations because it uses structure-aware features instead of raw intensities (He et al., 17 Sep 2025).

Efficiency claims are tied to freezing and precomputing the SAM encoder. The framework emphasizes that the encoder is frozen, features can be precomputed, and the method adds no major runtime burden during registration. On ACDC, feature extraction takes about 0.94 seconds per sample, while inference time is 0.32 s, which is presented as competitive with strong baselines (He et al., 17 Sep 2025).

7. Interpretation, scope, and limitations

SAMIR’s central claim is that a pretrained foundation model can serve as a source of robust anatomical representation for deformable registration. The reported empirical pattern supports this: the largest relative improvement appears on abdomen CT, where inter-subject deformations are large and raw intensity similarity is a weak registration signal. This suggests that foundation-model-derived features may be especially useful when anatomical structure is more stable than image appearance (He et al., 17 Sep 2025).

The framework should not be conflated with segmentation systems whose names resemble it. A separate paper introduces SAMIRA, a conversational VR system for interactive medical image segmentation (Spiegler et al., 12 May 2025). SAMIR, by contrast, is a registration framework that uses SAM-based feature extraction and coarse-to-fine deformation modeling (He et al., 17 Sep 2025).

Several practical boundaries are explicit in the paper. SAM is used as a frozen encoder rather than being retrained specifically for medical registration. The 3D adaptation is slice-wise before volumetric fusion, which preserves compatibility with the original encoder but may limit how much true volumetric context is captured prior to the 3D head. The weakly supervised loss includes $u(x)$7, but the framework’s motivation is to reduce dependence on labels at inference time, not necessarily to eliminate all supervision during training (He et al., 17 Sep 2025).

A plausible implication is that SAMIR occupies an intermediate position between purely intensity-driven registration and heavily supervised anatomy-aware registration. It does not require masks or landmarks to drive inference-time alignment, yet it does not discard classical registration ingredients such as multi-scale deformation estimation, smoothness regularization, and NCC. Its contribution is therefore less a replacement of registration methodology than a reweighting of the feature hierarchy on which registration is built.

In summary, SAMIR is defined by four coupled design choices: a frozen SAM encoder for structure-aware anatomical representation, a lightweight 3D head for volumetric refinement, a pyramid deformation field predictor for global-to-local alignment, and a hierarchical feature consistency objective for semantic matching across scales. Within the reported experiments, these elements together yield higher Dice and competitive efficiency on both intra-subject cardiac MRI registration and inter-subject abdomen CT registration (He et al., 17 Sep 2025).