---
title: 'FractMorph: Dual-Parallel Transformer for DIR'
url: https://www.emergentmind.com/topics/fractmorph
type: topic
---

# FractMorph: Dual-Parallel Transformer for DIR

FractMorph is a 3D dual-parallel transformer architecture for deformable image registration (DIR) that uses multi-domain fractional Fourier transform (FrFT) branches inside transformer cross-attention to capture fine local, semi-global, and large global deformations within a single end-to-end framework [2508.12445]. It operates on paired fixed and moving medical volumes, enriches cross-image feature matching through Fractional Cross-Attention (FCA), and predicts a dense deformation field with a lightweight U-Net style network. On the ACDC cardiac MRI dataset, it is reported to achieve an overall Dice Similarity Coefficient (DSC) of \(86.45\%\), an average per-structure DSC of \(75.15\%\), and a \(95\)th-percentile Hausdorff distance (HD95) of \(1.54\) mm on the authors’ data split [2508.12445].

## 1. Concept and problem setting

FractMorph addresses deformable image registration, in which a moving image \(I_m\) is aligned to a fixed image \(I_f\) by predicting a dense deformation field \(\phi\) such that the warped moving image \(I_m \circ \phi\) matches \(I_f\) [2508.12445]. The method is motivated by a specific limitation of prior learned DIR systems: CNN-based models are effective at capturing fine local deformations but often miss long-range context, whereas transformer-based methods model global correspondences more effectively but can lose local precision and do not necessarily represent different deformation scales within one unified design [2508.12445].

The model is formulated as an unsupervised registration system. During training, it does not use segmentation supervision; ground-truth segmentations are reserved for evaluation only [2508.12445]. Its stated objective is to model local, semi-global, and global deformation cues simultaneously by combining spatial, spectral, and intermediate fractional-domain information inside the feature-matching mechanism itself, rather than by relying on scenario-specific tuning or hierarchical multi-scale registration pipelines [2508.12445].

A central design choice is that FractMorph processes the fixed and moving volumes in parallel, rather than concatenating them into a single stream at the outset. This makes inter-image correspondence an explicit architectural concern throughout the network. The resulting formulation is therefore neither a purely convolutional registration model nor a standard single-stream vision transformer applied to 3D medical images; it is a dual-stream correspondence-learning system whose defining operation is Fractional Cross-Attention [2508.12445].

## 2. Network architecture and feature flow

FractMorph begins by dividing each 3D image into non-overlapping cubic patches. For an image \(I \in \mathbb{R}^{H \times W \times D}\) with patch size \(P_x \times P_y \times P_z\), the number of patches is

\[
N = \frac{H \cdot W \cdot D}{P_x \cdot P_y \cdot P_z}.
\]

The patch set is written as

\[
\mathcal{X} = \{\,\mathbf{x}^1,\mathbf{x}^2,\ldots,\mathbf{x}^N\},\quad \mathbf{x}^i\in\mathbb{R}^{P_x\times P_y\times P_z},
\]

and each patch is projected to a token by

\[
\mathbf{z}^i = \mathbf{x}^i \mathbf{E}, \quad \mathbf{z}^i \in \mathbb{R}^d.
\]

This yields token sequences \(\mathbf{Z}_f\) and \(\mathbf{Z}_m\) for the fixed and moving images, respectively [2508.12445].

The core backbone is a hierarchical dual-parallel transformer. At level \(l\), the fixed and moving features are denoted

\[
\mathbf{F}^l_m, \mathbf{F}^l_f \in \mathbb{R}^{H_l \times W_l \times D_l \times C_l}.
\]

The encoder applies several transformer blocks per level and downsamples by a factor of \(2\) in each spatial dimension while doubling the channel count:

\[
H_{l+1}=\tfrac{H_l}{2},\;W_{l+1}=\tfrac{W_l}{2},\;D_{l+1}=\tfrac{D_l}{2},\;C_{l+1}=2\,C_l.
\]

A symmetric decoder mirrors this structure, upsamples the coarse representations, fuses them with corresponding encoder skip connections, and refines both streams with additional transformer blocks [2508.12445].

Interaction between the streams occurs throughout the transformer, not only at the end. At each stage, fixed and moving features are first enriched through FrFT-based parallel branches and then exchanged through cross-attention in both directions. After decoding, the final fixed and moving features are concatenated, normalized, and passed through reverse patch embedding. A lightweight 3D U-Net style network—composed of three convolutional downsampling layers, a bottleneck, and three deconvolutional upsampling layers—then predicts the dense deformation field \(\phi\) from the transformer-enriched representation [2508.12445].

This architecture places transformers and convolutions in different roles. The transformer layers perform cross-image matching under multi-domain feature representations, whereas the CNN head converts those features into a dense, locally smooth displacement field at high spatial resolution. The paper presents this division as a way to preserve both correspondence modeling and displacement-field regularity within one network [2508.12445].

## 3. Fractional Cross-Attention and multi-domain FrFT branches

The defining module of FractMorph is the Fractional Cross-Attention block. FCA combines two operations: multi-domain feature extraction using FrFT branches, followed by cross-attention between the enriched fixed and moving streams [2508.12445].

The paper defines the 1D \(p\)-th order FrFT of a signal \(x(t)\) as

\[
X_{p}(u) = \mathcal{F}_{p}\{x\}(u) = \int_{-\infty}^{\infty} K_{p}(t,u)\,x(t)\,\mathrm{d}t,
\]

with kernel, for \(\alpha \neq n\pi\),

\[
K_p(t,u) \;=\; A_\alpha\, e^{\,j\!\bigl(\tfrac{t^2\cot\alpha}{2} -u\,t\,\csc\alpha +\tfrac{u^2\cot\alpha}{2}\bigr)},
\]

and normalization factor

\[
A_\alpha = \sqrt{\frac{1 - j\,\cot\alpha}{2\pi}}.
\]

The transform interpolates between the spatial and Fourier domains. In the paper’s framing, \(p=0\) corresponds to the spatial/original domain, \(p=1\) to the standard Fourier transform, and \(p=0.5\) to an intermediate fractional domain [2508.12445].

FractMorph implements 3D FrFT by separability, applying 1D FrFT successively along the \(x\), \(y\), and \(z\) axes. The discrete 3D transform is written as

\[
X_p[m_x,m_y,m_z] = \mathcal{F}^{(z)}_{p}\!\left\{ \mathcal{F}^{(y)}_{p}\!\left\{ \mathcal{F}^{(x)}_{p}\{x\} \right\} \right\}[m_x,m_y,m_z],
\]

with inverse

\[
x[n_x,n_y,n_z] = \mathcal{F}^{(x)}_{-p}\!\left\{ \mathcal{F}^{(y)}_{-p}\!\left\{ \mathcal{F}^{(z)}_{-p}\{X_p\} \right\} \right\}[n_x,n_y,n_z].
\]

Inside each FCA block, the normalized feature tensor is passed through four parallel branches: \(\mathrm{FrFT}_{0^\circ}\), \(\mathrm{FrFT}_{45^\circ}\), \(\mathrm{FrFT}_{90^\circ}\), and a \(\log(|\mathrm{FrFT}_{90^\circ}|)\) branch [2508.12445]. The paper interprets these branches as capturing local spatial, semi-global mixed spatial-frequency, global spectral, and fine spectral-magnitude information, respectively. For the \(90^\circ\) branch, the log-magnitude transform is given by

\[
A = \log \big( 1 + | X_{90^\circ}(u) | \big),
\]

with inverse

\[
| X_{90^\circ}(u) | = \exp(A) - 1.
\]

Each branch applies FrFT, convolution, ReLU, and inverse FrFT. The spatial branch uses kernel size \(3\), while the spectral branches use kernel size \(1\), consistent with the paper’s use of the Fourier convolution theorem,

\[
\mathcal{F}\{f * g\} = \mathcal{F}\{f\} \cdot \mathcal{F}\{g\},
\]

to motivate minimal spectral-domain kernels [2508.12445].

After branch processing, the outputs and the skip connection are concatenated along channels, normalized, and fused by pointwise convolution. The enriched moving and fixed features are then matched by cross-attention. For moving-to-fixed attention, the paper gives

\[
\text{Cross-Attention}(m \rightarrow f) = \text{softmax} \left( \frac{Q_m K_f^T}{\sqrt{d_k}} \right) V_f,
\]

where \(Q_m \in \mathbb{R}^{N_l \times C}\), \(K_f, V_f \in \mathbb{R}^{N_l \times C}\), and \(d_k = C/h\) with \(h\) attention heads [2508.12445]. The reverse fixed-to-moving attention is computed in parallel. These outputs are reshaped back to volumetric tensors, combined with residual connections, layer normalization, and feed-forward MLP blocks.

The FCA design is therefore the point at which FractMorph departs most sharply from standard transformer registration models. Cross-image attention is not applied to purely spatial tokens; it is applied to features that have already been expanded across multiple fractional Fourier domains [2508.12445].

## 4. Training objective, warping model, and implementation

FractMorph predicts a dense deformation field \(\phi\) and uses a spatial transformer to warp the moving image. At voxel \(p\), the deformation is

\[
p' = p + \phi(p),
\]

and the warped image is

\[
I_w(p) = I_m \circ \phi(p).
\]

The paper describes trilinear interpolation by summing over the neighborhood \(\mathcal{N}(p')\) of the deformed position \(p'\) [2508.12445].

Training is unsupervised. The total objective is the sum of an image-similarity term and a deformation smoothness term,

\[
\mathcal{L}_{\text{total}}(I_f, I_m, \phi) = \mathcal{L}_{\text{similarity}}(I_f, I_m \circ \phi) + \lambda\, \mathcal{L}_{\text{smooth}}(\phi).
\]

The similarity term is based on local cross-correlation. With local means \(\hat{I}_f(p)\) and \(\widehat{I_m \circ \phi}(p)\) over an \(n^3\) neighborhood, the cross-correlation is

\[
\begin{split}
CC(I_f,I_m\circ\phi)= \sum_{p\in\Omega} 
\frac{\bigl(\sum_{p_i}[I_f(p_i)-\hat I_f(p)]\,[\,(I_m\circ\phi)(p_i)-\widehat{I_m\circ\phi}(p)\,]\bigr)^2}
{\bigl(\sum_{p_i}[I_f(p_i)-\hat I_f(p)]^2\bigr)\, 
\bigl(\sum_{p_i}[(I_m\circ\phi)(p_i)-\widehat{I_m\circ\phi}(p)]^2\bigr)}.
\end{split}
\]

The similarity loss is

\[
\mathcal{L}_{\text{similarity}}(I_f, I_m, \phi) = -CC(I_f, I_m \circ \phi),
\]

and the smoothness penalty is

\[
\mathcal{L}_{\text{smooth}}(\phi) = \sum_{p \in \Omega} \| \nabla \phi(p) \|^2.
\]

The smoothness weight is fixed to \(\lambda = 1\) for all learning-based methods in the reported experiments [2508.12445].

The ACDC cardiac MRI volumes are normalized to \([0,1]\) and resized to \(16 \times 128 \times 128\), corresponding to voxel size \(10 \times 1.8 \times 1.8\) mm [2508.12445]. Patch size is set to

\[
P_x = P_y = P_z = 4,
\]

and the token embedding dimension is

\[
d = 48.
\]

All learning-based models are trained for \(400\) epochs with Adam and batch size \(1\) [2508.12445]. The paper specifies NVIDIA GeForce RTX 3080 hardware with \(12\) GB VRAM, CUDA 12.2, and PyTorch 2.2.1 for the learned methods. It does not report data augmentation for FractMorph [2508.12445].

## 5. Empirical evaluation on ACDC and architectural ablations

The experimental evaluation uses the ACDC dataset, which contains \(150\) patients with cine cardiac MRI at end-diastole and end-systole, with manual segmentations for the left ventricular cavity, myocardium, and right ventricular cavity [2508.12445]. The authors split the data by the dataset’s five groups, selecting from each group \(18\) cases for training, \(2\) for validation, and \(10\) for testing, for totals of \(90\), \(10\), and \(50\), respectively [2508.12445].

Registration is evaluated by warping moving labels and comparing them against fixed labels using overall DSC, average DSC across structures, HD95, the percentage of voxels with non-positive Jacobian determinant, and the standard deviation of the Jacobian determinant [2508.12445]. The paper defines

\[
DSC = \frac{2 |X \cap Y|}{|X| + |Y|},
\]

and reports HD95 as the \(95\)th-percentile Hausdorff distance.

The main quantitative results place FractMorph at the top of the authors’ comparison on their data split [2508.12445].

| Method | Registration metrics | Efficiency metrics |
|---|---|---|
| FractMorph | Overall DSC \(86.45 \pm 4.72\), Avg. DSC \(75.15 \pm 8.95\), HD95 \(1.54 \pm 0.78\) mm | \(63{,}910{,}483\) params, \(1.00\) s, \(594\) MB |
| FractMorph-Light | Overall DSC \(86.32 \pm 4.86\), Avg. DSC \(74.97 \pm 9.07\), HD95 \(1.57 \pm 0.82\) mm | \(29{,}630{,}931\) params, \(0.98\) s, \(320\) MB |
| VoxelMorph | Overall DSC \(86.08 \pm 5.99\), Avg. DSC \(74.88 \pm 8.69\), HD95 \(1.58 \pm 0.68\) mm | \(327{,}331\) params, \(0.02\) s, \(181\) MB |
| XMorpher | Overall DSC \(84.96 \pm 4.04\), Avg. DSC \(70.09 \pm 9.53\), HD95 \(1.82 \pm 0.77\) mm | \(15{,}093{,}891\) params, \(0.46\) s, \(207\) MB |
| TransMorph | Overall DSC \(82.87 \pm 7.22\), Avg. DSC \(68.11 \pm 11.70\), HD95 \(2.05 \pm 0.88\) mm | \(46{,}689{,}459\) params, \(0.30\) s, \(491\) MB |

The topology-related metrics show that FractMorph does not achieve the lowest reported folding rate or Jacobian variability. Fourier-Net reports \(0.00 \pm 0.00\) for the fraction of non-positive Jacobian determinant and \(0.03 \pm 0.01\) for the Jacobian standard deviation, while FractMorph reports \(0.05 \pm 0.04\) and \(0.15 \pm 0.03\), respectively [2508.12445]. The paper nevertheless argues that FractMorph improves alignment while keeping deformation plausibility at a competitive level.

The architectural ablations isolate the role of the FrFT branches and the FCA skip connection. The full model with all four branches reports overall DSC \(86.45 \pm 4.72\), average DSC \(75.15 \pm 8.95\), and HD95 \(1.54 \pm 0.78\) mm [2508.12445]. Removing \(\mathrm{FrFT}_{45^\circ}\) lowers performance to overall DSC \(86.12\), average DSC \(74.66\), and HD95 \(1.65\) mm, while removing \(\mathrm{FrFT}_{90^\circ}\) and keeping the log branch yields overall DSC \(86.11\), average DSC \(74.05\), and HD95 \(1.68\) mm [2508.12445]. The paper states that removing the \(45^\circ\) or \(90^\circ\) branch causes more than \(0.33\%\) drop in overall DSC and more than \(0.11\) mm increase in HD95.

The skip connection inside FCA also has a measurable effect. With the skip connection, the model reports overall DSC \(86.45 \pm 4.72\), average DSC \(75.15 \pm 8.95\), and HD95 \(1.54 \pm 0.78\) mm; without it, the values drop to \(85.45 \pm 4.54\), \(74.42 \pm 9.14\), and \(1.65 \pm 0.80\) mm [2508.12445]. This indicates that the residual path is not merely conventional transformer scaffolding but materially supports the registration accuracy of the FCA block.

## 6. FractMorph-Light, interpretation, and limitations

FractMorph-Light is a reduced-cost variant of the model that modifies the channel allocation inside the FrFT feature-extractor branches [2508.12445]. The paper defines a channel coefficient \(\alpha\): the main model uses \(\alpha = 1\), whereas FractMorph-Light uses \(\alpha = 1/3\). For branch processing with input shape \(C \times D \times H \times W\), the branch parameter count is

\[
36\,\alpha^2 C^2,
\]

and the branch convolution FLOPs are

\[
36\,\alpha^2 C^2 D H W.
\]

Thus, with \(\alpha = 1/3\), the branch parameter and convolution FLOP cost become \(1/9\) of those of the main model [2508.12445]. The empirical result is a substantial memory reduction—from \(594\) MB to \(320\) MB—and a reduction in parameters from \(63{,}910{,}483\) to \(29{,}630{,}931\), while maintaining nearly identical registration accuracy [2508.12445].

The paper’s interpretation of the model is that FrFT creates a continuous family of representational domains between image space and Fourier space, so FractMorph is not limited to a simple local/global dichotomy. Instead, it explicitly incorporates local, intermediate, and global deformation cues in each FCA block [2508.12445]. This suggests that its performance gain comes less from raw model size than from the structured way in which different deformation scales are exposed to cross-image attention.

The reported limitations are primarily computational. The paper identifies runtime overhead from FrFT operations as the main drawback and states that future work will focus on accelerating FrFT [2508.12445]. In the reported benchmarks, FractMorph is markedly slower than lightweight learned baselines such as VoxelMorph and Fourier-Net, though still much faster than traditional SyN and Demons implementations on the reported hardware [2508.12445]. The published evaluation is on ACDC cardiac MRI, and the paper does not present multi-modal or broader cross-dataset studies [2508.12445].

In the literature of learned deformable registration, FractMorph is therefore best understood as a transformer-based registration model whose novelty lies in relocating spectral-domain reasoning into the correspondence module itself. Rather than using frequency features as a peripheral augmentation, it makes fractional-domain processing a first-class component of cross-attention. Its encyclopedic significance derives from that architectural move: FractMorph is an attempt to formalize multi-scale, multi-domain registration as a single transformer problem, rather than as a sequence of separate local and global alignment stages [2508.12445].

Source: https://www.emergentmind.com/topics/fractmorph