FPRAN: Feature Pyramid Registration in Multimodal Models

• The paper introduces FPRAN as an auxiliary branch that performs coarse-to-fine registration of CT and MRI feature pyramids to mitigate cross-view spatial discrepancies.
• It utilizes four Spatial Alignment Modules with STN-based warping and cross-attention fusion to achieve robust alignment between differing imaging modalities.
• FPRAN’s integration with the ABPDC module shows significant performance gains in challenging cross-view tasks, notably improving postoperative neck pain recovery predictions.

Feature Pyramid Registration Auxiliary Network (FPRAN) is a multiscale registration branch introduced within a multimodal cross-view model for predicting postoperative neck pain recovery in cervical spondylosis patients from preoperative imaging in the MMCSD dataset. Its specific function is to mitigate cross-view spatial discrepancy between modalities, especially the mismatch between axial CT and sagittal MRI, in a setting where ordinary multimodal fusion is inadequate because the modalities differ both in grayscale/texture statistics and in anatomical plane. In the overall framework, FPRAN operates on intermediate CT and MRI feature pyramids, performs coarse-to-fine alignment through four Spatial Alignment Modules (SAMs), and is optimized jointly with the classification head through similarity and deformation regularization terms rather than as a standalone registration algorithm (Shan et al., 2 Sep 2025).

1. Problem setting and intended role

FPRAN is defined for a multimodal, cross-view prediction problem rather than for generic deformable registration. The broader task is to predict postoperative neck pain recovery in cervical spondylosis patients from preoperative multimodal imaging. The available data include axial CT bone-window, axial CT soft-tissue-window, axial MRI T2, and sagittal MRI T1/T2. Within this setting, the authors identify two distinct obstacles to multimodal fusion: CT and MRI have very different grayscale and texture statistics, and the modalities are often acquired in different anatomical planes, so corresponding structures do not line up spatially (Shan et al., 2 Sep 2025).

This division of difficulties determines the division of labor in the full model. Adaptive Bidirectional Pyramid Difference Convolution (ABPDC) is introduced to address inter-modal imaging differences, whereas FPRAN is introduced to address inter-view and inter-plane spatial discrepancies. The distinction is central: FPRAN is not the component responsible for modality-invariant texture extraction, and ABPDC is not the component responsible for cross-plane structural registration. The architecture therefore separates appearance mismatch from geometric mismatch instead of treating multimodal fusion as a single undifferentiated problem (Shan et al., 2 Sep 2025).

A common misconception is to regard FPRAN as a generic feature-fusion block. In the cited formulation, it is more specific than that. A fusion network that merely concatenates or combines features assumes at least rough semantic correspondence across spatial locations; in this application, that assumption breaks down because the same anatomy may appear under different orientations and with different structural detail. FPRAN is introduced precisely to reduce this mismatch before or during multimodal fusion (Shan et al., 2 Sep 2025).

2. Architectural definition inside the prediction framework

Within the full framework, two separate backbone networks first process CT and MRI inputs independently, with ordinary convolutions replaced by ABPDC blocks. These backbones produce two four-level feature pyramids,

$$F_{\mathrm{CT}}=\{f_{\mathrm{CT}^{i}}\}_{i=1}^{4},\qquad F_{\mathrm{MRI}}=\{f_{\mathrm{MRI}^{i}}\}_{i=1}^{4},$$

where $i$ indexes pyramid levels from fine to coarse according to the text (Shan et al., 2 Sep 2025).

FPRAN is defined as a Feature Pyramid Registration Auxiliary Network consisting of four Spatial Alignment Modules, one per pyramid level. Its defining principle is feature pyramid registration: registration is performed not at a single resolution, but progressively across the multiscale feature hierarchy generated by the backbones. The alignment starts at the coarsest level and is then refined toward finer levels. This follows the usual coarse-to-fine intuition that large structural discrepancies are easier to estimate on low-resolution, semantically abstract features, while higher-resolution levels provide finer corrective detail (Shan et al., 2 Sep 2025).

The network does not initially operate on raw images. It acts first on intermediate multiscale feature maps, and only the final deformation field is applied to the original CT image. Classification, meanwhile, is produced from the top-level features,

$$\mathbf{p} = \mathbf{W}\!\begin{bmatrix} f^{4}_{\mathrm{CT}} \ f^{4}_{\mathrm{MRI}} \end{bmatrix} + \mathbf{b}.$$

This means prediction and alignment are learned jointly. In this formulation, FPRAN is auxiliary in the sense that it is not the main prediction head; it is an alignment branch that shapes feature learning and regularizes the multimodal representation (Shan et al., 2 Sep 2025).

3. Coarse-to-fine registration mechanism

The deformation field is estimated recursively across the pyramid. The update rule is given as

$\psi_i = \begin{cases} \mathrm{SAM}(f_{\mathrm{CT}^{i}}, f_{\mathrm{MRI}^{i}}), & i=4,\[0.3em] \mathrm{SAM}(f_{\mathrm{CT}^{i}}\circ\psi_{i+1}, f_{\mathrm{MRI}^{i}}) + \psi_{i+1}, & i=1,2,3, \end{cases}$

where $\circ$ denotes STN-based warping. The intended interpretation is that level 4 produces an initial displacement $\psi_4$; level 3 warps CT features with $\psi_4$, predicts a residual correction, and adds it to the coarser estimate; the same pattern is repeated down to level 1, yielding the final deformation $\psi_1$ (Shan et al., 2 Sep 2025).

The coarsest-level SAM receives the most explicit description. Backbone features $f_{\mathrm{CT}^{4}}$ and $f_{\mathrm{MRI}^{4}}$ are first enhanced by 3D convolutions to produce learnable spatial representations,

$i$0

The exact kernel sizes, channel counts, normalization layers, and activations of these 3D convolutions are not specified. The first distinctive operation is an affine transformation that reorients CT features from the axial plane into the sagittal plane:

$i$1

where $i$2 is a base rotation matrix and $i$3 is a learnable refinement network of three convolutional layers. This step is intended to ensure that CT features are geometrically consistent with MRI features before deformable refinement. The exact form of $i$4, the affine parameterization, and the detailed structure of $i$5 are not reported (Shan et al., 2 Sep 2025).

After affine reorientation, the transformed CT feature $i$6 and MRI feature $i$7 are fused through a cross-attention mechanism to generate the coarse displacement field:

$i$8

The text does not provide the internal mathematical form of $i$9: there are no query-key-value equations, no attention-head counts, and no tensor-shape details. What is explicit is that the coarse displacement is produced by cross-attention fusion of transformed CT and MRI features (Shan et al., 2 Sep 2025).

At finer scales, the same pattern is applied recursively. The level-3 update is written explicitly as

$$\mathbf{p} = \mathbf{W}\!\begin{bmatrix} f^{4}_{\mathrm{CT}} \ f^{4}_{\mathrm{MRI}} \end{bmatrix} + \mathbf{b}.$$0

This refinement process is repeated down to the first level. The final deformation field is then applied to the original CT image,

$$\mathbf{p} = \mathbf{W}\!\begin{bmatrix} f^{4}_{\mathrm{CT}} \ f^{4}_{\mathrm{MRI}} \end{bmatrix} + \mathbf{b}.$$1

again through STN-based transformation. The warped image is not described as the direct classification input; its primary function is to support the alignment-related auxiliary objective (Shan et al., 2 Sep 2025).

4. Auxiliary objective, loss design, and optimization

FPRAN is called an auxiliary network because it adds an alignment task coupled to the main classification objective. The model overview states that $$\mathbf{p} = \mathbf{W}\!\begin{bmatrix} f^{4}_{\mathrm{CT}} \ f^{4}_{\mathrm{MRI}} \end{bmatrix} + \mathbf{b}.$$2, $$\mathbf{p} = \mathbf{W}\!\begin{bmatrix} f^{4}_{\mathrm{CT}} \ f^{4}_{\mathrm{MRI}} \end{bmatrix} + \mathbf{b}.$$3, and $$\mathbf{p} = \mathbf{W}\!\begin{bmatrix} f^{4}_{\mathrm{CT}} \ f^{4}_{\mathrm{MRI}} \end{bmatrix} + \mathbf{b}.$$4 are jointly used in the loss function, enabling simultaneous optimization of classification and cross-modal alignment. The total loss is

$$\mathbf{p} = \mathbf{W}\!\begin{bmatrix} f^{4}_{\mathrm{CT}} \ f^{4}_{\mathrm{MRI}} \end{bmatrix} + \mathbf{b}.$$5

The classification term is the standard cross-entropy

$$\mathbf{p} = \mathbf{W}\!\begin{bmatrix} f^{4}_{\mathrm{CT}} \ f^{4}_{\mathrm{MRI}} \end{bmatrix} + \mathbf{b}.$$6

with $$\mathbf{p} = \mathbf{W}\!\begin{bmatrix} f^{4}_{\mathrm{CT}} \ f^{4}_{\mathrm{MRI}} \end{bmatrix} + \mathbf{b}.$$7 the class label and $$\mathbf{p} = \mathbf{W}\!\begin{bmatrix} f^{4}_{\mathrm{CT}} \ f^{4}_{\mathrm{MRI}} \end{bmatrix} + \mathbf{b}.$$8 the predicted class probability for the two pain outcome classes (Shan et al., 2 Sep 2025).

The alignment term uses normalized cross-correlation between the MRI image and the warped CT image,

$$\mathbf{p} = \mathbf{W}\!\begin{bmatrix} f^{4}_{\mathrm{CT}} \ f^{4}_{\mathrm{MRI}} \end{bmatrix} + \mathbf{b}.$$9

where the notation indicates averaging over the spatial domain $\psi_i = \begin{cases} \mathrm{SAM}(f_{\mathrm{CT}^{i}}, f_{\mathrm{MRI}^{i}}), & i=4,\[0.3em] \mathrm{SAM}(f_{\mathrm{CT}^{i}}\circ\psi_{i+1}, f_{\mathrm{MRI}^{i}}) + \psi_{i+1}, & i=1,2,3, \end{cases}$0 with normalization factor $\psi_i = \begin{cases} \mathrm{SAM}(f_{\mathrm{CT}^{i}}, f_{\mathrm{MRI}^{i}}), & i=4,\[0.3em] \mathrm{SAM}(f_{\mathrm{CT}^{i}}\circ\psi_{i+1}, f_{\mathrm{MRI}^{i}}) + \psi_{i+1}, & i=1,2,3, \end{cases}$1, though $\psi_i = \begin{cases} \mathrm{SAM}(f_{\mathrm{CT}^{i}}, f_{\mathrm{MRI}^{i}}), & i=4,\[0.3em] \mathrm{SAM}(f_{\mathrm{CT}^{i}}\circ\psi_{i+1}, f_{\mathrm{MRI}^{i}}) + \psi_{i+1}, & i=1,2,3, \end{cases}$2 is not further defined. Deformation smoothness is encouraged through

$\psi_i = \begin{cases} \mathrm{SAM}(f_{\mathrm{CT}^{i}}, f_{\mathrm{MRI}^{i}}), & i=4,\[0.3em] \mathrm{SAM}(f_{\mathrm{CT}^{i}}\circ\psi_{i+1}, f_{\mathrm{MRI}^{i}}) + \psi_{i+1}, & i=1,2,3, \end{cases}$3

The accompanying description interprets this as penalizing gradients of the displacement field for smoothness, although the displayed equation uses $\psi_i = \begin{cases} \mathrm{SAM}(f_{\mathrm{CT}^{i}}, f_{\mathrm{MRI}^{i}}), & i=4,\[0.3em] \mathrm{SAM}(f_{\mathrm{CT}^{i}}\circ\psi_{i+1}, f_{\mathrm{MRI}^{i}}) + \psi_{i+1}, & i=1,2,3, \end{cases}$4 without explicitly writing them as partial derivatives (Shan et al., 2 Sep 2025).

The reported loss formulation has an important omission: no weighting coefficients are given between $\psi_i = \begin{cases} \mathrm{SAM}(f_{\mathrm{CT}^{i}}, f_{\mathrm{MRI}^{i}}), & i=4,\[0.3em] \mathrm{SAM}(f_{\mathrm{CT}^{i}}\circ\psi_{i+1}, f_{\mathrm{MRI}^{i}}) + \psi_{i+1}, & i=1,2,3, \end{cases}$5, $\psi_i = \begin{cases} \mathrm{SAM}(f_{\mathrm{CT}^{i}}, f_{\mathrm{MRI}^{i}}), & i=4,\[0.3em] \mathrm{SAM}(f_{\mathrm{CT}^{i}}\circ\psi_{i+1}, f_{\mathrm{MRI}^{i}}) + \psi_{i+1}, & i=1,2,3, \end{cases}$6, and $\psi_i = \begin{cases} \mathrm{SAM}(f_{\mathrm{CT}^{i}}, f_{\mathrm{MRI}^{i}}), & i=4,\[0.3em] \mathrm{SAM}(f_{\mathrm{CT}^{i}}\circ\psi_{i+1}, f_{\mathrm{MRI}^{i}}) + \psi_{i+1}, & i=1,2,3, \end{cases}$7. As written, the three terms are simply summed with equal weight. If weighting hyperparameters were used in practice, they are not reported. Likewise, no staged training schedule is described; the wording implies end-to-end joint optimization of prediction and alignment (Shan et al., 2 Sep 2025).

The implementation details that are reported are global to the whole model rather than FPRAN alone. The model is implemented with PyTorch and MONAI, optimized by AdamW, trained with batch size 16, initial learning rate $\psi_i = \begin{cases} \mathrm{SAM}(f_{\mathrm{CT}^{i}}, f_{\mathrm{MRI}^{i}}), & i=4,\[0.3em] \mathrm{SAM}(f_{\mathrm{CT}^{i}}\circ\psi_{i+1}, f_{\mathrm{MRI}^{i}}) + \psi_{i+1}, & i=1,2,3, \end{cases}$8, Step Decay with factor $\psi_i = \begin{cases} \mathrm{SAM}(f_{\mathrm{CT}^{i}}, f_{\mathrm{MRI}^{i}}), & i=4,\[0.3em] \mathrm{SAM}(f_{\mathrm{CT}^{i}}\circ\psi_{i+1}, f_{\mathrm{MRI}^{i}}) + \psi_{i+1}, & i=1,2,3, \end{cases}$9, and 100 epochs on two RTX 3090 GPUs. These settings matter because FPRAN is trained jointly with the rest of the network rather than as a separate pretraining stage (Shan et al., 2 Sep 2025).

5. Empirical evidence and interaction with ABPDC

The direct evidence for FPRAN comes from ablation on six multi-view modality combinations pairing sagittal MRI, either T1 or T2, with axial MRI T2, CT bone, or CT soft tissue. This setup directly reflects the network’s stated motivation: cross-view fusion is most difficult when sagittal MRI must be combined with axial CT or axial MRI. Looking only at the effect of FPRAN without ABPDC, the reported results are as follows (Shan et al., 2 Sep 2025).

Modality pair	Baseline $\circ$0 FPRAN only	Reported pattern
T2sag + T2tra	ACC 68.00 $\circ$1 70.00; AUC 68.11 $\circ$2 69.23	modest gain
T2sag + CTbone	ACC 66.00 $\circ$3 74.00; AUC 65.71 $\circ$4 78.53	strongest isolated effect
T2sag + CTtissue	ACC 62.00 $\circ$5 68.00; AUC 61.94 $\circ$6 63.62	clear gain
T1sag + T2tra	ACC 70.00 $\circ$7 74.00; AUC 72.76 $\circ$8 76.83	clear gain
T1sag + CTbone	ACC 76.00 $\circ$9 78.00; AUC 72.76 $\psi_4$0 75.63	small gain
T1sag + CTtissue	ACC 70.00 $\psi_4$1 74.00; AUC 71.96 $\psi_4$2 77.35	clear gain

These results support the authors’ claim that FPRAN consistently achieves superior performance, but the gains are not uniform across every metric. Some F1 scores improve only marginally, and some decrease slightly. The clearest isolated benefit appears on cross-view CT/MRI pairs, especially T2sag + CTbone, where FPRAN alone yields $\psi_4$3 accuracy and $\psi_4$4 AUC. That pattern is consistent with the architectural motivation: FPRAN is most valuable when spatial mismatch is severe, such as sagittal MRI paired with axial CT (Shan et al., 2 Sep 2025).

The ablation table also shows that FPRAN and ABPDC are complementary rather than redundant. For T1sag + T2tra, performance improves from 70.00 baseline to 78.00 with ABPDC alone, 74.00 with FPRAN alone, and 80.00 with both. For T1sag + CTbone, it goes from 76.00 baseline to 76.00 with ABPDC alone, 78.00 with FPRAN alone, and 82.00 with both. For T2sag + CTtissue, it rises from 62.00 baseline to 68.00 with either module alone and 70.00 with both. This establishes that FPRAN is not a replacement for ABPDC; the two modules target different failure modes and work synergistically when combined (Shan et al., 2 Sep 2025).

At the whole-model level, the best reported performance reaches 82% accuracy. On T1sag + T2tra, the proposed method achieves 80.0 ACC / 82.6 AUC / 80.7 F1, outperforming ResNet, ViT, EfficientNet, DenseNet, PBTC TransNet, and CoPAS. On T1sag + CTbone, it reaches 82.0 ACC / 80.3 AUC / 83.0 F1. These results are not isolated validations of FPRAN alone, but they indicate that the complete multimodal cross-view framework materially benefits from the alignment machinery (Shan et al., 2 Sep 2025).

Notably, the effectiveness evidence is numerical rather than visual. The work does not provide dedicated qualitative visualizations of deformation fields, correspondence maps, before-and-after alignment overlays, heatmaps, or FPRAN-specific feature visualizations. As a result, its empirical standing rests mainly on ablation and predictive performance rather than on direct qualitative assessment of registration behavior (Shan et al., 2 Sep 2025).

6. Relation to pyramid registration literature, misconceptions, and limitations

FPRAN belongs to a broader class of coarse-to-fine pyramid registration designs, but its placement is distinctive. It is not presented as a standalone classical image registration algorithm; instead, it is embedded inside a predictive network and trained jointly with classification. It also performs registration in feature space across a pyramid rather than only at image level. This places it near other multiscale registration frameworks that rely on progressive deformation refinement, feature-space warping, or auxiliary pyramid mechanisms, including the Cascaded Feature Warping Network (Zhang et al., 2021), Dual-PRNet (Kang et al., 2019), the shared auxiliary decoder and pyramid decoder framework of “Deformable Image Registration with Multi-scale Feature Fusion from Shared Encoder, Auxiliary and Pyramid Decoders” (Zhou et al., 2024), FF-PNet (Zhang et al., 8 May 2025), Pyramid Attention Network (Wang et al., 2024), MsMorph (Nan et al., 2024), PULPo (Siegert et al., 2024), and TCIP (Wu et al., 9 Oct 2025).

The point of contact with those systems is the feature-pyramid or pyramid-registration principle: multi-resolution representations are used to estimate and refine deformation progressively from coarse to fine. What distinguishes FPRAN in the cited multimodal neck-pain model is its explicit response to a known axial-to-sagittal geometric discrepancy through an initial affine orientation transform before deformable refinement, and its role as an auxiliary branch inside a multimodal classifier rather than a registration-only network. This suggests that FPRAN occupies a hybrid position between multimodal feature alignment and embedded deformable registration (Shan et al., 2 Sep 2025).

Several misconceptions are clarified by the source description. FPRAN is not the mechanism that handles CT/MRI appearance discrepancy; that role is assigned to ABPDC. It is not the main classification pathway; it is an auxiliary alignment branch. It is not shown to be a general-purpose registration model superior to established baselines, because no direct comparison against dedicated registration methods is reported. The article describing it explicitly cautions that one should be careful not to overstate novelty beyond what is shown (Shan et al., 2 Sep 2025).

The limitations of the presentation are substantial. Important implementation details are absent: the exact architecture of SAM, the cross-attention formulation, the dimensions and channels of pyramid levels, the affine parameterization, interpolation details of warping, and any loss weights. Computational overhead is not reported, nor are runtime or memory costs. The smoothness regularizer notation is slightly ambiguous. Because no visual alignment results are shown, failure cases and robustness in unusual anatomies cannot be directly assessed. These omissions do not negate the reported ablation gains, but they constrain reproducibility and make it difficult to isolate which parts of FPRAN drive performance most strongly (Shan et al., 2 Sep 2025).

In concise technical terms, FPRAN is best understood as a coarse-to-fine multiscale feature registration branch for cross-view multimodal prediction. It takes CT and MRI feature pyramids from ABPDC-based backbones, applies a coarse affine reorientation of CT features, estimates a displacement field through cross-attention at the coarsest level, refines that field through four pyramid-level SAMs using STN warping, and is trained jointly with classification using NCC similarity and deformation smoothness losses. Its contribution is most evident on sagittal-MRI/axial-CT combinations, where ordinary fusion is weakest and alignment matters most (Shan et al., 2 Sep 2025).