PaIR-Net: Pairwise Image Ranking Network

• PaIR-Net is a pairwise image ranking network designed to assess longitudinal changes by comparing image pairs without explicit registration.
• It employs a Siamese ResNet-18 backbone with feature subtraction and a linear scoring layer to quantify change and generate saliency maps.
• Training includes both supervised regression and self-supervised temporal ordering, achieving high correlation metrics and effective change localization in biomedical imaging.

Searching arXiv for the specified paper and closely related PaIR-Net variants to ground the article. {"queries":[{"query":"(Kim et al., 2023) Learning to Compare Longitudinal Images PaIRNet"},{"query":"PaIR-Net arXiv (Kim et al., 2023) Pairwise Image Ranking Network"},{"query":"PAIR-Net arXiv egocentric speaker detection (Wang et al., 2 Jun 2025)"},{"query":"Pair then Relation Pair-Net arXiv (Wang et al., 2023)"}]} PaIR-Net, short for Pairwise Image Ranking Network, is a deep learning model for longitudinal image comparison that was introduced in "Learning to Compare Longitudinal Images" (Kim et al., 2023). It is designed for settings in which a series of images from the same individuals are acquired at different time-points and the analytical objective is to detect meaningful temporal change while discounting nuisance variation. The method departs from the classical pipeline of pre-processing followed by statistical analysis by learning a pairwise comparison function directly from image pairs, either with supervision or in a self-supervised temporal-ordering setup. In the formulation reported in the paper, PaIRNet is used on biomedical image sequences and is presented as a mechanism for localizing and quantifying longitudinal changes without explicit registration or intensity normalization (Kim et al., 2023).

1. Problem setting and conceptual basis

Longitudinal studies are described as a setting in which a series of images from the same set of individuals are acquired at different time-points, with applications to the study and characterization of temporal dynamics in biomedical data (Kim et al., 2023). The classical approach normalizes nuisance variations such as image orientation or contrast differences via pre-processing, and then performs statistical analysis to detect changes of interest at the individual or population level. The paper identifies two difficulties in that pipeline: normalizing nuisance variation can be hard when there are many idiosyncratic changes, and downstream statistical modeling can be limiting.

PaIRNet addresses this by learning to compare image pairs directly. In the self-supervised setup, the model is trained to temporally order images, which requires learning to recognize time-irreversible changes. The central assumption used to justify nuisance suppression is that nuisance factors such as pose, contrast, brightness, and translation are statistically independent of chronological time across the population. Under that assumption, subtracting feature representations of two images tends to cancel out features that are constant or random with respect to time and to amplify features that consistently correlate with time ordering.

This suggests that PaIRNet is best suited to trajectories with monotonic, irreversible temporal progression, which is the regime used throughout the reported experiments. A plausible implication is that the method is not framed as a generic change detector for arbitrary reversible dynamics, but rather as a pairwise ranking model for clinically or biologically directional change.

2. Network architecture and scoring function

PaIRNet uses a ResNet-18 convolutional backbone consisting of all convolutional and residual blocks up to the last global-pooling layer (Kim et al., 2023). Two input images, $I_1$ and $I_2$, from the same subject at two time-points are passed through the same feature extractor $f(\cdot)$ in a classical Siamese setup, yielding

$$f(I_1)\in\mathbb R^{512}, \qquad f(I_2)\in\mathbb R^{512}.$$

The pairwise representation is the feature difference

$d = f(I_1)-f(I_2)\in\mathbb R^{512}.$

A single fully-connected linear layer with weight vector $w\in\mathbb R^{512}$ and no bias produces the scalar score

$R(I_1,I_2)=w^\top d.$

The model therefore has a deliberately minimal comparison head: rather than concatenating features or using a deeper relation module, it uses direct subtraction followed by a linear ranking layer. In classification mode, $R$ is passed through a sigmoid $\sigma(\cdot)$ to give a probability that $I_1$ comes after $I_2$0 in time. In regression mode, $I_2$1 predicts the magnitude of change in a clinical variable $I_2$2, such as tumor size difference or phase difference.

The input-output contract is correspondingly simple. The input consists of two 2D images, with examples in the paper including mid-axial MRI slices, synthetic objects, and time-lapse embryos. The output is either a scalar regression estimate of longitudinal change magnitude or a binary temporal-ordering probability. This architectural economy is central to the paper’s positioning of PaIRNet as a simple machine learning-based approach rather than a reconstruction-heavy or registration-dependent framework.

3. Training modes and optimization

PaIRNet is trained in two distinct ways: supervised regression and self-supervised temporal ordering (Kim et al., 2023).

In the supervised setting, the ground truth for a pair $I_2$3 is the numerical difference

$I_2$4

The objective is the mean-squared error

$I_2$5

This mode uses known clinical or temporal targets and learns the ranking score as a direct predictor of inter-time-point change magnitude.

In the self-supervised setting, if $I_2$6 denotes the acquisition time of $I_2$7, a binary label is defined by

$I_2$8

Training then minimizes binary cross-entropy on the sigmoid of the score:

$I_2$9

Here the supervision signal is only temporal ordering, not a measured clinical progression variable.

The reported optimization details are specific. The optimizer is Adam with $f(\cdot)$0 and $f(\cdot)$1. The batch size is 64 ordered pairs. The learning rate is grid-searched over $f(\cdot)$2 and selected by lowest validation loss; typical best rates are $f(\cdot)$3 or $f(\cdot)$4 for supervised PaIRNet and $f(\cdot)$5 or $f(\cdot)$6 for self-supervised PaIRNet. Early stopping halts training if validation loss does not improve for 5 consecutive epochs. No additional regularizer beyond standard Adam weight decay, if used, is required. Data augmentation is applied independently to each image and includes random rotation uniform in $f(\cdot)$7, random translation such as $f(\cdot)$8 px, and random brightness and contrast jitter with factors in $f(\cdot)$9.

4. Nuisance variation and change localization

A defining feature of PaIRNet is that it does not require explicit registration or intensity normalization (Kim et al., 2023). The paper attributes this to two coupled mechanisms. First, the subtraction $$f(I_1)\in\mathbb R^{512}, \qquad f(I_2)\in\mathbb R^{512}.$$0 tends to cancel out features that are constant or random with respect to time. Second, exposure to random augmentations during training encourages invariance to nuisance transforms. In the examples discussed in the paper, nuisance factors include pose, contrast, brightness, and translation.

PaIRNet is also designed to localize where change occurs. The paper introduces a modified Class Activation Map for this purpose. Let $$f(I_1)\in\mathbb R^{512}, \qquad f(I_2)\in\mathbb R^{512}.$$1 be the $$f(I_1)\in\mathbb R^{512}, \qquad f(I_2)\in\mathbb R^{512}.$$2th channel activation at spatial location $$f(I_1)\in\mathbb R^{512}, \qquad f(I_2)\in\mathbb R^{512}.$$3 in the final convolutional layer for image $$f(I_1)\in\mathbb R^{512}, \qquad f(I_2)\in\mathbb R^{512}.$$4. The channel weight for change is defined as the absolute contribution of that channel to the pairwise difference score:

$$f(I_1)\in\mathbb R^{512}, \qquad f(I_2)\in\mathbb R^{512}.$$5

where $$f(I_1)\in\mathbb R^{512}, \qquad f(I_2)\in\mathbb R^{512}.$$6 is the $$f(I_1)\in\mathbb R^{512}, \qquad f(I_2)\in\mathbb R^{512}.$$7th feature and $$f(I_1)\in\mathbb R^{512}, \qquad f(I_2)\in\mathbb R^{512}.$$8 is the corresponding fully connected weight. The change-saliency map is then

$$f(I_1)\in\mathbb R^{512}, \qquad f(I_2)\in\mathbb R^{512}.$$9

This map can be visualized over an up-sampled image to localize evolving structures. The qualitative examples are specific: on Starmen, saliency maps ring the raising arm and are invariant to subject-specific pose or random rotation; on Tumor, the method highlights disc region growth while ignoring surrounding brain anatomy shifts; on Embryo, it focuses on cell-mass shape changes rather than dish background drift; on Aging Brain, it highlights enlarging ventricles and hippocampal atrophy zones. The paper also notes an important qualification for localization metrics: Dice Similarity on Tumor is low overall because CAM highlights discriminative subregions rather than full lesion extent.

5. Experimental configuration and empirical results

The evaluation uses four datasets, all described as having monotonic, irreversible temporal progression (Kim et al., 2023).

Dataset	Data summary	Ground-truth signal
Starmen	1 000 subjects × 10 time-points of “arm raising”; Train/Val/Test = 400/100/500 subjects	$d = f(I_1)-f(I_2)\in\mathbb R^{512}.$0, an affine reparam. of frame index
Tumor	72 subjects, 3–5 slices each, total 292 images; tumor size $d = f(I_1)-f(I_2)\in\mathbb R^{512}.$1 px	Tumor disc growth size
Embryo	698 embryos, 16 annotated phases; total 7 784 frames; avg 12.2 phases/embryo	Phase progression
Aging Brain	272 healthy subjects, total 754 $d = f(I_1)-f(I_2)\in\mathbb R^{512}.$2 slices; avg 2.77 time-points each; age range 42–86 yrs	Age difference

The reported evaluation metrics are Pearson correlation $d = f(I_1)-f(I_2)\in\mathbb R^{512}.$3 between predicted $d = f(I_1)-f(I_2)\in\mathbb R^{512}.$4 and ground-truth $d = f(I_1)-f(I_2)\in\mathbb R^{512}.$5, binary AUC for ordering accuracy in the self-supervised setting, and Dice Similarity for localization on Tumor. Pearson correlation is defined in the paper as

$d = f(I_1)-f(I_2)\in\mathbb R^{512}.$6

The main quantitative findings are consistent across the four datasets. Supervised PaIRNet achieves $d = f(I_1)-f(I_2)\in\mathbb R^{512}.$7–$d = f(I_1)-f(I_2)\in\mathbb R^{512}.$8 on Starmen and Tumor and outperforms cross-sectional regression especially in high inter-subject-variation tasks such as Brain aging. Self-supervised PaIRNet, trained only with temporal ordering, attains $d = f(I_1)-f(I_2)\in\mathbb R^{512}.$9–$w\in\mathbb R^{512}$0 and $w\in\mathbb R^{512}$1 on all datasets, nearly matching the supervised version. The baselines LSSL-CL and LVAE have much lower correlations, approximately $w\in\mathbb R^{512}$2–$w\in\mathbb R^{512}$3, and are described as unable to handle raw nuisance variations. For localization on Tumor, Dice Similarity is approximately $w\in\mathbb R^{512}$4–$w\in\mathbb R^{512}$5, but PaIRNet’s Dice exceeds cross-sectional regression across thresholds.

These results support two specific claims made by the paper: PaIRNet can quantify longitudinal change with high correlation to true $w\in\mathbb R^{512}$6, and it can localize discriminative regions associated with progression. A plausible implication is that temporal ordering alone can be a strong supervisory signal when the underlying process is irreversible.

6. Interpretation, scope, and nomenclature

The paper’s conclusion presents PaIRNet as a simple way to learn longitudinal change directly from pairs of images, without preprocessing or handcrafted registration (Kim et al., 2023). Its empirical behavior is characterized in terms of two outputs: quantification of progression through the ranking score and localization of evolving structures through the weighted CAM. The method is therefore positioned between classical longitudinal image analysis and more complex self-supervised reconstruction-based methods.

Its scope is also bounded by its assumptions. The datasets are all monotonic and irreversible, and the self-supervised formulation relies on learning time-irreversible changes from temporal ordering. The nuisance-handling argument depends on nuisance factors being statistically independent of chronological time across the population. This suggests that deployment outside that regime would require checking whether those assumptions continue to hold.

A common source of confusion is nomenclature. The label “PaIR-Net,” “PAIR-Net,” “Pair-Net,” and “PairNet” is used by several unrelated models in the arXiv literature. These include an audio-visual active speaker detection model that integrates Whisper and AV-HuBERT for Ego4D ASD (Wang et al., 2 Jun 2025), a panoptic scene graph generation framework centered on a Pair Proposal Network (Wang et al., 2023), shallow pairwise neural networks on partitioned subspaces for regression and on-device applications (Zhang, 2020, Zhang, 2020), a trainable pairwise image recognition network based on binary class-pair blocks (Geidarov, 29 May 2025), and an instance- and pair-aware dynamic network for re-identification (Jiao et al., 2021). In strict usage, however, PaIR-Net in the context of longitudinal biomedical imaging refers to the Pairwise Image Ranking Network introduced in "Learning to Compare Longitudinal Images" (Kim et al., 2023).