Deformable Refinement Network (DeRefNet)

• The paper presents DeRefNet as a model-driven framework that recovers sub-pixel target positions in mixed infrared images.
• It integrates sparsity-driven feature extraction, positional encoding, and temporal deformable alignment to enhance multi-frame unmixing.
• Empirical results on SeqCSIST show improved CSO-mAP and real-time performance, validating its design over traditional detectors.

Deformable Refinement Network (DeRefNet) is a model-driven deep learning framework proposed for Sequential Closely-Spaced Infrared Small Target Unmixing, a setting in which multiple distant infrared small targets are visually collapsed into a single blurred or mixed spot because of optical diffraction, detector resolution limits, and overlapping point spread functions. DeRefNet takes a short sequence of low-resolution target-centered patches and predicts a high-resolution response map for the middle frame, with the explicit aim of recovering target multiplicity and sub-pixel target locations. Its architecture combines a sparsity-driven feature extraction module, a positional / temporal encoding module, and a Temporal Deformable Feature Alignment (TDFA) module, making it the paper’s mechanism for exploiting multi-frame information in a task where ordinary infrared small target detection is insufficient (Zhai et al., 13 Jul 2025). In a broader methodological sense, its deformable alignment stage belongs to the lineage established by deformable convolutional networks, which replace fixed geometric sampling with learned offsets and bilinear sampling to improve spatial adaptation (Dai et al., 2017).

1. Problem domain and definitional scope

DeRefNet was introduced together with the task of Sequential Closely-Spaced Infrared Small Target Unmixing. In this setting, CSIST denotes Closely-Spaced Infrared Small Target groups: several distant infrared objects whose spatial separation is below the resolving capability of the imaging system, so that they appear in the image as a single mixed spot rather than as separable objects. The task is explicitly distinguished from ordinary infrared small target detection, because traditional IRSTD assumes a one-to-one correspondence between a visible spot and a real target, whereas CSIST breaks that assumption and requires the system to determine how many targets are present and recover their individual positions at sub-pixel precision (Zhai et al., 13 Jul 2025).

The model is defined on a short temporal window. Given the low-resolution sequence

$\{L_{t-N}, \ldots, L_{t+N}\},$

where the middle frame is $L_t$, DeRefNet predicts a high-resolution center-frame response

$$H_t = f_{DeRefNet}(L_{t-N}, \ldots, L_{t+N}). \tag{1}$$

The output $H_t$ lies in $\mathbb{R}^{C \times cH \times cW}$, where $c$ is the unmixing ratio. In the reported experiments, the practical configuration is $11 \times 11$ input patches and $33 \times 33$ outputs, so $c=3$ (Zhai et al., 13 Jul 2025).

A common misconception is to treat DeRefNet as a conventional detector. The paper instead describes it as a downstream post-processing model that operates after a detector has already cropped a target patch. It does not define separate objectness or box-regression heads; rather, it predicts a high-resolution unmixing image or target response map for the center frame, from which sub-pixel target positions are subsequently localized (Zhai et al., 13 Jul 2025).

2. Physical imaging model and model-driven initialization

A central property of DeRefNet is that it is not introduced as a purely data-driven feature stack. Its front end is tied to an explicit optical image-formation model. For a sub-pixel target at $(x_t,y_t)$ with brightness $L_t$0, the paper uses a Gaussian approximation of the point spread function:

$L_t$1

The sensor response integrates the PSF over a pixel support:

$L_t$2

From this construction the paper forms a sensing matrix $L_t$3, yielding the degradation model

$L_t$4

Here $L_t$5 is the latent high-resolution target distribution and $L_t$6 is the observed low-resolution mixed image (Zhai et al., 13 Jul 2025).

DeRefNet begins by computing a least-squares initialization operator:

$L_t$7

with

$L_t$8

Each frame is then lifted to an initial high-resolution estimate

$L_t$9

This initialization is one of the paper’s defining model-driven elements: the network does not start from arbitrary latent features, but from a physically motivated inverse mapping derived from the sequential sensing model (Zhai et al., 13 Jul 2025).

This suggests that DeRefNet is best understood not as generic video super-resolution, but as an inverse imaging system specialized for mixed infrared target groups. The initialization step is intended to regularize the later deformable refinement by starting from a sequence of coarse high-resolution hypotheses that already encode the forward operator $$H_t = f_{DeRefNet}(L_{t-N}, \ldots, L_{t+N}). \tag{1}$$0.

3. Architectural composition

The paper identifies three principal ingredients in DeRefNet: a sparsity-driven feature extraction module, a positional / temporal encoding module, and the Temporal Deformable Feature Alignment module. At the system level, five consecutive low-resolution target patches are first converted to $$H_t = f_{DeRefNet}(L_{t-N}, \ldots, L_{t+N}). \tag{1}$$1 by $$H_t = f_{DeRefNet}(L_{t-N}, \ldots, L_{t+N}). \tag{1}$$2, then processed by a deep-unfolding front end, then modulated by temporal encoding, then aligned and aggregated across time, and finally fused by a tail network with residual blocks to produce the center-frame high-resolution response $$H_t = f_{DeRefNet}(L_{t-N}, \ldots, L_{t+N}). \tag{1}$$3 (Zhai et al., 13 Jul 2025).

The feature extractor is formulated by unfolding an ISTA-like sparse recovery procedure. For the sequence of initialized inputs, the paper writes

$$H_t = f_{DeRefNet}(L_{t-N}, \ldots, L_{t+N}). \tag{1}$$4

At iteration $$H_t = f_{DeRefNet}(L_{t-N}, \ldots, L_{t+N}). \tag{1}$$5, each frame update begins with

$$H_t = f_{DeRefNet}(L_{t-N}, \ldots, L_{t+N}). \tag{1}$$6

followed by the sparse reconstruction problem

$$H_t = f_{DeRefNet}(L_{t-N}, \ldots, L_{t+N}). \tag{1}$$7

The paper replaces the hand-crafted sparsifying transform $$H_t = f_{DeRefNet}(L_{t-N}, \ldots, L_{t+N}). \tag{1}$$8 with a learnable nonlinear transform $$H_t = f_{DeRefNet}(L_{t-N}, \ldots, L_{t+N}). \tag{1}$$9, producing

$H_t$0

and, using the cited approximation,

$H_t$1

reformulates the update as

$H_t$2

The corresponding shrinkage step is

$H_t$3

hence

$H_t$4

In learnable stage-specific form,

$H_t$5

The paper argues that this unfolding-based design is preferable to generic ResNet-like backbones for CSIST because the latent high-resolution target map is naturally sparse (Zhai et al., 13 Jul 2025).

Temporal information is injected after feature extraction through

$H_t$6

followed by multiplicative modulation

$H_t$7

A 2D convolution then expands channels to produce the center-frame feature $H_t$8 and the reference features $H_t$9 (Zhai et al., 13 Jul 2025).

4. Temporal Deformable Feature Alignment

The Temporal Deformable Feature Alignment (TDFA) module is the part of DeRefNet that gives the architecture its name. It is designed to align each neighboring-frame feature with the middle-frame feature without explicit optical flow. For each reference frame $\mathbb{R}^{C \times cH \times cW}$0, the module first forms an attention-guided aggregate:

$\mathbb{R}^{C \times cH \times cW}$1

The paper describes this selective-attention block verbally: both features are reduced by convolution, concatenated, processed by max-pooled and average-pooled descriptors, activated by Sigmoid, and then reweighted so that the center frame receives average-pooled weights while the reference frame receives max-pooled weights before concatenation (Zhai et al., 13 Jul 2025).

From the aggregated feature, a convolution predicts deformable sampling parameters:

$\mathbb{R}^{C \times cH \times cW}$2

where $\mathbb{R}^{C \times cH \times cW}$3. The aligned reference feature is then computed by bilinear interpolation-based deformable sampling:

$\mathbb{R}^{C \times cH \times cW}$4

and, at each spatial location $\mathbb{R}^{C \times cH \times cW}$5,

$\mathbb{R}^{C \times cH \times cW}$6

Here $\mathbb{R}^{C \times cH \times cW}$7 is the regular grid offset, $\mathbb{R}^{C \times cH \times cW}$8 is the learned dynamic offset, and $\mathbb{R}^{C \times cH \times cW}$9 is the sampling weight (Zhai et al., 13 Jul 2025).

This mechanism follows the broader deformable-convolution principle in which fixed geometric sampling is replaced by learned, input-conditioned offsets and bilinear interpolation, a design originally introduced to overcome the limitations of CNN modules with fixed geometric structures (Dai et al., 2017). In DeRefNet, the same idea is specialized for sequential CSIST unmixing: instead of assuming that neighboring-frame evidence can be fused on a rigid spatial grid, TDFA adaptively resamples each reference feature map so that the subtle sub-pixel changes across frames become more usable for center-frame reconstruction (Zhai et al., 13 Jul 2025).

The paper’s ablations compare deformable alignment with optical flow and report 50.55 CSO-mAP for “Deep unfolding + optical flow” and 50.67 for “Deep unfolding + deformable alignment,” supporting the claim that deformable alignment is better suited to the subtle, ambiguous inter-frame displacements in CSIST (Zhai et al., 13 Jul 2025).

5. Training objective, SeqCSIST dataset, and evaluation protocol

DeRefNet is trained with three losses. The constraint loss enforces approximate invertibility of the learned transform pair:

$c$0

The alignment loss penalizes discrepancy between aligned reference features and the temporally encoded center feature:

$c$1

The regression loss supervises the final reconstruction:

$c$2

These are combined as

$c$3

with

$c$4

The paper reports that this weighting yields the best CSO-mAP in its loss-weight ablation (Zhai et al., 13 Jul 2025).

The accompanying benchmark, SeqCSIST, contains 5,000 trajectories, 100,000 total frames, and 20 frames per trajectory. Every 5 consecutive frames form one input sequence, and each trajectory yields 16 sequences through sliding windows. The split is 70% training (3500 trajectories), 15% validation (750 trajectories), and 15% testing (750 trajectories). Low-resolution image size is $c$5, output size is $c$6, and the target count per image varies randomly between 2, 3, and 4. Target intensities lie in $c$7 (Zhai et al., 13 Jul 2025).

Training uses Adam in MMEngine with learning rate $c$8, batch size 20, and five consecutive frames per sample. The paper also reports that all spatial feature extraction uses 32-channel convolution and the tail network contains 5 residual blocks. It does not report the number of epochs, weight decay, or a specific learning-rate schedule in the provided summary (Zhai et al., 13 Jul 2025).

Evaluation is centered on CSO-mAP together with thresholded AP values such as AP$c$9, AP$11 \times 11$0, AP$11 \times 11$1, AP$11 \times 11$2, and AP$11 \times 11$3, reflecting the sub-pixel localization nature of the task (Zhai et al., 13 Jul 2025).

6. Empirical performance, interpretation, and relation to adjacent literature

On SeqCSIST, DeRefNet reports CSO-mAP = 51.55, with AP$11 \times 11$4, AP$11 \times 11$5, AP$11 \times 11$6, AP$11 \times 11$7, and AP$11 \times 11$8. The abstract states that the method improves mAP by 5.3\% over prior methods. The benchmark summary in the provided details lists ISTA-Net+ at 51.02 CSO-mAP as the strongest prior entry shown there. This suggests that the paper’s headline 5.3% claim may be using a baseline definition or aggregation not identical to the specific excerpted table, although the reported result remains the best in the benchmark summary presented here (Zhai et al., 13 Jul 2025).

The ablations isolate the major components. Replacing standard residual blocks with the deep unfolding extractor improves CSO-mAP from 47.96 to 50.27. Adding deformable alignment gives 50.67, and adding the time encoder raises this to 51.39. A dynamic deformable alignment variant, labeled DDA, reaches 51.09 CSO-mAP with 0.28M parameters and 6.14G FLOPs, compared with 50.67, 0.23M parameters, and 6.44G FLOPs for the corresponding DA setting. The paper also reports robustness figures of 47.48 mAP on a hybrid synthetic-real background dataset, 47.23 mAP under Gaussian noise up to $11 \times 11$9, and 49.79 mAP when target count is increased to the denser 2–8 setting. Runtime is reported as 367 FPS, despite 15.70G FLOPs and a parameter count of 0.89M (Zhai et al., 13 Jul 2025).

DeRefNet should also be situated within a broader family of DeRefNet-like methods that do not use the exact name. The deformable refinement component in TSAIN aligns and extracts relevant content from input frames via deformable convolution in electron microscopic interpolation (Wang et al., 2021). PDWN performs coarse-to-fine deformable offset refinement for video interpolation (Chen et al., 2021). DFAR combines deformable temporal alignment with a feature refinement module for moving infrared dim-small target detection (Luo et al., 2024). RRN recursively refines deformation vector fields across scales in unsupervised lung CT registration (He et al., 2021). These systems indicate that “deformable refinement” is a wider architectural pattern spanning interpolation, detection, and registration, whereas DeRefNet in the strict sense refers specifically to the sequential CSIST unmixing model introduced in 2025 (Zhai et al., 13 Jul 2025).

A final distinction is terminological. Earlier work established the underlying deformable sampling machinery and many later systems used deformable refinement ideas, but the exact name “Deformable Refinement Network (DeRefNet)” is the designation used in the SeqCSIST paper itself. In that paper, the term refers specifically to a model-driven, multi-frame unmixing architecture whose defining contribution is the coupling of sparse unfolding with Temporal Deformable Feature Alignment for sub-pixel recovery of latent closely spaced infrared targets (Zhai et al., 13 Jul 2025).