---
title: Deformable Refinement Network (DeRefNet)
url: https://www.emergentmind.com/topics/deformable-refinement-network-derefnet
type: topic
---

# Deformable Refinement Network (DeRefNet)

Deformable Refinement Network (DeRefNet) is a model-driven deep learning framework proposed for **Sequential Closely-Spaced Infrared Small Target Unmixing**, a setting in which multiple distant infrared small targets are visually collapsed into a single blurred or mixed spot because of optical diffraction, detector resolution limits, and overlapping point spread functions. DeRefNet takes a short sequence of low-resolution target-centered patches and predicts a high-resolution response map for the middle frame, with the explicit aim of recovering target multiplicity and **sub-pixel** target locations. Its architecture combines a **sparsity-driven feature extraction module**, a **positional / temporal encoding module**, and a **Temporal Deformable Feature Alignment (TDFA) module**, making it the paper’s mechanism for exploiting multi-frame information in a task where ordinary infrared small target detection is insufficient [2507.09556]. In a broader methodological sense, its deformable alignment stage belongs to the lineage established by deformable convolutional networks, which replace fixed geometric sampling with learned offsets and bilinear sampling to improve spatial adaptation [1703.06211].

## 1. Problem domain and definitional scope

DeRefNet was introduced together with the task of **Sequential Closely-Spaced Infrared Small Target Unmixing**. In this setting, **CSIST** denotes **Closely-Spaced Infrared Small Target** groups: several distant infrared objects whose spatial separation is below the resolving capability of the imaging system, so that they appear in the image as a single **mixed spot** rather than as separable objects. The task is explicitly distinguished from ordinary infrared small target detection, because traditional IRSTD assumes a **one-to-one correspondence** between a visible spot and a real target, whereas CSIST breaks that assumption and requires the system to determine **how many** targets are present and recover their **individual positions** at **sub-pixel precision** [2507.09556].

The model is defined on a short temporal window. Given the low-resolution sequence
$$
\{L_{t-N}, \ldots, L_{t+N}\},
$$
where the middle frame is \(L_t\), DeRefNet predicts a high-resolution center-frame response
$$
H_t = f_{DeRefNet}(L_{t-N}, \ldots, L_{t+N}).
\tag{1}
$$
The output \(H_t\) lies in \(\mathbb{R}^{C \times cH \times cW}\), where \(c\) is the unmixing ratio. In the reported experiments, the practical configuration is \(11 \times 11\) input patches and \(33 \times 33\) outputs, so \(c=3\) [2507.09556].

A common misconception is to treat DeRefNet as a conventional detector. The paper instead describes it as a **downstream post-processing model** that operates after a detector has already cropped a target patch. It does not define separate objectness or box-regression heads; rather, it predicts a **high-resolution unmixing image** or **target response map** for the center frame, from which sub-pixel target positions are subsequently localized [2507.09556].

## 2. Physical imaging model and model-driven initialization

A central property of DeRefNet is that it is not introduced as a purely data-driven feature stack. Its front end is tied to an explicit optical image-formation model. For a sub-pixel target at \((x_t,y_t)\) with brightness \(a_i\), the paper uses a Gaussian approximation of the point spread function:
$$
\text{PSF}(x, y; x_t, y_t) = a_i \cdot \frac{1}{2\pi\sigma^2} \exp\left( -\frac{(x - x_t)^2 + (y - y_t)^2}{2\sigma^2} \right).
\tag{2}
$$
The sensor response integrates the PSF over a pixel support:
$$
r(x, y) = \iint_{\text{pixel}} \text{PSF}(u, v; x_t, y_t) \, du \, dv.
\tag{3}
$$
From this construction the paper forms a sensing matrix \(\Phi\), yielding the degradation model
$$
L = \Phi H_{\text{GT}}.
\tag{4}
$$
Here \(H_{\text{GT}}\) is the latent high-resolution target distribution and \(L\) is the observed low-resolution mixed image [2507.09556].

DeRefNet begins by computing a least-squares initialization operator:
$$
Q_{\text{init}} = \arg\min_Q \|QL - H\|^2 = L^T (LL^T)^{-1} H,
\tag{5}
$$
with
$$
L = [L_{t-N}, \ldots, L_{t+N}], \quad H = [H_{t-N}, \ldots, H_{t+N}].
$$
Each frame is then lifted to an initial high-resolution estimate
$$
H_i^{(0)} = Q_{\text{init}} L_i.
\tag{6}
$$
This initialization is one of the paper’s defining model-driven elements: the network does not start from arbitrary latent features, but from a physically motivated inverse mapping derived from the sequential sensing model [2507.09556].

This suggests that DeRefNet is best understood not as generic video super-resolution, but as an inverse imaging system specialized for mixed infrared target groups. The initialization step is intended to regularize the later deformable refinement by starting from a sequence of coarse high-resolution hypotheses that already encode the forward operator \(\Phi\).

## 3. Architectural composition

The paper identifies three principal ingredients in DeRefNet: a **sparsity-driven feature extraction module**, a **positional / temporal encoding module**, and the **Temporal Deformable Feature Alignment** module. At the system level, five consecutive low-resolution target patches are first converted to \(H_i^{(0)}\) by \(Q_{\text{init}}\), then processed by a deep-unfolding front end, then modulated by temporal encoding, then aligned and aggregated across time, and finally fused by a tail network with residual blocks to produce the center-frame high-resolution response \(H_t\) [2507.09556].

The feature extractor is formulated by unfolding an ISTA-like sparse recovery procedure. For the sequence of initialized inputs, the paper writes
$$
H_{t-N}^{(k)}, \ldots, H_{t+N}^{(k)} = f_{FE}(H_{t-N}^{(0)}, \ldots, H_{t+N}^{(0)}).
\tag{7}
$$
At iteration \(k\), each frame update begins with
$$
R_i^{(k)} = H_i^{(k-1)} - \rho \Phi^T (\Phi H_i^{(k-1)} - L_i),
\tag{8}
$$
followed by the sparse reconstruction problem
$$
H_i^{(k)} = \arg\min_{H_{\text{GT}i}} \frac{1}{2} \|H_{\text{GT}i} - R_i^{(k)}\|_2^2 + \lambda \|\Psi H_{\text{GT}i}\|_1.
\tag{9}
$$
The paper replaces the hand-crafted sparsifying transform \(\Psi\) with a learnable nonlinear transform \(G(\cdot)\), producing
$$
H_i^{(k)} = \arg\min_{H_{\text{GT}i}} \frac{1}{2} \|H_{\text{GT}i} - R_i^{(k)}\|_2^2 + \lambda \|G(H_{\text{GT}i})\|_1,
\tag{10}
$$
and, using the cited approximation,
$$
\|G(H_{\text{GT}i}) - G(R_i^{(k)})\|_2^2 \approx \alpha \|H_{\text{GT}i} - R_i^{(k)}\|_2^2,
\tag{11}
$$
reformulates the update as
$$
H_i^{(k)} = \arg\min_{H_{\text{GT}i}} \frac{1}{2} \|G(H_{\text{GT}i}) - G(R_i^{(k)})\|_2^2 + \theta \|G(H_{\text{GT}i})\|_1.
\tag{12}
$$
The corresponding shrinkage step is
$$
G(H_i^{(k)}) = \text{soft}(G(R_i^{(k)}), \theta),
\tag{13}
$$
hence
$$
H_i^{(k)} = G^{-1}(\text{soft}(G(R_i^{(k)}), \theta)).
\tag{14}
$$
In learnable stage-specific form,
$$
H_{i}^{(k)} = \tilde{G}^{(k)}\left(\text{soft}\left(G^{(k)}(R_i^{(k)}), \theta^{(k)}\right)\right).
\tag{15}
$$
The paper argues that this unfolding-based design is preferable to generic ResNet-like backbones for CSIST because the latent high-resolution target map is naturally sparse [2507.09556].

Temporal information is injected after feature extraction through
$$
T = Sigmoid(MLP(Encoder(t))),
\tag{16}
$$
followed by multiplicative modulation
$$
H^{'} = H^{(k)} T.
\tag{17}
$$
A 2D convolution then expands channels to produce the center-frame feature \(F_t\) and the reference features \(F_i\) [2507.09556].

## 4. Temporal Deformable Feature Alignment

The **Temporal Deformable Feature Alignment (TDFA)** module is the part of DeRefNet that gives the architecture its name. It is designed to align each neighboring-frame feature with the middle-frame feature without explicit optical flow. For each reference frame \(i \neq t\), the module first forms an attention-guided aggregate:
$$
F_{aggi} = SelectiveAttention(F_i, F_t).
\tag{18}
$$
The paper describes this selective-attention block verbally: both features are reduced by convolution, concatenated, processed by max-pooled and average-pooled descriptors, activated by Sigmoid, and then reweighted so that the center frame receives average-pooled weights while the reference frame receives max-pooled weights before concatenation [2507.09556].

From the aggregated feature, a convolution predicts deformable sampling parameters:
$$
\eta = conv(F_{aggi}),
\tag{19}
$$
where \(\eta = \{\Delta p_k\}\). The aligned reference feature is then computed by bilinear interpolation-based deformable sampling:
$$
H_{i}^{''} = \phi(F_i, \eta),
\tag{20}
$$
and, at each spatial location \(p\),
$$
H_{i}^{''}(p) = \sum_{k=1}^{K^2} \omega_k \cdot F_i(p + p_k + \Delta p_k).
\tag{21}
$$
Here \(p_k\) is the regular grid offset, \(\Delta p_k\) is the learned dynamic offset, and \(\omega_k\) is the sampling weight [2507.09556].

This mechanism follows the broader deformable-convolution principle in which fixed geometric sampling is replaced by learned, input-conditioned offsets and bilinear interpolation, a design originally introduced to overcome the limitations of CNN modules with **fixed geometric structures** [1703.06211]. In DeRefNet, the same idea is specialized for sequential CSIST unmixing: instead of assuming that neighboring-frame evidence can be fused on a rigid spatial grid, TDFA adaptively resamples each reference feature map so that the subtle sub-pixel changes across frames become more usable for center-frame reconstruction [2507.09556].

The paper’s ablations compare deformable alignment with optical flow and report **50.55** CSO-mAP for “Deep unfolding + optical flow” and **50.67** for “Deep unfolding + deformable alignment,” supporting the claim that deformable alignment is better suited to the subtle, ambiguous inter-frame displacements in CSIST [2507.09556].

## 5. Training objective, SeqCSIST dataset, and evaluation protocol

DeRefNet is trained with three losses. The **constraint loss** enforces approximate invertibility of the learned transform pair:
$$
\mathcal{L}_{\text{constraint}} = \frac{1}{X Y} \sum_{i=1}^{X} \sum_{i=1}^{L} \left\| \tilde{G}^{(k)}\left(G^{(k)}(s_i)\right) - s_i \right\|_2^2.
\tag{22}
$$
The **alignment loss** penalizes discrepancy between aligned reference features and the temporally encoded center feature:
$$
\mathcal{L}_{\text{align}} = \frac{1}{2N} \sum_{i=t-N, i\neq t}^{t+N} \left\| H_{i}^{''} - H_{t}^{'} \right\|_1.
\tag{23}
$$
The **regression loss** supervises the final reconstruction:
$$
\mathcal{L}_{\text{regression}} = \frac{1}{(T-4)*(M/T)} \sum_{k=1}^{M/T} \sum_{i=2}^{T-2} \left\| H_{ki} - s_{ki} \right\|_2^2.
\tag{24}
$$
These are combined as
$$
\mathcal{L}_{all} = \beta \mathcal{L}_{constraint} + \gamma \mathcal{L}_{align} + \zeta \mathcal{L}_{regression},
\tag{25}
$$
with
$$
\beta = 0.01,\quad \gamma = 0.01,\quad \zeta = 1.
$$
The paper reports that this weighting yields the best CSO-mAP in its loss-weight ablation [2507.09556].

The accompanying benchmark, **SeqCSIST**, contains **5,000 trajectories**, **100,000 total frames**, and **20 frames per trajectory**. Every **5 consecutive frames** form one input sequence, and each trajectory yields **16 sequences** through sliding windows. The split is **70% training** (3500 trajectories), **15% validation** (750 trajectories), and **15% testing** (750 trajectories). Low-resolution image size is \(11 \times 11\), output size is \(33 \times 33\), and the target count per image varies randomly between **2, 3, and 4**. Target intensities lie in \([220,250]\) [2507.09556].

Training uses **Adam** in **MMEngine** with learning rate \(10^{-4}\), batch size **20**, and five consecutive frames per sample. The paper also reports that all spatial feature extraction uses **32-channel convolution** and the tail network contains **5** residual blocks. It does **not** report the number of epochs, weight decay, or a specific learning-rate schedule in the provided summary [2507.09556].

Evaluation is centered on **CSO-mAP** together with thresholded AP values such as **AP\(_{05}\)**, **AP\(_{10}\)**, **AP\(_{15}\)**, **AP\(_{20}\)**, and **AP\(_{25}\)**, reflecting the sub-pixel localization nature of the task [2507.09556].

## 6. Empirical performance, interpretation, and relation to adjacent literature

On SeqCSIST, DeRefNet reports **CSO-mAP = 51.55**, with **AP\(_{05}=1.00\)**, **AP\(_{10}=14.40\)**, **AP\(_{15}=54.90\)**, **AP\(_{20}=90.40\)**, and **AP\(_{25}=97.10\)**. The abstract states that the method improves **mAP by 5.3\%** over prior methods. The benchmark summary in the provided details lists **ISTA-Net+** at **51.02** CSO-mAP as the strongest prior entry shown there. This suggests that the paper’s headline 5.3% claim may be using a baseline definition or aggregation not identical to the specific excerpted table, although the reported result remains the best in the benchmark summary presented here [2507.09556].

The ablations isolate the major components. Replacing standard residual blocks with the deep unfolding extractor improves CSO-mAP from **47.96** to **50.27**. Adding deformable alignment gives **50.67**, and adding the time encoder raises this to **51.39**. A dynamic deformable alignment variant, labeled **DDA**, reaches **51.09** CSO-mAP with **0.28M** parameters and **6.14G FLOPs**, compared with **50.67**, **0.23M** parameters, and **6.44G FLOPs** for the corresponding DA setting. The paper also reports robustness figures of **47.48 mAP** on a hybrid synthetic-real background dataset, **47.23 mAP** under Gaussian noise up to \(\sigma=5\), and **49.79 mAP** when target count is increased to the denser **2–8** setting. Runtime is reported as **367 FPS**, despite **15.70G FLOPs** and a parameter count of **0.89M** [2507.09556].

DeRefNet should also be situated within a broader family of **DeRefNet-like** methods that do not use the exact name. The deformable refinement component in TSAIN aligns and extracts relevant content from input frames via deformable convolution in electron microscopic interpolation [2101.06771]. PDWN performs coarse-to-fine deformable offset refinement for video interpolation [2104.01517]. DFAR combines deformable temporal alignment with a feature refinement module for moving infrared dim-small target detection [2407.07289]. RRN recursively refines deformation vector fields across scales in unsupervised lung CT registration [2106.07608]. These systems indicate that “deformable refinement” is a wider architectural pattern spanning interpolation, detection, and registration, whereas **DeRefNet** in the strict sense refers specifically to the sequential CSIST unmixing model introduced in 2025 [2507.09556].

A final distinction is terminological. Earlier work established the underlying deformable sampling machinery and many later systems used deformable refinement ideas, but the exact name **“Deformable Refinement Network (DeRefNet)”** is the designation used in the SeqCSIST paper itself. In that paper, the term refers specifically to a model-driven, multi-frame unmixing architecture whose defining contribution is the coupling of sparse unfolding with **Temporal Deformable Feature Alignment** for sub-pixel recovery of latent closely spaced infrared targets [2507.09556].

Source: https://www.emergentmind.com/topics/deformable-refinement-network-derefnet