---
title: Sequential CSIST Unmixing for Infrared Targets
url: https://www.emergentmind.com/topics/sequential-csist-unmixing
type: topic
---

# Sequential CSIST Unmixing for Infrared Targets

Sequential CSIST Unmixing denotes a multi-frame infrared vision task in which a highly dense Closely-Spaced Infrared Small Target group, observed as a single mixed spot because of limited optical focal length and detector resolution, is resolved into its constituent targets through sub-pixel localization. It was introduced as detecting all targets in the form of sub-pixel localization from a highly dense CSIST group, with the middle frame of an odd-length sequence taken as the reconstruction target and neighboring frames used as temporal support [2507.09556].

## 1. Physical basis and task scope

The task is motivated by long-range infrared imaging conditions in which multiple nearby small targets do not appear as separate objects. Instead, optical diffraction spreads each point target over adjacent pixels through a Point Spread Function approximated by a 2D Gaussian, and sufficiently small inter-target spacing produces energy aliasing. The formulation in the sequential benchmark explicitly links this to the Airy disk and Rayleigh criterion: the Airy disk captures approximately 84% of the total energy, its radius is about \(1.91\) times the standard deviation of the PSF, and when the inter-target distance falls below \(1R\), energy aliasing occurs [2507.09556].

Under this imaging regime, a single observed spot may correspond to multiple physical targets. The practical objective is therefore not merely target presence detection, but recovery of the number of hidden constituent targets and their precise sub-pixel locations. The sequential version is defined for an odd number of frames and uses spatiotemporal support from neighboring frames to unmix the middle frame. This distinguishes the task from conventional infrared small target detection, which assumes a one-to-one correspondence between an image detection and a real-world target, and from hyperspectral unmixing, where the latent variables are spectral signatures and abundance fractions rather than overlapping point-source responses [2507.09556].

The underlying point-source model is written as
\[
\text{PSF}(x, y; x_t, y_t) = a_i \cdot \frac{1}{2\pi\sigma^2} \exp\left( -\frac{(x - x_t)^2 + (y - y_t)^2}{2\sigma^2} \right),
\]
with \((x_t,y_t)\) denoting sub-pixel target location, \(a_i\) target brightness, and \(\sigma\) the diffusion variance parameter. Pixel intensities are obtained by integrating this response over the pixel support [2507.09556].

## 2. Observation models and inverse formulations

Single-frame CSIST unmixing papers formulate the problem as sparse inverse reconstruction on a high-resolution grid. In DISTA-Net, the discretized forward model is
\[
\mathbf{z}=\mathbf{G}(\Omega)\tilde{\mathbf{s}}+\mathbf{w},
\]
where \(\Omega\) is a fine sub-pixel grid, \(\tilde{\mathbf{s}}\) is a sparse high-resolution source vector, and the nonzero count, nonzero locations, and nonzero amplitudes encode target number, sub-pixel positions, and radiation intensities, respectively [2505.19148]. DSCSNet uses a closely related compressed-sensing formulation,
\[
\mathbf{y} = \boldsymbol{\Phi} \cdot \mathcal{H}(\mathbf{x}) + \mathbf{n},
\qquad
\|\mathbf{x}\|_0 = K \ll N,
\]
where \(\mathbf{x}\) is the sparse high-resolution target map, \(\mathcal{H}\) models PSF blur, and \(\boldsymbol{\Phi}\) is the measurement or downsampling operator [2603.21192].

Sequential CSIST Unmixing augments this sparse inverse model with temporal support. The benchmark formulation is
\[
H_t = f_{DeRefNet}(L_{t-N}, \ldots, L_{t+N}),
\]
where \(\{L_{t-N}, \ldots, L_{t+N}\}\) are \(2N+1\) low-resolution video frames, \(L_t \in \mathbb{R}^{C \times H \times W}\) is the middle frame, and \(H_t \in \mathbb{R}^{C \times cH \times cW}\) is the predicted unmixing response for that middle frame [2507.09556]. In experiments, \(c=3\), so an \(11\times 11\) input patch is mapped to a \(33\times 33\) output response [2507.09556].

This formulation places sequential CSIST unmixing in a multi-frame reconstruction regime. The latent representation remains a sparse high-resolution response, but the evidence now comes from a short temporal window rather than a single patch. A plausible implication is that sub-pixel inter-frame motion provides complementary mixtures of the same hidden target configuration, which can be exploited if alignment is accurate.

## 3. SeqCSIST dataset and benchmark protocol

SeqCSIST is presented as the first dataset specifically for Sequential CSIST Unmixing. It contains **5,000 trajectories** and **100,000 frames total**, with **20 frames** per trajectory. Each frame is an **\(11 \times 11\)** low-resolution crop. Every 5 consecutive frames form one input sequence, and each trajectory therefore yields **16 sequences** through sliding windows over frames \(1\!-\!5, 2\!-\!6, \dots, 16\!-\!20\) [2507.09556].

| Component | Specification |
|---|---|
| Trajectories | 5,000 |
| Total frames | 100,000 |
| Frames per trajectory | 20 |
| Input sequence length | 5 |
| Input patch size | \(11 \times 11\) |
| Output/GT size | \(33 \times 33\) |
| Core target count per image | 2, 3, or 4 |
| Split | 70% train / 15% val / 15% test |

The benchmark is synthetic. Target intensity values are random in \([220, 250]\), diffusion variance is **0.5 pixel**, and targets move along random trajectories such as quadratic functions, circles, and straight lines. The dataset also imposes explicit spatial and temporal constraints: targets are positioned relative to a reference point; relative directions are randomly selected from up, down, left, and right; the initial distance from each target point to the reference point is **0.3 pixel**; the minimum distance between target points is \(0.3\sqrt{2}\) pixel; the reference point moves by \(+0.04\) or \(-0.04\) pixels per frame depending on initial \(x\); and each target point additionally moves a random distance in \((0, +0.0014)\) pixel away from the reference point [2507.09556].

Ground truth is provided as high-resolution coordinate labels and associated XML files that document coordinates and intensity values of all targets on the imaging plane. The \(33\times 33\) representation is used for visibility and evaluation, with super-resolution or unmixing ratio \(c=3\) [2507.09556].

The primary metric is **CSO-mAP**, which aggregates AP scores over multiple localization distance thresholds \(\delta_k\). Reported columns are AP\(_{05}\), AP\(_{10}\), AP\(_{15}\), AP\(_{20}\), and AP\(_{25}\). The benchmark description states that predictions are categorized as TP or FP, a precision-recall curve is generated by varying confidence thresholds, AP is the area under that curve, and CSO-mAP averages AP over the different distance thresholds [2507.09556]. The text does not provide a full matching algorithm, detailed score-generation mechanism, or explicit one-to-one assignment rule.

## 4. Single-frame CSIST foundations and the meaning of “sequential”

Before the multi-frame formulation, CSIST unmixing was primarily studied as a single-frame inverse problem. DISTA-Net formulates closely-spaced infrared small target unmixing as sparse reconstruction on a fine sub-pixel grid and unfolds ISTA into a multi-stage network with a dynamic transform module, a dynamic soft-thresholding module, and an inverse transform module. Its sequentiality is stage-wise: the estimate evolves through
\[
\tilde{\mathbf{s}}^{(0)} \rightarrow \mathbf{r}^{(1)} \rightarrow \tilde{\mathbf{s}}^{(1)} \rightarrow \cdots \rightarrow \mathbf{r}^{(N)} \rightarrow \tilde{\mathbf{s}}^{(N)},
\]
rather than across video frames [2505.19148].

DSCSNet follows a related but ADMM-based route. It introduces a Dynamic Sparse Compressed Sensing Network for Close Small Object Unmixing, embeds a strict \(\ell_1\)-norm sparsity constraint into the auxiliary-variable update step, and adds a self-attention-based dynamic thresholding mechanism together with Dynamic Information Reorganization. Like DISTA-Net, it is iterative within a frame, not temporal across frames [2603.21192].

Sequential CSIST Unmixing, as defined by SeqCSIST and DeRefNet, changes the meaning of “sequential.” The sequential element is no longer only stage-wise iterative refinement; it is multi-frame processing in which neighboring frames support the unmixing of the middle frame. The paper explicitly states that it is the first endeavor to address the CSIST Unmixing task within a multi-frame paradigm [2507.09556].

This distinction addresses a common misconception. In the CSIST literature, “sequential” can refer either to unfolded iterative refinement inside one image or to genuine temporal aggregation across video frames. DISTA-Net and DSCSNet exemplify the former; SeqCSIST and DeRefNet exemplify the latter.

## 5. DeRefNet architecture and optimization

DeRefNet is the benchmark method proposed for Sequential CSIST Unmixing. It is described as a model-driven deep learning framework with three principal modules: a sparsity-driven feature extraction module, a positional or temporal encoding module, and a Temporal Deformable Feature Alignment (TDFA) module [2507.09556].

The pipeline begins with a linear initialization inspired by ISTA. The initialized frame-specific estimate is denoted \(H_i^{(0)}\), and stage-wise updates follow the unfolded reconstruction pattern
\[
R_i^{(k)} = H_i^{(k-1)} - \rho \Phi^T (\Phi H_i^{(k-1)} - L_i),
\]
followed by a learned transform and shrinkage stage,
\[
H_{i}^{(k)} = \tilde{G}^{(k)}\left(\text{soft}\left({G}^{(k)}({R}_i^{(k)}), \theta^{(k)}\right)\right).
\]
Here \(\rho\) is the gradient step size, \(G^{(k)}\) and \(\tilde G^{(k)}\) are learned analysis and synthesis transforms, and \(\theta^{(k)}\) is a learned threshold [2507.09556].

Temporal information is encoded as
\[
T = Sigmoid(MLP(Encoder(t))),
\qquad
H^{'} = H^{(k)}T.
\]
These temporally modulated features are then fed to TDFA. For each reference frame, TDFA first constructs an aggregated representation
\[
F_{aggi} = SelectiveAttention(F_{i}, F_{t}),
\]
predicts offsets
\[
\eta = conv(F_{aggi}),
\]
and performs deformable alignment through
\[
H_{i}^{''}(p) = \sum_{k=1}^{K^2} \omega_k \cdot F_{i}(p + p_k + \Delta p_k).
\]
The middle-frame feature and aligned reference features are finally fused by tail convolution and a cascade of \(n\) residual blocks; in experiments, \(n=5\) [2507.09556].

Training uses three losses. The **constraint loss** regularizes the learned transform pair toward approximate invertibility, the **alignment loss** encourages aligned reference features to match the middle-frame feature, and the **regression loss** supervises the output map against ground truth. The total loss is
\[
\mathcal{L}_{all} = \beta \mathcal{L}_{constraint} + \gamma \mathcal{L}_{align} + \zeta \mathcal{L}_{regression},
\]
with default weights \(\beta = 0.01\), \(\gamma = 0.01\), and \(\zeta = 1\) [2507.09556].

The reported optimization and implementation settings are: optimizer **Adam**, framework **MMEngine**, learning rate \(10^{-4}\), batch size **20**, processed sequence length **5 consecutive frames**, super-resolution ratio \(c=3\), input size \(11\times11\), ground-truth output size \(33\times33\), and **32** channels in spatial feature extraction [2507.09556].

## 6. Empirical findings, robustness, and limitations

On the SeqCSIST benchmark, DeRefNet achieves **CSO-mAP 51.55**, with AP\(_{05}\) **1.00**, AP\(_{10}\) **14.40**, AP\(_{15}\) **54.90**, AP\(_{20}\) **90.40**, and AP\(_{25}\) **97.10**. The strongest baselines in the benchmark table are **ISTA-Net+** at **51.02**, **GMFN** at **50.94**, and **FISTA-Net** at **50.61**. The abstract reports that mean Average Precision is improved by **5.3\%**, while the table shows a smaller absolute margin over the strongest listed baseline [2507.09556].

The reported efficiency figures are **0.89M** parameters, **15.70G** FLOPs, and **367** FPS. In ablations, replacing stacked residual blocks with deep unfolding improves CSO-mAP from **47.96** to **50.27**; adding deformable alignment yields **50.67** relative to **50.55** for optical flow in the compared setting; adding the time encoder raises performance to **51.39**; and the default loss weighting gives the best result at **51.55** [2507.09556].

Robustness studies indicate moderate degradation under domain shift and noise. On the hybrid real-background dataset, performance decreases from **51.55** to **47.48**. Under additive Gaussian noise, reported CSO-mAP values are **49.09** at \(\sigma=2\), **48.40** at \(\sigma=3\), **48.41** at \(\sigma=4\), and **47.23** at \(\sigma=5\). When target density is increased from 2–4 to 2–8 targets, CSO-mAP becomes **49.79**, a **1.76\%** absolute drop from the original setting [2507.09556].

The limitations are correspondingly clear. The benchmark is primarily synthetic; the core target-count range is limited to **2–4** in the main dataset; the method assumes that input has already been cropped around a candidate patch; the benchmark description does not fully specify post-processing from response maps to discrete point predictions; and the evaluation text does not provide a complete matching algorithm or score-generation mechanism. These constraints do not invalidate the task formulation, but they delimit the current scope of the benchmark and its immediate transferability [2507.09556].

## 7. Relation to broader sequential unmixing literature

Outside infrared CSIST, “sequential unmixing” has an older meaning in hyperspectral and spectroscopic imaging. One line of work formulates multitemporal hyperspectral unmixing as online stochastic optimization over image sequences, with a perturbed linear mixing model,
\[
Y_t = (M+dM_t)A_t + \mathbf{B}_t,
\]
temporal smoothness penalties, running sufficient statistics, and image-by-image updates of shared endmembers [1510.05893]. Another line reformulates linear spectral unmixing as a linear Gaussian state-space problem,
\[
s_t = s_{t-1} + v_t,
\qquad
y_t = H_t s_t + e_t,
\]
so that endmembers can be tracked spectrum by spectrum with a Kalman filter [2407.15636].

These formulations are task-distinct from CSIST unmixing. In hyperspectral sequential unmixing, the hidden variables are endmember spectra and abundances; in Sequential CSIST Unmixing, they are the constituent point targets hidden inside a mixed infrared spot. The resemblance lies in the incremental structure: a latent representation is refined under new observations, and constraints or priors regulate that refinement. This suggests a conceptual family resemblance rather than an identity of problem setting.

The broader constrained-unmixing literature also contributes reusable optimization templates. Blind and fully constrained hyperspectral unmixing uses group sparsity, positivity, and sum-to-one constraints within ADMM and reweighted schemes [1403.0289], while fast fully constrained least squares can be implemented through Dykstra’s alternating projection onto convex sets in a reduced subspace [1505.01740]. In the infrared CSIST setting, these works are not direct task formulations, but they exemplify how constraint enforcement, sparse regularization, and operator splitting can be organized in sequential or unfolded pipelines.

Source: https://www.emergentmind.com/topics/sequential-csist-unmixing