Sequential CSIST Unmixing for Infrared Targets

• The paper introduces Sequential CSIST Unmixing as a multi-frame infrared vision task that achieves sub-pixel localization of densely packed small targets.
• It combines sparse inverse reconstruction with temporal alignment via models like DeRefNet to disentangle overlapping energy from closely spaced targets.
• Empirical results on the SeqCSIST dataset demonstrate enhanced precision, robust performance under noise, and efficient real-time processing.

Sequential CSIST Unmixing denotes a multi-frame infrared vision task in which a highly dense Closely-Spaced Infrared Small Target group, observed as a single mixed spot because of limited optical focal length and detector resolution, is resolved into its constituent targets through sub-pixel localization. It was introduced as detecting all targets in the form of sub-pixel localization from a highly dense CSIST group, with the middle frame of an odd-length sequence taken as the reconstruction target and neighboring frames used as temporal support (Zhai et al., 13 Jul 2025).

1. Physical basis and task scope

The task is motivated by long-range infrared imaging conditions in which multiple nearby small targets do not appear as separate objects. Instead, optical diffraction spreads each point target over adjacent pixels through a Point Spread Function approximated by a 2D Gaussian, and sufficiently small inter-target spacing produces energy aliasing. The formulation in the sequential benchmark explicitly links this to the Airy disk and Rayleigh criterion: the Airy disk captures approximately 84% of the total energy, its radius is about $1.91$ times the standard deviation of the PSF, and when the inter-target distance falls below $1R$, energy aliasing occurs (Zhai et al., 13 Jul 2025).

Under this imaging regime, a single observed spot may correspond to multiple physical targets. The practical objective is therefore not merely target presence detection, but recovery of the number of hidden constituent targets and their precise sub-pixel locations. The sequential version is defined for an odd number of frames and uses spatiotemporal support from neighboring frames to unmix the middle frame. This distinguishes the task from conventional infrared small target detection, which assumes a one-to-one correspondence between an image detection and a real-world target, and from hyperspectral unmixing, where the latent variables are spectral signatures and abundance fractions rather than overlapping point-source responses (Zhai et al., 13 Jul 2025).

The underlying point-source model is written as

$$\text{PSF}(x, y; x_t, y_t) = a_i \cdot \frac{1}{2\pi\sigma^2} \exp\left( -\frac{(x - x_t)^2 + (y - y_t)^2}{2\sigma^2} \right),$$

with $(x_t,y_t)$ denoting sub-pixel target location, $a_i$ target brightness, and $\sigma$ the diffusion variance parameter. Pixel intensities are obtained by integrating this response over the pixel support (Zhai et al., 13 Jul 2025).

2. Observation models and inverse formulations

Single-frame CSIST unmixing papers formulate the problem as sparse inverse reconstruction on a high-resolution grid. In DISTA-Net, the discretized forward model is

$$\mathbf{z}=\mathbf{G}(\Omega)\tilde{\mathbf{s}}+\mathbf{w},$$

where $\Omega$ is a fine sub-pixel grid, $\tilde{\mathbf{s}}$ is a sparse high-resolution source vector, and the nonzero count, nonzero locations, and nonzero amplitudes encode target number, sub-pixel positions, and radiation intensities, respectively (Han et al., 25 May 2025). DSCSNet uses a closely related compressed-sensing formulation,

$$\mathbf{y} = \boldsymbol{\Phi} \cdot \mathcal{H}(\mathbf{x}) + \mathbf{n}, \qquad \|\mathbf{x}\|_0 = K \ll N,$$

where $1R$0 is the sparse high-resolution target map, $1R$1 models PSF blur, and $1R$2 is the measurement or downsampling operator (Tang et al., 22 Mar 2026).

Sequential CSIST Unmixing augments this sparse inverse model with temporal support. The benchmark formulation is

$1R$3

where $1R$4 are $1R$5 low-resolution video frames, $1R$6 is the middle frame, and $1R$7 is the predicted unmixing response for that middle frame (Zhai et al., 13 Jul 2025). In experiments, $1R$8, so an $1R$9 input patch is mapped to a $$\text{PSF}(x, y; x_t, y_t) = a_i \cdot \frac{1}{2\pi\sigma^2} \exp\left( -\frac{(x - x_t)^2 + (y - y_t)^2}{2\sigma^2} \right),$$0 output response (Zhai et al., 13 Jul 2025).

This formulation places sequential CSIST unmixing in a multi-frame reconstruction regime. The latent representation remains a sparse high-resolution response, but the evidence now comes from a short temporal window rather than a single patch. A plausible implication is that sub-pixel inter-frame motion provides complementary mixtures of the same hidden target configuration, which can be exploited if alignment is accurate.

3. SeqCSIST dataset and benchmark protocol

SeqCSIST is presented as the first dataset specifically for Sequential CSIST Unmixing. It contains 5,000 trajectories and 100,000 frames total, with 20 frames per trajectory. Each frame is an $$\text{PSF}(x, y; x_t, y_t) = a_i \cdot \frac{1}{2\pi\sigma^2} \exp\left( -\frac{(x - x_t)^2 + (y - y_t)^2}{2\sigma^2} \right),$$1 low-resolution crop. Every 5 consecutive frames form one input sequence, and each trajectory therefore yields 16 sequences through sliding windows over frames $$\text{PSF}(x, y; x_t, y_t) = a_i \cdot \frac{1}{2\pi\sigma^2} \exp\left( -\frac{(x - x_t)^2 + (y - y_t)^2}{2\sigma^2} \right),$$2 (Zhai et al., 13 Jul 2025).

Component	Specification
Trajectories	5,000
Total frames	100,000
Frames per trajectory	20
Input sequence length	5
Input patch size	$$\text{PSF}(x, y; x_t, y_t) = a_i \cdot \frac{1}{2\pi\sigma^2} \exp\left( -\frac{(x - x_t)^2 + (y - y_t)^2}{2\sigma^2} \right),$$3
Output/GT size	$$\text{PSF}(x, y; x_t, y_t) = a_i \cdot \frac{1}{2\pi\sigma^2} \exp\left( -\frac{(x - x_t)^2 + (y - y_t)^2}{2\sigma^2} \right),$$4
Core target count per image	2, 3, or 4
Split	70% train / 15% val / 15% test

The benchmark is synthetic. Target intensity values are random in $$\text{PSF}(x, y; x_t, y_t) = a_i \cdot \frac{1}{2\pi\sigma^2} \exp\left( -\frac{(x - x_t)^2 + (y - y_t)^2}{2\sigma^2} \right),$$5, diffusion variance is 0.5 pixel, and targets move along random trajectories such as quadratic functions, circles, and straight lines. The dataset also imposes explicit spatial and temporal constraints: targets are positioned relative to a reference point; relative directions are randomly selected from up, down, left, and right; the initial distance from each target point to the reference point is 0.3 pixel; the minimum distance between target points is $$\text{PSF}(x, y; x_t, y_t) = a_i \cdot \frac{1}{2\pi\sigma^2} \exp\left( -\frac{(x - x_t)^2 + (y - y_t)^2}{2\sigma^2} \right),$$6 pixel; the reference point moves by $$\text{PSF}(x, y; x_t, y_t) = a_i \cdot \frac{1}{2\pi\sigma^2} \exp\left( -\frac{(x - x_t)^2 + (y - y_t)^2}{2\sigma^2} \right),$$7 or $$\text{PSF}(x, y; x_t, y_t) = a_i \cdot \frac{1}{2\pi\sigma^2} \exp\left( -\frac{(x - x_t)^2 + (y - y_t)^2}{2\sigma^2} \right),$$8 pixels per frame depending on initial $$\text{PSF}(x, y; x_t, y_t) = a_i \cdot \frac{1}{2\pi\sigma^2} \exp\left( -\frac{(x - x_t)^2 + (y - y_t)^2}{2\sigma^2} \right),$$9; and each target point additionally moves a random distance in $(x_t,y_t)$0 pixel away from the reference point (Zhai et al., 13 Jul 2025).

Ground truth is provided as high-resolution coordinate labels and associated XML files that document coordinates and intensity values of all targets on the imaging plane. The $(x_t,y_t)$1 representation is used for visibility and evaluation, with super-resolution or unmixing ratio $(x_t,y_t)$2 (Zhai et al., 13 Jul 2025).

The primary metric is CSO-mAP, which aggregates AP scores over multiple localization distance thresholds $(x_t,y_t)$3. Reported columns are AP$(x_t,y_t)$4, AP$(x_t,y_t)$5, AP$(x_t,y_t)$6, AP$(x_t,y_t)$7, and AP$(x_t,y_t)$8. The benchmark description states that predictions are categorized as TP or FP, a precision-recall curve is generated by varying confidence thresholds, AP is the area under that curve, and CSO-mAP averages AP over the different distance thresholds (Zhai et al., 13 Jul 2025). The text does not provide a full matching algorithm, detailed score-generation mechanism, or explicit one-to-one assignment rule.

4. Single-frame CSIST foundations and the meaning of “sequential”

Before the multi-frame formulation, CSIST unmixing was primarily studied as a single-frame inverse problem. DISTA-Net formulates closely-spaced infrared small target unmixing as sparse reconstruction on a fine sub-pixel grid and unfolds ISTA into a multi-stage network with a dynamic transform module, a dynamic soft-thresholding module, and an inverse transform module. Its sequentiality is stage-wise: the estimate evolves through

$(x_t,y_t)$9

rather than across video frames (Han et al., 25 May 2025).

DSCSNet follows a related but ADMM-based route. It introduces a Dynamic Sparse Compressed Sensing Network for Close Small Object Unmixing, embeds a strict $a_i$0-norm sparsity constraint into the auxiliary-variable update step, and adds a self-attention-based dynamic thresholding mechanism together with Dynamic Information Reorganization. Like DISTA-Net, it is iterative within a frame, not temporal across frames (Tang et al., 22 Mar 2026).

Sequential CSIST Unmixing, as defined by SeqCSIST and DeRefNet, changes the meaning of “sequential.” The sequential element is no longer only stage-wise iterative refinement; it is multi-frame processing in which neighboring frames support the unmixing of the middle frame. The paper explicitly states that it is the first endeavor to address the CSIST Unmixing task within a multi-frame paradigm (Zhai et al., 13 Jul 2025).

This distinction addresses a common misconception. In the CSIST literature, “sequential” can refer either to unfolded iterative refinement inside one image or to genuine temporal aggregation across video frames. DISTA-Net and DSCSNet exemplify the former; SeqCSIST and DeRefNet exemplify the latter.

5. DeRefNet architecture and optimization

DeRefNet is the benchmark method proposed for Sequential CSIST Unmixing. It is described as a model-driven deep learning framework with three principal modules: a sparsity-driven feature extraction module, a positional or temporal encoding module, and a Temporal Deformable Feature Alignment (TDFA) module (Zhai et al., 13 Jul 2025).

The pipeline begins with a linear initialization inspired by ISTA. The initialized frame-specific estimate is denoted $a_i$1, and stage-wise updates follow the unfolded reconstruction pattern

$a_i$2

followed by a learned transform and shrinkage stage,

$a_i$3

Here $a_i$4 is the gradient step size, $a_i$5 and $a_i$6 are learned analysis and synthesis transforms, and $a_i$7 is a learned threshold (Zhai et al., 13 Jul 2025).

Temporal information is encoded as

$a_i$8

These temporally modulated features are then fed to TDFA. For each reference frame, TDFA first constructs an aggregated representation

$a_i$9

predicts offsets

$\sigma$0

and performs deformable alignment through

$\sigma$1

The middle-frame feature and aligned reference features are finally fused by tail convolution and a cascade of $\sigma$2 residual blocks; in experiments, $\sigma$3 (Zhai et al., 13 Jul 2025).

Training uses three losses. The constraint loss regularizes the learned transform pair toward approximate invertibility, the alignment loss encourages aligned reference features to match the middle-frame feature, and the regression loss supervises the output map against ground truth. The total loss is

$\sigma$4

with default weights $\sigma$5, $\sigma$6, and $\sigma$7 (Zhai et al., 13 Jul 2025).

The reported optimization and implementation settings are: optimizer Adam, framework MMEngine, learning rate $\sigma$8, batch size 20, processed sequence length 5 consecutive frames, super-resolution ratio $\sigma$9, input size $$\mathbf{z}=\mathbf{G}(\Omega)\tilde{\mathbf{s}}+\mathbf{w},$$0, ground-truth output size $$\mathbf{z}=\mathbf{G}(\Omega)\tilde{\mathbf{s}}+\mathbf{w},$$1, and 32 channels in spatial feature extraction (Zhai et al., 13 Jul 2025).

6. Empirical findings, robustness, and limitations

On the SeqCSIST benchmark, DeRefNet achieves CSO-mAP 51.55, with AP$$\mathbf{z}=\mathbf{G}(\Omega)\tilde{\mathbf{s}}+\mathbf{w},$$2 1.00, AP$$\mathbf{z}=\mathbf{G}(\Omega)\tilde{\mathbf{s}}+\mathbf{w},$$3 14.40, AP$$\mathbf{z}=\mathbf{G}(\Omega)\tilde{\mathbf{s}}+\mathbf{w},$$4 54.90, AP$$\mathbf{z}=\mathbf{G}(\Omega)\tilde{\mathbf{s}}+\mathbf{w},$$5 90.40, and AP$$\mathbf{z}=\mathbf{G}(\Omega)\tilde{\mathbf{s}}+\mathbf{w},$$6 97.10. The strongest baselines in the benchmark table are ISTA-Net+ at 51.02, GMFN at 50.94, and FISTA-Net at 50.61. The abstract reports that mean Average Precision is improved by 5.3\%, while the table shows a smaller absolute margin over the strongest listed baseline (Zhai et al., 13 Jul 2025).

The reported efficiency figures are 0.89M parameters, 15.70G FLOPs, and 367 FPS. In ablations, replacing stacked residual blocks with deep unfolding improves CSO-mAP from 47.96 to 50.27; adding deformable alignment yields 50.67 relative to 50.55 for optical flow in the compared setting; adding the time encoder raises performance to 51.39; and the default loss weighting gives the best result at 51.55 (Zhai et al., 13 Jul 2025).

Robustness studies indicate moderate degradation under domain shift and noise. On the hybrid real-background dataset, performance decreases from 51.55 to 47.48. Under additive Gaussian noise, reported CSO-mAP values are 49.09 at $$\mathbf{z}=\mathbf{G}(\Omega)\tilde{\mathbf{s}}+\mathbf{w},$$7, 48.40 at $$\mathbf{z}=\mathbf{G}(\Omega)\tilde{\mathbf{s}}+\mathbf{w},$$8, 48.41 at $$\mathbf{z}=\mathbf{G}(\Omega)\tilde{\mathbf{s}}+\mathbf{w},$$9, and 47.23 at $\Omega$0. When target density is increased from 2–4 to 2–8 targets, CSO-mAP becomes 49.79, a 1.76\% absolute drop from the original setting (Zhai et al., 13 Jul 2025).

The limitations are correspondingly clear. The benchmark is primarily synthetic; the core target-count range is limited to 2–4 in the main dataset; the method assumes that input has already been cropped around a candidate patch; the benchmark description does not fully specify post-processing from response maps to discrete point predictions; and the evaluation text does not provide a complete matching algorithm or score-generation mechanism. These constraints do not invalidate the task formulation, but they delimit the current scope of the benchmark and its immediate transferability (Zhai et al., 13 Jul 2025).

7. Relation to broader sequential unmixing literature

Outside infrared CSIST, “sequential unmixing” has an older meaning in hyperspectral and spectroscopic imaging. One line of work formulates multitemporal hyperspectral unmixing as online stochastic optimization over image sequences, with a perturbed linear mixing model,

$\Omega$1

temporal smoothness penalties, running sufficient statistics, and image-by-image updates of shared endmembers (Thouvenin et al., 2015). Another line reformulates linear spectral unmixing as a linear Gaussian state-space problem,

$\Omega$2

so that endmembers can be tracked spectrum by spectrum with a Kalman filter (Kouakou et al., 2024).

These formulations are task-distinct from CSIST unmixing. In hyperspectral sequential unmixing, the hidden variables are endmember spectra and abundances; in Sequential CSIST Unmixing, they are the constituent point targets hidden inside a mixed infrared spot. The resemblance lies in the incremental structure: a latent representation is refined under new observations, and constraints or priors regulate that refinement. This suggests a conceptual family resemblance rather than an identity of problem setting.

The broader constrained-unmixing literature also contributes reusable optimization templates. Blind and fully constrained hyperspectral unmixing uses group sparsity, positivity, and sum-to-one constraints within ADMM and reweighted schemes (Ammanouil et al., 2014), while fast fully constrained least squares can be implemented through Dykstra’s alternating projection onto convex sets in a reduced subspace (Wei et al., 2015). In the infrared CSIST setting, these works are not direct task formulations, but they exemplify how constraint enforcement, sparse regularization, and operator splitting can be organized in sequential or unfolded pipelines.