Holistic SIRST Evaluation (HSE) Metric

• HSE is a unified evaluation metric for single-frame infrared small target detection, integrating multi-threshold pixel-level confidence and target-level detection into one score.
• It computes HSE-P by integrating the pixel-level precision-recall curve to address false positives amid severe class imbalance and complex backgrounds.
• HSE-T aggregates object-level performance via connected-component matching over thresholds, ensuring robust detection consistency and fair model comparison.

Searching arXiv for the cited papers to ground the article in current preprints. Holistic SIRST Evaluation (HSE) is a unified evaluation metric for single-frame infrared small target (SIRST) detection that was introduced to address the fragmented and threshold-sensitive assessment regime common in the field. It is defined in “Rethinking Infrared Small Target Detection: A Foundation-Driven Efficient Paradigm” as a multi-threshold metric that evaluates both pixel-level confidence quality and target-level detection robustness, and combines them into a single scalar score, $\text{HSE} = \text{HSE-P} \times \text{HSE-T}$ (Yu et al., 5 Dec 2025). In the SIRST setting, where targets are tiny, positive pixels are extremely rare, backgrounds are complex, and outputs are usually probability maps, HSE is intended to provide a stable and comprehensive basis for fair model comparison.

1. Problem setting and motivation

SIRST (Single-frame InfraRed Small Target) detection aims to detect extremely small, dim objects in single infrared images. The task is characterized by targets that are often only a few to a few dozen pixels, severe class imbalance, complex backgrounds such as clouds, sea clutter, terrain, and sensor noise, and dense per-pixel confidence maps as the typical model output (Yu et al., 5 Dec 2025).

The difficulty of evaluation follows directly from these properties. A small change in threshold can cause large changes in detection rate and false alarms, while different metrics emphasize different failure modes. Traditionally, SIRST papers report a heterogeneous mix of pixel-level metrics such as IoU and normalized IoU (nIoU), target-level metrics such as detection probability $P_d$ and false alarm rate $F_a$, and sometimes ROC-AUC over many thresholds. The reported problems are fourfold: the evaluation system is fragmented and heterogeneous; many metrics are single-threshold sensitive; some metrics are purely pixel-level or purely target-level; and ROC/PR behavior under extreme imbalance can be misleading, especially for ROC-AUC (Yu et al., 5 Dec 2025).

The motivation for HSE is therefore not merely to add another score, but to replace a non-unified evaluation culture with a metric that is multi-threshold, holistic, and unified. In the terminology of the original definition, HSE evaluates both pixel-level confidence and target-level robustness, integrates over many decision thresholds, and provides a unified scalar score that can be used as a primary metric for SIRST (Yu et al., 5 Dec 2025).

A related line of work on robust SIRST training underscores why such a metric is needed. “SIRST-5K: Exploring Massive Negatives Synthesis with Self-supervised Learning for Robust Infrared Small Target Detection” emphasizes that limited dataset scale, limited diversity of noise and background, imbalanced positive vs negative samples, and problematic traditional augmentation make it hard to holistically evaluate SIRST models. It further argues that existing datasets lack massive, diverse negatives needed to probe false alarms in a realistic way (Lu et al., 2024). This suggests that HSE is not only an evaluation formalism, but also a response to dataset and protocol limitations already visible in prior SIRST research.

2. Formal structure of the metric

HSE has two sub-metrics. HSE-P is the pixel-level part and is defined as the area under the pixel-level Precision–Recall curve computed from all pixels of the dataset. HSE-T is the target-level part and is defined as a discrete integral of target-level Precision–Recall pairs across a set of thresholds. The final score is their product (Yu et al., 5 Dec 2025).

$$\boxed{ \text{HSE} = \text{HSE-P} \times \text{HSE-T} }$$

With a dataset of $N$ images, predicted probability maps $P_i \in [0,1]^{H_i \times W_i}$, ground-truth masks $G_i \in \{0,1\}^{H_i \times W_i}$, flattened pixels

$$\{(p_k, g_k)\}_{k=1}^K = \text{Flatten}\big(\{(P_i,G_i)\}_{i=1}^N\big),$$

and a threshold set

$$\mathcal{T} = \{t_j\}_{j=1}^M, \quad t_1 > t_2 > \dots > t_M,$$

the two components are defined as follows (Yu et al., 5 Dec 2025):

$$\boxed{ \text{HSE-P} = \int_{0}^{1} P(R)\, \mathrm{d}R }$$

$P_d$0

with $P_d$1 or equivalent boundary handling, and $P_d$2, $P_d$3 denoting target-level precision and recall at threshold $P_d$4.

The multiplicative fusion is explicitly intended to enforce joint competence. A low value in one component reduces the overall score, so a detector cannot compensate poor target-level robustness with strong pixel ranking, or vice versa. This is central to the “holistic” designation: the metric is not an arithmetic aggregation of loosely related quantities, but a coupling of two AP-like evaluations at different semantic levels (Yu et al., 5 Dec 2025).

3. Pixel-level component: HSE-P

HSE-P treats SIRST as a pixel-wise binary classification problem over the entire dataset. “Pixel-level confidence” refers to the continuous output values of the detector for each pixel. Instead of immediately thresholding them, HSE-P constructs a global Precision–Recall curve over all pixels in the dataset (Yu et al., 5 Dec 2025).

At any threshold $P_d$5, the binary prediction for pixel $P_d$6 is

$P_d$7

The counts are then

$P_d$8

From these,

$P_d$9

Sweeping $F_a$0 from 1 down to 0 yields a Precision–Recall curve $F_a$1, and HSE-P is the area under that curve: $F_a$2

In practice, the integral is implemented numerically. The description given is to sort pixels by decreasing confidence, compute precision and recall at each rank or at discrete thresholds, and approximate the integral by a discrete sum: $F_a$3 where $F_a$4 are recall values sorted from high to low. The paper does not specify the exact discretization scheme for HSE-P, but Algorithm 1 states: “Step HSE-P-2: Compute the Precision–Recall curve and integrate it to obtain HSE-P according to Eq.(11).” It is described as conceptually identical to computing Average Precision (AP) over all pixels (Yu et al., 5 Dec 2025).

The rationale for choosing a PR-based quantity rather than ROC-AUC is explicit. ROC-AUC is described as insensitive to false positives when negatives dominate, as in SIRST, and the paper notes that a detector can have huge numbers of false positives yet show a very high AUC. HSE-P is therefore intended to better reflect pixel-level false alarms and confidence calibration (Yu et al., 5 Dec 2025).

4. Target-level component: HSE-T

HSE-T evaluates object-level detection performance across thresholds. At a given threshold $F_a$5, predictions are binarized,

$F_a$6

and connected components are extracted from both prediction and ground truth via $F_a$7. For target-level evaluation, the paper defines predicted components collectively as connected components in thresholded predictions and ground-truth components as connected components in $F_a$8, with centroid-based matching under a Euclidean tolerance $F_a$9 (Yu et al., 5 Dec 2025).

The matching protocol is one-to-one. For each ground-truth target centroid $$\boxed{ \text{HSE} = \text{HSE-P} \times \text{HSE-T} }$$0, unmatched predicted centroids $$\boxed{ \text{HSE} = \text{HSE-P} \times \text{HSE-T} }$$1 are examined, and if

$$\boxed{ \text{HSE} = \text{HSE-P} \times \text{HSE-T} }$$2

then one successful match is counted and the prediction is marked as matched so that it cannot match multiple ground truths. The paper characterizes this centroid-based matching as simple, robust to tiny shapes, and not dependent on IoU thresholding, which can be unstable for extremely small objects (Yu et al., 5 Dec 2025).

For a threshold $$\boxed{ \text{HSE} = \text{HSE-P} \times \text{HSE-T} }$$3, let $$\boxed{ \text{HSE} = \text{HSE-P} \times \text{HSE-T} }$$4 be the number of matched predicted–ground-truth pairs, $$\boxed{ \text{HSE} = \text{HSE-P} \times \text{HSE-T} }$$5 the number of predicted components, and $$\boxed{ \text{HSE} = \text{HSE-P} \times \text{HSE-T} }$$6 the total number of ground-truth components. Then target-level precision and recall are

$$\boxed{ \text{HSE} = \text{HSE-P} \times \text{HSE-T} }$$7

$$\boxed{ \text{HSE} = \text{HSE-P} \times \text{HSE-T} }$$8

HSE-T is then the discrete integral

$$\boxed{ \text{HSE} = \text{HSE-P} \times \text{HSE-T} }$$9

The authors describe HSE-T as measuring target-level detection robustness because it reflects how well a model consistently detects targets across many thresholds, not just at one operating point. Because it integrates precision and recall over the entire threshold range, it penalizes models whose performance is unstable or highly sensitive to threshold choice. Compared with single-threshold $N$0 and $N$1, HSE-T is described as less sensitive to the exact threshold value and more informative about how well detection performance generalizes across different operating points (Yu et al., 5 Dec 2025).

This formulation also clarifies the relation between HSE-T and legacy SIRST metrics. The recall term acts as a $N$2 analog, while the precision term absorbs the false-alarm burden that would otherwise be reflected through $N$3. The difference is that HSE-T does this over a threshold sequence rather than at an arbitrary fixed threshold.

5. Practical computation, outputs, and implementation constraints

The computational pattern of HSE is AP-like at two levels. HSE-P is a pixel-wise PR integral over thresholds, and HSE-T is an object-wise PR integral over thresholds. The generic discrete approximation used in both cases is

$N$4

where $N$5 and $N$6 are precision and recall at different operating points (Yu et al., 5 Dec 2025).

For HSE-P, the required inputs are predicted probability maps and binary ground-truth masks for a test set. All images are flattened, a PR curve is computed over all pixels, and the curve is integrated. If an implementation already has a library function to compute Average Precision for binary classification, the paper notes that it can be directly used on all pixels (Yu et al., 5 Dec 2025).

For HSE-T, the required inputs additionally include a threshold set $N$7 and a centroid matching tolerance $N$8. Ground-truth connected components and centroids are precomputed, thresholded prediction components are extracted for each $N$9, one-to-one centroid matching is performed, target-level precision and recall are computed, and the discrete integral is taken over the resulting PR sequence (Yu et al., 5 Dec 2025).

The paper does not lock down a specific set $P_i \in [0,1]^{H_i \times W_i}$0, but states that $P_i \in [0,1]^{H_i \times W_i}$1 is provided as an input argument. It further notes that common choices are uniformly spaced thresholds or denser sampling in the high-confidence region, and that typical values of $P_i \in [0,1]^{H_i \times W_i}$2 in segmentation / detection benchmarks are 50–200. Because HSE-T is computed exactly over $P_i \in [0,1]^{H_i \times W_i}$3, consistency across compared methods is necessary (Yu et al., 5 Dec 2025).

The outputs are reported as a scalar HSE-P in $P_i \in [0,1]^{H_i \times W_i}$4, a scalar HSE-T in $P_i \in [0,1]^{H_i \times W_i}$5, and a scalar HSE in $P_i \in [0,1]^{H_i \times W_i}$6 as well, with the caveat that implementation may keep them in $P_i \in [0,1]^{H_i \times W_i}$7 before percentage scaling. The authors also speak of “HSE series metrics,” meaning the tuple $P_i \in [0,1]^{H_i \times W_i}$8 (Yu et al., 5 Dec 2025).

Several caveats are explicit. HSE relies on accurate masks and connected components; mislabeled or merged targets affect both sub-metrics. The choice of $P_i \in [0,1]^{H_i \times W_i}$9 matters: too small and small localization errors count as misses; too large and distinct nearby detections may be merged. PR-based metrics remain preferable to ROC under imbalance, but still need enough positives to be reliable. If many targets are essentially undetectable, recall will be low for all methods and HSE will primarily reflect relative differences in easier targets (Yu et al., 5 Dec 2025).

6. Relation to prior SIRST metrics, empirical behavior, and broader evaluation practice

HSE was proposed against the background of the usual SIRST quartet $G_i \in \{0,1\}^{H_i \times W_i}$0. Relative to those metrics, HSE-P uses the full probability map instead of a binarized map at a fixed threshold, incorporates both precision and recall over all thresholds, and is more sensitive to pixel-level false positives than ROC-AUC. HSE-T aggregates object-level performance over thresholds and replaces brittle single values of $G_i \in \{0,1\}^{H_i \times W_i}$1 and $G_i \in \{0,1\}^{H_i \times W_i}$2 at an arbitrary threshold. The authors state that HSE metrics therefore give stable model rankings even when thresholds or operating points change (Yu et al., 5 Dec 2025).

The paper provides a specific ROC limitation example. In a toy example labeled Fig. 8, Case I has many false positives visually yet achieves ROC-AUC $G_i \in \{0,1\}^{H_i \times W_i}$3, while Case II is visually much better with fewer false positives but has ROC-AUC $G_i \in \{0,1\}^{H_i \times W_i}$4. This is used to argue that ROC-AUC cannot be trusted as the main evaluation for SIRST, and to motivate PR-based HSE-P and HSE-T instead (Yu et al., 5 Dec 2025).

Empirically, HSE is used to evaluate the Foundation-Driven Efficient Paradigm (FDEP) and its components. The paper reports that when embedding FDEP into various backbones—ACM, ALCNet, MLCL-Net, ALCL-Net, DNA-Net, GGL-Net, UIU-Net, and MSDA-Net—HSE-P, HSE-T, and HSE consistently improve. The same metric family is also used to quantify gains from adding frozen VFM features, the Semantic Alignment Modulation Fusion (SAMF) module, and the Collaborative Optimization-based Implicit Self-Distillation (CO-ISD) strategy. The authors argue that because HSE integrates over thresholds, it is less affected by specific decision rules and provides a stable and comprehensive basis for comparing these variants (Yu et al., 5 Dec 2025).

A broader perspective emerges when HSE is read alongside SIRST-5K. That work does not explicitly use the term “Holistic SIRST Evaluation (HSE),” but it frames robust assessment in terms of scale, diverse noise conditions, diverse target appearances, and the joint use of IoU, $G_i \in \{0,1\}^{H_i \times W_i}$5, and $G_i \in \{0,1\}^{H_i \times W_i}$6 (Lu et al., 2024). It also argues that models trained on small, homogeneous datasets cannot be robustly tested under different noise levels, different background types and clutter structures, and varied target sizes and intensities, and that existing datasets lack massive, diverse negatives needed to probe false alarms realistically (Lu et al., 2024). A plausible implication is that HSE can be understood as the metric counterpart to that dataset-and-protocol critique: it supplies the unified score that a more diverse and stress-oriented SIRST benchmark requires.

Common misconceptions follow predictable lines. One is to treat HSE as merely another AP score. In the original formulation, this is incomplete: HSE is the product of a pixel-level PR integral and a target-level PR integral, so it is explicitly designed to prevent strong pixel ranking from masking weak object-level robustness, or the reverse (Yu et al., 5 Dec 2025). Another misconception is to assume that HSE eliminates all design choices. The paper does not claim that. Threshold sampling, connected-component definition, and centroid tolerance remain implementation variables, although the metric is intended to be markedly less brittle than single-threshold evaluation.

7. Position within the FDEP framework and significance for SIRST research

Within “Rethinking Infrared Small Target Detection: A Foundation-Driven Efficient Paradigm,” HSE is not an auxiliary appendix metric; it is presented as the mechanism that “unif[ies] the fragmented evaluation system” by performing “multi-threshold integral evaluation at both pixel-level confidence and target-level robustness” (Yu et al., 5 Dec 2025). The metric is therefore structurally tied to the paper’s larger argument that better feature transfer and efficient semantic distillation should be validated by an evaluation protocol aligned with the operating conditions of SIRST.

The same paper positions HSE as the primary, holistic indicator of performance while still reporting conventional metrics such as IoU, nIoU, $G_i \in \{0,1\}^{H_i \times W_i}$7, and $G_i \in \{0,1\}^{H_i \times W_i}$8. This dual reporting is significant because it preserves compatibility with prior literature while redefining the principal comparison axis. The authors emphasize that HSE agrees with visual quality improvements, including fewer false positives and fewer missed targets, while being robust to threshold choice and capturing both segmentation quality and detection robustness (Yu et al., 5 Dec 2025).

In the wider SIRST literature, robust training strategies such as those used to construct Synthetic SIRST-5K—real-world Noise2Noise displacement, negative augmentation, and self-supervised learning with Soft-IoU supervision—seek to improve $G_i \in \{0,1\}^{H_i \times W_i}$9, reduce $$\{(p_k, g_k)\}_{k=1}^K = \text{Flatten}\big(\{(P_i,G_i)\}_{i=1}^N\big),$$0, and raise IoU under broader noise and clutter conditions (Lu et al., 2024). This suggests that HSE is especially well matched to research programs centered on robustness, calibration, and cross-condition consistency, because those properties are not fully captured by a single threshold or by a purely pixel-level or purely target-level metric.

For researchers and benchmark designers, the significance of HSE lies in its attempt to formalize what SIRST evaluation had previously treated piecemeal. It encodes the view that a useful detector must rank target pixels well, suppress false alarms under extreme imbalance, and preserve stable target-level detectability across thresholds. In that sense, HSE marks a shift from metric aggregation by custom reporting to an explicit unified criterion for fair model comparison (Yu et al., 5 Dec 2025).