SFAE: Source Finder Accuracy Evaluator

• SFAE is a tool for evaluating radio astronomy source finders by measuring completeness, reliability, and parameterisation accuracy against a known truth catalogue.
• It employs a systematic methodology including potential-match identification, threshold-based matching, and the Hungarian Algorithm to resolve ambiguous detections.
• Its diagnostic outputs from simulated ASKAP data provide actionable insights to improve survey pipelines and optimize source finder performance.

Searching arXiv for the SFAE paper and closely related source-finder evaluation work. SFAE, the Source Finder Accuracy Evaluator, is a software tool and evaluation method for measuring how accurately a radio astronomy source-finding algorithm performs on spectral-line data cubes. It defines source-finder accuracy in terms of completeness, reliability, and accuracy of the parameterisation of the sources that were found, by comparing a source finder’s output catalogue against a catalogue of known truth on the same image. In the formulation introduced in "Assessing The Accuracy Of Radio Astronomy Source Finding Algorithms" (Westerlund et al., 2012), SFAE is intended for survey-scale settings in which data volumes from instruments such as ASKAP and SKA render manual inspection impractical.

1. Purpose and survey context

SFAE was created for an observational regime in which source finding must be both automated and sufficiently accurate that human verification is minimized. Its intended uses include comparing source finding algorithms for survey pipelines, determining which kinds of radio sources a particular source finder can accurately locate, identifying failure modes such as missed sources, false detections, merges, and splits, exploring optimal parameter settings, and assessing how well a source finder measures source parameters such as flux, position, and width (Westerlund et al., 2012).

The evaluator is therefore not a detection algorithm in its own right. Its role is post hoc and comparative: a source finder is run on a data cube exactly as it would be used in production, and SFAE then measures the quality of the resulting catalogue relative to a reference catalogue treated as truth. This design makes SFAE particularly relevant for survey engineering, where the central question is not only whether detections are produced, but whether the resulting catalogue is usable for downstream scientific analysis.

2. Accuracy criteria and reported quantities

SFAE evaluates source finders through three core metrics: completeness, reliability, and source parameterisation accuracy (Westerlund et al., 2012).

Quantity	Definition	Interpretation
Completeness	$C=\frac{n_r}{n_d}$	Fraction of real sources recovered
Reliability	$R=\frac{n_r}{n_t}$	Fraction of detections that are real
Parameterisation accuracy	Reference value versus measured test value for each matched source	Accuracy of source-property estimation

Here, $n_d$ is the number of sources present in the data cube, $n_r$ is the number of real sources recovered by the source finder, and $n_t$ is the total number of sources found by the source finder, including true and false positives. Completeness answers how many real sources are found; reliability answers how many detections are genuine; parameterisation accuracy addresses how well the finder measures quantities such as spatial position, spectral or frequency position, size or extent, peak flux, and integrated flux.

A central feature of SFAE is that these quantities are not restricted to global catalogue summaries. Completeness and reliability can also be computed as a function of source properties such as peak flux, integrated flux, signal-to-noise ratio, and spectral width. The binning convention is asymmetric by construction: completeness binning should use the reference catalogue values, whereas reliability binning should use the test catalogue values. This separation preserves the semantics of the two measures, since completeness is conditioned on what is truly present and reliability on what is actually reported.

A common simplification in source-finder evaluation is to reduce performance to a single scalar score. SFAE explicitly rejects that reduction by treating missed detections, false detections, and parameter biases as distinct failure modes. This suggests a more diagnostically useful notion of accuracy than a single aggregate metric.

3. Catalogue comparison and potential-match construction

SFAE operates on three inputs: a data cube, a reference catalogue containing the true sources in that cube, and a test catalogue produced by running the source finder on the same cube (Westerlund et al., 2012). The reference catalogue is assumed to be ground truth.

Its workflow has three main stages: identifying potential matches, resolving ambiguous matches into final matches, and analysing the matched catalogues to compute accuracy metrics. The first stage establishes whether a reference source and a test source could plausibly correspond to the same astrophysical object. A pair is considered a potential match if it is sufficiently close in spatial position along the beam’s major axis, sufficiently close in spatial position along the beam’s minor axis, and sufficiently close in frequency position.

The thresholds are specified directly. Spatial separation along each beam axis must be less than or equal to the beam size in that axis, and frequency separation must be less than or equal to the source’s full width at half maximum (FWHM). The paper notes that these thresholds are ideally based on the expected error of the source finder, if known.

Whenever a source has more than one potential match, SFAE forms a confused set: a cluster of reference and test objects linked through potential matches. This addresses ambiguous cases in which one test source could match multiple reference sources or one reference source could match multiple test sources. Such ambiguity is especially important in the presence of source blending and fragmentation, because naive nearest-neighbour matching can obscure the distinction between correct localisation and structural misrepresentation.

4. One-to-one optimum matching and treatment of ambiguous detections

After potential matches are established, SFAE resolves them into a final one-to-one correspondence using the Hungarian Algorithm on a weighted bipartite graph (Westerlund et al., 2012). One vertex set represents reference sources, the other represents test sources, and edges exist only for potential matches.

The matching obeys three rules: each source can appear in at most one final match, matches are allowed only between pairs already identified as potential matches, and matching must be symmetric. The optimization objective is two-stage: first maximize the number of matched pairs, then, among those solutions, minimize the total distance.

The distance metric is defined from normalized offsets in three coordinates: difference along the beam major axis, difference along the beam minor axis, and difference in frequency, scaled respectively by the beam widths and the frequency resolution. If the frequency resolution is unknown, the channel width may be used instead. The paper notes that other parameters, such as flux, could in principle be included, but were deliberately excluded so that parameterisation errors do not affect source matching.

This design has direct implications for how SFAE handles merged sources and split sources. If multiple true sources are reported as one detection, or one true source is reported as multiple detections, SFAE uses the reference catalogue as truth and may therefore leave some sources unmatched. That behavior is conservative rather than permissive: it records that the source finder failed to represent source structure correctly, rather than forcing a misleading match. In that sense, unmatched objects can be diagnostically meaningful rather than merely residual bookkeeping.

5. Diagnostic outputs and demonstration on simulated ASKAP data

SFAE can produce several diagnostic views: overall completeness and reliability, completeness versus source properties, reliability versus source properties, distributions of matched and unmatched sources in parameter space, and parameter comparison plots for matched pairs (Westerlund et al., 2012). These outputs help identify what types of sources are missed, what types are falsely detected, whether the source finder systematically biases measured parameters, and what selection cuts might improve catalogue reliability.

The demonstration in the paper uses a simulated ASKAP spectral-line data cube and the source finder Duchamp. The cube had angular size $1.42 \times 1.42$ degrees, angular resolution 30 arcsec, pixel size 10 arcsec, frequency range 1327.39 to (1422.0175) MHz, frequency resolution or channel width 92.5 kHz, and dimensions $512 \times 512 \times 1024$. The simulation used reduced noise to make testing easier. The source catalogue came from the SKADS S3-SAX simulation, with mostly point sources and some extended sources, and the image was a dirty image that had to be cleaned before source finding.

The reference catalogue was generated from a corresponding noiseless model cube using Duchamp with a very low threshold. The test catalogue was generated by running Duchamp on the cleaned noisy cube. Duchamp was run with its default parameters, except for using a trous wavelet reconstruction in three dimensions, a three-sigma threshold for final objects, and a direct flux threshold of $1\,\mu\mathrm{Jy}$ for the reference model.

The reported catalogue sizes were 235 for the reference catalogue and 384 for the test catalogue, with 190 matched pairs. From these values, the paper reports

$C=70.6\% \qquad R=49.9\%.$

The one-to-one matching did not exclude any sources from being matched. The authors also manually checked the matching and found that SFAE matched all 235 reference objects to the same test object as the manual process, indicating that the matching procedure behaved correctly on the test case.

The demonstration further shows that completeness and reliability vary with peak signal-to-noise ratio. These plots indicate the SNR regime in which the source finder becomes effective and where false detections dominate, which in turn can help estimate the survey’s usable flux limit. The paper also examines matched and unmatched sources as a function of integrated flux and FWHM line width, revealing which source populations are missed and where algorithmic improvements may be required.

For parameterisation accuracy, the paper illustrates the comparison using integrated flux. In this example, Duchamp generally underestimates integrated flux, with the bias becoming larger for faint sources. The paper suggests that such behavior may motivate improved parameterisation algorithms, a different source-measurement method, or a statistical correction.

6. Computational profile, assumptions, and limitations

The overall computational complexity of SFAE is given as

$O(N^3),$

where $R=\frac{n_r}{n_t}$0 is the number of catalogue objects (Westerlund et al., 2012). The breakdown reported in the paper is $R=\frac{n_r}{n_t}$1 for reading data, $R=\frac{n_r}{n_t}$2 for potential-match comparison, $R=\frac{n_r}{n_t}$3 for Hungarian matching, and up to $R=\frac{n_r}{n_t}$4 or $R=\frac{n_r}{n_t}$5 for later analyses, depending on the task.

Despite the cubic worst-case bound, the implementation is reported to be practical for the data sizes considered. On the ASKAP test case, SFAE took less than 3 seconds on a single quad-core CPU, whereas Duchamp itself took about 90 minutes on the same system for the test cube. The paper estimates that even for ASKAP-scale cubes, SFAE’s runtime remains modest. This suggests that, in practice, the dominant computational burden in survey pipelines remains source finding rather than evaluation.

The method also makes several explicit assumptions. The reference catalogue is treated as ground truth; the source finder should be run on the data cube as it would be used in production; matching is based mainly on position and frequency rather than flux or other measured parameters; and binning by source properties assumes those properties are measured consistently enough to support meaningful statistics. Merged and split sources remain difficult edge cases; SFAE handles them conservatively but not perfectly. The authors further note that SFAE could be extended to more advanced cross-matching or used with multiple data cubes, but that such extensions lie beyond the scope of the paper.

Taken together, these assumptions define the methodological boundaries of SFAE. It is a rigorous evaluator under a known-truth regime, not a universal source-catalogue reconciliation framework. Within that regime, however, it provides a structured basis for comparing source finders, tuning their parameters, and characterizing the operating conditions under which survey catalogues become scientifically reliable.