SNAD Anomaly Detection Pipeline

• SNAD Anomaly Detection Pipeline is a modular framework that identifies rare astrophysical anomalies using standardized feature extraction and multiple unsupervised machine learning methods.
• The pipeline transforms light curves into 42 dimensionless features and applies algorithms like Isolation Forest, LOF, GMM, and One-Class SVM to robustly detect outliers.
• Expert validation with a bi-dimensional discriminant (periodogram amplitude versus reduced chi-square) minimizes false positives and highlights scientifically significant events.

The SNAD (SuperNova Anomaly Detection) anomaly detection pipeline is a comprehensive methodology for identifying rare or unexpected astrophysical phenomena in large-scale photometric surveys. Designed to process millions of light curves from facilities such as the Zwicky Transient Facility (ZTF), the pipeline orchestrates feature extraction, unsupervised machine learning, and expert validation in sequential modules, enabling the systematic discovery of scientifically interesting anomalies while minimizing false positives due to instrumental artifacts.

1. Pipeline Structure and Workflow

The SNAD anomaly detection pipeline operates in three main stages:

1. Feature Extraction: Each light curve from the survey is transformed into a standardized feature vector. In the ZTF DR3 application, 42 features were computed per object, encompassing photometric amplitudes, statistical moments (such as standard deviation, skew, and kurtosis), temporal statistics (e.g., the von Neumann statistic $\eta$), and periodogram-derived characteristics (including periodogram amplitude and frequency-domain percentiles). Normalization ensures all features are dimensionless and comparable (2012.01419).
2. Unsupervised Outlier Detection: Multiple machine learning algorithms are employed to search for outliers within the high-dimensional feature space. Outliers are defined as objects not conforming to the density or distribution expected for normal sources. The algorithms used include Isolation Forest (IF), Local Outlier Factor (LOF), Gaussian Mixture Models (GMM), and One-Class Support Vector Machines (O-SVM). Each leverages a distinct perspective on anomalousness, enhancing the pipeline's robustness through diversity.
3. Expert-Guided Identification: The ensemble of candidate anomalies produced by the algorithms is subjected to scrutiny by domain experts via a dedicated web interface. This step involves light curve visualization, cross-matching with astrophysical catalogs (e.g., SIMBAD, VSX), and the review of diagnostic plots. Experts classify candidates as either bogus (i.e., non-astrophysical artifacts) or potentially astrophysically interesting, enabling targeted follow-up.

2. Unsupervised Learning Algorithms and Their Integration

The choice and combination of outlier detection algorithms are central to SNAD's performance (2012.01419):

• Isolation Forest (IF): Constructs random trees that partition the feature space; anomalies are isolated faster and thus have shorter average path lengths.
• Local Outlier Factor (LOF): Quantifies an object's local density relative to its neighbors, detecting outliers in heterogeneous-density regions.
• Gaussian Mixture Model (GMM): Models the feature space as a sum of multi-variate Gaussians; low-probability points under all components are anomalous.
• One-Class SVM (O-SVM): Finds a (possibly nonlinear) boundary enclosing the normal data, with points outside considered outliers.

Anomalies returned by these methods are collated, and top-ranking candidates are selected for expert analysis. The use of four different methods mitigates the limitations inherent in any single approach.

3. Expert Analysis and Bi-Dimensional Filtering

A major challenge identified in the pipeline is the high rate of false positives due to artifacts such as image subtraction failures, CCD defects, overlapping sources, and transient image noise. Expert inspection revealed that 68% of initially flagged anomalies were bogus, while 32% corresponded to real, variable astrophysical sources (with 24% previously catalogued and 8% non-catalogued, including novel events) (2012.01419).

Subsequent data exploration by experts led to the identification of an effective bi-dimensional discriminant in the feature space: plotting the periodogram amplitude against the reduced $\chi^2$ of the light curve fit. This relationship can be formalized as:

$$\chi^2_{\rm red} = \frac{1}{N-1}\sum_{i=1}^{N} \left(\frac{m_i - \bar{m}}{\delta_i}\right)^2$$

where $m_i$ are measured magnitudes, $\delta_i$ their errors, and $\bar{m}$ the weighted mean. Artefacts tend to cluster at low periodogram amplitude and high reduced $\chi^2$, enabling efficient filtering of bogus candidates with minimal scientific loss.

4. Scientific Yield and Implications

The pipeline demonstrated its capacity for discovery by surfacing previously unknown variable objects—specifically, a spectroscopically confirmed RS Canum Venaticorum system, several nova candidates, and a red dwarf flare among non-catalogued outliers (2012.01419). These findings illustrate the power of machine learning to reveal rare or unexpected events hidden among millions of time series.

Beyond immediate discoveries, the high incidence of artifacts among flagged anomalies has informed improvements to upstream data calibration and subtraction procedures, with the bi-dimensional filter providing actionable guidance for ongoing and future surveys.

Moreover, the scalable architecture and expert-machine synergy embodied by SNAD serve as a prototype for alert brokers and anomaly filters critical to next-generation surveys (e.g., LSST), where data volumes will preclude exhaustive human inspection.

5. Codebase and Supporting Resources

The SNAD pipeline is released under an open-source license, with source code, feature extraction routines, and detailed instructions available at https://github.com/snad-space/zwad. A companion web viewer (https://ztf.snad.space/) allows expert users to inspect, cross-match, and annotate candidate anomalies with minimal friction. These tools lower the barrier to entry for other research groups and promote reproducibility and community follow-up (2012.01419).

6. Broader Methodological Context

The SNAD pipeline's staged structure exemplifies a modular and extensible approach to anomaly detection. Its integration of unsupervised algorithms, domain-specific feature engineering, and human-in-the-loop validation is consistent with best practices for machine learning in high-throughput scientific domains. The use of multi-method ensembles, iterative feedback, and intuitive expert-driven feature selection aligns the pipeline's findings with astrophysical relevance, a necessity given the overwhelming preponderance of non-scientific outliers in astronomical data streams.

The identification of practical discriminants (e.g., the periodogram amplitude vs. reduced $\chi^2$ plane) underscores the value of embedding interpretable submodules within complex ML pipelines, facilitating transparent decision-making and knowledge transfer to future experiments.

7. Conclusions and Relevance to Future Surveys

The SNAD anomaly detection pipeline operationalizes a rigorous, reproducible workflow for the identification and vetting of anomalous time-domain phenomena in astronomy. Its structured approach—from normalized feature extraction, through unsupervised ensemble detection, to expert adjudication and rational post-hoc filtering—enables both breadth and depth in the search for astrophysical novelties. These attributes position SNAD as an enabling technology for scientific exploitation of current and forthcoming massive surveys, where the ability to surface the rarest anomalies with high confidence is both scientifically urgent and technically demanding (2012.01419).