---
title: 'FlareBench: Solar Flare Prediction Benchmark'
url: https://www.emergentmind.com/topics/flarebench
type: topic
---

# FlareBench: Solar Flare Prediction Benchmark

In current arXiv usage, **FlareBench** is explicitly the name of a public benchmark for solar flare prediction introduced alongside the Deep Space Weather Model, where full-disk, multi-wavelength solar images are used to forecast the maximum flare class expected within the next 24 hours across a full 11-year solar activity cycle [2508.07847]. The designation is not unique across research domains: the release site `flarebench.github.io` belongs to the unrelated FLARE long-video audiovisual retrieval benchmark [2605.10228], while the flare-removal study "Toward Real Flare Removal: A Comprehensive Pipeline and A New Benchmark" presents a new benchmark but the available description does not confirm "FlareBench" as its official title [2306.15884].

## 1. Terminology and scope

The name has been applied in more than one context, but in space-weather research it denotes a benchmark for long-range solar flare prediction from synchronized SDO/AIA and SDO/HMI imagery. The overlap in naming matters because the underlying tasks, modalities, and evaluation criteria are entirely different.

| Name usage | Domain | Source |
|---|---|---|
| FlareBench | Solar flare prediction from multi-wavelength solar images | [2508.07847] |
| FLARE release at `flarebench.github.io` | Long-video audiovisual retrieval | [2605.10228] |
| "A New Benchmark" without confirmed official name "FlareBench" | Real flare removal in photography | [2306.15884] |

Within solar physics, FlareBench is designed around **full-disk**, **multi-wavelength**, **time-series** prediction rather than active-region patch classification alone. Its prediction target is the largest flare class occurring anywhere on the solar disk within the next 24 hours, which makes it a full-disk exceedance benchmark rather than a region-association benchmark [2508.07847].

## 2. Benchmark design and scientific motivation

FlareBench was created because prior datasets often cover short time spans, limited activity states, or small and low-resolution sunspot patches, leading to biased evaluations and limited generalization. The benchmark addresses these gaps by spanning the full solar cycle from June 2011 to November 2022, using both photospheric magnetic-field information and coronal multi-wavelength imaging, and emphasizing long-range spatio-temporal modeling through hour-cadence sequences and time-series cross-validation [2508.07847].

The benchmark differs from several earlier resources in a specific way. NOAA/GOES catalogs provide labels only; SHARP emphasizes active-region patches and parameters; FLARECAST centers on magnetogram-based predictors; and short-duration, patch-based datasets do not provide full-disk, synchronized, multi-channel imaging over a complete cycle. FlareBench instead offers **10 synchronized channels per time step**, continuous coverage across a full activity cycle, and a protocol centered on **reliability** and **equitable skill across classes**, not accuracy alone [2508.07847].

This design places FlareBench at the intersection of operational forecasting and end-to-end representation learning. Earlier forecasting studies frequently evaluated handcrafted active-region descriptors. For example, a FLARECAST/SMART-based study used feature vectors of up to 171 active-region properties extracted from SDO/HMI magnetograms, with 4,442 samples at 24-hour cadence and random 66%/34% train/test splits averaged over 100 realizations [1812.07258]. FlareBench moves the evaluation substrate from such feature tables to synchronized full-disk image sequences, thereby changing both the input space and the leakage-control assumptions.

## 3. Data sources, preprocessing, and label construction

FlareBench uses **SDO/AIA Level 1 images** at nine wavelengths and **SDO/HMI "1K" line-of-sight magnetograms** from JSOC. The AIA channels are the EUV bands 94 Å, 131 Å, 171 Å, 193 Å, 211 Å, 304 Å, and 335 Å, together with the UV band 1600 Å and the visible band 4500 Å. Labels are derived from **GOES XRS 1–8 Å** science-level X-ray flux, using GOES-15 for 2011–2020 and GOES-16 for 2021–2022 [2508.07847].

Each sample is a sequence at **1-hour cadence**, and each time step contains **10 channels**: one HMI magnetogram plus nine AIA images. The reported supervised experiments use a look-back length of \(k=4\) hours, while pretraining uses \(m=672\) hours, approximately 28 days, to capture solar-rotation-scale dependencies. Original images are \(1024 \times 1024\) and are resized to \(256 \times 256\). Preprocessing includes masking the HMI timestamp in the bottom-left corner, cropping AIA edges and resizing with bilinear interpolation to align AIA and HMI scales, per-channel standardization, and synchronization of HMI, AIA, and labels to the 1-hour cadence [2508.07847].

The benchmark begins with **100,801** samples. It excludes **2,440** samples with missing labels and **2,524** samples with more than 25% missing input channels per sample, leaving **95,837** samples for training and evaluation. Across the retained samples, the average missing rate is **0.93% \((\pm 4.03\%)\)**; if fewer than 25% of channels are missing in a sample, zero-padding is applied [2508.07847].

Labels correspond to the **largest flare class within the next 24 hours**. The classes are \(X\), \(M\), \(C\), and \(O\), defined from GOES XRS flux \(I\) as follows:
- \(X: I > 10^{-4}\,\mathrm{W/m^2}\)
- \(M: 10^{-5} < I \le 10^{-4}\,\mathrm{W/m^2}\)
- \(C: 10^{-6} < I \le 10^{-5}\,\mathrm{W/m^2}\)
- \(O: I \le 10^{-6}\,\mathrm{W/m^2}\)

The final class counts are **\(X=1{,}750\)**, **\(M=13{,}263\)**, **\(C=34{,}978\)**, and **\(O=47{,}775\)**. The distribution is highly imbalanced and varies annually; 2014 had 6.0% X-class flares, whereas 2018–2020 had no X-class events [2508.07847].

A key structural property of FlareBench is that labeling is **full-disk** rather than active-region-specific. The benchmark predicts the maximum class anywhere on the disk within 24 hours and does not spatially associate labels with particular active regions [2508.07847].

## 4. Forecasting task, validation protocol, and metric system

The benchmark task is **multiclass classification**: given a short history of synchronized multi-wavelength images, predict the largest flare class that will occur in the next 24 hours. To reduce temporal leakage and assess generalization across different phases of the solar cycle, FlareBench uses **three-fold time-series cross-validation**. The test sets include observations from 2021–2022, and a reported Deep SWM evaluation uses the period **2019-12-01 to 2022-11-30** [2508.07847].

Because rare \(X\)- and \(M\)-class events can make naive oversampling unstable, the associated Deep SWM study uses a two-stage training procedure, **Classifier Re-training** following Kang et al. (2020), to mitigate imbalance without heavy oversampling [2508.07847].

The protocol centers on three metrics. The first is the **Gandin-Murphy-Gerrity score (GMGS)**, an equitable multiclass weighted accuracy:
$$
\mathrm{GMGS} = \mathrm{tr}(S^T \cdot P),
$$
where \(P\) is the contingency table and \(S\) is a scoring matrix whose entries are defined from class frequencies:
$$
s_{ii} = \frac{1}{I-1}\left[\sum_{k=1}^{i-1} a_k^{-1} + \sum_{k=i}^{I-1} a_k\right],
$$
$$
s_{ij} = \frac{1}{I-1}\left[\sum_{k=1}^{i-1} a_k^{-1} + \sum_{k=i}^{j-1} (-1) + \sum_{k=j}^{I-1} a_k\right], \quad i<j,
$$
with
$$
a_i = \frac{1 - \sum_{k=1}^{i} p_k}{\sum_{k=1}^{i} p_k}, \qquad p_i = \sum_{j=1}^{I} p_{ij}.
$$

The second is **Brier Skill Score for M-or-above events**, written \( \mathrm{BSS}_{\ge M} \):
$$
\mathrm{BSS} = \frac{\mathrm{BS} - \mathrm{BS}_c}{0 - \mathrm{BS}_c},
$$
with
$$
\mathrm{BS} = \sum_{n=1}^{N} \sum_{i=1}^{I} \big(p(\hat{y}_{ni}) - y_{ni}\big)^2,
\qquad
\mathrm{BS}_c = \sum_{n=1}^{N} \sum_{i=1}^{I} (f - y_{ni})^2,
$$
where \(f\) is the climatological event rate [2508.07847].

The third is **True Skill Statistic for M-or-above events**, written \( \mathrm{TSS}_{\ge M} \). In solar-flare forecasting, the standard form is
$$
\mathrm{TSS} = \frac{TP}{TP+FN} - \frac{FP}{FP+TN},
$$
a metric long argued to be less sensitive than HSS to event/no-event imbalance [1202.5995].

Notably, the FlareBench protocol in the Deep SWM paper does **not** define or report HSS, Precision, Recall, F1, AUROC, Average Precision, Negative Log-Likelihood, Expected Calibration Error, or reliability diagrams. The benchmark centers on **GMGS**, **\( \mathrm{BSS}_{\ge M} \)**, and **\( \mathrm{TSS}_{\ge M} \)** [2508.07847].

## 5. Baselines, Deep SWM, and reported performance

The Deep SWM study evaluates FlareBench with several baselines. For context it reports physical-feature methods such as **DeFN**, **DeFN-R**, and **Flare Transformer**, although these were not fully rerun on FlareBench because the required physical features were unavailable. The reproduced end-to-end baseline is **CNN-LSTM**, trained and tested under the same conditions as the proposed method [2508.07847].

Human expert performance, drawn from Kubo et al. (2017) and Murray et al. (2017) over 2000–2015, is reported as **GMGS = 0.48**, **\( \mathrm{BSS}_{\ge M} = 0.16 \)**, and **\( \mathrm{TSS}_{\ge M} = 0.50 \)**. On FlareBench, Deep SWM yields the following mean \(\pm\) standard deviation over time-series cross-validation folds for the test period 2019-12-01 to 2022-11-30: **GMGS \(= 0.582 \pm 0.032\)**, **\( \mathrm{BSS}_{\ge M} = 0.334 \pm 0.299 \)**, and **\( \mathrm{TSS}_{\ge M} = 0.543 \pm 0.074 \)**. Relative to CNN-LSTM on the same test period, the reported gains are **GMGS +0.267**, **\( \mathrm{BSS}_{\ge M} +0.062 \)**, and **\( \mathrm{TSS}_{\ge M} +0.213 \)**. Relative to the Flare Transformer, evaluated on a different test period, the reported gains are **GMGS +0.079**, **\( \mathrm{BSS}_{\ge M} +0.252 \)**, and **\( \mathrm{TSS}_{\ge M} +0.013 \)**. Relative to human experts, the gains are **GMGS +0.102**, **\( \mathrm{BSS}_{\ge M} +0.174 \)**, and **\( \mathrm{TSS}_{\ge M} +0.043 \)** [2508.07847].

Ablation studies identify the **Long-range Temporal State Space Model**, specifically the **S5-based** component, as the module that improves GMGS the most. In the reported setting, **S5 outperforms attention and time-varying SSM (Mamba)**, which the authors interpret as supporting the value of time-invariant MIMO SSMs for continuous, multi-channel solar image sequences [2508.07847].

The benchmark is distributed through the Deep SWM project page, which hosts the benchmark description, code, and model details. Source data come from JSOC and GOES XRS. Reported implementation details include **AdamW** with \(\beta_1 = 0.9\), \(\beta_2 = 0.95\), batch size 32, learning rate \(4.0 \times 10^{-5}\), weight decay \(5.0 \times 10^{-2}\), \(k=4\) for supervised training, \(m=672\) for pretraining, selection by best validation GMGS, and three-fold time-series cross-validation. The hardware is reported as **Nvidia H200**, with training taking about **3 hours** and inference about **12 ms per sample** [2508.07847].

## 6. Benchmark lineage, methodological significance, and limitations

FlareBench sits within a longer methodological trajectory in solar flare forecasting. Earlier work argued that benchmark quality depends critically on standardized event definitions, leakage-aware temporal splits, and rare-event verification. "Toward Reliable Benchmarking of Solar Flare Forecasting Methods" made the case for adopting **TSS** as a standard comparison metric because it is unbiased under changing event/no-event sample ratios, and reported threshold-scanned Poisson-McIntosh baselines for 24-hour flare prediction [1202.5995]. The "All-Clear" workshop comparison later showed that direct head-to-head evaluation on common data sets and common labels can overturn claims of superiority, with no one method clearly outperforming all others [1608.06319]. Operational benchmarking work then formalized full-disk exceedance evaluation, issuance-time alignment, climatology references, reliability diagrams, ROC/AUC, and uniform thresholding conventions for probabilistic operational systems [1907.02905].

Against that background, FlareBench extends the benchmark concept in two directions. First, it replaces region-centric tabular predictors with **full-disk, synchronized, multi-wavelength image sequences**. Second, it spans a **complete 11-year solar activity cycle**, which directly addresses temporal bias and enables evaluation across quiet and active epochs [2508.07847].

Its limitations are equally explicit. The reported experiments use **compressed \(256 \times 256\)** images, which may limit fine-grained spatial detail. Class imbalance remains substantial. Samples with more than 25% missing channels are excluded, while the remainder are zero-padded. Labels are full-disk rather than active-region-specific, so the model must learn from global context without explicit spatial association [2508.07847].

A plausible implication is that FlareBench is best understood not as a replacement for earlier active-region benchmarks, but as a higher-level protocol for **end-to-end spatio-temporal forecasting under realistic operational variability**. That interpretation is consistent with the benchmark’s emphasis on multiclass equitable skill, reliability-focused Brier scoring for \(M\)-or-above events, and solar-cycle-spanning time-series validation [2508.07847].

Source: https://www.emergentmind.com/topics/flarebench