FlareBench: Solar Flare Prediction Benchmark

• FlareBench is a public benchmark for forecasting maximum solar flare classes using synchronized, full-disk, multi-wavelength images across an 11-year solar cycle.
• It employs SDO/AIA and SDO/HMI imagery with rigorous time-series cross-validation to mitigate temporal leakage and address class imbalance.
• The benchmark uses advanced metrics such as GMGS, BSS, and TSS to evaluate deep learning models, demonstrating significant improvements over traditional methods and expert forecasts.

In current arXiv usage, FlareBench is explicitly the name of a public benchmark for solar flare prediction introduced alongside the Deep Space Weather Model, where full-disk, multi-wavelength solar images are used to forecast the maximum flare class expected within the next 24 hours across a full 11-year solar activity cycle (Nagashima et al., 11 Aug 2025). The designation is not unique across research domains: the release site flarebench.github.io belongs to the unrelated FLARE long-video audiovisual retrieval benchmark (You et al., 11 May 2026), while the flare-removal study "Toward Real Flare Removal: A Comprehensive Pipeline and A New Benchmark" presents a new benchmark but the available description does not confirm "FlareBench" as its official title (Jin et al., 2023).

1. Terminology and scope

The name has been applied in more than one context, but in space-weather research it denotes a benchmark for long-range solar flare prediction from synchronized SDO/AIA and SDO/HMI imagery. The overlap in naming matters because the underlying tasks, modalities, and evaluation criteria are entirely different.

Name usage	Domain	Source
FlareBench	Solar flare prediction from multi-wavelength solar images	(Nagashima et al., 11 Aug 2025)
FLARE release at flarebench.github.io	Long-video audiovisual retrieval	(You et al., 11 May 2026)
"A New Benchmark" without confirmed official name "FlareBench"	Real flare removal in photography	(Jin et al., 2023)

Within solar physics, FlareBench is designed around full-disk, multi-wavelength, time-series prediction rather than active-region patch classification alone. Its prediction target is the largest flare class occurring anywhere on the solar disk within the next 24 hours, which makes it a full-disk exceedance benchmark rather than a region-association benchmark (Nagashima et al., 11 Aug 2025).

2. Benchmark design and scientific motivation

FlareBench was created because prior datasets often cover short time spans, limited activity states, or small and low-resolution sunspot patches, leading to biased evaluations and limited generalization. The benchmark addresses these gaps by spanning the full solar cycle from June 2011 to November 2022, using both photospheric magnetic-field information and coronal multi-wavelength imaging, and emphasizing long-range spatio-temporal modeling through hour-cadence sequences and time-series cross-validation (Nagashima et al., 11 Aug 2025).

The benchmark differs from several earlier resources in a specific way. NOAA/GOES catalogs provide labels only; SHARP emphasizes active-region patches and parameters; FLARECAST centers on magnetogram-based predictors; and short-duration, patch-based datasets do not provide full-disk, synchronized, multi-channel imaging over a complete cycle. FlareBench instead offers 10 synchronized channels per time step, continuous coverage across a full activity cycle, and a protocol centered on reliability and equitable skill across classes, not accuracy alone (Nagashima et al., 11 Aug 2025).

This design places FlareBench at the intersection of operational forecasting and end-to-end representation learning. Earlier forecasting studies frequently evaluated handcrafted active-region descriptors. For example, a FLARECAST/SMART-based study used feature vectors of up to 171 active-region properties extracted from SDO/HMI magnetograms, with 4,442 samples at 24-hour cadence and random 66%/34% train/test splits averaged over 100 realizations (Piana et al., 2018). FlareBench moves the evaluation substrate from such feature tables to synchronized full-disk image sequences, thereby changing both the input space and the leakage-control assumptions.

3. Data sources, preprocessing, and label construction

FlareBench uses SDO/AIA Level 1 images at nine wavelengths and SDO/HMI "1K" line-of-sight magnetograms from JSOC. The AIA channels are the EUV bands 94 Å, 131 Å, 171 Å, 193 Å, 211 Å, 304 Å, and 335 Å, together with the UV band 1600 Å and the visible band 4500 Å. Labels are derived from GOES XRS 1–8 Å science-level X-ray flux, using GOES-15 for 2011–2020 and GOES-16 for 2021–2022 (Nagashima et al., 11 Aug 2025).

Each sample is a sequence at 1-hour cadence, and each time step contains 10 channels: one HMI magnetogram plus nine AIA images. The reported supervised experiments use a look-back length of $k=4$ hours, while pretraining uses $m=672$ hours, approximately 28 days, to capture solar-rotation-scale dependencies. Original images are $1024 \times 1024$ and are resized to $256 \times 256$. Preprocessing includes masking the HMI timestamp in the bottom-left corner, cropping AIA edges and resizing with bilinear interpolation to align AIA and HMI scales, per-channel standardization, and synchronization of HMI, AIA, and labels to the 1-hour cadence (Nagashima et al., 11 Aug 2025).

The benchmark begins with 100,801 samples. It excludes 2,440 samples with missing labels and 2,524 samples with more than 25% missing input channels per sample, leaving 95,837 samples for training and evaluation. Across the retained samples, the average missing rate is 0.93% $(\pm 4.03\%)$; if fewer than 25% of channels are missing in a sample, zero-padding is applied (Nagashima et al., 11 Aug 2025).

Labels correspond to the largest flare class within the next 24 hours. The classes are $X$, $M$, $C$, and $O$, defined from GOES XRS flux $I$ as follows:

• $m=672$0
• $m=672$1
• $m=672$2
• $m=672$3

The final class counts are $m=672$4, $m=672$5, $m=672$6, and $m=672$7. The distribution is highly imbalanced and varies annually; 2014 had 6.0% X-class flares, whereas 2018–2020 had no X-class events (Nagashima et al., 11 Aug 2025).

A key structural property of FlareBench is that labeling is full-disk rather than active-region-specific. The benchmark predicts the maximum class anywhere on the disk within 24 hours and does not spatially associate labels with particular active regions (Nagashima et al., 11 Aug 2025).

4. Forecasting task, validation protocol, and metric system

The benchmark task is multiclass classification: given a short history of synchronized multi-wavelength images, predict the largest flare class that will occur in the next 24 hours. To reduce temporal leakage and assess generalization across different phases of the solar cycle, FlareBench uses three-fold time-series cross-validation. The test sets include observations from 2021–2022, and a reported Deep SWM evaluation uses the period 2019-12-01 to 2022-11-30 (Nagashima et al., 11 Aug 2025).

Because rare $m=672$8- and $m=672$9-class events can make naive oversampling unstable, the associated Deep SWM study uses a two-stage training procedure, Classifier Re-training following Kang et al. (2020), to mitigate imbalance without heavy oversampling (Nagashima et al., 11 Aug 2025).

The protocol centers on three metrics. The first is the Gandin-Murphy-Gerrity score (GMGS), an equitable multiclass weighted accuracy:

$1024 \times 1024$0

where $1024 \times 1024$1 is the contingency table and $1024 \times 1024$2 is a scoring matrix whose entries are defined from class frequencies:

$1024 \times 1024$3

$1024 \times 1024$4

with

$1024 \times 1024$5

The second is Brier Skill Score for M-or-above events, written $1024 \times 1024$6:

$1024 \times 1024$7

with

$1024 \times 1024$8

where $1024 \times 1024$9 is the climatological event rate (Nagashima et al., 11 Aug 2025).

The third is True Skill Statistic for M-or-above events, written $256 \times 256$0. In solar-flare forecasting, the standard form is

$256 \times 256$1

a metric long argued to be less sensitive than HSS to event/no-event imbalance (Bloomfield et al., 2012).

Notably, the FlareBench protocol in the Deep SWM paper does not define or report HSS, Precision, Recall, F1, AUROC, Average Precision, Negative Log-Likelihood, Expected Calibration Error, or reliability diagrams. The benchmark centers on GMGS, $256 \times 256$2, and $256 \times 256$3 (Nagashima et al., 11 Aug 2025).

5. Baselines, Deep SWM, and reported performance

The Deep SWM study evaluates FlareBench with several baselines. For context it reports physical-feature methods such as DeFN, DeFN-R, and Flare Transformer, although these were not fully rerun on FlareBench because the required physical features were unavailable. The reproduced end-to-end baseline is CNN-LSTM, trained and tested under the same conditions as the proposed method (Nagashima et al., 11 Aug 2025).

Human expert performance, drawn from Kubo et al. (2017) and Murray et al. (2017) over 2000–2015, is reported as GMGS = 0.48, $256 \times 256$4, and $256 \times 256$5. On FlareBench, Deep SWM yields the following mean $256 \times 256$6 standard deviation over time-series cross-validation folds for the test period 2019-12-01 to 2022-11-30: GMGS $256 \times 256$7, $256 \times 256$8, and $256 \times 256$9. Relative to CNN-LSTM on the same test period, the reported gains are GMGS +0.267, $(\pm 4.03\%)$0, and $(\pm 4.03\%)$1. Relative to the Flare Transformer, evaluated on a different test period, the reported gains are GMGS +0.079, $(\pm 4.03\%)$2, and $(\pm 4.03\%)$3. Relative to human experts, the gains are GMGS +0.102, $(\pm 4.03\%)$4, and $(\pm 4.03\%)$5 (Nagashima et al., 11 Aug 2025).

Ablation studies identify the Long-range Temporal State Space Model, specifically the S5-based component, as the module that improves GMGS the most. In the reported setting, S5 outperforms attention and time-varying SSM (Mamba), which the authors interpret as supporting the value of time-invariant MIMO SSMs for continuous, multi-channel solar image sequences (Nagashima et al., 11 Aug 2025).

The benchmark is distributed through the Deep SWM project page, which hosts the benchmark description, code, and model details. Source data come from JSOC and GOES XRS. Reported implementation details include AdamW with $(\pm 4.03\%)$6, $(\pm 4.03\%)$7, batch size 32, learning rate $(\pm 4.03\%)$8, weight decay $(\pm 4.03\%)$9, $X$0 for supervised training, $X$1 for pretraining, selection by best validation GMGS, and three-fold time-series cross-validation. The hardware is reported as Nvidia H200, with training taking about 3 hours and inference about 12 ms per sample (Nagashima et al., 11 Aug 2025).

6. Benchmark lineage, methodological significance, and limitations

FlareBench sits within a longer methodological trajectory in solar flare forecasting. Earlier work argued that benchmark quality depends critically on standardized event definitions, leakage-aware temporal splits, and rare-event verification. "Toward Reliable Benchmarking of Solar Flare Forecasting Methods" made the case for adopting TSS as a standard comparison metric because it is unbiased under changing event/no-event sample ratios, and reported threshold-scanned Poisson-McIntosh baselines for 24-hour flare prediction (Bloomfield et al., 2012). The "All-Clear" workshop comparison later showed that direct head-to-head evaluation on common data sets and common labels can overturn claims of superiority, with no one method clearly outperforming all others (Barnes et al., 2016). Operational benchmarking work then formalized full-disk exceedance evaluation, issuance-time alignment, climatology references, reliability diagrams, ROC/AUC, and uniform thresholding conventions for probabilistic operational systems (Leka et al., 2019).

Against that background, FlareBench extends the benchmark concept in two directions. First, it replaces region-centric tabular predictors with full-disk, synchronized, multi-wavelength image sequences. Second, it spans a complete 11-year solar activity cycle, which directly addresses temporal bias and enables evaluation across quiet and active epochs (Nagashima et al., 11 Aug 2025).

Its limitations are equally explicit. The reported experiments use compressed $X$2 images, which may limit fine-grained spatial detail. Class imbalance remains substantial. Samples with more than 25% missing channels are excluded, while the remainder are zero-padded. Labels are full-disk rather than active-region-specific, so the model must learn from global context without explicit spatial association (Nagashima et al., 11 Aug 2025).

A plausible implication is that FlareBench is best understood not as a replacement for earlier active-region benchmarks, but as a higher-level protocol for end-to-end spatio-temporal forecasting under realistic operational variability. That interpretation is consistent with the benchmark’s emphasis on multiclass equitable skill, reliability-focused Brier scoring for $X$3-or-above events, and solar-cycle-spanning time-series validation (Nagashima et al., 11 Aug 2025).