Ensemble reliability and the signal-to-noise paradox in large-ensemble subseasonal forecasts

Abstract: Recent studies have illustrated the existence of a `signal-to-noise paradox' (SNP) in some ensemble forecasting systems that manifests as situations where the correlation between the forecast ensemble mean and the observed truth is larger than the correlation between the forecast ensemble mean and individual forecast members. A well-calibrated forecast system that simultaneously satisfies climatological and ensemble variance reliability criteria will not exhibit an SNP if sample statistics can be evaluated using a sufficiently large ensemble size ($N$) over a sufficiently large number of independent cases ($M$). However, when $M$ is finite, an apparent SNP will sometimes occur as a natural consequence of sampling uncertainty, even in a perfectly reliable ensemble with many members. In this study, we evaluate the forecast skill, reliability characteristics, and signal-to-noise properties of three large-scale atmospheric circulation indices in 100-member subseasonal reforecasts with the ECMWF IFS. Daily mean NAO forecasts generally satisfy unbiased reliability criteria within the tolerance of our estimated sampling uncertainties. Nevertheless, NAO forecasts in this dataset exhibit symptoms of the SNP at subseasonal lead times. However, we do not find robust evidence for an underestimation of the magnitude of predictable signals and do not exclude the possibility that the apparent paradox in this dataset is a consequence of observational sampling uncertainties that are insensitive to ensemble size and common to all comparisons for this set of forecast start dates and lead times. Furthermore, we demonstrate that this apparent SNP can be eliminated with an unbiased reliability calibration. However, this is achieved through overfitting such that calibrated forecasts inherit the large sampling uncertainties present in the observations and thus exhibit unphysical variations with lead time.