SleepBench: Open Benchmark for Sleep Analysis

• SleepBench is a comprehensive benchmark for sleep analysis that standardizes preprocessing, consensus labeling, and subject-level splits across diverse PSG datasets.
• It aggregates large-scale recordings from public sources to evaluate tasks such as sleep staging, event detection, and disease prediction using robust metrics.
• The framework supports controlled comparisons of self-supervised and foundation models under heterogeneous conditions including missing channels and variable scorer agreement.

Searching arXiv for the specified papers to ground the article in the cited literature. SleepBench is a benchmark framework for automated analysis of sleep recordings, especially polysomnography (PSG), designed to support controlled and reproducible evaluation under heterogeneous cohorts, channel configurations, and task settings. In the literature, the term has at least two closely related instantiations. One originates from the Dreem Open Datasets and their multi-scorer benchmarking framework for sleep staging, centered on DOD-H and DOD-O and formalized around consensus scoring, subject-wise evaluation, and comparison to human technologists (Guillot et al., 2019). A later and explicitly named version defines SleepBench as a “large-scale, fully open benchmark assembled to study pre-training and scaling of sleep foundation models (FMs),” aggregating 166,500 hours of PSG recordings from nine public datasets and extending evaluation beyond sleep staging to event detection and disease prediction (Shuai et al., 27 Feb 2026). A broader evaluation framework, SLEEPYLAND, does not name itself SleepBench, but is presented as already instantiating many benchmark hallmarks through a 24-dataset, consensus-aware, uncertainty-informed evaluation regime (Rossi et al., 10 Jun 2025).

1. Origins and conceptual scope

The earliest foundation for SleepBench is the Dreem Open Datasets study, which introduced two publicly available datasets, DOD-H and DOD-O, each scored by five sleep technologists from different sleep centers, together with a framework for comparing automated approaches to a consensus of multiple human scorers (Guillot et al., 2019). In that setting, the central problem was that automated sleep stage classifiers had usually been compared against a single human annotation despite inter-rater agreement of about 85% only. The Dreem framework therefore emphasized multi-scored evaluation, consensus labeling, weighted metrics, and subject-wise splits.

A later extension of the benchmarking idea appears in SLEEPYLAND, an open-source sleep staging evaluation framework designed to address fair model evaluation, generalization across diverse datasets, model bias, and variability in human annotations (Rossi et al., 10 Jun 2025). It centralizes and harmonizes approximately 220,000 hours of in-domain sleep recordings and approximately 84,000 hours of out-of-domain sleep recordings, spanning a broad range of ages, sleep-wake disorders, and hardware setups. SLEEPYLAND standardizes AASM stage labels W/N1/N2/N3/REM, evaluates single-channel and multi-channel EEG/EOG configurations, and introduces consensus-aware and uncertainty-aware metrics on multi-annotator datasets.

The most explicit formulation is given by "OSF: On Pre-training and Scaling of Sleep Foundation Models" (Shuai et al., 27 Feb 2026), which states that SleepBench is a “comprehensive, fully open-source benchmark” built to study pre-training and scaling of sleep foundation models. In that work, SleepBench addresses real-world heterogeneity in PSG across cohorts and devices, including differences in channel montages, sampling rates, demographics, and label taxonomies. It is intended to enable controlled comparisons of self-supervised objectives, scaling behaviors, and missing-channel robustness.

Taken together, these works suggest that SleepBench is best understood not as a single immutable dataset, but as a benchmark lineage organized around standardized preprocessing, rigorous splits, consensus-aware scoring where possible, and evaluation under distributional and hardware variability.

2. Dataset foundations and benchmark composition

The Dreem-based benchmark core comprises two PSG datasets. DOD-H contains 25 volunteers recruited at IRBA (France), described as healthy sleepers with no sleep complaints and PSG-confirmed absence of sleep disorder (Guillot et al., 2019). DOD-O contains 55 patients with clinical suspicion for sleep-related breathing disorder, recorded at the Stanford Sleep Medicine Center, with exclusions including sleep disorders other than OSA (Guillot et al., 2019). Both were sampled at 250 Hz, scored according to the AASM Scoring Manual v2.5, and segmented into 30 s epochs. DOD-H uses a Compumedics Siesta device with 12 EEG derivations, left and right EOG, chin EMG, and ECG; DOD-O uses a Somnomedics Somno HD device with 8 EEG derivations, left and right EOG, chin EMG, and ECG (Guillot et al., 2019).

The explicitly named SleepBench in OSF aggregates 166,500 hours of PSG recordings, approximately 20 million 30-second epochs, from 21,482 nights across 9 public datasets hosted by the National Sleep Research Resource (NSRR) (Shuai et al., 27 Feb 2026). The included datasets are SHHS, NCHSDB, WSC, CCSHS, CFS, MROS, MESA, CHAT, and SOF. Its in-domain partition consists of SHHS, NCHSDB, WSC, CCSHS, and CFS, while MROS, MESA, CHAT, and SOF are strictly held out for out-of-domain evaluation (Shuai et al., 27 Feb 2026).

SLEEPYLAND, by contrast, covers 24 datasets, including 17 in-domain cohorts and 7 out-of-domain cohorts (Rossi et al., 10 Jun 2025). Its out-of-domain datasets include BSWR, DCSM, DOD-H, DOD-O, PHYS, SEDF-SC, and SEDF-ST. It includes adult, pediatric, elderly, healthy, mixed cohorts, narcolepsy-related subcohorts, severe OSA, and broad real-world clinical populations. It operates across EEG and EOG derivations and explicitly excludes chin EMG to maximize dataset inclusion (Rossi et al., 10 Jun 2025).

The following table summarizes the three benchmark formulations described in the cited literature.

Framework	Core scale	Primary emphasis
Dreem Open Datasets framework (Guillot et al., 2019)	DOD-H: 25 subjects; DOD-O: 55 subjects	Multi-scorer sleep staging benchmark with consensus evaluation
SLEEPYLAND (Rossi et al., 10 Jun 2025)	approximately 220,000 ID hours and approximately 84,000 OOD hours across 24 datasets	Fair cross-dataset sleep staging evaluation, consensus, uncertainty, and bias
SleepBench in OSF (Shuai et al., 27 Feb 2026)	166,500 hours, 21,482 nights, 9 datasets	Fully open benchmark for sleep foundation models, scaling, SSL, and missing-channel robustness

A plausible implication is that the Dreem datasets supply a consensus-rich staging benchmark, whereas OSF’s SleepBench generalizes the benchmarking objective to multi-task foundation-model evaluation under broader physiological and cohort heterogeneity.

3. Consensus labeling, scorer agreement, and uncertainty

A defining feature of the Dreem-based SleepBench formulation is its use of five independent scorers per record (Guillot et al., 2019). Every record is annotated by five scorers, enabling robust estimation beyond a single scorer’s variability. For DOD-H, the Soft-Agreement values of the scorers are 0.87, 0.91, 0.92, 0.84, and 0.92, leading to a consensus built from scorers 1, 2, 3, and 5; for DOD-O, the values are 0.88, 0.87, 0.88, 0.88, and 0.91, leading to a consensus built from scorers 1, 2, 4, and 5 (Guillot et al., 2019). Ties in majority voting occur in 7.3% of epochs in DOD-H and 9.9% in DOD-O.

The consensus strategy is defined per epoch by majority vote among the selected scorers, with ties broken by selecting the label from the scorer with the highest Soft-Agreement on that record (Guillot et al., 2019). Epochs are further weighted by the fraction of scorers who agreed with the consensus label. The Soft-Agreement metric is defined by

$$v^{(t)} = \sum_{i\ne j}\hat{y}_i[:,t], \qquad a_j(t)=\frac{v^{(t)}[y_j(t)]}{\max_k v^{(t)}[k]},$$

and

$$\mathrm{Soft\text{-}Agreement}_j=\frac{1}{T}\sum_{t=1}^{T} a_j(t).$$

Its interpretation is that a value of 1.0 means scorer $j$’s label always aligns with the majority or one of tied majorities (Guillot et al., 2019).

SLEEPYLAND adopts the same multi-annotator logic on DOD-H and DOD-O, using both discrete consensus hypnograms and soft-consensus hypnodensity graphs built from scorers’ probabilistic votes (Rossi et al., 10 Jun 2025). In that framework, consensus is defined via majority voting among the four most reliable scorers, with ties resolved by the most reliable scorer. SLEEPYLAND further introduces Averaged Cosine Similarity (ACS) between a model hypnodensity and the soft-consensus hypnodensity, and it reports that models align more with consensus than with any single scorer (Rossi et al., 10 Jun 2025).

SLEEPYLAND also adds explicit uncertainty quantification. Entropy of the ensemble output is computed as

$H(p)=-\sum_{c=1}^{C} p_c \log p_c,$

and inter-model divergence is computed through pairwise cosine distances

$d_{m,n}^t = 1 - \cos(\hat{y}_m^t,\hat{y}_n^t).$

A logistic regression using entropy and divergence features predicts consensus disagreement, achieving mean ROC AUC 0.823 on DOD-H and 0.828 on DOD-O, outperforming entropy-only and divergence-only variants (Rossi et al., 10 Jun 2025). This suggests a benchmark evolution from consensus matching alone toward explicit modeling of human uncertainty.

4. Preprocessing, splits, and evaluation protocols

In the Dreem framework, common preprocessing used for all benchmarked models consists of a band-pass filter $[0.4,18]$ Hz, resampling to 100 Hz, clipping and dividing by 500 to remove extremes, and zero-padding at both ends to accommodate context windows (Guillot et al., 2019). A context window $k=21$ is used, corresponding to 10 past epochs, the current epoch, and 10 future epochs. The common training setup uses Adam with $\mathrm{lr}=0.001$, $\beta_1=0.9$, $\beta_2=0.999$, batch size 32, early stopping if validation accuracy does not improve for more than 15 epochs, and up to 100 epochs total (Guillot et al., 2019). DOD-H uses a leave-one-out-like split per iteration with 18 train, 6 validation, and 1 test subject; DOD-O uses a 10-fold subject-wise protocol with 37 train, 12 validation, and 6 test subjects (Guillot et al., 2019).

SLEEPYLAND defines a different but equally standardized protocol (Rossi et al., 10 Jun 2025). All deep models use consistent splits per dataset and identical data sampling, preprocessing, augmentation, and optimization under a unified TensorFlow 2 framework. Preprocessing follows Perslev et al. (U-Sleep) with minor adjustments, no additional filtering, robust scaling per instance, on-the-fly spectrogram generation for SleepTransformer, and standardized AASM W/N1/N2/N3/REM labels. Training segments have length 35 epochs (17.5 min) for all models. Training uses a consistent batch size of 64, early stopping with patience 200, up to 10,000 training “epochs” where one “epoch” processes $$\mathrm{Soft\text{-}Agreement}_j=\frac{1}{T}\sum_{t=1}^{T} a_j(t).$$0 sleep segments, a cross-entropy objective, and the same augmentation with 0.1 probability of noise or channel dropout, omitting channel dropout for single-channel models (Rossi et al., 10 Jun 2025). Outputs are aggregated by majority vote across available channel derivations.

OSF’s SleepBench standardizes a 12-channel montage spanning brain, respiration, cardiac, and somatic groups: C3–A2, C4–A1, E1–A2, E2–A1, abdominal effort, thoracic effort, nasal pressure, snore, ECG, EMG-chin, EMG-left leg, and EMG-right leg (Shuai et al., 27 Feb 2026). Raw channels are resampled to target rates, each night is z-score normalized per channel, respiratory channels additionally undergo area-dependent z-score normalization, normalized signals are clipped to $$\mathrm{Soft\text{-}Agreement}_j=\frac{1}{T}\sum_{t=1}^{T} a_j(t).$$1, and signals are segmented into non-overlapping 30-second epochs. Within epochs, all channels are resampled to 64 Hz for model input, and missing channels are zero-padded (Shuai et al., 27 Feb 2026). SleepBench also defines four missing-channel stress-test configurations: head-band device only, sleep disorder study, sleep micro-architecture study, and in-home screening (Shuai et al., 27 Feb 2026).

Across these variants, a common benchmark principle is subject-level or recording-level split integrity to avoid leakage, together with preprocessing harmonization strong enough to support cross-method comparison without erasing clinically meaningful signal variation.

5. Tasks, metrics, and model families

The Dreem-based staging benchmark is defined on five classes, with labels $$\mathrm{Soft\text{-}Agreement}_j=\frac{1}{T}\sum_{t=1}^{T} a_j(t).$$2, $$\mathrm{Soft\text{-}Agreement}_j=\frac{1}{T}\sum_{t=1}^{T} a_j(t).$$3, $$\mathrm{Soft\text{-}Agreement}_j=\frac{1}{T}\sum_{t=1}^{T} a_j(t).$$4, $$\mathrm{Soft\text{-}Agreement}_j=\frac{1}{T}\sum_{t=1}^{T} a_j(t).$$5, and $$\mathrm{Soft\text{-}Agreement}_j=\frac{1}{T}\sum_{t=1}^{T} a_j(t).$$6 (Guillot et al., 2019). Reported metrics include per-class precision, recall, F1, macro- and micro-aggregations, accuracy, Cohen’s $$\mathrm{Soft\text{-}Agreement}_j=\frac{1}{T}\sum_{t=1}^{T} a_j(t).$$7, and confusion matrices. Precision, recall, and F1 are defined as

$$\mathrm{Soft\text{-}Agreement}_j=\frac{1}{T}\sum_{t=1}^{T} a_j(t).$$8

with

$$\mathrm{Soft\text{-}Agreement}_j=\frac{1}{T}\sum_{t=1}^{T} a_j(t).$$9

and

$j$0

Metrics are computed subject-wise rather than epoch-wise, with epoch weighting by consensus vote strength (Guillot et al., 2019).

SLEEPYLAND also evaluates sleep staging on five AASM classes and uses Macro-F1, per-class F1, accuracy, Cohen’s $j$1, and ACS, together with ROC AUC for uncertainty prediction (Rossi et al., 10 Jun 2025). The framework releases pre-trained models based on U-Sleep, DeepResNet, and SleepTransformer, and introduces SOMNUS, an ensemble across architectures and across channel setups using soft voting. For each epoch $j$2, if model $j$3 outputs class probabilities $j$4, soft voting computes

$j$5

The ensemble is unweighted (Rossi et al., 10 Jun 2025).

OSF’s SleepBench broadens the task space beyond staging (Shuai et al., 27 Feb 2026). Epoch-level tasks are sleep staging in four classes—Wake, Light sleep $j$6, Deep sleep $j$7, and REM—plus arousal detection, hypopnea detection, oxygen desaturation detection, and central apnea detection. At the patient level on MROS, it also evaluates coronary disease, diabetes, and hypertension using aggregated frozen-encoder embeddings (Shuai et al., 27 Feb 2026). Missing-channel evaluation is an explicit part of the benchmark design.

The OSF benchmark evaluates four self-supervised objective families, all instantiated with the same ViT encoder: contrastive learning (SimCLR), self-distillation (DINO), reconstruction-based methods (MAE and VQ-VAE), and autoregression (Shuai et al., 27 Feb 2026). It defines channel-invariant augmentations through time-wise block masking and channel masking, with 50% of channels randomly dropped per view and masking ratio $j$8 for contiguous temporal blocks (Shuai et al., 27 Feb 2026). This makes SleepBench not only a leaderboard framework but also an experimental platform for analyzing representation learning recipes under realistic physiological heterogeneity.

6. Empirical results and benchmark significance

On the Dreem datasets, human performance against consensus is reported as F1 $j$9, Accuracy $H(p)=-\sum_{c=1}^{C} p_c \log p_c,$0, and $H(p)=-\sum_{c=1}^{C} p_c \log p_c,$1 on DOD-H, and F1 $H(p)=-\sum_{c=1}^{C} p_c \log p_c,$2, Accuracy $H(p)=-\sum_{c=1}^{C} p_c \log p_c,$3, and $H(p)=-\sum_{c=1}^{C} p_c \log p_c,$4 on DOD-O (Guillot et al., 2019). SimpleSleepNet achieves F1 $H(p)=-\sum_{c=1}^{C} p_c \log p_c,$5, Accuracy $H(p)=-\sum_{c=1}^{C} p_c \log p_c,$6, and $H(p)=-\sum_{c=1}^{C} p_c \log p_c,$7 on DOD-H, and F1 $H(p)=-\sum_{c=1}^{C} p_c \log p_c,$8, Accuracy $H(p)=-\sum_{c=1}^{C} p_c \log p_c,$9, and $d_{m,n}^t = 1 - \cos(\hat{y}_m^t,\hat{y}_n^t).$0 on DOD-O (Guillot et al., 2019). The study concludes that many methods can reach human-level performance on both datasets, and that SimpleSleepNet exceeds average human performance for both healthy volunteers and patients suffering from OSA.

Common confusion patterns in the Dreem benchmark are that N1 is often misclassified as Wake or N2, and N3 is misclassified as N2, with error rates and variance increasing in OSA (Guillot et al., 2019). Single-channel performance degrades relative to multimodal PSG by 3.3 points F1 on DOD-H and 3.9 points on DOD-O, but remains near average human (Guillot et al., 2019). Transfer learning across healthy and OSA cohorts is strongly asymmetric: training on healthy and testing on OSA gives F1 $d_{m,n}^t = 1 - \cos(\hat{y}_m^t,\hat{y}_n^t).$1 versus $d_{m,n}^t = 1 - \cos(\hat{y}_m^t,\hat{y}_n^t).$2 when trained on OSA, whereas training on OSA and testing on healthy gives F1 $d_{m,n}^t = 1 - \cos(\hat{y}_m^t,\hat{y}_n^t).$3 versus $d_{m,n}^t = 1 - \cos(\hat{y}_m^t,\hat{y}_n^t).$4 when trained on healthy (Guillot et al., 2019). This was interpreted in the source as strong domain dependence, reinforcing the need for diverse cohorts and consistent protocols.

SLEEPYLAND reports that SOMNUS achieves Macro-F1 scores between 68.7% and 87.2% across 24 datasets and outperforms individual models in 94.9% of cases (Rossi et al., 10 Jun 2025). On the multi-annotator OOD datasets, SOMNUS exceeds the best human scorer, with DOD-H MF1 85.2% versus best human 80.8%, $d_{m,n}^t = 1 - \cos(\hat{y}_m^t,\hat{y}_n^t).$5, and ACS $d_{m,n}^t = 1 - \cos(\hat{y}_m^t,\hat{y}_n^t).$6, and DOD-O MF1 80.2% versus best human 75.9%, $d_{m,n}^t = 1 - \cos(\hat{y}_m^t,\hat{y}_n^t).$7, and ACS $d_{m,n}^t = 1 - \cos(\hat{y}_m^t,\hat{y}_n^t).$8 (Rossi et al., 10 Jun 2025). The framework also reports inter-model soft-agreement exceeding human soft-agreement: DOD-H humans $d_{m,n}^t = 1 - \cos(\hat{y}_m^t,\hat{y}_n^t).$9 versus models $[0.4,18]$0, and DOD-O humans $[0.4,18]$1 versus models $[0.4,18]$2 (Rossi et al., 10 Jun 2025).

OSF’s SleepBench yields three central findings: existing foundation models fail to generalize to missing channels at inference, channel-invariant feature learning is essential, and scaling sample size, model capacity, and multi-source data mixture consistently improves downstream performance (Shuai et al., 27 Feb 2026). On OOD MROS under linear probing, OSF achieves sleep staging AUC/AUPRC of 97.3/90.4, arousal 92.8/88.3, hypopnea 77.7/62.2, oxygen desaturation 81.5/81.0, and central apnea 97.3/70.7 (Shuai et al., 27 Feb 2026). Under full fine-tuning on the same cohort, it reports sleep staging 97.9/92.0, arousal 94.6/91.0, hypopnea 85.0/69.2, oxygen desaturation 83.5/83.1, and central apnea 98.1/75.5 (Shuai et al., 27 Feb 2026). Cross-cohort staging results are also reported as state of the art across all listed cohorts.

These results collectively position SleepBench as a benchmark family in which consensus-aware human comparison, cross-dataset robustness, and missing-channel stress testing are treated as first-class evaluation targets rather than afterthoughts.

7. Limitations, controversies, and future directions

Several limitations recur across the literature. In the Dreem benchmark, the total sample size is 25 healthy subjects and 55 OSA subjects, and the source explicitly notes that larger and more diverse cohorts in age, sex, comorbidities, and devices would strengthen generalization (Guillot et al., 2019). Device and montage differences between DOD-H and DOD-O affect transferability, and consensus ties remain even with five scorers per record. The study also notes that baseline models are reimplementations, so minor differences may exist relative to original publications (Guillot et al., 2019).

SLEEPYLAND identifies additional issues. Chin EMG is excluded to maximize inclusion across datasets, some cohorts lack detailed hardware and sampling documentation, pediatric AASM criteria differ from adult rules, and rare disorders remain underrepresented relative to common OSA and aging populations (Rossi et al., 10 Jun 2025). It further shows that ensemble methods improve robustness but do not remove demographic and clinical biases linked to age, gender, AHI, and PLMI, and that clinical marker estimates exhibit systematic errors such as oversmoothing and reduced sensitivity to microarousals (Rossi et al., 10 Jun 2025). Regulatory certification and integration into clinical PSG platforms are described as pending.

OSF’s SleepBench, although fully open, also has constraints (Shuai et al., 27 Feb 2026). Its standardized montage excludes some commonly used channels such as fingertip $[0.4,18]$3 waveform, disease labels may be noisy because they are derived from questionnaire and medication variables, and the corpus is primarily NSRR-based, mostly U.S., and PSG-grade. Generalization to other vendors and non-PSG settings such as home wearables remains to be validated (Shuai et al., 27 Feb 2026).

Future directions are correspondingly diverse. The Dreem-based formulation recommends montage-agnostic models, explicit domain adaptation across centers and devices, probabilistic consensus beyond majority vote, stage-wise uncertainty, and incorporation of respiratory channels when relevant (Guillot et al., 2019). SLEEPYLAND proposes standardized reporting including bias flags, human-in-the-loop workflows for uncertainty-flagged epochs, bias-aware training, subgroup fine-tuning, and exploration of shorter staging epochs and self-supervised foundations such as SleepFM (Rossi et al., 10 Jun 2025). OSF proposes stronger robustness to absent task-critical channels, expansion to additional modalities such as $[0.4,18]$4 waveform and airflow thermistors, richer subject-level outcomes, longitudinal modeling, and explicit scaling-law characterization (Shuai et al., 27 Feb 2026).

A plausible implication is that the future of SleepBench lies in convergence between these strands: the Dreem emphasis on multi-annotator consensus, the SLEEPYLAND emphasis on fair and bias-aware cross-dataset evaluation, and the OSF emphasis on fully open multi-task benchmarking for scalable sleep foundation models.