---
title: 'SleepBench: Open Benchmark for Sleep Analysis'
url: https://www.emergentmind.com/topics/sleepbench
type: topic
---

# SleepBench: Open Benchmark for Sleep Analysis

Searching arXiv for the specified papers to ground the article in the cited literature.
SleepBench is a benchmark framework for automated analysis of sleep recordings, especially polysomnography (PSG), designed to support controlled and reproducible evaluation under heterogeneous cohorts, channel configurations, and task settings. In the literature, the term has at least two closely related instantiations. One originates from the Dreem Open Datasets and their multi-scorer benchmarking framework for sleep staging, centered on DOD-H and DOD-O and formalized around consensus scoring, subject-wise evaluation, and comparison to human technologists [1911.03221]. A later and explicitly named version defines SleepBench as a “large-scale, fully open benchmark assembled to study pre-training and scaling of sleep foundation models (FMs),” aggregating 166,500 hours of PSG recordings from nine public datasets and extending evaluation beyond sleep staging to event detection and disease prediction [2603.00190]. A broader evaluation framework, SLEEPYLAND, does not name itself SleepBench, but is presented as already instantiating many benchmark hallmarks through a 24-dataset, consensus-aware, uncertainty-informed evaluation regime [2506.08574].

## 1. Origins and conceptual scope

The earliest foundation for SleepBench is the Dreem Open Datasets study, which introduced two publicly available datasets, DOD-H and DOD-O, each scored by five sleep technologists from different sleep centers, together with a framework for comparing automated approaches to a consensus of multiple human scorers [1911.03221]. In that setting, the central problem was that automated sleep stage classifiers had usually been compared against a single human annotation despite inter-rater agreement of about 85% only. The Dreem framework therefore emphasized multi-scored evaluation, consensus labeling, weighted metrics, and subject-wise splits.

A later extension of the benchmarking idea appears in SLEEPYLAND, an open-source sleep staging evaluation framework designed to address fair model evaluation, generalization across diverse datasets, model bias, and variability in human annotations [2506.08574]. It centralizes and harmonizes approximately 220,000 hours of in-domain sleep recordings and approximately 84,000 hours of out-of-domain sleep recordings, spanning a broad range of ages, sleep-wake disorders, and hardware setups. SLEEPYLAND standardizes AASM stage labels W/N1/N2/N3/REM, evaluates single-channel and multi-channel EEG/EOG configurations, and introduces consensus-aware and uncertainty-aware metrics on multi-annotator datasets.

The most explicit formulation is given by "OSF: On Pre-training and Scaling of Sleep Foundation Models" [2603.00190], which states that SleepBench is a “comprehensive, fully open-source benchmark” built to study pre-training and scaling of sleep foundation models. In that work, SleepBench addresses real-world heterogeneity in PSG across cohorts and devices, including differences in channel montages, sampling rates, demographics, and label taxonomies. It is intended to enable controlled comparisons of self-supervised objectives, scaling behaviors, and missing-channel robustness.

Taken together, these works suggest that SleepBench is best understood not as a single immutable dataset, but as a benchmark lineage organized around standardized preprocessing, rigorous splits, consensus-aware scoring where possible, and evaluation under distributional and hardware variability.

## 2. Dataset foundations and benchmark composition

The Dreem-based benchmark core comprises two PSG datasets. DOD-H contains 25 volunteers recruited at IRBA (France), described as healthy sleepers with no sleep complaints and PSG-confirmed absence of sleep disorder [1911.03221]. DOD-O contains 55 patients with clinical suspicion for sleep-related breathing disorder, recorded at the Stanford Sleep Medicine Center, with exclusions including sleep disorders other than OSA [1911.03221]. Both were sampled at 250 Hz, scored according to the AASM Scoring Manual v2.5, and segmented into 30 s epochs. DOD-H uses a Compumedics Siesta device with 12 EEG derivations, left and right EOG, chin EMG, and ECG; DOD-O uses a Somnomedics Somno HD device with 8 EEG derivations, left and right EOG, chin EMG, and ECG [1911.03221].

The explicitly named SleepBench in OSF aggregates 166,500 hours of PSG recordings, approximately 20 million 30-second epochs, from 21,482 nights across 9 public datasets hosted by the National Sleep Research Resource (NSRR) [2603.00190]. The included datasets are SHHS, NCHSDB, WSC, CCSHS, CFS, MROS, MESA, CHAT, and SOF. Its in-domain partition consists of SHHS, NCHSDB, WSC, CCSHS, and CFS, while MROS, MESA, CHAT, and SOF are strictly held out for out-of-domain evaluation [2603.00190].

SLEEPYLAND, by contrast, covers 24 datasets, including 17 in-domain cohorts and 7 out-of-domain cohorts [2506.08574]. Its out-of-domain datasets include BSWR, DCSM, DOD-H, DOD-O, PHYS, SEDF-SC, and SEDF-ST. It includes adult, pediatric, elderly, healthy, mixed cohorts, narcolepsy-related subcohorts, severe OSA, and broad real-world clinical populations. It operates across EEG and EOG derivations and explicitly excludes chin EMG to maximize dataset inclusion [2506.08574].

The following table summarizes the three benchmark formulations described in the cited literature.

| Framework | Core scale | Primary emphasis |
|---|---:|---|
| Dreem Open Datasets framework [1911.03221] | DOD-H: 25 subjects; DOD-O: 55 subjects | Multi-scorer sleep staging benchmark with consensus evaluation |
| SLEEPYLAND [2506.08574] | approximately 220,000 ID hours and approximately 84,000 OOD hours across 24 datasets | Fair cross-dataset sleep staging evaluation, consensus, uncertainty, and bias |
| SleepBench in OSF [2603.00190] | 166,500 hours, 21,482 nights, 9 datasets | Fully open benchmark for sleep foundation models, scaling, SSL, and missing-channel robustness |

A plausible implication is that the Dreem datasets supply a consensus-rich staging benchmark, whereas OSF’s SleepBench generalizes the benchmarking objective to multi-task foundation-model evaluation under broader physiological and cohort heterogeneity.

## 3. Consensus labeling, scorer agreement, and uncertainty

A defining feature of the Dreem-based SleepBench formulation is its use of five independent scorers per record [1911.03221]. Every record is annotated by five scorers, enabling robust estimation beyond a single scorer’s variability. For DOD-H, the Soft-Agreement values of the scorers are 0.87, 0.91, 0.92, 0.84, and 0.92, leading to a consensus built from scorers 1, 2, 3, and 5; for DOD-O, the values are 0.88, 0.87, 0.88, 0.88, and 0.91, leading to a consensus built from scorers 1, 2, 4, and 5 [1911.03221]. Ties in majority voting occur in 7.3% of epochs in DOD-H and 9.9% in DOD-O.

The consensus strategy is defined per epoch by majority vote among the selected scorers, with ties broken by selecting the label from the scorer with the highest Soft-Agreement on that record [1911.03221]. Epochs are further weighted by the fraction of scorers who agreed with the consensus label. The Soft-Agreement metric is defined by
$$
v^{(t)} = \sum_{i\ne j}\hat{y}_i[:,t], \qquad
a_j(t)=\frac{v^{(t)}[y_j(t)]}{\max_k v^{(t)}[k]},
$$
and
$$
\mathrm{Soft\text{-}Agreement}_j=\frac{1}{T}\sum_{t=1}^{T} a_j(t).
$$
Its interpretation is that a value of 1.0 means scorer $j$’s label always aligns with the majority or one of tied majorities [1911.03221].

SLEEPYLAND adopts the same multi-annotator logic on DOD-H and DOD-O, using both discrete consensus hypnograms and soft-consensus hypnodensity graphs built from scorers’ probabilistic votes [2506.08574]. In that framework, consensus is defined via majority voting among the four most reliable scorers, with ties resolved by the most reliable scorer. SLEEPYLAND further introduces Averaged Cosine Similarity (ACS) between a model hypnodensity and the soft-consensus hypnodensity, and it reports that models align more with consensus than with any single scorer [2506.08574].

SLEEPYLAND also adds explicit uncertainty quantification. Entropy of the ensemble output is computed as
$$
H(p)=-\sum_{c=1}^{C} p_c \log p_c,
$$
and inter-model divergence is computed through pairwise cosine distances
$$
d_{m,n}^t = 1 - \cos(\hat{y}_m^t,\hat{y}_n^t).
$$
A logistic regression using entropy and divergence features predicts consensus disagreement, achieving mean ROC AUC 0.823 on DOD-H and 0.828 on DOD-O, outperforming entropy-only and divergence-only variants [2506.08574]. This suggests a benchmark evolution from consensus matching alone toward explicit modeling of human uncertainty.

## 4. Preprocessing, splits, and evaluation protocols

In the Dreem framework, common preprocessing used for all benchmarked models consists of a band-pass filter $[0.4,18]$ Hz, resampling to 100 Hz, clipping and dividing by 500 to remove extremes, and zero-padding at both ends to accommodate context windows [1911.03221]. A context window $k=21$ is used, corresponding to 10 past epochs, the current epoch, and 10 future epochs. The common training setup uses Adam with $\mathrm{lr}=0.001$, $\beta_1=0.9$, $\beta_2=0.999$, batch size 32, early stopping if validation accuracy does not improve for more than 15 epochs, and up to 100 epochs total [1911.03221]. DOD-H uses a leave-one-out-like split per iteration with 18 train, 6 validation, and 1 test subject; DOD-O uses a 10-fold subject-wise protocol with 37 train, 12 validation, and 6 test subjects [1911.03221].

SLEEPYLAND defines a different but equally standardized protocol [2506.08574]. All deep models use consistent splits per dataset and identical data sampling, preprocessing, augmentation, and optimization under a unified TensorFlow 2 framework. Preprocessing follows Perslev et al. (U-Sleep) with minor adjustments, no additional filtering, robust scaling per instance, on-the-fly spectrogram generation for SleepTransformer, and standardized AASM W/N1/N2/N3/REM labels. Training segments have length 35 epochs (17.5 min) for all models. Training uses a consistent batch size of 64, early stopping with patience 200, up to 10,000 training “epochs” where one “epoch” processes $10^6$ sleep segments, a cross-entropy objective, and the same augmentation with 0.1 probability of noise or channel dropout, omitting channel dropout for single-channel models [2506.08574]. Outputs are aggregated by majority vote across available channel derivations.

OSF’s SleepBench standardizes a 12-channel montage spanning brain, respiration, cardiac, and somatic groups: C3–A2, C4–A1, E1–A2, E2–A1, abdominal effort, thoracic effort, nasal pressure, snore, ECG, EMG-chin, EMG-left leg, and EMG-right leg [2603.00190]. Raw channels are resampled to target rates, each night is z-score normalized per channel, respiratory channels additionally undergo area-dependent z-score normalization, normalized signals are clipped to $[-6,6]$, and signals are segmented into non-overlapping 30-second epochs. Within epochs, all channels are resampled to 64 Hz for model input, and missing channels are zero-padded [2603.00190]. SleepBench also defines four missing-channel stress-test configurations: head-band device only, sleep disorder study, sleep micro-architecture study, and in-home screening [2603.00190].

Across these variants, a common benchmark principle is subject-level or recording-level split integrity to avoid leakage, together with preprocessing harmonization strong enough to support cross-method comparison without erasing clinically meaningful signal variation.

## 5. Tasks, metrics, and model families

The Dreem-based staging benchmark is defined on five classes, with labels $0=W$, $1=N1$, $2=N2$, $3=N3$, and $4=REM$ [1911.03221]. Reported metrics include per-class precision, recall, F1, macro- and micro-aggregations, accuracy, Cohen’s $\kappa$, and confusion matrices. Precision, recall, and F1 are defined as
$$
\mathrm{Precision} = \frac{TP}{TP + FP}, \qquad
\mathrm{Recall} = \frac{TP}{TP + FN}, \qquad
\mathrm{F1} = \frac{2\cdot \mathrm{Precision}\cdot \mathrm{Recall}}{\mathrm{Precision} + \mathrm{Recall}},
$$
with
$$
\mathrm{Macro\text{-}F1} = \frac{1}{C}\sum_{c=1}^{C}\mathrm{F1}_c,
$$
and
$$
\kappa=\frac{p_o-p_e}{1-p_e}.
$$
Metrics are computed subject-wise rather than epoch-wise, with epoch weighting by consensus vote strength [1911.03221].

SLEEPYLAND also evaluates sleep staging on five AASM classes and uses Macro-F1, per-class F1, accuracy, Cohen’s $\kappa$, and ACS, together with ROC AUC for uncertainty prediction [2506.08574]. The framework releases pre-trained models based on U-Sleep, DeepResNet, and SleepTransformer, and introduces SOMNUS, an ensemble across architectures and across channel setups using soft voting. For each epoch $t$, if model $m$ outputs class probabilities $p_m(c\mid x_t)$, soft voting computes
$$
p(y=c\mid x_t)=\frac{1}{M}\sum_{m=1}^{M} p_m(c\mid x_t).
$$
The ensemble is unweighted [2506.08574].

OSF’s SleepBench broadens the task space beyond staging [2603.00190]. Epoch-level tasks are sleep staging in four classes—Wake, Light sleep $(N1+N2)$, Deep sleep $(N3)$, and REM—plus arousal detection, hypopnea detection, oxygen desaturation detection, and central apnea detection. At the patient level on MROS, it also evaluates coronary disease, diabetes, and hypertension using aggregated frozen-encoder embeddings [2603.00190]. Missing-channel evaluation is an explicit part of the benchmark design.

The OSF benchmark evaluates four self-supervised objective families, all instantiated with the same ViT encoder: contrastive learning (SimCLR), self-distillation (DINO), reconstruction-based methods (MAE and VQ-VAE), and autoregression [2603.00190]. It defines channel-invariant augmentations through time-wise block masking and channel masking, with 50% of channels randomly dropped per view and masking ratio $r\sim U[0.3,0.6]$ for contiguous temporal blocks [2603.00190]. This makes SleepBench not only a leaderboard framework but also an experimental platform for analyzing representation learning recipes under realistic physiological heterogeneity.

## 6. Empirical results and benchmark significance

On the Dreem datasets, human performance against consensus is reported as F1 $=86.8\%\pm 7.6$, Accuracy $=86.5\%\pm 8.1$, and $\kappa=79.9\%\pm 11.3$ on DOD-H, and F1 $=84.8\%\pm 8.6$, Accuracy $=85.0\%\pm 8.9$, and $\kappa=76.5\%\pm 12.3$ on DOD-O [1911.03221]. SimpleSleepNet achieves F1 $89.9\%\pm 4.1$, Accuracy $89.9\%\pm 4.2$, and $\kappa 84.6\%\pm 6.5$ on DOD-H, and F1 $88.3\%\pm 9.0$, Accuracy $88.7\%\pm 8.2$, and $\kappa 82.3\%\pm 11.2$ on DOD-O [1911.03221]. The study concludes that many methods can reach human-level performance on both datasets, and that SimpleSleepNet exceeds average human performance for both healthy volunteers and patients suffering from OSA.

Common confusion patterns in the Dreem benchmark are that N1 is often misclassified as Wake or N2, and N3 is misclassified as N2, with error rates and variance increasing in OSA [1911.03221]. Single-channel performance degrades relative to multimodal PSG by 3.3 points F1 on DOD-H and 3.9 points on DOD-O, but remains near average human [1911.03221]. Transfer learning across healthy and OSA cohorts is strongly asymmetric: training on healthy and testing on OSA gives F1 $62.6\%\pm 14.9$ versus $88.3\%\pm 9.0$ when trained on OSA, whereas training on OSA and testing on healthy gives F1 $84.8\%\pm 7.8$ versus $89.9\%\pm 4.1$ when trained on healthy [1911.03221]. This was interpreted in the source as strong domain dependence, reinforcing the need for diverse cohorts and consistent protocols.

SLEEPYLAND reports that SOMNUS achieves Macro-F1 scores between 68.7% and 87.2% across 24 datasets and outperforms individual models in 94.9% of cases [2506.08574]. On the multi-annotator OOD datasets, SOMNUS exceeds the best human scorer, with DOD-H MF1 85.2% versus best human 80.8%, $\kappa 88.9\pm 4.7$, and ACS $0.947\pm 0.021$, and DOD-O MF1 80.2% versus best human 75.9%, $\kappa 85.4\pm 8.0$, and ACS $0.939\pm 0.024$ [2506.08574]. The framework also reports inter-model soft-agreement exceeding human soft-agreement: DOD-H humans $0.894\pm 0.028$ versus models $0.944\pm 0.019$, and DOD-O humans $0.885\pm 0.014$ versus models $0.902\pm 0.081$ [2506.08574].

OSF’s SleepBench yields three central findings: existing foundation models fail to generalize to missing channels at inference, channel-invariant feature learning is essential, and scaling sample size, model capacity, and multi-source data mixture consistently improves downstream performance [2603.00190]. On OOD MROS under linear probing, OSF achieves sleep staging AUC/AUPRC of 97.3/90.4, arousal 92.8/88.3, hypopnea 77.7/62.2, oxygen desaturation 81.5/81.0, and central apnea 97.3/70.7 [2603.00190]. Under full fine-tuning on the same cohort, it reports sleep staging 97.9/92.0, arousal 94.6/91.0, hypopnea 85.0/69.2, oxygen desaturation 83.5/83.1, and central apnea 98.1/75.5 [2603.00190]. Cross-cohort staging results are also reported as state of the art across all listed cohorts.

These results collectively position SleepBench as a benchmark family in which consensus-aware human comparison, cross-dataset robustness, and missing-channel stress testing are treated as first-class evaluation targets rather than afterthoughts.

## 7. Limitations, controversies, and future directions

Several limitations recur across the literature. In the Dreem benchmark, the total sample size is 25 healthy subjects and 55 OSA subjects, and the source explicitly notes that larger and more diverse cohorts in age, sex, comorbidities, and devices would strengthen generalization [1911.03221]. Device and montage differences between DOD-H and DOD-O affect transferability, and consensus ties remain even with five scorers per record. The study also notes that baseline models are reimplementations, so minor differences may exist relative to original publications [1911.03221].

SLEEPYLAND identifies additional issues. Chin EMG is excluded to maximize inclusion across datasets, some cohorts lack detailed hardware and sampling documentation, pediatric AASM criteria differ from adult rules, and rare disorders remain underrepresented relative to common OSA and aging populations [2506.08574]. It further shows that ensemble methods improve robustness but do not remove demographic and clinical biases linked to age, gender, AHI, and PLMI, and that clinical marker estimates exhibit systematic errors such as oversmoothing and reduced sensitivity to microarousals [2506.08574]. Regulatory certification and integration into clinical PSG platforms are described as pending.

OSF’s SleepBench, although fully open, also has constraints [2603.00190]. Its standardized montage excludes some commonly used channels such as fingertip $\mathrm{SpO}_2$ waveform, disease labels may be noisy because they are derived from questionnaire and medication variables, and the corpus is primarily NSRR-based, mostly U.S., and PSG-grade. Generalization to other vendors and non-PSG settings such as home wearables remains to be validated [2603.00190].

Future directions are correspondingly diverse. The Dreem-based formulation recommends montage-agnostic models, explicit domain adaptation across centers and devices, probabilistic consensus beyond majority vote, stage-wise uncertainty, and incorporation of respiratory channels when relevant [1911.03221]. SLEEPYLAND proposes standardized reporting including bias flags, human-in-the-loop workflows for uncertainty-flagged epochs, bias-aware training, subgroup fine-tuning, and exploration of shorter staging epochs and self-supervised foundations such as SleepFM [2506.08574]. OSF proposes stronger robustness to absent task-critical channels, expansion to additional modalities such as $\mathrm{SpO}_2$ waveform and airflow thermistors, richer subject-level outcomes, longitudinal modeling, and explicit scaling-law characterization [2603.00190].

A plausible implication is that the future of SleepBench lies in convergence between these strands: the Dreem emphasis on multi-annotator consensus, the SLEEPYLAND emphasis on fair and bias-aware cross-dataset evaluation, and the OSF emphasis on fully open multi-task benchmarking for scalable sleep foundation models.

Source: https://www.emergentmind.com/topics/sleepbench