BirdSet: Avian Audio Classification Benchmark

• BirdSet is a benchmark dataset for avian bioacoustics that standardizes species-level classification under realistic passive acoustic monitoring conditions.
• It integrates weakly labeled focal recordings with expert-annotated soundscapes, harmonizing taxonomy and enabling multi-label, segment-based evaluations.
• BirdSet supports diverse research regimes including self-supervised learning, transfer learning, calibration analysis, and active learning for domain adaptation.

BirdSet is a large-scale benchmark dataset and evaluation framework for audio classification in avian bioacoustics, designed to standardize species-level classification under realistic passive acoustic monitoring (PAM) conditions. It consolidates weakly labeled focal recordings from Xeno-Canto for training, strongly labeled expert-annotated soundscape datasets for evaluation, harmonized taxonomy via eBird species codes, and a reproducible pipeline delivered through Hugging Face and a public codebase (Rauch et al., 2024). Its defining methodological role is to expose the covariate shift between focal recordings and deployment-time soundscapes, while supporting multi-label segment-based evaluation, subpopulation specialization, self-supervised learning, calibration analysis, transfer learning, and active learning (Rauch et al., 2024).

1. Origin, scope, and benchmark design

BirdSet was introduced to address fragmentation in avian bioacoustic research, where training and evaluation data were often disparate, preprocessing pipelines were ad hoc, and strongly labeled soundscape test sets were scarce (Rauch et al., 2024). The benchmark is explicitly centered on practical PAM: models are typically trained on focal, weakly labeled bird recordings and evaluated on omnidirectional soundscapes with overlapping vocalizations, environmental noise, and domain shift across habitats, devices, and geographies (Rauch et al., 2024).

The benchmark’s scope is quantitatively broad. The original dataset paper reports over 6,800 recording hours for training from nearly 10,000 classes and more than 400 hours across eight strongly labeled evaluation datasets (Rauch et al., 2024). In the detailed dataset description, the main training source is XCL, a Xeno-Canto snapshot containing 528,434 recordings and approximately 9,734 bird species after removal of CC-ND licensed items; XCM is a curated subset of 89,798 recordings spanning 409 species aligned to the evaluation sets; XCS is a 5,000-recording mini-set for rapid development; and VOX contributes 20,331 non-bird recordings for augmentation (Rauch et al., 2024). Later calibration work describes the training source as 712,433 annotations and approximately 7,200 hours, with validation and eight expert-annotated test sets used for calibration analysis, indicating a protocol view organized around annotations and 5 s segments rather than only source recordings (Schwinger et al., 11 Nov 2025).

BirdSet is both a dataset collection and a standardized protocol. The benchmark fixes evaluation around soundscape generalization, species-level labels, and threshold-free metrics such as macro mAP, cmAP, and AUROC, depending on the study (Rauch et al., 2024). This makes it suitable for multiple methodological regimes rather than a single leaderboard task. The official paper explicitly identifies multi-label classification, covariate shift, and self-supervised learning as primary use cases (Rauch et al., 2024).

2. Data sources, subsets, and annotation structure

BirdSet combines focal-recording training data with strongly labeled soundscapes. Training data come from Xeno-Canto and are weakly labeled at the recording level, whereas the evaluation soundscapes are annotated by experts with second-accurate time-frequency bounding boxes and species codes (Rauch et al., 2024). All audio is hosted as .ogg at 32 kHz, labels are standardized to eBird species codes, and metadata are harmonized across sources (Rauch et al., 2024).

The eight evaluation datasets span diverse habitats and geographies. The official dataset description gives the following subset-level summary (Rauch et al., 2024):

Subset	Duration	Species
PER	21 h	132
NES	34 h	89
UHH	~51 h	25
HSN	~16.7 h	21
NBP	~0.78 h	51
POW	~6.4 h	48
SSW	285 h	81
SNE	33 h	56

The same source also reports segment and annotation counts. PER contains 14,798 bounding boxes and 15,120 five-second segments; NES contains 6,952 labels and 24,480 segments; UHH contains 59,583 labels and 36,637 segments; HSN contains more than 10,000 labels and 12,000 segments; NBP contains 563 segments; POW contains 16,052 annotations and 4,560 segments; SSW contains 50,760 annotations and 205,200 segments; and SNE contains 20,147 annotations and 23,756 segments (Rauch et al., 2024). Dedicated Xeno-Canto training subsets are also defined per evaluation set, with sizes such as 16,802 XC clips for PER, 16,117 for NES, 3,626 for UHH, 5,460 for HSN, 24,327 for NBP, 14,911 for POW, 28,403 for SSW, and 19,390 for SNE (Rauch et al., 2024). The comparative review restates these dedicated-train counts and describes BirdSet as combining XCL, XCM, and per-test dedicated training sets, with aggregate reporting often excluding POW because it is treated as auxiliary or validation in some studies (Schwinger et al., 2 Aug 2025).

BirdSet’s label granularity is species-level and inherently multi-label at the segment level: a five-second segment may contain zero, one, or multiple species (Rauch et al., 2024). Overlapping calls are handled by segment labeling rules, and the soundscape annotations additionally retain event timestamps and bounding boxes so that BirdSet can support both multi-label segment classification and event-oriented analyses (Rauch et al., 2024). This structure is central to later work: AudioProtoPNet treats BirdSet as a multi-task, multi-label benchmark for cross-region generalization (Heinrich et al., 2024); calibration work emphasizes overlapping vocalizations, long-tailed species distributions, and the shift from directional focal recordings to omnidirectional soundscapes (Schwinger et al., 11 Nov 2025); and active learning work uses fixed 5 s PerchV2 embeddings on BirdSet subsets HSN, POW, and UHH, stressing variation in label density and class cardinality across structured terrestrial soundscapes (Dubus et al., 6 Jul 2026).

3. Evaluation protocols, preprocessing conventions, and metrics

BirdSet’s canonical evaluation protocol is segment-based and threshold-independent. The benchmark adopts five-second segments without overlap, following BirdCLEF conventions, while also exposing raw event timestamps for alternative analyses (Rauch et al., 2024). Multi-label classification is the dominant formulation, and the original paper foregrounds macro-averaged AP and AUROC for fair comparison under class imbalance and domain shift (Rauch et al., 2024).

The official paper states the multi-label binary cross-entropy objective as

$$\mathcal{L}_{\mathrm{BCE}} = - \frac{1}{N} \sum_{n=1}^{N} \sum_{c=1}^{C} \big[ y_{n,c} \log(\hat{p}_{n,c}) + (1 - y_{n,c}) \log(1 - \hat{p}_{n,c}) \big].$$

It also defines class-wise average precision and macro mean average precision through

$$AP_c = \sum_{k} (R_k - R_{k-1}) P_k,\qquad \mathrm{mAP} = \frac{1}{C} \sum_{c=1}^{C} AP_c,$$

and summarizes AUROC via threshold-dependent TPR and FPR, with

$$\mathrm{TPR} = \frac{TP}{TP + FN},\qquad \mathrm{FPR} = \frac{FP}{FP + TN}.$$

Thresholded metrics such as precision, recall, and F1 are included for completeness, but BirdSet emphasizes thresholdless metrics for robust cross-dataset comparability (Rauch et al., 2024).

Subsequent work uses several closely related metric conventions. The probing study on bioacoustic transfer reports macro-averaged mean average precision on BirdSet and interprets it as reducing bias toward common species by averaging AP per class or species (Miron et al., 11 May 2026). Calibration work uses class mean Average Precision (cmAP) together with threshold-free calibration metrics including ECE, MCS, OCS, and UCS, computed globally, per dataset, and per class (Schwinger et al., 11 Nov 2025). The comparative review instead treats macro-AUROC as the primary BirdSet aggregate metric, with cmAP5 in the appendix, and excludes POW from aggregate BirdSet scores because it is used as auxiliary validation (Schwinger et al., 2 Aug 2025). Perch 2.0 reports ROC-AUC, cmAP, and Top-1 Accuracy and argues that ROC-AUC is “the most stable and informative” BirdSet metric for its analysis (Merriënboer et al., 6 Aug 2025). This variation does not change the underlying multi-label soundscape problem, but it does mean that cross-paper comparisons depend on the precise protocol being followed.

BirdSet deliberately leaves spectrogram parameterization to the model or study, shipping raw audio plus event-detection metadata rather than imposing a single frontend (Rauch et al., 2024). Later evaluations therefore differ in input conventions. The comparative review evaluates models at their native preprocessing settings, such as 32 kHz mel spectrograms for BirdMAE and ConvNeXtBS, PCEN mel spectrograms for Perch, and 16 kHz BEATs-style spectrograms for BEATs-derived encoders (Schwinger et al., 2 Aug 2025). Probing work notes that CNN encoders operate on spectrogram inputs, while transformer encoders may operate on raw or featureized audio according to pretraining, without BirdSet-specific frontend modifications beyond probe-side alignment (Miron et al., 11 May 2026). AudioProtoPNet likewise focuses on spectrogram-based modeling but does not specify STFT or mel parameters in the reported text (Heinrich et al., 2024).

4. Training regimes and benchmark usage across studies

BirdSet supports several distinct training scenarios. The official paper identifies three major regimes: universal pretraining on XCL, subpopulation specialization using XCM or per-test dedicated XC subsets with logit restriction, and robustness-oriented augmentation with VOX backgrounds, no-call segments, mixup, and event-centered extraction (Rauch et al., 2024). This structure has been adopted and specialized differently across later work.

One recurring regime is frozen-encoder transfer. The probing study freezes base encoders and trains only probe parameters on BirdSet, with fully fine-tuned models serving as a top-line condition (Miron et al., 11 May 2026). “Last-layer” probes use only the final block or layer, whereas “all-layer” probes use the last layer of every block, with 11 layers for BEATs and BirdAVES, 10 for EAT, and 15 for EfficientNet (Miron et al., 11 May 2026). To combine heterogeneous internal representations, learned adapters map layer outputs to a unified time-by-feature format, after which layer aggregation and temporal pooling are applied (Miron et al., 11 May 2026).

A second regime is direct evaluation of pretrained classification heads. Perch 2.0 explicitly forgoes BirdSet fine-tuning and evaluates the pretrained prototype-learning classifier head directly, using frozen embeddings and BirdSet’s official metrics (Merriënboer et al., 6 Aug 2025). In aggregate, it reports AUROC 0.908, cmAP 0.431, and Accuracy 0.665 for the random-window self-distilled variant, outperforming Perch 1.0 and several strong baselines in AUROC without any BirdSet fine-tuning (Merriënboer et al., 6 Aug 2025). The comparative review later argues that training even a small downstream linear classifier on BirdSet is systematically better than restricted zero-training evaluation for models such as Perch and ConvNeXtBS (Schwinger et al., 2 Aug 2025).

A third regime is domain-specific self-supervised pretraining followed by either fine-tuning or frozen-feature adaptation. Bird-MAE replaces AudioSet pretraining with BirdSet-derived XCL-1.6M, adjusts image size, decoder, masking ratio, epochs, batch size, normalization, and pretraining mixup, and then evaluates either full supervised fine-tuning or frozen-feature probing on BirdSet’s downstream tasks (Rauch et al., 17 Apr 2025). The paper reports state-of-the-art MAP across all eight datasets under fine-tuning with a prototypical head and strong gains from prototypical probing in the frozen-encoder setting (Rauch et al., 17 Apr 2025).

BirdSet is also used beyond pure classification transfer. FINCH augments a frozen BEATs-based audio classifier with a metadata-only contextual predictor using latitude, longitude, day-of-year, and hour-of-day features (Ovanger et al., 3 Feb 2026). Calibration work treats BirdSet as a multi-label uncertainty benchmark and studies post hoc correction using small calibration sets (Schwinger et al., 11 Nov 2025). Active learning work places BirdSet inside the BioDCASE 2026 Task 4 BaseAL framework, using fixed PerchV2 embeddings, a small multilabel head, and iterative batch acquisition under a fixed annotation budget (Dubus et al., 6 Jul 2026). A plausible implication is that BirdSet has become not just a static dataset, but an infrastructure benchmark for studying transfer, calibration, fusion, interpretability, and annotation efficiency under a shared ecological task formulation.

5. Representation learning, probing, and transfer findings

BirdSet has been central to a sequence of studies on how audio representations transfer to bioacoustic multi-label soundscape classification. A key finding is that the downstream head matters as much as the encoder.

The attentive probing study shows that current benchmarks may misrepresent encoder quality when relying on a last-layer linear probe (Miron et al., 11 May 2026). On BirdSet, multi-layer probing improves macro mAP for all tested models; across transformer encoders, all-layer probing improves macro mAP by about +0.03 absolute over last-layer probing, while EfficientNet benefits by about +0.02 mAP (Miron et al., 11 May 2026). Attention-based probes outperform linear probes for transformer encoders including BEATs, EAT, BirdAVES, and NatureBEATs, whereas EfficientNet does not benefit from attention, likely because CNN features are less explicitly temporally structured and adapter costs are much larger (Miron et al., 11 May 2026). The paper attributes this to distributed species-relevant information across layers and to the loss of temporal structure under simple clip-level averaging (Miron et al., 11 May 2026). For BirdSet specifically, multi-layer attentive probing is recommended for transformer encoders, especially self-supervised ones (Miron et al., 11 May 2026).

The comparative review reaches a consistent conclusion at the model-selection level. Using frozen encoders and either linear probing or attentive probing, it reports BirdMAE as the strongest aggregate BirdSet model under attentive probing with macro-AUROC 86.54, while ConvNeXtBS and Perch are strongest under linear probing with 85.75 and 85.63 respectively (Schwinger et al., 2 Aug 2025). The same study shows large AP-induced improvements for transformers: BirdMAE rises from 77.66 to 86.54 AUROC, BEATs from 72.70 to 82.28, and BEATsNLM from 80.10 to 84.55 when attentive pooling replaces simple linear probing (Schwinger et al., 2 Aug 2025). CNN attentive probing did not help ConvNeXtBS in that setup (Schwinger et al., 2 Aug 2025). This suggests that BirdSet strongly rewards spatially or temporally aware token aggregation when the encoder is transformer-based.

Bird-MAE pushes this line further by specializing masked autoencoder pretraining to BirdSet rather than relying on general-domain Audio-MAE pipelines (Rauch et al., 17 Apr 2025). With BirdSet-derived pretraining and domain-tailored augmentations, Bird-MAE-L with a prototypical head reports MAP values such as 55.26 on POW, 55.28 on HSN, 34.64 on PER, 41.50 on NES, 30.17 on UHH, 71.69 on NBP, 40.82 on SSW, and 33.82 on SNE, surpassing Audio-MAE and several supervised baselines in the reported comparisons (Rauch et al., 17 Apr 2025). In the frozen-feature setting, prototypical probing dramatically improves over linear probing, for example from 12.44 to 49.03 MAP on HSN for Bird-MAE-L, narrowing the gap to fine-tuning to 6.25 points on that task (Rauch et al., 17 Apr 2025). The paper summarizes that prototypical probes outperform linear probing by up to 37 percentage points in MAP and narrow the gap to fine-tuning to approximately 3.3 percentage points on average across BirdSet tasks (Rauch et al., 17 Apr 2025).

Perch 2.0 establishes a complementary result from the supervised pretraining side. Without any BirdSet fine-tuning, the random-window self-distilled variant attains mean BirdSet AUROC 0.908, cmAP 0.431, and Top-1 Accuracy 0.665, improving markedly over Perch 1.0 and surpassing reported baselines in AUROC (Merriënboer et al., 6 Aug 2025). The paper attributes these gains to multi-taxa supervised pretraining, prototype-learning self-distillation, source-prediction training, and a generalized multi-source mixup scheme (Merriënboer et al., 6 Aug 2025). This suggests that BirdSet is sensitive both to sophisticated transfer heads and to the quality of pretraining objectives.

6. Interpretability, calibration, metadata fusion, and active learning

BirdSet has also become a substrate for methodological work beyond raw discrimination performance.

AudioProtoPNet adapts ProtoPNet to BirdSet’s multi-label bird sound classification problem by replacing the standard classification layer with a prototype-learning classifier on top of a ConvNeXt backbone (Heinrich et al., 2024). The model learns class-specific prototypes in latent spectrogram space, compares them with normalized spatial patches using cosine similarity, aggregates them by global max pooling, and produces multi-label sigmoid outputs through a fixed per-class prototype connection structure (Heinrich et al., 2024). Interpretability is provided through similarity maps and projection of prototypes to nearest training patches, with highlighted diagnostic regions based on the smallest rectangle containing pixels above the 95th percentile of activation (Heinrich et al., 2024). The abstract reports that the model was trained on the BirdSet training dataset containing 9,734 bird species and over 6,800 hours of recordings, and that it outperformed Perch with average AUROC 0.90 and cmAP 0.42 across the seven BirdSet test datasets (Heinrich et al., 2024).

Uncertainty calibration has been studied systematically on BirdSet as well. The calibration benchmark evaluates Perch v2, ConvNeXt$_{BS}$, AudioProtoPNet-20, and BirdMAE-XCL using threshold-free calibration metrics alongside cmAP (Schwinger et al., 11 Nov 2025). Globally across all test datasets, it reports cmAP values of 41.36 for AudioProtoPNet, 39.16 for Perch v2, 35.57 for BirdMAE, and 32.20 for ConvNeXt$_{BS}$; ECE values of 0.92, 1.10, 5.25, and 7.62 respectively; and MCS values showing that Perch v2 and ConvNeXt$_{BS}$ are underconfident while AudioProtoPNet and BirdMAE are overconfident (Schwinger et al., 11 Nov 2025). The study finds substantial dataset-specific variability, notes that global aggregation can mask miscalibration, and shows that a small labeled calibration set can materially improve calibration, with per-class Platt scaling on 10 minutes of labeled audio per dataset often giving the most consistent gains (Schwinger et al., 11 Nov 2025).

BirdSet also supports contextual fusion with spatiotemporal metadata. FINCH fuses a frozen BEATs-based audio classifier with a metadata-only BirdSet contextual predictor trained on

$$f_{\text{meta}} = \Big[\sin\!\left(\tfrac{2\pi d}{365}\right), \cos\!\left(\tfrac{2\pi d}{365}\right), \sin\!\left(\tfrac{2\pi h}{24}\right), \cos\!\left(\tfrac{2\pi h}{24}\right), \tfrac{\text{lat}}{90}, \tfrac{\text{lon}}{180}\Big],$$

using adaptive log-linear evidence fusion (Ovanger et al., 3 Feb 2026). The core fusion rule is

$$\log \tilde{p}_{\omega}(y\mid x,s) = \log p_{\theta}(y\mid x) + \omega(x,s)\,\log p_{\psi}(y\mid s),$$

with a bounded, per-sample gate and an explicit audio-only fallback when $\omega(x,s)=0$ (Ovanger et al., 3 Feb 2026). On BirdSet subsets PER, NES, UHH, and SSW, FINCH shows mixed but sometimes strong subset-level gains, such as UHH performance of 0.927 AUROC, 0.536 cmAP, and 0.747 Top-1 Accuracy, while performing poorly on SSW retrieval and detection, reflecting the heterogeneous utility of weak metadata-only priors across BirdSet subsets (Ovanger et al., 3 Feb 2026). The paper emphasizes that BirdSet context is weak and heterogeneous because AdaSTEM priors are unavailable in this setting (Ovanger et al., 3 Feb 2026).

Finally, BirdSet has been used to study annotation efficiency. The active learning paper evaluates an adaptive diversity-uncertainty strategy with Maximum Marginal Relevance on HSN, POW, and UHH using fixed PerchV2 embeddings and a total annotation budget of 500 samples per subset (Dubus et al., 6 Jul 2026). Uncertainty is defined as summed Bernoulli entropy across classes, diversity as nearest-labeled squared Euclidean distance in embedding space, and acquisition as

$$s_i(t) = \alpha_t \,\tilde{h}_i + \big(1 - \alpha_t\big)\,\tilde{d}_i,$$

with an adaptive schedule and redundancy control via greedy MMR (Dubus et al., 6 Jul 2026). The method improves AULC and best macro mAP over Random, Margin-only, and CoreSet, with especially large gains on HSN: AULC 0.632 versus 0.556 for CoreSet and 0.382 for Random, and best macro mAP 0.716 versus 0.696 and 0.477 respectively (Dubus et al., 6 Jul 2026). This suggests that BirdSet’s structured terrestrial soundscapes are well suited to acquisition strategies that combine uncertainty, diversity, and redundancy control.

7. Limitations, protocol effects, and benchmark significance

BirdSet’s central strength is that it operationalizes realistic domain shift, but that same design introduces several recurrent caveats. The training data are weakly labeled focal recordings, with possible unlabeled background birds and heterogeneous recording practices, while the evaluation data are strongly labeled soundscapes from omnidirectional microphones in site-specific environments (Rauch et al., 2024). This induces the intended shift $$AP_c = \sum_{k} (R_k - R_{k-1}) P_k,\qquad \mathrm{mAP} = \frac{1}{C} \sum_{c=1}^{C} AP_c,$$0 and also creates label-noise and calibration challenges (Rauch et al., 2024).

Class imbalance is intrinsic: species prevalence differs strongly across sites and globally, and later work quantifies long-tailed behavior through imbalance indices such as $$AP_c = \sum_{k} (R_k - R_{k-1}) P_k,\qquad \mathrm{mAP} = \frac{1}{C} \sum_{c=1}^{C} AP_c,$$1 for the XCL train split and dataset-specific values between 0.54 and 0.92 across evaluation sets (Schwinger et al., 11 Nov 2025). Macro-averaged metrics partly mitigate this, but calibration work notes that rare-species calibration estimation can be unstable and that stronger per-class analysis is needed (Schwinger et al., 11 Nov 2025). Probe studies likewise note that benchmark conclusions can depend heavily on downstream head design, training budget, and layer access; BirdNET and Perch could not be included in some inner-layer probing analyses because intermediate representations were not extractable in their TensorFlow implementations (Miron et al., 11 May 2026).

Protocol differences across papers are also consequential. Some studies report all eight evaluation datasets, whereas others treat POW as validation or auxiliary and exclude it from aggregate test metrics (Schwinger et al., 2 Aug 2025). Some evaluate pretrained heads directly (Merriënboer et al., 6 Aug 2025), others retrain only probes (Miron et al., 11 May 2026, Schwinger et al., 2 Aug 2025), and others perform full fine-tuning with extensive domain augmentations (Rauch et al., 17 Apr 2025). Metrics vary among macro mAP, cmAP, AUROC, cmAP5, and Top-1 Accuracy (Rauch et al., 2024, Merriënboer et al., 6 Aug 2025, Schwinger et al., 2 Aug 2025, Schwinger et al., 11 Nov 2025). This does not undermine BirdSet’s role, but it does mean that BirdSet results are most informative when the exact task, split handling, and adaptation protocol are specified.

Within these constraints, BirdSet has become a reference benchmark for avian bioacoustic classification. It supports model development ranging from supervised CNNs and transformer encoders to prototype-based interpretable systems, metadata fusion methods, calibration procedures, and active learning strategies (Rauch et al., 2024, Heinrich et al., 2024, Ovanger et al., 3 Feb 2026, Dubus et al., 6 Jul 2026). The later probing and review literature converges on a broader methodological lesson: BirdSet does not merely rank encoders; it reveals how representation quality, probe design, temporal aggregation, calibration, and domain adaptation interact under realistic PAM deployment conditions (Miron et al., 11 May 2026, Schwinger et al., 2 Aug 2025).