---
title: 'BirdSet Benchmark: Avian Audio & Vision'
url: https://www.emergentmind.com/topics/birdset-benchmark
type: topic
---

# BirdSet Benchmark: Avian Audio & Vision

The BirdSet Benchmark defines a suite of large-scale, standardized datasets and evaluation protocols for bird-focused audio classification and, more recently, vision-based tracking tasks. Its primary objective is to enable rigorous, repeatable benchmarking of machine learning models under realistic acoustic and visual conditions reflecting the diversity, data imbalance, and domain shift inherent in avian monitoring. The BirdSet framework encompasses both multi-label audio classification (across nearly 10,000 species and hundreds of hours of annotated soundscapes) and UAV-based multi-object tracking of wild birds, supporting both supervised and self-supervised learning paradigms, few-shot adaptation, and interpretable modeling approaches. By providing diverse and richly labeled datasets—alongside specialized metrics such as cmAP, AUROC, and SO-HOTA—BirdSet catalyzes advances in biodiversity monitoring, conservation, and applied machine learning for environmental audio and video analysis.

## 1. Dataset Structure and Composition

BirdSet’s core audio benchmark unifies nearly 10,000 avian species, over 528,000 labelled focal recordings, and 448 hours of strongly labeled soundscape evaluations within a single protocol [2403.10380]. Its main training partition (“XCL”) is drawn from the Xeno-Canto collection, comprising 528,434 weakly-annotated audio clips covering 9,734 species and around 7,200 hours. Eight passive soundscape test splits span diverse biomes (Amazon Basin, Andes, Pacific, North America, Eurasia, Pacific islands), each offering species ground-truth at 5 s granularity with multi-label annotation.

Test set examples:

| Split | #Species | #5s Clips | Hours  |
|-------|----------|-----------|--------|
| PER   |   132    |  15,120   | 21.0   |
| NES   |    89    |  24,480   | 34.0   |
| UHH   |    25    |  36,637   | 50.9   |
| HSN   |    21    |  12,000   | 16.7   |
| NBP   |    51    |     563   |  0.8   |
| POW   |    48    |   4,560   |  6.3   |
| SSW   |    81    | 205,200   | 285    |
| SNE   |    56    |  23,756   | 33.0   |

The benchmark also incorporates a background noise set (VOX) and three “focal” training splits (small, medium, full). Annotation for soundscape evaluation uses second-level bounding boxes for event onsets/offsets, transformed for task consistency into fixed 5 s multi-label clips [2403.10380].

The vision extension to BirdSet, SMOT4SB, is designed specifically for UAV-based small multi-object tracking. It consists of 211 sequences and 108,192 annotated frames at native 1920×1080 px and 3840×2160 px resolutions, with 371,690 bounding boxes and 2,240 unique tracks. Environmental and behavioral diversity is explicitly encoded: birds are tracked across a variety of habitats (forests, agricultural, urban) and conditions (lighting, flocking, UAV and subject motion) [2507.12832].

## 2. Task Definitions and Supported Evaluation Protocols

BirdSet’s canonical audio task is multi-label classification: given a 5 s clip $x$, predict binary presence/absence of each species $y\in\{0,1\}^C$. Scenarios include end-to-end supervised learning, few-shot adaptation with as few as 1–10 labeled examples per class, and self-supervised pretraining (e.g., masked prediction or contrastive objectives) [2504.12880].

In vision, the task is Small Multi-Object Tracking (SMOT) for freely moving birds against moving UAV backgrounds, with unique persistent IDs and axis-aligned bounding box annotations. The benchmark mandates strict video-level data splits, use of standard COCO formatting, and explicit train/public-test/private-test partitions to support unseen generalization [2507.12832].

BirdSet protocols emphasize domain shift: models must generalize from focal, high-SNR directional training data to omnidirectional, noisy, multi-source, multi-label test domains with often severely skewed species distributions [2511.08261].

## 3. Evaluation Metrics and Quantitative Benchmarks

Metrics are chosen for multi-label, long-tailed, and covariate-shifted settings:

- **Classwise Mean Average Precision (cmAP):**
  $$
  \mathrm{cmAP} = \frac{1}{C}\sum_{c=1}^C \mathrm{AP}_c
  $$
  where $\mathrm{AP}_c$ is area under the precision–recall curve for class $c$ [2403.10380].

- **Macro-Averaged AUROC:**
  $$
  \text{AUROC}_{\mathrm{mean}} = \frac1C\sum_{c=1}^C \mathrm{AUROC}_c
  $$

- **Macro-F1** and **label-free metrics** for robust performance assessment under extreme class imbalance.

- **SO-HOTA (Small Object HOTA)** for SMOT4SB replaces conventional HOTA’s IoU-matching with Dot Distance:
  - $$
    \mathrm{DotD}(A,B) = \exp\left(-\frac{D}{S}\right)
    $$
    where $D$ is center distance and $S$ is the root-mean-area.
  - SO-HOTA aggregates $TP_{\mathrm{DotD}}$ based detection and association at thresholds $\alpha\in\{0.05,...,0.95\}$:
    $$
    \mathrm{SO\textrm{-}HOTA}_\alpha = \sqrt{\mathrm{SO\textrm{-}DetA}_\alpha \cdot \mathrm{SO\textrm{-}AssA}_\alpha}
    $$
    $$
    \mathrm{SO\textrm{-}HOTA} = \frac{1}{19} \sum_\alpha \mathrm{SO\textrm{-}HOTA}_\alpha
    $$
    [2507.12832]

- **Calibration metrics** (Expected Calibration Error, Miscalibration Score) capture reliability under prediction uncertainty, crucial in rare-species and low-SNR conditions [2511.08261].

## 4. Baseline Models and State-of-the-Art Results

For audio, baseline and top-performing methods include:

- **Deep CNN and Transformer architectures** (EfficientNet-B3, ConvNeXt-BS, AST, ViT) [2403.10380].
- **Self-supervised models**: Bird-MAE (domain-adapted Masked Autoencoder) and AudioProtoPNet (interpretable prototype learning model) [2504.12880, 2404.10420].
- **Linear and attentive probing**, as well as parameter-efficient prototypical probing, are supported for foundation model adaptation [2508.01277].

Representative results:
- **Bird-MAE ViT-L/16**: establishes new SOTA, mAP 55.3% on the POW split, outperforming Perch by absolute gains up to +16.2% on some splits [2504.12880].
- **AudioProtoPNet**: achieves AUROC 0.842 and cmAP 0.675 on seven-split averages, exceeding Perch by +3.3% (AUROC) and +6.5% (cmAP) [2404.10420].
- **Calibration**: Perch v2 and ConvNeXt_BS exhibit global underconfidence (ECE ≈ 1%), AudioProtoPNet and BirdMAE tend towards overconfidence (ECE up to 7.6%) [2511.08261].
- **SMOT4SB baseline**: YOLOX+OC-SORT reaches SO-HOTA 9.90; challenge winner achieves 50.59 (a 5.1× improvement) via YOLOv8-SOD+YOLOv8-SMOT tracker with patch-based “SliceTrain” augmentation [2507.12832].

## 5. Domain Shift, Data Augmentation, and Best Practices

BirdSet explicitly models and quantifies domain shift: the generalization challenge from directional, single-class, high-quality training audio to complex, overlapping, multi-source, and low-SNR soundscape evaluation [2504.12880, 2511.08261]. In object tracking, the shift is from typical planar, single-object, or urban tracking (e.g., VisDrone, UAVDT) towards naturalistic, 3D “motion entanglement” (free UAV plus bird movement, flocking) [2507.12832].

Best practices established in the literature include:

- **SliceTrain**: patch-based samplers for small/faint object emphasis in vision [2507.12832].
- **Copy/Paste augmentation**: synthesis of rare poses and distraction backgrounds using real or synthetic bird cutouts.
- **Temporal and motion-aware trackers**: integration of motion-direction EMA, affine compensation, and multi-detector ensembling (adaptive weighted fusion) for robust tracking [2507.12832].
- **Audio data augmentations**: SpecAugment, MixUp, and augmentation in spectrogram or waveform space.
- **Video-level splits and submission limits**: to prevent overfitting and HARKing, maintaining valid generalization assessment [2507.12832].
- **Reporting protocols**: models must supply cmAP, AUROC, SO-HOTA, and calibration plots for comprehensive evaluation.

## 6. Interpretability and Model Adaptation

BirdSet has catalyzed the development of interpretable machine learning methods tailored for avian monitoring. Notably:

- **AudioProtoPNet** provides prototype traceability, allowing event-level and class-level human inspection via back-projection of prototype activations onto spectrograms. Local and global explanations facilitate scientific diagnosis and knowledge transfer [2404.10420].
- **Bird-MAE prototypical probing**: adapts masked autoencoder representations to few-shot and low-resource tasks, allowing parameter-efficient downstream adaptation within 3–5% mAP of full fine-tuning, and significantly exceeding linear/MLP probe performance (up to +37% mAP on select tasks) [2504.12880].
- **Calibration analysis**: class-level and split-level reliability assessments identify under- and overconfidence patterns, enabling domain-specific recalibration via (for example) Platt/temperature scaling with minimal labeled sets [2511.08261].

## 7. Position Within the Bioacoustic Benchmarking Landscape

BirdSet extends coverage far beyond AudioSet by providing a nearly 18.5× class increase (9,734 bird species vs. 527 classes in AudioSet) and a rich diversity of real-world, strongly-labeled evaluation deployments (448 h in eight splits vs. 60 h in AudioSet-strong sets) [2403.10380]. The benchmark’s unified labeling schema, focus on both strong- and weak-label settings, and its extensibility with vision-based tracking (SMOT4SB) distinguish it from earlier single-region or single-modality datasets such as LaTOT, VISO, and VisDrone [2507.12832].

By supporting robust generalization evaluation, self-supervised learning, data efficiency studies, prototype-based interpretability, and precise calibration assessment, BirdSet sets the standard for algorithmic development and comparative research in avian monitoring and related bioacoustic fields [2508.01277, 2511.08261, 2404.10420, 2507.12832].

---

**References:**  
- "BirdSet: A Large-Scale Dataset for Audio Classification in Avian Bioacoustics" [2403.10380]  
- "MVA 2025 Small Multi-Object Tracking for Spotting Birds Challenge: Dataset, Methods, and Results" [2507.12832]  
- "Can Masked Autoencoders Also Listen to Birds?" [2504.12880]  
- "AudioProtoPNet: An interpretable deep learning model for bird sound classification" [2404.10420]  
- "Foundation Models for Bioacoustics -- a Comparative Review" [2508.01277]  
- "Uncertainty Calibration of Multi-Label Bird Sound Classifiers" [2511.08261]

Source: https://www.emergentmind.com/topics/birdset-benchmark