---
title: BEANS Benchmark for Bioacoustics
url: https://www.emergentmind.com/topics/beans-benchmark
type: topic
---

# BEANS Benchmark for Bioacoustics

BEANS (Benchmark of Animal Sounds) is a standardized, multi-dataset, multi-task benchmark suite for the systematic evaluation of machine learning models in computational bioacoustics. Designed to address the fragmentation of ML applications in biology, BEANS and its extension BEANS-Zero provide curated, public datasets with unified annotation and evaluation standards, supporting quantitative assessment of algorithms for animal sound classification, detection, and auxiliary behaviors such as counting and captioning. The BEANS series has catalyzed progress in generalization across taxa and tasks, and has served as the foundation for comparing both classical and foundation models in bioacoustic research [2210.12300, 2411.07186, 2512.03563, 2508.01277].

## 1. Origin, Purpose, and Evolution

BEANS was first introduced to establish a common ground for quantifying ML progress in bioacoustics by standardizing datasets, tasks, and evaluation [2210.12300]. Its motivation stems from the fragmentation and lack of comparability in bioacoustic ML studies, which historically relied on diverse, small, and inconsistent datasets. The objectives of BEANS include:

- Providing a unified collection of public datasets covering diverse animal taxa (birds, mammals, anurans, insects).
- Defining reproducible supervised tasks with fixed splits and standardized evaluation.
- Supplying open-source baselines and metric computation tools.
- Facilitating out-of-the-box development and evaluation of generic, cross-species models.

The BEANS benchmark was subsequently extended with BEANS-Zero [2411.07186], specifically advancing the field through the introduction of task diversity (multi-task, zero-shot, captioning, and counting) and challenging generalization protocols (zero-shot splits across hundreds of species).

## 2. Dataset Structure and Task Taxonomy

BEANS aggregates datasets encompassing a wide phylogenetic range and recording conditions. The original benchmark included 12 datasets—5 for classification, 5 for event detection, and 2 auxiliary non-bioacoustic datasets—for a total coverage of birds, land and marine mammals, insects, and other taxa [2210.12300, 2508.01277].

**Task Types:**
- **Classification (CLS):** Single-label, multi-class assignment to audio clips.
- **Detection (DET):** Multi-label, multi-class event detection in sliding windows over long recordings.
- **Novel tasks in BEANS-Zero:** Bird lifestage, call-type (call vs. song), open-ended captioning, and zebra finch individual counting ("zf-indv"), each with unique label types and biological relevance [2411.07186].

**BEANS-Zero expands task coverage and introduces species-level zero-shot splits:**

| Dataset/Task      | Samples  | Label Kind                | Zero-Shot?     |
|-------------------|----------|---------------------------|----------------|
| esc50 (CLS)       | 400      | 50 environmental sounds   | No             |
| watkins (CLS)     | 339      | 31 marine mammal species  | No             |
| cbi (CLS)         | 3,620    | 264 bird species          | No             |
| humbugdb (CLS)    | 1,859    | 14 mosquito species       | No             |
| dcase (DET)       | 13,688   | 20 bird/mammal species    | No             |
| enabirds (DET)    | 4,543    | 34 bird labels            | No             |
| unseen-cmn (CLS)  | 931      | 300 bird/other species    | Yes            |
| unseen-sci (CLS)  | 931      | 300 bird/other species    | Yes            |
| lifestage (CLS)   | 493      | 3 (juvenile, adult, other)| No             |
| call-type (CLS)   | 15,439   | 2 (call, song)            | No             |
| captioning (CAP)  | 29,002   | Open-ended text           | No             |
| zf-indv (CLS)     | 2,346    | 4 (# zebra finch)         | No             |

Annotation sources include expert labels, automated metadata extraction, and large language model (LLM)-generated captions [2411.07186].

## 3. Evaluation Protocols and Metrics

BEANS imposes strictly defined training/validation/test splits, prohibiting information leakage for fair comparison [2210.12300].

**Classification:**
Given $N$ test samples:
\[
\mathrm{Accuracy} = \frac{1}{N} \sum_{i=1}^N \mathbf{1}(y_i = \hat y_i)
\]

**Detection:**
Multi-label detection is evaluated using macro-averaged F1-score or mean average precision (mAP):

\[
F_{1, \mathrm{macro}} = \frac{1}{C} \sum_{c=1}^C F_1^{(c)}
\]
\[
\mathrm{mAP} = \frac{1}{11C} \sum_{c=1}^C \sum_{r\in\{0,0.1,\dots,1.0\}} P_c(r)
\]

**Captioning:**
Open-ended captions are evaluated using SPIDEr, a composite of CIDEr and SPICE metrics [2411.07186]:
\[
\mathrm{SPIDEr} = 0.5 \times (\mathrm{CIDEr} + \mathrm{SPICE})
\]

For novel zero-shot splits (e.g., unseen-cmn/unseen-sci), all labels from hundreds of species are withheld from model training to test "out-of-the-box" generalization. Post-processing matches text outputs to labels using Levenshtein distance [2411.07186].

## 4. Baseline Methods and State-of-the-Art Comparisons

Baseline evaluations in BEANS include non-neural models (e.g., SVM, XGBoost on MFCC features), standard CNNs (ResNet18/50/152), VGGish pretrained on AudioSet, and Transformer-based audio encoders [2210.12300, 2512.03563, 2508.01277]. 

Advanced evaluations leverage foundation models such as BEATs, BirdMAE, and large audio-language models (e.g., NatureLM-audio [2411.07186]).

**Key Protocols:**
- **Linear Probing (LP):** A frozen encoder's clip-level embedding is classified via a linear layer trained on BEANS [2508.01277].
- **Attentive Probing (AP):** Retains patch-wise features and applies a trainable self-attention head before classification, unlocking additional representational power in Transformer-based models.

**Performance Summary (Top-1 Accuracy, AP indicates attentive probing) [2508.01277]:**

| Model              | WTK | BAT | CBI | DOG | HUM | Avg |
|--------------------|-----|-----|-----|-----|-----|-----|
| ConvNeXt (LP)      |98.9 |93.7 |98.9 |99.3 |96.2 |97.4 |
| Perch (LP)         |98.4 |89.0 |99.0 |99.5 |95.6 |96.3 |
| BEATs$_{\rm NLM}$(AP) |99.5 |96.9 |98.9 |99.8 |97.8 |98.6|
| BirdMAE (AP)       |99.5 |96.8 |98.0 |99.3 |97.3 |98.2 |

NatureLM-audio establishes new state-of-the-art results in BEANS-Zero, including zero-shot classification (e.g., 11.6% on unseen-cmn vs. 3.4% for CLAP; 0.755 accuracy on cbi) and open-captioning (SPIDEr=0.494 vs. 0.009 for SALMONN) [2411.07186].

## 5. Model Design Insights and Adaptation Practices

BEANS benchmarks have informed several model architecture and adaptation strategies:

- Domain-general foundation models (e.g., audio Transformers with large-scale self-supervised pretraining) can match or outperform taxon-specific models in cross-taxon settings when attentive head adaptation is employed [2508.01277].
- For pure bird identification (CBI subset), supervised bird models (ConvNeXt$_{BS}$, Perch, BirdMAE) provide superior linear probing performance.
- Lightweight adaptation (training only classifier heads, LP or AP) suffices to reach strong performance, minimizing the need for full fine-tuning.
- State-space sequence models (BioMamba) offer competitive accuracy to Transformers with lower VRAM scaling, beneficial for long bioacoustic recordings [2512.03563].

## 6. Limitations, Challenges, and Future Directions

Limitations identified through BEANS and BEANS-Zero evaluations include:

- Poor performance persists for underrepresented taxa (e.g., mosquitoes, rare underwater noise) [2411.07186].
- Existing metrics may be suboptimal for heavily imbalanced or open-set recognition scenarios.
- Detection tasks with overlapping or simultaneous calls remain challenging and may require integrated source separation [2210.12300].

Proposed future directions:

- Expanding taxonomic and ecological coverage, especially toward amphibians, invertebrates, and nonsong vocalizations [2411.07186].
- Adding context and behavior labels, and tackling complex soundscapes with multiple, overlapping vocalizations.
- Introducing multi-lingual and cross-modal (audio/text) evaluation.
- Incorporating generative tasks such as "denoise and reconstruct" for probing model synthesis capacities.
- Developing more fine-grained time-localization and unsupervised discovery protocols [2210.12300, 2512.03563].

## 7. Benchmark Ecosystem, Codebase, and Community

The BEANS codebase provides dataset management scripts, preprocessing pipelines, training/evaluation recipes, and a public leaderboard. Datasets are standardized (mono/WAV, fixed sample rates, consistent splits) and annotation scripts unify label formats. The benchmark is open-source and available at https://github.com/earthspecies/beans [2210.12300].

Typical workflow involves data preparation, optional feature extraction, model training/evaluation, and leaderboard submission. The suite is designed for reproducibility, extensibility, and fair comparison, serving as an anchor for annual challenges and collaborative extension [2210.12300].

---

**References:**  
- [2210.12300] BEANS: The Benchmark of Animal Sounds  
- [2411.07186] NatureLM-audio: an Audio-Language Foundation Model for Bioacoustics  
- [2512.03563] State Space Models for Bioacoustics: A comparative Evaluation with Transformers  
- [2508.01277] Foundation Models for Bioacoustics – a Comparative Review

Source: https://www.emergentmind.com/topics/beans-benchmark