OpenBEATs: Open-Source Audio Encoder

Updated 6 July 2026

OpenBEATs is a fully open-source general-purpose audio encoder employing masked token prediction and a ViT-style architecture to support diverse non-speech audio domains such as environmental sounds, bioacoustics, and music.
It uses an iterative tokenizer training process on 128-bin log-mel spectrogram patches and Transformer encoder blocks with multi-head self-attention to achieve scalable and reproducible performance.
Comprehensive evaluations across 25 datasets and tasks in environmental sound, bioacoustics, music, and audio reasoning demonstrate its competitive performance and practical significance for audio representation research.

Searching arXiv for “OpenBEATs” and closely related papers to ground the article in the relevant literature. arXiv search query: OpenBEATs OpenBEATs is a fully open-source general-purpose audio encoder that extends BEATs through multi-domain masked token prediction, a reproducible ESPnet pretraining pipeline, and scaling from a 90M-parameter base model to a 300M-parameter large model (Bharadwaj et al., 18 Jul 2025). In the current literature, the exact name most directly denotes this encoder. The surrounding nomenclature is not entirely uniform, however: the reproducibility platform described in 2017 is named the BEAT platform rather than “OpenBEATs,” and that paper explicitly notes that “OpenBEATs” is not its official name (Anjos et al., 2017).

1. Nomenclature and scope

OpenBEATs most properly refers to the audio encoder introduced in 2025 under the title “OpenBEATs: A Fully Open-Source General-Purpose Audio Encoder” (Bharadwaj et al., 18 Jul 2025). Its immediate lineage is BEATs, a masked-token-prediction audio representation model. The stated purpose of OpenBEATs is to make BEATs “truly accessible and extensible” by releasing the full pretraining pipeline, expanding pretraining beyond AudioSet, and evaluating the resulting representations across a broad multi-domain benchmark suite.

The term nonetheless appears in adjacent contexts. The 2017 BEAT platform is an open-source web-based system for reproducible ML and PR research, but its paper states that “OpenBEATs” is not an official term for that platform (Anjos et al., 2017). More recent literature also uses closely related BEAT/BEATS naming for systems in beat tracking, EEG acquisition, and challenge submissions. This suggests that “OpenBEATs” functions partly as a naming attractor around “open” BEAT-derived systems, but the canonical usage in audio representation learning is the 2025 encoder.

Within audio ML, OpenBEATs therefore denotes neither a beat tracker nor a general reproducibility platform. It denotes a foundation-model-style encoder for non-speech audio understanding, pretrained with masked token prediction on environmental sound, bioacoustics, and music.

2. Motivation and relation to BEATs

The immediate motivation for OpenBEATs is a pair of limitations attributed to BEATs: the absence of open-source pretraining code, and pretraining only on AudioSet (Bharadwaj et al., 18 Jul 2025). The first limitation constrained experimentation with tokenizer design, masking, and iteration schedules. The second limited downstream transfer, especially outside environmental audio.

This design choice was not merely procedural. Prior evaluation of audio foundation models for hearables showed that BEATs substantially outperformed Wav2Vec2.0, HuBERT, and WavLM on a benchmark probing scene context, speech sources, and technical acoustic properties such as DRR, RT60, and SNR (Gröger et al., 10 Feb 2025). In that study, BEATs led across all tasks, with reverberation-property advantages “as high as 30 percentage points,” despite the competing models being more speech-centric. The same work argued that foundation models trained on diverse audio collections are particularly effective for encoding environment properties relevant to hearable steering. OpenBEATs can be understood as a direct response to that observation: if BEATs-style masked token prediction is already strong, then an open implementation trained on broader audio domains should have broader utility.

The resulting project frames masked token prediction as a unifying self-supervised objective across modalities, while arguing that its application to general audio understanding had remained underexplored except for BEATs (Bharadwaj et al., 18 Jul 2025). OpenBEATs retains the BEATs paradigm but changes the two conditions most likely to determine transfer: reproducibility of the training pipeline and diversity of the pretraining corpus.

3. Architecture and pretraining pipeline

OpenBEATs operates on audio resampled to 16 kHz and transformed into 128-bin log-mel spectrograms (Bharadwaj et al., 18 Jul 2025). The spectrogram is partitioned into $16 \times 16$ time-frequency patches. A BEATs patch corresponds to about 175 ms temporal resolution, which the paper argues is better suited to non-speech audio than phoneme-scale resolution; speech is therefore excluded from the main pretraining recipe.

The encoder is ViT-style. Patches are embedded and passed through Transformer encoder blocks with multi-head self-attention and positional encodings preserving time-frequency order. Two principal configurations are reported.

Variant	Parameters	Configuration
OpenBEATs-Base	90M	hidden size 768, FFN 3072, 12 layers, 12 heads
OpenBEATs-Large	300M	hidden size 1024, FFN 4096, 24 layers, 16 heads

The model uses iterative tokenizer/encoder training. In iteration 1, masking uses tokens from a randomly initialized tokenizer; from iteration 2 onward, a trained tokenizer provides discrete targets (Bharadwaj et al., 18 Jul 2025). The tokenizer quantizes patch embeddings against a codebook of size $K=1024$ , initialized by K-Means and updated by EMA. With $\bar{\cdot}$ denoting $L_2$ normalization, token assignment is

$z_n = \arg\min_t \| \bar v_t - \bar e_n \|_2^2.$

Encoder pretraining uses masked token prediction:

$L_{\mathrm{MLM}} = - \sum_{n \in M} \log p_\theta(z_n \mid \tilde X),$

where $\tilde X$ is the partially observed patch sequence and $z_n$ is the tokenizer target for masked position $n$ .

Tokenizer training combines cosine alignment to teacher embeddings with vector-quantization stabilization terms:

$L_{\mathrm{KD}} = - \sum_n \cos(\hat o_n, o_n) + \| \mathrm{sg}[\bar e_n] - \bar v_{z_n} \|_2^2 + \| \bar e_n - \mathrm{sg}[\bar v_{z_n}] \|_2^2.$

Optimization uses AdamW for 400k updates, with 40k warmup steps. The reported learning rates are $K=1024$ 0 for OpenBEATs-Base and $K=1024$ 1 for OpenBEATs-Large (Bharadwaj et al., 18 Jul 2025). Variable-length input sequences are supported throughout.

4. Pretraining data and model variants

The main OpenBEATs paper expands pretraining from AudioSet-only BEATs to a multi-domain corpus of approximately 20k hours and 7.3M clips spanning environmental sound, bioacoustics, and music (Bharadwaj et al., 18 Jul 2025). The reported composition is as follows.

Domain	Representative datasets	Scale
Music	FMA	7.7k hours, 2.8M instances
Environmental sound	AudioSet, FreeSound, BBC Sound Effects, CochlScene, Epic-Kitchens	11.0k hours total, 4.2M instances
Bioacoustics	iNaturalist Sounds	0.8k hours, 0.3M instances

The environmental subset includes AudioSet at 5.6k hours and 2.0M clips, FreeSound at 4.1k hours and 1.7M clips, BBC Sound Effects at 1.0k hours and 0.4M clips, and smaller scene/action datasets at 0.3k hours and 0.1M clips (Bharadwaj et al., 18 Jul 2025). FreeSound and BBC Sound Effects are filtered with the WavCaps pipeline. Data are unified into a single pretraining pool; the public description does not specify a specialized domain-balancing rule beyond broad inclusion.

A later CMU–AIST challenge report extends the same lineage to a 74,187-hour corpus containing speech, music, and sound, and releases two 300M checkpoints: OpenBEATs-ICME, trained on a speech-heavy $K=1024$ 2 mixture, and OpenBEATs-ICME-SOUND, trained on a balanced $K=1024$ 3 mixture (Bharadwaj et al., 22 Jan 2026). Those models preserve the BEATs/OpenBEATs masked acoustic token prediction formulation while changing scale and domain proportions. The report states that the speech-heavy mixture improves speech tasks, while the balanced mixture tends to improve general sound and music tasks.

5. Evaluation and empirical profile

OpenBEATs is evaluated across six task types, twenty five datasets, and three audio domains, including audio reasoning tasks such as audio question answering, entailment, and captioning (Bharadwaj et al., 18 Jul 2025). The benchmark covers environmental sound, bioacoustics, reasoning, and music, with both linear probing and fine-tuning protocols.

In linear probing on X-ARES sound tasks, OpenBEATs-Large reports DESED 0.565, UrbanSound8K 0.866, FSD-50K 0.434, ESC-50 0.859, FSD2018-Kaggle 0.544, Clotho retrieval 0.046, and average 0.552 (Bharadwaj et al., 18 Jul 2025). The corresponding averages given for BEATs (iter3), Dasheng-1.2B, and EAT-L are 0.509, 0.570, and 0.510. The paper interprets this as competitive performance against a model exceeding one billion parameters, at one-fourth the size and with substantially less pretraining data.

Under full fine-tuning, OpenBEATs-Large reaches ESC-50 95.8% accuracy, FSD-50K 57.5 mAP at iter2 and 56.8 mAP at iter3, AudioSet-2M 42.2/42.1 mAP, and AudioSet-20K 32.4/33.1 mAP (Bharadwaj et al., 18 Jul 2025). The same paper notes that reproduced BEATs AudioSet numbers differ from originally reported ones because YouTube copies are non-static; all IDs are released to make that variance explicit.

The strongest relative gains are in bioacoustics and several reasoning tasks. On the BEANS suite, OpenBEATs-Large reports, among others, Watkins 88.8, Dogs 95.7, Bats 79.7, DCASE21 49.5 mAP, Gibbons 50.2 mAP, Hiceas 69.1 mAP, and Enabirds 54.4 mAP (Bharadwaj et al., 18 Jul 2025). The paper states that OpenBEATs achieves state-of-the-art performance on six bioacoustics datasets and two environmental sound datasets, and that it outperforms text-supervised audio-text models such as LAION-CLAP and BioLingual on many BEANS tasks.

For reasoning, the paper reports AudioQA accuracies of 12.7/11.5 for open questions and 59.9/58.7 for yes-no questions with OpenBEATs-Large, compared with 10.5/9.7 and 56.7/57.5 for BEATs (Bharadwaj et al., 18 Jul 2025). On Audio Entailment, OpenBEATs-Base reaches 73.2 with BERT text embeddings and 75.2 with CLAP text embeddings; OpenBEATs-Large reaches 73.2 and 74.2. On Clotho AAC captioning, however, OpenBEATs-Large scores 37.5 CIDEr, trailing BEATs at 39.0 and Dasheng-1.2B at 47.7.

Music-transfer performance is also competitive. OpenBEATs-Large reports GTZAN genre accuracy 89.1, NSynth-Instrument 81.7, and NSynth-Pitch 92.7, while BEATs reports 87.1, 79.9, and 93.1 respectively (Bharadwaj et al., 18 Jul 2025). This pattern indicates broad musical transfer with a slight deficit on one pitch-classification benchmark.

The ICME 2025 extension further shows that simple feature-level concatenation with Dasheng 1.2B can improve transfer on several tasks, including NSynth-Instruments 0.729, ESC-50 0.904, FSD18-Kaggle 0.764, GTZAN 0.898, CREMA-D 0.815, and LibriCount 0.747 (Bharadwaj et al., 22 Jan 2026). The report describes this as a late-fusion ensemble formed by upsampling Dasheng outputs to match the OpenBEATs frame count and concatenating along the channel axis.

6. Reproducibility, limitations, and significance

OpenBEATs is defined as much by its release model as by its architecture. The project releases complete pretraining code in ESPnet, evaluation scripts, pretrained and fine-tuned checkpoints, and training logs, together with pointers and YouTube IDs used for AudioSet copies (Bharadwaj et al., 18 Jul 2025). A single evaluation script is described for the 25-dataset suite. The CMU–AIST extension likewise releases trained checkpoints publicly via Hugging Face (Bharadwaj et al., 22 Jan 2026). This open release directly addresses the closed pretraining pipeline of BEATs.

Several limitations are stated explicitly. Captioning under the reported frozen-encoder-plus-transformer-decoder recipe trails both BEATs and Dasheng (Bharadwaj et al., 18 Jul 2025). Speech is excluded from the main OpenBEATs pretraining design because the 175 ms patch resolution is argued to be mismatched to speech’s finer temporal granularity. The paper also identifies AudioSet evaluation instability arising from differing YouTube copies, and leaves broader ablations on mask ratio, masking span, positional encodings, and tokenizer variants for future work.

The evidence base nonetheless supports a stable interpretation of OpenBEATs as a reproducible, extensible BEATs successor for non-speech audio representation learning. The hearables study on DEAR had already shown that BEATs-style representations can encode scene context, speech presence, and acoustic parameters such as DRR, RT60, and SNR without explicit supervision for those properties (Gröger et al., 10 Feb 2025). OpenBEATs generalizes that line of work by making the full training stack public and by demonstrating cross-domain transfer into environmental sound, bioacoustics, music, and audio reasoning (Bharadwaj et al., 18 Jul 2025). A plausible implication is that OpenBEATs occupies a transitional position between task-specific audio encoders and more general audio foundation models: it preserves a single masked-token objective, but derives much of its breadth from domain-diverse data and systematically open evaluation.