ISPA: Inter-Species Phonetic Alphabet for Transcribing Animal Sounds (2402.03269v1)

Abstract: Traditionally, bioacoustics has relied on spectrograms and continuous, per-frame audio representations for the analysis of animal sounds, also serving as input to machine learning models. Meanwhile, the International Phonetic Alphabet (IPA) system has provided an interpretable, language-independent method for transcribing human speech sounds. In this paper, we introduce ISPA (Inter-Species Phonetic Alphabet), a precise, concise, and interpretable system designed for transcribing animal sounds into text. We compare acoustics-based and feature-based methods for transcribing and classifying animal sounds, demonstrating their comparable performance with baseline methods utilizing continuous, dense audio representations. By representing animal sounds with text, we effectively treat them as a "foreign language," and we show that established human language ML paradigms and models, such as LLMs, can be successfully applied to improve performance.

• The paper introduces the ISPA framework, converting animal sounds into text to standardize bioacoustic transcription.
• It details two methods, ISPA-A and ISPA-F, which leverage acoustic features and clustering with Viterbi decoding.
• Experimental results show ISPA-F using AVES rivals continuous audio models and enables efficient integration with language models.

Inter-Species Phonetic Alphabet: A Novel Approach to Transcribing Animal Sounds

The paper "ISPA: Inter-Species Phonetic Alphabet for Transcribing Animal Sounds" introduces a novel framework for bioacoustic transcription, leveraging methodologies from human language processing. The proposed Inter-Species Phonetic Alphabet (ISPA) aims to provide a standardized, interpretable, yet concise system for transcribing animal vocalizations into text, offering a departure from traditional, dense, and continuous audio representations.

Motivation and Concept

Traditional bioacoustic analysis has predominantly employed spectrograms and sonograms to visualize and analyze animal sounds. However, the lack of a concise and standardized transcription system analogous to the International Phonetic Alphabet (IPA) for human speech has limited further linguistic-style inquiries and applications using current LLMs. ISPA seeks to address these limitations by treating animal sounds "as a foreign language."

The transcription process converts audio recordings into text, allowing the use of Machine Learning (ML) techniques more commonly applied to natural languages. Through the newly developed ISPA, both acoustics-based and feature-based transcription methods are explored.

Methods and Techniques

The paper describes two main methodologies:

1. ISPA-A (Acoustics-Based Transcription):
   • ISPA-A translates acoustic features into text by encoding spectral bandwidth, pitch, length, and pitch slope into concise tokens. This approach is akin to automatic music transcription (AMT), focusing on musical notes but adapted for animal sounds, accounting for more frequent pitch variations and the need to encode timbral differences vital for animal communication.
2. ISPA-F (Feature-Based Transcription):
   • This method utilizes audio feature representations, such as MFCC and AVES, transformed into text tokens. Through segmentation and k-means clustering, ISPA-F creates discrete interpretable segments that may not be directly read from acoustic data. By integrating a Viterbi algorithm, continuous feature vectors are converted into sequences of text that a LLM can process.

Experimental Results

The experiments conducted evaluated ISPA-encoded text representations on bioacoustic classification tasks using various datasets representative of environmental and animal sounds—ranging from ESC50 and the Watkins Marine Mammal Sound Database to Egyptian fruit bat calls and humbug mosquitoes.

Remarkable results were recorded for the ISPA-F approach, particularly when paired with the AVES feature set, which approached or even exceeded baseline models based on continuous audio input. The experiments also demonstrated that pre-trained LLMs like RoBERTa can efficaciously harness ISPA transcripts, especially post-finetuning on large datasets such as FSD50K.

Implications and Future Directions

The paper emphasizes ISPA's potential beyond classification tasks. By framing bioacoustic sounds in text format, ISPA encourages the exploration of multimodal processing, improved detection capabilities, audio-captioning applications, and even generative models. This paradigm shift could transform bioacoustic studies akin to recent advancements in NLP. While classification was the main focus of this work, future endeavors could delve into synthesis tasks like audio generation from the transcribed text.

The introduction of ISPA transforms the handling of animal sounds, facilitating the application of LLM architectures in bioacoustics. As ML models continue to evolve, further refinements in transcription fidelity and adaptability across species could augment ecological research and species communication studies, reinforcing ISPA as a significant tool for interdisciplinary applications across AI and animal communication research.