LabelBuddy: Audio Annotation Platform

• LabelBuddy is an open-source, web-based system for music and audio annotation that integrates collaborative labeling, auto-tagging, and AI-assisted pre-annotation.
• The platform decouples the user interface from model inference using Django for project management and Dockerized Flask APIs for scalable, containerized AI integration.
• It supports human-in-the-loop workflows, enabling iterative model refinement and consensus-based evaluation to enhance music information retrieval and audio-language tasks.

LabelBuddy is an open-source, web-based system for music and audio annotation that combines collaborative labeling, auto-tagging, and AI-assisted pre-annotation within a single platform. It is designed for music and broader audio tasks, including language-related description, subjective evaluation, and future agentic workflows, and it is architected to decouple the user interface from model inference through containerized backends. In the literature of Music Information Retrieval (MIR) and audio AI, it is positioned as infrastructure for human-aligned, language-centric annotation in settings shaped by modern machine learning models, Large Audio-LLMs (LALMs), and human-in-the-loop alignment workflows (Prokopiou et al., 4 Mar 2026).

1. Motivation and problem setting

LabelBuddy is introduced against a shift in MIR from static discriminative tagging toward richer generative and reasoning-based approaches. The motivating claim is that contemporary systems, including LALMs such as Music Flamingo, Qwen-Audio, and Audio Flamingo 3, require richer natural-language descriptions, multi-label and region-based annotations, and alignment with human intent, often through Reinforcement Learning from Human Feedback. The paper further identifies a lack of open-source infrastructure for subjective, language-heavy audio annotation, noting that existing tools are often task-specific, static, and fragmented across waveform editors, text platforms, and listening-test software (Prokopiou et al., 4 Mar 2026).

A second motivation is the need for human-in-the-loop pipelines in which annotators verify and improve AI suggestions rather than create labels entirely from scratch. In this formulation, annotation is not only a dataset-building activity but also part of an iterative alignment loop in which verified labels are reused for fine-tuning and model improvement. A third motivation is the “crisis of metrics” in generative music: metrics such as Fréchet Audio Distance are described as often failing to correlate with human perception, while preference collection is frequently separated from annotation proper. LabelBuddy is therefore framed as a means to integrate annotation, review, and eventual preference aggregation more tightly into RLHF-style workflows (Prokopiou et al., 4 Mar 2026).

This positioning distinguishes LabelBuddy from older audio-specific annotation utilities and from general-purpose human-in-the-loop platforms. Audio-specific open tools such as Audino, BAT, Aubio, and Gecko are described as lacking both decoupled AI-assist architectures and built-in collaborative consensus mechanisms. General HITL systems such as Prodigy and Label Studio CE are described as stronger on AI integration, but not as audio/music-specific, and collaborative or consensus-oriented features are often associated with enterprise tiers rather than open-source community editions. LabelBuddy is presented as combining four properties at once: audio specificity, decoupled AI-assist via containers, open-source availability, and collaborative consensus (Prokopiou et al., 4 Mar 2026).

2. Architecture and data model

The architectural core of LabelBuddy is the separation of interaction and inference. The frontend and application server are implemented as a Django web application, while inference is performed by one or more Docker containers exposing REST APIs through Flask. Annotators and managers interact with the Django server for project management and annotation, and when AI assistance is requested, Django serializes the audio and sends it to a Dockerized model container, which returns predictions such as tags, temporal regions, or captions. The design goal is to avoid hard-coding any MIR-specific algorithm inside Django itself; inference logic is pushed entirely into external containers (Prokopiou et al., 4 Mar 2026).

The backend data model is organized around three main entities: Projects, Users, and Tasks. Projects encapsulate datasets, configurations, tasks, and label ontologies, and may be associated with models defined through YAML. Users are governed by Role-Based Access Control with three explicit roles. A Manager has control over project setup, model configuration, task assignment, quality control, and export. An Annotator can access only assigned tasks, perform annotation, and request AI suggestions. A Reviewer accesses completed tasks for quality control and consensus formation. The paper also associates RBAC with privacy and data leakage prevention, since annotators see only relevant tasks (Prokopiou et al., 4 Mar 2026).

Tasks typically correspond to pieces of audio such as tracks or clips and can be assigned to one or multiple annotators. Task state is stored as new, in progress, completed, or reviewed. Annotations themselves are stored as JSON objects containing temporal boundaries and label metadata, including tag identifiers, free-text labels, or captions. Managers can export annotation data, including consensus results, to CSV and JSON/JSONL, and the paper explicitly states that these exported datasets can be plugged into downstream training pipelines for MIR models, LALMs, and related tasks (Prokopiou et al., 4 Mar 2026).

The containerized inference engine is configured through YAML files specifying the Docker image, input schema, output schema, and resource requirements. A representative configuration specifies an image such as "my-repo/music-flamingo:v1", an input schema such as "audio": "wav", an output schema such as a text field labeled "Caption", and resource requirements such as "gpu": "true". This scheme supports model isolation, remote or cloud deployment on AWS or Azure, and model agnosticism, since any model wrapped in Docker plus Flask and conforming to the declared schemas can be integrated (Prokopiou et al., 4 Mar 2026).

3. Annotation workflow and representational scheme

The operational workflow begins with project creation and schema definition. A manager creates a Project from the dashboard, defines the labeling ontology or text fields, uploads audio files in formats such as WAV or MP3, and assigns tasks either through a shared pool or as disjoint assignments. The frontend uses wavesurfer.js for waveform visualization, with scrolling, zooming, and editable regions defined by dragging and resizing (Prokopiou et al., 4 Mar 2026).

For annotators, the typical loop is structured but flexible. An annotator opens a task in the annotation interface, listens to the entire clip or selected sections, and may trigger AI-assisted pre-annotation through an on-demand prediction or pre-annotate button. Predictions then appear as colored waveform regions with labels and, for language tasks, as editable text fields. The annotator can adjust boundaries, add or remove labels, or edit captions to correct hallucinations. When the task is marked completed, it moves to the review interface (Prokopiou et al., 4 Mar 2026).

Review and consensus formation are central rather than auxiliary. Reviewers can inspect annotations from one or multiple annotators, examine differences in labels or captions, approve or reject individual regions, provide feedback, and resolve conflicts by choosing or merging proposals. The current system concentrates on providing infrastructure for multi-user, role-separated consensus, while formal inter-annotator agreement metrics such as Fleiss’ $\kappa$ are described as part of the future evaluation plan rather than as built-in outputs of the current version (Prokopiou et al., 4 Mar 2026).

LabelBuddy’s internal representation is deliberately flexible. JSON annotations include start_time, end_time, labels or tags, and text fields such as "Caption". Project ontologies may contain genre tags, instrument labels, descriptive attributes such as mood, tempo, and environment, and higher-level language fields such as captions or comments. The paper explicitly states that this representation supports static tag classification, region-based Sound Event Detection, multimodal audio-language alignment, and future preference labels such as A-vs-B choices. This suggests that LabelBuddy treats annotation not as a fixed tag matrix but as a configurable representational layer aligned with heterogeneous MIR and audio-language tasks (Prokopiou et al., 4 Mar 2026).

4. AI assistance, model integration, and iterative training

AI assistance in LabelBuddy is formulated as on-demand pre-annotation. After opening a task, the annotator can request a prediction; Django then serializes the audio and any required metadata, sends them to the assigned model container, and receives predictions that may take the form of tags per temporal region, temporal regions with labels, or natural-language descriptions. These outputs are rendered back into the interface as editable regions or text. The intended shift is from label creation to label verification, with the paper arguing that this reduces cognitive load and improves efficiency (Prokopiou et al., 4 Mar 2026).

The paper names several categories of supported models. Classical MIR models include YOHO, musicnn, and PANNs. LALMs include Music Flamingo, Audio Flamingo 3, and Qwen-Audio. More generally, any model that can be wrapped in Docker and Flask and made to satisfy the declared input/output schemas can participate in the system. Because the protocol is RESTful and schema-based, model integration is defined operationally rather than by commitment to a single inference framework (Prokopiou et al., 4 Mar 2026).

LabelBuddy also closes the loop between annotation and model refinement. On the Model Page, managers may monitor training metrics such as loss and accuracy, download weight files from containers, and trigger fine-tuning jobs using validated project data while specifying parameters such as number of epochs and learning rates. The workflow is explicit: pretrained models generate initial annotations; humans verify and correct them; approved labels are exported; models are fine-tuned; and improved models supply better pre-annotations in subsequent rounds. This is a pragmatic formulation of continuous human-guided adaptation rather than a one-shot annotation pipeline (Prokopiou et al., 4 Mar 2026).

The paper’s case study is music captioning. A manager defines a captioning container, for example wrapping Music Flamingo, and exposes a text output labeled "Caption". Annotators then work in a “Human-Verify” loop: instead of writing captions from scratch, they trigger pre-annotation, receive a candidate caption such as "A lo-fi hip-hop track with a slow tempo and vinyl crackle.", and edit it to fix hallucinations or add detail, for example by changing "vinyl crackle" to "rain sounds". When multiple annotators caption the same track, a reviewer compares semantic content and resolves disagreements into a consensus caption. The resulting dataset can then be exported as JSONL or CSV containing (audio_path, text_caption) pairs for audio-to-text training (Prokopiou et al., 4 Mar 2026).

The paper does not report a large-scale user study, but it outlines a planned pilot evaluation on DCASE 2024 data. The proposed measures are annotation efficiency through time-on-task with and without AI assistance, inter-annotator agreement through metrics such as Fleiss’ $\kappa$, and downstream impact through training baseline SED models and measuring improvements in PSDS or similar metrics. At present, these are described as future validation rather than published performance results (Prokopiou et al., 4 Mar 2026).

5. Position within annotation-system research

Within the narrower comparison developed in the LabelBuddy paper, the system combines properties that were presented as missing in existing audio tools: audio specificity, decoupled AI assistance, open-source availability, and collaborative consensus. The contrast with WebMUSHRA and GoListen is equally important: those systems support listening tests, but they are characterized as sitting outside the annotation pipeline and not integrating directly with tagging or model training. LabelBuddy’s intended role is to unify annotation and evaluation inside one extensible environment (Prokopiou et al., 4 Mar 2026).

In the broader annotation-systems literature, neighboring tools illuminate what LabelBuddy is and is not. Ruler, introduced under the framework of data programming by demonstration, focuses on interactive synthesis of labeling functions for document classification through span-level demonstrations rather than on audio-centric collaborative annotation (Evensen et al., 2020). CodeLabeller is a web-based platform for Java design pattern and summary annotation, emphasizing project/file hierarchy, confidence fields, and crowdsourcing-oriented task assignment rather than waveform-based audio interaction (Nazar et al., 2021). BakuFlow concentrates on semi-automatic computer-vision labeling, with label propagation between frames, a live magnifier, and YOLOE-based auto-labeling for object detection and tracking (Lin et al., 10 Jun 2025). BiLA, by contrast, addresses online aggregation of noisy crowd labels through variational Bayesian inference and worker confusion modeling, rather than domain-specific annotation interfaces (Hong et al., 2018).

This broader comparison suggests a useful characterization. LabelBuddy belongs to the family of annotation infrastructures that treat labeling as an iterative socio-technical process involving interface design, model assistance, and quality control, but its distinctive niche is the audio domain, especially the conjunction of waveform-centric interaction, language-centric outputs, and containerized inference. Unlike general weak-supervision systems, it is not primarily about synthesizing labeling functions; unlike code or vision annotation platforms, it is not built around source-file hierarchies or bounding-box propagation. Its central object is the collaboratively reviewed audio annotation task, potentially spanning temporal regions, tags, captions, and future preference data (Prokopiou et al., 4 Mar 2026).

6. Limitations, roadmap, and significance

The current system has several explicit limitations. The paper reports no large-scale evaluation, only a case study and planned experiments. The current implementation focuses on region-based tagging and captioning via pre-annotation. Subjective evaluation tools such as pairwise comparisons and Bayesian Bradley–Terry aggregation are described as planned rather than implemented. Complex musical structures such as bars, beats, and sections are not yet deeply integrated, although the architecture is described as allowing such extensions. Agentic conversational workflows are likewise part of the roadmap rather than present functionality (Prokopiou et al., 4 Mar 2026).

The future roadmap is technically specific. One direction is agentic reasoning and conversational assistance, extending the backend API so that models can respond to annotator queries and provide “Chain-of-Thought” style explanations for suggestions. Another direction is a native “Pairwise Preference” interface for A/B comparisons of clips or model outputs, with preference aggregation via Bayesian Bradley–Terry models. In the standard Bradley–Terry formulation, the probability that item $i$ is preferred over $j$ is

$$P(i \succ j) = \frac{\exp(\beta_i)}{\exp(\beta_i) + \exp(\beta_j)}.$$

The paper presents this as a building block for turning LabelBuddy into an RLHF workbench for generative music models such as MusicRL (Prokopiou et al., 4 Mar 2026).

A further planned extension is perceptual grounding through timestamp-required question answering. The motivating concern, drawn from RUListening, is that models may rely too heavily on textual priors rather than actual audio evidence. The proposed remedy is to require semantic claims to be tied to explicit time ranges, so that statements such as the presence of a saxophone solo are grounded in the audio timeline. This suggests a more stringent coupling between language output and auditory evidence, and a corresponding shift in quality control from purely semantic plausibility to temporal accountability (Prokopiou et al., 4 Mar 2026).

The significance of LabelBuddy therefore lies less in a single algorithm than in an infrastructural synthesis. It is a web-based, open-source, audio-specific annotation platform built around decoupled containerized inference, AI-assisted pre-annotation, multi-user roles, and collaborative consensus. It is explicitly designed to handle rich, language-centric, human-aligned labels rather than only static tags, and it seeks to bridge annotation and subjective evaluation, human intent and machine representation, and traditional MIR tasks and emerging LALM or agentic workflows. In that sense, LabelBuddy functions as a general-purpose annotation substrate for the next generation of multimodal, subjective, and model-coupled audio datasets (Prokopiou et al., 4 Mar 2026).