voxmap-studio: An open-source speaker diarization annotation tool with built-in cost instrumentation

Abstract: Labeling speaker diarization data is costly, yet annotation tools rarely measure that cost. We present voxmap-studio, an open-source, React-based diarization annotation tool integrated with the pyannote-based diarization ecosystem. Its canvas is initialized by a fast stride-accelerated diarization engine so that the annotator corrects a hypothesis rather than drawing every speaker turn by hand, and the tool records annotation cost - typed edit-operation counts and time - as a first-class output, enabling quantitative comparison of how much different forms of assistance actually help. Export is gated on per-segment human confirmation and guarded by injected "phantom" attention checks, which prevent unverified automatic output from being released as ground truth. In a preliminary study on nine AMI audio files, unassisted manual annotation was the costliest and least accurate, and automatic initialization shifted the work from creating turns to correcting them; highlighting uncertain segments gave the lowest cost in our small sample. The tool and its instrumentation are open source.

• The paper introduces voxmap-studio, which quantifies annotation cost by tracking edit operations and editing time to refine diarization workflows.
• The methodology combines automatic initialization with human confirmation and attention checks to ensure annotation integrity and cost efficiency.
• Empirical results reveal that uncertainty highlighting minimizes annotation errors and cost, while bulk gallery operations may unexpectedly increase operational effort.

Comprehensive Analysis of voxmap-studio: An Open-Source Speaker Diarization Annotation Tool with Cost Instrumentation

Motivation and Contributions

Speaker diarization data annotation is labor-intensive, driving up the cost and limiting the availability of high-quality labeled corpora. Existing annotation frameworks, notably gryannote, facilitate the correction of automatic diarization hypotheses but do not quantify the annotation cost, leaving critical gaps in workflow optimization and empirical evaluation. The paper introduces voxmap-studio (2606.26842), which explicitly instruments annotation cost (i.e., edit operation counts and editing time), thereby enabling rigorous comparison and refinement of human-in-the-loop annotation strategies. Key technical contributions include:

• Full instrumentation of annotation cost with typed edit operations and time as first-class outputs.
• Workflow combining automatic initialization with confirmation-gated and attention-checked export, guaranteeing annotation integrity.
• Empirical evaluation demonstrating cost reductions and accuracy improvements tied to system-assistive features.

System Architecture and Workflow

voxmap-studio is implemented as a browser-based, React application, departing from autogenerated Python GUIs to optimize for precise waveform editing and fluid user interaction. The annotation interface is initialized by a stride-accelerated diarization pipeline, leveraging pyannote segmentation models for low-latency, real-time factor initialization on commodity hardware. Annotators correct the initial hypothesis rather than annotating from scratch, significantly reducing manual effort.

Figure 1: The voxmap-studio interface visualizes automatic output, allows detailed editing, tracks annotation cost, and exports RTTM annotations.

The interface supports granular segment manipulation (resize, split, delete, reassign), facilitated by keyboard shortcuts and interactive speaker galleries. Label assistance is provided via segment similarity-based uncertainty highlighting, identifying likely intrusions and borderline assignments. Gallery and recommendation functions expedite bulk operations, clustering segments by speaker and suggesting optimal assignments.

The tool records every edit operation and editing time, exporting these metrics in a JSON sidecar for downstream analysis. Export gating via per-segment confirmation and injected phantom attention checks prevents unverified automatic output from entering the ground-truth corpus, ensuring annotation integrity. Phantoms are silent-gap inserts flagged for annotators, measuring attention and preventing careless rubber-stamping.

Cost Instrumentation and Empirical Study

voxmap-studio’s hallmark is quantitative cost tracing. The tool meticulously records editOps (operation counts, reflecting user input, not affected segments), editing time per audio minute, fraction of audio listened at full speed, and attention check outcomes.

The paper presents a preliminary cost study on nine AMI audio files with three annotation conditions:

• C1 (Manual): No automatic initialization; annotators generate all speaker turns manually.
• C2 (Engine + Uncertainty): Automatic initialization with uncertainty highlighting.
• C3 (Engine + Gallery + Recommend): All C2 features plus gallery-based bulk labeling and embedding-based recommendations.

Results demonstrate that manual annotation (C1) incurs substantially higher editOps (761) and macro DER (0.177) compared to automatic initialization with uncertainty highlighting (C2: 278 editOps, 0.079 DER). Notably, gallery and recommendation functions (C3) did not yield lower cost than uncertainty highlighting alone (C2), with editOps rising to 418 and DER at 0.093. This inversion is attributed to a shift in operational effort toward hypothesis correction (resize/split/delete/reassign) post-initialization, and the increased interaction cost of gallery-based edits.

Figure 2: Distribution of edit operations per annotation condition, highlighting the transition from creation-dominated manual annotation to correction-centric semi-automatic workflows.

The cost instrumentation enables rigorous delineation of workflow properties: C1 is dominated by creation (81% of editOps), while C2/C3 pivot toward correction. The uncertainty-highlight condition minimizes both cost and error rate within the experimental setting.

Implications for Diarization Annotation and AI Systems

The explicit instrumentation of annotation cost introduces an empirical framework for evaluating annotation workflows, challenging assumptions about the utility of assistive features. The findings that bulk gallery/recommendation operations can result in increased annotation cost, contrary to intuitive expectations, signal the necessity for empirical optimization and the potential for assistive features to be counterproductive if not tuned for annotator efficiency.

Confirmation gating and attention checks enforce annotation provenance, preventing leakage of unverified automatic output and establishing higher corpus integrity standards. The export workflow, embedding per-segment confirmation attributes and integrity hashes, facilitates robust reference annotation and downstream reproducibility.

The tool’s modular design and exported cost metrics lay the groundwork for broader adoption, enabling cross-session speaker asset integration and extension to new tasks such as speaker-attributed recognition. Efforts toward back-end portability (optimized CPU/AMD/NPU inference) address practical deployment constraints in heterogeneous computing environments.

Prospects for Future Research

The portability of the backend, integration of cross-session speaker assets, and extension beyond diarization represent promising directions. Quantitative cost instrumentation can be leveraged to optimize annotation pipelines, adapt assistive features dynamically, and scale human-in-the-loop annotation in large-scale speech corpora. In the broader context of human-computer interaction and supervised learning, such instrumentation facilitates workflow benchmarking, data quality assurance, and iterative refinement of annotation protocols. The precise, actionable analytics produced can inform best practices in both supervised speech tasks and broader annotation regimes throughout AI research and industry applications.

Conclusion

voxmap-studio introduces a rigorous, instrumented approach to speaker diarization annotation, quantifying annotation cost and enforcing annotation integrity through confirmation and attention checks. Empirical results challenge the notion that increased annotation assistance monotonically reduces cost, underscoring the value of workflow instrumentation for empirical optimization. The framework provides practical and theoretical advancements in the creation, validation, and benchmarking of labeled speech corpora, and holds promise for extension to broader supervised annotation tasks in AI.