MuScriptor: An Open Model for Multi-Instrument Music Transcription
Abstract: Existing methods for automatic music transcription are often limited to single-instrument recordings or fail on complex, real music mixes. Although previous work utilizes synthetic training data, the resulting models generalize poorly, leading to largely unusable transcription output in realistic, multi-instrument settings. In this work, we analyze the effectiveness of synthetic data for pre-training while combining it with fine-tuning on real music audio and post-training using reinforcement learning. We further introduce conditioning on instrument presence to customize transcriptions. Finally, we release MuScriptor, an open-weight multi-instrument music transcription model that works on real-world music recordings from across a diverse range of musical genres.
Summary
- The paper introduces MuScriptor, a transformer-based model that leverages large-scale real and synthetic data with RL post-training for improved multi-instrument transcription.
- It employs instrument conditioning via learned embeddings to achieve more temporally consistent and customizable transcriptions across diverse genres.
- Quantitative results show substantial improvements in F1 metrics, setting a new baseline over prior state-of-the-art methods.
MuScriptor: An Open Model for Multi-Instrument Music Transcription
Problem Statement and Context
Automatic Music Transcription (AMT) for real-world, multi-instrument audio remains an unresolved technical challenge, largely due to limitations in dataset scale, annotation quality, and the domain shift between synthesized and real recordings. While piano- and instrument-specific transcription models have seen considerable progress, general-purpose systems capable of accurate transcription across diverse, professional mixes lag behind. Existing approaches such as MT3 partially address data scarcity via synthetic pre-training, but their generalization to in-the-wild audio is poor, resulting in high error rates on practical tasks. This work introduces MuScriptor, a model designed to close this gap by leveraging large-scale real and synthetic data, a straightforward transformer backbone, reinforcement learning post-training, and optional instrument conditioning.
Data Curation and Instrument Diversity
A central contribution is the compilation and alignment of a large-scale real-world dataset spanning 170,000 music recordings (over 11,000 hours) and a synthetic corpus comprising 1.45M uniquely augmented MIDI tracks. The real audio dataset is notable for its genre coverage (e.g., classical, metal, jazz) and annotation diversity, made possible through advanced synchronization techniques utilizing chroma and onset-based dynamic time warping. The synthetic dataset pipeline augments MIDI files through pitch, velocity, tempo modulation, and soundfont randomization, supporting robust pre-training and augmentation.
Statistical analyses confirm substantial instrument variety; 17 instrument groups appear in more than 5% of tracks, and the number of simultaneously active instruments per track reflects realistic multitrack complexity.
Figure 1: Piano roll for guitar notes in a test song, visualizing MuScriptor’s true positives (blue), false negatives (green), and false positives (red), with vertical lines denoting evaluation segments.
Model Architecture and Training Regimes
The MuScriptor model uses a decoder-only transformer architecture, scalable from 60M to 1.3B parameters, conditioned on a pair of inputs: a 5-second segment mel-spectrogram and an optional list of present instruments. Instrument conditioning is injected through learned embeddings and concatenated as a prefix to the spectrogram encoding, following standard sequence modeling practices.
During training, MuScriptor employs a hybrid regime:
- Synthetic pre-training yields a general initialization, critical when annotated real data is limited.
- Extensive fine-tuning on real audio addresses the synthetic-to-real domain gap.
- Reinforcement learning (RL) post-training on a high-quality subset employs a GRPO-like algorithm to further improve F1 scores, especially onset and offset precision.
Classifier-free guidance (CFG) and segment-level instrument conditioning further control and stabilize model predictions.
Performance Evaluation: Quantitative and Qualitative Results
Synthetic Pre-training and Real Data Fine-Tuning
Experiments demonstrably validate that pre-training solely on synthetic data yields strong baselines, but significant improvements are obtained only when fine-tuning on real audio with aligned annotations. Notably, with only 1% of the real dataset, pre-training improves offset F1 from 9.9 to 33.4. However, even with abundant synthetic data, exclusive reliance on it leads to severe degradation when applied to real audio. The combination of both is optimal; at full scale, pre-training plus real data achieves an offset F1 of 42.3 versus 41.0 without pre-training.
Reinforcement Learning Post-Training
A distinct boost is provided by reinforcement learning over a hand-curated, high-quality set. The RL phase delivers an absolute improvement up to 6–8 F1 points on several metrics, notably in reducing both false negatives and incorrect note onsets.
Instrument Conditioning and Model Scalability
Instrument conditioning at inference yields measurable gains across all F1 metrics. It enables:
- Custom transcription (targeting only selected instruments)
- More temporally consistent instrument predictions across segment boundaries.
Scaling the model from 60M to 1.3B parameters reveals a monotonic improvement; the largest model reaches a frame F1 of 68.7. Input representation ablation demonstrates that standard mel-spectrograms outperform magnitude CQT, Encodec, and learned music embeddings (MERT) for this task.
Benchmarking and Cross-Domain Evaluation
On the novel internal test set, MuScriptor exceeds prior state-of-the-art baselines (notably YourMT3+), achieving substantial improvements in both segmental (e.g., onset, offset, frame) and aggregated (multi) F1 across all evaluation protocols. Cross-domain tests on held-out public datasets (e.g., RWC, Bach10, Dagstuhl ChoirSet) reveal larger improvements in frame and multi-F1 scores, although onset and offset metrics remain challenging for certain repertoires, reflecting annotation ambiguities (e.g., in choir music).
Theoretical and Practical Implications
Key claims:
- Training on real data is necessary for accurate transcription of real music; synthetic pre-training is not sufficient.
- RL-based post-training produces further measurable improvements, a previously unexplored strategy in AMT.
- Instrument-conditioning allows fine control and cross-segment consistency.
The open-weight release of MuScriptor removes a significant barrier in MIR research by democratizing access to a robust multi-instrument transcription model trained at scale. From a practical standpoint, MuScriptor enables downstream applications such as automated musicology, large-scale symbolic corpus construction, and data-efficient MIR systems requiring accurate note events as input.
On the theoretical side, the research establishes that architectural complexity (e.g., additional encoders, MoE) provides only marginal benefit compared to the impact of training data scale/diversity, robust alignment, and RL finetuning. The findings indicate that improvements in data quality, reward shaping, and better handling of overlapping note events (currently limited by the tokenization scheme) are promising future directions for sequence-to-sequence AMT.
Limitations and Future Directions
MuScriptor’s tokenization does not accommodate overlapping notes of the same pitch/instrument—a restriction affecting theoretical upper bounds on achievable F1. Allowing for concurrent note events and scaling the instrument taxonomy further would address real-world musical complexity. Extending the segment size for faster inference and exploring richer conditioning (e.g., song section, genre) are direct future steps. Additionally, expanding RL-style post-training with larger, more diverse reward functions and datasets holds promise for cross-domain robustness.
Conclusion
MuScriptor establishes a new performance baseline for open-weight, general-purpose multi-instrument music transcription. By combining large-scale real and synthetic data, an unembellished transformer backbone, reinforcement learning post-training, and flexible instrument conditioning, it surpasses prior art in both quantitative and qualitative evaluation. This work demonstrates the decisive role of data quality/scale and suggests clear avenues for progress in data curation and output expressivity for future music transcription models.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Explain it Like I'm 14
Explaining “MuScriptor: An Open Model for Multi-Instrument Music Transcription”
Overview: What is this paper about?
This paper introduces MuScriptor, a computer model that listens to real songs and writes down the notes being played, kind of like turning a recording into digital sheet music (called MIDI). Unlike many older systems that handle just one instrument (like solo piano), MuScriptor can handle full songs with many instruments—guitars, drums, piano, voice, and more. The team also made the model freely available so others can use and improve it.
Key goals and questions
The researchers focused on a few big questions:
- Can we train a model to accurately transcribe real, mixed music with many instruments?
- Does training first on “fake” (computer-made) music help, and how much does it help compared to training on real recordings?
- Can we make the model better by giving it a hint about which instruments are in the song?
- Can the model improve further by “practice with feedback” (a type of training called reinforcement learning)?
- Can we build a simple, strong system that works across many music styles without fancy tricks?
How did they do it? (Methods in simple terms)
Think of the model as a very attentive listener and note-taker:
- Audio to pictures: The model first turns short chunks of the song (5 seconds) into a picture-like view of sound called a “mel-spectrogram.” Imagine a heatmap that shows how strong different pitches are over time.
- Notes as words: Instead of scanning every tiny slice of audio, the model writes a sequence of “tokens” (like words) that describe what happened: which note, when it started and ended, and which instrument played it. That’s how it creates a MIDI-like transcription.
- Transformers: The model uses a “Transformer” (a powerful type of AI that’s good at understanding sequences—like language). Here, the sequence is the stream of music events.
- Two kinds of training data:
- Synthetic data (computer-made): 1.45 million MIDI files turned into audio using many different sounds and random changes (tempo, pitch, instruments). This is like giving the model tons of practice sheets.
- Real data (recordings): 170,000 real songs (about 11,000 hours) with known, aligned notes. This is like practicing with actual performances.
- Instrument hints (conditioning): You can tell the model which instruments are present in the song (e.g., “piano, bass, drums”). That helps the model focus and keeps its guesses more consistent.
- Reinforcement learning (practice with feedback): After normal training, the model “plays back” its guesses and gets a score for how well it matched the true notes (starts, stops, and overall activity). It then adjusts to do better next time—like a student practicing and getting graded.
To check accuracy, the team measured:
- How well the model gets note starts (onsets),
- How well it gets note ends (offsets),
- How well it matches the overall notes over time (frames),
- How well it does on drums,
- And how well it matches both notes and the right instruments together.
Main findings: What worked and why it matters
- Synthetic pre-training helps—but isn’t enough on its own:
- Training first on lots of computer-made music gives the model a good foundation.
- But to work well on real songs, the model needs fine-tuning on real recordings.
- Real data is crucial:
- Adding large amounts of real music data made a big difference, especially for precise note timings and correct instruments.
- Reinforcement learning adds another boost:
- After the main training, “practice with feedback” improved note detection further, making the transcriptions cleaner and more reliable.
- Instrument hints improve accuracy:
- Telling the model which instruments are in the song (when that info is known) leads to better and more stable results.
- Simple inputs work best:
- The basic mel-spectrogram input (that sound heatmap) outperformed fancier alternatives they tried.
- Scales up well:
- Bigger versions of the model performed better, but even smaller versions worked well.
Overall, MuScriptor beat a strong previous system on many tests, and it works across a wide range of genres (from classical to metal).
Why this is important
- For musicians: It can help create sheet music-like files (MIDI) from recordings, making it easier to learn parts, remix, or analyze songs.
- For teachers and students: It can speed up music study and practice by highlighting parts and instruments.
- For researchers and developers: It opens doors to better music search, chord/key detection, and new music-making tools.
- For the community: The model and its weights are open, so others can build on it.
Final takeaway and future impact
MuScriptor shows that:
- A smart mix of synthetic pre-training, lots of real training data, and reinforcement learning can produce a reliable, multi-instrument music transcription system.
- Simple, solid design choices (like spectrogram inputs and a Transformer) can go a long way.
- Sharing the model openly can accelerate progress in music tech.
The authors suggest future improvements like recognizing more instrument types, handling overlapping notes of the same pitch and instrument better, and processing longer audio segments faster.
Knowledge Gaps
Knowledge gaps, limitations, and open questions
Below is a concise, actionable list of what remains missing, uncertain, or unexplored in the paper.
- Tokenization constraint on overlapping notes: The current event vocabulary cannot represent multiple simultaneous notes of the same instrument and pitch; evaluation removes such cases. Open tasks: design a tokenization that supports unisons/pedaled overlaps, retrain, and re-evaluate without filtering; quantify overlap prevalence across genres.
- Limited instrument taxonomy (36 groups): Mapping 128 GM instruments into 36 subgroups collapses distinctions (e.g., similar timbres) and may hurt instrument attribution. Open tasks: expand/reharmonize the taxonomy, report per-instrument confusion matrices, and assess gains versus added complexity.
- Dataset coverage and bias: The real dataset’s instrument distribution is long-tailed and focused on Western/pop/classical styles. Open tasks: measure per-instrument and per-genre performance; add/assess non-Western instruments, microtonal systems, and extended techniques; explore rebalancing and curriculum strategies for rare classes.
- Annotation alignment noise: Most real labels are created via audio–symbolic synchronization (DTW/measure interpolation) then filtered heuristically. Open tasks: quantify residual alignment error, publish alignment QC metrics, train/evaluate with controlled noise levels, and test noise-robust objectives.
- Synthetic-to-real domain gap: Despite large-scale augmentation (soundfonts, detuning, symbolic edits), pretraining alone remains insufficient. Open tasks: ablate each augmentation’s contribution; add production effects (mixing, compression, distortion, spatialization), stems, and mastering chains; develop “realism” metrics to predict transfer effectiveness.
- Small-scale RL post-training (300 tracks): The RL phase uses a tiny high-quality set and omits KL/clipping, risking overfitting/instability. Open tasks: ablate , temperature, and reward composition; add KL regularization or reference policy constraints; study generalization and drift across styles; scale RL data and test curriculum RL.
- Reward design lacks instrument correctness: RL reward sums onset/frame/offset F1 but excludes instrument attribution (Multi F1). Open tasks: incorporate instrument-conditioned rewards (including drums class-wise), assess trade-offs, and verify that improved rewards lift Multi F1 without harming timing.
- Segment-level conditioning dependence: Instrument conditioning assumes track-level instrument lists are known and correct. Open tasks: evaluate robustness to missing/misleading conditioning; train/jointly learn an instrument-presence detector; explore per-segment conditioning or confidence-weighted conditioning.
- Five-second context and boundary effects: Short windows can hurt onset/offset precision and cross-segment consistency. Open tasks: increase context length or use overlapping windows with sequence-level decoding; test track-level decoding, beam search, or global consistency constraints (e.g., CRFs/post-hoc smoothing).
- Audio input limitations (16 kHz mono mel): Mono and 16 kHz may discard spatial/high-frequency cues. Open tasks: evaluate stereo and higher sample rates; test multi-resolution front-ends, learned encoders, or fusion of mel with CQT; measure impact by instrument class and onset sharpness.
- Decoding strategy underexplored: Inference uses argmax with classifier-free guidance (CFG); beam search, sampling, and calibration are not examined. Open tasks: study guidance strength, beam sizes, and N-best rescoring; analyze precision/recall trade-offs and why RL reduces CFG benefits.
- Drum transcription scope: Evaluation is onset-only and aggregates drums; sustained cymbals and articulations (rolls/flams) are not modeled. Open tasks: track duration for sustained percussive events where relevant, report per-drum-class F1, and extend tokens for articulations.
- Expressive/performance nuances: Pitch bends, vibrato, microtonality, and piano pedaling are not represented. Open tasks: add continuous pitch tokens/controls, pedaling/CC events, and evaluate on guitar solos, vocal vibrato, and non-tempered music.
- Evaluation limitations and standardization: Heavy reliance on an internal test set; external benchmarks are few and onset/offset metrics struggle for styles like choir. Open tasks: help establish/open a diverse, public, multi-instrument benchmark; explore tolerance curves, soft/perceptual metrics, and human-in-the-loop evaluation for ambiguous onsets/offsets.
- Failure-mode analysis: On some public datasets, onset/offset F1 lags despite strong frame/Multi F1. Open tasks: conduct detailed error analyses by instrument, genre, and acoustics (reverb, legato); target style-specific remedies (e.g., choir phrasing, string legato).
- Cross-segment instrument consistency: The paper notes fluctuations mitigated by conditioning but does not quantify track-level consistency. Open tasks: define metrics for instrument assignment consistency across segments; explore global decoding constraints or post-processing.
- Data de-duplication and leakage: Train–test separation used title-based filtering; duplicates across alternate titles/arrangements may remain. Open tasks: perform audio/MIDI fingerprinting and melody hashing de-duplication; quantify residual leakage risk.
- Efficiency and deployment: No latency/throughput metrics are reported; the 1.3B model may be impractical for real-time use. Open tasks: benchmark inference speed/memory; explore distillation, pruning, and quantization; characterize quality–efficiency trade-offs.
- Per-instrument performance reporting: Results are largely instrument-agnostic or aggregated; fine-grained score breakdowns are missing. Open tasks: publish per-instrument F1 and confusion matrices; identify instruments that dominate errors and target data/architecture fixes.
- Robustness to effects and production: Claims of handling distortion and complex mixes are qualitative. Open tasks: create effect-stress benchmarks (varying distortion/reverb/compression), measure degradation curves, and train with targeted augmentations.
- Hybrid pipelines with source separation: The model omits testing source-separation-assisted transcription, which might boost multi-instrument attribution. Open tasks: evaluate stem-informed transcription versus end-to-end, and study fusion strategies.
- CFG formulation for autoregressive AMT: Applying CFG to both spectrogram and instrument conditions is described but not formalized or justified theoretically. Open tasks: clarify the training/inference formulation for CFG in AR models, analyze when/why it helps, and provide ablations across guidance strengths.
Practical Applications
Immediate Applications
The following use cases can be deployed now using MuScriptor’s open weights/code, instrument-conditioned transcription, and demonstrated performance on real-world, multi-instrument mixes.
- Audio-to-MIDI transcription inside DAWs
- Sector: Software, Music production
- Tools/products/workflows: VST3/AU/AAX plugin or command-line tool that converts mixed audio into multi-track MIDI; selective transcription via instrument conditioning (e.g., “transcribe only drums and bass”); post-processing to quantize and export to notation (MuseScore, Sibelius) or DAW tracks.
- Assumptions/dependencies: Known or estimated track-level instrument list improves stability; 5 s windowing requires overlap-add stitching; current tokenization cannot represent overlapping notes for same pitch/instrument; best accuracy on Western, multi-instrument genres seen in training.
- Catalog transcription and metadata enrichment at scale
- Sector: Music industry, Streaming, Libraries/Archives
- Tools/products/workflows: Batch process catalogs to produce MIDI/token indexes (melody, harmony, rhythm, instrumentation) for search/retrieval, playlisting, recommendation, and A&R analytics; melody/harmony fingerprints for cover/derivative detection.
- Assumptions/dependencies: Compute throughput for 11k+ hours-scale processing; legal policies on creating and storing symbolic representations from copyrighted audio; improved stability if basic instrument metadata is available.
- Rapid dataset bootstrapping for MIR research
- Sector: Academia (MIR, signal processing)
- Tools/products/workflows: Generate pseudo-labels (notes, onsets/offsets, instrument tags) on unlabeled audio to train/benchmark downstream tasks (key, chords, melody extraction, beat tracking); apply MuScriptor’s synthetic pretraining + real fine-tuning recipe and GRPO-style post-training for other MIR tasks.
- Assumptions/dependencies: Pseudo-label noise management; task-specific filtering/quality thresholds; performance varies across genres (e.g., lower onset/offset precision for choral).
- Transcription-assisted music education
- Sector: Education
- Tools/products/workflows: Student apps that turn recordings into lead sheets or parts; instrument-conditioned part extraction for practice (e.g., auto-generated violin part from an orchestral recording); progress tracking via note/frame F1-like metrics.
- Assumptions/dependencies: Notation rendering pipeline; potential difficulty for highly expressive/vocal music (onset/offset timing less reliable).
- Drum and rhythm part extraction for producers and drummers
- Sector: Music production, Education
- Tools/products/workflows: Extract drum MIDI from full mixes to reprogram kits, layer samples, or create practice loops; generate click/guide tracks aligned with detected onsets.
- Assumptions/dependencies: Drum F1 is moderate; genre-dependent performance; best with stable timekeeping.
- Musicology and archival analysis
- Sector: Academia, Cultural heritage
- Tools/products/workflows: Large-scale analysis of orchestration, voice-leading patterns, harmonic progressions; cross-era instrumentation statistics; corpus creation for comparative studies.
- Assumptions/dependencies: Bias toward genres present in training; choir/legato styles have weaker onset/offset precision; tokenization limits in overlapping note cases.
- API-based transcription for developer ecosystems
- Sector: Software/SaaS
- Tools/products/workflows: Hosted inference API for batch or on-demand transcription; instrument-conditioned endpoints; pipelines integrating MuScriptor with source separation, quantization, and notation export.
- Assumptions/dependencies: Latency/throughput SLAs; GPU/CPU cost control; quality monitoring on user submissions.
- Loop/beat-to-MIDI conversion for soundware and remix workflows
- Sector: Music production, Sample packs
- Tools/products/workflows: Convert audio loops to MIDI for flexible remixing, key/tempo changes, and arrangement; organize libraries by extracted rhythmic/harmonic content.
- Assumptions/dependencies: Accurate tempo alignment and quantization post-processing; best performance with clear, non-overlapping instrument textures.
- Accessibility: audio-to-readable music formats
- Sector: Accessibility/Assistive tech
- Tools/products/workflows: Convert recordings to simplified scores or Braille music via MIDI-to-notation; instrument conditioning to prioritize target parts for learners.
- Assumptions/dependencies: Notation simplification rules; handling expressive timing; segment boundary consistency.
- Improved content identification and sample clearance support
- Sector: Legal/Policy, Music industry
- Tools/products/workflows: Symbolic matching (melodic/harmonic) across catalogs to flag potential infringements and assist musicologists in determining similarity.
- Assumptions/dependencies: Policies around “symbolic fingerprints” from copyrighted audio; false positive/negative management due to transcription errors.
Long-Term Applications
The following opportunities require further research, scaling, or development (e.g., extended tokenization, more diverse training, real-time constraints).
- Real-time, low-latency live transcription and on-stage accompaniment
- Sector: Live performance, Software, Robotics
- Tools/products/workflows: Streaming inference with sub-second latency to create live lead sheets, click tracks, or automatic band-in-a-box accompaniments; robotic or virtual performers following transcribed structure.
- Assumptions/dependencies: Longer-context modeling without 5 s boundary artifacts; efficient streaming decoders; improved onset/offset accuracy under stage acoustics; robust instrument detection without metadata.
- Fully editable DAW reconstructions from mixed audio
- Sector: Music production
- Tools/products/workflows: End-to-end pipeline that turns a song into quantized, multi-track MIDI plus articulations; enables remixing, reorchestration, and sound replacement at scale.
- Assumptions/dependencies: Tokenization supporting overlapping notes and articulations; tighter onset/offset and sustain modeling (e.g., pedals); integration with source separation to disambiguate parts.
- Cross-genre and global instrument coverage
- Sector: Education, Industry, Academia
- Tools/products/workflows: Extend instrument vocabulary beyond MT3_FULL_PLUS to non-Western instruments, microtonal systems, and extended techniques; per-culture transcription services and study tools.
- Assumptions/dependencies: New annotated datasets across cultures; flexible pitch representations (microtones); instrument recognition models for expanded classes.
- Automated grading and assessment in music education
- Sector: Education
- Tools/products/workflows: Objective performance scoring from audio via alignment of student recordings to target scores (intonation, rhythm, articulation); feedback generation.
- Assumptions/dependencies: Higher temporal precision; robust handling of expressive timing and ornamentation; fairness and transparency standards for assessment.
- Rights management and regulatory frameworks for symbolic derivatives
- Sector: Policy/Legal
- Tools/products/workflows: Standards for storage, sharing, and use of machine-derived symbolic transcriptions; guidelines on fair use and privacy; industry-wide metadata schemas for symbolic fingerprints in streaming.
- Assumptions/dependencies: Stakeholder consensus; jurisdiction-specific IP interpretations; traceability and audit tools for derived data.
- Foundation pipelines for audio–symbolic–audio editing and generation
- Sector: Generative AI, Software
- Tools/products/workflows: Train sequence models that ingest MuScriptor transcriptions to enable structure-aware audio generation/editing (e.g., change orchestration while preserving form); cross-modal editors (score ↔ audio).
- Assumptions/dependencies: Stable, high-recall transcriptions for training; integration with audio generative models; standardized symbolic representations across tools.
- Unified cross-modal music translation and retrieval
- Sector: Software, Academia
- Tools/products/workflows: Systems that jointly handle score images, symbolic music, and performance audio for retrieval and translation (e.g., query by humming → score snippets → audio exemplars).
- Assumptions/dependencies: Larger multi-modal datasets; alignment robustness; domain-bridging architectures combining MuScriptor-like transcription with OCR/OMR and synthesis.
- Source-aware multi-instrument transcription via hybrid pipelines
- Sector: Software, Music production
- Tools/products/workflows: Combine source separation with MuScriptor to improve per-instrument note accuracy, reduce interference, and support overlapping note cases.
- Assumptions/dependencies: High-quality separation models; error propagation management; retraining with separated stems.
- Orchestration/style transfer and arrangement assistants
- Sector: Music production, Education
- Tools/products/workflows: Learn mappings from transcribed multi-instrument scores to target ensembles/styles; auto-arrange band/orchestra parts from a demo recording.
- Assumptions/dependencies: Large paired corpora of arrangements; improved instrument labeling and articulation extraction; user-in-the-loop controls.
- Clinical and therapeutic session analysis
- Sector: Healthcare
- Tools/products/workflows: Quantify engagement, timing, and pitch/rhythm features from music therapy recordings for progress tracking and research.
- Assumptions/dependencies: Domain adaptation to small-ensemble/acoustic contexts; privacy and data handling compliance; clinician-validated metrics.
- Longer-context, faster inference with global coherence
- Sector: Software, Industry
- Tools/products/workflows: Whole-track or minute-scale inference to stabilize instrument assignments and eliminate segment boundary artifacts; accelerated decoding for batch pipelines.
- Assumptions/dependencies: Efficient architectures and caching; memory-optimized tokenization; training on longer windows.
- Advanced tokenization for concurrent note activity and articulations
- Sector: Academia, Software
- Tools/products/workflows: New vocabularies that encode overlapping notes, pedals, and performance nuances; improved alignment metrics and RL rewards reflecting musical semantics.
- Assumptions/dependencies: Redesign of event schemas; retraining with enriched annotations; updated evaluation benchmarks.
Notes on feasibility across applications:
- Compute/latency: 1.3B-parameter model offers best accuracy; smaller (60–300M) variants suit edge or low-latency use with modest accuracy trade-offs.
- Instrument conditioning: Works best when track-level instrument lists are available; otherwise pair with an instrument recognition model.
- Domain coverage: Performance is strongest on genres present in training; choir/legato styles show lower onset/offset precision; non-Western and microtonal music require additional data and modeling.
- Legal/ethical: Transcribing copyrighted recordings raises policy questions for storage/use of symbolic derivatives; establish internal guidelines and rights checks before deployment.
Glossary
- AdamW: A variant of the Adam optimizer that decouples weight decay from gradient updates to improve generalization; "the AdamW optimizer with "
- argmax decoding: A greedy decoding strategy that selects the highest-probability token at each step without sampling; "performing argmax decoding."
- audio-symbolic music synchronization: Aligning audio recordings with symbolic scores (e.g., MIDI) so that notes match in time; "Most of this data is obtained via audio-symbolic music synchronization."
- autoregressive distribution: A distribution where each token is generated conditional on the preceding tokens; "by sampling from the autoregressive distribution with a fixed temperature ()."
- classifier-free-guidance: A technique that steers generation using unconditional and conditional passes to strengthen conditioning signals; "we apply classifier-free-guidance \cite{cfg} to both conditions with a strength of ."
- chroma features: Pitch-class energy representations over 12 semitone classes used for alignment and analysis; "dynamic time warping using chroma and onset features"
- Constant-Q Transform (CQT): A time–frequency transform with logarithmically spaced frequency bins, useful for music; "a magnitude CQT with the same frame rate of 100\,Hz and three bins per octave (252 bins in total) performed slightly worse in our experiments."
- cosine schedule: A learning-rate schedule that decays following a cosine curve; "a learning rate of $1e-4$ with a linear warmup of 2000 steps and a cosine schedule."
- cross-entropy loss: A standard classification loss measuring the divergence between predicted and target distributions; "using a standard cross-entropy loss."
- decoder-only Transformer: A Transformer architecture composed solely of a decoder stack for autoregressive generation; "opt for a simple yet effective decoder-only transformer architecture."
- detuning: Slightly shifting pitch away from standard tuning to increase realism/robustness; "applies a random detuning on the synthesized audio."
- domain shift: A mismatch between training and deployment data distributions that harms generalization; "a critical domain shift between synthesized MIDI and the complexities of professional music productions."
- Drums Onset F1: An F1 metric evaluating the correctness of predicted drum hit onsets; "Drums Onset F1: We separately report the onset F1 of predicted drum notes."
- dynamic time warping (DTW): An algorithm for aligning sequences with non-linear time variations; "dynamic time warping using chroma and onset features"
- Encodec: A neural audio codec that compresses audio into discrete representations; "Encodec \cite{encodec}, with a frame rate of 50\,Hz"
- Frame F1: Frame-wise F1 metric measuring overlap of active pitches over time; "Frame F1: To account for overall pitch activity, we compute frame-wise F1 at a resolution of $62.5$\,ms."
- Group Relative Policy Optimization (GRPO): A reinforcement learning method that normalizes rewards within groups to stabilize policy gradients; "Reinforcement Learning via Group Relative Policy Optimization (GRPO)"
- hierarchical attention transformer: A Transformer that aggregates information at multiple temporal/frequency scales; "a hierarchical attention transformer operating in the time-frequency domain"
- hop size: The number of samples between successive frames in a time–frequency transform; "a hop size of 160 samples (yielding a 100Hz frame rate)"
- instrument-agnostic metrics: Evaluation measures that do not require correct instrument labeling, focusing on pitch/timing; "three standard instrument-agnostic metrics from the mir_eval library"
- instrument conditioning: Supplying the model with the set of instruments present to guide transcription; "For the instrument conditioning, we provide the model with the set of instruments present in the full track"
- KL divergence penalty: A regularizer that penalizes deviation from a reference policy in RL; "no importance-sampling ratio clipping or KL divergence penalty against a frozen reference policy is applied."
- linear warmup: A training schedule that increases the learning rate linearly for initial steps; "a linear warmup of 2000 steps"
- mel filter-bank: A set of filters spaced on the mel scale used to convert spectra to mel-spectrograms; "a mel filter-bank with 512 bins."
- mel-spectrogram: A spectrogram whose frequency axis is mapped to the mel scale, approximating human pitch perception; "a mel-spectrogram representing a 5-second audio excerpt"
- MERT: A large self-supervised acoustic music embedding model used as an alternative input representation; "MERT \cite{mert}, with a frame rate of 75\,Hz"
- Mixture of Experts (MoE): A model architecture that routes inputs to specialized expert sub-networks; "a Mixture of Experts (MoE) layer"
- MIDI-like tokens: Discrete event tokens representing musical notes, timings, and instruments; "a discrete stream of MIDI-like tokens"
- MT3 (Multi-Task Multitrack Music Transcription): An encoder–decoder Transformer framework for multi-instrument transcription from spectrograms; "the Multi-Task Multitrack Music Transcription (MT3) framework"
- MT3_FULL_PLUS taxonomy: A grouped instrument taxonomy used for tokenization and conditioning; "MT3_FULL_PLUS taxonomy"
- Multi F1: An F1 metric requiring correct pitch, timing, and instrument assignment; "Multi F1: An extension of the Offset F1 metric that additionally requires the instrument predicted for a note to be correct"
- MusicFM: A self-supervised audio foundation model used as an encoder in recent systems; "MusicFM, a self-supervised audio foundation model."
- Offset F1: A note-level F1 metric requiring both correct onset and offset within tolerances; "Offset F1: This metric requires, in addition to the onset criterion, that the predicted offset falls within a tolerance of"
- Onset F1: A note-level F1 metric requiring correct pitch and onset timing within a set tolerance; "Onset F1: Following the evaluation protocol of MT3 \cite{mt3}, a predicted note is considered correct if its pitch matches the reference and its onset is within a \,ms tolerance."
- Onsets and Frames (architecture): A model that separates onset detection from frame-wise pitch activity for piano transcription; "Onsets and Frames architecture"
- piano roll: A time–pitch matrix representation marking active notes over frames; "piano roll representation of transcriptions"
- policy gradient algorithm: An RL method that directly optimizes the parameters of a stochastic policy toward higher reward; "using a policy gradient algorithm that combines"
- prefix conditioning: Supplying conditioning embeddings as a prefix sequence to the model input; "used as prefix conditioning for the transformer model."
- REINFORCE estimator: A classic policy gradient estimator using Monte Carlo returns; "the REINFORCE estimator~\cite{reinforce}"
- reinforcement learning: Training with reward feedback to optimize model behavior beyond supervised targets; "post-training using reinforcement learning."
- self-supervised (learning): Learning representations from unlabeled data via pretext tasks; "a self-supervised audio foundation model."
- soundfont: A collection of sampled instrument sounds used to synthesize MIDI; "soundfont (from a collection of over 250 soundfonts)"
- STFT (Short-Time Fourier Transform): A time–frequency transform computed over sliding windows; "an STFT with $n_{\text{FFT}=2048$, a hop size of 160 samples (yielding a 100Hz frame rate)"
- teacher forcing: A training strategy for sequence models where ground-truth tokens are fed at each step; "most AMT models rely strictly on supervised teacher forcing"
- temperature (sampling): A scaling factor applied to logits to control randomness in sampling; "with a fixed temperature ()."
- time dilation factor: A constraint limiting how much alignment can stretch or compress during synchronization; "maximum time dilation factor"
- time-frequency domain: A representation analyzing signal energy across time and frequency; "operating in the time-frequency domain"
- tokenization scheme: The design for mapping musical events into discrete tokens; "because the tokenization scheme does not allow for multiple notes"
- timbre: The characteristic tone color or quality that distinguishes instrument sounds; "a vast range of timbres"
Collections
Sign up for free to add this paper to one or more collections.
