Automatic Piano Transcription

• Automatic Piano Transcription is the process of converting piano audio into symbolic representations such as note events and MIDI data.
• It addresses challenges like dense polyphony, overlapping resonances, and dynamic range variations using advanced acoustic modeling and decoding strategies.
• Recent approaches leverage CNNs, RNNs, and transformers along with efficient beam search and robust signal representations to enhance transcription accuracy.

Automatic Piano Transcription (APT) converts a piano performance recorded in audio into a symbolic representation, typically note events or MIDI capturing pitch, onset, offset, and often velocity and pedal-related information. Across the literature, the task appears in several closely related formulations: framewise prediction of 88-key piano rolls, event-sequence generation with MIDI-like tokens, score-level quantization from performance representations, and score-informed refinement of expressive attributes (Sigtia et al., 2014, Hawthorne et al., 2021, He et al., 11 Aug 2025, Zeng et al., 28 Sep 2025). The central difficulty is that piano audio combines dense polyphony, long decays, overlapping resonances, expressive timing, and wide dynamic range, so successful systems typically combine acoustic modeling, temporal or symbolic priors, and increasingly explicit strategies for robustness and efficiency (Edwards et al., 2024).

1. Task definition and scope

APT is usually treated as the piano-specialized case of Automatic Music Transcription, with subtasks that include pitch or note activity, onsets, offsets, pedaling, and dynamics or velocity (He et al., 11 Aug 2025). Common symbolic targets include event lists, piano-rolls, and framewise activation matrices followed by event-level refinements (Edwards et al., 2024). In framewise formulations, the acoustic input is a time series of spectrogram frames and the symbolic output is a time series of 88-dimensional binary vectors, one bit per piano key per frame (Sigtia et al., 2014).

The literature also contains narrower and broader task variants. In monophonic settings, Scorpiano treats APT as a DSP pipeline of onset detection, tempo estimation, beat-duration computation, pitch detection, and score generation for simple piano melodies (Sofronievski et al., 2021). At the other extreme, some recent work defines APT as symbolic performance-to-score transcription: the input is expressive piano performance MIDI and the output is notation-oriented score tokens, without raw audio in the APT module itself (Zeng et al., 28 Sep 2025).

A major distinction is therefore between framewise and sequence formulations. Framewise systems predict pitch activity at each analysis frame and then assemble note events through decoding or post-processing (Sigtia et al., 2014). Sequence-to-sequence systems instead model the conditional distribution over a variable-length event sequence, so timing, pitch, offset, and velocity dependencies are learned directly in the decoder (Hawthorne et al., 2021). This split structures much of the modern APT literature.

2. Signal representations and symbolic vocabularies

APT front ends vary substantially, but most modern systems use spectrogram-like time-frequency representations. A hybrid recurrent model from 2014 uses normalized magnitude spectrograms from audio downsampled to 16 kHz, with a 64 ms window and a 32 ms hop for training and validation, and a 10 ms hop for test (Sigtia et al., 2014). A generic encoder-decoder Transformer uses 16,000 Hz audio, STFT with FFT length 2048 and hop size 128, followed by 512 mel bands and log scaling; each input segment contains 511 spectrogram frames plus an EOS embedding, corresponding to 4.088 s (Hawthorne et al., 2021). The hFT-Transformer also operates at 16 kHz, but with a 256-bin log-mel front end, 16 ms hop size, and chunked tensors of shape $(B, N, F, 65)$ built from local temporal context around each frame (Toyama et al., 2023).

Constant-Q representations are used when harmonic alignment is treated as a primary design objective. HRplus and HRplus-hybrid use CQT with 48 bins per octave, 352 total bins, hop length 320 samples at 16 kHz, and high-resolution onset and offset targets defined by a peaked function $g(\Delta_i)$ over a window $\pm J$ frames (Mi et al., 2024). A simpler CQT+CNN pipeline for synthetic classical piano uses 44.1 kHz audio, contiguous 2756-sample frames, and 192-by-11 CQT patches for 72-way framewise pitch activity over C2–C8 (Telila et al., 7 May 2025). Audio-to-score work has also used HCFP features with $H=6$ harmonics, $F=352$ frequency bins, and 20 ms frame shift as input to a DeepLabv3+-based multipitch front end (Shibata et al., 2020).

Symbolic outputs are equally heterogeneous. In framewise systems, the dominant representation is an 88-key piano roll or a set of per-frame onset, offset, frame, and velocity targets (Sigtia et al., 2014, Ou et al., 2022). In event-based systems, the output vocabulary usually includes time, note on, note off, and velocity tokens. The 2021 seq2seq Transformer uses 128 Note tokens, 128 Velocity tokens, 6,000 absolute Time tokens quantized in 10 ms bins, and EOS; velocity persists until updated, and note-offs are represented by Note tokens emitted after a Velocity=0 token (Hawthorne et al., 2021). A hierarchical language-model approach uses a token dictionary of size 1265 with special tokens, query tokens $q_p$, $q_v$, $q_f$, 1001 time tokens per 10-second segment, 128 pitch tokens, 128 velocity tokens, and a “note-sustain” token for boundary continuity (Li et al., 6 Jan 2025). An efficient sparse-attention seq2seq system uses absolute Time, Note On, Note Off, Velocity, and BOS/EOS/PAD tokens with 20 ms time resolution (Wei et al., 11 Sep 2025).

These design choices reflect piano-specific acoustics. Long decays, expressive timing, and polyphonic chord structures are repeatedly identified as core complications (Sigtia et al., 2014). CQT-based systems argue that a logarithmic frequency axis better matches musical intervals and harmonic spacing, reducing false positives in which overtones are mistaken for separate fundamentals (Mi et al., 2024).

3. Modeling paradigms

One long-standing line of work is instrument-dependent generative modeling with explicit spectro-temporal templates. A studio-oriented ADMM framework builds variable-length patterns from single-note recordings of the target piano and optimizes a combination of generalized KL divergence, sparsity, total diagonal variation, and temporal-semantic regularizers resembling factorial scaled HMM constraints (Ewert et al., 2016). On Yamaha Disklavier recordings from MAPS, it reports an onset F-measure of 93–95%. A later extension adds equal-loudness thresholding based on the Glasberg–Moore loudness model and LSTM or BLSTM re-weighting of note constellations, improving F-measure from 95% to 97% on MAPS ENSTDkCl (Ewert et al., 2017). Semi-supervised CNMF is a related template-based alternative: it learns one isolated-note template per key, fixes the dictionary at test time, infers activations with KL-based CNMF, and uses adaptive onset picking; this improves over prior low-rank factorization baselines on MAPS but shows strong train-test mismatch effects on MAESTRO (Wu et al., 2022).

A second line combines acoustic classifiers with symbolic priors. The hybrid recurrent neural network formalizes the joint model as

$$P(z,x) \propto P(z_1;\Theta_l)\frac{P(z_1|x_1)}{P(z_1)} \prod_{t=2}^{T} P(z_t|\mathcal{A}_t)\frac{P(z_t|x_t)}{P(z_t)},$$

where $P(z_t|x_t)$ is a frame-level acoustic term, $g(\Delta_i)$0 is a music LLM, and $g(\Delta_i)$1 is the empirical marginal over target vectors (Sigtia et al., 2014). Because piano frames are 88-dimensional binary vectors, the LLM uses RNN-NADE rather than independent sigmoids, so it can represent joint pitch co-occurrence and temporal dependencies exactly enough for beam search (Sigtia et al., 2014). A later high-resolution system retains a multitask CNN-GRU architecture for onset, offset, and frame activity but replaces the velocity branch with a Transformer encoder, finding that self-attention helps velocity estimation more than frame or offset modeling (Ou et al., 2022).

Recent neural systems move further toward end-to-end event modeling. The 2021 seq2seq Transformer treats APT as spectrogram-to-event translation optimized by teacher-forced cross-entropy,

$g(\Delta_i)$2

and argues that absolute-time tokens avoid cumulative drift associated with relative time-shift outputs (Hawthorne et al., 2021). The hFT-Transformer splits attention hierarchically: a frequency-axis Transformer encoder operates on spectral tokens, a Transformer decoder converts frequency representations into pitch-wise tokens, and a time-axis Transformer encoder models long-range temporal dependencies over those pitch tokens (Toyama et al., 2023). HRplus and HRplus-hybrid instead retain high-resolution onset/offset labeling and use a musically aligned CQT front end, harmonic-spacing-matched dilation rates, and either a CRNN or a CRNN plus non-autoregressive Transformer decoder (Mi et al., 2024). A hierarchical language-model system attaches decoder-only Transformers to pretrained roll-based encoders and factorizes note prediction into onset+pitch, then velocity, then offset, explicitly reducing sequence length and computational burden relative to flattened note-event streams (Li et al., 6 Jan 2025).

An alternative rethinks the objective rather than only the architecture. The optimal-transport approach models onset and offset predictions as distributions over the time–pitch lattice and minimizes the cost of transporting predicted mass to ground-truth mass under a pitch-faithful, temporally capped transport cost (Wei et al., 17 May 2026). In that work, a CRNN with harmonics-aware attention and frequency-grouped LSTMs attains state-of-the-art onset detection on MAESTRO.

4. Decoding, note formation, and score constraints

Decoding is a decisive part of APT because raw frame or token predictions must be converted into coherent note events. In the hybrid RNN, greedy chronological decoding is explicitly described as suboptimal because the history $g(\Delta_i)$3 is itself uncertain; the paper therefore uses a high-dimensional beam search with beam width $g(\Delta_i)$4, acoustic-driven branching, and 70 ms minimum-duration pruning (Sigtia et al., 2014). The resulting search is computationally heavy: decoding the first 30 seconds of all 50 MAPS test tracks took 20 hours on a CPU (Sigtia et al., 2014).

High-resolution frame-based systems often use task-specific note search. The hFT-Transformer detects onset and offset candidates as local maxima above 0.5, refines their times with adjacent frames, reads velocity at the onset frame, and takes the earlier of two offset candidates: one from the offset head and one from the frame head (Toyama et al., 2023). It also uses half-stride inference, keeping only the central portion of each chunk to reduce boundary errors (Toyama et al., 2023). HRplus adopts the same decoding family with thresholds 0.4 for onset, offset, and frame, velocity threshold 0, and continuous onset/offset estimation inherited from the high-resolution baseline (Mi et al., 2024).

Event-based seq2seq models shift complexity into the decoder itself. The generic encoder-decoder Transformer uses greedy autoregressive argmax until EOS; after non-overlapping segmentation, note-offs lacking prior note-ons are removed, same-pitch re-articulations terminate the currently active note, and active notes are ended at the segment boundary if no note-off appears (Hawthorne et al., 2021). The hierarchical language-model system likewise avoids thresholding and frame-to-note merging: stage 1 emits onset and pitch tokens, stage 2 predicts velocities conditioned on those notes, and stage 3 predicts offsets conditioned on onset, pitch, and velocity (Li et al., 6 Jan 2025). Efficient sparse-attention seq2seq transcription pairs NoteOn and NoteOff tokens by pitch and uses the most recent Time token to anchor local cross-attention windows during decoding (Wei et al., 11 Sep 2025).

Some systems impose score-level structure after note recovery. Audio-to-score transcription with non-local musical statistics first performs DNN-based multipitch detection, then applies a metrical HMM and an MRF for rhythm quantization, metre identification, hand separation, voice separation, and MusicXML export (Shibata et al., 2020). Symbolic performance-to-score transcription instead lets an autoregressive score decoder map expressive timing and duration directly into notation-oriented tokens, eliminating hand-crafted quantization heuristics and note-level alignment procedures in the APT stage itself (Zeng et al., 28 Sep 2025).

5. Datasets, evaluation protocols, and representative results

MAPS and MAESTRO dominate contemporary audio-based evaluation. MAPS contains 270 piano pieces with aligned MIDI, including 210 synthesized renderings and 60 real-piano recordings; one common split is 200 training tracks, 20 validation tracks, and 50 test tracks, with evaluation on the first 30 seconds of each test item (Sigtia et al., 2014). MAESTRO contains about 200 hours of aligned virtuosic piano audio and MIDI, and version 3.0.0 adds 26 hours and additional pedal annotations (Hawthorne et al., 2021). Robustness work also introduces Studio MAESTRO, a full re-performance of MAESTRO on a Yamaha Disklavier in a professional studio, plus six Pianoteq renderings per piece for timbral diversity (Edwards et al., 2024). Symbolic performance-to-score work uses ASAP as paired data, MuseScore as large unpaired score data, and ATEPP for out-of-distribution style analysis (Zeng et al., 28 Sep 2025).

Evaluation is similarly stratified. Framewise systems report MIREX-style frame and note metrics, often with precision, recall, and accuracy (Sigtia et al., 2014). Event-based systems commonly use mir_eval with onset tolerance ±50 ms, offset tolerance equal to the larger of ±50 ms or ±20% of note duration, and a velocity tolerance of 0.1 on normalized velocity or ±10% of ground-truth velocity (Hawthorne et al., 2021, Li et al., 6 Jan 2025, Mi et al., 2024). Symbolic score-transcription work instead uses notation-oriented metrics such as MUSTER and ScoreSimilarity, which measure edit errors, onset and offset deviation, duration error, staff assignment, stem direction, and pitch spelling (Zeng et al., 28 Sep 2025).

A representative empirical snapshot is shown below.

System	Dataset	Reported result
Hybrid DNN+RNN with MLM	MAPS	Hybrid F-measure 69.62 / 64.69 for frame / onset-only note (Sigtia et al., 2014)
hFT-Transformer with half-stride	MAESTRO v3.0.0	Frame 93.24, Note 97.44, Note w/ Offset 90.53, Note w/ Offset & Velocity 89.48 (Toyama et al., 2023)
Robust data-centric model	MAPS zero-shot	Precision 89.5, Recall 87.4, F1 88.4 for note onsets without MAPS training (Edwards et al., 2024)
HRplus	MAESTRO v3	Onset F1 97.39, Onset+Offset F1 84.76, Onset+Offset+Velocity F1 82.96 (Mi et al., 2024)
SFT-CRNN with OT loss	MAESTRO	Onset F1 98.36, Note+Offset F1 90.78 (Wei et al., 17 May 2026)

Additional subtask-specific results show similar specialization. Score-informed BiLSTM refinement of High-Resolution Piano Transcription reduces velocity MAE from 5.05 to 3.48 and raises Recall from 90.6% to 95.6% on MAESTRO test when ground-truth onsets are used (He et al., 11 Aug 2025). In symbolic performance-to-score transcription, adding full unpaired data improves MUSTER $g(\Delta_i)$5 from 13.30 to 12.48 and ScoreSimilarity pitch-spelling error from 8.44 to 6.24 on ASAP (Zeng et al., 28 Sep 2025). This suggests that onset recovery is no longer the only bottleneck; duration, velocity, notation, and domain shift remain decisive evaluation axes.

6. Robustness, efficiency, and extended objectives

A recurring theme is that high in-distribution accuracy can hide severe acoustic overfitting. Training only on Studio MAESTRO yields about 97.3 F1 on studio-held-out audio but only about 80.8 on original MAESTRO, despite identical musical content, showing that acoustic conditions alone can dominate generalization (Edwards et al., 2024). The same study finds that micro pitch-shift and reverberation are the most important augmentations for out-of-distribution robustness, and that a model trained with diverse acoustic sources and augmentation reaches 88.4 note-onset F1 on MAPS without seeing MAPS training data (Edwards et al., 2024). A different strategy pretrains a compact CRNN on synthesized multi-instrument audio and then fine-tunes for piano, yielding faster convergence and modest improvements on MAPS relative to piano-only training from scratch (Leś et al., 2023). Template-based CNMF, by contrast, performs well under matched piano conditions but degrades strongly under cross-piano and cross-dataset mismatch (Wu et al., 2022).

Efficiency has become an equally visible concern. The 2021 seq2seq Transformer segments audio into approximately 4.09 s windows because full self-attention has quadratic cost and is therefore best suited to offline transcription (Hawthorne et al., 2021). Sparse-attention seq2seq transcription addresses this directly with sliding-window self-attention, token-type-conditioned hybrid global-local cross-attention, and hierarchical pooling, reporting more than 2× speedup and significant GPU memory reduction while maintaining performance comparable to a full-attention baseline (Wei et al., 11 Sep 2025). Earlier beam-search hybrids illustrate the opposite end of the trade-off: global search improved coherence but remained far from real-time on CPU hardware (Sigtia et al., 2014). Real-time deployment has nonetheless begun to appear in interactive music systems. In “Dialogue in Resonance,” an online neural autoregressive piano transcription model is integrated into a Python–Max/MSP performance environment and used to drive a Yamaha Disklavier with reported model latency of approximately 350 ms (Bang et al., 22 May 2025).

APT has also expanded beyond note onset recovery toward richer musical semantics. Score-informed velocity correction treats timing as already aligned and uses onset, frame, and sustain-excluding frame masks from the score to refine note dynamics (He et al., 11 Aug 2025). Audio-to-score transcription adds metre, bar lines, downbeats, hand separation, and voice assignment, and shows that non-local musical statistics can raise downbeat F-measure on J-pop from 69.4 to 85.6 without changing the underlying note transcription error rate (Shibata et al., 2020). Symbolic performance-to-score transcription further separates score content from performance style, using transformer encoders and decoders plus unpaired reconstruction losses to recover notation-oriented structure without fine-grained note-level alignment (Zeng et al., 28 Sep 2025).

Across these strands, several limitations remain stable. Duration modeling and false alarms remain difficult in framewise systems (Sigtia et al., 2014). Pedal handling is frequently simplified or omitted in seq2seq vocabularies and high-efficiency architectures (Hawthorne et al., 2021, Wei et al., 11 Sep 2025). Some models are evaluated only on MAESTRO-like conditions, while others require perfect timing correction or instrument-specific calibration (He et al., 11 Aug 2025, Ewert et al., 2016). A plausible implication is that the field’s remaining gains will depend at least as much on representation design, data diversity, and decoding constraints as on raw backbone scale.