MIDI Velocity Prediction

• MIDI Velocity Prediction is the task of estimating, correcting, generating, or transferring the expressive dynamics encoded in note events.
• Techniques range from acoustics-specific transcription with CNN/BiLSTM architectures to generative models like variational autoencoders and GRU-based systems.
• Comparative evaluations use metrics such as L1 error, MAE, and perceptual listening tests, highlighting that velocity does not directly equate to perceived loudness.

Searching arXiv for the specified MIDI velocity prediction papers and closely related work. MIDI velocity prediction is the estimation, correction, generation, or transfer of the MIDI velocity parameter attached to note events, usually with the objective of recovering expressive dynamics while preserving note identity and timing. Across recent work, the problem is not treated as a single canonical task: it appears as acoustics-specific note transcription for piano, score-informed correction of automatic music transcription (AMT) outputs, symbolic reconstruction inside variational generative models, dense “velocity filling” on piano rolls, explicit probabilistic sub-event modeling for real-time generation, and heuristic characterization of expressive performance at the track level rather than note level (Simonetta et al., 2022, He et al., 11 Aug 2025, Brunner et al., 2018, He et al., 11 Aug 2025, Shepardson et al., 2024, Lee et al., 24 Feb 2025). A recurring theme is that MIDI velocity is an abstract control parameter in the range $[0,127]$, not a universally standardized physical quantity, and that its relationship to perceived loudness, articulation, and expressive intent depends on representation, instrument, and acoustic context (Simonetta et al., 2022, Qu et al., 2022).

1. Conceptual scope and problem formulations

The literature distinguishes several formulations of MIDI velocity prediction. In acoustics-specific piano transcription, the task is to infer “a velocity representation suitable for faithful resynthesis in a known acoustic environment,” rather than merely to assign a context-invariant integer to each note (Simonetta et al., 2022). In score-informed correction, the task is narrower: a system such as HPT first produces preliminary note velocities from audio, and a subsequent model refines those values under the assumption of perfect timing correction (He et al., 11 Aug 2025). In MIDI velocity filling, the input is symbolic MIDI with velocity removed or flattened, and the objective is to restore note-level expressive dynamics without altering pitch or timing (He et al., 11 Aug 2025).

Other formulations are generative rather than corrective. MIDI-VAE models velocity as a continuous reconstruction target inside a shared latent space for pitch, velocity, and instrument rolls; style transfer then changes velocity together with pitch and instrumentation (Brunner et al., 2018). Notochord treats velocity as an explicit sub-event $v$ in a structured event tuple $(\Delta t, v, p, \alpha)$, with continuous density modeling and support for arbitrary sub-event query order during interactive generation (Shepardson et al., 2024). Drum transcription work extends onset classification by adding a velocity head so that resynthesized outputs better reflect human-performed dynamics (Callender et al., 2020).

A separate but related line of work uses velocity statistics descriptively rather than predictively. The GigaMIDI study introduces the Distinctive Note Velocity Ratio (DNVR) to classify tracks as non-expressive or expressive in velocity, making velocity a track-level expressive-performance cue rather than a note-wise regression target (Lee et al., 24 Feb 2025). This suggests that “MIDI velocity prediction” functions as an umbrella term spanning transcription, rendering, correction, generation, transfer, and expressive-performance analysis.

2. Representations of velocity in symbolic and audio-conditioned systems

A central source of methodological divergence is representation. In standard MIDI usage, velocity is an integer from $0$ to $127$, but individual papers encode or normalize it differently depending on task (Simonetta et al., 2022, Lee et al., 24 Feb 2025, He et al., 11 Aug 2025). Acoustics-specific piano estimation predicts one scalar velocity per note, with a final Sigmoid output interpreted on the target velocity scale (Simonetta et al., 2022). Score-HPT trains on normalized velocities in $[0,1]$ and evaluates on denormalized velocities in $[0,127]$ (He et al., 11 Aug 2025). The U-Net colorization approach similarly normalizes the velocity roll from the integer MIDI range $[0,127]$ to approximately $[0,1)$ (He et al., 11 Aug 2025).

In MIDI-VAE, velocity is not only a dynamics value but part of a symbolic scheme that also captures note duration. The model defines a velocity roll $V \in [0,1]^{n_P \cdot n_B \cdot n_T}$, maps MIDI velocities $v$0 to $v$1 into $v$2, and assigns special semantics to values below $v$3: they indicate either no note or a held note from the previous step (Brunner et al., 2018). In that representation, the velocity roll encodes onset/hold structure as well as dynamic intensity. Notochord makes a different choice: velocity is continuous, dequantized, and modeled analogously to time; note-off events are encoded by setting velocity to zero and release velocities are ignored (Shepardson et al., 2024).

Event-based generative work on expressive piano performance generation introduces yet another representation. There, velocity is not predicted as a raw continuous MIDI number but as a relative change category with respect to the previous note, and the categories are quantized into uneven chunks based on Weber’s law (Liu, 2024). This suggests a design preference for local dynamic contour over absolute dynamic level, although the paper does not provide velocity-specific quantitative evaluation (Liu, 2024).

These choices affect what counts as “prediction.” A model may predict a note-level scalar, reconstruct a dense piano-roll image, sample from a continuous density, or generate a velocity-change token. Consequently, reported metrics are only partially comparable across papers.

3. Architectural families and learning objectives

Audio-conditioned note-level estimation for piano is exemplified by the acoustics-specific modular architecture. The pipeline takes an audio recording plus aligned score or MIDI timing information, uses NMF for note separation, computes the first 13 MFCCs over 30 frames for each separated note, and feeds the result to a two-stage CNN composed of an encoder and a performer module built from Residual CNN blocks with skip connections, batch normalization, ReLU activations, and grouped convolutions (Simonetta et al., 2022). The encoder is intended to infer an interpretation-like latent representation, while the performer maps that representation to the final velocity in the target acoustic context (Simonetta et al., 2022).

Score-informed correction uses a different decomposition. Score-HPT concatenates the audio-based preliminary velocity predictions from HPT with score-derived onset, frame, or onset-excluded frame matrices, and processes the resulting sequence with a BiLSTM having 256 hidden units per direction, followed by a fully connected Sigmoid layer with 88 outputs (He et al., 11 Aug 2025). The system is explicitly a correction module rather than a replacement for the AMT backbone (He et al., 11 Aug 2025).

Image-based filling reframes the problem as dense prediction on a sparse $v$4 piano-roll image with onset, frame, and velocity channels. The U-Net model uses $v$5 convolutions, $v$6 max pooling, transpose convolutions with $v$7 kernels and stride 2, skip connections, and windowed scaled dot-product attention over non-overlapping $v$8 windows, with best result reported for $v$9 windows (He et al., 11 Aug 2025). Its loss function is tailored to sparsity: a combined objective of BCE and cosine similarity, onset masking, and a V-shaped weighting centered at velocity 64 normalized as 0.5 (He et al., 11 Aug 2025).

Generative models are more heterogeneous. MIDI-VAE uses three parallel RNN encoder/decoder pairs for pitch, velocity, and instrument rolls, concatenates their encoder outputs, predicts latent Gaussian parameters $(\Delta t, v, p, \alpha)$0, and reconstructs velocity with an MSE term inside the full VAE loss (Brunner et al., 2018). Notochord uses a GRU backbone with an MLP head per sub-event; velocity is modeled by a discretized mixture of logistics, trained under maximum likelihood rather than MSE or cross-entropy (Shepardson et al., 2024). Drum transcription adapts Onsets and Frames by retaining the onset stack and adding a velocity stack with a Dense(88) velocity head trained using Mean Squared Error, with the velocity loss multiplied by 0.5 to prioritize hit recognition (Callender et al., 2020).

The diversity of objectives is correspondingly large. The acoustics-specific piano model uses $(\Delta t, v, p, \alpha)$1 velocity regression, optional Cross-Entropy for context classification, and RotoGrad for multi-task stabilization (Simonetta et al., 2022). Score-HPT uses a BCE loss on onset positions only (He et al., 11 Aug 2025). MIDI-VAE uses MSE for the velocity roll, with $(\Delta t, v, p, \alpha)$2 in the reported final models (Brunner et al., 2018). Notochord uses density-based negative log-likelihood (Shepardson et al., 2024). U-Net velocity filling uses masked and weighted BCE plus cosine similarity (He et al., 11 Aug 2025). These differences reflect distinct assumptions about continuity, sparsity, calibration, and interaction with note detection.

4. Acoustics, loudness, and expressive intent

One of the most consequential results in the area is that MIDI velocity should not be conflated with perceived loudness. The acoustics-specific piano study argues that standard AMT fails for resynthesis because the mapping from MIDI velocity to sound is not standardized across instruments or virtual pianos, and because performers adapt their playing to room, instrument, and recording context (Simonetta et al., 2022). It therefore separates “performance,” as the context-dependent physical act captured in audio, from “interpretation,” as the context-independent artistic layer, and seeks an acoustics-specific MIDI representation that preserves interpretation under a target context (Simonetta et al., 2022).

The psychoacoustic loudness study sharpens this distinction by explicitly separating physical intensity, perceptual loudness, and MIDI velocity (Qu et al., 2022). Piano tones are parameterized as $(\Delta t, v, p, \alpha)$3, and the goal is first to estimate loudness $(\Delta t, v, p, \alpha)$4, then to adjust velocities across player pianos or acoustic environments so that the perceptual effect is preserved (Qu et al., 2022). Equal-loudness contours are measured via the point of subjective equality using a randomized maximum likelihood sequential procedure, and a machine-learning model based on 8 mel-frequency bins, window size 2048, hop size 512, sample rate 22050 Hz, and the first 5 time frames after onset is trained from pairwise comparison data (Qu et al., 2022).

The loudness results show that intensity-based baselines fail more clearly than perceptual models for cross-pitch loudness comparisons, while the proposed parametric and non-parametric loudness models support better velocity remapping in downstream performance control transfer (Qu et al., 2022). In a listening test with 21 participants rating faithfulness, naturalness, and overall musicality, all loudness-based methods significantly outperform the intensity-based and raw transfer baselines with $(\Delta t, v, p, \alpha)$5 (Qu et al., 2022). This directly informs MIDI velocity prediction: when the output is intended for resynthesis, accurate velocity is not merely a regression problem on integers but an acoustically and perceptually conditioned control problem.

A common misconception is therefore that velocity prediction can be solved by matching amplitude or intensity alone. The piano loudness and acoustics-specific transcription papers both reject that view, albeit in different formulations (Qu et al., 2022, Simonetta et al., 2022).

5. Evaluation practices and empirical findings

Evaluation protocols vary with task. For acoustics-specific piano estimation, the main metric is average $(\Delta t, v, p, \alpha)$6 error on velocity prediction. The study trains 36 model shapes, four strategy variants per shape, and 144 trained models in total, although some are discarded for memory or invalid-loss issues (Simonetta et al., 2022). There is no single best strategy across all shapes, but acoustics-specific strategies generally improve performance; in the win analysis, the non-acoustics-specific Single-w/o baseline is best in only one configuration. Shapiro-Wilk rejects normality, Kruskal-Wallis is not significant overall with $(\Delta t, v, p, \alpha)$7, but pairwise Wilcoxon tests with Bonferroni-Holm correction show significant differences between Multiple-w/o and Single-w/o, and between Multiple-with and Single-w/o. An oracle-style selection of the best strategy per model shape yields highly statistically significant improvements with $(\Delta t, v, p, \alpha)$8 (Simonetta et al., 2022).

Score-HPT evaluates velocity correction on MAESTRO, SMD, and MAPS using MAE, STD of absolute error, and Recall derived from mir_eval Notew/OffVelo with 50 ms onset/offset tolerance and velocity tolerance of $(\Delta t, v, p, \alpha)$9 (He et al., 11 Aug 2025). On the MAESTRO test set, Original HPT has Recall $0$0; HPT with ground-truth onsets reaches MAE $0$1, STD $0$2, Recall $0$3; and Score-HPT variants improve further, with audio + onset obtaining MAE $0$4, STD $0$5, Recall $0$6, and audio + onset + frame$0$7 also reaching Recall $0$8 (He et al., 11 Aug 2025). On SMD, Score-HPT improves over HPT but does not beat FiLM U-Net; for example, HPT reports MAE $0$9, STD $127$0, Recall $127$1, whereas Score-HPT audio + onset + frame$127$2 reports MAE $127$3, STD $127$4, Recall $127$5, while FiLM U-Net reports MAE $127$6, STD $127$7, Recall $127$8 (He et al., 11 Aug 2025).

The U-Net image-colorization approach reports denormalized MAE, MSE, $127$9, $[0,1]$0, and Recall with 10% error tolerance on both SMD and MAESTRO (He et al., 11 Aug 2025). On SMD, the U-Net achieves MAE $[0,1]$1, MSE $[0,1]$2, $[0,1]$3 $[0,1]$4, Recall $[0,1]$5, and $[0,1]$6 $[0,1]$7, outperforming Flat, Seq2Seq, and ConvAE baselines on every objective metric (He et al., 11 Aug 2025). On MAESTRO test, the same model reports MAE $[0,1]$8, MSE $[0,1]$9, $[0,127]$0 $[0,127]$1, Recall $[0,127]$2, and $[0,127]$3 $[0,127]$4 (He et al., 11 Aug 2025).

Drum transcription provides an important counterpoint regarding metrics. OaF-Drums reports E-GMD F-measure $[0,127]$5 and velocity-aware E-GMD F-measure $[0,127]$6, demonstrating that velocity prediction is substantially harder than onset classification (Callender et al., 2020). Yet listening tests on 2,976 pairwise questions show that OaF-Drums with predicted velocity receives 919 wins, compared with 456 for the same model with velocity fixed constant, 677 for DT-Ensemble, and 372 for ADTLib (Callender et al., 2020). Kruskal-Wallis yields $[0,127]$7, and all pairwise post-hoc Wilcoxon differences are significant with $[0,127]$8 (Callender et al., 2020). This establishes a broader methodological point: standard classifier metrics such as accuracy and F-measure may not be sufficient proxies for downstream perceptual quality when the output is rendered as MIDI (Callender et al., 2020).

MIDI-VAE, although not a standalone supervised predictor, offers evidence that velocity patterns are learnable and style-relevant. Reported test velocity decoder MSE values are 0.029 for CvJ, 0.036 for CvP, 0.048 for JvP, and 0.033 for BvM (Brunner et al., 2018). In style-transfer experiments, velocity classifier accuracies decrease after transfer—e.g., $[0,127]$9 for CvJ and $[0,127]$0 for BvM—supporting the claim that latent manipulations alter velocity patterns in musically meaningful ways (Brunner et al., 2018).

6. Expressiveness detection, limitations, and open directions

Velocity is widely treated as an expressive cue, but its diagnostic power is not absolute. The GigaMIDI work defines DNVR as the normalized count of distinct velocity levels,

$[0,127]$1

and uses it as the velocity axis in a four-class expressive-performance scheme: NE, EO, EV, and EP (Lee et al., 24 Feb 2025). The optimal DNVR threshold is $[0,127]$2, with $[0,127]$3; DNVR accuracy is $[0,127]$4, improving over the distinct velocity baseline at $[0,127]$5 but trailing onset-based heuristics such as DNODR at $[0,127]$6 and NOMML at $[0,127]$7 (Lee et al., 24 Feb 2025). The paper notes that some non-expressive score-derived MIDI tracks still contain velocity variation because notation may encode traditional dynamics. This directly challenges the simplified assumption that “more velocity variation” is sufficient evidence of expressive performance (Lee et al., 24 Feb 2025).

Several limitations recur across predictive systems. Score-HPT assumes perfectly time-aligned MIDI score information and perfect timing correction, a strong assumption that the authors explicitly flag as limiting real-world applicability (He et al., 11 Aug 2025). The U-Net filling study is piano-only and attributes broader multi-instrument evaluation limits to dataset availability (He et al., 11 Aug 2025). The expressive piano generation paper reports only qualitative evidence, notes that the model was trained only two epochs, and acknowledges that generated sequences may sound incoherent and random at certain points (Liu, 2024). The psychoacoustic loudness work models only pitch and velocity, not duration or pedal, and measures equal-loudness contours on only 9 pitches before generalizing to all 88 keys (Qu et al., 2022). The acoustics-specific transcription paper suggests extensibility to parameters such as pedaling and more accurate note offset inference, but presents velocity as the demonstrated task (Simonetta et al., 2022).

Taken together, these works indicate that MIDI velocity prediction is not a solved scalar-regression problem. It is a family of tasks whose appropriate formulation depends on whether the objective is faithful resynthesis, AMT refinement, symbolic rendering, style transfer, interactive generation, or expressive-performance analysis. A plausible implication is that progress in the field depends less on a single universal architecture than on matching representation, loss, and evaluation protocol to the intended downstream use.