RL-Duet: Real-Time Deep RL Music Accompaniment

• The paper introduces the first deep reinforcement learning framework for online music accompaniment, optimizing duet improvisation via an ensemble-based reward model.
• It formalizes accompaniment as a Markov Decision Process and employs an on-policy actor–critic system for fast, fine-grained, real-time responses.
• The framework demonstrates superior melodic, harmonic, and stylistic compatibility over MLE and rule-based baselines, validated by both objective metrics and listener studies.

RL-Duet is a deep reinforcement learning framework for online, real-time music accompaniment generation. Distinct from offline music generation methods, RL-Duet addresses the interactive, sequential nature of duet improvisation by coupling a policy network with a learned, ensemble-based reward model, enabling the agent to respond to a live human part at a fine temporal granularity. The model demonstrates superior melodic, harmonic, and stylistic compatibility compared to maximum likelihood and rule-based baselines, validated through both objective measures and human preference studies (Jiang et al., 2020).

1. Formalization as a Markov Decision Process

RL-Duet casts online accompaniment generation as a Markov Decision Process (MDP). The state space at time $t$, $s_t = (s_t^h, s_t^m)$, encodes two clipped sequences: $s_t^h = h_0 : h_{t-1}$ (the human's preceding notes) and $s_t^m = m_0 : m_{t-1}$ (the machine's own generated notes prior to $t$). Each sequence is maintained within a sliding window of fixed length $L$, with note tokens drawn from a discrete vocabulary covering pitches, holds, rests, and beat subdivisions (typical vocabulary size: 50–90 tokens).

At each sixteenth-note step $t$, the agent observes $s_t$, receives a new human note $h_t$, and selects an action $m_t \in A$ (the next machine note). The process advances chronologically, with the updated state $s_{t+1}$ reflecting both agents’ outputs up to $t$. Transitions thus simultaneously encode human and agent histories, supporting fine-grained responsiveness required for real-time duet performance.

2. Ensemble-Based Learned Reward Model

The reward signal in RL-Duet eschews hand-crafted music-theoretic criteria in favor of a data-driven, ensemble-based construction. The immediate reward at step $t$ is defined as:

$$r_t = \frac{1}{K} \sum_{k=1}^K f_k(m_t \mid \text{context}_k) + r_\text{rule}(m_t)$$

where $K=6$ neural “reward specialist” sub-models $f_k$ are pre-trained by maximum likelihood on Bach chorale corpora, each conditioned on distinct contextual views:

• Joint pre-context predictor $p(m_t \mid s_t)$
• Joint pre- and post-context predictor $p(m_t : m_{t+\Delta} \mid \text{augmented } s_t)$
• Horizontal (intra-machine) Cloze: $p(m_t : m_{t+\Delta} \mid$ machine context$)$
• Vertical (inter-part harmony): $p(m_t : m_{t+\Delta} \mid$ human context$)$
• Plus three variants of the pre-context model trained with different learning rates

The ensemble captures both melodic (horizontal) and harmonic (vertical) compatibilities. An auxiliary penalty $r_\text{rule}(m_t) = -1$ discourages excessive repetition of the same pitch. Each $f_k$ is trained to predict masked targets from its specific contextual window, enforcing musicological coherence both within and between parts.

3. Policy Network and Actor–Critic Architecture

The policy and value functions are parameterized via an actor–critic architecture sharing a unified “backbone”. The input layer maps machine and human note tokens, as well as beat subdivisions, to learned embeddings of dimension $\approx 256$.

Separate branches process the embedded human, machine, and beat sequences through two-layer bi-directional GRUs. Outputs are summarized temporally using both max-pooling and attention to produce fixed-length vectors. These context vectors are concatenated and fed in parallel to the actor network (producing a softmax over action tokens, yielding $\pi_{\theta_a}(a_t \mid s_t)$) and the critic head (outputting a scalar value estimate $V_{\theta_v}(s_t)$).

4. Training Methodology

Training employs on-policy actor–critic techniques with Generalized Advantage Estimation (GAE) [Schulman et al. 2016]. Key mathematical definitions include:

• Return: $R_t = \sum_{i=0}^{T-t} \gamma^i r_{t+i}$
• TD error: $\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)$
• GAE: $$\hat{A}_t = \sum_{l=0}^{T-t} (\gamma \lambda)^l \delta_{t+l}$$

The policy gradient maximizes

$$J(\theta_a) = \mathbb{E}_\pi \left[ \sum_t \log \pi_{\theta_a}(a_t \mid s_t) \hat{A}_t \right]$$

and the critic loss is

$$L_v(\theta_v) = \mathbb{E} \left[ \sum_t \left( V_{\theta_v}(s_t) - R_t \right)^2 \right].$$

Notable hyperparameters include discount $\gamma=0.5$ and GAE $\lambda=1.0$. The generation network is initialized from reward-model (1) pretraining at learning rate $0.01$, and training covers 100,000 generated duets with on-policy updates and validation-based learning rate tuning. Exploration relies solely on on-policy sampling.

5. Real-Time Online Duet Generation

During inference, RL-Duet operates with bounded latency. At each time $t$, the system receives the incoming human note $h_t$, forms the clipped state window $s_t$, and samples $m_t$ from the policy $\pi(a_t \mid s_t)$. The system does not employ beam search or lookahead, limiting computational delay to tens of milliseconds per step on typical hardware. Temporal dependencies and stylistic continuity are maintained via the recurrent and attention-driven backbone, with only the most recent $L$ steps stored in memory.

6. Experimental Protocol and Benchmarks

The system is trained and validated on 327/37 SATB Bach chorales, employing random (human, machine) part assignments and pitch transposition within MIDI range 36–81. For evaluation, 37 additional chorales yield 460 fixed duet pairs, initializing the machine part with ground truth for the initial two measures.

Baselines include:

1. MLE RNN: identical architecture to the RL-Duet actor, trained via cross-entropy.
2. RL-Rules: a SequenceTutor variant utilizing both the learned ensemble reward and ~10 hand-coded music rules.

Objective metrics comprise pitch count per bar (PC/bar), average pitch interval (PI), average inter-onset interval (IOI), pitch-class histogram (PCH), note-length histogram (NLH), and Earth-mover’s distance (EMD) for PCH and NLH distributions. RL-Duet exhibits minimal deviation from ground truth (e.g., PC/bar $\Delta=+0.12$ vs MLE $\Delta=-0.90$; EMD(PCH)$=0.0057$, superior to baselines).

Subjective evaluation engaged 125 online listeners, stratified by instrument training and classical listening background, in 2000 forced-choice pairwise comparisons (each subject compared two 8-second clips, same human part, one from RL-Duet, one from MLE). RL-Duet was preferred in 58.4% of comparisons, with preference correlating positively to participants’ musical expertise.

7. Contributions, Limitations, and Prospective Directions

Key contributions include:

• Introduction of the first deep RL framework for real-time, truly online music accompaniment.
• Data-driven, ensemble-based reward model jointly optimizing horizontal and vertical musical compatibility without extensive manual rule specification.
• Demonstrated improvements over MLE and rule-based RL baselines in both objective metrics and subjective listener preference.

Limitations are notable: the current system fixes the human part during training and evaluation; a fully interactive end-to-end human–machine duet system with live feedback remains unimplemented. Robustness of the reward ensemble could be enhanced by adversarial defenses. Prospective work includes extension to multi-instrumentation, richer rhythmic vocabularies, and further real-time latency reduction (Jiang et al., 2020).