ACCompanion Framework: Real-Time Accompaniment

• ACCompanion framework is a modular system that delivers real-time, expressive piano accompaniment through integrated score following, probabilistic tempo inference, and machine-learned generation.
• Its architecture features discrete modules including MIDI handling, score tracking via HMM or OLTW, and an expressivity engine based on the Basis-Mixer network, ensuring low latency and high alignment accuracy.
• The system supports advanced sensorimotor synchronization and extensibility with neural models, making it robust for both experimental and live performance environments.

The ACCompanion framework refers to a class of systems for real-time, expressive, and adaptive musical accompaniment, primarily for piano, that tightly integrate symbolic score following, probabilistic tempo inference, and machine-learned expressive generation. The term encompasses implementations ranging from early prototypes to highly robust public-performance-ready software, each characterized by a modular architecture and rigorous mathematical underpinnings for hybrid score-following and expressivity modeling (Cancino-Chacón et al., 2023, Cancino-Chacón et al., 2017).

1. Architectural Overview

The ACCompanion system consistently employs a modular pipeline architecture, typically decomposed into the following real-time modules:

• MIDI Handler: Aggregates, timestamps, and forwards incoming MIDI events from the soloist. Batching occurs in fixed 10 ms windows to accommodate polyphony; processed events are delivered with end-to-end latency constrained to under 20 ms.
• Score Follower: Aligns live performance with the notated score, emitting current score positions and estimates of tempo (beat period). Implemented alternately as a switching Kalman-filter Hidden Markov Model (HMM) or an Online Linear Time Warping (OLTW) ensemble for complex alignments.
• Accompanist (Encoder/Decoder): Computes expressive performance parameters—tempo, dynamics, articulation—from both score and observed performance, and generates corresponding accompaniment events from these descriptors.
• Expressivity Engine (v0.1 and earlier): Employs the Basis-Mixer neural architecture for phrase-level and note-level expressive targets.
• MIDI Output & Visualization: Outputs generated accompaniment MIDI, often paired with real-time visual feedback (e.g., piano roll interface).

The data flow is strictly sequential: MIDI Input → Score Follower → Expressivity Encoder → Accompaniment Decoder → Output (Cancino-Chacón et al., 2023, Cancino-Chacón et al., 2017).

2. Score Following Algorithms

The ACCompanion's core innovation is probabilistic real-time score tracking. Two principal alignment strategies are implemented:

• HMM-based Score Follower: Models the soloist's position as a discrete latent variable $s_t$ (score position) and jointly infers continuous tempo state $z_t$ (beat period), maintaining distributions via forward filtering. Observation model includes pitch likelihoods and Gaussian models of inter-onset intervals (IOIs), parameterized by current tempo estimate. State transitions admit self-loops (insertions), forward skips (deletions), and small backward moves.

$$p(s_t, z_t | s_{t-1}, z_{t-1}) = A(s_{t-1}, s_t)\,\mathcal{N}(z_t; F(s_t)z_{t-1}, Q(s_t))$$

$$p(y_t | s_t, z_t) = p(p_t | s_t)\,\mathcal{N}(\Delta t_\text{perf}; z_t \delta^\text{score}(s_t), R)$$

• Online Linear Time Warping (OLTW): Maintains sliding windows over one or more reference recordings of the score. Multiple OLTW followers are run in parallel; their average provides robust location estimates in highly ornamented or rhythmically variable passages.

Selecting between HMM and OLTW is context-dependent; the latter demonstrates superior robustness to trills and ornaments, as shown empirically (Cancino-Chacón et al., 2023).

3. Expressive Performance Generation

Expressive accompaniment in ACCompanion is accomplished via an explicit Encoder/Decoder split:

• Encoder: Extracts low-level descriptors from the soloist's live performance, including:
  • Tempo ($b_t$): Beat period, computed as the ratio of live IOI to notated IOI.
  • Dynamics ($v_t$): Median or mean MIDI velocity.
  • Articulation ($a_t$): Log-ratio of performed-to-notated durations.
• Decoder: Computes, for each upcoming accompaniment note, the actual onset, velocity, and duration:

$$\begin{align*} \tau_j^\text{acc} &= \tau_t^\text{solo} + b_t \Delta_t^\text{score} \ v_j^\text{acc} &= v_j^\text{base} \cdot \frac{v_t}{\bar{v}_\text{ref}} \ d_j^\text{acc} &= d_j^\text{s} \cdot b_t \cdot 2^{a_t} \end{align*}$$

Smoothing and clamping are enforced to ensure musical plausibility.

In earlier implementations (notably v0.1), the Basis-Mixer network outputs five expressive control targets, distributed between onsetwise (RNN) and notewise (feedforward) architectures (Cancino-Chacón et al., 2017).

4. Sensorimotor Synchronization and Tempo Prediction

ACCompanion incorporates several sensorimotor synchronization (SMS) models for real-time prediction of soloist tempo and onset. These include:

• Reactive (R) and Moving Average (MA): Simple extrapolation based on last observed beat period.
• Linear SMS (L) and Linear Tempo Expectation (LTE): Employ correction terms to compensate for asynchronies and anticipate future tempo, with LTE using a rehearsal-derived expectation function.
• Joint Adaptation–Anticipation (JADAM) and Kalman Tempo (KT): Blend adaptation to recent data with anticipation, or employ a 1D Kalman filter for hidden tempo state estimation.

Empirical results demonstrate LTE's superior prediction accuracy, minimizing onset and beat period error in complex piano repertoire (Cancino-Chacón et al., 2023):

Model	Onset error (ms)	Tempo error (ms/beat)
LTE	23.3	63.3
L	81.9	173
KT	1154	177
R	4280	209

5. System Evaluation, Robustness, and Latency

Performance evaluation spans both technical and user-centered criteria:

• Latency: Aggregated system latency remains below 15 ms (key-to-speaker) (Cancino-Chacón et al., 2023).
• Alignment Accuracy: On challenging musical material, OLTW achieves sub-100 ms alignment 86.7% of the time; HMM lags significantly in highly ornamented contexts.
• Expressivity: Subjective feedback highlights convincing handling of rubato and fermatas; musicians note the lack of bidirectional non-verbal cues (visual, haptic) as a limiting factor.
• Robustness: The modular architecture allows OLTW and HMM states to absorb and correct for performance errors, spontaneous ornaments, and unexpected tempo changes.

Real-world deployments include keynotes, exhibitions, and public performances, with documented musician testimonials citing both strengths (expressivity, reliability) and areas for improvement (cueing, communication) (Cancino-Chacón et al., 2023).

6. Implementation, Extensibility, and Future Directions

ACCompanion reference implementations are in Python 3 with Mido (MIDI I/O) and Partitura (score parsing); C extensions and GPU/CPP ports are considered for future upgrades. System modularity explicitly supports:

• Replacing monophonic with polyphonic score-followers.
• Swapping neural expressivity engines (e.g., Transformer-based models).
• Adding novel expressive targets—including pedal control and sound color.
• Enabling plugin APIs for third-party contributions (Cancino-Chacón et al., 2017).

Planned advances include end-to-end learning architectures, integration of visual or haptic cues, and explicit modeling of accompanist–soloist social interaction feedback loops. Open-source release and systematic user evaluation are scheduled (Cancino-Chacón et al., 2023).

7. Summary and Significance

ACCompanion establishes a reference pipeline for real-time, musically expressive, automatic accompaniment grounded in probabilistic score alignment, adaptive tempo inference, and expressive performance synthesis. Distinguishing features are its support for high reactivity to the soloist, robustness to performance idiosyncrasies, and flexible expressiveness enveloping tempo, dynamics, and articulation. Extensive empirical results confirm high alignment accuracy and low latency under concert conditions. Future directions aim at deepening the system's anticipatory and communicative faculties (Cancino-Chacón et al., 2023, Cancino-Chacón et al., 2017).