- The paper introduces a co-designed communication model that represents improvisation through Initiate and Acknowledge actions forming proposal, stability, and negotiation states.
- The H2H dataset contains 37 clips totaling 6 hours and 8 minutes of synchronized audio-visual duo improvisation, with 1,368 annotations from six pairings of five expert musicians.
- The paper finds strong agreement for stability states at 81.3% median IoU but much lower agreement for proposals at 23.6% and negotiations at 0%, demonstrating that musical intention and perceived communication often diverge.
Motivation and problem statement
Real-time AI improvisation systems can produce locally coherent musical output, but they typically treat communication as an afterthought: interaction strategies such as trading, call-and-response, or feedback-button negotiation are layered post-hoc onto generative algorithms through explicit controls and predefined modes (2608.13957). The authors argue this is a design choice rather than an inevitability, and that a key obstacle is the absence of a formalized, machine-readable model of how human musicians actually communicate during improvisation. They address this gap by studying free (non-idiomatic) improvisation — following Bailey's definition, where no pre-agreed material, tempo, key, or roles constrain the performance — because it strips away stylistic scaffolds and forces communication itself to carry much of the musical structure.
A co-designed communication model
The communication model was developed through a six-month bi-monthly co-design process involving expert improvisers alongside researchers in music information retrieval, machine learning, and HCI. Pilot playing sessions followed by joint recording reviews surfaced a central design insight: musicians sometimes hold divergent interpretations of the same moment — what one intends as a proposal of a new idea, the other may perceive as continuation or as a signal to move elsewhere. This asymmetry between produced and perceived intention directly motivated the two-sided annotation scheme.
The formalization represents duo improvisation as point-in-time actions on a one-dimensional timeline:
- Initiate (i): introduction of a new musical idea or sonic space; consecutive initiations occur when ideas go unacknowledged.
- Acknowledge (a): a response that takes up a preceding initiation from the partner; it never introduces new material.
Sequences of these actions compose three states: proposal (an initiation awaiting acknowledgment), stability (both musicians developing an accepted idea, whether jointly or in parallel streams), and negotiation (one or more unacknowledged initiations in search of alignment). An improvisation is thus an alternating sequence of states, and each musician annotates timestamps of actions and actors for both their own intentions and their perception of their partner's.
The H2H Music Improvisation dataset
The dataset comprises 37 clips totaling 6 hours 8 minutes of audio-visual duo free improvisation, recorded across 6 pairings drawn from 5 expert improvisers (guitar, trombone ×2, saxophone, trumpet). Recording prioritized per-player stems without cross-bleed: musicians performed in acoustically isolated rooms connected visually through a window and via low-latency monitoring, at 48 kHz/16-bit mono audio synchronized with two professional camcorders. Four pairs were first-time collaborators and two had prior shared performance history, enabling future analysis of how familiarity shapes communication dynamics. In total the dataset contains 1,368 annotated actions, with each musician annotating both self-intentions and perceived partner intentions post-hoc in Reaper — post-hoc annotation being necessary because any in-session task would alter the music itself.
Beyond improvisation research, the clean stems and synchronized video support multi-modal work relating gesture to music development and multi-modal source separation tasks.
Intention–perception alignment results
The two-sided annotations permit direct quantification of the intention–perception gap using overall state agreement and per-state Intersection-over-Union (IoU). The results are striking:
| State |
Median agreement |
| Stability |
81.3% |
| Proposal |
23.6% |
| Negotiation |
0% |
Stability is reliably co-perceived once established. Proposal agreement is low, reflecting frequent disagreement about when or which initiation was taken up. Negotiation shows essentially no overlap — 27 of 37 clips have zero negotiation IoU — which the authors interpret not as annotation error but as evidence that one musician's experience of putting forward unaccepted initiations may simply not register as negotiation for the partner. These findings empirically validate the co-design decision to collect two-sided annotations: a single annotator's perspective cannot capture improvisational communication. At the clip level, paired annotations also mirror a known challenge in music structure annotation, where annotators agree on broad segmentation while diverging on hierarchical granularity and state assignment.
Qualitative insights from co-design
Thematic analysis of session discussions yielded several findings relevant to system design. Musicians reported attending primarily to texture rather than pitch, citing note length, onset density, rhythmic and melodic shape, energy, timbre, and groove as primary reference dimensions. Signaling mechanisms include increased energy, repetition with variation, silence (which reliably predicts a forthcoming initiation), and switching to repetitive accompaniment as endorsement; endings require dedicated signals such as decrescendo fades or sudden stops, often accompanied by visual contact. Role dynamics operate along multiple independent axes simultaneously — one musician may lead melodically while following rhythmically — and role cannot be inferred from sonic output alone. Social hierarchy also matters: across all pairings, guitar1 initiated approximately 63% of perceived initiations on average, and less experienced players described deferring to senior musicians.
Limitations
The authors are explicit about several constraints. Two observed interaction patterns fall outside the discrete event model: rapid acknowledgment without uptake, and gradual convergence of parallel streams without any discrete initiate/acknowledge event — suggesting the model may not capture continuous, emergent exchanges. Toward the end of hour-long sessions, musicians reported "creative fatigue," and day-to-day mental state independently affected creativity, both of which introduce variability into the data. Horn recordings occasionally exhibit audio clipping due to wide dynamic range. Finally, the participant pool is small and demographically skewed: five musicians, four man-identifying, all formally trained, all based in Atlanta, with only three active in free improvisation specifically.
Conclusion
This work contributes a machine-readable communication model for free improvisation — built from initiation/acknowledgment actions composing into proposal, stability, and negotiation states — together with the first annotated audio-visual dataset for free improvisation, offering six hours of clean-stem recordings with per-player intention and perception annotations. Its central empirical result, that paired musicians agree strongly on stability but diverge sharply on proposal and negotiation states, establishes the asymmetry between produced and perceived intention as a measurable phenomenon rather than anecdote. The open question the paper leaves is how such a communication model can be incorporated at the algorithm-design stage of ML-based generative systems, so that machine improvisers communicate by design rather than through post-hoc interaction modes.