LK_Jam: Real-Time AI Co-Performer

• LK_Jam is a real-time, bidirectional human–AI interactive music generation system that co-performs with live musicians by facilitating turn-taking and role awareness.
• It employs a role-aware GRU architecture with sparse event representation to capture micro-timing, harmonic context, and phrase boundaries in live improvisation.
• The system integrates a lightweight, lock-free C++/JUCE plugin with a three-stage training curriculum to ensure predictable, ultra-low latency performance in DAWs.

LK_Jam is a real-time, bidirectional human–AI interactive music generation system designed specifically for live improvisation, with jazz “trading fours” as its reference use case; it is implemented as a lightweight C++/JUCE audio plugin (and host) that can run inside a DAW and behave as an AI co-performer rather than an offline composition tool (Liu et al., 19 Jun 2026). In the supplied research record, the same label also appears as an interpretive shorthand in several jamming and anti-jamming contexts rather than as a single standardized term. This suggests that “LK_Jam” is domain-dependent: in music-AI research it denotes a role-aware GRU-based co-performer, whereas in communications, radar, and electronic-warfare literature it can denote link-level, learning-based, or likelihood-based jammer frameworks (Liu et al., 2017).

1. Interactive-music definition and problem setting

Within music-AI research, LK_Jam addresses the gap between powerful but offline, unidirectional generative models and the strict requirements of live, embodied musical interaction: ultra-low latency, turn-taking, and role-aware “conversation” between a human player and the AI (Liu et al., 19 Jun 2026). The system is framed as a move toward “virtual synergy”: the AI has to listen, analyze and respond in a way that fits into the temporal, harmonic, and interactive constraints of a live performance, rather than just extending a sequence.

The system models distinct roles, human versus AI, and phrase boundaries, allowing it to behave as an improvising partner with clear turns. It uses a multi-dimensional sparse event stream instead of a dense time grid, builds on a role-aware GRU architecture chosen for its strictly $O(1)$ single-step complexity and suitability for CPU-critical audio threads, and is engineered with a JUCE-based, lock-free, multi-threaded architecture and RTNeural inference for predictable, allocation-free performance inside DAWs (Liu et al., 19 Jun 2026).

Its immediate performance setting is monophonic improvisation in which a human and an AI alternate loops or choruses, typically in jazz-like settings, but the turn-taking logic is described as generalizable to other interactive music contexts. A plausible implication is that LK_Jam is less a generic offline symbolic generator than a tightly constrained co-performance system whose musical semantics and systems design were developed together.

2. Sparse event representation, harmonic context, and role semantics

LK_Jam abandons the computationally expensive fixed time-grid and instead represents music as a timestamp-based sparse event stream in which each actual MIDI note onset becomes one time step; there is no explicit representation of rests as grid cells (Liu et al., 19 Jun 2026). This design is motivated by jazz improvisation’s micro-timing, including swing, laid-back feel, and off-grid syncopation: a coarse grid destroys these nuances, while a very fine grid causes data sparsity and large sequence lengths.

At each note event $t$, the system constructs a composite feature vector

$$\begin{aligned} x_t = \text{Concat}\big( &E_{pitch}(p_t), E_{vel}(v_t), E_{time}(\Delta t_t), \ &C_t, E_{role}(r_t), E_{phrase}(loc_t) \big). \end{aligned}$$

Here, $E_{pitch}(p_t)$ is an embedding of MIDI pitch; $E_{vel}(v_t)$ is an embedding of velocity $0\text{–}127$; $E_{time}(\Delta t_t)$ embeds the time shift between the current and previous note onset; $C_t$ is a continuous harmonic context embedding following the “Theory Structured Harmonic Embeddings” idea; $E_{role}(r_t)$ is a role-aware encoding, typically with $r_t \in \{0,1\}$ distinguishing human versus AI events; and $t$0 is a phrase position identifier such as $t$1 (Liu et al., 19 Jun 2026).

Conceptually, each event can be viewed as the tuple

$t$2

which allows the model to see melody, dynamics, timing, harmony, and interaction context in a single time step. This representation is explicitly intended to capture turn-taking and phrase boundaries through $t$3 and $t$4, and to preserve expressive micro-timing through $t$5.

Turn-taking is encoded at the data level. Training sequences interleave human and AI events, with role labels $t$6 and phrase positions $t$7, and the “Trading Fours” scenario is generalized as loop-level role alternation: even loops for human and odd loops for AI, or vice versa. Appendix examples described in the report show human motif events with $t$8 and AI response events with $t$9, both under the same harmonic context $$\begin{aligned} x_t = \text{Concat}\big( &E_{pitch}(p_t), E_{vel}(v_t), E_{time}(\Delta t_t), \ &C_t, E_{role}(r_t), E_{phrase}(loc_t) \big). \end{aligned}$$0, with Start, Continue, and End phrase markers. The intended learned mapping is

$$\begin{aligned} x_t = \text{Concat}\big( &E_{pitch}(p_t), E_{vel}(v_t), E_{time}(\Delta t_t), \ &C_t, E_{role}(r_t), E_{phrase}(loc_t) \big). \end{aligned}$$1

with explicit awareness of “who played what, when, and where in the phrase” (Liu et al., 19 Jun 2026).

3. Role-aware GRU architecture and autoregressive generation

The core model is a lightweight GRU. The report does not specify exact layer counts or hidden sizes, but it explicitly emphasizes a small footprint to satisfy tight real-time constraints (Liu et al., 19 Jun 2026). The recurrent state update is described as

$$\begin{aligned} x_t = \text{Concat}\big( &E_{pitch}(p_t), E_{vel}(v_t), E_{time}(\Delta t_t), \ &C_t, E_{role}(r_t), E_{phrase}(loc_t) \big). \end{aligned}$$2

with the standard GRU gating equations given for reference: $$\begin{aligned} x_t = \text{Concat}\big( &E_{pitch}(p_t), E_{vel}(v_t), E_{time}(\Delta t_t), \ &C_t, E_{role}(r_t), E_{phrase}(loc_t) \big). \end{aligned}$$3 LK_Jam does not introduce new GRU gates; instead, role awareness is injected through the concatenated input embeddings $$\begin{aligned} x_t = \text{Concat}\big( &E_{pitch}(p_t), E_{vel}(v_t), E_{time}(\Delta t_t), \ &C_t, E_{role}(r_t), E_{phrase}(loc_t) \big). \end{aligned}$$4 and $$\begin{aligned} x_t = \text{Concat}\big( &E_{pitch}(p_t), E_{vel}(v_t), E_{time}(\Delta t_t), \ &C_t, E_{role}(r_t), E_{phrase}(loc_t) \big). \end{aligned}$$5 (Liu et al., 19 Jun 2026).

After the GRU layer or layers, a linear projection and softmax produce distributions for the next note attributes, especially pitch: $$\begin{aligned} x_t = \text{Concat}\big( &E_{pitch}(p_t), E_{vel}(v_t), E_{time}(\Delta t_t), \ &C_t, E_{role}(r_t), E_{phrase}(loc_t) \big). \end{aligned}$$6 The system then samples or selects appropriate pitch, and possibly other attributes depending on decoder design. During training, the model sees sequences that alternately contain human and AI segments, along with $$\begin{aligned} x_t = \text{Concat}\big( &E_{pitch}(p_t), E_{vel}(v_t), E_{time}(\Delta t_t), \ &C_t, E_{role}(r_t), E_{phrase}(loc_t) \big). \end{aligned}$$7 and $$\begin{aligned} x_t = \text{Concat}\big( &E_{pitch}(p_t), E_{vel}(v_t), E_{time}(\Delta t_t), \ &C_t, E_{role}(r_t), E_{phrase}(loc_t) \big). \end{aligned}$$8, and learns to interpret human segments as context and AI segments as targets conditioned on harmonic context $$\begin{aligned} x_t = \text{Concat}\big( &E_{pitch}(p_t), E_{vel}(v_t), E_{time}(\Delta t_t), \ &C_t, E_{role}(r_t), E_{phrase}(loc_t) \big). \end{aligned}$$9. At inference, the plugin captures MIDI events during a human turn, buffers them in the human role, and at the turn boundary the inference thread processes those events with $E_{pitch}(p_t)$0 and generates a response sequence labeled as $E_{pitch}(p_t)$1 for the next loop (Liu et al., 19 Jun 2026).

The training pipeline is autoregressive sequence modeling on the sparse event stream. Input at each step is $E_{pitch}(p_t)$2; the target is typically the next event’s pitch and possibly other attributes. Teacher forcing is combined with Scheduled Sampling, gradually increasing the probability of using the model’s own previous predictions instead of ground truth in order to mitigate exposure bias and improve coherence over long phrases. The report states that there is no explicit custom loss formula beyond standard cross-entropy on the discrete outputs, while role and phrase embeddings function as implicit structural constraints (Liu et al., 19 Jun 2026).

4. Real-time architecture, lock-free communication, and audio safety

LK_Jam is implemented in C++ using the JUCE framework and follows a classic triple-thread design: a high-priority Audio Thread, a background Inference or Worker Thread, and a UI or Message Thread (Liu et al., 19 Jun 2026). The Audio Thread captures incoming MIDI, tracks host transport, PPQ, BPM, and bar or beat indices, and performs minimal state scheduling and playback of already-generated AI MIDI events; it must avoid blocking operations, locks, or dynamic allocation. The Worker Thread concatenates features, runs GRU forward passes via RTNeural, and generates a full phrase of AI response during the short window between turns. The UI Thread handles graphics and visualization and is designed to be light and non-blocking, for example by updating at approximately $E_{pitch}(p_t)$3 Hz and capping note updates per cycle.

Communication between threads relies on lock-free, single-producer single-consumer queues and atomic operations. For heavier but infrequently updated data such as chord progressions, LK_Jam uses atomic double buffering: two buffers are pre-allocated, the UI writes into the inactive buffer, and an atomic pointer swap makes the new harmonic context visible to the Audio Thread instantly, without locks or allocations (Liu et al., 19 Jun 2026). The report’s engineering claim is that this prevents the Audio Thread from ever blocking on locks or memory allocation and thereby reduces dropout risk.

Host synchronization is tied to the DAW timeline rather than an internal free-running clock. In host sync mode, the plugin queries the host’s PPQ position and BPM, enters a PreRoll state one bar before a loop start to collect necessary harmonic context, and switches role precisely when the loop position hits the next bar boundary (Liu et al., 19 Jun 2026). This means that role switch and turn-taking are driven directly by the DAW timeline, which is significant for synchronization with backing tracks or click.

The inference backend is RTNeural, described as a C++ library designed for real-time neural networks in audio. LK_Jam uses compile-time topology solidification through C++ templates, zero-allocation runtime with all memory allocated upfront, and an implementation characterized as “wait-free” from the perspective of the Audio Thread, with no internal thread pools, hidden synchronization, or dynamic memory operations. Vectorized math via Eigen or xsimd is also noted (Liu et al., 19 Jun 2026).

The report’s $E_{pitch}(p_t)$4 claim is explicitly from the audio thread’s point of view. A GRU’s single-step forward pass has constant complexity with respect to sequence length, and because the system uses a sparse event stream and performs decoding on the Worker Thread, the per-callback overhead seen by the Audio Thread is fixed and bounded. This is contrasted with embedding a large Transformer model or a general-purpose runtime in the audio thread, which would exhibit $E_{pitch}(p_t)$5 or $E_{pitch}(p_t)$6 behavior with respect to sequence length or attention context and may allocate memory or spawn threads at unpredictable times (Liu et al., 19 Jun 2026).

5. Curriculum design, data hierarchy, and evaluation protocol

The dataset is constructed hierarchically to support a three-stage training curriculum (Liu et al., 19 Jun 2026). The Basic Layer consists of algorithmically generated monophonic melodies over simple, canonical progressions such as ii–V–I; it is highly constrained, using chord tones only and no non-chord tones or complex syncopation, with the stated goal of learning atomic harmonic mapping between chord context $E_{pitch}(p_t)$7 and pitch. The Advanced Layer incorporates public jazz solo datasets and more complex harmonic scenarios such as I–vi–ii–V and blues forms; it introduces non-chord tones, chromatic approaches, bebop scales, syncopations, and other idiomatic devices, with the goal of learning stylistic vocabulary and self-motif development within a single turn. The Expert Layer comprises custom, professional-level human–computer call-and-response corpora in which motif pairs are labeled with explicit relationships such as imitation, inversion, motif response, and conflict, with the goal of learning cross-turn interplay and the “listen-analyze-respond” cycle at an expert level. All datasets are augmented by 12-key transposition.

The three training stages are named Stage 1: Atomic Harmonization, Stage 2: Stylistic Wandering (Linear Vocabulary Expansion), and Stage 3: Expert-Level Interaction Alignment (Liu et al., 19 Jun 2026). In Stage 1, the model learns the mapping $E_{pitch}(p_t)$8 in a simple environment. In Stage 2, it is exposed to real jazz solos and more complex patterns, including chromatic approaches and enclosures, bebop scales and standard jazz vocabulary, syncopated rhythms, rhythmic displacement, and motif sequencing, with emphasis on phrase self-motif development. In Stage 3, supervised fine-tuning explicitly activates role-aware encoding $E_{pitch}(p_t)$9; sequences contain clearly labeled human and AI parts, and motif pairs across turns are aligned according to interaction types such as imitation, inversion, and complementary response. The stated objective is to break the model’s tendency for infinite continuation and enforce closed-loop dialogue.

Optimization is described as standard autoregressive cross-entropy over next-event predictions, especially pitch, with perplexity tracked over stages (Liu et al., 19 Jun 2026). Planned ablations remove role encoding $E_{vel}(v_t)$0 or phrase encoding $E_{vel}(v_t)$1 to test their impact on thematic coherence and phrase structure. The authors hypothesize that without phrase markers melodies become overly long and lack clear cadences, and without role markers interaction logic breaks down and the model reverts to unidirectional continuation. Subjective evaluation is planned using a six-dimension framework comprising motif response strategy, phrase boundary, harmonic adherence, linear ornamentation, rhythmic density, and melodic contour.

6. Reported musical behavior, limitations, and implications

The report is explicit that it does not provide detailed benchmark numbers, but it argues from architectural analysis that peak CPU usage on the audio thread is strictly bounded and independent of the length or complexity of AI phrases because decoding is offloaded to the Worker Thread and RTNeural provides zero-allocation, constant-time inference per step (Liu et al., 19 Jun 2026). It also states that, compared to architectures using external Python processes or general inference frameworks with inter-process communication or hidden thread pools, the design significantly reduces the risk of dropouts and jitter in VST3 environments.

Preliminary observations and design intent indicate that the AI produces harmonically coherent lines that respect chord changes, particularly after Stage 1 and Stage 2 training; that explicit role and phrase encodings help generate phrases with clear starts, developments, and cadences rather than endless note streams; that in Stage 3 the system can emulate interaction strategies such as imitation and inversion in a recognizably conversational way; and that micro-timing and groove are preserved through $E_{vel}(v_t)$2 modeling rather than rigid grid quantization (Liu et al., 19 Jun 2026). In the example interaction scenario, a guitarist or keyboardist loads LK_Jam as a MIDI plugin in a DAW, loops over a chord progression such as a 32-bar AABA form, improvises during the human turn, and then hears an AI phrase generated at the loop boundary using the buffered human events, $E_{vel}(v_t)$3, $E_{vel}(v_t)$4 labels, and harmonic context $E_{vel}(v_t)$5.

Several limitations are explicitly acknowledged. The current architecture is monophonic and does not generate multi-voice, chord voicing, or polyphonic accompaniment. GRUs may suffer memory decay over very long sequences, which may limit very long-form structural planning. Training data is heavily jazz and improvisation oriented, and generalization to other styles, instruments, or ensembles is untested. The report does not yet present systematic tests for out-of-distribution human input such as very dense or sparse playing or extreme tempos. Evaluation remains mostly architectural and qualitative, with large-scale user studies and blind listening tests planned but not yet reported (Liu et al., 19 Jun 2026).

The broader implication drawn in the report is that LK_Jam exemplifies how Embodied AI ideas—real-time interaction, role awareness, and situated behavior—can be realized in a music AI co-performer, and that the combination of role identifiers and phrase markers in the core representation may be applicable to other interactive systems such as co-improvising robots and real-time generative accompanists (Liu et al., 19 Jun 2026). A plausible implication is that the project’s main contribution lies in the coupling of symbolic representation, recurrent modeling, and real-time audio-systems engineering rather than in scale alone.

7. Other research uses of “LK_Jam”

The broader literature supplied here does not use “LK_Jam” as a single universally fixed term. Instead, several papers use it interpretively for distinct jamming-related mechanisms. This suggests that the label functions as a context-sensitive shorthand outside the music-AI setting.

Context	Interpretation of “LK_Jam”	Paper
Music AI	Real-time, bidirectional human–AI interactive music generation system	(Liu et al., 19 Jun 2026)
Secure SWIPT	Link-level or physical-layer jamming–aided security mechanism	(Liu et al., 2017)
Radar anti-jamming	Learning-based anti-jamming framed as Online Convex Optimization	(Liu et al., 2024)
Wireless anti-jamming	Learning-based jammer exploitation with ambient backscatter and dueling DQN	(Huynh et al., 2019)
Multi-UAV electronic attack	Link-level jamming via cooperative UAV scheduling	(Jiang et al., 2023)
Radar jammer detection	Likelihood-based jammer detector based on sparse learning	(Yan et al., 2020)

In secure SWIPT, the label is used to describe a wireless-powered cooperative jamming mechanism in which a full-duplex jammer harvests energy from the transmitter and transmits artificial interference to degrade the eavesdropper while also charging the energy receiver; the goal is secrecy-rate maximization through joint optimization of transmitter power and jammer power under harvested-energy constraints (Liu et al., 2017). In radar anti-jamming, it is used to contextualize an Online Convex Optimization framework against a DRFM-based intelligent jammer, with domain-knowledge-enhanced gradient estimators and sub-linear static regret and universal regret results (Liu et al., 2024).

In wireless anti-jamming for communications, the label is associated with a design that combines ambient backscattering and a deep dueling neural network architecture so that a legitimate transmitter can learn the jammer’s strategy and either adapt the rate or transmit information on the jamming signal itself; the reported gains include up to $E_{vel}(v_t)$6 improvement in average throughput and $E_{vel}(v_t)$7 reduction in packet loss (Huynh et al., 2019). In multi-UAV cooperative jamming, it denotes a link-level jamming problem in which UAV deployment and directional antenna orientations are jointly optimized to minimize targets’ average SINR by an ADMM-based iterative algorithm (Jiang et al., 2023). In radar array processing, it denotes a likelihood-based jammer detector in which sparse cyclic estimation and likelihood ratio tests are used to detect multiple noise-like jammers and estimate their angles of arrival (Yan et al., 2020).

Accordingly, the most stable and direct use of the name is the 2026 music-AI system “LK Jam: System Architecture and Implementation of a Real-Time Human-AI Interactive Music Generation System using Role-Aware GRU” (Liu et al., 19 Jun 2026). In other fields, “LK_Jam” is best understood as an interpretive convenience rather than a standardized canonical designation.