Flute X GPT: Adaptive Flute Instruction

• Flute X GPT is a real-time LLM-driven interface blending GPT-4 with multi-modal hardware to deliver adaptive, sensor-based flute instruction.
• It employs a robust architecture integrating haptic gloves, sensor-augmented instruments, and dynamic feedback (audio and visual) to tailor instructional strategies using real-time performance data.
• The system exemplifies innovative LLM orchestration in interactive education, demonstrating rapid response latency and emergent workflow adaptations.

Flute X GPT is a “secretary-level” LLM-Agent User Interface (LAUI) that integrates a GPT-4-powered agent with multi-modal software and hardware for real-time, adaptive flute instruction. The system exemplifies an LLM agent’s capacity to coordinate a complex workflow, proactively discover novel instructional strategies, and mediate user interaction without requiring the learner’s prior technical knowledge. It is built atop the Music X Machine platform, incorporating haptic gloves, sensor-augmented instruments, real-time feedback modalities, and robotic output, orchestrated via an LLM-driven prompt manager and state machine (Chin et al., 19 May 2024).

1. System Architecture

Flute X GPT comprises three core components: the LLM agent (GPT-4 with function calling), a prompt manager/conversation parser/state machine, and the Music X Machine plus hardware stack (including text-to-speech, speech-to-text, and a robot teacher interface). Communications flow bidirectionally among these layers, facilitating continuous context-aware interaction. The prompt manager relays a fixed “system principles” message to the LLM, managing state, parsing user speech and sensor events, and routing output to hardware.

Input streams include user speech (via Whisper S2T) and performance data (timing, pitch, gesture). The LLM’s output is parsed into three channels: “thoughts” (internal reasoning, delimited by triple-quotes), “actions” (function calls), and “speech” (robot dialogue via text-to-speech). The architecture is designed for immediate loopback, with the agent re-queried after every speech or action unless a wait() is called, supporting sub-second response latency.

The hardware stack encompasses:

• Haptic gloves with servo-driven finger rings for force/hint-based feedback modes
• Sensor-augmented flute with capacitive finger and breath sensors
• Visual feedback (annotated staff overlays)
• Multi-stream audio mixing (reference, user, metronome)
• Timing and pitch error classification for real-time note assessment
• A monophonic song database (POP909) split for segmental practice

2. LLM Agent Orchestration

The LLM agent operates under a single, comprehensive “system principles” prompt (~1,000 words), codifying its instructional role, pedagogical strategies (e.g., Challenge Point Theory, scaffolding, fading), the Music X Machine function API, and stringent turn-taking logic. At runtime, the agent processes each user or system event, reasons stepwise in hidden “thoughts” (e.g., “Student is struggling with rests...”), selects from the defined function repertoire, and generates contextually appropriate robot speech.

Function calls available to the agent include but are not limited to InterruptSession(), ModifyTempo(), SetHapticMode(), and StartSession(). User-facing speech is generated separately, explicitly describing actions and next steps. Prompt-engineering leverages JSON-schema function definitions and segregation of internal “thought” from output “speech,” preserving transparency in decision processes and facilitating reliable parsing.

Latency is actively managed via full conversation history streaming, token-wise generation (<300 ms per token), and online estimates of text-to-speech compute time for batched audio synthesis.

3. Multi-Modal Feedback, Sensing, and Scoring

Flute X GPT’s feedback ecosystem integrates several streams:

• Haptic: Servo-driven glove modes including force, hint (triggers on note onset), fixed/adaptive timing (time-locked or user-paced).
• Visual: Note-by-note overlay with real-time error annotation; transparent to the LLM, which configures display as pedagogically warranted.
• Audio: Synchronized streams—reference audio, live user input, metronome—mixed algorithmically.
• Note classification: Rule-based formulas for timing (Δt = t_measured – t_expected; on_time if |Δt| ≤ ε, else early/late) and pitch (Δp = p_measured – p_target; classified into correct, octave_wrong (±12 semitones), or unrelated).

Classification streams are periodically summarized in explicit language ("The third note was 120 ms late, pitch correct") for the LLM agent, obviating the need for pose estimation or advanced DSP. This enables the agent to reactively scaffold exercises, interrupt to address error streaks, or dynamically reconfigure hardware modalities.

4. User Interaction Paradigms and Emergent Workflows

Interaction in Flute X GPT is conversational and context-sensitive, with the agent proactively guiding onboarding, practice sessions, and reactive interventions. The onboarding sequence introduces users to hardware and options (“Please put on the haptic gloves… would you like to hear the reference audio?”). During practice, the agent monitors real-time feedback events, interrupting as necessary (e.g., slowing tempo, enabling hint mode after repeated timing errors).

The LAUI approach enables “emergent workflows” beyond those anticipated in the original GUI, such as:

• Disabling visual feedback to encourage auditory learning
• Dynamically switching between haptic modes
• Segmenting practice dynamically in response to detected errors
• Explicitly querying the user about pedagogical focus (intonation vs. rhythm)

Example agent reasoning in response to a user statement (“I get lost on the rests.”):

SetHapticMode("hint"), ModifyTempo(0.7), ToggleVisual(true) “I’ll apply hint-mode guidance at each note onset and slow the tempo to 70%. You’ll see each note light up visually. Ready to try again?” wait()

This interaction flow showcases how the agent reasons, configures multi-modal feedback, explains its rationales, and solicits confirmation in a closed loop (Chin et al., 19 May 2024).

5. Prompt Management and System Design Strategies

Prompt management is central to robustness and flexibility. The system principle message, function API (described in JSON schema), and turn-taking policies are relayed as a single “system” message at session start. The manager parses every LLM output, separating “thoughts,” determinable actions, and user-directed speech, and executes prescribed hardware or interface modifications accordingly.

Engineered strategies include:

• Streaming and batching LLM responses for minimal end-to-end latency
• Explicit segmentation of internal (“thought”) and external (“speech”) agent outputs to avoid leakage of hidden state
• Evaluation and routing of function calls to the Music X Machine and robotic subsystems, guaranteeing rapid execution

This suggests that prompt and output mediation via an explicit manager layer enhances both agent transparency and system reliability, particularly in interactive, real-time educational environments.

6. Evaluation, Observed Outcomes, and Limitations

Flute X GPT has been demonstrated in three video trials: one fully scripted and two live improvisations with developer participants. Observations include:

• Seamless agent clarification of vague user intents (e.g., explicit dialogues on pedagogical priority)
• Generation of previously non-programmed workflow combinations (e.g., novel mixes of haptic and visual guidance)
• End-to-end system latency from speech to action of less than ~1 s, maintained via streaming
• Positive pilot feedback on agent persona (perceived as “considerate” and “multi-step ahead”)

No formal user paper or quantitative performance metrics (e.g., learning gain or retention) are yet available; future work is expected to address controlled evaluation of learning outcomes, reliability across expanded workflow spaces, long-term user modeling (potentially with retrieval-augmented memory), and generalization to polyphonic or alternative modalities (Chin et al., 19 May 2024).

7. Broader Implications and Future Directions

Flute X GPT demonstrates a transition from passive LLM agent execution to a secretary-level LAUI, capable of both system mastery and individualized user modeling. Implications include:

• Feasibility of LLM-mediated, hardware-in-the-loop adaptive instruction for complex sensorimotor skills
• Novel user experience models that relieve non-expert users from mastering the technical substrate
• Scalability challenges as the space of possible workflow combinations grows
• Need for robust trust, interpretability, and safety in agent pedagogical decisions

Broader generalization to other domains (music with polyphonic input, dance, handwriting) will require adaptation of the hardware stack and input event modeling. There is a recognized need for controlled, user-centered studies to validate learning efficacy, user satisfaction, and safety (Chin et al., 19 May 2024).