Opening the Design Space: Two Years of Performance with Intelligent Musical Instruments

Abstract: Machine generation of symbolic music and digital audio are hot topics but there have been relatively few digital musical instruments that integrate generative AI. Present musical AI tools are not artist centred and do not support experimentation or integrating into musical instruments or practices. This work introduces an inexpensive generative AI instrument platform based on a single board computer that connects via MIDI to other musical devices. The platform uses artist-collected datasets with models trained on a regular computer. This paper asks what the design space of intelligent musical instruments might look like when accessible and portable AI systems are available for artistic exploration. I contribute five examples of instruments created and tested through a two-year first-person artistic research process. These show that (re)mapping can replace retraining for discovering AI interaction, that fast input interleaving is a new co-creative strategy, that small-data AI models can be a transportable design resource, and that cheap hardware can lower barriers to inclusion. This work could enable artists to explore new interaction and performance schemes with intelligent musical instruments.

• The paper presents a novel low-cost generative AI platform using Raspberry Pi for real-time musical performance.
• It highlights how flexible MIDI mapping enables dynamic, co-creative interactions between human performers and AI.
• It demonstrates that artist-captured, small-data models can expand instrument design while addressing ethical and sustainability concerns.

Expanding the Design Space of Intelligent Musical Instruments with Embedded Generative AI

Introduction and Context

The paper "Opening the Design Space: Two Years of Performance with Intelligent Musical Instruments" (2604.23583) critically examines the practical integration of generative AI into digital musical instruments (DMIs) over a sustained two-year period of artistic research. Despite significant progress in symbolic music and audio generation via machine learning, real-world adoption of generative AI as a co-creative agent in live performance instruments remains rare. Existing tools often cater to studio production or data analysis, lacking seamless integration with the material practices and interfaces of performing musicians.

This work introduces a low-cost, battery-powered generative AI platform based on Raspberry Pi single-board computers, focusing on accessibility, portability, and use with artist-curated datasets. Five case-study instruments and a sequence of performance/recording experiences articulate how flexible mapping, co-creative strategies, and hardware constraints open new interaction and design possibilities.

Figure 1: The generative AI interactive music platform connects via MIDI to synths and controllers and requires minimal hardware resources.

System Architecture and Implementation

The foundation of the platform is a Python-based implementation of a mixture density recurrent neural network (MDRNN), optimized for efficient real-time symbolic gesture and control data generation. Artist-sourced performance data is used for initial training on conventional computers, after which the models are exported (to Keras or TensorFlow Lite formats) for inference on the Raspberry Pi device.

Key points of the architecture:

• Hardware Platform: Compatibility with any 64-bit Raspberry Pi, including the low-end Zero 2 W, enables affordable, portable deployments ($<15$ USD per unit), with robust MIDI communications via USB or serial UART.
• Software Stack: The system is provided as a bootable OS image, requiring minimal configuration via a web interface. This enables direct mapping of AI-generated outputs to arbitrary device controls and parameters, facilitating rapid prototyping and reconfiguration for different performance scenarios.
• Interfacing and Mapping: Flexible mapping is central to the system. The platform can interleave AI-generated MIDI control and note events with live human input, mapped to any hardware or DAW environment supporting the MIDI protocol.

Figure 2: Battery-powered performance setup—the system runs on a Raspberry Pi 4, supports configuration via web UI, and is fully portable.

Performance benchmarks demonstrate that even the most constrained Raspberry Pi boards achieve inference times $<5$ ms for models up to 128 LSTM units, well below the real-time threshold for musical interaction. The Raspberry Pi 5 further reduces boot and inference times, impacting stage readiness.

Figure 3: Model inference times demonstrate sub-5ms prediction across Raspberry Pi variants; even the minimal Pi Zero 2 W is suitable for real-time applications.

Case Studies in Intelligent Instrument Design

The paper presents an autoethnographic sequence of five evolving instrument prototypes, illustrating how mapping, interface design, and performance context interplay with AI capabilities.

Intelligent Volca: Proof of Concept

An initial experiment uses the system as a pitch/rhythm controller for the Korg Volca FM synthesizer, leveraging a simple MDRNN model trained on expressive controller data. While the AI's actions are one-way due to hardware MIDI limitations, the experiment validates the feasibility of a battery-powered embedded AI agent performing in real-world contexts.

Figure 4: A Raspberry Pi Zero 2 W autonomously drives a hardware synth’s pitch and rhythm, with the musician directly editing timbre.

Intelligent MicroFreak & S-1: Two-way Co-Creativity

Subsequent instruments take advantage of hardware synths capable of full-duplex MIDI over USB (e.g., Arturia MicroFreak, Roland S-1), allowing fast call-and-response interactions between human and AI. The AI is mapped to simultaneous control of multiple timbral parameters—extending the expressive palette beyond what a human can manipulate physically—while human performers focus on notes or particular controls.

Figure 5: Co-creative setups with hardware synths; the AI system and human each manipulate different facets (notes vs. timbre) in real time.

Intelligent DAW: Mapping over Re-invention

By connecting to an iPad or DAW host via MIDI, the generative AI system is decoupled from any fixed sound engine, leveraging software synths within professional production environments. Here, the focus shifts to rapid remapping of AI-generated signals without model retraining—demonstrating that flexible mappings are often sufficient to yield new and productive interaction schemes.

Figure 6: Software-based performance: the AI platform routes MIDI to multiple virtual instruments within a DAW, highlighting mapping flexibility.

Intelligent Setups: Multi-Controller Integration

The platform’s extended mapping supports ensemble arrangements with multiple controllers (e.g., Behringer X-Touch, Keith McMillen QuNeo) feeding both control and feedback through LEDs, touchpads, and synths concurrently. This opens up gestural affordances, enables nuanced feedback and agency-tracking, and supports workflow migration across synth and controller types—demonstrated in varied festival performances.

Figure 7: Final-stage setup leveraging Roland S-1, QuNeo, and Raspberry Pi 5, deployed in professional festival environments, showing advanced mapping and feedback strategies.

Design Implications and Theoretical Impact

Flexible Mapping over Retraining

The most substantive empirical finding is that expanding the platform’s mapping capabilities often supersedes the need for retraining. Remapping AI control signals to different instrument parameters, visual indicators, or interface modalities yields new affordances with minimal time and energy investment, avoiding the resource-intensive retraining cycles typical of larger AI models [fiebrink_years_ml; stefansdottir_intelligent_2025]. This approach is both environmentally sustainable and musically adaptive.

Transportable, Artist-Captured AI Models

Small-data, artist-trained generative models, as opposed to large, centrally-trained industrial models, emerge as reusable design materials—similar to hardware modules or effects pedals—capable of being deployed and remapped across a suite of instruments. This local approach directly addresses ongoing ethical concerns regarding AI model provenance and sustainability [Vigliensoni:2022; ethical_genaudio_2023; sustainable-internet-of-musical-things].

Fast Human-AI Interleaving

Very rapid interleaving of human and machine control, facilitated by continuous monitoring and low-latency inference, enables new co-creative performance paradigms beyond classic call-and-response or agent-based interaction. The AI acts as an autonomous, dynamically configurable parametric process, with the human performer able to seize or yield control instantaneously via their own gestural inputs.

Accessibility, Inclusion, and Sustainability

The affordability and open-access nature of the platform lower barriers to both instrument prototyping and performance, supporting individual, collaborative, and educational deployment at scale. MIDI’s continued ubiquity ensures cross-generational interface compatibility, while the open-source release maximizes transferability.

Practical and Theoretical Implications

From a design research and HCI perspective, the paper substantiates the claim that embodied, artist-centric practices with small-data AI both expand the configuration space for intelligent musical instruments and instantiate new types of musical agency [stefansdottir_intelligent_2025; taxonomy-music-interactions-parkinson-2025]. It advances a pragmatic methodology—prioritizing mapping, modularity, and sustainability—that resonates with ongoing discourses in creativity support tools and co-creative AI in the arts [cst-landscape-in-hci-2019; design-principles-genai-2024].

Potential future research includes systematic multi-user studies, longitudinal model updating workflows, and further exploration of agency and control dynamics in ensemble contexts.

Conclusion

This work demonstrates that a generative AI platform, built on low-cost embedded hardware and emphasizing flexible input/output mapping, can substantially expand the design and performance space of digital musical instruments. Small-data, artist-owned models not only sidestep ethical and sustainability challenges of large-scale AI but also reveal rich avenues for co-creative practice and rapid prototyping. The findings encourage an ongoing shift toward artist-driven, inclusive, and sustainable intelligent instrument design within the computer music and HCI communities.