DeformTune: Embodied AI Music Generation

• DeformTune is a tactile, embodied interface that maps four pressure sensor inputs to a latent space in MeasureVAE, enabling real-time AI music generation.
• It employs a discretized latent code of 10^4 settings, with each sensor controlling distinct musical attributes like rhythmic complexity and note range.
• User studies show that while the system fosters creative exploration, improved explainability through multimodal feedback is crucial for novice users.

DeformTune is a prototype system aimed at enabling intuitive, embodied, and explainable interaction with AI-driven music generation, especially for users without formal musical training. The system integrates a tactile, deformable hardware interface with the latent code–controlled MeasureVAE model, allowing real-time manipulations of musical features through physical gestures. It is positioned as a response to the complexity and lack of transparency in conventional text-prompt or instrument-style AI music interfaces, emphasizing accessibility and explainability for novice users (Xu et al., 31 Jul 2025).

1. System Architecture and Interaction Paradigm

DeformTune comprises two principal components: a multi-sensor deformable interface and a discrete, controllable generative music model. The interface consists of four pressure sensors constructed from conductive fabric, Velostat, copper tape, and foam, all connected via an Arduino. Each sensor captures a continuous pressure value $p \in [0, 1]$, which is smoothed and quantized to integer levels in the range 1–10 using the mapping $x = \lfloor 9p + 1 \rfloor$.

These four sensor outputs collectively define a 4-dimensional control vector that directly indexes the latent space of the MeasureVAE model, which is discretized into $10^4$ possible settings. Each unique latent vector is pre-correlated with a generated MIDI phrase, enabling deterministic, real-time selection and playback of musical materials based on tactile input alone. This physical-to-latent coupling provides an immediate, embodied mapping from user deformation gestures to musical semantics, bypassing the need for symbolic commands or parameter fiddling.

2. Underlying Generative Model and Latent Space Structuring

MeasureVAE, the backbone generative engine, is a variational autoencoder (VAE) architecture whose latent space is regularized and axis-aligned so that each dimension corresponds to an interpretable musical attribute. In DeformTune, these are:

• Rhythmic complexity
• Note range
• Note density
• Average interval jump

During generation, the selected latent code $z \in \mathbb{Z}_{10}^4$ is used to deterministically select from a set of pre-generated samples, guaranteeing a one-to-one mapping from multi-sensor state to musical output.

This internal structure ensures that each sensor on the tactile interface controls a distinctly perceivable musical property. For example, increased pressure on the sensor mapped to “rhythmic complexity” will continually shift generated music toward sparser or denser patterns, while the “note range” sensor will extend or contract melodic pitch contours.

3. User Study: Usability and Explainability Outcomes

A pilot user paper was conducted with eleven adult non-musician participants. Participants explored the interface freely and then performed a creative generation task (composing a 10-second ringtone fragment), followed by completion of the UEQ-short and RiCE usability questionnaires and a semi-structured interview.

Thematic analysis identified several recurring issues:

• Users often found the mapping between pressure and musical result ambiguous, noting that although the device was responsive, the causal structure ("which axis does what?") was rarely obvious.
• Several participants struggled to reproduce or deliberately traverse to a previously-experienced musical result.
• Some confusion arose from the physical interface affordances; for example, the use of conductive fabric led to incorrect assumptions about gesture modalities (e.g., expecting sliding rather than pressing).

While users characterized the system as playful and appreciated the direct physical engagement, perceived pragmatic usability lagged behind creative expressivity. These findings indicate that transparency of the mapping (i.e., explainability in the sense of traceable, predictable causality) is essential for empowering creative novices.

4. Explainability Strategies and Interaction Design Opportunities

The paper proposes concrete strategies to enhance transparency and foster more explainable interaction:

• Multimodal Feedback: Overlaying visual indicators, LED cues, or on-screen graphics to show current sensor values and corresponding latent-space movement. This can clarify the current mapping and help users form reliable action–effect models.
• Progressive (Layered) Interaction Support: Introducing guidance that scaffolds users through system features. For example, context-sensitive haptic or auditory “micro-feedback” can assist during exploration, while more detailed explanations (of each sensor’s effect) are accessible post hoc.
• Guidance and Onboarding: Providing animated or interactive tutorials to help users internalize how latent dimensions correspond to musical attributes and encouraging systematic exploration of the action space.

Collectively, these approaches are aimed at transitioning the system from a “black box” to a more interpretable “gray box,” where users retain creative agency but can also cultivate a functional, perceptual understanding of the generative process.

5. Limitations and Expressive Constraints

Several limitations are inherent in the current DeformTune design:

• The mapping from analog sensor input to latent code is quantized; expressive range is limited by the granularity of latent discretization and pre-generated output (here, $10^4$ possible samples).
• The current hardware form factor supports only pressure-press gestures; these are mapped to four independent features, precluding more complex, hierarchically organized or continuous controls.
• There remains a gap between low-level sensor data and high-level musical intent: users can sense some degree of control, but crafting nuanced, repeatable outputs or thoroughly understanding latent–music relations remains nontrivial.
• The need to pre-generate and correlate MIDI files to each latent combination restricts the spontaneous exploration of the full generative model capacity.

6. Future Directions

Building on these findings, several promising developmental pathways are articulated:

• Enhanced Real-time Responsiveness: Optimizing the computation and mapping pipeline to ensure minimized latency and stable, predictable parameter response.
• Richer Feedback Modalities: Integrating dynamic visualizations, richer sound cues, or haptic feedback to provide immediate informational scaffolding during gesture.
• Expanded Physical Gestures: Incorporating gesture types beyond pressing—such as sliding, rotation, or multi-sensor combinations—to extend control over additional musical parameters (e.g., timbre or harmony).
• Layered Onboarding Experiences: Implementing guided exploration modes that introduce users gradually to each underlying sensor–latent mapping, facilitating the construction of robust mental models without overwhelming cognitive load.
• Adaptive Explainability: Supporting both functional (“why did this fragment sound like that?”) and technical explanations, potentially with interactive explanations or query-based exploration of the system’s generative inner workings.

7. Significance and Context within Explainable AI Music

DeformTune exemplifies a shift toward physically grounded, accessible interfaces for AI-mediated creative tasks, moving beyond text or code-based paradigms. By mapping hardware sensor states to semantically meaningful latent dimensions in a generative model, and foregrounding multimodal explainability strategies, the system addresses persistent barriers faced by non-expert users. The user paper surfaces important challenges relating to action–effect ambiguity and the necessity of structured onboarding, contributing empirical evidence on XAI system affordances for creative novices. The findings and proposed design directions lay groundwork for future research in embodied, explainable AI-assisted creation targeted at broad, non-technical user populations (Xu et al., 31 Jul 2025).