TofuML: Embodied Interactive ML

• TofuML is a spatio-physical interactive ML system that uses a toy-like device and spatial paper mat to facilitate sound classification for novices.
• It maps the core ML loop—data collection, training, inference, and iteration—into a physical workflow that enhances engagement and conceptual clarity.
• The system employs a lightweight CNN feature extractor and Random Forest to quickly bridge sound inputs to physical actions.

Searching arXiv for the specified TofuML paper and related context. TofuML is a spatio-physical interactive machine learning (IML) system designed to support the interactive exploration of machine learning for novices through a tangible, toy-like interface rather than a conventional GUI (Kawabe et al., 1 Aug 2025). It centers on sound classification and maps the core ML loop—data collection, model training, inference, and iteration—onto a physical workflow involving a small device and a paper mat. In this design, users teach the system by associating sounds with actions, then observe predictions as physical motion and effects. The system is intended for non-experts and general users, especially those with little or no ML background, and it investigates whether physical embodiment and spatial interaction can lower barriers to ML participation while increasing engagement, curiosity, and creative ideation (Kawabe et al., 1 Aug 2025).

1. Definition and design rationale

TofuML is presented as a physical interactive ML device composed of a small cube-like handheld device, a paper mat with spatially arranged action zones, and a workflow for collecting training data, training a model, and running inference/evaluation (Kawabe et al., 1 Aug 2025). The system’s central proposition is that ML can be experienced as an embodied interaction rather than as a screen-centered procedure. Instead of selecting labels on a display and inspecting probabilities only on a screen, users place the device on specific regions of the mat to define labels, record sound samples, and watch the device execute predictions physically.

The motivating critique is directed at conventional GUI-based IML systems. Such systems may simplify data collection, but they can still leave users feeling conceptually distant from the model. TofuML addresses this by emphasizing a toy-like, approachable, and nurturing interaction style in which users feel they are “teaching” the device. The focus is therefore not solely on efficiency, but on whether a physical and spatial interface can make ML feel less like an opaque technical artifact and more like an accessible object-based activity (Kawabe et al., 1 Aug 2025).

The choice of sound classification is integral to this rationale. Sound production is described as easy and playful, and classification is one of the most fundamental ML tasks. This suggests that TofuML is intended as an entry point into the supervised learning loop rather than as a general-purpose ML engineering environment. The paper is explicit that the system intentionally omits advanced steps such as hyperparameter tuning, architecture design, and data augmentation, thereby foregrounding the core interaction cycle over full-stack model development (Kawabe et al., 1 Aug 2025).

2. Spatio-physical interface and embodied interaction

The device is a small cuboid, toy-like physical object built from M5Stack Core2 as the main controller, a toio module for wheel-based motion, and a custom base part containing physical buttons, speakers/LRA, microphone, and connectivity components (Kawabe et al., 1 Aug 2025). Its output channels include wheel movement, body vibration, LED lights, and an on-device screen showing labels/probabilities. A defining feature of the system is that model output is not merely displayed; it is performed.

The paper mat serves as a printed spatial interface compatible with toio’s optical sensing of an invisible pattern. Different regions correspond to six action labels: shake, go forward, light up, turn left, go backward, and turn right (Kawabe et al., 1 Aug 2025). By placing the device on a particular zone, a user selects the current label for the next training example. The mat therefore functions as a physical label selector, converting label assignment into a spatial gesture rather than a menu action.

This interface structure gives TofuML a characteristic “spatio-physical” grammar. Label selection is embodied as placement in space; inference is embodied as motion, vibration, or light; and the model’s behavior is experienced as a coupling between sound input and observable action. The paper’s interpretation is that such coupling supports emotional engagement and a stronger sense of training interaction than GUI-only alternatives (Kawabe et al., 1 Aug 2025). A plausible implication is that the system’s pedagogical value depends not only on whether predictions are correct, but also on how concretely users can perceive the relationship between labeled examples and enacted outputs.

3. Interactive machine learning workflow

TofuML implements a simple iterative IML loop. In training mode, the user first chooses an action by moving the device to the appropriate area of the paper mat. The device then shows the action name and a face/emoticon on its screen. The user presses a record button and produces a 1-second sound sample, which is stored as training data for the selected action (Kawabe et al., 1 Aug 2025). After collecting examples for multiple actions, the user presses a physical button to train the model on a local server. Once training is complete, the system switches to inference mode.

In inference, the device records audio for 1 second every 2.5 seconds, classifies each recording, and executes the action with the highest probability; the screen also shows the probability distribution for debugging or transparency (Kawabe et al., 1 Aug 2025). The reported end-to-end delay from recording to output action is about 3 seconds. If performance is unsatisfactory, users can return to training mode, add more data, retrain, delete the latest sample, or reset all samples. The workflow is thus explicitly iterative and supports trial-and-error interaction rather than a single train-once deployment pattern.

The paper summarizes the process formally by defining an action set $A = \{a_1, a_2, \dots, a_k\}$ and a training set $D = \{(x_i, a_i)\}$, where each sound clip $x_i$ is associated with an action label $a_i$ (Kawabe et al., 1 Aug 2025). Data collection expands the dataset by

D \leftarrow D \cup \{(x_i, a_j)\}

and training yields a classifier $D = \{(x_i, a_i)\}$0 while inference selects $D = \{(x_i, a_i)\}$1 This operationalization captures the central loop of the system: label by spatial placement, record sound, retrain, infer, act (Kawabe et al., 1 Aug 2025).

4. Technical architecture and model pipeline

The system is split between device-side interaction and server-side ML. On the device side, TofuML captures audio, handles user interface operations, drives movement, and renders outputs. On the server side, it performs ML training and inference. Communication between device and server occurs over Wi‑Fi, while the toio module communicates with the M5Stack Core2 over Bluetooth (Kawabe et al., 1 Aug 2025). This architecture allows the physical device to remain compact while offloading the computationally heavier components of the learning pipeline.

The classifier follows a lightweight two-stage structure. First, the system applies feature extraction using a convolutional neural network pre-trained on AudioSet, following TensorFlow’s official sample implementation. Second, a Random Forest is trained on the extracted features (Kawabe et al., 1 Aug 2025). The paper states that this design was selected because it is computationally lightweight and trains quickly on CPU, which reduces waiting time during interactive use.

The pipeline is summarized in the paper as $D = \{(x_i, a_i)\}$2 where $x$ is the recorded sound, $f(x)$ is the learned audio embedding or feature vector, and $\hat{y}$ is the predicted action label (Kawabe et al., 1 Aug 2025). The system therefore does not train an end-to-end audio network during interaction; rather, it reuses a pretrained representation and learns a lightweight downstream classifier. This suggests a design balance between responsiveness and conceptual simplicity: users interact with a complete trainable system, but latency is constrained by a pipeline chosen for quick CPU-side retraining.

5. Empirical studies and observed effects

The paper reports two empirical studies. The first is a comparative study against a GUI-based version intended to isolate interface effects while keeping the classification model and timing the same (Kawabe et al., 1 Aug 2025). The GUI version reproduced the core functionality as a web app with six action zones as on-screen buttons, recording controls, a train button, inference with probability display, and a return-to-recording button. The study involved 19 participants, aged 19–53; 12 had no AI/ML experience, while others ranged from studying ML to developing ML. Each participant used both systems for 10–15 minutes, with order counterbalanced, and then completed a questionnaire and interview (Kawabe et al., 1 Aug 2025).

The questionnaire employed eight custom pairwise-comparison items, including overall enjoyment, attractive design, enjoyment of recording, pleasure in observing output, enjoyment of trial-and-error, sense of achievement when model works, contribution to learning about AI/ML, and ease of use, scored from $-2$ for preferring the GUI to $+2$ for preferring TofuML (Kawabe et al., 1 Aug 2025). TofuML was preferred on nearly all dimensions, with the strongest preferences for observing output and sense of achievement, whereas ease of use slightly favored the GUI. Qualitatively, participants described TofuML as cuter, more approachable, and more like nurturing a living thing, while the GUI was viewed as efficient and easy to understand, but also impersonal and less exciting (Kawabe et al., 1 Aug 2025).

The second study was a public event deployment conducted at a science museum in a glass-walled room, intended to recruit a more diverse and less self-selected population (Kawabe et al., 1 Aug 2025). The format was hackathon-style and open-ended: participants could enter and leave freely, pursue their own goals, and choose their own sound-to-action mappings, with staff only providing technical support. The study involved 41 participants, aged 2–53; 24 had no prior ML experience; and the average experience time was 1096 seconds (Kawabe et al., 1 Aug 2025). Participants received the device, paper mat, worksheet, and sound-making objects, and the study used pre- and post-experience questionnaires, worksheet analysis, and optional semi-structured interviews.

The reported findings from the public deployment concern engagement, technical understanding, and ideation. Participants reported high engagement and found the toy-like feel, cute screen expressions, simplicity, and direct action-behavior linkage compelling. They also learned that data quantity matters, data quality matters, similar-sounding samples may not work well, clean and reproducible sounds improve accuracy, and machine learning can work with only a few examples in some cases (Kawabe et al., 1 Aug 2025). In ideation, participants proposed a wide range of imagined AI applications, including refrigerator content scanning for recipes and expiration dates, detecting vegetable freshness, allergen warning systems, voice-based control for non-verbal individuals, emotion recognition from voice, robots that greet people, cleaning robots, caregiving and school-support tools for children, and vibration-based music access for hearing-impaired users (Kawabe et al., 1 Aug 2025). The study therefore frames TofuML not only as an educational interface but also as a scaffold for thinking about personally meaningful AI uses.

6. Interpretation, limitations, and research significance

The paper characterizes TofuML as a system that increases engagement through physical embodiment, toy-like appearance, motion and feedback, a nurturing/teaching metaphor, and visible success or failure (Kawabe et al., 1 Aug 2025). At the same time, it identifies a trade-off between engagement and other desiderata. The physical workflow can be less convenient than a GUI; users may dislike moving the device for each recording, worry about bumping or dropping it, or feel frustration when the toy “fails” visibly. This duality is central to the paper’s interpretation: the very inconvenience of physical interaction may increase attachment and investment, but it can also reduce usability (Kawabe et al., 1 Aug 2025).

A further theme concerns data collection quality and explainability. The system encourages active, playful collection of labeled data, yet users sometimes choose acoustically similar sounds for different labels, which can hurt model performance. The paper also notes that the interface can leave the classifier somewhat of a black box; participants wanted more transparency, and the authors discuss the possible addition of explanation tools such as Grad-CAM and LIME, while warning that these might increase cognitive load and reduce approachability (Kawabe et al., 1 Aug 2025). This suggests that novice-oriented IML design involves a nontrivial optimization problem between conceptual understanding and experiential accessibility.

The stated limitations are explicit. TofuML is restricted to simple classification, mainly sound classification; it offers a simplified ML workflow without hyperparameter tuning, architecture design, or data augmentation; and it does not support standardized task-level performance comparison because participants defined their own tasks in the public deployment (Kawabe et al., 1 Aug 2025). The GUI comparison is acknowledged as imperfect because mapping physical interaction to GUI form is inherently arbitrary, and the public study does not directly compare against the GUI, so some effects of the physical interface are not fully isolated from the event context. The paper also notes hardware constraints, including issues related to vibration noticeability, button placement, durability, and movement safety (Kawabe et al., 1 Aug 2025).

Within interactive ML research, TofuML’s significance lies in its proposition that novice-facing systems may need to optimize not only for efficiency or transparency, but also for approachability, curiosity, and emotional connection (Kawabe et al., 1 Aug 2025). The paper concludes that physical UI and GUI serve different roles: the former is better for sparking initial interest and emotional engagement, while the latter may be preferable for efficiency once users become more technically confident. This suggests a staged design model for interactive ML education and prototyping in which embodied systems function as introductory interfaces and more conventional tools support later refinement.