---
title: 'TofuML: Embodied Interactive ML'
url: https://www.emergentmind.com/topics/tofuml
type: topic
---

# TofuML: Embodied Interactive ML

Searching arXiv for the specified TofuML paper and related context.
TofuML is a **spatio-physical interactive machine learning (IML) system** designed to support the **interactive exploration of machine learning for novices** through a tangible, toy-like interface rather than a conventional GUI [2508.00252]. It centers on **sound classification** and maps the core ML loop—data collection, model training, inference, and iteration—onto a physical workflow involving a small device and a paper mat. In this design, users teach the system by associating sounds with actions, then observe predictions as **physical motion and effects**. The system is intended for **non-experts** and **general users**, especially those with little or no ML background, and it investigates whether **physical embodiment and spatial interaction** can lower barriers to ML participation while increasing engagement, curiosity, and creative ideation [2508.00252].

## 1. Definition and design rationale

TofuML is presented as a **physical interactive ML device** composed of a **small cube-like handheld device**, a **paper mat** with spatially arranged action zones, and a workflow for **collecting training data**, **training a model**, and **running inference/evaluation** [2508.00252]. The system’s central proposition is that ML can be experienced as an embodied interaction rather than as a screen-centered procedure. Instead of selecting labels on a display and inspecting probabilities only on a screen, users place the device on specific regions of the mat to define labels, record sound samples, and watch the device execute predictions physically.

The motivating critique is directed at conventional GUI-based IML systems. Such systems may simplify data collection, but they can still leave users feeling conceptually distant from the model. TofuML addresses this by emphasizing a **toy-like**, **approachable**, and **nurturing** interaction style in which users feel they are “teaching” the device. The focus is therefore not solely on efficiency, but on whether a physical and spatial interface can make ML feel less like an opaque technical artifact and more like an accessible object-based activity [2508.00252].

The choice of **sound classification** is integral to this rationale. Sound production is described as easy and playful, and classification is one of the most fundamental ML tasks. This suggests that TofuML is intended as an entry point into the supervised learning loop rather than as a general-purpose ML engineering environment. The paper is explicit that the system intentionally omits advanced steps such as **hyperparameter tuning**, **architecture design**, and **data augmentation**, thereby foregrounding the core interaction cycle over full-stack model development [2508.00252].

## 2. Spatio-physical interface and embodied interaction

The device is a **small cuboid, toy-like physical object** built from **M5Stack Core2** as the main controller, a **toio** module for wheel-based motion, and a custom base part containing **physical buttons**, **speakers/LRA**, **microphone**, and connectivity components [2508.00252]. Its output channels include **wheel movement**, **body vibration**, **LED lights**, and an on-device screen showing **labels/probabilities**. A defining feature of the system is that model output is not merely displayed; it is **performed**.

The **paper mat** serves as a printed spatial interface compatible with toio’s optical sensing of an invisible pattern. Different regions correspond to six action labels: **shake**, **go forward**, **light up**, **turn left**, **go backward**, and **turn right** [2508.00252]. By placing the device on a particular zone, a user selects the current label for the next training example. The mat therefore functions as a **physical label selector**, converting label assignment into a spatial gesture rather than a menu action.

This interface structure gives TofuML a characteristic “spatio-physical” grammar. Label selection is embodied as placement in space; inference is embodied as motion, vibration, or light; and the model’s behavior is experienced as a coupling between sound input and observable action. The paper’s interpretation is that such coupling supports emotional engagement and a stronger sense of training interaction than GUI-only alternatives [2508.00252]. A plausible implication is that the system’s pedagogical value depends not only on whether predictions are correct, but also on how concretely users can perceive the relationship between labeled examples and enacted outputs.

## 3. Interactive machine learning workflow

TofuML implements a simple iterative IML loop. In **training mode**, the user first chooses an action by moving the device to the appropriate area of the paper mat. The device then shows the action name and a face/emoticon on its screen. The user presses a **record button** and produces a **1-second sound sample**, which is stored as training data for the selected action [2508.00252]. After collecting examples for multiple actions, the user presses a physical button to train the model on a **local server**. Once training is complete, the system switches to **inference mode**.

In inference, the device records audio for **1 second every 2.5 seconds**, classifies each recording, and executes the action with the highest probability; the screen also shows the probability distribution for debugging or transparency [2508.00252]. The reported end-to-end delay from recording to output action is about **3 seconds**. If performance is unsatisfactory, users can return to training mode, add more data, retrain, delete the latest sample, or reset all samples. The workflow is thus explicitly iterative and supports trial-and-error interaction rather than a single train-once deployment pattern.

The paper summarizes the process formally by defining an action set \(A = \{a_1, a_2, \dots, a_k\}\) and a training set \(D = \{(x_i, a_i)\}\), where each sound clip \(x_i\) is associated with an action label \(a_i\) [2508.00252]. Data collection expands the dataset by
```latex
D \leftarrow D \cup \{(x_i, a_j)\}
```
and training yields a classifier
```latex
\hat{m} = \text{Train}(D)
```
while inference selects
```latex
\hat{y} = \arg\max_{a \in A} p(a \mid x).
```
This operationalization captures the central loop of the system: **label by spatial placement, record sound, retrain, infer, act** [2508.00252].

## 4. Technical architecture and model pipeline

The system is split between device-side interaction and server-side ML. On the device side, TofuML captures audio, handles user interface operations, drives movement, and renders outputs. On the server side, it performs ML **training and inference**. Communication between device and server occurs over **Wi‑Fi**, while the toio module communicates with the M5Stack Core2 over **Bluetooth** [2508.00252]. This architecture allows the physical device to remain compact while offloading the computationally heavier components of the learning pipeline.

The classifier follows a lightweight two-stage structure. First, the system applies **feature extraction** using a convolutional neural network **pre-trained on AudioSet**, following TensorFlow’s official sample implementation. Second, a **Random Forest** is trained on the extracted features [2508.00252]. The paper states that this design was selected because it is computationally lightweight and trains quickly on CPU, which reduces waiting time during interactive use.

The pipeline is summarized in the paper as
```latex
x \xrightarrow{\text{Audio feature extractor}} f(x) \xrightarrow{\text{Random Forest}} \hat{y}
```
where \(x\) is the recorded sound, \(f(x)\) is the learned audio embedding or feature vector, and \(\hat{y}\) is the predicted action label [2508.00252]. The system therefore does not train an end-to-end audio network during interaction; rather, it reuses a pretrained representation and learns a lightweight downstream classifier. This suggests a design balance between responsiveness and conceptual simplicity: users interact with a complete trainable system, but latency is constrained by a pipeline chosen for quick CPU-side retraining.

## 5. Empirical studies and observed effects

The paper reports two empirical studies. The first is a **comparative study against a GUI-based version** intended to isolate interface effects while keeping the classification model and timing the same [2508.00252]. The GUI version reproduced the core functionality as a web app with six action zones as on-screen buttons, recording controls, a train button, inference with probability display, and a return-to-recording button. The study involved **19 participants**, aged **19–53**; **12 had no AI/ML experience**, while others ranged from studying ML to developing ML. Each participant used both systems for **10–15 minutes**, with order counterbalanced, and then completed a questionnaire and interview [2508.00252].

The questionnaire employed eight custom pairwise-comparison items, including **overall enjoyment**, **attractive design**, **enjoyment of recording**, **pleasure in observing output**, **enjoyment of trial-and-error**, **sense of achievement when model works**, **contribution to learning about AI/ML**, and **ease of use**, scored from \(-2\) for preferring the GUI to \(+2\) for preferring TofuML [2508.00252]. TofuML was preferred on nearly all dimensions, with the strongest preferences for **observing output** and **sense of achievement**, whereas **ease of use** slightly favored the GUI. Qualitatively, participants described TofuML as **cuter**, **more approachable**, and **more like nurturing a living thing**, while the GUI was viewed as **efficient** and **easy to understand**, but also **impersonal** and **less exciting** [2508.00252].

The second study was a **public event deployment** conducted at a **science museum** in a **glass-walled room**, intended to recruit a more diverse and less self-selected population [2508.00252]. The format was **hackathon-style** and open-ended: participants could enter and leave freely, pursue their own goals, and choose their own sound-to-action mappings, with staff only providing technical support. The study involved **41 participants**, aged **2–53**; **24 had no prior ML experience**; and the average experience time was **1096 seconds** [2508.00252]. Participants received the device, paper mat, worksheet, and sound-making objects, and the study used pre- and post-experience questionnaires, worksheet analysis, and optional semi-structured interviews.

The reported findings from the public deployment concern **engagement**, **technical understanding**, and **ideation**. Participants reported high engagement and found the toy-like feel, cute screen expressions, simplicity, and direct action-behavior linkage compelling. They also learned that **data quantity matters**, **data quality matters**, **similar-sounding samples may not work well**, **clean and reproducible sounds improve accuracy**, and **machine learning can work with only a few examples in some cases** [2508.00252]. In ideation, participants proposed a wide range of imagined AI applications, including **refrigerator content scanning for recipes and expiration dates**, **detecting vegetable freshness**, **allergen warning systems**, **voice-based control for non-verbal individuals**, **emotion recognition from voice**, **robots that greet people**, **cleaning robots**, **caregiving and school-support tools for children**, and **vibration-based music access for hearing-impaired users** [2508.00252]. The study therefore frames TofuML not only as an educational interface but also as a scaffold for thinking about personally meaningful AI uses.

## 6. Interpretation, limitations, and research significance

The paper characterizes TofuML as a system that increases **engagement** through **physical embodiment**, **toy-like appearance**, **motion and feedback**, a **nurturing/teaching metaphor**, and visible success or failure [2508.00252]. At the same time, it identifies a trade-off between engagement and other desiderata. The physical workflow can be less convenient than a GUI; users may dislike moving the device for each recording, worry about bumping or dropping it, or feel frustration when the toy “fails” visibly. This duality is central to the paper’s interpretation: the very inconvenience of physical interaction may increase attachment and investment, but it can also reduce usability [2508.00252].

A further theme concerns **data collection quality** and **explainability**. The system encourages active, playful collection of labeled data, yet users sometimes choose acoustically similar sounds for different labels, which can hurt model performance. The paper also notes that the interface can leave the classifier somewhat of a black box; participants wanted more transparency, and the authors discuss the possible addition of explanation tools such as **Grad-CAM** and **LIME**, while warning that these might increase cognitive load and reduce approachability [2508.00252]. This suggests that novice-oriented IML design involves a nontrivial optimization problem between conceptual understanding and experiential accessibility.

The stated limitations are explicit. TofuML is restricted to **simple classification**, mainly **sound classification**; it offers a **simplified ML workflow** without hyperparameter tuning, architecture design, or data augmentation; and it does not support standardized task-level performance comparison because participants defined their own tasks in the public deployment [2508.00252]. The GUI comparison is acknowledged as imperfect because mapping physical interaction to GUI form is inherently arbitrary, and the public study does not directly compare against the GUI, so some effects of the physical interface are not fully isolated from the event context. The paper also notes **hardware constraints**, including issues related to vibration noticeability, button placement, durability, and movement safety [2508.00252].

Within interactive ML research, TofuML’s significance lies in its proposition that novice-facing systems may need to optimize not only for efficiency or transparency, but also for **approachability**, **curiosity**, and **emotional connection** [2508.00252]. The paper concludes that **physical UI** and **GUI** serve different roles: the former is better for sparking initial interest and emotional engagement, while the latter may be preferable for efficiency once users become more technically confident. This suggests a staged design model for interactive ML education and prototyping in which embodied systems function as introductory interfaces and more conventional tools support later refinement.

Source: https://www.emergentmind.com/topics/tofuml