Iola Walker: A Mobile Footfall Detection System for Music Composition

Abstract: This project is the first of several experiments composing music that changes in response to biosignals. The system is dubbed "iola walker" in reference to a common polyrhythm, the hemiola. A listener goes for a walk, and the Iola Walker app detects their walking pace. Iola Walker picks up footfalls using a foot-mounted accelerometer, processing the signals in real time using a recurrent neural network in an Android app. The Android app outputs a MIDI event for each footfall. The iola walker player, which might be a VST running in a DAW, plays the version of the next music passage with underlying polyrhythms closest to the listener's walking pace. This paper documents the process of training the model to detect the footfalls in real time. The model is trained on accelerometer data from an Mbient Labs foot-mounted IMU at 200~Hz, with the ground truth for footfalls annotated by pressing the volume-up button on the Android device when the foot hits the ground. To collect training data, I walked around my neighborhood clicking the volume-up button each time my foot hit the ground. Several methods were tried for detecting footfalls in real time from sensor data, including ones based on digital signal processing techniques and traditional machine learning techniques.