MUSTEM: A Dual-Modality System for Vibrotactile and Visual Translation of Music as an Assistive Technology (2511.12045v1)

Abstract: The emotional and structural experience of music remains a significant accessibility challenge for the deaf and hard of hearing community. This paper introduces MUSTEM (Multisensorial Emotional Translation), a novel system designed to translate music into a rich, coherent, and scientifically-grounded sensory experience. We present a dual-modality approach addressing this challenge through two interconnected components. First, a low-cost, portable hardware prototype that performs real-time audio analysis, mapping distinct frequency bands (sub-bass, bass, mid-range, treble) to a four-channel vibrotactile system, allowing users to feel the music's rhythmic and foundational structure. Second, to overcome the processing limitations of embedded hardware, we developed a high-fidelity software simulation that demonstrates the full potential of the visual translation. This assistive dashboard decodes musical components - such as rhythm, harmony, and frequency spectrum - into an intuitive and educational visual interface. MUSTEM offers a comprehensive framework for sensory substitution, presenting a viable and accessible pathway for the deaf community to experience music not just as vibration, but as a structured, substantiated and emotionally resonant visual and tactile language. Preliminary feedback from seven deaf users suggests the system's spatial vibrotactile mapping is perceptible and engaging. All source code and hardware designs are released as open-source. Video demonstrations and open-source code are available on the project's official channel.