A Cross-Modal Approach to Silent Speech with LLM-Enhanced Recognition (2403.05583v1)

Abstract: Silent Speech Interfaces (SSIs) offer a noninvasive alternative to brain-computer interfaces for soundless verbal communication. We introduce Multimodal Orofacial Neural Audio (MONA), a system that leverages cross-modal alignment through novel loss functions--cross-contrast (crossCon) and supervised temporal contrast (supTcon)--to train a multimodal model with a shared latent representation. This architecture enables the use of audio-only datasets like LibriSpeech to improve silent speech recognition. Additionally, our introduction of LLM Integrated Scoring Adjustment (LISA) significantly improves recognition accuracy. Together, MONA LISA reduces the state-of-the-art word error rate (WER) from 28.8% to 12.2% in the Gaddy (2020) benchmark dataset for silent speech on an open vocabulary. For vocal EMG recordings, our method improves the state-of-the-art from 23.3% to 3.7% WER. In the Brain-to-Text 2024 competition, LISA performs best, improving the top WER from 9.8% to 8.9%. To the best of our knowledge, this work represents the first instance where noninvasive silent speech recognition on an open vocabulary has cleared the threshold of 15% WER, demonstrating that SSIs can be a viable alternative to automatic speech recognition (ASR). Our work not only narrows the performance gap between silent and vocalized speech but also opens new possibilities in human-computer interaction, demonstrating the potential of cross-modal approaches in noisy and data-limited regimes.

• The paper introduces MONA, a novel system aligning audio and EMG data using cross-contrast and supervised temporal contrast losses with dynamic time warping.
• It implements LLM-integrated scoring (LISA) to refine candidate sentences, significantly boosting recognition accuracy.
• Empirical results show a word error rate of 12.2% for silent speech and improvement from 23.3% to 3.7% for vocalized EMG recordings.

Enhanced Silent Speech Recognition through Cross-Modal Learning and LLM-Enhanced Scoring

Introduction

Silent Speech Interfaces (SSI) hold transformative potential for communication technologies, particularly for individuals with speech impairments or in situations where vocal communication is not possible. Despite the promise of SSIs, significant challenges have impeded their development, notably the absence of phonetic content and limited datasets for effective training. A paper presents an innovative approach that leverages cross-modal learning and the integration of LLMs to address these challenges, demonstrating significant improvements in silent speech recognition accuracy.

Background

The development of SSIs has seen various technological approaches, each with its unique advantages and limitations. Among these, lip reading and surface electromyography (EMG) have emerged as promising techniques for silent speech decoding. Unlike auditory methods, EMG captures muscle activity associated with speech articulation, offering a potential advantage in speech recognition tasks.

Prior research in the field of Automatic Speech Recognition (ASR) has achieved considerable success, largely benefiting from advanced algorithms, neural network architectures, and expansive training datasets. However, transferring these advances to SSIs has been constrained by the unique challenges silent speech presents.

Proposed Approach

The paper introduces a novel system termed Multimodal Orofacial Neural Audio (MONA) and incorporates a new scoring adjustment method utilizing LLMs, named LISA. This approach seeks to improve silent speech recognition accuracy by:

1. Cross-Modal Learning: MONA employs two unique loss functions—cross-contrast (crossCon) and supervised temporal contrast (supTcon)—to align latent representations across different modalities (audio and EMG) within a shared latent space. This alignment is facilitated through the novel application of dynamic time warping in conjunction with these contrastive loss functions, enabling effective model training using both synchronized and independent datasets.
2. LLM-Integrated Scoring Adjustment (LISA): Beyond the neural network's predictions, LISA leverages the capability of LLMs to refine and choose the best candidate sentences. This post-processing step significantly enhances the recognition accuracy by selecting the most linguistically probable and coherent sentences from a set of top predictions.

Empirical Evaluation

The proposed approach underwent rigorous evaluation using several benchmark datasets, notably the Gaddy Silent Speech dataset. Results from the paper highlight the effectiveness of the cross-modal learning and LISA:

• A significant reduction in word error rate (WER) for silent speech, achieving a record low WER of 12.2% on an open vocabulary.
• For vocalized EMG recordings, the approach demonstrated an improvement from a WER of 23.3% to an astonishingly low rate of 3.7%.

Implications and Future Directions

The findings from this paper represent a significant step forward in the realization of practical and accurate silent speech interfaces. By achieving a WER below the 15% threshold, this work signals a pivotal shift towards the broader applicability of SSIs in real-world scenarios. It not only underscores the potential of EMG as a viable modality for silent speech recognition but also illustrates the profound impact of integrating large-scale LLMs in refining speech recognition accuracy.

Looking ahead, the methodologies introduced in this paper have the potential to be extended to a wider range of speech modalities, paving the way for more robust and versatile silent speech interfaces. Furthermore, the use of cross-modal learning strategies in other data-limited domains suggests a promising avenue for future research in machine learning and human-computer interaction.

Conclusion

This paper showcases a significant leap in silent speech recognition technology, driven by innovative cross-modal learning techniques and the strategic integration of LLM-enhanced scoring. As research in this field progresses, the envisioned future where SSIs offer seamless and accurate communication for all individuals moves ever closer to reality.