The Brain's Bitter Lesson: Scaling Speech Decoding With Self-Supervised Learning (2406.04328v3)

Abstract: The past few years have produced a series of spectacular advances in the decoding of speech from brain activity. The engine of these advances has been the acquisition of labelled data, with increasingly large datasets acquired from single subjects. However, participants exhibit individual differences, such as anatomy, and datasets use varied scanners and task designs. As a result, prior work has struggled to leverage data from multiple subjects, multiple datasets, multiple tasks, and unlabelled datasets. In turn, the field has not benefited from the rapidly growing number of open neural data repositories to exploit large-scale data and deep learning. This gap exists for all neural data, but especially for magnetoencephalography (MEG), where the scale of individual datasets has not yet caught up with other modalities. To address this, we develop a set of neuroscience-inspired self-supervised objectives, together with a neural architecture, for representation learning from heterogeneous and unlabelled neural recordings. Experimental results with MEG show that representations learned with these objectives scale with data, generalise across subjects, datasets, and tasks, outperform using the raw input representation, and even surpass comparable self-supervised approaches. In addition, we set new benchmarks for two foundational speech decoding tasks. Collectively, these methods now unlock the potential for training speech decoding models with orders of magnitude more existing data.

• The paper introduces an innovative SSL framework that utilizes vast unlabelled MEG data to decode speech across diverse subjects.
• It employs neuroscience-inspired pretext tasks and domain-specific transformations to boost cross-dataset and cross-task generalization.
• Empirical results show logarithmic performance gains with increased data, setting new benchmarks for non-invasive brain-computer interfaces.

Insights into Scaling Speech Decoding with Self-Supervised Learning

This paper advances the field of speech decoding from brain activity by addressing the limitations imposed by reliance on labeled datasets. Traditional approaches, grounded in supervised learning, often struggle to generalize across subjects, datasets, and task variations due to individual anatomical differences and varied experimental designs. The authors propose an innovative framework leveraging self-supervised learning (SSL) to utilize unlabelled and heterogeneous magnetoencephalography (MEG) data for robust and scalable speech decoding.

The paper introduces neuroscience-inspired self-supervised objectives together with a novel neural architecture. The architecture facilitates representation learning from a vast and diverse array of unlabelled neural recordings, utilizing pretext tasks that uncover implicit labels through domain-specific transformations on input signals. This method enables the representation to scale with data and generalize effectively across different contexts, including novel subjects not seen during training—a significant progression from traditional methods that typically require retraining for each new subject.

Key Numerical Results and Claims

The empirical results underline substantial improvements achieved through the proposed SSL approach. Using aggregated data from open neural repositories and multiple datasets, the trained models set new benchmarks in two primary speech decoding tasks: speech detection and voicing classification. Representations learned from pretext tasks not only demonstrated superior scalability with increasing unlabelled data quantities but also enhanced cross-subject, cross-dataset, and cross-task generalization. Notably, performance increased logarithmically with more data, maintaining improvements even at volumes surpassing those used in prior surgical studies.

Strong claims are made regarding the efficiency of these self-supervised objectives in unlocking orders of magnitude more data for model training. The framework surpasses comparable state-of-the-art self-supervised methods, such as BIOT, especially in data-efficiency critical self-supervised methods for MEG, suggesting a potential shift in how the field may approach brain data scalability and model learning in the future.

Implications for AI Developments

This research underscores the applicability of the "bitter lesson" of AI, which posits that general methods leveraging large-scale computation outperform tailored model-based approaches. By effectively exploiting larger datasets through generic self-supervised tasks, the proposed method realizes greater scalability and generalization without conforming to the convention of tailored, dataset-specific models. In further practical applications, this approach could lead to non-invasive brain-computer interfaces (BCIs) capable of assisting patients with speech impairments by robustly decoding speech without the need for extensive subject-specific data.

Future Directions

The demonstrated scalability and generalizability suggest promising avenues for future research. Developing other pretext tasks that capture even more nuanced features of neural data might further amplify performance. Extending this framework to include pre-training functionalities across additional brain modalities and non-linguistic datasets could yield a truly universal brain-to-text translation technology, moving us closer to practical, non-invasive BCIs for communication rehabilitation. The approach could also inspire new methodologies that realize these benefits across other neural interfaces and cognitive tasks.

In summary, this paper offers a rigorous approach to scaling speech decoding models through self-supervised learning, marking substantial progress in generalization, scalability, and the practical utilization of large-scale, heterogeneous brain data in AI.