Zero-Shot Imagined Speech Decoding via Imagined-to-Listened MEG Mapping

Abstract: Decoding imagined speech from non-invasive brain recordings is challenging because imagined datasets are scarce and difficult to align temporally across subjects and sessions In this work, we propose a new approach to the decoding of imagined speech that leverages the richer and more reliably labeled recordings during listening to speech. We collected paired listened and imagined MEG recordings to rhythmic melodic and spoken stimuli from trained musicians. Using trained musicians helped improve temporal alignment across conditions. We then developed a three-stage decoding pipeline that revealed consistent and meaningful relationships between neural activity evoked by imagining and listening to the same stimuli. First, we trained six linear and neural models to map imagined MEG responses to listened responses. We evaluated these models against a null baseline from unseen subjects to validate that the predicted-listening responses preserve stimulus-specific information. In the second stage, we trained a contrastive word decoder exclusively on the listened MEG responses, and evaluated it using four embedding strategies including semantic, acoustic, and phonetic representations. In the third stage, we process the imagined MEG responses from held-out subjects through the mapping pipeline to compute the corresponding listening responses that are then decoded by the listened decoder. Using rank-based analysis, we show that the imagined words are decodable significantly above chance. We shall report here the results of a proof-of-concept implementation to decode imagined speech, where all evaluations are performed on held-out subjects. We also demonstrate that performance improves with training data size, suggesting that this approach is scalable and can directly be made applicable to realistic brain-computer interface scenarios.

• The paper introduces a zero-shot framework that maps imagined MEG signals to listened counterparts, enabling speech decoding without imagined labels.
• It evaluates six mapping architectures, highlighting that linear lag regression and RNNs scale effectively with more training data.
• The work employs a contrastive word-level decoder using hybrid embeddings (BERT, Wav2Vec2) to achieve robust above-chance decoding across subjects.

Zero-Shot Decoding of Imagined Speech via Imagined-to-Listened MEG Mapping

Experimental Design and Decoding Pipeline

This work addresses the persistent challenge of decoding imagined speech from non-invasive brain recordings, introducing a zero-shot framework that relies exclusively on magnetoencephalography (MEG) data elicited during overt listening. The experimental cohort comprised musically trained subjects, selected specifically to achieve precise temporal alignment between listened and imagined conditions. This facilitated the pairing of neural responses obtained during overt listening and silent imagery of temporally structured speech and melodic stimuli.

The decoding pipeline is structured in three clearly delineated stages. First, six mapping models—including both linear (e.g., lagged ridge regression) and nonlinear neural network architectures (shallow MLP, CNN1D, UNet, bidirectional GRU, TCN)—are trained to map subject-level imagined MEG time series onto their listened counterparts. Second, a contrastive word-level decoder is fitted using only listened MEG, leveraging state-of-the-art word encoders (BERT, Wav2Vec2, Whisper, and BERT+Wav2Vec2) to project neural response segments and corresponding word embeddings into a common representational space. Finally, for subject- and trial-level zero-shot decoding, imagined MEG from held-out participants is passed through the trained mapping model and the frozen listened decoder to produce and rank candidate word embeddings.

Figure 1: Experimental paradigm depicting collection of MEG data during listened and imagined trials, and the three-stage decoding pipeline for mapping, contrastive decoding, and zero-shot word-level retrieval.

Mapping from Imagined to Listened MEG Responses

Quantitative evaluation of the mapping stage shows all six architectures achieve significantly above-null mean per-channel Pearson correlation on both training and leave-one-subject-out (LOSO) held-out test conditions, with the linear lagged model yielding the highest raw correlations, and the RNN showing greatest separation from null baselines in cross-subject generalization. Notably, mapping performance scales monotonically with increasing training data, demonstrating the regime is data-limited rather than bounded by model class capacity.

Correlation-based classification on the predicted listened MEG (from mapped imagined responses) confirms that stimulus identity is preserved at a level significantly above chance, even in the absence of any imagined stimulus labels during either mapping or decoding. Fine-grained stimulus discrimination remains challenging, but classification is reliable at the melody-versus-poem category level. These results suggest that the mapping function recovers stimulus-discriminative structure latent in noisy imagined responses.

Figure 2: Mapping evaluation results showing prediction correlations, confusion matrices for stimulus classification, and data scaling analysis highlighting the monotonic benefit from increasing training set size.

Listened-Condition Contrastive Decoder

The second stage contrastive decoder maps listened MEG windows to word embeddings using four encoder strategies. The decoder’s performance is robustly above chance for all encoders, with hybrid semantic-phonetic embeddings (BERT+Wav2Vec2) yielding the highest recall at all rank thresholds. Analysis demonstrates that accuracy varies by word: semantic representations disambiguate distinctive content (BERT); phonetic encodings offer complementary improvements for underspecified lemma (Wav2Vec2); and a combined embedding maximizes per-word and global performance.

Figure 3: Contrastive decoder architecture and word-level decoding performance for listened MEG, comparing semantic, acoustic, phonetic, and concatenated embedding models.

Full Zero-Shot Pipeline for Imagined Speech Decoding

When the pipeline is composed end-to-end—applying the trained mapping model to imagined MEG and then the listened-decoder to the predicted-response—word-level rank-based decoding remains consistently above chance for all subjects, mapping architectures, and embedding strategies. This is a decisive demonstration of zero-shot imagined speech decoding without requiring any imagined data or labels in training. The sets of best-decodable words are consistent across subjects and architectures, and overlap significantly with those accessible to the decoder in the listened condition, substantiating the recovery of meaningful stimulus information.

Individual subject analysis confirms robust cross-subject generalization with limited inter-individual variability. Error analysis attributes the drop in full pipeline relative to the decoding ceiling (listened-only) to (1) additive noise introduced by imperfect mapping stage prediction and (2) limited vocabulary size constrained by the small-scale dataset.

Figure 4: Full pipeline rank-based word decoding CDFs and word consistency analysis, showcasing above-chance zero-shot decoding and cross-subject consistency of decodable words.

Figure 5: Subject-averaged normalized area under the decoding curve (AUC), validating robust performance across all held-out participants.

Model Analysis and Architectural Implications

A salient outcome is the competitive performance of linear mapping (lag regression) relative to complex neural models, indicating that the mapping between imagined and listened neural responses to rhythmic/speech stimuli is largely linear. While advances in sequence modeling (e.g., transformers) are not yet beneficial in this limited data regime, monotonic scaling trends point to likely future gains as larger paired MEG datasets become available.

Transformer-based mapping fails to generalize beyond null baselines under current data constraints, corroborating the status of the problem as regime-limited. In contrast, the sequential RNN and UNet models show some incremental advantage in generalization to unseen subjects, indicating that modeling context beyond per-channel, per-time dependencies is beneficial, but only marginally so at the current data scale.

Context, Theoretical and Practical Implications

This framework positions the reliably labeled, data-rich listened condition as a supervisory ground for training decoders intended for the scarce, noisy, and label-free imagined condition. The finding that above-chance decoding can be achieved without imagined labels—using only a mapping and universally trained decoder—suggests a pathway to generalizable non-invasive BCIs for speech prosthesis, particularly for locked-in or anarthric patients.

The result that model scaling and data scaling continue to raise performance indicates that the primary bottleneck is dataset size. Thus, expansion of temporally-aligned paired MEG datasets—and transfer learning from large-scale, listened-only MEG corpora—should yield substantial gains. The pipeline is designed modularly, enabling straightforward substitution of more powerful decoders or mapping networks as datasets grow.

Future Directions

Practical deployment in BCI will require (1) collection of large-scale, paired imagined-listened datasets with greater stimulus vocabulary, (2) optimization for latency and online operation (particularly for causal architectures like TCN), (3) integration of more sophisticated word/phrase-level supervision and generative reconstruction targets, and (4) extension to continuous or spontaneous imagined speech. Improved alignment, including modeling non-linear temporal warping, may further close the gap to the listened decoding ceiling.

From a theoretical viewpoint, the finding of approximately linear transformability between imagined and listened responses supports the hypothesis that the neural codes underpinning internal and external speech are highly overlapping, with systematic, learnable divergence.

Conclusion

This work establishes a proof of concept for zero-shot imagined speech decoding from non-invasive MEG using a direct mapping to listened responses followed by a contrastive, listened-trained decoder. Significant above-chance decoding is achieved across subjects and architectures, with robust word consistency and cross-encoder generalization. Linear mapping suffices in the current regime, but all approaches benefit with increasing data, indicating scalability. The methodology lays the foundation for label-efficient BCI systems for imagined speech and motivates further research in data scaling, architecture design, and the neural basis of speech imagery.

(2605.08075)