- The paper introduces a dual-stream framework that decodes semantic content and acoustic attributes from EEG or MEG, then uses frozen ASR, speech-codec, and TTS priors to generate intelligible speech.
- MindVoice substantially outperforms prior baselines, reaching ASR-BERTScore-F1 scores of 0.379 on EEG and 0.441 on MEG, with mean opinion scores above 4.2 despite higher mel-spectrogram error.
- The results show that pretrained discrete representations improve reconstruction, but hallucinations, shortened sentences, weak recovery of specific content words, and limited fine-grained temporal accuracy remain major challenges.
Reconstructing continuous, intelligible speech from non-invasive neural recordings such as EEG and MEG remains an underdetermined inverse problem: the recordings are noisy, spatially blurred, and only weakly preserve information about perceived speech. "MindVoice: Reconstructing Intelligible Speech from Non-invasive Neural Signals with Pretrained Priors" (2605.31173) addresses this problem by arguing that direct regression from neural activity to entangled speech representations is insufficient, and that pretrained generative priors must be used to compensate for missing semantic and acoustic information. The paper reports substantial gains over prior neuro-to-speech baselines on semantic accuracy, timbre similarity, and perceptual quality, while explicitly conceding that reconstruction fidelity remains limited.
Prior non-invasive decoding work has largely targeted paraspeech information—speech envelope reconstruction, auditory attention decoding, voice activity detection, phoneme classification, and word classification from closed vocabularies. The few attempts at continuous waveform reconstruction (e.g., FESDE) align EEG with deep speech features and decode with a vocoder; the authors argue this yields spectrally similar but unintelligible output because the learned mapping from noisy neural signals to entangled representations is imprecise. MindVoice instead decomposes reconstruction along two complementary axes, motivated by dual-stream accounts of speech perception: a semantic-level stream recovering linguistic content, and an acoustic-level stream recovering pitch- and timbre-related attributes. A final generation stage fuses both using a pretrained TTS model with in-context voice cloning.
Method
The semantic-level stream treats EEG/MEG as single-channel images, applies cascaded CNNs plus a channel-wise MLP to produce spatiotemporal neural tokens, and models inter-token dependencies with a Transformer. A vector-quantized autoencoder (VQ-AE) trained on mel-spectrograms maps speech into discrete semantic tokens via nearest-neighbor lookup over a 2048-entry learnable vocabulary, trained with reconstruction, codebook, and commitment losses. An MLP projector maps neural embeddings into the pre-quantization speech embedding space, supervised by an ℓ2​ alignment loss plus a next-token prediction loss through a frozen Whisper-base ASR model. The frozen ASR supplies language-modeling priors that complete incomplete semantic information; the authors deliberately avoid fine-tuning it, finding limited neural data insufficient for improvement.
The acoustic-level stream uses a pretrained codebook-based speech codec (from FishSpeech-s1-mini). A second embedder-projector pair predicts codec latents from neural signals, optimized with quantized-latent ℓ2​ loss and a temperature-scaled contrastive loss. The authors note that raw predictions collapse toward averaged features carrying only weak acoustic cues; quantization against the pretrained codebook injects priors and restores discriminability.
The speech reconstruction branch conditions a frozen TTS model autoregressively on the decoded text, using predicted acoustic embeddings as in-context voice-cloning prompts. The TTS model is not fine-tuned against ground truth, because doing so trades intelligibility for spectrogram similarity—a design choice consistent with the paper's central claim about the mismatch between spectral error and perceptual quality.
Quantitative results
Experiments use the Brennan EEG dataset (49 participants, 10.1 hours) and Gwilliams MEG-MASC dataset (27 participants, 49 hours), each under a Random split and a stricter Sentence split where no sentence identity appears in more than one split. Baselines are a mel-spectrogram regression pipeline with BigVGAN-v2 vocoding ("Vanilla") and FESDE. Representative results:
| Data / Split |
Method |
HuBERT ↑ |
ASR-BERTScore-F1 ↑ |
WavLM ↑ |
MOS ↑ |
mel-MSE ↓ |
| EEG / Random |
Vanilla |
0.136 |
0.144 |
0.070 |
1.28 |
0.321 |
| EEG / Random |
FESDE |
0.391 |
0.189 |
0.054 |
1.43 |
0.198 |
| EEG / Random |
MindVoice |
0.752 |
0.379 |
0.664 |
4.26 |
0.413 |
| MEG / Random |
FESDE |
0.532 |
0.219 |
0.331 |
1.41 |
0.187 |
| MEG / Random |
MindVoice |
0.829 |
0.441 |
0.777 |
4.35 |
0.445 |
| MEG / Sentence |
MindVoice |
0.820 |
0.324 |
0.758 |
4.34 |
0.457 |
MindVoice loses on mel-MSE—expected, since baselines optimize exactly that objective while MindVoice does not—and performs comparably on MCD. The paper frames this as evidence of a clear mismatch between point-wise spectral error and perceptual quality: lower spectral error does not imply intelligibility. Qualitative spectrogram comparisons support this, showing heavily over-smoothed baseline outputs lacking harmonic structure and transients, whereas MindVoice preserves periodic harmonic patterns and silent intervals despite higher MSE. ASR transcription of MindVoice output contains recognizable content, while FESDE output transcribes to short uninformative fragments.
Two additional trends emerge. First, all methods perform better on MEG than EEG, consistent with prior work attributing this to signal quality and data scale. Second, the Sentence split degrades performance substantially on EEG but only marginally on MEG (e.g., ASR-BERTScore-F1 drops from 0.441 to 0.324), suggesting larger, cleaner datasets mitigate generalization to unseen sentences—an implication that data scale, rather than architecture alone, partly governs whether the framework learns a stable neural-to-speech mapping.
Interpretability, preference analyses, and ablations
Channel-wise attention visualizations show the semantic stream concentrates responses on sparse channels while the acoustic stream distributes responses broadly, indicating partially distinct neural substrates are exploited for the two targets. Sentence-length regression reveals a compression effect (fitted slope 0.59 below identity), with longer sentences systematically shortened—a consequence of incomplete semantic information in the neural signal. Fine-grained linguistic analyses show grammatical function words are recovered more reliably than lexically specific content words; recovered and hallucinated words are both biased toward high Zipf frequency, so low-scoring reconstructions fall back on generic frequent vocabulary when target content is uncertain.
Ablations isolate contributions. Removing the acoustic stream leaves ASR-BERTScore-F1 unchanged (0.441) with only slight HuBERT/WavLM decreases, indicating the two streams are only weakly coupled and semantics drive intelligibility. Removing the VQ vocabulary degrades semantic metrics sharply (HuBERT 0.617, ASR-BERTScore 0.294), confirming the discrete-token interface is critical for injecting ASR priors. Most importantly, replacing MEG input with Gaussian noise collapses all metrics (HuBERT 0.243, ASR-BERTScore 0.110) and causes degeneration to fixed short phrases—addressing concerns raised in prior work that EEG-to-text evaluations can be confounded by insufficiently rigorous protocols, and demonstrating the reconstructions genuinely derive from neural signals.
Limitations and open questions
The paper is explicit about its constraints. The best baseline-scaled BERTScore-F1 of 0.441 means most reconstructions still contain substantial semantic discrepancies; incorporating pretrained priors necessarily introduces generative hallucination when neural evidence is insufficient, so outputs should be interpreted cautiously. The objective optimizes overall semantic and timbre similarity without guaranteeing correctness of semantic or acoustic units at each time step, making the method unsuitable for studies requiring fine-grained temporal fidelity. Scope is limited to auditory-perception-evoked signals; performance on overt or imagined speech is unknown. Open questions include reducing hallucinated content while improving accuracy, achieving temporally aligned fine-grained reconstruction, and extending the framework beyond listened speech toward practical non-invasive speech BCIs.
Conclusion
MindVoice reframes non-invasive speech reconstruction as a decomposition into semantic and acoustic subproblems, each augmented by frozen pretrained priors (ASR language modeling, codec quantization, TTS generation with voice cloning). It establishes state-of-the-art results on EEG and MEG benchmarks across semantic, timbre, and perceptual-quality metrics, and its ablations—including the noise-input control—support the claim that gains derive from genuine neural information combined with pretrained priors rather than evaluation artifacts. The work's honest accounting of limited reconstruction success rates and hallucination risk positions it as a rigorous baseline for subsequent research on non-invasive speech brain-computer interfaces.