Relating the Neural Representations of Vocalized, Mimed, and Imagined Speech

Abstract: We investigated the relationship among neural representations of vocalized, mimed, and imagined speech recorded using publicly available stereotactic EEG recordings. Most prior studies have focused on decoding speech responses within each condition separately. Here, instead, we explore how responses across conditions relate by training linear spectrogram reconstruction models for each condition and evaluate their generalization across conditions. We demonstrate that linear decoders trained on one condition generally transfer successfully to others, implying shared speech representations. This commonality was assessed with stimulus-level discriminability by performing a rank-based analysis demonstrating preservation of stimulus-specific structure in both within- and across-conditions. Finally, we compared linear reconstructions to those from a nonlinear neural network. While both exhibited cross-condition transfer, linear models achieve superior stimulus-level discriminability.

• The paper demonstrates that linear spectrogram decoders can successfully generalize across vocalized, mimed, and imagined speech, revealing a hierarchical representational organization.
• Mimed speech is found to occupy an intermediate representational position between vocalized and imagined speech, suggesting shared neural components.
• Linear models, although less accurate in overall correlation, better preserve sentence-level stimulus-specific structure, a finding with practical implications for neural decoding systems.

Overview

This paper investigates the relationships among the neural representations of three modes of speech production — vocalized, mimed, and imagined speech — using publicly available stereotactic EEG (sEEG) recordings (2602.22597). Rather than decoding each condition in isolation, as most prior work has done, the authors train linear spectrogram reconstruction models per condition and evaluate their generalization across conditions. The central finding is that decoders transfer successfully across all three conditions, indicating shared neural speech representations organized in a hierarchical fashion. A secondary contribution is a methodological comparison: although a nonlinear convolutional–recurrent network achieves higher reconstruction correlations, a linear decoder better preserves sentence-level stimulus-specific structure.

Data and modeling approach

The study uses the VocalMind dataset: sEEG recordings from a single participant producing 100 Mandarin Chinese sentences under vocalized, mimed, and imagined conditions, with two repetitions per condition. Preprocessed low-pass-filtered signals from 110 electrodes serve as neural input, and the NSL cortical model provides biologically motivated time–frequency stimulus representations.

The linear decoder reconstructs each spectrogram value from time-lagged neural responses via ridge regression, with regularization selected by grid search; separate transfer functions are fit for each condition ($G_v$, $G_m$, $G_i$). Statistical significance is assessed against null models trained on trial-shuffled data pairing mismatched responses and spectrograms. For comparison, the authors retrain the nonlinear CNN–RNN decoder from the original VocalMind work, which predicts mel-spectrograms that can be vocoded with HiFi-GAN.

Evaluation proceeds along two complementary axes. Envelope and spectrogram correlation quantify overall reconstruction similarity within and across conditions. Because correlation does not directly measure preservation of stimulus identity, the authors additionally perform a rank-based analysis: for each reconstruction, its envelope is correlated against all candidate target envelopes, yielding top-$k$ identification curves summarized by an area-under-curve (AUC) above chance ($k/N$).

Cross-condition generalization reveals a representational hierarchy

All within- and cross-condition reconstructions significantly exceed null performance ($p \ll 0.001$), establishing reliable cross-mode decoding. The pattern of transfer is asymmetric in an informative way:

• Vocalized-trained decoders perform best on vocalized data, followed by mimed, then imagined.
• Mimed-trained decoders perform equivalently on mimed and vocalized data ($p = 0.6$) but significantly worse on imagined data.
• Imagined-trained decoders perform best on vocalized data, with comparable performance on imagined and mimed data.

The authors interpret this asymmetry through a componential model in which imagined responses contain planning-related activity ($X_P$), mimed responses add articulatory activity ($X_A$), and vocalized responses add auditory feedback ($X_S$). Under this decomposition, cross-condition generalization depends on how much test-condition activity projects onto the subspace captured by the training-condition decoder. Mimed-trained decoders tolerate the extra sensory component during vocalization but lose the articulatory contribution when applied to imagined speech; imagined-trained decoders succeed on vocalized data to the extent that articulatory and sensory components project onto the planning subspace learned from imagined activity. The empirical ordering supports the hypothesis that mimed speech occupies an intermediate representational position between vocalized and imagined speech. Notably, the strong imagined-to-vocalized transfer indicates that the additional processes engaged during overt vocalization do not obscure planning-related information — a result relevant for BCI applications targeting individuals unable to speak.

Sentence-level discriminability

Rank analysis confirms that reconstructions retain stimulus-specific identity. Discriminability is consistently strongest when testing on vocalized responses: vocalized-trained models reach an AUC of 0.32 above chance, and both mimed- and imagined-trained models remain above chance on vocalized data. Mimed-trained decoders show similar discriminability on mimed (AUC = 0.13) and vocalized (AUC = 0.11) data but near-chance performance on imagined data (AUC = -0.01); imagined-trained decoders show comparable values on imagined (AUC = 0.07) and mimed (AUC = 0.08) data. This mirrors the correlation-based transfer structure, independently supporting the intermediate position of mimed speech.

The nonlinear network exhibits the same qualitative transfer pattern with slightly different details in the imagined-trained case, and achieves substantially higher within-condition discriminability on vocalized data (AUC = 0.59 versus 0.32 for the linear model). Across training–test pairs, however, the relationship between mean reconstruction correlation and discriminability differs systematically: while AUC increases with correlation for both architectures, the linear model shows a steeper relationship (Steiger's test, $G_m$0). This is the paper's strongest methodological claim: improvements in reconstruction accuracy do not translate proportionally into gains in sentence-level separation, and the interpretable linear model better preserves stimulus-specific structure despite lower raw correlation.

Limitations and open questions

Several constraints qualify these findings. All analyses rely on a single participant speaking Mandarin Chinese, so the generality of the observed hierarchy across subjects and languages remains untested. The componential account ($G_m$1, $G_m$2, $G_m$3) is a modeling assumption inferred indirectly from transfer behavior rather than demonstrated directly; the authors propose subspace analyses such as principal-component comparisons or subspace alignment to test it explicitly, and note that recordings during listened speech would be needed to isolate the hypothesized auditory feedback component. Imagined speech lacks external timing cues, complicating alignment and potentially depressing imagined-condition performance. Finally, why the linear and nonlinear models diverge in the correlation–discriminability relationship — and which neural components support cross-condition generalization in each architecture — remains unresolved.

Conclusion

Using sEEG recordings of vocalized, mimed, and imagined speech, this work demonstrates that linear spectrogram decoders generalize significantly across all three production modes, with transfer patterns consistent with a nested representational organization in which imagined speech engages a subset of the processes engaged by mimed speech, which in turn engages a subset of those engaged by vocalized speech. Rank analysis confirms that sentence-specific information survives reconstruction in both within- and cross-condition settings. The comparison with a nonlinear network yields a practically important result: higher reconstruction fidelity does not imply better stimulus-level discriminability, underscoring the value of interpretability alongside accuracy when evaluating neural decoding systems.