Contrastive Learning with Variational Regularization for Multi-Session EEG-to-Speech Decoding

Abstract: Reconstructing heard speech from non-invasive electroencephalography (EEG) is challenging due to a low signal-to-noise ratio (SNR) and inter-session variability. While trial averaging improves the SNR, it is difficult to apply to continuous speech. We instead use repeated EEG responses to the same stimulus across different sessions as positive pairs for contrastive learning, and introduce variational regularization that, combined with this contrastive objective, keeps the encoder representation space broad. Experiments on a Japanese EEG dataset show that combining the session-invariant strategy with variational regularization improves the character error rate (CER) while maintaining mel-spectrogram reconstruction fidelity. Session probing confirms that the encoder representations achieve session-invariance.

• The paper combines cross-session momentum contrastive learning with indirect variational regularization, reducing CER from 0.968 to 0.948 while preserving mel-spectrogram fidelity.
• Cross-session contrastive learning drove session-probing accuracy to 2.08%, statistically indistinguishable from the 2.2% chance level, demonstrating session-invariant EEG representations.
• The combined method mitigated representational collapse and restored latent-space variability, but decoding remained poorly intelligible and validation was limited to one participant and repeated stimuli.

Overview

This paper addresses a persistent obstacle in non-invasive speech decoding from electroencephalography (EEG): inter-session variability. The authors, Mizuno and Nakashika of The University of Electro-Communications, propose a training framework for EEG-to-speech reconstruction that combines two objectives: momentum contrastive learning (MoCo) using cross-session positive pairs, and variational regularization applied indirectly to the encoder output. The central claim is that these two components are complementary — neither alone yields consistent gains, but their combination improves character error rate (CER) while preserving mel-spectrogram reconstruction fidelity and achieving session-invariant encoder representations. Experiments are conducted on the SpREAD dataset, a multi-session Japanese auditory EEG corpus.

Method

The architecture builds on a Transformer-based sequence-to-sequence EEG-to-speech model that predicts 80-dimensional log-mel spectrograms, subsequently vocoded by HiFi-GAN. The baseline loss combines mel-spectrogram L1 reconstruction, stop-token binary cross-entropy, and guided attention losses.

The first contribution is a session-invariant contrastive strategy. Because each stimulus was presented in three separate sessions, EEG responses to the same linguistic content across different sessions form positive pairs; one recording serves as the anchor and the other two as positives. This differs from prior contrastive approaches based on temporal proximity, augmentation, cross-subject responses to the same stimulus, or unaligned same-subject segments. A momentum encoder updated by exponential moving average ($m = 0.999$) and a memory bank of size 4,096 supply negatives, with InfoNCE computed at temperature $\tau = 0.07$.

The second contribution is a stochastic regularization module (SRM). Since positive pairs lack temporal alignment, global average pooling summarizes the encoder output $\boldsymbol{h}$ into $\bar{\boldsymbol{h}}$, from which a diagonal Gaussian posterior $q_\psi(\boldsymbol{z} \mid \bar{\boldsymbol{h}})$ is predicted and sampled via reparameterization. A KL divergence term toward $\mathcal{N}(\mathbf{0}, \mathbf{I})$ regularizes this latent space. Crucially, $\boldsymbol{z}$ is used only for the contrastive and KL objectives and is not fed to the decoder, which cross-attends solely to the deterministic $\boldsymbol{h}$. This departs from standard VAEs, ContrastVAE, and variational information bottlenecks, where the latent variable participates in the generative pathway; here it acts purely as an indirect regularizer on encoder dispersion.

Experimental setup

Experiments use the SpREAD dataset: a single participant listening to 1,353 Japanese utterances from 18 speakers, each presented three times on separate days across 45 sessions over 9 days. EEG was recorded with a 64-channel Biosemi ActiveTwo system at 2,048 Hz, preprocessed to 512 Hz with 1–40 Hz bandpass filtering. Data were split into 3,195 training, 432 development, and 432 evaluation samples, with all three recordings of the same text assigned to the same split.

Evaluation covers four dimensions: CER via a Japanese ASR model, speaker identity via SECS (ECAPA-TDNN cosine similarity) and speaker probing accuracy (SPA, linear SVM on encoder states), acoustic fidelity via per-band Pearson correlation (PCC) of mel-spectrograms, and session-invariance via session probing accuracy against 45 session labels (chance ≈ 2.2%). Statistical significance uses Holm–Bonferroni-corrected Wilcoxon signed-rank tests for continuous metrics and McNemar's exact test for probing outcomes.

Results

The headline results show modest but statistically significant improvements from the combined objective:

Condition	CER ↓	SECS ↑	SPA (%) ↑	PCC ↑
Baseline	0.968 ± 0.121	0.190 ± 0.160	5.32 ± 1.08	0.282 ± 0.150
CL alone	0.956 ± 0.095	0.173 ± 0.158	4.86 ± 1.03	0.262 ± 0.143
VR alone	0.974 ± 0.113	0.193 ± 0.163	5.79 ± 1.12	0.277 ± 0.146
Proposed (CL + VR)	0.948 ± 0.097	0.176 ± 0.151	7.18 ± 1.24	0.274 ± 0.155

Three findings stand out. First, contrastive learning alone trades linguistic fidelity for acoustic fidelity: it significantly improved CER but significantly degraded PCC relative to the baseline. Second, variational regularization alone produced no significant improvement on any metric, and its session probing accuracy remained well above chance (~24.8%), indicating it did not reduce session dependence by itself. Third, the combination retained the CER gain and restored PCC to baseline level (paired comparison $p = 0.046$ versus VR alone), while SPA showed a non-significant trend toward improvement ($p = 0.087$, Cohen's $\tau = 0.07$0).

Session probing provides the most striking result: all contrastive-learning conditions dropped to statistically chance-level session decodability (e.g., 2.08% for the proposed method, indistinguishable from the 2.2% floor by binomial test), compared with roughly 23–25% without contrastive learning. This confirms that the cross-session positive-pair construction successfully removes session-specific information from the encoder representation.

The mechanism behind the PCC recovery is analyzed through latent distributions. Variational regularization alone caused complete posterior collapse — a failure mode previously reported for CR-VAE — though the collapse was confined to the VAE projection, leaving the encoder summary vector at std = 0.332. Contrastive learning alone also yielded a narrow distribution (std = 0.264). Combined, both $\tau = 0.07$1 (std = 1.099) and $\tau = 0.07$2 (std = 1.012) recovered near-Gaussian spread. Because $\tau = 0.07$3 is a linear projection of $\tau = 0.07$4, the KL gradient propagates directly to widen the encoder space, simultaneously addressing contrastive collapse and posterior collapse.

An ablation testing whether $\tau = 0.07$5 should be prepended to $\tau = 0.07$6 as decoder input found no significant difference on any metric under either base configuration, supporting the design choice of using $\tau = 0.07$7 solely as a regularization signal and avoiding unnecessary decoder attention computation.

Limitations and open questions

The paper concedes several constraints explicitly. All experiments use a single participant, which limits generalizability, though the authors argue this isolates inter-session variability from subject differences. The absolute decoding quality remains far from practical use: CER values near 0.95 indicate that reconstructed speech is barely intelligible, and SECS/PCC improvements are incremental rather than transformative. The SPA trend did not reach statistical significance, so claims about improved speaker identity encoding rest on effect sizes alone. Additionally, the framework presupposes repeated presentation of identical stimuli across sessions — a property not available in all recording paradigms — and the analysis does not establish how the method scales to more sessions, more subjects, or stimuli without cross-session repeats. Whether the observed complementarity between contrastive and variational objectives holds at higher SNR or with invasive recordings remains an open question the paper does not address.

Conclusion

This paper demonstrates that cross-session contrastive learning, which requires no temporal alignment across trials, can drive session information in EEG encoder representations to chance level, and that pairing it with an indirect variational regularization term prevents the representational collapse that otherwise degrades acoustic reconstruction fidelity. The complementary roles of the two objectives — supported by significant CER improvement, restored PCC, and floor-effect session probing — constitute a coherent contribution to multi-session EEG-to-speech decoding, albeit within the confines of a single-subject evaluation and near-chance intelligibility of the decoded speech.