MEBM-Phoneme: Multi-scale Enhanced BrainMagic for End-to-End MEG Phoneme Classification

Abstract: We propose MEBM-Phoneme, a multi-scale enhanced neural decoder for phoneme classification from non-invasive magnetoencephalography (MEG) signals. Built upon the BrainMagic backbone, MEBM-Phoneme integrates a short-term multi-scale convolutional module to augment the native mid-term encoder, with fused representations via depthwise separable convolution for efficient cross-scale integration. A convolutional attention layer dynamically weights temporal dependencies to refine feature aggregation. To address class imbalance and session-specific distributional shifts, we introduce a stacking-based local validation set alongside weighted cross-entropy loss and random temporal augmentation. Comprehensive evaluations on LibriBrain Competition 2025 Track2 demonstrate robust generalization, achieving competitive phoneme decoding accuracy on the validation and official test leaderboard. These results underscore the value of hierarchical temporal modeling and training stabilization for advancing MEG-based speech perception analysis.

• The paper introduces a 4.7-million-parameter end-to-end MEG decoder that combines BrainMagic mid-term encoding, multi-scale convolutions, depthwise fusion, and temporal attention for 39-class phoneme classification.
• The model achieves 60.95% ± 0.90 macro F1, 89.54% Top-3 accuracy, and 95.08% Top-5 accuracy on local validation, while ablations show that removing the BrainMagic encoder causes the largest performance loss.
• The training strategy improves noisy, imbalanced MEG data through session-aware validation, trial averaging, temporal jittering, and adaptive class weighting, but single-trial decoding and unexplained online test-set instability remain open challenges.

Overview

MEBM-Phoneme is an end-to-end neural decoder for phoneme classification from non-invasive magnetoencephalography (MEG), developed for Track 2 of the NeurIPS 2025 LibriBrain Competition (Landau et al., 11 Jun 2025, Özdogan et al., 2 Jun 2025). The system extends the BrainMagic backbone [defossez2023] with a short-term multi-scale convolutional pathway, a depthwise separable fusion layer, and a convolutional attention mechanism for temporal aggregation. Beyond architecture, the paper's principal methodological contributions lie in its data handling: a stacking-based session-aware local validation set designed to approximate the competition's holdout distribution, and a stochastic training protocol combining sample averaging, temporal jittering, and adaptive weighted cross-entropy to counteract severe class imbalance.

The motivation is straightforward: invasive neuroprostheses achieve high phoneme decoding accuracy by mapping neural activity directly to phoneme categories followed by language-model-based text reconstruction [metzger2023, willett2023, card2024], but non-invasive MEG suffers from substantially lower signal-to-noise ratio (SNR). The paper addresses this gap within the constraints of the LibriBrain benchmark, which provides over 50 hours of within-subject MEG from a single participant listening to audiobooks.

Model Architecture

The input is a 306-channel MEG sequence of $T = 125$ time points. A spatial attention module first re-weights sensor-wise activations, producing a projected representation $\mathbf{H}_s \in \mathbb{R}^{D \times T}$ with $D = 128$. Two parallel temporal streams then process this representation:

• Short-term pathway: twelve multi-scale dilated convolutional blocks capturing fine-grained local temporal dependencies across multiple receptive fields.
• Mid-term pathway: BrainMagic encoders extracting contextual features at longer time scales.

The two streams are concatenated along the channel dimension and fused via a depthwise separable convolution, which reduces computational cost and encourages disentanglement across temporal scales. A subsequent convolutional attention layer compresses channels to a single dimension through a 1D convolution, applies softmax along the temporal axis to obtain weights $\mathbf{W}_t$, and reweights each time step as $$\mathbf{H}_{\text{att}} = \mathbf{W}_t \odot \mathbf{H}_{\text{fused}}$$. Sum pooling collapses the temporal axis, and a linear softmax head outputs probabilities over 39 phoneme classes. The full model contains only 4.7M trainable parameters and trains in roughly three hours on a single NVIDIA A800 GPU.

Validation and Training Strategy

A central design choice is that validation and training samples share a unified construction rule based on per-class, per-session sample counts $n$, differing only in stochasticity. For validation, the number of averaged single-trial samples $n'$ follows a deterministic rule ($n'_{\text{val}} = 100$ if $n > 100$; $n'_{\text{val}} = n$ if $\mathbf{H}_s \in \mathbb{R}^{D \times T}$0; $\mathbf{H}_s \in \mathbb{R}^{D \times T}$1 if $\mathbf{H}_s \in \mathbb{R}^{D \times T}$2). Training uses the same thresholds but introduces randomness: $\mathbf{H}_s \in \mathbb{R}^{D \times T}$3 is drawn uniformly in $\mathbf{H}_s \in \mathbb{R}^{D \times T}$4 for mid-frequency classes and set to $\mathbf{H}_s \in \mathbb{R}^{D \times T}$5 for rare classes. At each iteration one phoneme class is sampled, and its instances are averaged according to this rule — a form of SNR boosting that also implicitly rebalances class frequencies.

Temporal robustness is addressed by jittering segment onsets uniformly within $\mathbf{H}_s \in \mathbb{R}^{D \times T}$6 samples before extracting a fixed 0.5 s window. The loss is an adaptively weighted cross-entropy whose per-class weights are empirically tuned to balance frequency and articulatory/acoustic confusability; notably, the weights span more than two orders of magnitude, from 0.05 for /ey/ to 10.00 for /uh/, indicating aggressive manual rebalancing rather than purely inverse-frequency weighting.

The offline validation set is built from Sherlock1 sessions 11–12 using eight independent sampling iterations per class, with fixed seeds and removal of classes lacking sufficient samples. This stacking-based construction is intended to statistically align the local evaluation with the official holdout distribution while mitigating session-specific bias.

Results and Ablation

On the local validation set, averaged over six random seeds, the full model achieves F1\textsubscript{macro} of 60.95% ± 0.90, Top-3 Acc\textsubscript{macro} of 89.54% ± 0.48, and Top-5 Acc\textsubscript{macro} of 95.08% ± 0.61. The ablation confirms consistent contributions from every component:

Variant	F1\textsubscript{macro} (%)	Top-3 Acc (%)	Top-5 Acc (%)
Full model	60.95 ± 0.90	89.54 ± 0.48	95.08 ± 0.61
w/o weighted loss	59.97 ± 0.90	88.87 ± 1.14	94.75 ± 0.63
w/o multi-scale conv	59.75 ± 0.68	88.98 ± 1.12	94.67 ± 1.03
w/o BM encoder	54.43 ± 2.07	84.96 ± 1.69	92.19 ± 1.28
w/o conv. attention	59.60 ± 0.82	88.47 ± 1.46	94.17 ± 1.13

The BM encoder is by far the most critical component: removing it costs 6.5 F1 points, roughly four times the degradation from removing any other module. This indicates that mid-term contextual modeling, not the added short-term machinery, carries most of the discriminative power, though the multi-scale branch, attention layer, and weighted loss each still yield measurable gains.

Two observations deserve emphasis. First, the large gap between Top-1 (implied by F1) and Top-5 accuracy (~34 points) suggests the model ranks the correct phoneme highly even when its top prediction errs, supporting the authors' claim that integration with a LLM exploiting linguistic context could yield substantial further improvements. Second, there is a notable discrepancy between local and online results: the model reached up to 72% decoding accuracy on the first half of the online test set but degraded on the second half. The authors attribute this partly to their submission strategy, but they do not fully explain it, and the claim that "results on the local evaluation set indicate strong generalization" sits uneasily against this half-to-half instability — a distributional shift across test halves that the session-aware validation design evidently did not capture.

Limitations and Open Questions

The paper concedes several limitations explicitly. Most fundamentally, all reported results rely on averaging multiple MEG trials to boost SNR; the authors identify accurate phoneme classification on single-trial, continuous MEG as the essential open problem for practical real-time speech decoding, deferring it to future work. Second, the adaptive loss weights are hand-tuned rather than learned or derived from a principled criterion, raising questions about transferability beyond this dataset and subject. Third, the explanation for the second-half online performance drop remains conjectural ("we conjecture... may be partly attributed to our submission strategy"), leaving open whether the model genuinely fails under distribution shift or whether the evaluation pipeline is at fault. Finally, the validation construction discards classes with insufficient samples, so the local metrics may not reflect performance on the rarest phonemes in the holdout set.

Conclusion

MEBM-Phoneme demonstrates that augmenting a mid-term contextual encoder with short-term multi-scale convolutions, attention-based temporal aggregation, and carefully engineered sampling and loss weighting yields competitive phoneme decoding on the LibriBrain 2025 benchmark, with ablations confirming each component's contribution and the BM encoder dominating performance. The work's most consequential limitation is its dependence on trial averaging: closing the gap between averaged-sample accuracy and single-trial, continuous-decoding performance remains unresolved, as does the unexplained asymmetry in online leaderboard results across test halves.