Measuring Robustness of Speech Recognition from MEG Signals Under Distribution Shift

Abstract: This study investigates robust speech-related decoding from non-invasive MEG signals using the LibriBrain phoneme-classification benchmark from the 2025 PNPL competition. We compare residual convolutional neural networks (CNNs), an STFT-based CNN, and a CNN--Transformer hybrid, while also examining the effects of group averaging, label balancing, repeated grouping, normalization strategies, and data augmentation. Across our in-house implementations, preprocessing and data-configuration choices matter more than additional architectural complexity, among which instance normalization emerges as the most influential modification for generalization. The strongest of our own models, a CNN with group averaging, label balancing, repeated grouping, and instance normalization, achieves 60.95% F1-macro on the test split, compared with 39.53% for the plain CNN baseline. However, most of our models, without instance normalization, show substantial validation-to-test degradation, indicating that distribution shift induced by different normalization statistics is a major obstacle to generalization in our experiments. By contrast, MEGConformer maintains 64.09% F1-macro on both validation and test, and saliency-map analysis is qualitatively consistent with this contrast: weaker models exhibit more concentrated or repetitive phoneme-sensitive patterns across splits, whereas MEGConformer appears more distributed. Overall, the results suggest that improving the reliability of non-invasive phoneme decoding will likely require better handling of normalization-related distribution shift while also addressing the challenge of single-trial decoding.

• The paper introduces instance-level normalization as the key driver for robust neural phoneme classification under distribution shifts in MEG signals.
• It demonstrates that group-based aggregation and label balancing are crucial for mitigating class imbalance and normalization-induced biases.
• Empirical results reveal that enhanced preprocessing strategies achieve up to 64.09% F1-macro, underscoring the limitations of relying solely on architectural complexity.

Measuring Robustness of Speech Recognition from MEG Signals Under Distribution Shift

Introduction

The paper "Measuring Robustness of Speech Recognition from MEG Signals Under Distribution Shift" (2604.04129) presents an empirical investigation into the generalization capabilities of neural phoneme classification models trained on non-invasive MEG recordings. Using the LibriBrain dataset and its associated PNPL competition protocol, the authors systematically evaluate architectural innovations, normalization strategies, group-based aggregation, label balancing, and multiple data augmentation tactics in the context of realistic distribution shifts between validation and test splits. The work uniquely foregrounds the critical impact of data configuration and normalization on model robustness, drawing attention to the inadequacy of incremental architectural complexity alone for achieving reliable generalization in neural decoding from noisy and variable MEG sources.

LibriBrain Dataset Characteristics and Data Distribution

The LibriBrain dataset comprises 1,622,678 phoneme-centered MEG windows collected from audiobook listening sessions, partitioned into train, validation, test, and a hidden holdout split for competitive evaluation. Each segment is 500 ms, across 306 sensors, with a 39-class phoneme inventory exhibiting a heavily imbalanced, long-tailed distribution.

Figure 1: Raw phoneme-class distribution for the train and validation splits, highlighting substantial class imbalance.

This imbalanced structure imposes specific challenges for both learning and evaluation. Standardization, group averaging, and label balancing methodologies are therefore critical for mitigating the statistical shift and improving representational invariance—especially under differential preprocessing between splits.

Model Architectures and Preprocessing Frameworks

The core architectural baselines include a ResNet-style CNN, an STFT-based CNN, and a CNN-Transformer hybrid. Key architectural and preprocessing details are as follows:

• Instance Normalization (IN): Applied per-sample and per-channel, IN eliminates between-split amplitude bias, directly targeting normalization-induced distribution shift.
• Group Averaging: Averages randomly grouped windows of the same phoneme class to enhance SNR and facilitate more stable learning.
• Label Balancing: Implements random oversampling to equalize class frequencies.
• Data Augmentation: Gaussian noise, temporal shift/masking, channel dropout, amplitude scaling, and frequency band perturbation are deployed with probability 0.3 per augmentation type.

Figure 2: Overall model architecture, comparing (a) ResNet CNN, (b) STFT-CNN, and (c) CNN-Transformer hybrid layouts.

The architectural variants show profound sensitivity to preprocessing strategies and normalization configurations, with the introduction of InstanceNorm yielding dramatic improvements in test and holdout generalization.

Empirical Results: Ablations and Generalization Trends

The results reveal that instance-level normalization is the single most impactful modification, with the best CNN configuration (with label balancing, repeated grouping, and IN) reaching 60.95% F1-macro on the test split, compared to 39.53% for the baseline without IN. MEGConformer, which applies instance-level normalization and dynamic grouping, achieves 64.09% F1-macro—a consistent high mark on both validation and test splits, denoting effective resistance to preprocessing-induced shift.

Figure 3: ResNet CNN architecture segment, illustrating convolutional backbone and residual path.

Architectural complexity alone—particularly STFT and hybrid Transformer variants—did not guarantee improved robustness without the accompanying normalization and grouping strategies. Ablations show substantial validation-to-test collapse for models without IN, indicating that distribution shift from differing normalization statistics is the dominant generalization bottleneck.

Saliency Map and Stability Analysis

Layer-wise saliency maps were computed per phoneme and per split to visualize class sensitivity profiles and probe the interpretability of learned representations. Saliency maps for MEGConformer and other models with InstanceNorm were highly stable across validation and test splits, unlike concentration-prone or shifting maps of non-IN variants.

Figure 4: Pearson correlation saliency matrix, evidencing robust cross-split stability for MEGConformer.

Quantitative analysis (Pearson/Spearman correlation) confirmed that models with IN maintained near-perfect correspondence (>0.95) of layer-phoneme saliency across splits, while non-IN architectures exhibited marked instability.

Key Claims and Contradictions

• Strong Claim: Instance-level normalization is the primary driver for cross-split generalization and saliency map stability, greatly outweighing incremental architectural complexity.
• Contradiction: Deep architectures and alternative label spaces (e.g., PanPhon or distribution mapping) do not confer robustness in the absence of normalization solutions; test improvements do not imply holdout transfer.

Practical and Theoretical Implications

Practically, the results prescribe that reliable neural speech decoding from MEG, particularly for future BCIs, will require robust normalization frameworks and denoising approaches capable of handling single-trial, low-SNR inputs. Theoretically, the saliency map analysis suggests that models exhibiting distributed, stable sensitivity profiles are more likely to generalize under domain shift. Strong reliance on group averaging, however, remains fundamentally incompatible with naturalistic applications, motivating ongoing research into adaptive normalization, stability-based objectives, and domain-invariant feature learning.

Future Directions

1. Single-Trial Decoding: Pursue advanced denoising and representation learning techniques to transcend group-averaged benchmarks.
2. Adaptive Normalization: Develop normalization mechanisms tailored to local distribution dynamics, minimizing split-induced shift.
3. Invariant Representation Learning: Incorporate explicit stability and invariance constraints to ensure network sensitivity aligns with domain-relevant neural processes.

Conclusion

The study rigorously demonstrates that effective MEG-based phoneme classification under distribution shift hinges predominantly on normalization-driven data configuration, with instance-level normalization emerging as the critical mechanism for robust generalization. Architectural innovation alone is incapable of bridging the validation-test gap in the absence of normalization and aggregation strategies. The findings solidify the necessity for further advances in adaptive normalization and single-trial decoding, with interpretability-driven analyses (such as saliency stability) providing valuable diagnostic tools for future neurolinguistic modeling and BCI development.