Robust Evaluation of Neural Encoding Models via ground-truth approximation

Abstract: Encoding models enable measurement of how our brains represent sensory inputs using electro-and magneto-encephalography (MEEG). Evaluating how closely encoding models reflect the underlying brain functions is a crucial premise for model interpretation and hypothesis testing. However, the ground-truth neural activity is unknown, preventing model evaluation with respect to the target neural signal. Existing evaluation metrics must therefore relate model's predictions to noisy MEEG measurements, where most variance is stimulus-unrelated. Here, I introduce an evaluation framework where model predictions are compared to a ground-truth approximation, obtained by aligning MEEG signals with predictions using canonical correlation analysis and via participant averaging. The resulting metric (CPA-PA) yields single-participant evaluations outperforming conventional scores by 300-1000% on synthetic EEG data and 250% on 34 real MEEG datasets (818 datapoints). These gains reflect increased sensitivity to stimulus-relevant neural activity and reduced dependence on SNR, establishing ground-truth approximation as a robust framework for evaluating encoding models.

• The paper introduces CPA and CPA-PA to robustly evaluate neural encoding models, achieving up to 3241% improvement in prediction correlation scores.
• The framework employs Canonical Correlation Analysis for denoising and alignment, preserving interpretability while reducing bias in low-SNR MEEG data.
• Empirical results on synthetic and real datasets validate enhanced sensitivity and reproducibility of model assessments across diverse experimental paradigms.

Robust Evaluation of Neural Encoding Models via Ground-Truth Approximation

Introduction

This work addresses a foundational challenge in computational sensory neuroscience: the robust evaluation of neural encoding models using electro- and magneto-encephalography (EEG/MEG, collectively MEEG). Standard evaluation metrics—typically Pearson or Spearman correlations between predicted and observed sensor-level neural signals—are confounded by substantial stimulus-unrelated variance in MEEG data, leading to severe underestimation of model performance. The paper presents a principled evaluation framework, leveraging Canonical Prediction Alignment (CPA) and its participant-averaged extension (CPA-PA), for comparing model predictions to a denoised, ground-truth-approximated neural signal. This framework demonstrably enhances the detection of stimulus-relevant neural activity, affording a significantly more faithful characterization of encoding model performance, especially in low-SNR scenarios and challenging experimental contexts.

Methodological Framework

The CPA metric utilizes Canonical Correlation Analysis (CCA) as a post-hoc denoising and alignment stage between neural predictions and measured signals. Encoding models, typically linear time-invariant (LTI) architectures such as Temporal Response Functions (TRF), are first trained in a cross-validation loop. Their predictions are then aligned with observed MEEG activity by projecting both to a shared latent canonical space derivable via CCA. Crucially, CCA parameters are estimated independently of model fitting to avoid information leakage and optimistic bias.

For group experiments with time-locked stimuli, CPA is further extended via Participant Averaging (PA), wherein MEEG data are averaged across participants before canonical projection. CPA-PA thus yields a denoised, ground-truth-approximated signal with maximized inter-participant consistency, further suppressing idiosyncratic, stimulus-irrelevant fluctuations.

CPA and CPA-PA preserve the interpretability and computational tractability of LTI models while enhancing denoising capacity compared to direct CCA mappings (e.g., CCA-1 and CCA-2). The separation between model estimation and evaluation reduces overfitting and maintains clear interpretative semantics for spatial and temporal weights.

Empirical Validation and Results

Synthetic Data

On rigorous synthetic datasets—where ground-truth signals were precisely defined and confounded by empirically realistic levels of SNR variation—the framework demonstrated large, systematic improvements relative to classical metrics:

• For SNRs as low as -50 dB, CPA-PA produced up to 3241% increases in prediction correlation scores compared to the best sensor-level metric (R-MAX).
• These improvements generalized to signal detectability, as quantified by match-mismatch classification (identifying true-vs-random stimulus/response correspondence), with up to 9235% increases in the lowest-SNR data.
• Critically, even when traditional evaluation metrics were at chance, CPA-PA reliably extracted the ground-truth, stimulus-relevant signal.

Real Datasets

Across 34 real MEEG datasets (31 EEG, 2 MEG, 1 fNIRS; speech, music, sign language, multimodal stimuli; N=818 participant-level entries), the framework achieved:

• Mean correlation gains of 68% for CPA over R-MAX, and 252% for CPA-PA.
• Detectability gains of 71% (CPA) and 210% (CPA-PA).
• These effects were confirmed across diverse paradigms, including low-SNR child and infant data, and demonstrated robust main effects in repeated-measures ANOVA analyses.

Single-Participant and Multivariate Analysis

The framework enhanced sensitivity to participant-specific neural variables even under strong SNR heterogeneity. For simulated individual differences in neural encoding (parameter $\beta$ controlling word onset TRF strength), CPA-based metrics consistently correlated with the ground-truth variable, whereas classical metrics became dominated by SNR-related variability.

The CPA approach systematically outperformed both traditional forward encoding metrics (R-MAX and R-AVG) and direct, high-dimensional CCA mappings—especially in the presence of multiple stimulus features and neural generators, where interpretability and risk of overfitting are paramount.

Methodological Comparison and Theoretical Implications

Direct CCA mappings, particularly with lagged representations of neural and stimulus data (CCA-2), achieve high variance explanation but dilute interpretability and inflate overfitting risk. CPA's constraint to spatial projection—without time-lag expansion—strikes a principled balance, consolidating stimulus-relevant variance into a smaller number of canonical components, thus maintaining interpretability.

CPA-PA leverages natural replications in experimental design (identically timed stimuli) for inter-participant denoising, but at the cost of assuming group homogeneity. Alternatives such as multiway CCA provide participant-specific maps, but at higher computational/interpretational cost.

Importantly, the ground-truth approximation paradigm operates as a model-agnostic, add-on evaluation layer, compatible with any encoding architecture, not restricted to TRFs or LTI models. It offers a principled route for retrospective re-evaluation of prior results, allowing for more accurate assessments of model explanatory capacity without modifications to the underlying model fitting process.

Open Data Resource and Implementation

The paper introduces a standardized collection of 34 MEEG and fNIRS datasets, preprocessed with consistent pipelines and provided alongside evaluation code. This resource (https://osf.io/c76p8/overview, https://diliberg.net) enables large-scale, reproducible benchmarking of encoding models and evaluation metrics, facilitating cross-study methodological synthesis.

Implications and Prospects

Practical

• Enhanced sensitivity allows robust model evaluation in previously inaccessible datasets: low-SNR infant/child data, short-duration or mobile EEG, and "in-the-wild" recordings.
• Retrospective analysis of existing models can uncover previously unrecognized neural signals, justifying more detailed interpretation of model weights and architectures.
• The separation of evaluation from estimation preserves critical properties like interpretability, which are essential in clinical and translational neuroscience.

Theoretical

• The framework introduces a paradigm shift for encoding model evaluation in noisy neural data, enabling direct quantification of representational strength without inflating the model class or sacrificing interpretability.
• Facilitates more accurate mapping between observed behavior or naturalistic stimuli and neural coding, supporting advanced investigations into individual difference, development, and clinical populations.

Future Directions

• Extension to deep, nonlinear encoding and decoding models, with exploration of nonlinear canonical alignment.
• Incorporation of more advanced multiway CCA and domain-general denoising approaches for group analysis.
• Further validation and adaptation to high-temporal-resolution modalities (e.g., source-localized MEG, electrocorticography).

Conclusion

This work establishes ground-truth approximation via CPA and CPA-PA as a robust, noise-agnostic methodology for evaluating neural encoding models. By providing sensitive, interpretable metrics that more faithfully reflect the true explanatory power of encoding models, this framework addresses a fundamental limitation of traditional sensor-space evaluation. The provided standardized dataset resource and open codebase facilitate reproducibility while promoting methodological rigor. The proposed framework is positioned to become a new standard in neural encoding evaluation, with implications for all domains leveraging high-dimensional, noisy neural signals measured in naturalistic paradigms.

Reference: "Robust Evaluation of Neural Encoding Models via ground-truth approximation" (2604.14694)