MEG-XL: Long-Context Brain-to-Text Decoder
- MEG-XL is a long-context self-supervised transformer model designed for data-efficient brain-to-text decoding from non-invasive MEG signals.
- It utilizes masked token prediction and a criss-cross attention mechanism to capture both temporal and spatial dependencies over extended neural contexts.
- The model demonstrates robust word decoding performance in low-data settings by pre-training on minute-scale MEG samples and fine-tuning for contextual word retrieval.
MEG-XL is a long-context self-supervised transformer model for magnetoencephalography (MEG) designed for data-efficient brain-to-text decoding, especially in regimes where only limited subject-specific supervision is available. It is pre-trained on 2.5 minutes of MEG context per sample, approximately $191$k tokens, using masked token prediction over discretized neural signals, and is then fine-tuned for contextual word decoding from non-invasive brain data. Its central claim is that pre-training on minute-scale neural context yields representations that transfer substantially better than short-context pre-training, allowing the model to match supervised performance with a fraction of the labeled data and to outperform existing brain foundation models in low-data settings (Jayalath et al., 2 Feb 2026).
1. Concept and motivation
MEG-XL addresses a specific bottleneck in clinical brain-to-text interfaces: paralysed patients cannot typically provide extensive training recordings, yet non-invasive speech decoding methods have often depended on many hours of subject-specific labeled data. The model is built around the observation that pre-training priors critically depend on context. Natural speech unfolds gradually over minutes, but most earlier methods pre-trained with only a few seconds of context. MEG-XL therefore shifts the scaling axis from short-window representation learning to long-context pre-training, using $150$ s samples that are $5$– longer than prior work and equivalent to $191$k tokens per sample (Jayalath et al., 2 Feb 2026).
The model operates in a domain where MEG offers high temporal resolution and non-invasive acquisition but remains noisier and lower in spatial resolution than invasive electrocorticography. In that setting, the stated goal is not open-vocabulary language generation but word decoding from perceived speech. MEG-XL is positioned against two limitations in the literature: short-context non-invasive decoders that discard extended neural context, and brain foundation models whose pre-training contexts are typically limited to $0.5$–$30$ s and often collapse in low-data transfer regimes. A plausible implication is that MEG-XL treats long-range temporal structure in neural activity as a first-class modeling target rather than as a nuisance to be windowed away.
2. Data representation and preprocessing
The MEG-XL pipeline begins from continuous MEG sampled originally at $1000$ Hz. For both pre-training and fine-tuning, the data are high-pass filtered at $0.1$ Hz, low-pass filtered at $40$ Hz, and resampled to $150$0 Hz. During pre-training, continuous $150$1-minute windows are taken as samples and divided into contiguous $150$2 s subsegments. Each $150$3 s subsegment is baseline-corrected using its first $150$4 s per channel, robustly scaled to inter-quartile range so that the median is $150$5, $150$6, and $150$7, and then clamped to $150$8 (Jayalath et al., 2 Feb 2026).
Tokenization is performed with a frozen single-channel neural codec, BioCodec, applied independently to each sensor. For an MEG sample $150$9, BioCodec uses residual vector quantization with $5$0 levels, codebook size $5$1 per level, and temporal downsampling factor $5$2. The tokenizer outputs
$5$3
A key design choice is that the tokenizer does not compress across channels. This preserves reconstruction quality at the cost of longer token sequences. The resulting token stream is therefore long both because the context window is $5$4 s and because each channel retains its own discrete representation.
Embeddings are formed by looking up codebook vectors at each residual quantization level, concatenating them, and projecting to the transformer dimension:
$5$5
Sensor metadata are then added. Position $5$6 and orientation $5$7 are encoded with Gaussian Fourier features,
$5$8
using separate encoders and projections for position and orientation, while sensor type is represented with a learned embedding. This makes the input explicitly sensor-aware and allows pre-training across datasets with heterogeneous layouts.
3. Architecture and self-supervised pre-training
The backbone is an $5$9-layer transformer that uses criss-cross attention to scale to minute-level context. Each layer splits the feature dimension into two halves and applies temporal attention across time within each channel and spatial attention across sensors at each timestep. If 0 is the input to layer 1, the update is
2
Temporal attention uses rotary positional embeddings, while the transformer stack uses pre-layer normalization with RMSNorm, feedforward blocks with hidden dimension 3, SELU activation, and residual connections around both attention and feedforward sublayers (Jayalath et al., 2 Feb 2026).
This factorization reduces the cost of full attention from 4 to
5
which is the key systems-level enabler for 6 s inputs without channel compression. The architecture is therefore not merely a standard transformer applied to longer sequences; it is structured explicitly around the separability of temporal and sensor dimensions in MEG.
Pre-training uses masked token prediction. The 7 s tokenized input is divided into contiguous 8 s blocks in time, and blocks are randomly selected and masked uniformly across all channels until approximately 9 of tokens are masked. The model predicts the masked discrete codes through a linear head:
$191$0
with average cross-entropy loss
$191$1
Masking whole $191$2 s intervals across all sensors prevents trivial interpolation in either time or space and forces the model to use longer-range context.
The pre-training corpus comprises approximately $191$3 hours of unlabeled MEG from more than $191$4 subjects drawn from CamCAN, MOUS, and SMN4Lang. Different datasets have different channel counts, so MEG-XL pads to a maximum channel count and uses a channel mask to exclude padded channels from attention and from the loss. This enables cross-dataset self-supervision without requiring a single sensor topology.
4. Fine-tuning for contextual word decoding
MEG-XL is fine-tuned on a contextual word decoding task based on sequences of $191$5 words. For each word $191$6, a $191$7 s MEG window $191$8 is extracted starting $191$9 s before word onset. The $0.5$0 windows are concatenated along time, yielding a $0.5$1 s sequence that matches the pre-training context length. This input is not fully continuous MEG, but it preserves within-word temporal evolution and relative position within the $0.5$2-word sequence (Jayalath et al., 2 Feb 2026).
Rather than directly classifying words, the model maps MEG segments into a semantic embedding space. Target word embeddings are obtained from a T5-LLM with a SigLIP-style objective, and repeated words are masked in the contrastive loss. After running the full sequence through the MEG-XL backbone, hidden states are averaged over the $0.5$3 s interval corresponding to each word and flattened across channels into a word-specific representation $0.5$4. A small MLP head predicts the target embedding,
$0.5$5
and inference uses nearest-neighbor retrieval with cosine similarity:
$0.5$6
The evaluation datasets are the following.
| Dataset | Subjects | Hours per subject |
|---|---|---|
| MEG-MASC | 27 | 2 |
| Armeni | 3 | 10 |
| LibriBrain | 1 | 52 |
All three use $0.5$7 train, $0.5$8 validation, and $0.5$9 test splits, with the same stimuli assigned to the same split across subjects to prevent leakage through stimulus memorization. The main metric is top-10 balanced accuracy on the top-50 most frequent words, computed as macro-averaged top-10 retrieval accuracy across word classes. Fine-tuning uses AdamW, a backbone learning rate of $30$0, an MLP-head learning rate of $30$1, weight decay $30$2, gradient clipping $30$3, and early stopping with patience $30$4.
5. Empirical performance and representation analysis
In direct comparison with six pre-trained baselines—BioCodec, EEGPT, BIOT, BBL, BrainOmni, and LaBraM—MEG-XL is the only foundation model reported to remain robust in low-data regimes. With $30$5 of the training data, MEG-XL achieves $30$6 on MEG-MASC, $30$7 on Armeni, and $30$8 on LibriBrain, against a random baseline of approximately $30$9. With $1000$0 of the training data, it achieves $1000$1, $1000$2, and $1000$3, respectively. BrainOmni rivals or slightly exceeds it on Armeni and LibriBrain in the full-data setting, but collapses to about $1000$4 on MEG-MASC, whereas MEG-XL remains stable across all three evaluation regimes (Jayalath et al., 2 Feb 2026).
Relative to the supervised contextual word decoder of d’Ascoli et al. (2025), MEG-XL shows a distinctly data-efficient profile. On MEG-MASC it outperforms the supervised model across the entire data range, with gains up to more than $1000$5 absolute accuracy at some training sizes. On Armeni it leads by approximately $1000$6 absolute up to around $1000$7 hours total and then converges. On LibriBrain it matches or slightly exceeds the supervised model for at most $1000$8 hours of training data, after which the fully supervised system pulls ahead. The reported summary is that pre-training can partially substitute for subject-specific recording depth, including scenarios characterized as approximately $1000$9 hour versus $0.1$0 hours.
The representation analysis isolates context length as a causal factor. Linear probing experiments with models pre-trained on different context lengths show that longer pre-training context improves downstream word decoding on all three datasets, with diminishing returns after roughly $0.1$1 s. Matched-context and full-context inference curves are almost identical, which indicates that additional inference-time context is useful only when the model has been trained to exploit it. Zero-shot masked token prediction on unseen datasets improves almost log-linearly with context length up to $0.1$2 s and does not saturate at that point. This suggests that the benefits of long-context pre-training are not confined to speech decoding alone.
Attention analysis reaches a similar conclusion. Using the temporal attention matrix
$0.1$3
the model defines mean attention distance
$0.1$4
and attention entropy
$0.1$5
Short-context models exhibit diffuse attention with high entropy even in early layers, whereas long-context models show local attention in early layers and progressively broader integration in deeper layers, with lower entropy overall. This suggests a learned local-to-global hierarchy rather than indiscriminate long-range mixing.
6. Limitations, future directions, and terminological ambiguity
The authors identify several limitations. MEG is bulky and expensive and is not yet suitable for daily clinical deployment. The task is perceived-speech decoding rather than imagined speech. The vocabularies are limited to $0.1$6 or $0.1$7 words, and non-invasive performance remains far from the throughput and accuracy required for assistive devices. The main experiments use $0.1$8 Hz resampling with a $0.1$9 Hz low-pass filter, which is slightly off Nyquist; a $40$0 Hz ablation shows similar trends but somewhat lower performance because the tested context is shorter. The representational content of the learned features—whether phonological, lexical, semantic, or dominated by lower-frequency envelope structure—also remains open (Jayalath et al., 2 Feb 2026).
The stated future directions are correspondingly concrete: extending long-context pre-training to imagined speech and clinical populations, applying similar frameworks to EEG, exploring contexts beyond $40$1 s as memory permits, scaling from small closed vocabularies toward open-vocabulary decoding, and developing norms and regulations as decoding capability improves. A plausible implication is that MEG-XL reframes non-invasive neural decoding as a long-context pre-training problem, much as long-context language modeling reframed the role of document-scale dependencies in text.
The term “MEG-XL” is not entirely unambiguous across the arXiv literature. In a distinct biomagnetism usage, it has been used in the sense of an ultra-sensitive, extended, multipurpose MEG system based on low-$40$2 dc SQUID current sensors, hybrid pick-up coil arrays, and ultra-low-noise dewars for MEG, ULF MRI, current density imaging, and neuronal current imaging (Körber, 2019). Separately, the phrase has been used analogically in charged lepton flavour violation discussions, where MEG II was described as a “MEG-XL precursor” because it pushes the classic stopped-muon $40$3 search concept toward the $40$4 frontier after the original MEG limit $40$5 at $40$6 confidence level (collaboration et al., 2021, Mori, 2016). In current usage, however, the named system “MEG-XL” refers specifically to the 2026 long-context brain-to-text model, while related long-context autoregressive work such as source-space next-brain-token prediction occupies a neighboring but distinct line of development (Csaky, 28 Jan 2026).