Brain-to-Text '24: Silent-Speech Benchmark

• The paper introduces DCoND, a context-aware diphone decoding approach that leverages local phonetic context to reduce phoneme error rates.
• The research employs a GRU-based RNN with CTC loss and a multi-stage phoneme-to-text pipeline to outperform traditional baseline methods.
• Brain-to-Text '24 benchmarks silent-speech decoding in an ALS subject, highlighting the effectiveness of LLM integration for error correction.

Brain-to-Text ’24 denotes a 2024 benchmark and associated line of silent-speech brain-computer interface research in which intended speech is reconstructed from neural activity recorded while a subject with ALS silently attempts to speak. In the benchmark’s strongest reported system, neural activity is decoded through context-aware phonetic units called diphones, then converted to text through a staged language-model pipeline. The central claim is that phoneme decoding is improved when local phonetic context is modeled explicitly rather than treating each phoneme as an isolated class, yielding state-of-the-art performance on the Brain-to-Text 2024 benchmark with a reported 15.34% Phoneme Error Rate and, with fine-tuned GPT-3.5 integration, 5.77% Word Error Rate (Li et al., 2024, Willett et al., 2024).

1. Benchmark setting and problem formulation

Brain-to-Text ’24 is a silent-speech decoding system: it reconstructs intended speech from neural activity recorded while a subject with ALS silently attempts to speak displayed sentences. The benchmark task is difficult because the neural signal is noisy, multivariate, temporally unaligned with the true phoneme sequence, and the neural-to-phoneme mapping is highly nonlinear. The official benchmark used a held-out private dataset of 1200 sentences and evaluated systems primarily by word error rate (WER) (Li et al., 2024, Willett et al., 2024).

The reported dataset configuration for the DCoND study comprises 8800 training sentences, 600 validation sentences, and 1200 competition/test sentences, with a vocabulary size of 125,000 words. Neural recordings were obtained from ventral premotor cortex (area 6V), and the neural features included multiunit threshold crossings and spike band power. Because the subject attempted speech silently, there was no exact neural-speech alignment, which is why CTC loss was used for training (Li et al., 2024).

The benchmark baseline was a two-stage system in which a recurrent neural network mapped neural activity to phoneme logits and a LLM converted phoneme logits into text. That baseline achieved 9.7% WER, and the competition was created to improve on this reference point (Willett et al., 2024). Brain-to-Text ’24 is therefore best understood both as a benchmarked task and as a concrete decoding paradigm: neural activity is first projected into an intermediate linguistic representation, then transformed into text by language modeling.

2. Context-aware diphones and the DCoND formulation

The key methodological contribution of the 2024 winning line of work is DCoND (Divide-and-Conquer Neural Decoder), which replaces direct monophone decoding with a context-aware intermediate target. The motivating observation is that the brain does not appear to encode a phoneme in isolation; instead, its representation depends on surrounding phonetic context via coarticulation. The paper supports this with neuroscience evidence and with empirical visualization showing that neural activity aligned to the same phoneme forms multiple subclusters depending on what phoneme came before it. A phoneme such as AH may appear in distinct neural subclusters such as DH → AH or SIL → AH, implying multiple neural modes for what is nominally the same monophone (Li et al., 2024).

DCoND formalizes monophone decoding as marginalization over context-dependent subclasses:

$$f(x) := p(z|X) = \sum_{S} p(z, S | X) = \sum_{s \in S} g_s^z(x)$$

Here $X \in \mathbb{R}^{T \times D}$ is the neural activity, $z$ is a phoneme class, and $S$ denotes the context variable. For diphones specifically, the formulation becomes

$p(Z=c_i|X) = \sum_{c_j\in S} p(c_j, c_i | X),$

where $p(c_j, c_i|X)$ is the probability that the neural activity encodes the diphone transition $c_j \rightarrow c_i$ (Li et al., 2024).

In implementation, the decoder outputs a $40 \times 40$ diphone probability matrix at each time step, with rows corresponding to the preceding phoneme and columns to the current phoneme. The single-phoneme probability is recovered by summing the joint probabilities column-wise. The divide-and-conquer procedure therefore has three steps: learn context-aware diphone classes, marginalize diphone probabilities back into phoneme probabilities, and use those phoneme probabilities downstream for text decoding (Li et al., 2024).

This representation is intentionally local rather than maximal. The paper also evaluates triphones, but reports that triphones generally underperform diphones in WER, likely because of sparsity and mismatch with the downstream language-model distribution. A plausible implication is that Brain-to-Text ’24 benefits from a context representation that is rich enough to capture coarticulation yet still compact enough to preserve stable estimation.

3. Neural decoder architecture and training

The neural stage takes multi-channel neural time series $X \in \mathbb{R}^{T \times D}$ as input and outputs diphone probabilities, followed by marginalized phoneme probabilities. The reported best-performing neural decoder is an RNN-based GRU model. The authors compare Transformer, LSTM, and GRU decoders and conclude that the GRU performs best, with the further conclusion that, for this benchmark and data scale, an RNN is a more appropriate neural decoder than a Transformer (Li et al., 2024).

Training uses CTC loss because there is no exact temporal alignment between neural bins and phoneme labels. The paper gives the CTC objective in unaligned-sequence form as

$p(Z|X) = \sum_{A \in \mathcal{A}_{(X,Z)} \prod_{t=1}^{T'} p(a_{t}|X),$

where $X \in \mathbb{R}^{T \times D}$0 is the set of valid alignments between $X \in \mathbb{R}^{T \times D}$1 and the target sequence $X \in \mathbb{R}^{T \times D}$2. In addition, DCoND uses a combined loss over both the monophone task and the diphone subclass task:

$X \in \mathbb{R}^{T \times D}$3

where $X \in \mathbb{R}^{T \times D}$4 is the loss for single-phoneme decoding, $X \in \mathbb{R}^{T \times D}$5 is the loss for diphone decoding, and $X \in \mathbb{R}^{T \times D}$6 balances the two. The paper notes that $X \in \mathbb{R}^{T \times D}$7 is increased gradually during training so that the model first learns subclass structure and then emphasizes the monophone objective (Li et al., 2024).

The reported training configuration is batch size 32, learning rate 0.02, Adam optimizer, and training on 2× RTX 2080 Ti for about 6–8 hours. The phoneme logits fed to the 5-gram model are temperature-scaled with $X \in \mathbb{R}^{T \times D}$8. The benchmark’s post-competition analysis further emphasizes that improvements in optimization, including learning-rate scheduling, were among the most effective ways to improve the recurrent baseline, whereas attempts to use deep state space models or transformers did not yet offer a benefit over the optimized RNN baseline (Li et al., 2024, Willett et al., 2024).

4. Phoneme-to-text decoding and large-language-model integration

Once phoneme probabilities are obtained, Brain-to-Text ’24 uses a staged phoneme-to-text pipeline. The first stage is candidate generation with a 5-gram model, which turns phoneme distributions into sentence candidates and assigns likelihoods. The second stage is rescoring with OPT, which re-scores candidate transcriptions and retains the highest-likelihood ones. The third stage is error correction with GPT-3.5, used as an ensemble-based corrector over multiple transcription candidates (Li et al., 2024).

Two LLM operating modes are defined. In DCoND-LI, GPT-3.5 is used through in-context learning: the prompt contains examples of multiple candidate transcriptions, their corresponding phoneme sequences, and the correct transcription for those examples, after which GPT-3.5 refines a new query without weight updates. In DCoND-LIFT, GPT-3.5 is fine-tuned on many prediction-correction pairs. The explicit enhancement over prior work is that GPT is given not only candidate transcriptions but also the decoded phoneme sequences. The paper states that this is stronger than using transcription candidates alone and treats the task as jointly recovering the correct transcription and phoneme sequence (Li et al., 2024).

The benchmark retrospective identifies this broader pattern as the major lesson of the competition: the largest improvements came from ensembling multiple independent decoders and merging their outputs with a fine-tuned LLM. All three top entrants used this general strategy. In the first-place DCoND-LIFT pipeline, the third step merged 10 sentence candidates produced from an ensemble of 10 independent RNN decoders, each with 5-gram hypothesis generation and OPT rescoring upstream (Willett et al., 2024).

This architecture makes the language-model stage more than a conventional post hoc spell-checker. It functions as a structured inference layer operating over multiple noisy hypotheses and, in the DCoND formulation, over phonetic evidence as well as orthographic hypotheses.

5. Quantitative results and ablation structure

The principal reported metrics are PER, WER, and P-WER. The strongest result reported for the full system is DCoND-LIFT WER = 5.77%, compared with 8.93% WER for the benchmark’s leading prior method LISA. The paper identifies this as an improvement of 3.16 absolute WER points. At the neural-decoder level, the context-aware diphone formulation achieves 15.34% PER compared with 16.62% PER for monophone-based decoding (Li et al., 2024).

A concise view of the headline results is as follows:

Method	PER	WER
NPTL	16.62	9.46
LISA	—	8.93
DCoND-L	15.34	8.06
DCoND-LI	—	7.29
DCoND-LIFT	—	5.77

The paper also reports P-WER = 8.02 for DCoND-L and P-WER = 11.33 for NPTL. It emphasizes that DCoND-L already beats NPTL while using the same general backbone and similar language-model stack, that DCoND-LI improves over DCoND-L through in-context GPT-3.5 refinement, and that DCoND-LIFT is best overall (Li et al., 2024).

The ablation results are tightly connected to the model’s design claims. For the loss-balance parameter $X \in \mathbb{R}^{T \times D}$9, the reported WERs are 8.47 at $z$0, 8.70 at $z$1, 8.06 at $z$2, 8.64 at $z$3, and 9.46 at $z$4, where $z$5 corresponds to the monophone-only baseline. Thus the best reported setting is $z$6. The paper also reports that removing OPT rescoring degrades WER, longer 5-gram dependencies help, and fine-tuning GPT-3.5 with phoneme inputs gives the best result (Li et al., 2024).

The benchmark synthesis reinforces the same hierarchy of effects. It reports that lower phoneme error rates from the RNN do not always lead to lower word error rates when the phoneme logits are decoded with the LLM, which is a reminder that Brain-to-Text ’24 is not a single-model optimization problem but a coupled neural-decoder and language-decoder system (Willett et al., 2024).

6. Position within brain-to-text research and principal limitations

Brain-to-Text ’24 sits within a broader progression of brain-to-language research. Earlier invasive systems translated cortical recordings into text through phoneme decoding plus language modeling or through direct character decoding with CTC and beam search. Examples include a phoneme-classification and particle-filtering framework using intracranial depth electrodes (Sheth et al., 2019) and Brain2Char, which decoded character sequences from ECoG using 3D Inception layers, bidirectional recurrent layers, dilated convolution layers, and language-model-weighted beam search (Sun et al., 2019). In non-invasive work, open-vocabulary EEG-to-text generation was explicitly advanced by BART-based EEG-to-text sequence-to-sequence modeling (Wang et al., 2021), by subject-dependent EEG representation learning integrated with BART and GPT-4 refinement (Amrani et al., 2023), and by DeWave, which translated full raw EEG periods without word-level order markers through a discrete codex representation (Duan et al., 2023). MEG work has likewise pursued unseen-text generation through multi-alignment with speech representations (Yang et al., 2024).

Against that background, Brain-to-Text ’24 is distinctive in four respects. First, it is centered on silent attempted speech in an ALS subject rather than natural reading or stimulus captioning. Second, it uses a benchmark explicitly built around speech restoration and held-out WER evaluation (Willett et al., 2024). Third, it argues that the decisive intermediate representation is not the monophone or the character but the context-dependent diphone (Li et al., 2024). Fourth, its strongest performance depends on a tightly integrated 5-gram + OPT + GPT-3.5 stack rather than on a single end-to-end generative model.

Several limitations are explicit or implicit in the reported studies. Evaluation is on a single subject / benchmark setting. The best overall result comes from combining multiple components, including GPT-based correction, so the end-to-end gain is not solely from diphone modeling. Triphones indicate that more context is not always better, because the class count becomes sparse and unstable. The benchmark report further notes that replacing the recurrent neural decoder with Mamba or transformers did not outperform the optimized RNN baseline, which suggests that current performance is still shaped heavily by data scale, local temporal structure, and optimization details rather than by architectural novelty alone (Li et al., 2024, Willett et al., 2024).

A common misconception is that Brain-to-Text ’24 is simply a language-model story. The benchmark lessons argue otherwise: the largest gains came from LLM-based ensembling and transcript merging, but performance gains were also found by improving how the recurrent neural decoder was trained and by using a diphone training objective (Willett et al., 2024). Another misconception is that richer phonetic context automatically improves decoding; the triphone ablations show that this is not generally the case (Li et al., 2024). The broader implication is that Brain-to-Text ’24 advances the field not by abandoning structured phonetic decoding, but by making that structure more neurally faithful and then coupling it to stronger language-level inference.