LightBeam: An Accurate and Memory-Efficient CTC Decoder for Speech Neuroprostheses

Abstract: A promising pathway for restoring communication in patients with dysarthria and anarthria is speech neuroprostheses, which directly decode speech from cortical neural activity. Two benchmarks, Brain-to-Text '24 and '25, released intracranial recordings from patients with dysarthria along with a baseline algorithm trained with Connectionist Temporal Classification (CTC). Despite significant innovation on these benchmarks, all leading published prior work relies on a WFST-based CTC decoder that requires ${\sim}$320 GB of RAM. These memory requirements limit accessibility for both patients and researchers. Here, we propose LightBeam, a non-WFST based CTC decoder that requires only ${\sim}$10 GB of RAM and achieves state-of-the-art performance on both benchmarks. LightBeam achieves this by integrating an LLM into the beam-search process via delayed fusion, obviating the prior need for using a large N-gram LM. LightBeam is implemented in Python and is open-source.

• The paper introduces a non-WFST CTC decoding pipeline that reduces memory usage from 320GB to 10GB while integrating LLMs directly into beam search.
• The paper employs delayed fusion LLM integration during beam search, significantly improving word error rates on Brain-to-Text benchmarks in real-time settings.
• The paper demonstrates that a GPU-optimized, open-source decoding framework enables adaptive, low-resource neural speech prostheses with practical clinical applications.

LightBeam: An Accurate and Memory-Efficient CTC Decoder for Speech Neuroprostheses

Introduction and Motivation

LightBeam is introduced as a response to the significant memory bottleneck posed by Weighted Finite-State Transducer (WFST)-based CTC decoders, notably in the deployment of speech neuroprostheses for clinical communication restoration in dysarthria and anarthria. As detailed in "LightBeam: An Accurate and Memory-Efficient CTC Decoder for Speech Neuroprostheses" (2603.14002), contemporary SOTA CTC decoders for neural speech decoding demand up to 320GB of RAM due to reliance on expansive WFST-based integration with large N-gram LLMs and a post hoc LLM rescoring step. This restricts the clinical and research accessibility of neuroprosthetic speech systems.

LightBeam directly addresses these limitations via a fundamentally different non-WFST-based CTC decoding pipeline, reducing peak RAM usage to approximately 10GB. This is accomplished by integrating LLMs into the beam search process via delayed fusion, supplanting the need for intractable WFST-LLM coupling. The system allows efficient real-time deployment on resource-constrained hardware, has an open-source implementation, and yields robust improvements in recognition performance on the current B2T ‘24 and ‘25 benchmarks.

Technical Framework

Datasets and Benchmarks

LightBeam is evaluated on Brain-to-Text ‘24 and ‘25, the leading benchmarks for neural speech decoding, featuring >10,000 spoken sentence attempts per subject acquired through high-density ventral motor cortex intracranial arrays. These datasets feature challenging conditions: small subject pools, non-ideal population sizes for LLM fine-tuning, and a high degree of input signal variability.

Encoder Architectures

Two canonical CTC encoder configurations are considered: the organizers' bidirectional GRU (non-streaming for B2T '24, unidirectional for B2T '25), and the time-masked Transformer, which supports bounded KV-cache and streaming, thus being more suitable for clinical adaptation. Both models output 41 tokens comprising phonemes, blank, and space indicators, and are trained under standard CTC loss regimes.

Baseline WFST-CTC Decoder

The WFST-based decoder establishes a strong baseline by composing phoneme, lexicon, and LLMs into a static graph, supporting efficient CPU-based beam search but demanding immense RAM for storing and traversing the WFST compiled with a large 5-gram model. LLM integration is only feasible as a post hoc rescoring phase, as WFSTs cannot natively represent neural LMs.

LightBeam Decoding Algorithm

LightBeam is constructed atop the GPU-optimized FlexCTC decoder. Core features include:

• Parallelized beam expansion and scoring, with token- and word-level insertion bonuses.
• Lexicon constraints imposed through state transition tables, supporting efficient GPU-masked candidate pruning.
• Nested beam search for rigorous homophone resolution, combined with shallow fusion of N-gram LM log-probabilities for orthography disambiguation.
• Delayed fusion LLM integration: At fixed intervals, LLMs (Llama 3.2, 1B) rescore candidate beams, fully replacing the N-gram LM’s influence in scoring hypotheses, and providing non-greedy, context-rich feedback to the decoder.
• Fine-tuned next-word prediction using the subject’s available text data adapts the LLM to individual language priors.
• A final generative error correction stage with a more capable LLM (Llama 3.1 8B) refines the selected outputs by mapping candidate sentences to ground-truth corrections.

Experimental Results

Resource Efficiency and Performance

LightBeam yields a dramatic reduction in RAM consumption (from $\sim$320GB to $\sim$10GB), with additional reductions in VRAM and maintenance of real-time processing speeds (RTF < 1 on RTX 5090 GPUs). On both B2T ‘24 and ‘25, LightBeam significantly improves Word Error Rate (WER) over both the original and faithfully-reproduced WFST baselines, across 10 random encoder seeds (statistical significance: paired one- and two-sample t-tests, $p < 0.05$).

Figure 1: LightBeam continues to match or outperform baseline WFST decoder across all benchmarks when both methods are paired with generative error correction. Results are shown with time-masked Transformer. Asterisks indicate significant improvement when performing paired t-test across $n=10$ seeds.

Encoder Generalization

Critically, LightBeam outperforms the WFST decoder not only with the canonical GRU encoder but also with the advanced time-masked Transformer architecture. In all cases, LightBeam establishes new lowest WERs for both B2T ‘24 and ‘25, even when compared to previously published SOTA results achieved by large-scale pretraining and proprietary LLMs. This is a direct product of more effective LLM-guided first-pass decoding rather than advances in encoder architecture.

Generative Error Correction and SOTA

The application of LLM-based generative error correction (GEC) in conjunction with LightBeam achieves further performance gains. The GEC-enhanced LightBeam system attains WERs as low as 2–5%, surpassing previously published neural speech decoding systems that relied on considerably larger models and longer pretraining, and without any non-open LLMs. Notably, the performance gains remain additive and complementary to advances in encoder design.

Ablation Studies and Latency Tradeoffs

Removing core LightBeam features—such as LLM rescoring during beam search or LLM next-word fine-tuning—results in significant performance degradation, confirming their necessity for SOTA decoding. The homophone multi-variant handling, however, is found to have negligible effect in this domain. Hyperparameter sweeps on the LLM rescore interval reveal a tunable tradeoff between WER and real-time latency (RTF): as rescoring becomes more frequent, WER improves at the cost of higher compute.

Figure 2: Impact of LLM rescore interval on validation WER and RTF across both datasets. An encoder frame is output every 100 ms in B2T '24, and every 80 ms in B2T '25. Shaded regions indicate SEM across $n=10$ seeds. LLM rescore interval was set to 10 and 15 throughout the study for B2T '24 and '25, respectively.

Theoretical and Practical Implications

LightBeam demonstrates that WFST-based decoders are not necessary for SOTA performance in neural speech decoding and that LLMs can be effectively integrated into first-pass decoding with aggressive resource reductions. This paradigm enables local deployment on commodity hardware, reducing latency and privacy risks associated with remote decoding. The delayed fusion approach exposes new opportunities for balancing latency, resource use, and accuracy in real-world settings, and is model-agnostic—future improvements in open-source LLMs can be directly integrated. The results also suggest that tight LLM integration will be advantageous in long-context and low-data settings, with important implications for adaptive clinical devices.

Limitations revolve around increased latency compared to WFSTs, potential output “jitter” when N-gram and LLM scores diverge, and the requirement for initial per-subject LLM calibration. Nonetheless, LightBeam maintains real-time operation and adaptive potential for future extensions.

Conclusion

LightBeam introduces a resource-efficient, open-source, and SOTA-capable CTC decoding pipeline for neural speech prostheses, obviating the need for unscalable WFST-based systems and explicit N-gram LM dependency. Direct LLM integration into beam search yields robust, significant improvements over entrenched methods, supporting practical clinical and research utility. The modular delayed fusion framework is suitable for further innovation—including online adaptation, improved clinical calibration, and direct integration with more capable or specialized LLMs. LightBeam thus represents a substantive advance toward democratizing neural speech decoding research and translation for patient-facing applications.