Papers
Topics
Authors
Recent
Search
2000 character limit reached

Language Generation from Brain Recordings

Published 16 Nov 2023 in cs.CL | (2311.09889v5)

Abstract: Generating human language through non-invasive brain-computer interfaces (BCIs) has the potential to unlock many applications, such as serving disabled patients and improving communication. Currently, however, generating language via BCIs has been previously successful only within a classification setup for selecting pre-generated sentence continuation candidates with the most likely cortical semantic representation. Inspired by recent research that revealed associations between the brain and the large computational LLMs, we propose a generative language BCI that utilizes the capacity of a LLM jointly with a semantic brain decoder to directly generate language from functional magnetic resonance imaging (fMRI) input. The proposed model can generate coherent language sequences aligned with the semantic content of visual or auditory language stimuli perceived, without prior knowledge of any pre-generated candidates. We compare the language generated from the presented model with a random control, pre-generated language selection approach, and a standard LLM, which generates common coherent text solely based on the next word likelihood according to statistical language training data. The proposed model is found to generate language that is more aligned with semantic stimulus in response to which brain input is sampled. Our findings demonstrate the potential and feasibility of employing BCIs in direct language generation.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (59)
  1. Speech synthesis from neural decoding of spoken sentences. Nature, 568(7753):493–498, 2019.
  2. Brain2word: decoding brain activity for language generation. arXiv preprint arXiv:2009.04765, 2020.
  3. Deep artificial neural networks reveal a distributed cortical network encoding propositional sentence-level meaning. Journal of Neuroscience, 41(18):4100–4119, 2021.
  4. Training language models to summarize narratives improves brain alignment. In Eleventh International Conference on Learning Representations. OpenReview. net, 2023.
  5. Scaling laws for language encoding models in fmri. arXiv preprint arXiv:2305.11863, 2023.
  6. Principal component analysis. Wiley interdisciplinary reviews: computational statistics, 2(4):433–459, 2010.
  7. The neurobiology of semantic memory. Trends in cognitive sciences, 15(11):527–536, 2011.
  8. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
  9. Deep language algorithms predict semantic comprehension from brain activity. Scientific reports, 12(1):16327, 2022.
  10. Brains and algorithms partially converge in natural language processing. Communications biology, 5(1):134, 2022.
  11. Andy Clark. Whatever next? predictive brains, situated agents, and the future of cognitive science. Behavioral and brain sciences, 36(3):181–204, 2013.
  12. Left insula activation: a marker for language attainment in bilinguals. Proceedings of the National Academy of Sciences, 101(42):15265–15270, 2004.
  13. Two distinct neural timescales for predictive speech processing. Neuron, 105(2):385–393, 2020.
  14. Kunihiko Fukushima. Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biological cybernetics, 36(4):193–202, 1980.
  15. Predictability and surprise in large generative models. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 1747–1764, 2022.
  16. The role of left prefrontal cortex in language and memory. Proceedings of the national Academy of Sciences, 95(3):906–913, 1998.
  17. Shared computational principles for language processing in humans and deep language models. Nature neuroscience, 25(3):369–380, 2022.
  18. Neurocomputational models of language processing. Annual Review of Linguistics, 8:427–446, 2022.
  19. Loghub: A large collection of system log datasets towards automated log analytics. arXiv preprint arXiv:2008.06448, 2020.
  20. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  21. The neural bases of sentence comprehension: a fmri examination of syntactic and lexical processing. Cerebral cortex, 11(3):223–237, 2001.
  22. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.
  23. Testing the correlation of word error rate and perplexity. Speech Communication, 38(1-2):19–28, 2002.
  24. Reconstructing meaning from bits of information. Nature communications, 10(1):927, 2019.
  25. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691, 2021.
  26. Words and the world: Predictive coding and the language-perception-cognition interface. Current Directions in Psychological Science, 24(4):279–284, 2015.
  27. Topographic mapping of a hierarchy of temporal receptive windows using a narrated story. Journal of Neuroscience, 31(8):2906–2915, 2011.
  28. Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81, 2004.
  29. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 61–68, 2022.
  30. A natural language fmri dataset for voxelwise encoding models. Scientific Data, 10(1):555, 2023.
  31. Cogtaskonomy: Cognitively inspired task taxonomy is beneficial to transfer learning in nlp. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 904–920, 2022.
  32. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1–35, 2023.
  33. Gpt understands, too. AI Open, 2023.
  34. Broca’s area and the language instinct. Nature neuroscience, 6(7):774–781, 2003.
  35. Neuroprosthesis for decoding speech in a paralyzed person with anarthria. New England Journal of Medicine, 385(3):217–227, 2021.
  36. Predicting human brain activity associated with the meanings of nouns. science, 320(5880):1191–1195, 2008.
  37. A comprehensive overview of large language models. arXiv preprint arXiv:2307.06435, 2023.
  38. The “narratives” fmri dataset for evaluating models of naturalistic language comprehension. Scientific data, 8(1):250, 2021.
  39. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744, 2022.
  40. Decoding vowels and consonants in spoken and imagined words using electrocorticographic signals in humans. Journal of neural engineering, 8(4):046028, 2011.
  41. Converging evidence for the neuroanatomic basis of combinatorial semantics in the angular gyrus. Journal of Neuroscience, 35(7):3276–3284, 2015.
  42. Toward a universal decoder of linguistic meaning from brain activation. Nature communications, 9(1):963, 2018.
  43. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318, 2002.
  44. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
  45. The neural architecture of language: Integrative modeling converges on predictive processing. Proceedings of the National Academy of Sciences, 118(45):e2105646118, 2021.
  46. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008–3021, 2020.
  47. Native language, gender, and functional organization of the auditory cortex. Proceedings of the National Academy of Sciences, 96(18):10460–10465, 1999.
  48. Towards sentence-level brain decoding with distributed representations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7047–7054, 2019.
  49. Neural encoding and decoding with distributed sentence representations. IEEE Transactions on Neural Networks and Learning Systems, 32(2):589–603, 2020.
  50. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.
  51. Semantic reconstruction of continuous language from non-invasive brain recordings. Nature Neuroscience, pages 1–9, 2023.
  52. Mariya Toneva. Bridging Language in Machines with Language in the Brain. PhD thesis, Carnegie Mellon University, 2021.
  53. Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain). Advances in neural information processing systems, 32, 2019.
  54. Higher language ability is related to angular gyrus activation increase during semantic processing, independent of sentence incongruency. Frontiers in human neuroscience, 10:110, 2016.
  55. Attention is all you need. Advances in neural information processing systems, 30, 2017.
  56. High-performance brain-to-text communication via handwriting. Nature, 593(7858):249–254, 2021.
  57. Unicorn: Unified cognitive signal reconstruction bridging cognitive signals and human language. arXiv preprint arXiv:2307.05355, 2023.
  58. A survey of stimulation methods used in ssvep-based bcis. Computational intelligence and neuroscience, 2010, 2010.
  59. Towards brain-to-text generation: Neural decoding with pre-trained encoder-decoder models. In NeurIPS 2021 AI for Science Workshop, 2021.

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up