Assembling the Mind's Mosaic: Towards EEG Semantic Intent Decoding

Abstract: Enabling natural communication through brain-computer interfaces (BCIs) remains one of the most profound challenges in neuroscience and neurotechnology. While existing frameworks offer partial solutions, they are constrained by oversimplified semantic representations and a lack of interpretability. To overcome these limitations, we introduce Semantic Intent Decoding (SID), a novel framework that translates neural activity into natural language by modeling meaning as a flexible set of compositional semantic units. SID is built on three core principles: semantic compositionality, continuity and expandability of semantic space, and fidelity in reconstruction. We present BrainMosaic, a deep learning architecture implementing SID. BrainMosaic decodes multiple semantic units from EEG/SEEG signals using set matching and then reconstructs coherent sentences through semantic-guided reconstruction. This approach moves beyond traditional pipelines that rely on fixed-class classification or unconstrained generation, enabling a more interpretable and expressive communication paradigm. Extensive experiments on multilingual EEG and clinical SEEG datasets demonstrate that SID and BrainMosaic offer substantial advantages over existing frameworks, paving the way for natural and effective BCI-mediated communication.

• The paper introduces Semantic Intent Decoding (SID), which decomposes EEG signals into compositional semantic units to enable natural language reconstruction.
• BrainMosaic leverages a hybrid ModernTCN-Transformer module and LLM-based constrained generation to achieve superior performance metrics like UMA, MUS, and SRS.
• The study demonstrates open-vocabulary generalization and improved interpretability, paving the way for scalable BCIs to aid communication-impaired individuals.

Semantic Intent Decoding: A Principled Framework for EEG-Based Natural Language Generation

Introduction

The translation of neural activity to natural language remains an open challenge in neuroscience and neurotechnology, with major implications for the development of brain–computer interfaces (BCIs) for individuals with severe motor or language impairments. Historically, BCI research has favored speech decoding paradigms focused on articulatory and phonetic reconstruction. However, these approaches neglect the distributed and flexible nature of semantic processing in the brain. The paper "Assembling the Mind’s Mosaic: Towards EEG Semantic Intent Decoding" (2601.20447) introduces Semantic Intent Decoding (SID), a novel framework that addresses this gap by mapping EEG/SEEG signals into a set-based, compositional semantic space, subsequently reconstructing coherent utterances reflecting the decoded intent.

Theoretical Foundations of SID

SID is constructed from three linguistic and neuroscientific principles: semantic compositionality, semantic space continuity and expandability, and fidelity in natural language reconstruction.

1. Semantic compositionality motivates representing communicative intent as a variable-length, permutation-invariant set of semantic units. This approach is supported by psycholinguistic evidence indicating that comprehension does not require strict word order, particularly in short utterances, and that meaning is inherently multi-dimensional and chunked in working memory.
2. Continuity and expandability postulate that the semantic space is open and continuous, mirroring findings from neuroimaging and behavioral studies indicating that semantic representation in the brain is gradual, distributed, and rapidly integrates novel concepts. SID therefore decodes meaning as coordinates in a vector-space embedding, rather than as a fixed label from a closed set.
3. Fidelity asserts that reconstructed sentences must remain faithful to the decoded semantics and adhere to natural language syntax, leveraging LLMs only as constrained generators based on the set of retrieved semantic units.

Together, these principles circumvent the rigidity of fixed-class concept classifiers and the opacity of black-box neural-to-language mappings.

The BrainMosaic Architecture

The paper details BrainMosaic, a deep learning architecture that realizes SID. The system follows a multi-stage pipeline:

1. Semantic Decomposer: EEG/SEEG signals are processed by a hybrid ModernTCN and Transformer module, decomposing neural activity into $K$ semantic unit embeddings using set-based bipartite matching strategies inspired by DETR. This set-based approach achieves permutation invariance and supports variable cardinality (Figure 1).

   Figure 1: Overview of the Semantic Intent Decoding (SID) framework.

2. Semantic Retriever: Each candidate semantic unit embedding is aligned to a continuous, open-vocabulary embedding space (such as Doubao or Qwen3), using a combination of cosine similarity and binary classification heads to detect active units. The system also predicts holistic sentence-level semantic and categorical attributes for enhanced reconstruction and supervision.
3. Semantic Decoder: The final set of semantic units, filtered by confidence, is transformed into a structured prompt for an LLM. The LLM reconstructs the natural language utterance, constrained by the retrieved semantics and optional sentence-level attributes.

This modular composition allows for interpretability at both the unit (concept) level and the final utterance.

Experimental Results

SID and BrainMosaic are evaluated on a suite of public (Chisco, ChineseEEG-2, ZuCo 1.0/2.0) and private SEEG datasets, covering both Mandarin and English stimuli and a range of task paradigms (reading, imagined speech, clinical judgments).

Key findings include:

• Strong numerical improvements: BrainMosaic substantially outperforms fixed-class (Multi-Cls), sequential decoder (Seq-Decode), and unconstrained generation (Neuro2Semantic) baselines in both concept-level and sentence-level metrics (UMA, MUS, SRS, BERTScore-F1). For example, in the Chisco dataset, BrainMosaic achieves UMA of 0.5617 ($\pm$0.0085) and SRS of 0.6206 ($\pm$0.0034), both significant improvements over baselines.
• Open-vocabulary generalization: The continuous semantic space enables robust open-set decoding. When the retrieval vocabulary is expanded with tens of thousands of unseen high-frequency words, UMA and SRS exhibit only modest degradation, and MUS remains stable, indicating that the model retrieves semantically proximal units for OOV cases instead of defaulting to random selection.
• Scalability with data: When the training data proportion is reduced in the large-vocabulary setting, performance scales predictably, with no evidence of catastrophic forgetting of previously learned units. This confirms the expandability of the SID’s continuous semantic space.
• Fidelity and interpretability: Compared to unconstrained neural-to-text baselines, semantic-constrained generation via LLMs yields higher semantic reconstruction fidelity and more interpretable intermediate representations, as confirmed by SRS scores and the provision of transparent semantic unit outputs.
• Ablations: Removing the set-based decomposer, continuous semantic retriever, or LLM-based natural language decoder each results in significant performance loss, establishing the necessity of all components.

Qualitative analyses (Figure 2) reinforce that MUS and SRS scores track genuine semantic alignment rather than superficial token overlap, outperforming traditional NLP measures (BLEU, WER) in capturing paraphrastic or contextually appropriate outputs.

Figure 2: Qualitative examples of two-level evaluation metrics. Underlines mark meaningful Chinese or English word groups.

Regional Neurophysiological Analysis

On the clinical SEEG dataset, channel-level and electrode-level analyses using gradient-based saliency highlight that the superior and middle temporal cortex, primarily in the left hemisphere, contribute maximally to semantic intent decoding. There is a significant positive correlation ($P \leq 0.01$) between electrode saliency and decoding performance (UMA/MUS), aligning with established functional neuroanatomy of language and semantics (Figure 3).

Figure 3: Regional contribution analysis. (a,b) Spatial distribution of electrodes and SEEG channels on the MNI template brain (top and left views). (c) Gradient-based channel saliency, with red indicating higher contribution. (d) Positive correlations ($P \leq 0.01$) between electrode saliency and single-electrode decoding performance (UMA/MUS).

Implications and Future Directions

Theoretical implications: The SID framework instantiates a compositional view of semantic processing, operationalizing core principles from linguistics and cognitive neuroscience for BCI language generation. SID enables the parsing of neural signals into modular conceptual elements and provides transparent, interpretable links between low-level neural data and high-level language.

Practical implications: On the application side, BrainMosaic suggests a scalable and modular pipeline for restoring communication in individuals with locked-in syndrome, aphasia, or other severe speech impairments. The open-vocabulary, compositional architecture addresses the central limitations of prior fixed-label and black-box deep learning decoders, supporting more flexible and adaptive BCI systems.

Future directions:

• Improved semantic decomposition methodologies, potentially leveraging multi-modal or hierarchical latent variable models.
• Further investigation of semantic intent representations across distributed neural circuits, exploiting high-density SEEG or combined hemodynamic/oscillatory approaches.
• Extension to real-time, interactive BCI systems enabling back-and-forth communication and integration with external knowledge bases.
• Integration of richer, multimodal semantic spaces via advances in foundation model embeddings.

Conclusion

Semantic Intent Decoding and its implementation in BrainMosaic represent a principled, compositional, and interpretable approach to neural-to-language translation. By uniting set-based decomposition, continuous semantic space retrieval, and constrained natural language generation, this framework overcomes the limitations of previous fixed-class and black-box neural decoders. The results across diverse datasets and tasks demonstrate that SID supports robust, open-vocabulary, and faithful BCI-mediated communication grounded in both neuroscientific theory and modern machine learning.

(2601.20447)