BrainMosaic: Semantic Intent Decoding

• BrainMosaic is a deep learning architecture that translates EEG/SEEG signals into natural language through compositional semantic intent decoding.
• It employs a modular pipeline of semantic decomposition, optimal set matching via the Hungarian algorithm, and LLM-based sentence reconstruction.
• Experimental validation shows superior UMA, MUS, and SRS performance over conventional BCI frameworks, enhancing interpretability and adaptability.

BrainMosaic is a deep learning architecture for semantic intent decoding from neural signals, specifically EEG and SEEG, enabling the translation of brain activity into natural language by modeling meaning as flexible sets of compositional semantic units. Designed to overcome the interpretability and semantic expressiveness limitations of conventional brain–computer interface (BCI) decoding frameworks, BrainMosaic operationalizes the Semantic Intent Decoding (SID) paradigm, providing a compositional, continuous, and extendable approach to BCI-mediated communication (Li et al., 28 Jan 2026).

1. Conceptual Foundations: Semantic Intent Decoding (SID)

BrainMosaic is a concretization of the SID framework, which is based on three core principles:

1. Compositionality: Communicative intent $I$ is modeled as a finite, unordered set of semantic units, denoted $S = \{u_1,\,u_2,\,\dots,\,u_n\}$. This mirrors psycholinguistic theories where meaning is constructed from flexible “chunks” rather than linear, class-constrained units.
2. Continuity and Expandability: Each semantic unit $u$ exists as a vector in a continuous open embedding space $\mathcal{V} \subset \mathbb{R}^d$, where cosine similarity $\mathrm{sim}(E(u_i),E(u_j))$ quantifies graded semantic similarity.
3. Fidelity: The system enforces semantic and linguistic faithfulness by ensuring generated sentences are grounded in decoded semantic units and manifested as grammatically and contextually appropriate natural language.

The SID pipeline comprises three sequential modules:

• Semantic Decomposer: Extracts candidate semantic representations from neural data.
• Semantic Retriever (Set Matching): Selects the optimal set of semantic units using bipartite set matching.
• Semantic-Guided Reconstruction: Assembles the final output sentence from the retrieved set.

2. BrainMosaic Architecture

BrainMosaic employs a modular architecture, structured as follows:

2.1. EEG/SEEG Encoder

A ModernTCN (Temporal Convolutional Network) combined with a Transformer encodes raw, multi-channel time series signals $x \in \mathbb{R}^{C \times T}$ into feature tokens $X \in \mathbb{R}^{N \times D}$. A set of $K$ learnable slot queries then attends over $X$ to produce $K$ candidate embeddings: $S = \{u_1,\,u_2,\,\dots,\,u_n\}$0.

2.2. Semantic Unit Matcher (Retriever)

For each candidate slot embedding $S = \{u_1,\,u_2,\,\dots,\,u_n\}$1, the retriever performs bipartite optimal assignment (using the Hungarian algorithm) to align $S = \{u_1,\,u_2,\,\dots,\,u_n\}$2 with a continuous text embedding $S = \{u_1,\,u_2,\,\dots,\,u_n\}$3 or a “no-object” class, producing slot activity scores $S = \{u_1,\,u_2,\,\dots,\,u_n\}$4 and a global sentence embedding $S = \{u_1,\,u_2,\,\dots,\,u_n\}$5 augmented with attribute predictions.

2.3. Semantic-Guided Sentence Reconstructor

Retrieved units $S = \{u_1,\,u_2,\,\dots,\,u_n\}$6 and global attributes $S = \{u_1,\,u_2,\,\dots,\,u_n\}$7 are converted into a structured prompt, $S = \{u_1,\,u_2,\,\dots,\,u_n\}$8, which is then fed into a LLM, such as GPT-4o-mini, to generate the final natural language output: $S = \{u_1,\,u_2,\,\dots,\,u_n\}$9

3. Mathematical Formulation and Optimization

3.1. Set Matching and Assignment

The assignment problem is: $u$0 where $u$1 is the per-slot matching loss, $u$2 is an optimal permutation of slots, and $u$3 is the symmetric group on $u$4 elements.

3.2. Objective Losses

• Hungarian Loss: Aggregates per-slot assignments.
• Per-Slot Matching Loss:

$u$5

with $u$6 as the binary activity label and $u$7 is cosine similarity.

• Global Intent Loss:

$u$8

where $u$9 is cross-entropy.

• Total Retriever Loss:

$\mathcal{V} \subset \mathbb{R}^d$0

• Total Training Loss:

$\mathcal{V} \subset \mathbb{R}^d$1

with $\mathcal{V} \subset \mathbb{R}^d$2 a contrastive margin loss for “no-object” slots.

3.3. Metrics

• Unit Matching Accuracy (UMA):

$\mathcal{V} \subset \mathbb{R}^d$3

• Mean Unit Similarity (MUS):

$\mathcal{V} \subset \mathbb{R}^d$4

• Sentence Reconstruction Similarity (SRS):

$\mathcal{V} \subset \mathbb{R}^d$5

• BERTScore-F1 is used as a secondary measure; perplexity/BLEU and WER are de-emphasized due to limited semantic fidelity capture.

4. Experimental Validation

BrainMosaic was evaluated using multiple datasets:

• Public multilingual EEG: Chisco (Chinese, imagined speech), ChineseEEG-2 (Chinese), ZuCo 1.0/2.0 (English).
• Private clinical SEEG: 1 subject, 515 imagined Chinese sentences across five topics.

Key comparative baselines include Cls-Align (fixed-label classification with embedding alignment), Multi-Cls (multi-label classification), Seq-Decode (LSTM-based sequential decoding), and Neuro2Semantic (end-to-end embedding alignment plus a generator).

Level	Cls-Align	Multi-Cls	Seq-Decode	BrainMosaic
UMA	—	0.0359 ± 0.0006	0.1786 ± 0.0126	0.6596 ± 0.0102
MUS	—	0.6739 ± 0.0061	0.6503 ± 0.0115	0.8124 ± 0.0108
SRS	0.5976 ± 0.0030	0.4400 ± 0.0082	0.5104 ± 0.0114	0.6651 ± 0.0045
BERT-F1	—	0.6173 ± 0.0075	0.6126 ± 0.0121	0.6629 ± 0.0137

All leading metrics (UMA > 0.65, SRS > 0.62 for clinical SEEG) reflect pronounced superiority over prior paradigms ($\mathcal{V} \subset \mathbb{R}^d$6).

As the retrieval vocabulary expands (base to base +30,000 words), UMA and MUS decline modestly, while SRS remains robust. Scaling the training data from 10% to 100% yields monotonic increases in both UMA and SRS, supporting continuous space learnability.

5. Interpretability and Open-Set Generalization

BrainMosaic enables granular inspection at the semantic unit level. The intermediate unit set $\mathcal{V} \subset \mathbb{R}^d$7 permits transparent error analysis; slot activity $\mathcal{V} \subset \mathbb{R}^d$8 reflects per-unit confidence. The continuous nature of the embedding space enables retrieval of unseen concepts via nearest-neighbor search in $\mathcal{V} \subset \mathbb{R}^d$9. Use of definition-based embedding (rather than word-only) further extends semantic coverage.

Open-set generalization is supported by the embedding architecture, enabling BrainMosaic to effectively decode out-of-vocabulary meanings—a critical limitation in conventional classification approaches.

6. Regional Contributions and Neurophysiological Correlates

Gradient-based saliency analysis in the clinical SEEG data shows moderate correlation ($\mathrm{sim}(E(u_i),E(u_j))$0) between slot decoding and regional neural activity, with strongest contributions from the superior temporal gyrus/sulcus (STG/STS) and middle temporal cortex. These findings align with established roles for these regions in semantic representation and decoding.

7. Significance and Implications

BrainMosaic operationalizes a major advance in BCI decoding by integrating set-based compositionality, continuous embeddings, and LLM-constrained language generation. The architecture’s alignment with psycholinguistic theory, its transparency at the semantic chunk level, and its adaptability across languages and open vocabularies constitute significant steps for natural and interpretable BCI-mediated communication (Li et al., 28 Jan 2026). A plausible implication is the potential for more naturalistic, expressive BCIs capable of adapting to user-specific or context-dependent semantics without reengineering for each new vocabulary.