- The paper introduces a two-stage framework that aligns fMRI activity with CLIP and LLaVA embedding spaces before generating Japanese inner-speech descriptions through a frozen multimodal LLM, avoiding word-level temporal alignment and decoder fine-tuning.
- MindAlign consistently outperforms randomized-fMRI controls and often improves on fMRI-only decoding, with BERT-Scores of roughly 0.72–0.78 but substantially lower BLEU and ROUGE-L scores that indicate semantic paraphrasing rather than exact word recovery.
- The paper demonstrates partial cross-subject transfer, with all off-diagonal BERT-Scores exceeding 0.69, while its small five-participant dataset, delayed written-recall labels, and reliance on visual inputs leave subject-independent decoding and fidelity to actual inner speech unresolved.
Overview and Motivation
MindAlign is a two-stage brain-to-language framework that decodes freely generated inner speech in Japanese from fMRI recordings, without requiring word-level temporal alignment or fine-tuning of the underlying LLM (2606.20696). The work addresses three persistent obstacles in non-invasive language decoding: the absence of overt linguistic output during inner speech, limited training data (roughly 2,000 samples from five participants), and large inter-subject variability. Unlike prior sentence-decoding pipelines such as those of Tang et al. or BP-GPT, which require frame-accurate alignment among audio, tokens, and TRs, MindAlign operates on a decoupled representation-alignment design: a subject-specific encoder maps fMRI activity into a CLIP-guided semantic space, and a frozen multimodal LLM (LLaVA-1.6-vicuna-7B) is prompted via soft prefix tuning conditioned on both neural and visual inputs.
Position Relative to Prior Work
The paper's related-work survey distinguishes two dominant research routes. Image-reconstruction systems (MindEye, MindBridge, BrainCLIP) achieve strong visual decoding but are trained on the Natural Scenes Dataset and target image reconstruction rather than subject-generated language. Sentence-level systems (Tang et al., BP-GPT, BrainLLM) pursue open-vocabulary text but suffer from temporal-mismatch and error-compounding problems; BrainLLM's PCA compression from 50,000 to 1,000 voxels, for example, discards cortical topology and temporal dynamics. Multimodal captioning systems (BrainChat, MindLLM) typically rely on external image–text corpora as pseudo-labels, meaning the decoded text is not the participant's own language. Mind Captioning generates descriptive captions from fMRI but targets generic scene descriptions rather than internal verbalization.
MindAlign claims two distinguishing features: the decoding target is the participant's own inner speech (post-scan written descriptions), and the framework requires no audio–token–TR alignment. The paper identifies two gaps it aims to fill: reliance on externally provided captions rather than participant-generated language, and difficulty preserving subject-specific semantics.
Method
Stage 1 (cross-modal alignment) uses a two-layer Bi-LSTM (512 hidden units per direction) over 8-TR fMRI sequences, restricted to voxels selected by category-based explainable variance (EV) filtering. The encoder output is supervised with a hybrid objective combining InfoNCE contrastive loss, cosine similarity, MSE, and an auxiliary COCO category classification loss, using CLIP ViT-L/14 image embeddings as intermediate anchors and LLaVA input-layer embeddings (global-average-pooled over tokens) as linguistic targets. Regularization includes intra-category Mixup and spatially constrained noise injection.
Stage 2 (multimodal fusion generative decoding) constructs a soft prefix of length 8 in LLaVA's input embedding space. A neural stream (fMRI-derived 4096-d vector) and a visual stream (frozen CLIP image features) are each projected to half the prefix dimension, concatenated, and passed through a fusion MLP. Only the projection and fusion modules are trained; the LLaVA backbone remains frozen. Training uses teacher-forced cross-entropy with prefix positions masked at −100.
Dataset
The dataset comprises five Japanese participants (LD0001–LD0006, with LD0003 withdrawing) who silently described COCO images in Japanese during scanning at TR = 1 s. Each image trial involved four 8-second description segments (whole image plus three bounded objects), yielding 560 image trials over seven scanning days and approximately 2,000 usable samples. Ground-truth labels are post-scan written descriptions produced more than one month after scanning to reduce retrospective adjustment effects.
The paper validates these labels against overt speech collected outside the scanner: true inner-speech/spoken pairs yield a Sentence-BERT cosine of 0.7231 versus 0.0761 for random pairings (a 17.8% BERT-Score gain over the 0.6600 random baseline). This is an important assumption to note: the decoding target is a delayed written recall of inner speech, and the validation establishes semantic consistency with overt speech but cannot verify verbatim fidelity to what was verbalized in the scanner.
Main Results
The evaluation uses six metrics (TF-IDF cosine, ChrF, Levenshtein similarity, BLEU, ROUGE-L, BERT-Score) under a strictly subject-wise protocol. Key findings:
| Comparison |
Evidence |
Interpretation |
| fMRI+Image vs. Random fMRI+Image |
Real fMRI beats Gaussian-noise fMRI for all subjects (e.g., BERT-Score 0.7800 vs. 0.7364 for LD0006) |
Neural signals carry semantic information beyond visual priors |
| fMRI+Image vs. fMRI-only |
Consistent gains from adding the image (e.g., LD0005 ROUGE-L 0.3949 vs. 0.3230) |
Visual input anchors noisy neural semantics |
| Random fMRI+Image vs. original LLaVA |
Prefix tuning alone improves fit (0.7417 vs. 0.6945 for LD0002) |
Part of the gain reflects domain adaptation |
The central claim—that the model reads structured neural content rather than exploiting visual priors alone—rests on the real-vs-random fMRI ablation, and the consistent degradation under random neural input supports it. However, the paper also candidly notes a caveat in the qualitative analysis: the random-fMRI baseline produces fluent output because the decoder learns to ignore uninformative neural input and rely on image features, so fluency alone is not evidence of decoding.
A notable pattern is the divergence between lexical and semantic metrics: BLEU and ROUGE-L remain low (roughly 0.08–0.20 and 0.25–0.42, respectively), while BERT-Score is consistently high (0.72–0.78). The authors attribute this to gist-level paraphrasing rather than word-for-word reconstruction, which is arguably the appropriate characterization for brain-driven generation, but it also means the framework does not recover exact wording. Absolute lexical scores remain modest, and no statistical significance tests are reported for the subject-wise comparisons.
Category-level analysis shows fMRI+Image outperforming the random baseline in most COCO categories (e.g., bus: 0.796 vs. 0.711; chair: 0.779 vs. 0.741), with a small number of categories (broccoli) favoring the random control, indicating the gains, while moderate, are not uniformly attributable to neural signal.
Cross-Subject Transfer
The decoupled architecture enables a modular transfer experiment: a Stage-1 encoder from one subject paired with a Stage-2 projector trained on another. All off-diagonal BERT-Scores exceed 0.69, and some pairings exceed other subjects' within-subject performance (LD0006 encoder + LD0005 projector: 0.7371, above LD0001's within-subject 0.7215). This supports the claim that Stage 2 learns a subject-independent semantic-to-language mapping while Stage 1 acts as a subject-specific neural normalizer. The transfer is not lossless—within-subject performance is generally highest on the diagonal (up to 0.7683 for LD0006)—so subject-specific alignment remains necessary for best results, and the cross-subject matrix is evaluated without statistical testing.
Limitations and Open Questions
The paper concedes several limitations. Inter-subject variance is substantial (BERT-Score 0.7215 to 0.7800), attributed to BOLD SNR differences and variable task compliance in the subjective inner-speech task; the framework's success therefore depends on participants producing consistent, vivid mental verbalization, which cannot be directly verified. Stage-1 validation loss remains high (3.70 for LD0001), indicating that direct fMRI-to-embedding alignment under limited supervision is lossy, and the authors note this is expected for a representation module rather than a generative model. In the fMRI-only Stage-2 setting, validation loss bottoms out early (epoch 4) and then mildly degrades, suggesting limited generalization without visual anchoring. The dataset is small (five participants, ~2,000 samples), one participant withdrew, and the dataset release is stated as planned pending permissions rather than already available. Open questions include whether the semantic-to-language projection scales to larger cohorts, whether the delayed-written-recall target introduces systematic bias relative to in-scanner inner speech, and whether the framework extends beyond Japanese or beyond COCO-style stimuli.
Conclusion
MindAlign demonstrates that embedding-level alignment between fMRI activity and a frozen multimodal LLM's input space can support free-form decoding of participant-generated inner speech without temporal supervision or decoder fine-tuning. Its strongest empirical support is the consistent advantage of real over random neural input and the partial cross-subject transferability of the Stage-2 projector, which together indicate that fMRI signals modulate semantic content beyond image-driven priors. The results are strongest at the semantic (BERT-Score) level and modest at the lexical level, leaving exact-word recovery and full subject-agnostic decoding as open problems.
