---
title: Brain Question Answering (BQA)
url: https://www.emergentmind.com/topics/brain-question-answering-bqa
type: topic
---

# Brain Question Answering (BQA)

Searching arXiv for recent papers on Brain Question Answering and adjacent benchmarks/frameworks.
Brain Question Answering (BQA) denotes a family of question-answering tasks in which the conditioning signal is brain-related rather than conventional web text alone. In current research usage, the term covers at least two distinct formulations: answering questions *from* neural activity, typically fMRI acquired while a subject views natural images, and answering questions *about* brain images, especially 3D MRI volumes, with clinically grounded visual question answering protocols [2406.07584], [2605.29588], [2605.20525]. A separate, earlier lineage uses “brain” in a brain-inspired modeling sense rather than as the data source: Brain2Text is a modular machine-comprehension system for Russian text QA trained with the Informational Neurobayesian Approach on small corpora [1804.00551]. This suggests that BQA is an umbrella term whose precise meaning depends on whether the primary object of inference is neural representation, brain imaging, or a brain-inspired QA pipeline.

## 1. Conceptual scope and task formulations

Across contemporary work, BQA is usually formalized as conditional answer generation or answer selection from a question and a brain-derived input. In multimodal brain-imaging benchmarks, the task can be written as a function $f_\theta:(\{I_k\},q)\rightarrow a$, where $\{I_k\}$ comprises one or more brain-related images and $q$ is a natural-language question; correctness is determined by exact agreement with a clinician-validated answer label [2511.00846]. In fMRI-based BQA, the input is instead a neural measurement vector or volume, such as $x\in\mathbb{R}^V$ or $b\in\mathbb{R}^V$, together with the question, and the target is an answer sequence or short answer class [2605.29588], [2406.07584].

| Formulation | Primary input | Representative sources |
|---|---|---|
| Brain-signal BQA | fMRI + question | [2406.07584], [2605.29588], [2605.29591] |
| Brain-imaging BQA | Brain MRI or multimodal brain images + question | [2605.20525], [2605.17140], [2509.25889], [2511.00846] |
| Brain-inspired text QA | Text document + question | [1804.00551] |

The distinction is not merely terminological. fMRI-based BQA is fundamentally a neural decoding problem constrained by low signal-to-noise ratio, individual variability, and the mismatch between population-level hemodynamic signals and symbolic linguistic output [2605.29588]. MRI-grounded BQA is instead a clinically situated VQA problem in which the central difficulty is faithful 3D or multi-sequence visual grounding, often under severe shortcut risk and with answer spaces shaped by radiological conventions [2605.20525], [2605.17140]. Brain2Text occupies a third position: it is a seven-module pipeline for machine comprehension of text that uses analytically computed information-theoretic weights rather than back-propagation, and its relevance is historical and conceptual rather than neuroimaging-specific [1804.00551].

## 2. Decoding answers from neural activity

The first explicit implementation of fMRI question answering in the provided literature is BrainChat, which combines Masked Brain Modeling, contrastive alignment to the CoCa vision-language embedding space, and a generative Brain Decoder with cross-attention from text to fMRI embeddings [2406.07584]. Its encoder operates on preprocessed fMRI volumes partitioned into non-overlapping patches, applies a 75% masking ratio during self-supervised pretraining, and minimizes an MSE reconstruction loss before moving to multimodal alignment and caption-style generation. For question answering, the system encodes the textual prompt “Question: \<Q\> Answer:” and autoregressively generates answer tokens without access to the original image. On VQA-style fMRI QA, BrainChat reports accuracy of 0.405–0.417 across ten settings, with best 0.417 for CoCa-Large with no pretraining [2406.07584].

Brain-IT-VQA extends this line by making the intermediate neural representation more structured and by introducing a more controlled benchmark. The model maps an fMRI vector with approximately 40k selected voxels into $K=128$ Brain Tokens with embedding dimension $d=768$, applies self-attention over Brain Tokens, and then uses two question-conditioned cross-attention branches: a CLIP-aligned pathway that produces visual-token embeddings and a direct-conditioning pathway that produces soft prompt tokens, whose average is prepended to a frozen language model [2605.29588]. Token generation follows the usual autoregressive form $P(w_t\mid z,w_{<t})=\mathrm{softmax}(g(z,w_{<t}))$, with scaled dot-product attention used for the cross-modal fusion blocks. Under this design, Brain-IT-VQA reaches 56.95% VQA-v2 accuracy for subject 1 and 73.78 ± 0.92% short-answer accuracy on NSD-VQA, significantly outperforming MindLLM in 14/23 categories by paired bootstrap testing with 10,000 samples at $p<0.05$ [2605.29588].

Mind-Omni reframes fMRI BQA as one task inside a unified discrete-diffusion system for seven encoding and decoding tasks. Its Brain Tokenizer converts continuous fMRI into a sequence of $L=64$ discrete tokens using a codebook of $K=128$ entries with dimension $D=16$, trained by vector quantization, semantic alignment, and perceptual alignment losses [2605.29591]. BQA instruction tuning uses 58K reasoning question-answer pairs derived from NSD-linked image-text data, and the target answer tokens are generated under a masked discrete-diffusion objective conditioned on brain tokens and question tokens. On the reported multi-subject BQA benchmark, Mind-Omni records BLEU1 23.18, METEOR 50.13, ROUGE 52.91, SPICE 43.28, CLIPScore 70.65, RefCLIPScore 76.72, and LLM-Judge 24.37%, outperforming OneLLM on METEOR, ROUGE, SPICE, CLIPScore, and LLM-Judge while rivaling UMBRAE despite being self-contained and 442M in size [2605.29591].

A distinctive feature of the fMRI BQA literature is its interpretive ambition. Brain-IT-VQA explicitly treats BQA as a tool for studying neural representations rather than only as a predictive benchmark. Its masking-based attribution analysis partitions 40k voxels into 128 functional clusters and estimates cluster-level contributions to question-category accuracy via ridge regression over random masking experiments. The reported pattern is category-specific: object-presence questions implicate areas such as EBA, OPA, and FFA; food questions are more distributed across ventral visual cortex; and fine-grained color decoding depends on specialized clusters in the V4 complex [2605.29588]. This makes BQA relevant to cognitive neuroscience as well as to brain-computer interfacing.

## 3. Question answering about brain images

In clinical neuroimaging, BQA is largely instantiated as VQA over 3D MRI. NeuroQA is the broadest benchmark in the supplied corpus: 56,953 QA pairs from 12,977 unique subjects across 12 datasets, spanning ages 5–104 and five clinical domains—Alzheimer’s, Parkinson’s, tumors, white matter disease, and neurodevelopment [2605.20525]. Every item is paired with a full 3D volume rather than a 2D slice, and the benchmark covers 11 clinically grounded reasoning skills distributed over 203 templates. Of these templates, 131 are image-grounded and answerable from a three-plane viewer, whereas 72 are image-informed and require FreeSurfer volumetry or external clinical instruments such as CDR or UPDRS. Answers are distributed across Yes/No, four-option multiple choice, and open-ended formats, with open text scored by exact match and token-level $F_1$ [2605.20525].

UCSF-PDGM-VQA narrows the problem to neuro-oncology. It contains 2,387 closed-ended QA pairs from 473 pre-operative glioma MRI studies in the public UCSF-PDGM collection, with each study containing on average 23 distinct imaging series and approximately 151 slices per series [2605.17140]. The questions are grounded in four core sequences—T1 pre-contrast, T1 post-contrast, T2, and T2 FLAIR—and target tumor size and segmentation, anatomical localization, mass effect and midline shift, enhancement pattern and vascularity, and edema and surrounding tissue changes. Longitudinal questions were excluded in this first release. The paper positions the benchmark as a clinically relevant stress test for VLMs under multi-sequence, three-dimensional MRI interpretation [2605.17140].

mpLLM focuses on multiparametric 3D brain MRI and introduces a prompt-conditioned hierarchical mixture-of-experts architecture designed to fuse multiple interrelated 3D modalities without image-report pretraining [2509.25889]. To address sparse paired supervision, it derives synthetic but clinically motivated QA pairs from segmentation annotations using rule-based measurements of volume, region, shape, and spread, then paraphrases 15 base templates with approximately 3,000 ChatGPT-4o rewrites per template. The reported datasets comprise GLI with 38,904 QA pairs across 1,621 scans, MET with 11,718 QA pairs across 651 scans, and GoAT with 24,318 QA pairs across 1,351 scans, all using T1, T1Gd, T2, and FLAIR at 1 mm³ resampled to $32\times256\times256$ [2509.25889].

OmniBrainBench generalizes beyond MRI alone and frames BQA as a full clinical-continuum benchmark across 15 brain imaging modalities collected from 30 verified medical sources, yielding 9,527 validated QA pairs and 31,706 images [2511.00846]. It organizes evaluation into five clinical phases and 15 specialized tasks, ranging from anatomical structure identification and modality recognition to disease diagnosis reasoning, risk stratification, treatment plan selection, and postoperative outcome assessment. All questions follow a standardized five-option multiple-choice format constructed by rule-based option generation together with GPT-5 distractor creation [2511.00846]. A plausible implication is that the field is moving from narrow lesion description toward workflow-level evaluation.

## 4. Benchmark construction, shortcut control, and evaluation methodology

A central methodological concern in BQA is whether a system answers from the brain-derived input or from textual priors. NeuroQA makes this issue explicit through answer-distribution refinement and an image-grounding protocol. In its raw template pool, text-only baselines achieved more than 80% closed-format accuracy, revealing exploitable priors [2605.20525]. Stage 5 refinement therefore removes templates with more than 95% answer dominance, downsamples templates with 70–95% dominance, enforces 50/50 Yes/No balance per template per dataset, and redistributes multiple-choice positions to a uniform 25%. The resulting closed-format text-only accuracy drops to 44.6%, only 5.1 percentage points above the 39.5% random-chance floor, and the benchmark defines a Shortcut Score
$$
SS = 1 - \frac{Acc_{\mathrm{model}} - Acc_{\mathrm{textonly}}}{100 - Acc_{\mathrm{textonly}}},
$$
with $SS=1$ at text-only performance and $SS>1$ indicating performance below the floor [2605.20525].

NeuroQA also formalizes image necessity through three stress tests. A model is certified image-grounded only if image-absent accuracy falls toward chance, the fabrication rate of confident visual claims without image remains near zero, and the hallucination rate of visual claims contradicting FreeSurfer ground truth remains near zero [2605.20525]. This protocol responds to a recurrent problem in medical VQA benchmarks: strong nominal accuracy can coexist with weak visual dependence.

Comparable control logic appears in fMRI BQA. NSD-VQA was created precisely because prior image-fMRI VQA datasets offered only a few broad and weakly controlled questions per image [2605.29588]. The benchmark instead provides approximately 20 question-answer pairs per image across 20 controlled categories, with template-based generation used to isolate visual attributes, balanced answer distributions in which no answer dominates more than 70% of instances, and a minimum support of at least 50 examples per question type. Counts were double-checked with Gemma-4-31B-it and synonyms were merged via embedding clustering [2605.29588]. This controlled design is intended to support both more reliable decoding evaluation and more interpretable neuroscience analysis.

UCSF-PDGM-VQA and OmniBrainBench emphasize clinical validation and filtering. In UCSF-PDGM-VQA, GPT-4o initially generates up to 20 QA pairs per report, GPT-5.2 rescored answerability and flagged ambiguities, a keyword filter removes postsurgical, recurrent, or non-brain concepts, and a neuroradiology fellow independently vetted the final 75 pairs in the spot-check stage [2605.17140]. OmniBrainBench applies rule-based brain-only screening, GPT-5 reformulation, clustering-based de-duplication, and final radiologist validation, while all raw 3D volumes are slice-selected in three orthogonal planes under board-certified radiologist guidance [2511.00846]. Across both lines, benchmark engineering is treated as a primary research problem rather than as a preprocessing footnote.

## 5. Empirical findings, ceilings, and recurrent failure modes

The most consistent empirical result in brain-imaging BQA is that contemporary models often fail to demonstrate genuine visual grounding. On NeuroQA Test-Public closed-format items, random performance is 39.5%, the text-only majority-per-template baseline is 49.4%, the supervised multi-task 3D CNN reaches 43.7%, and the best zero-shot VLM, Gemini-3.1-Pro, reaches 47.5%; no evaluated model surpasses either the 49.4% text-only floor or the clinician visual reference [2605.20525]. Two clinicians independently scoring 100 frozen items on a three-plane NIfTI viewer achieved 46.7% and 51.1%, for a mean of 48.9% with Cohen’s $\kappa=0.40$, and the gap was concentrated in image-informed categories such as Reasoning and Severity as well as in complex Location questions [2605.20525]. The benchmark therefore exposes both model weakness and a nontrivial perceptual ceiling for humans when forced to answer from imaging alone.

UCSF-PDGM-VQA sharpens this critique through the language of modality collapse. The best reported model scores are 63.57% for MedGemma-1.5 with multi-slice input, 63.20% for Lingshu-32B with montage input, and 59.14% for LLaVA-Med-1.5, against a neuroradiology fellow upper bound of 87.88% on a 75-question subset [2605.17140]. More strikingly, some VLMs perform better with blank images than with the actual MRI, as in the reported 66% versus 61% pattern for Lingshu and LLaVA-Med, while the text-only Qwen3-8B baseline attains 52.57% on the full dataset—roughly twice random—and answer-order reshuffling causes 10–20% drops due to positional bias [2605.17140]. These are direct indications that zero-shot multiple-choice accuracy can be dominated by priors, formatting effects, or answer-distribution cues.

OmniBrainBench reaches a similar conclusion at a broader clinical scale. Among 24 MLLMs, the best proprietary model, Gemini-2.5-Pro, achieves 66.58% overall accuracy, whereas the physician reference reaches 91.35% [2511.00846]. The strongest open-source and medical-specialized models remain materially below physician performance, and the hardest deficits occur in complex preoperative and prognostic tasks: the highest reported model score in Risk Stratification is approximately 40.8%, and Preoperative Assessment tops out at approximately 47.0% [2511.00846]. The benchmark explicitly interprets this as a visual-to-clinical reasoning gap.

The fMRI BQA literature reports a different performance profile. Brain-IT-VQA shows that coarse semantics and binary judgments are substantially easier to decode than fine-grained attributes: scene and person Yes/No categories reach 93% accuracy, whereas color is 48%, food 54%, and action 66%, with counting at 71.6% and spatial position at 73.6% [2605.29588]. This supports the claim that fMRI robustly preserves broad semantic and scene-level information while leaving fine-grained color, pose, and counting information relatively weak. BrainChat’s lower VQA-style accuracy range of 0.405–0.417 illustrates the difficulty of open generative QA from brain signals without the stronger structural constraints introduced by later models and benchmarks [2406.07584].

## 6. Limitations, controversies, and future directions

Several limitations recur across the literature. NeuroQA notes a T1-only limitation for white matter hyperintensity signal items, acknowledges that some Measurement templates are deliberately beyond human visual estimation and are tagged `human_assessable = false`, and states that residual text priors cannot be entirely removed without discarding clinically meaningful items [2605.20525]. UCSF-PDGM-VQA is restricted to a single institution, single anatomy, single disease family, single preoperative time point, and closed-ended QA, while mpLLM notes the absence of demographic metadata in its synthetic protocol and explicitly reserves autonomous decision-making claims [2605.17140], [2509.25889]. OmniBrainBench broadens modality coverage but still evaluates predominantly multiple-choice behavior rather than extended longitudinal interaction or full report synthesis [2511.00846].

A major controversy concerns what should count as success. One tradition emphasizes answer accuracy, sometimes augmented by BLEU, METEOR, CIDEr, SPICE, CLIPScore, or LLM-as-Judge metrics in generative settings [2605.29591]. Another insists that accuracy without demonstrated grounding is insufficient, hence NeuroQA’s stress tests and UCSF-PDGM-VQA’s blank-image and reshuffle ablations [2605.20525], [2605.17140]. This suggests that BQA evaluation is bifurcating into two requirements: task performance and evidence that the answer is causally tied to the relevant brain-derived input.

Future directions are correspondingly diverse. In clinical MRI BQA, proposed next steps include multimodal T1+T2+FLAIR sub-tracks for white matter disease, broader clinician studies with at least five board-certified neuroradiologists over 250 or more items, longitudinal-focused sub-benchmarks, open-ended report generation, treatment planning, confidence calibration, and selective prediction [2605.20525], [2605.17140]. In fMRI BQA, the agenda includes dynamic stimuli such as video, more abstract queries involving intent or emotion, cross-subject transfer, higher-field fMRI or intracranial recordings, and integration of EEG, MEG, or ECoG into unified tokenized frameworks [2605.29588], [2605.29591]. BrainChat adds a distinct ethical register, emphasizing privacy, misuse risk associated with unintended “mind-reading,” and the need for clinical trials before augmentative communication deployment [2406.07584]. Collectively, these directions indicate that BQA is evolving from benchmark construction toward a broader program of grounded neural decoding, clinically faithful visual reasoning, and safety-aware human-AI collaboration.

Source: https://www.emergentmind.com/topics/brain-question-answering-bqa