Brain Question Answering (BQA)

Updated 5 July 2026

Brain Question Answering (BQA) is a field using brain signals and imaging to generate answers, combining neural decoding with visual grounding.
It encompasses distinct modalities such as fMRI-based decoding, MRI visual question answering, and brain-inspired text QA pipelines tailored to clinical and cognitive tasks.
Recent models like BrainChat and Brain-IT-VQA demonstrate innovative methodologies and promising accuracy, while highlighting challenges in visual grounding and data alignment.

Searching arXiv for papers on Brain Question Answering and adjacent benchmarks/frameworks. Brain Question Answering (BQA) denotes a family of question-answering tasks in which the conditioning signal is brain-related rather than conventional web text alone. In current research usage, the term covers at least two distinct formulations: answering questions from neural activity, typically fMRI acquired while a subject views natural images, and answering questions about brain images, especially 3D MRI volumes, with clinically grounded visual question answering protocols (Huang, 2024, Beliy et al., 28 May 2026, Abbasi et al., 19 May 2026). A separate, earlier lineage uses “brain” in a brain-inspired modeling sense rather than as the data source: Brain2Text is a modular machine-comprehension system for Russian text QA trained with the Informational Neurobayesian Approach on small corpora (Artemov et al., 2018). This suggests that BQA is an umbrella term whose precise meaning depends on whether the primary object of inference is neural representation, brain imaging, or a brain-inspired QA pipeline.

1. Conceptual scope and task formulations

Across contemporary work, BQA is usually formalized as conditional answer generation or answer selection from a question and a brain-derived input. In multimodal brain-imaging benchmarks, the task can be written as a function $f_\theta:(\{I_k\},q)\rightarrow a$ , where $\{I_k\}$ comprises one or more brain-related images and $q$ is a natural-language question; correctness is determined by exact agreement with a clinician-validated answer label (Peng et al., 2 Nov 2025). In fMRI-based BQA, the input is instead a neural measurement vector or volume, such as $x\in\mathbb{R}^V$ or $b\in\mathbb{R}^V$ , together with the question, and the target is an answer sequence or short answer class (Beliy et al., 28 May 2026, Huang, 2024).

Formulation	Primary input	Representative sources
Brain-signal BQA	fMRI + question	(Huang, 2024, Beliy et al., 28 May 2026, Lu et al., 28 May 2026)
Brain-imaging BQA	Brain MRI or multimodal brain images + question	(Abbasi et al., 19 May 2026, Ghosh et al., 16 May 2026, Vepa et al., 30 Sep 2025, Peng et al., 2 Nov 2025)
Brain-inspired text QA	Text document + question	(Artemov et al., 2018)

The distinction is not merely terminological. fMRI-based BQA is fundamentally a neural decoding problem constrained by low signal-to-noise ratio, individual variability, and the mismatch between population-level hemodynamic signals and symbolic linguistic output (Beliy et al., 28 May 2026). MRI-grounded BQA is instead a clinically situated VQA problem in which the central difficulty is faithful 3D or multi-sequence visual grounding, often under severe shortcut risk and with answer spaces shaped by radiological conventions (Abbasi et al., 19 May 2026, Ghosh et al., 16 May 2026). Brain2Text occupies a third position: it is a seven-module pipeline for machine comprehension of text that uses analytically computed information-theoretic weights rather than back-propagation, and its relevance is historical and conceptual rather than neuroimaging-specific (Artemov et al., 2018).

2. Decoding answers from neural activity

The first explicit implementation of fMRI question answering in the provided literature is BrainChat, which combines Masked Brain Modeling, contrastive alignment to the CoCa vision-language embedding space, and a generative Brain Decoder with cross-attention from text to fMRI embeddings (Huang, 2024). Its encoder operates on preprocessed fMRI volumes partitioned into non-overlapping patches, applies a 75% masking ratio during self-supervised pretraining, and minimizes an MSE reconstruction loss before moving to multimodal alignment and caption-style generation. For question answering, the system encodes the textual prompt “Question: <Q> Answer:” and autoregressively generates answer tokens without access to the original image. On VQA-style fMRI QA, BrainChat reports accuracy of 0.405–0.417 across ten settings, with best 0.417 for CoCa-Large with no pretraining (Huang, 2024).

Brain-IT-VQA extends this line by making the intermediate neural representation more structured and by introducing a more controlled benchmark. The model maps an fMRI vector with approximately 40k selected voxels into $K=128$ Brain Tokens with embedding dimension $d=768$ , applies self-attention over Brain Tokens, and then uses two question-conditioned cross-attention branches: a CLIP-aligned pathway that produces visual-token embeddings and a direct-conditioning pathway that produces soft prompt tokens, whose average is prepended to a frozen LLM (Beliy et al., 28 May 2026). Token generation follows the usual autoregressive form $P(w_t\mid z,w_{<t})=\mathrm{softmax}(g(z,w_{<t}))$ , with scaled dot-product attention used for the cross-modal fusion blocks. Under this design, Brain-IT-VQA reaches 56.95% VQA-v2 accuracy for subject 1 and 73.78 ± 0.92% short-answer accuracy on NSD-VQA, significantly outperforming MindLLM in 14/23 categories by paired bootstrap testing with 10,000 samples at $p<0.05$ (Beliy et al., 28 May 2026).

Mind-Omni reframes fMRI BQA as one task inside a unified discrete-diffusion system for seven encoding and decoding tasks. Its Brain Tokenizer converts continuous fMRI into a sequence of $L=64$ discrete tokens using a codebook of $\{I_k\}$ 0 entries with dimension $\{I_k\}$ 1, trained by vector quantization, semantic alignment, and perceptual alignment losses (Lu et al., 28 May 2026). BQA instruction tuning uses 58K reasoning question-answer pairs derived from NSD-linked image-text data, and the target answer tokens are generated under a masked discrete-diffusion objective conditioned on brain tokens and question tokens. On the reported multi-subject BQA benchmark, Mind-Omni records BLEU1 23.18, METEOR 50.13, ROUGE 52.91, SPICE 43.28, CLIPScore 70.65, RefCLIPScore 76.72, and LLM-Judge 24.37%, outperforming OneLLM on METEOR, ROUGE, SPICE, CLIPScore, and LLM-Judge while rivaling UMBRAE despite being self-contained and 442M in size (Lu et al., 28 May 2026).

A distinctive feature of the fMRI BQA literature is its interpretive ambition. Brain-IT-VQA explicitly treats BQA as a tool for studying neural representations rather than only as a predictive benchmark. Its masking-based attribution analysis partitions 40k voxels into 128 functional clusters and estimates cluster-level contributions to question-category accuracy via ridge regression over random masking experiments. The reported pattern is category-specific: object-presence questions implicate areas such as EBA, OPA, and FFA; food questions are more distributed across ventral visual cortex; and fine-grained color decoding depends on specialized clusters in the V4 complex (Beliy et al., 28 May 2026). This makes BQA relevant to cognitive neuroscience as well as to brain-computer interfacing.

3. Question answering about brain images

In clinical neuroimaging, BQA is largely instantiated as VQA over 3D MRI. NeuroQA is the broadest benchmark in the supplied corpus: 56,953 QA pairs from 12,977 unique subjects across 12 datasets, spanning ages 5–104 and five clinical domains—Alzheimer’s, Parkinson’s, tumors, white matter disease, and neurodevelopment (Abbasi et al., 19 May 2026). Every item is paired with a full 3D volume rather than a 2D slice, and the benchmark covers 11 clinically grounded reasoning skills distributed over 203 templates. Of these templates, 131 are image-grounded and answerable from a three-plane viewer, whereas 72 are image-informed and require FreeSurfer volumetry or external clinical instruments such as CDR or UPDRS. Answers are distributed across Yes/No, four-option multiple choice, and open-ended formats, with open text scored by exact match and token-level $\{I_k\}$ 2 (Abbasi et al., 19 May 2026).

UCSF-PDGM-VQA narrows the problem to neuro-oncology. It contains 2,387 closed-ended QA pairs from 473 pre-operative glioma MRI studies in the public UCSF-PDGM collection, with each study containing on average 23 distinct imaging series and approximately 151 slices per series (Ghosh et al., 16 May 2026). The questions are grounded in four core sequences—T1 pre-contrast, T1 post-contrast, T2, and T2 FLAIR—and target tumor size and segmentation, anatomical localization, mass effect and midline shift, enhancement pattern and vascularity, and edema and surrounding tissue changes. Longitudinal questions were excluded in this first release. The paper positions the benchmark as a clinically relevant stress test for VLMs under multi-sequence, three-dimensional MRI interpretation (Ghosh et al., 16 May 2026).

mpLLM focuses on multiparametric 3D brain MRI and introduces a prompt-conditioned hierarchical mixture-of-experts architecture designed to fuse multiple interrelated 3D modalities without image-report pretraining (Vepa et al., 30 Sep 2025). To address sparse paired supervision, it derives synthetic but clinically motivated QA pairs from segmentation annotations using rule-based measurements of volume, region, shape, and spread, then paraphrases 15 base templates with approximately 3,000 ChatGPT-4o rewrites per template. The reported datasets comprise GLI with 38,904 QA pairs across 1,621 scans, MET with 11,718 QA pairs across 651 scans, and GoAT with 24,318 QA pairs across 1,351 scans, all using T1, T1Gd, T2, and FLAIR at 1 mm³ resampled to $\{I_k\}$ 3 (Vepa et al., 30 Sep 2025).

OmniBrainBench generalizes beyond MRI alone and frames BQA as a full clinical-continuum benchmark across 15 brain imaging modalities collected from 30 verified medical sources, yielding 9,527 validated QA pairs and 31,706 images (Peng et al., 2 Nov 2025). It organizes evaluation into five clinical phases and 15 specialized tasks, ranging from anatomical structure identification and modality recognition to disease diagnosis reasoning, risk stratification, treatment plan selection, and postoperative outcome assessment. All questions follow a standardized five-option multiple-choice format constructed by rule-based option generation together with GPT-5 distractor creation (Peng et al., 2 Nov 2025). A plausible implication is that the field is moving from narrow lesion description toward workflow-level evaluation.

4. Benchmark construction, shortcut control, and evaluation methodology

A central methodological concern in BQA is whether a system answers from the brain-derived input or from textual priors. NeuroQA makes this issue explicit through answer-distribution refinement and an image-grounding protocol. In its raw template pool, text-only baselines achieved more than 80% closed-format accuracy, revealing exploitable priors (Abbasi et al., 19 May 2026). Stage 5 refinement therefore removes templates with more than 95% answer dominance, downsamples templates with 70–95% dominance, enforces 50/50 Yes/No balance per template per dataset, and redistributes multiple-choice positions to a uniform 25%. The resulting closed-format text-only accuracy drops to 44.6%, only 5.1 percentage points above the 39.5% random-chance floor, and the benchmark defines a Shortcut Score

$\{I_k\}$ 4

with $\{I_k\}$ 5 at text-only performance and $\{I_k\}$ 6 indicating performance below the floor (Abbasi et al., 19 May 2026).

NeuroQA also formalizes image necessity through three stress tests. A model is certified image-grounded only if image-absent accuracy falls toward chance, the fabrication rate of confident visual claims without image remains near zero, and the hallucination rate of visual claims contradicting FreeSurfer ground truth remains near zero (Abbasi et al., 19 May 2026). This protocol responds to a recurrent problem in medical VQA benchmarks: strong nominal accuracy can coexist with weak visual dependence.

Comparable control logic appears in fMRI BQA. NSD-VQA was created precisely because prior image-fMRI VQA datasets offered only a few broad and weakly controlled questions per image (Beliy et al., 28 May 2026). The benchmark instead provides approximately 20 question-answer pairs per image across 20 controlled categories, with template-based generation used to isolate visual attributes, balanced answer distributions in which no answer dominates more than 70% of instances, and a minimum support of at least 50 examples per question type. Counts were double-checked with Gemma-4-31B-it and synonyms were merged via embedding clustering (Beliy et al., 28 May 2026). This controlled design is intended to support both more reliable decoding evaluation and more interpretable neuroscience analysis.

UCSF-PDGM-VQA and OmniBrainBench emphasize clinical validation and filtering. In UCSF-PDGM-VQA, GPT-4o initially generates up to 20 QA pairs per report, GPT-5.2 rescored answerability and flagged ambiguities, a keyword filter removes postsurgical, recurrent, or non-brain concepts, and a neuroradiology fellow independently vetted the final 75 pairs in the spot-check stage (Ghosh et al., 16 May 2026). OmniBrainBench applies rule-based brain-only screening, GPT-5 reformulation, clustering-based de-duplication, and final radiologist validation, while all raw 3D volumes are slice-selected in three orthogonal planes under board-certified radiologist guidance (Peng et al., 2 Nov 2025). Across both lines, benchmark engineering is treated as a primary research problem rather than as a preprocessing footnote.

5. Empirical findings, ceilings, and recurrent failure modes

The most consistent empirical result in brain-imaging BQA is that contemporary models often fail to demonstrate genuine visual grounding. On NeuroQA Test-Public closed-format items, random performance is 39.5%, the text-only majority-per-template baseline is 49.4%, the supervised multi-task 3D CNN reaches 43.7%, and the best zero-shot VLM, Gemini-3.1-Pro, reaches 47.5%; no evaluated model surpasses either the 49.4% text-only floor or the clinician visual reference (Abbasi et al., 19 May 2026). Two clinicians independently scoring 100 frozen items on a three-plane NIfTI viewer achieved 46.7% and 51.1%, for a mean of 48.9% with Cohen’s $\{I_k\}$ 7, and the gap was concentrated in image-informed categories such as Reasoning and Severity as well as in complex Location questions (Abbasi et al., 19 May 2026). The benchmark therefore exposes both model weakness and a nontrivial perceptual ceiling for humans when forced to answer from imaging alone.

UCSF-PDGM-VQA sharpens this critique through the language of modality collapse. The best reported model scores are 63.57% for MedGemma-1.5 with multi-slice input, 63.20% for Lingshu-32B with montage input, and 59.14% for LLaVA-Med-1.5, against a neuroradiology fellow upper bound of 87.88% on a 75-question subset (Ghosh et al., 16 May 2026). More strikingly, some VLMs perform better with blank images than with the actual MRI, as in the reported 66% versus 61% pattern for Lingshu and LLaVA-Med, while the text-only Qwen3-8B baseline attains 52.57% on the full dataset—roughly twice random—and answer-order reshuffling causes 10–20% drops due to positional bias (Ghosh et al., 16 May 2026). These are direct indications that zero-shot multiple-choice accuracy can be dominated by priors, formatting effects, or answer-distribution cues.

OmniBrainBench reaches a similar conclusion at a broader clinical scale. Among 24 MLLMs, the best proprietary model, Gemini-2.5-Pro, achieves 66.58% overall accuracy, whereas the physician reference reaches 91.35% (Peng et al., 2 Nov 2025). The strongest open-source and medical-specialized models remain materially below physician performance, and the hardest deficits occur in complex preoperative and prognostic tasks: the highest reported model score in Risk Stratification is approximately 40.8%, and Preoperative Assessment tops out at approximately 47.0% (Peng et al., 2 Nov 2025). The benchmark explicitly interprets this as a visual-to-clinical reasoning gap.

The fMRI BQA literature reports a different performance profile. Brain-IT-VQA shows that coarse semantics and binary judgments are substantially easier to decode than fine-grained attributes: scene and person Yes/No categories reach 93% accuracy, whereas color is 48%, food 54%, and action 66%, with counting at 71.6% and spatial position at 73.6% (Beliy et al., 28 May 2026). This supports the claim that fMRI robustly preserves broad semantic and scene-level information while leaving fine-grained color, pose, and counting information relatively weak. BrainChat’s lower VQA-style accuracy range of 0.405–0.417 illustrates the difficulty of open generative QA from brain signals without the stronger structural constraints introduced by later models and benchmarks (Huang, 2024).

6. Limitations, controversies, and future directions

Several limitations recur across the literature. NeuroQA notes a T1-only limitation for white matter hyperintensity signal items, acknowledges that some Measurement templates are deliberately beyond human visual estimation and are tagged human_assessable = false, and states that residual text priors cannot be entirely removed without discarding clinically meaningful items (Abbasi et al., 19 May 2026). UCSF-PDGM-VQA is restricted to a single institution, single anatomy, single disease family, single preoperative time point, and closed-ended QA, while mpLLM notes the absence of demographic metadata in its synthetic protocol and explicitly reserves autonomous decision-making claims (Ghosh et al., 16 May 2026, Vepa et al., 30 Sep 2025). OmniBrainBench broadens modality coverage but still evaluates predominantly multiple-choice behavior rather than extended longitudinal interaction or full report synthesis (Peng et al., 2 Nov 2025).

A major controversy concerns what should count as success. One tradition emphasizes answer accuracy, sometimes augmented by BLEU, METEOR, CIDEr, SPICE, CLIPScore, or LLM-as-Judge metrics in generative settings (Lu et al., 28 May 2026). Another insists that accuracy without demonstrated grounding is insufficient, hence NeuroQA’s stress tests and UCSF-PDGM-VQA’s blank-image and reshuffle ablations (Abbasi et al., 19 May 2026, Ghosh et al., 16 May 2026). This suggests that BQA evaluation is bifurcating into two requirements: task performance and evidence that the answer is causally tied to the relevant brain-derived input.

Future directions are correspondingly diverse. In clinical MRI BQA, proposed next steps include multimodal T1+T2+FLAIR sub-tracks for white matter disease, broader clinician studies with at least five board-certified neuroradiologists over 250 or more items, longitudinal-focused sub-benchmarks, open-ended report generation, treatment planning, confidence calibration, and selective prediction (Abbasi et al., 19 May 2026, Ghosh et al., 16 May 2026). In fMRI BQA, the agenda includes dynamic stimuli such as video, more abstract queries involving intent or emotion, cross-subject transfer, higher-field fMRI or intracranial recordings, and integration of EEG, MEG, or ECoG into unified tokenized frameworks (Beliy et al., 28 May 2026, Lu et al., 28 May 2026). BrainChat adds a distinct ethical register, emphasizing privacy, misuse risk associated with unintended “mind-reading,” and the need for clinical trials before augmentative communication deployment (Huang, 2024). Collectively, these directions indicate that BQA is evolving from benchmark construction toward a broader program of grounded neural decoding, clinically faithful visual reasoning, and safety-aware human-AI collaboration.