Do sentence captions recruit the same neural populations as viewing natural scenes?

Determine whether reading the sentence captions corresponding to the Natural Scenes Dataset images recruits the same neural populations that are engaged during visual scene viewing of those images, specifically the cortical regions identified via searchlight representational similarity analysis as encoding visually evoked responses during natural scene viewing.

Background

The paper shows that similarity judgements for images and their corresponding sentence captions converge behaviorally and predict similar patterns of visually evoked fMRI activity in mid- and high-level visual cortex. However, the fMRI data analyzed involve only visual scene viewing; no fMRI was collected during sentence reading by the same participants for the same stimuli. Thus, while linguistic similarity judgments align with brain activity elicited by viewing images, it remains unresolved whether reading the captions elicits activity in the same cortical populations identified during visual scene viewing.

The authors note related work suggesting cross-modal decoding between vision and language and partial convergence of representations, but emphasize that directly testing sentence-reading fMRI with these specific stimuli and analyses is necessary to address whether the same neural populations are recruited.