SceneQA: Grounded Scene Question Answering

Updated 8 July 2026

SceneQA is a family of question answering tasks that relies on explicit scene representations, enabling grounded reasoning over objects, geometry, and spatial relations.
It spans diverse settings—from image-based scene graphs and 3D indoor reconstructions to video scenes—coupling language with structured spatial and temporal cues.
Benchmark analyses highlight challenges in object grounding, precise localization, and temporal alignment, driving research towards richer multimodal and context-aware models.

SceneQA denotes a family of question-answering problems in which answers are grounded in an explicit scene representation rather than inferred from isolated image regions or language priors alone. In the literature, the term spans at least three major settings: reasoning over structured scene graphs in images, scene-level question answering over full 3D reconstructions, and scene-centric reasoning over videos in which the relevant unit is a coherent scene rather than a single frame (Hildebrandt et al., 2020, Azuma et al., 2021, Yang et al., 5 Aug 2025). Across these settings, the common requirement is that language understanding be coupled to objects, attributes, relations, geometry, motion, or viewpoint, so that answering depends on grounded scene comprehension.

1. Scope, definitions, and naming

SceneQA is not a single benchmark or modality. In image-based work, it refers to question answering over a scene graph, represented as a directed multigraph whose entities and typed relations support explicit multi-hop reasoning (Hildebrandt et al., 2020). In 3D indoor work, it refers to question answering over an entire RGB-D reconstruction or point cloud, often with explicit grounding in 3D object boxes rather than only answer prediction (Azuma et al., 2021). In situated 3D work, the task further conditions answers on an agent’s position and orientation, so that the answer depends on an egocentric situation rather than only on global scene content (Ma et al., 2022). In recent video work, SceneQA has also been defined as scene-localized long-video question answering, where the model must localize the relevant scene, perceive fine-grained details within it, and then reason over that scene context (Yang et al., 5 Aug 2025).

A source of bibliographic ambiguity is the name “ScanQA.” “3D Question Answering” introduces a dataset called ScanQA with 10,062 QA pairs across 806 scenes and evaluates the transformer-based 3DQA-TR model (Ye et al., 2021). “ScanQA: 3D Question Answering for Spatial Scene Understanding” later formalizes 3D-QA as object-grounded scene-level question answering over whole RGB-D scans and reports 800 scenes, 41,363 questions, and 58,191 answers (Azuma et al., 2021). The two works are related in topic but not identical in dataset scale or task framing.

A recurring misconception is that SceneQA is merely VQA with larger visual context. The surveyed papers argue otherwise. Scene-graph formulations emphasize explicit object–relation traversal rather than monolithic image embeddings (Hildebrandt et al., 2020). Indoor 3D formulations emphasize full-scene geometry, occlusion handling, directional alignment, and box grounding beyond frame-centric 2D reasoning (Azuma et al., 2021). Situated 3D formulations require viewpoint inference from text, and long-video formulations replace keyframe retrieval with scene localization and within-scene reasoning (Ma et al., 2022, Yang et al., 5 Aug 2025).

2. Representational substrates and grounding mechanisms

The representational core of SceneQA varies by domain, but object-centric structure is persistent. In scene-graph reasoning for image QA, the scene graph is formalized as $SG \subset \mathcal{E} \times \mathcal{R} \times \mathcal{E}$ , with inverse-complete edges, hub connections, self-loops, and typed object, attribute, and relation nodes. Reasoning becomes path construction over this graph, and answer probabilities aggregate over trajectories that terminate at candidate answer nodes (Hildebrandt et al., 2020). This formulation makes interpretability a first-class property: answers are supported by traversable chains of objects, attributes, and relations.

In 3D indoor SceneQA, the dominant substrate is the full 3D scene. ScanQA defines the task as predicting an answer from a point cloud $p$ and question $q$ , with the model often required to localize the relevant objects through 3D proposals and bounding boxes, not merely classify an answer (Azuma et al., 2021). The corresponding box representation is standard, $B=(c,d,\theta)$ with center, size, and yaw, and training typically couples answer loss with object classification, localization confidence, and 3D detection losses. “3D Question Answering” similarly uses object proposals from a 3D detector, but separates appearance and geometry into dual encoders whose outputs are fused by a 3D-Linguistic BERT (Ye et al., 2021).

Situated formulations add an explicit situation variable. SQA3D represents each instance using a scene $S$ , pose $x=(p,R)\in SE(3)$ , situation text, and question, so that answering requires understanding not only what is in the scene but what is visible or relevant from the described egocentric pose (Ma et al., 2022). MSQA generalizes this further to an interleaved multi-modal tuple $(P,S,q,a)$ in which the scene point cloud, situation text-image description, location, orientation, and interleaved question jointly determine the answer through $A^*=\arg\max_A p_\theta(A\mid P,S,Q)$ (Linghu et al., 2024).

Video-centered SceneQA introduces new representational demands. LVSQA treats scenes, not frames, as the unit of understanding, and SLFG constructs scene-localized frame groups whose descriptions are abstracted to scene summaries and retrieved by cosine similarity against a question-side scene description (Yang et al., 5 Aug 2025). 4DP-QA adds explicit world-centric motion supervision: apparent motion is distinguished from “True-Motion Tracking,” in which a 3D trajectory is projected through a fixed reference camera so that camera motion is factored out (Cho et al., 10 Jun 2026). EgoTextVQA, by contrast, binds scene understanding to timestamped egocentric video and OCR-like scene-text evidence, with the target answer conditioned on video content visible up to the question time (Zhou et al., 11 Feb 2025).

A systems-oriented formulation appears in the 3D Queryable Scene Representation framework, which unifies panoptic reconstruction, segmented point clouds, and 3D scene graphs under shared object identifiers so that natural-language queries can retrieve geometric, visual, and semantic information from the same scene representation (Li et al., 24 Sep 2025). This suggests a convergence between benchmark-driven SceneQA and queryable world models for robotic task planning.

3. Benchmark landscape

The benchmark ecosystem now spans indoor scans, situated embodied settings, autonomous driving, city-scale point clouds, egocentric video, long video, and fine-grained traffic reasoning. Representative scales and task emphases are summarized below.

Benchmark or formulation	Scale	Distinguishing property
ScanQA	800 scenes, 41,363 questions, 58,191 answers	Whole-scene 3D QA with 3D object grounding (Azuma et al., 2021)
SQA3D	650 scenes, $\sim$ 6.8k situations, $\sim$ 33.4k questions	Egocentric situated reasoning from pose-conditioned scene context (Ma et al., 2022)
MSQA / MSNN	251K QA pairs across 1,734 scenes; 34K navigation samples across 378 scenes	Interleaved text-image-point-cloud inputs and situated next-step navigation (Linghu et al., 2024)
City-3DQA	193 city scenes, 450k QA pairs, 6 cities	City-scale 3D multimodal QA with scene semantics and human-environment interaction (Sun et al., 2024)
NuScenes-QA / NuScenes-MQA / NuScenes-SpatialQA	459,941 QA pairs; 1,459,933 markup QAs; $p$ 03.5M spatial QA pairs	Autonomous-driving SceneQA with multimodal, markup, and spatial-reasoning variants (Qian et al., 2023, Inoue et al., 2023, Tian et al., 4 Apr 2025)
EgoTextVQA	1,507 videos, 7,064 questions	Egocentric scene-text-aware video QA under question timestamps (Zhou et al., 11 Feb 2025)
LVSQA / 4DP-QA / FGTR-Bench	100 videos and 500 QA pairs; 400K training + 2.2K benchmark; 40,236 MCQs + 4,947 blind test	Scene-localized long-video QA, 4D motion QA, and fine-grained traffic evidence evaluation (Yang et al., 5 Aug 2025, Cho et al., 10 Jun 2026, Xiu et al., 5 Jul 2026)

The indoor 3D branch remains foundational. ScanQA explicitly couples answer prediction with 3D grounding over whole RGB-D indoor scans and argues that explicit grounding curbs reliance on textual priors (Azuma et al., 2021). SQA3D shifts the problem from third-person scene understanding to situated egocentric reasoning, and the resulting human–model gap is correspondingly larger (Ma et al., 2022). MSQA scales this situated formulation through automatically constructed scene graphs, vision-language attribute extraction, and interleaved inputs (Linghu et al., 2024).

Autonomous-driving SceneQA has diversified into several subfamilies. NuScenes-QA emphasizes programmatically generated scene-graph questions over camera and LiDAR inputs (Qian et al., 2023). NuScenes-MQA uses markup-embedded answers to evaluate caption generation and QA simultaneously, with tags such as <obj>, <cnt>, <ans>, <cam>, <dst>, and <loc> (Inoue et al., 2023). NuScenes-SpatialQA isolates spatial understanding and reasoning through a 3D scene-graph pipeline that computes qualitative and quantitative relations from LiDAR ground truth (Tian et al., 4 Apr 2025). City-3DQA extends the problem from roadside scenes to city-scale point clouds, long-range spatial relations, and human–environment interaction semantics (Sun et al., 2024).

Video variants now expose limitations that are not visible in static benchmarks. EgoTextVQA focuses on reading and reasoning over scene text in dynamic first-person videos (Zhou et al., 11 Feb 2025). LVSQA argues that long-video QA should localize and reason over coherent scenes rather than retrieve isolated frames (Yang et al., 5 Aug 2025). 4DP-QA introduces motion-centric supervision that disentangles camera and object motion (Cho et al., 10 Jun 2026). FGTR-Bench, although single-image and multiple-choice, recasts fine-grained traffic question answering around localized critical evidence and same-scene distractors (Xiu et al., 5 Jul 2026).

4. Model families and learning paradigms

One major SceneQA family reasons over explicit graphs. In “Scene Graph Reasoning for Visual Question Answering,” a GAT encodes scene-graph neighborhoods, a Transformer encodes the question, and an RL agent uses an LSTM history state to walk from a hub node to an answer node under REINFORCE. The state is $p$ 1, the action space consists of outgoing relation–target pairs, and the terminal reward is $p$ 2 if the final node is the answer and $p$ 3 otherwise (Hildebrandt et al., 2020). This design prioritizes path interpretability and relational compositionality.

A second family is object-centric 3D fusion. ScanQA starts from VoteNet proposals, encodes the question with GloVe and a bidirectional LSTM, and uses MCAN-inspired transformer fusion to produce question-aware proposal features and a fused descriptor for both localization and answer prediction. The overall objective is explicitly joint, $p$ 4, reflecting the premise that answering and grounding should share representation learning (Azuma et al., 2021). “3D Question Answering” uses a dual-stream alternative: geometry and appearance are encoded separately, object-level spatial embeddings are derived from normalized box descriptors, and fusion is handled by a 3D-Linguistic BERT operating over language tokens and 3D object elements (Ye et al., 2021).

A third family reframes SceneQA as generation rather than classification. Gen3DQA replaces fixed-vocabulary answer classification with a transformer encoder–decoder that generates free-form answers from 3D scene context and the question. Training proceeds in stages: SoftGroup proposal pretraining, cross-entropy training for answer generation plus localization, then reinforcement learning on sentence-level CIDEr with a pragmatic VQG reward that scores how well the generated answer permits reconstruction of the original question (Dwedari et al., 2023). The technical claim is not merely that sequence generation is possible, but that direct optimization on language rewards improves answer quality and context-awareness in 3D QA.

A fourth family augments scene representations with structured semantics at larger scales. Sg-CityU constructs a city-level scene graph over point-cloud instances, encodes it with GCN layers, fuses graph, point-cloud, and question features through stacked self- and cross-attention, and predicts an answer by cross-entropy over a constrained answer set (Sun et al., 2024). MSR3D, designed for MSQA, uses object-centric point-cloud tokens, a Spatial Transformer over inter-object relations, interleaved 2D image tokens, and a situation-normalization strategy that rotates and translates the scene so that the agent is at the origin and faces the positive $p$ 5-axis (Linghu et al., 2024).

Temporal SceneQA has produced distinct architectural responses. SLFG is explicitly training-free and operates outside the base MLLM: frames are sampled, grouped, captioned, abstracted into scene-level summaries, ranked by cosine similarity, and then dynamically reorganized before being fed to an unchanged long-video model (Yang et al., 5 Aug 2025). 4DP-QA, by contrast, adds supervision rather than only preprocessing: training data includes both screen-space tracks and fixed-reference true-motion tracks, which teach models to reason about motion in world-centric rather than purely image-centric coordinates (Cho et al., 10 Jun 2026).

Recent safety-critical traffic work shifts attention from global scene priors to localized evidence. TSR-MLLM inserts a Text-Guided Small-Object Focus module at the decoder boundary of Qwen3-VL-4B, computes query–vision salience, selects a sparse Top- $p$ 6 subset of vision slots, and applies gated residual updates only to those slots while leaving text tokens unchanged (Xiu et al., 5 Jul 2026). The accompanying diagnosis—“critical evidence dilution”—extends SceneQA from a question of what representations are available to a question of how query-conditioned evidence is amplified before deep decoding.

5. Evaluation methodology and empirical trends

Evaluation protocols differ sharply across SceneQA variants, reflecting different views of what it means to answer correctly. ScanQA combines exact-match answer metrics with caption-style metrics and explicit 3D localization metrics. On the test split with object annotations, the ScanQA model reports $p$ 7, $p$ 8, $p$ 9, $q$ 0, $q$ 1, $q$ 2, $q$ 3, and $q$ 4, while on the test split without object annotations it reports $q$ 5, $q$ 6, and $q$ 7 (Azuma et al., 2021). The same work reports that random image sampling plus MCAN trails 3D full-scene reasoning, whereas an oracle image setup can rival or surpass some metrics, underscoring the role of perfect object identification.

Generative 3D SceneQA changes both training and evaluation emphasis. Gen3DQA does not report EM@1 or EM@10 because it generates answer sequences rather than classifying from a fixed answer vocabulary. On ScanQA, it reaches $q$ 8 on the test set with object IDs and $q$ 9 without object IDs, with validation localization $B=(c,d,\theta)$ 0 (Dwedari et al., 2023). The ablations indicate that localization training improves generation quality and that the VQG-based pragmatic reward yields consistent gains.

Situated and multi-modal benchmarks show that SceneQA remains far from solved even when the scene is static. SQA3D reports that the best evaluated model reaches 47.20% overall accuracy, while amateur human participants reach 90.06% (Ma et al., 2022). MSQA replaces exact match with a GPT-based correctness score and reports that the best MSR3D setting achieves an overall score of 56.48 on MSQA, while MSQA pretraining improves MSNN next-step navigation accuracy up to 48.4 (Linghu et al., 2024). These results imply that explicit situation modeling improves both question answering and action selection, but the agent-centric grounding problem remains difficult.

In outdoor and city-scale settings, the numerical picture is mixed. City-3DQA reports $B=(c,d,\theta)$ 1 and $B=(c,d,\theta)$ 2 on the sentence-wise split, and $B=(c,d,\theta)$ 3 and $B=(c,d,\theta)$ 4 on the city-wise split, with a minimal city-wise robustness gap (Sun et al., 2024). NuScenes-SpatialQA shows that qualitative spatial understanding is materially better than quantitative metric grounding: SpatialRGPT reaches 59.79% average qualitative accuracy, but LLaVA-v1.6-Mistral-7B leads the quantitative setting with 35.5% tolerance-based accuracy and MAE 33.3, while chain-of-thought prompting often degrades performance (Tian et al., 4 Apr 2025). The implication is that current VLMs are more reliable at categorical relation judgments than at calibrated metric estimation.

Video SceneQA exposes a different failure profile. On EgoTextVQA, even the strongest evaluated model, Gemini 1.5 Pro, reaches 33.4% accuracy outdoors and 34.4% indoors under the benchmark’s semantic-judge protocol, while humans achieve 43.1% and 27.7% respectively on a subset (Zhou et al., 11 Feb 2025). Auxiliary OCR-like scene text yields the largest single improvement, but text alone is insufficient. On long-video SceneQA, SLFG improves LLaVA-Video on LVSQA from 59.8 to 63.4 and improves LLaVA-OneVision from 56.2 to 61.0, indicating that scene-localized grouping is more effective than treating frame selection as the primary problem (Yang et al., 5 Aug 2025). On 4DP-QA-Bench, fine-tuning on 4DP-QA raises Qwen2.5-VL-7B from 46.6% to 84.3% and NVILA-Lite-8B from 42.3% to 84.4%, showing that explicit motion-grounded supervision transfers strongly to dynamic SceneQA (Cho et al., 10 Jun 2026).

Fine-grained traffic evidence benchmarks further reveal that strong average VLMs can still fail on localized safety-critical cues. On FGTR-Bench, TSR-MLLM reaches 74.1% overall on the blind test, exceeding the strongest matched 4B baseline by 2.1 points, with particularly clear gains on Nighttime Signal and Holistic Sign tracks (Xiu et al., 5 Jul 2026). This does not replace the broader SceneQA literature, but it sharpens a specific empirical conclusion: scene-level reasoning quality can be bottlenecked by failure to amplify tiny, question-relevant evidence.

6. Limitations, recurring failure modes, and research directions

Several limitations recur across otherwise different SceneQA formulations. The first is representational bottleneck. Scene-graph QA benefits from explicit structure, but the authors note that scene graph quality is the weakest part of the pipeline when graphs are generated automatically (Hildebrandt et al., 2020). In 3D QA, grounding helps, but localization accuracy remains modest and multi-object questions remain difficult; ScanQA error analyses identify misinterpretation of relations such as “corner” and “between,” mislocalization under occlusion, and lexical confusions such as “seat” $B=(c,d,\theta)$ 5 “table” (Azuma et al., 2021). Gen3DQA similarly notes that proposal quality caps the utility of target-conditioned signals (Dwedari et al., 2023).

The second limitation is ambiguity induced by viewpoint, phrasing, or scene scale. SQA3D formalizes this through situation descriptions, where omitting the situation produces measurable degradation (Ma et al., 2022). MSQA argues that interleaved text-only situation descriptions are insufficiently disambiguating and that mixing images with text is necessary for some situated references (Linghu et al., 2024). City-3DQA highlights city-scale sparsity, long-range relations, and usage-label ambiguity, while NuScenes-SpatialQA shows that even strong VLMs remain unreliable on quantitative distance, angle, and offset estimation (Sun et al., 2024, Tian et al., 4 Apr 2025).

The third limitation is temporal grounding. EgoTextVQA finds that current MLLMs often fail to align the relevant evidence with the question timestamp, and the same question can require different answers at different times (Zhou et al., 11 Feb 2025). LVSQA argues that keyframe-centric retrieval is misaligned with real long-video understanding, which instead requires scene localization, detail perception, and reasoning within coherent temporal context (Yang et al., 5 Aug 2025). 4DP-QA adds that many video models still entangle camera and object motion because they learn screen-space trajectories rather than world-centric motion (Cho et al., 10 Jun 2026).

The fourth limitation is excessive reliance on scene priors. This issue appears in different guises: ScanQA introduces explicit 3D grounding to curb reliance on textual priors (Azuma et al., 2021); FGTR-Bench frames the problem as “critical evidence dilution,” where background tokens dominate small but decisive cues (Xiu et al., 5 Jul 2026). A plausible implication is that future SceneQA systems will need both richer scene representations and stricter control over how questions gate evidence selection.

Current research directions in the literature are correspondingly diverse. Proposed directions include stronger 3D–2D fusion, scene-graph and multimodal pretraining adapted to 3D objects and point clouds, end-to-end differentiable box regression tied to language, hierarchical or dynamic scene graphs for large-scale environments, richer temporal and embodied reasoning, joint VQA–VQG training, open-vocabulary answer generation, and broader multimodal integration including OCR, audio, radar, and video memory (Azuma et al., 2021, Dwedari et al., 2023, Sun et al., 2024, Zhou et al., 11 Feb 2025). Across these proposals, the underlying trend is consistent: SceneQA is moving from answer prediction over static percepts toward grounded reasoning over structured, situated, and temporally coherent scene models.