Situated Question Answering (SQA)

• Situated Question Answering (SQA) is a task where answers depend on both linguistic input and explicit situational context, integrating multimodal sensory and environmental data.
• It spans diverse scenarios including 2D diagrams and 3D scenes, addressing spatial, temporal, and context-dependent challenges in applications like navigation or diagram interpretation.
• Current methodologies leverage probabilistic models and vision-language transformers, yet encounter challenges in spatial reasoning, egocentric viewpoint integration, and dynamic context adaptation.

Situated Question Answering (SQA) refers to the family of question answering tasks in which the correct answer depends not solely on linguistic input, but also on some explicit representation of “situation,” such as a physical or virtual environment, multimodal sensory data, or extra-linguistic context (e.g., the time and place where a question is asked). SQA tasks require models to integrate, ground, and reason over both linguistic and non-linguistic context, encompassing complex phenomena such as spatial relations, egocentric viewpoint, scenario adaptation, and the effects of world dynamics. The SQA paradigm is distinct from traditional QA in its explicit modeling of environmental, visual, or contextual grounding and its operationalization in both naturalistic and synthetic settings.

1. Formal Definitions and Problem Scope

The canonical SQA instance supplies as input a tuple $(q, s, e, A)$, where $q$ is a natural language question, $s$ a situation specification (such as agent pose, timestamp, or scenario text), $e$ an explicit representation of environment or context (e.g., image, 3D scene, temporal/geographical indicator), and $A$ the set of candidate answers or a method for open-answer generation.

The objective is to induce a mapping

$$a^* = \underset{a \in A}{\arg\max} \;\; P(a | q, s, e)$$

with the environment $e$ and situation $s$ varying by task. In SQA3D, $s = \langle s^{pos}, s^{ori}, s^{desc} \rangle$ (agent position, orientation, and textual description) and $e$ is a 3D scene; in “Semantic Parsing to Probabilistic Programs” (Krishnamurthy et al., 2016), $q$0 is a diagram with extracted structure; in SituatedQA (Zhang et al., 2021) $q$1 encodes time or place.

The critical distinction: the answer depends not only on question semantics but also on the represented situation and context—often requiring situated perception and joint reasoning.

2. Taxonomy and Dataset Landscape

SQA task formulations are heterogenous, with primary axes including:

• Modality of Situation/Environment:
  • 2D images or diagrams (P³, GeoSQA, science diagram QA) (Krishnamurthy et al., 2016, Huang et al., 2019).
  • 3D environments (SQA3D, MSQA, Multi-CLIP, cdViews) (Ma et al., 2022, Linghu et al., 2024, Delitzas et al., 2023, Wang et al., 28 May 2025).
  • Textual or event context (SituatedQA) (Zhang et al., 2021).
• Scope of Situational Context:
  • Egocentric pose or simulated agent location (Ma et al., 2022, Linghu et al., 2024).
  • Temporal or geographical context (Zhang et al., 2021).
  • Scenario-based case studies integrating background and domain knowledge (Huang et al., 2019).

Major public datasets are summarized below.

Dataset	Domain/Modality	Size	Context Type	Key Challenge
SQA3D	3D scenes (ScanNet)	33.4K QA pairs	3D pose/situation	Egocentric spatial reasoning
MSQA	3D scenes (multi-src)	251K QA pairs	Interleaved multi	Multi-modal, navigation
SituatedQA	Web open-retrieval	8.9K questions	Time/location	Context-sensitive answers
GeoSQA	Diagrams + text	4,110 QA	Scenario text/img	Scenario-specific adaptation
P³	Science diagrams	1,500 QA	Diagram structure	Joint vision/semantics

The research community has increasingly adopted large, richly annotated datasets to probe the situated and multi-modal capacities of contemporary AI systems (Ma et al., 2022, Linghu et al., 2024, Delitzas et al., 2023); however, the domain is characterized by an evolving landscape of modalities and situational abstractions.

3. Modeling Approaches and Architectural Innovations

3.1 Probabilistic and Compositional Models

Early SQA models, exemplified by P³ (Krishnamurthy et al., 2016), formalize SQA as semantic parsing into probabilistic programs. Each parse yields logical forms instantiated against environmental data with execution traces representing possible interpretations (due to visual ambiguity or environmental uncertainty). Feature-rich log-linear models score parses, leveraging both lexico-syntactic and vision-language features, and global graph-level constraints (e.g. cycle counts in food webs).

Inference proceeds by beam search over both semantic parses and their grounded executions, yielding

$q$2

where $q$3 encodes the parse and execution trace.

3.2 Vision-Language Transformers

Modern SQA in 3D scenes adopts a multi-stage pipeline uniting point cloud encoding (VoteNet, PointNet++), text encoding (CLIP, LLM subwords), cross-modal transformers for fusion, and specialized heads for answer selection and pose regression (Ma et al., 2022, Delitzas et al., 2023, Linghu et al., 2024). “Multi-CLIP” (Delitzas et al., 2023) aligns 3D scenes with both language and multi-view 2D image representations via InfoNCE contrastive pre-training, which improves downstream exact match and localization metrics on SQA3D. MSQA’s interleaved input design introduces sequence modeling over mixed tokens (text, image crops, point cloud objects) directly in a prefix-LM setup (Linghu et al., 2024).

3.3 2D Model Adaptation via View Selection

A contrasting approach, cdViews (Wang et al., 28 May 2025), circumvents explicit 3D reasoning by rendering 2D views from the 3D environment, selecting a maximally informative and diverse subset, and prompting frozen 2D vision-LLMs. Key contributions are the viewSelector, which ranks views by answer-relevance via cross-attention scoring, and viewNMS, which enforces spatial diversity over camera poses. This pipeline achieves or surpasses 3D-fusion models when deployed as a zero-shot system.

4. Evaluation Protocols and Benchmarks

SQA tasks employ domain-specific evaluation regimes. Metrics include:

• Classification Accuracy (EM@1): Proportion of correct top-1 predictions over all QA pairs (e.g., 47.2% on SQA3D, human ceiling 90.1%) (Ma et al., 2022).
• Answer Correctness Scores: Human or LLM-graded answer plausibility (e.g., MSQA uses averaged 1–5 ratings for situation/question/answer clarity) (Linghu et al., 2024).
• Localization Metrics: For pose prediction, accuracy within specified thresholds (e.g., [email protected], Acc@1m for position error; Acc@15° for orientation) (Ma et al., 2022, Delitzas et al., 2023).
• Navigation Benchmarks: For situated navigation, next-step accuracy against A*-computed optimal paths (Linghu et al., 2024).
• Context Sensitivity: In context-dependent web QA, exact match stratified by context type (temporal, geographical) and frequency (common vs. rare) (Zhang et al., 2021).

Notably, the performance gap between current models and human participants remains substantial (≈43 pp on SQA3D), especially in fine-grained spatial, commonsense, and multi-hop reasoning categories (Ma et al., 2022). In scenario-based SQA (GeoSQA), even strong retrieval and entailment models perform near random chance, indicating pronounced limitations of current architectures in grounding general knowledge to case-specific context (Huang et al., 2019).

5. Challenges, Failure Modes, and Insights

Principal challenges in SQA include:

• Situation Misunderstanding: Mis-localization or mis-interpretation of the agent’s pose or context leads to attention failures and erroneous inference over objects or relations (Ma et al., 2022).
• Spatial and Commonsense Reasoning: Multi-hop spatial queries, egocentric transformations, and contextual affordance judgments are often beyond the reach of current systems; question categories such as “What object lies between X and Y?” demand integration across modalities and abstractions (Ma et al., 2022, Linghu et al., 2024).
• Scenario Adaptation: Scenario-based SQA demands integrating retrieved domain knowledge with scenario particulars—existing NLI and reading comprehension models underperform when presented with this requirement (Huang et al., 2019).
• Contextual Dynamics: In SituatedQA, context-dependent questions exhibit answer drift over time and location; existing models, even when provided with up-to-date retrieval corpora, persistently lag on “updated” or “rare” contexts, and often return valid answers from incorrect contexts (Zhang et al., 2021).

A plausible implication is that effective SQA requires not only representational capacity for multi-modal fusion and situational grounding, but also explicit mechanisms for context disambiguation and dynamic knowledge integration.

6. Future Directions and Open Problems

Ongoing and proposed research directions include:

• Larger-Scale, More Diverse Multimodal Datasets: Expansion to synthetic and real scenes, additional situation types, and richer modalities (e.g., video, sensor streams) to improve domain robustness (Linghu et al., 2024).
• End-to-End Differentiable Models: Moving beyond feature-engineered and beam-search architectures to fully neural, gradient-based pipelines integrating parsing and grounding (Krishnamurthy et al., 2016).
• Human Feedback and RLHF: Incorporating reinforcement learning from human feedback into data and model training regimes to enhance QA quality and contextual appropriateness (Linghu et al., 2024).
• Cross-Modal and Spatial Reasoning Modules: Development of spatially aware attention blocks or graph-based neural modules to facilitate the fusion of text, images, and 3D structure (Delitzas et al., 2023, Linghu et al., 2024).
• Joint Learning Paradigms: Simultaneously learning vision and language components under weak QA supervision rather than explicit grounding annotation (Krishnamurthy et al., 2016).
• Dynamic Context Benchmarks: Periodic dataset and corpus updates, explicit context-annotation, and fine-grained evaluation (e.g., stable vs. updated, common vs. rare) for long-term benchmarking relevance (Zhang et al., 2021).

7. Comparative Table of Key SQA Benchmarks and Model Highlights

Task/Paper	Modality	Core Model Paradigm	Benchmark/Test Score	Main Challenge
P³ (Krishnamurthy et al., 2016)	Diagrams (2D)	CCG + probabilistic programs	48.7% (Dev)	Semantic parsing under vision uncertainty
SQA3D (Ma et al., 2022)	3D scenes	3D+language transformers	47.2% (Test, EM@1)	Egocentric & spatial reasoning
Multi-CLIP (Delitzas et al., 2023)	3D scenes	3D⇔2D⇔Text contrastive pretrain	48.02% (Test, EM@1)	Knowledge transfer from 2D CLIP
cdViews (Wang et al., 28 May 2025)	3D via 2D views	View selection + 2D LVLM	56.9% (Test, EM@1 SQA)	Zero-shot 2D model adaptation
MSQA (Linghu et al., 2024)	3D, Interleaved	Multimodal LM (Vicuna+LoRA)	56.5% overall (FT model)	Interleaved joint multimodal input
SituatedQA (Zhang et al., 2021)	Text+context	DPR/BART, context-enriched	23.0%/26.5% (DPR, Test)	Temporal/geographical context dependence
GeoSQA (Huang et al., 2019)	Diagrams + text	Retrieval/NLI/RC (various)	~26% (Textbook)	Scenario grounding, domain adaptation

These results highlight the persistent gap between specialized and generalist systems, and the necessity of models that couple domain knowledge, cross-modal perception, and dynamic situational reasoning.