SceneCritic: A Symbolic Evaluator for 3D Indoor Scene Synthesis

Abstract: LLMs and Vision-LLMs (VLMs) increasingly generate indoor scenes through intermediate structures such as layouts and scene graphs, yet evaluation still relies on LLM or VLM judges that score rendered views, making judgments sensitive to viewpoint, prompt phrasing, and hallucination. When the evaluator is unstable, it becomes difficult to determine whether a model has produced a spatially plausible scene or whether the output score reflects the choice of viewpoint, rendering, or prompt. We introduce SceneCritic, a symbolic evaluator for floor-plan-level layouts. SceneCritic's constraints are grounded in SceneOnto, a structured spatial ontology we construct by aggregating indoor scene priors from 3D-FRONT, ScanNet, and Visual Genome. SceneOnto traverses this ontology to jointly verify semantic, orientation, and geometric coherence across object relationships, providing object-level and relationship-level assessments that identify specific violations and successful placements. Furthermore, we pair SceneCritic with an iterative refinement test bed that probes how models build and revise spatial structure under different critic modalities: a rule-based critic using collision constraints as feedback, an LLM critic operating on the layout as text, and a VLM critic operating on rendered observations. Through extensive experiments, we show that (a) SceneCritic aligns substantially better with human judgments than VLM-based evaluators, (b) text-only LLMs can outperform VLMs on semantic layout quality, and (c) image-based VLM refinement is the most effective critic modality for semantic and orientation correction.

• The paper presents SceneCritic, a symbolic evaluator that operates directly on structured floor-plan layouts, bypassing unstable VLM-based methods.
• The approach leverages a comprehensive spatial ontology built from multiple datasets to assess semantic coherence, orientation correctness, and overlap detection.
• The system demonstrates high human-alignment with up to 94% agreement on easy scenes, providing transparent and detailed error localization.

SceneCritic: Towards Robust Symbolic Evaluation in 3D Indoor Scene Synthesis

Introduction and Motivation

Evaluation of 3D indoor scene layouts generated by LLM- or VLM-driven systems remains fundamentally limited by reliance on visual rendering and model-based judges. These approaches, such as VLM scorers, are sensitive to rendering viewpoint, prompt specification, and inherent model instability, introducing variance and hallucination that undermines both reproducibility and fidelity to human judgments. "SceneCritic: A Symbolic Evaluator for 3D Indoor Scene Synthesis" (2604.13035) addresses this critical gap by proposing SceneCritic, a symbolic and dataset-grounded evaluator that operates directly on structured floor-plan layouts. The system eschews rendered 2D or 3D views, instead leveraging semantic and geometric priors extracted from large-scale 3D and vision-language datasets to provide object- and relationship-level assessments with transparent error localization.

SceneCritic Ontology Construction

SceneCritic constructs a comprehensive spatial ontology by aggregating and reconciling statistics from three major datasets—3D-FRONT (for synthetic, well-annotated layouts), ScanNet (for real-world RGBD reconstructions), and Visual Genome (for image-based object relationships). The ontology embeds four key relational priors:

• Object dimensions (width, height, depth, with full percentile distributions and counts)
• Support relations (surface on which objects rest, extracted via geometric reasoning and filtered annotations)
• Co-occurrence statistics (conditional probabilities, normalized PMI, and room-type conditional statistics)
• Orientation preferences (back-to-wall, faces-center, and pairwise face-to-object orientation, all measured with angular statistics)

Evaluations proceed by traversing per-room-type relational graphs, whose nodes represent objects and whose edges encode the learned priors.

Figure 1: Relational sub-graph for three room types, with node frequencies and max conditional co-occurrence; dotted lines indicate categories shared across room types.

Figure 2: (Left) Object dimension distributions; (Center) 73.6% of object-pair edges have positive semantic association; (Right) Variance in canonical orientation strategies by object class.

Such a construction ensures evaluator extensibility: new datasets can be integrated by mapping category-level statistics, immediately supporting unseen room types and objects.

Symbolic Evaluation Pipeline

SceneCritic structurally verifies scene layouts along three canonical axes, each directly grounded in the constructed ontology:

• Semantic Coherence: Validates object combinations, functional groupings, scales within learned physical bounds, co-occurrence proximity, and total object completeness. Formally, the system calculates for each pair if joint placement is statistically plausible, requires spatial proximity, or is missing essential groupings.
• Orientation Correctness: Validates placement orientation using yaw-based angular checks for canonical placement (e.g., back-to-wall for beds, faces-center for TVs).
• Overlap and Collision: Separates coarse (AABB) and precise (OBB, using SAT) detection of placement violations, incorporating both proximity and rotation-aware spatial overlap.

Evaluation is both fine-grained and interpretable: every object and relationship is assigned a verdict with exact constraint violation information, rather than black-box scene-level scores.

Limitations of VLM-Based Evaluation

Empirical analysis demonstrates that VLM-based evaluation is fundamentally unstable:

• Output scene scores vary dramatically depending on rendered viewpoint (see Figure 3).
• Human-aligned judgments are not recoverable via prompt engineering or repeated runs, as model outputs fluctuate due to occlusion, projection ambiguity, and prompt phrasing.
• VLMs occasionally hallucinate or miss prominent geometric violations—in some instances, assigning higher scores to unacceptably erroneous layouts when viewed from a masking viewpoint.

Figure 3: VLM evaluation yields inconsistent scores for different views; SceneCritic provides stable, violation-localized assessment directly over the object layout.

Quantitatively, SceneCritic demonstrates up to 94% agreement with human annotators on easy scenes, compared to VLMs' 58%. On complex layouts, SceneCritic still maintains 83% agreement, with VLMs dropping to near random selection (47%). In all cases, disagreement magnitude is smaller for SceneCritic than VLMs, and critical overlap violations go undetected by vision-based models (see (2604.13035), Table 2).

Human Alignment and Numerical Results

Human studies were conducted using 16 annotators with almost 600 judgments. Key findings include:

• Agreement with human judgments (Easy / Complex): SceneCritic 94.44% / 83.33%; VLM 58.82% / 47.06%.
• Perfect detection of overlap: SceneCritic achieves 100% agreement on overlap errors, verifying its geometric reasoning.
• Score alignment: Method rankings from VLMs reverse by viewpoint and fail to correlate with human preferences; SceneCritic scores robustly reflect semantic, geometric, and orientation plausibility as seen by annotators.

Figure 4: SceneCritic diagnoses detailed errors (e.g., desks not backed to walls) and highlights semantic, orientation, and spatial violations in extended scenes.

Probing Refinement and Critic Modalities

The paper proposes a refinement testbed enabling scene generation models to undergo iterative improvement with feedback from either heuristic, text (LLM), or image (VLM) critics. Key observations:

• Text-only LLMs can outperform VLMs on semantic layout reasoning, especially when models have strong reasoning-oriented RL post-training (e.g., DeepSeek-3.2V achieves the highest semantic scores on both heuristic and LLM-based feedback, exceeding proprietary VLMs on average).
• Orientation remains challenging for all methods; math-oriented GRPO objectives yield superior orientation scores even in smaller parameter models.
• Image-based VLM refinement is most effective for correcting placement and orientation errors, especially as the size and complexity of scenes grows. The combination of image and text may degrade performance, as textual cues can dominate.
• Refinement trajectories: SceneCritic's stepwise feedback allows detailed study of error correction, plateaus, and trade-offs (e.g., fixing one overlap may induce semantic implausibility elsewhere).

Figure 5: Selected hyperparameter choices in SceneCritic evaluation align with human judgments, while alternatives result in systematic evaluation biases.

Implications and Future Directions

SceneCritic offers a paradigm shift for evaluation in 3D scene synthesis. The symbolic, explicable, and dataset-grounded framework provides:

• Trustworthy, decomposable, and extensible evaluation: Essential for tracking progress in large-scale generative pipelines and multi-agent reasoning or simulation platforms.
• Diagnostics for generation failures: Design of new architectures and training pipelines is informed by constraint-level failure modes supplied by SceneCritic.
• Immediate extensibility: Integration of new domains, room types, or object categories is a matter of dataset mapping, not retraining.
• Benchmarking post-training strategies: SceneCritic enables comparison across RLHF, AIF, and verifiable reward learning strategies, elucidating which objectives yield genuine reasoning improvements.

The future of spatial evaluation likely lies in tight integration of symbolic priors with neural score prediction, potentially enabling differentiable feedback loops within generative models—SceneCritic’s ontology and evaluation axes provide a model-agnostic substrate for this direction.

Conclusion

SceneCritic institutionalizes robust, transparent, and human-aligned evaluation for 3D scene layout generation. By grounding symbolic evaluation in aggregated, real-world statistics and explicitly modeling semantic, geometric, and orientation constraints, SceneCritic surpasses the stability and diagnostic depth of both LLM and VLM judges, and enables granular benchmarking across critic modalities, model architectures, and post-training strategies. The approach is immediately extensible and geared toward the practical demands of large-scale embodied AI, simulation, and interactive generation systems (2604.13035).