SceneTeract: 3D Scene Verification

• SceneTeract is a verification framework that decomposes natural language instructions into atomic actions and evaluates their feasibility in 3D environments.
• It employs a two-stage pipeline integrating semantic planning with Vision-Language Models and geometric grounding using predicate checks like navigability and reachability.
• Empirical results demonstrate its effectiveness in diagnosing agent affordance gaps and enhancing task accuracy through reward-based VLM post-training.

SceneTeract is a verification framework for 3D scene functionality, designed to bridge high-level semantic reasoning with explicit geometric and physical simulation under agent-specific constraints. SceneTeract systematically decomposes natural language activities into atomic actions, performing multi-level validation to assess each action's functional feasibility in a given 3D scene and for a specified agent embodiment. The approach identifies not only the feasibility verdict per action but also exposes detailed failure diagnostics, providing a fine-grained understanding of agentic affordances and grounding gaps in current Vision-LLMs (VLMs) (Maillard et al., 31 Mar 2026).

1. Architectural Overview

SceneTeract operates as a two-stage verification pipeline that integrates:

• Semantic Planning: A Vision-LLM (VLM), denoted Φ, parses a target activity T (free-form language; e.g., "make a cup of tea then watch TV"), an agent profile A (explicit embodiment parameters), and a 3D scene S (geometry G and objects O) into an ordered sequence of atomic actions π = [(a⁽¹⁾, o⁽¹⁾), ..., (a⁽ᴷ⁾, o⁽ᴷ⁾)]. Actions are drawn from a fixed primitive library, grouped into four families: Mobility, Contact, Handling, Perception (see Table 1 of (Maillard et al., 31 Mar 2026)).
• Geometric Grounding: For each (action, object) pair, a geometric verifier Ψ checks a sequence of agent-aware predicates (navigability, reachability, interactability, clearance, visibility). Each step t yields a boolean v⁽ᵗ⁾ and a per-predicate failure reason ε⁽ᵗ⁾, assembled into a diagnostic trace R (Algorithm 1).

This architecture enables hierarchical property checks, with short-circuiting to terminate early if fundamental constraints (e.g., navigation) fail.

2. Mathematical and Algorithmic Foundations

2.1 Agent Embodiment

The agent profile A = ⟨w_clear, h_eye, h_arm, r_arm, m_loco⟩ specifies clearance requirements (w_clear), eye-level (h_eye), shoulder height (h_arm), reach radius (r_arm), and locomotion mode (walk or wheel).

2.2 Geometric Reasoning Suite

• Navigability: Constructs a binary navigation map (M) by projecting floors/obstacles, applying w_clear-based erosion, and extracting connected components. Navigability is determined by the existence of a path from the agent’s current position to an interaction zone Z_o associated with the target object (Algorithms 2–3). Formally, is_Navigable_To(o) = ∃x ∈ Z_o such that L[x] = ℓ_agent.
• Reachability: For the agent shoulder point s and the shifted object geometry G_o^shifted, reachability is satisfied if d_min = min_{p ∈ P, q ∈ G_o^shifted}‖p−q‖ ≤ r_arm. Both standing and crouch heights are checked (Algorithm 4).
• Interactability: Multi-view segmentation identifies handles or contact points using Molmo (a point-prompt VLM), SAM for mask extraction, and 3D deprojection. Reachability is then reassessed on relevant contact regions (Algorithm 5).
• Clearance: For Handling actions, a 3D box (V_clear) is attached to the object, and checked for collisions using trimesh, ensuring no occlusion with walls/objects and containment within room bounds (Algorithm 6).
• Visibility: Simulates agent eye positions and projects rays to object keypoints. Visibility holds if the fraction of clear rays surpasses threshold τ = 0.15 (Algorithm 7).

Predicates are selectively applied per action family as per Table 2.

3. Evaluation Procedures and Diagnostic Reporting

Algorithm 1 (GroundPlan) processes the action-object sequence π, consulting property requirements for each atomic action and collating verdicts with diagnostic messages. The evaluation pipeline outputs, for each action, a tuple (a⁽ᵗ⁾, o⁽ᵗ⁾, v⁽ᵗ⁾, ε⁽ᵗ⁾), as well as a global verdict v = ∧_t v⁽ᵗ⁾.

Atomic actions that fail record specific failure reasons (e.g., insufficient reach, blocked path, lack of clearance), supporting both aggregate statistics and qualitative scene debugging.

4. Empirical Results and Functional Affordance Analysis

4.1 Scene Auditing on Synthetic Environments

On a dataset of 1,132 3D-FRONT living/dining rooms (across three agent profiles), SceneTeract performed 3,396 verifications:

Agent Profile	Task Succ. (%)	Navigability (%)	Reachability (%)	Interactability (%)	Clearance (%)	Visibility (%)
Adult	59.0	84.7	91.9	75.6	67.3	~98
Child	66.0	89.2	93.7	73.4	66.3	~98
Wheelchair	42.5	73.0	80.7	64.1	66.6	~98

Frequent failure modes included blocked hallways (navigability), unreachable high shelves (reachability), obstructed handles (interactability), and cluttered object fronts (clearance) (Fig. 5).

4.2 Vision-LLM Functional Reasoning

Using SceneTeract-generated ground truth, several proprietary (Gemini-3 Flash, Gemini-3.1 Pro, Claude 4.6 Sonnet) and open (Gemma 3 12B/4B, Qwen3 VL-8B/4B, Ministral 3 3B) VLMs were evaluated in two modes:

• Direct: holistic task-level yes/no.
• Decomposed: step-wise yes/no for each atomic action.

Principal metrics (Table 4) included action/task accuracy, false positive rate (FP), Matthews Correlation Coefficient (MCC), Consistency, Horizon Stability Index (HSI), and Inclusivity Gap (InGap).

Key findings:

• Direct mode yields low task accuracy (54–63%) and high FP (>20%), resulting in low MCC (often <0.2, sometimes negative).
• Decomposition improves FP and MCC substantially (e.g. Gemini-3 Flash: FP 35.7→12.0, MCC 0.144→0.390).
• Substantial Inclusivity Gaps (5–20%) persist, even in strong models, indicating agent-profile-sensitivity.
• Some models (Claude 4.6) exhibit high internal consistency (>80%), suggesting emergent stepwise decomposition.
• Semantic competence alone is consistently insufficient for predicting physical feasibility.

5. Reward-Shaping and VLM Post-Training

SceneTeract serves as a reward engine for policy optimization, facilitating distillation of geometric constraints into VLM reasoning. Using Group Relative Policy Optimization (GRPO):

• Base Model: Qwen3-VL-4B-Instruct, tuned with LoRA (q_proj, v_proj, r=8, α=32).
• Training Dataset: Decomposed atomic-action set (80/20 scene split), batch=12, G=12, steps=2,800, lr=2e-5.
• Reward Structure:

r_format (+0.5 correct tag use, –1.0 on “prompt-echo”), r_correct (+2.0 True, +3.0 False predictions), r_spatial (+1.0 if negative and geometric failure cited).

The group-relative advantage computes as A_i = (r_i – μ_r)/(σ_r+ε).

Post-training evaluation demonstrates marked improvements:

Metric	Pre (Decomposed)	Post (Decomposed)
MCC	0.172	0.364
FP	22.8%	14.1%
Task Accuracy	61.4%	69.2%
Consistency	~98%	~98%

Qualitative inspection of model > traces reveals increased use of geometric detail (“too far by 0.15 m,” “narrow path”), indicating internalization of SceneTeract's constraints.

6. Significance, Applications, and Outlook

SceneTeract provides a rigorous, interpretable framework for evaluating functional affordances in 3D environments under rich agent constraints, yielding insights unattainable by pure semantic or perceptual models. The diagnostic granularity offered by per-action, per-predicate failure traces facilitates synthetic scene auditing, accessibility validation, and design feedback.

As a benchmarking tool, SceneTeract exposes systematic grounding mismatches across leading VLMs, demonstrating that semantic understanding alone remains insufficient for robust embodied reasoning. Its role as a reward engine for VLM post-training results in measurable gains in physical grounding, suggesting scalable pathways for integrating geometric reasoning into large vision-LLMs.

A plausible implication is that SceneTeract’s approach—explicit decomposition, agent-sensitive verification, and geometric constraint distillation—sets a foundation for future embodied AI systems that demand reliable, user-conditional task feasibility in real-world or simulated 3D spaces (Maillard et al., 31 Mar 2026).