---
title: 'SceneTeract: 3D Scene Verification'
url: https://www.emergentmind.com/topics/sceneteract
type: topic
---

# SceneTeract: 3D Scene Verification

SceneTeract is a verification framework for 3D scene functionality, designed to bridge high-level semantic reasoning with explicit geometric and physical simulation under agent-specific constraints. SceneTeract systematically decomposes natural language activities into atomic actions, performing multi-level validation to assess each action's functional feasibility in a given 3D scene and for a specified agent embodiment. The approach identifies not only the feasibility verdict per action but also exposes detailed failure diagnostics, providing a fine-grained understanding of agentic affordances and grounding gaps in current Vision-Language Models (VLMs) [2603.29798].

## 1. Architectural Overview

SceneTeract operates as a two-stage verification pipeline that integrates:

- **Semantic Planning**: A Vision-Language Model (VLM), denoted Φ, parses a target activity T (free-form language; e.g., "make a cup of tea then watch TV"), an agent profile A (explicit embodiment parameters), and a 3D scene S (geometry G and objects O) into an ordered sequence of atomic actions  
  π = [(a⁽¹⁾, o⁽¹⁾), ..., (a⁽ᴷ⁾, o⁽ᴷ⁾)].  
Actions are drawn from a fixed primitive library, grouped into four families: Mobility, Contact, Handling, Perception (see Table 1 of [2603.29798]).

- **Geometric Grounding**: For each (action, object) pair, a geometric verifier Ψ checks a sequence of agent-aware predicates (navigability, reachability, interactability, clearance, visibility). Each step t yields a boolean v⁽ᵗ⁾ and a per-predicate failure reason ε⁽ᵗ⁾, assembled into a diagnostic trace R (Algorithm 1).

This architecture enables hierarchical property checks, with short-circuiting to terminate early if fundamental constraints (e.g., navigation) fail.

## 2. Mathematical and Algorithmic Foundations

### 2.1 Agent Embodiment

The agent profile  
A = ⟨w_clear, h_eye, h_arm, r_arm, m_loco⟩  
specifies clearance requirements (w_clear), eye-level (h_eye), shoulder height (h_arm), reach radius (r_arm), and locomotion mode (walk or wheel).

### 2.2 Geometric Reasoning Suite

- **Navigability**: Constructs a binary navigation map (M) by projecting floors/obstacles, applying w_clear-based erosion, and extracting connected components. Navigability is determined by the existence of a path from the agent’s current position to an interaction zone Z_o associated with the target object (Algorithms 2–3). Formally,  
  is_Navigable_To(o) = ∃x ∈ Z_o such that L[x] = ℓ_agent.

- **Reachability**: For the agent shoulder point s and the shifted object geometry G_o^shifted, reachability is satisfied if  
  d_min = min_{p ∈ P, q ∈ G_o^shifted}‖p−q‖ ≤ r_arm.  
Both standing and crouch heights are checked (Algorithm 4).

- **Interactability**: Multi-view segmentation identifies handles or contact points using Molmo (a point-prompt VLM), SAM for mask extraction, and 3D deprojection. Reachability is then reassessed on relevant contact regions (Algorithm 5).

- **Clearance**: For Handling actions, a 3D box (V_clear) is attached to the object, and checked for collisions using trimesh, ensuring no occlusion with walls/objects and containment within room bounds (Algorithm 6).

- **Visibility**: Simulates agent eye positions and projects rays to object keypoints. Visibility holds if the fraction of clear rays surpasses threshold τ = 0.15 (Algorithm 7).

Predicates are selectively applied per action family as per Table 2.

## 3. Evaluation Procedures and Diagnostic Reporting

Algorithm 1 (GroundPlan) processes the action-object sequence π, consulting property requirements for each atomic action and collating verdicts with diagnostic messages. The evaluation pipeline outputs, for each action, a tuple (a⁽ᵗ⁾, o⁽ᵗ⁾, v⁽ᵗ⁾, ε⁽ᵗ⁾), as well as a global verdict  
  v = ∧_t v⁽ᵗ⁾.

Atomic actions that fail record specific failure reasons (e.g., insufficient reach, blocked path, lack of clearance), supporting both aggregate statistics and qualitative scene debugging.

## 4. Empirical Results and Functional Affordance Analysis

### 4.1 Scene Auditing on Synthetic Environments

On a dataset of 1,132 3D-FRONT living/dining rooms (across three agent profiles), SceneTeract performed 3,396 verifications:

| Agent Profile | Task Succ. (%) | Navigability (%) | Reachability (%) | Interactability (%) | Clearance (%) | Visibility (%) |
|---------------|----------------|------------------|------------------|---------------------|---------------|----------------|
| Adult         | 59.0           | 84.7             | 91.9             | 75.6                | 67.3          | ~98            |
| Child         | 66.0           | 89.2             | 93.7             | 73.4                | 66.3          | ~98            |
| Wheelchair    | 42.5           | 73.0             | 80.7             | 64.1                | 66.6          | ~98            |

Frequent failure modes included blocked hallways (navigability), unreachable high shelves (reachability), obstructed handles (interactability), and cluttered object fronts (clearance) (Fig. 5).

### 4.2 Vision-Language Model Functional Reasoning

Using SceneTeract-generated ground truth, several proprietary (Gemini-3 Flash, Gemini-3.1 Pro, Claude 4.6 Sonnet) and open (Gemma 3 12B/4B, Qwen3 VL-8B/4B, Ministral 3 3B) VLMs were evaluated in two modes:

- **Direct**: holistic task-level yes/no.
- **Decomposed**: step-wise yes/no for each atomic action.

Principal metrics (Table 4) included action/task accuracy, false positive rate (FP), Matthews Correlation Coefficient (MCC), Consistency, Horizon Stability Index (HSI), and Inclusivity Gap (InGap).

Key findings:

- Direct mode yields low task accuracy (54–63%) and high FP (>20%), resulting in low MCC (often <0.2, sometimes negative).
- Decomposition improves FP and MCC substantially (e.g. Gemini-3 Flash: FP 35.7→12.0, MCC 0.144→0.390).
- Substantial Inclusivity Gaps (5–20%) persist, even in strong models, indicating agent-profile-sensitivity.
- Some models (Claude 4.6) exhibit high internal consistency (>80%), suggesting emergent stepwise decomposition.
- Semantic competence alone is consistently insufficient for predicting physical feasibility.

## 5. Reward-Shaping and VLM Post-Training

SceneTeract serves as a reward engine for policy optimization, facilitating distillation of geometric constraints into VLM reasoning. Using Group Relative Policy Optimization (GRPO):

- **Base Model**: Qwen3-VL-4B-Instruct, tuned with LoRA (q_proj, v_proj, r=8, α=32).
- **Training Dataset**: Decomposed atomic-action set (80/20 scene split), batch=12, G=12, steps=2,800, lr=2e-5.
- **Reward Structure**:  
  r_format (+0.5 correct tag use, –1.0 on “prompt-echo”),  
  r_correct (+2.0 True, +3.0 False predictions),  
  r_spatial (+1.0 if negative and geometric failure cited).

The group-relative advantage computes as  
  A_i = (r_i – μ_r)/(σ_r+ε).

Post-training evaluation demonstrates marked improvements:

| Metric           | Pre (Decomposed) | Post (Decomposed) |
|------------------|------------------|-------------------|
| MCC              | 0.172            | 0.364             |
| FP               | 22.8%            | 14.1%             |
| Task Accuracy    | 61.4%            | 69.2%             |
| Consistency      | ~98%             | ~98%              |

Qualitative inspection of model <think> traces reveals increased use of geometric detail (“too far by 0.15 m,” “narrow path”), indicating internalization of SceneTeract's constraints.

## 6. Significance, Applications, and Outlook

SceneTeract provides a rigorous, interpretable framework for evaluating functional affordances in 3D environments under rich agent constraints, yielding insights unattainable by pure semantic or perceptual models. The diagnostic granularity offered by per-action, per-predicate failure traces facilitates synthetic scene auditing, accessibility validation, and design feedback.

As a benchmarking tool, SceneTeract exposes systematic grounding mismatches across leading VLMs, demonstrating that semantic understanding alone remains insufficient for robust embodied reasoning. Its role as a reward engine for VLM post-training results in measurable gains in physical grounding, suggesting scalable pathways for integrating geometric reasoning into large vision-language models.

A plausible implication is that SceneTeract’s approach—explicit decomposition, agent-sensitive verification, and geometric constraint distillation—sets a foundation for future embodied AI systems that demand reliable, user-conditional task feasibility in real-world or simulated 3D spaces [2603.29798].

Source: https://www.emergentmind.com/topics/sceneteract