Agentic Scene Policies (ASP)

• Agentic Scene Policies (ASP) are frameworks that use structured scene representations as the policy interface between high-level reasoning and conventional motion planning in robotics.
• They integrate object grounding, spatial reasoning, and affordance-aware interactions to convert scene models into actionable control for both manipulation and 3D scene generation.
• The modular, iterative design of ASP enables robust zero-shot behavior in short-horizon tasks while highlighting challenges in long-horizon, complex scenarios.

Agentic Scene Policies (ASP) denotes a class of agent architectures in which a structured scene representation serves as the operative interface between high-level reasoning and action. In its explicit robotics formulation, ASP is a modular, language-conditioned framework that treats the scene representation itself as the “policy interface” between an LLM-based agent and the robot, coupling open-vocabulary grounding, spatial reasoning, and affordance-aware interaction with conventional motion planning (Morin et al., 23 Sep 2025). Closely aligned work applies the same policy view to 3D scene generation, where an agent repeatedly reads scene state, selects tools, verifies outcomes, and updates spatial memory rather than producing a scene in one shot (Liu et al., 26 May 2025). The terminology is not uniform across the literature: in some adjacent work, “ASP” instead stands for Answer Set Programming, used as a symbolic reasoning substrate for scene interpretation and projection in semi-autonomous driving (Suchan et al., 2023).

1. Definition and conceptual scope

In the robotics paper that names the framework explicitly, ASP is presented as an alternative to monolithic vision-language-action policies. The central claim is that many language queries can be solved through a repeated three-step process consisting of object grounding, spatial reasoning, and part-level interaction, all expressed as queries over a structured scene representation and then executed through conventional motion planning (Morin et al., 23 Sep 2025). The resulting policy is not an end-to-end actuator mapping; it is a tool-using agent whose competence depends on the quality and queryability of the scene model.

This framing distinguishes ASP from systems that rely primarily on task-specific fine-tuning. The motivation is that VLAs can be promising end-to-end policies but may struggle with open-ended instructions, novel objects, and complex multi-step reasoning. ASP instead treats perception, reasoning, and action as modular operations over a reusable world model. A plausible implication is that the scene representation is elevated from a perceptual by-product to the principal control surface of the agent.

Related work broadens this scope from robot action to scene synthesis. In that literature, the “policy” is a controller over scene state: a VLM or LLM repeatedly reads spatial context, decides what scene action to take, executes generation or editing operations, verifies the result, and updates the context for subsequent decisions (Liu et al., 26 May 2025). In this broader sense, ASP is less a single algorithm than a design pattern for agentic interaction with structured scene state.

2. Scene representation as the policy interface

The core ASP abstraction in robotics is an ObjectMap whose entries combine geometry, semantics, and affordances within one data structure. Each object contains a 3D point cloud in the scene frame, RGB and depth crops, a feature vector, and a list of affordances; each affordance contains a point cloud, a natural-language part description, and a linked skill (Morin et al., 23 Sep 2025). This unification is what allows the same representation to support open-vocabulary retrieval, relational queries, and action selection.

The agent itself does not reason directly over raw RGB-D streams. Instead, it operates through a symbolic State containing held_object and an inventory of grounded object keys, and it interacts with the world through typed tools that return structured outputs with explicit success or failure signals (Morin et al., 23 Sep 2025). The scene representation therefore mediates both epistemic state and execution state.

Three tool classes define the interface. Object retrieval searches the ObjectMap using an open-vocabulary text query, computes CLIP similarity between the query and object features, selects the top k objects, and uses a VLM classifier, specifically Gemini 2.5, to confirm relevance. Spatial understanding computes quantities such as distance_to, distance_between, left_of, right_of, and size_of from point clouds and centroids. Interact maps a natural-language action description to an affordance and then to a skill such as grasp, place, drop, grasp_part, tip_push, pinch_pull, or hook_pull (Morin et al., 23 Sep 2025).

Affordance inference is explicitly two-stage. Gemini 2.5 first predicts a list of (skill, part) pairs from RGB crops and the action description; it is then prompted again with the part and crop to produce a bounding box, which is segmented with SAM 2.1, lifted to 3D with depth, and merged across views using IoU (Morin et al., 23 Sep 2025). The representation is therefore not merely object-centric; it is part-centric and skill-linked.

3. Execution loop, zero-shot behavior, and empirical profile

ASP executes language-conditioned behavior through iterative tool use. A query is resolved by retrieving candidate objects, optionally applying spatial disambiguation, detecting the relevant affordance when the task requires part-level interaction, and then invoking a skill wrapper that executes via motion planning (Morin et al., 23 Sep 2025). When a tool operation creates uncertainty about the scene, remapping can be triggered; in tabletop mode, if a skill updates the map, the agent inventory is cleared and the next retrieval call rebuilds the map from the current RGB-D observation.

For mobile ASP, affordances also shape navigation. The framework estimates a preferred viewing pose from the affordance point cloud, projects the affordance normal onto the horizontal plane to obtain a preferred position $\mathbf{p}_{aff} \in \mathbb{R}^2$, and searches over candidate poses $$(\mathbf{p} \in \mathbb{R}^2, \theta \in [0, 2\pi))$$ on a circle of radius $r$ by minimizing

$$\mathcal{F}(\mathbf{p}, \theta) + \lambda_{aff} \| \mathbf{p} - \mathbf{p}_{aff} \|_2 .$$

This injects affordance reasoning directly into viewpoint selection and navigation (Morin et al., 23 Sep 2025).

The reported evaluation emphasizes zero-shot operation. On tabletop manipulation, the benchmark comprises 15 manipulation tasks and 540 trials total. ASP outperforms the VLA baselines on 13 of the 15 tabletop queries, while the VLA baselines have an overall success rate around 20%; the paper also reports 67% and 75% progression figures for the two VLA methods in the analyzed setting (Morin et al., 23 Sep 2025). The affordance ablation, ASP (No Aff), shows that explicit affordance reasoning materially improves performance on tasks involving the keyboard, power adapter, thumbtack, and drawer-related interactions.

The same framework extends to room-level problems. Mobile tasks include phone dialing, cabinet opening, double-pick tasks, and spatial disambiguation such as placing the egg that is near the tomato in the pan. The map in this setting is built from 1 to 5 keyframes via teleoperation (Morin et al., 23 Sep 2025). Failure analysis identifies perception errors, wrong affordance detection, incorrect handle use, and navigation or localization issues, and the authors state that ASP currently works best for short-horizon language-conditioned manipulation rather than long-horizon household tasks.

4. Agentic scene generation as scene-state control

A parallel literature applies the ASP idea to 3D scene generation by treating scene synthesis itself as a policy over structured state. In “Agentic 3D Scene Generation with Spatially Contextualized VLMs,” the state is a dynamic spatial context

$C = (S, P, G),$

where $S$ is a scene portrait, $P$ is a semantically labeled point cloud, and $G$ is a scene hypergraph encoding unary, binary, and higher-order constraints (Liu et al., 26 May 2025). The VLM repeatedly reads this context, acts through asset generation, layout, environment code, and pose refinement, verifies the result, and updates the context. The paper reports best generation scores of CLIP: 0.385, BLIP: 0.737, and LPIPS: 0.571, and attributes downstream gains in interactive editing and path planning to the injected spatial context.

SAGE presents a more explicitly tool-oriented realization of the same idea. It describes scene generation as a policy over tools and actions, implemented through the Model Context Protocol (MCP), where the agent is the MCP client and scene generators, critics, and simulator are MCP tools or servers (Xia et al., 10 Feb 2026). Its architecture combines a Scene Initializer, Asset Placer, Asset Mover, Asset Remover, a visual critic, and a physics critic with simulation-in-the-loop via Isaac Sim. The system scales one valid scene into many variants through object configuration-level, object category-level, and scene layout-level augmentation, producing SAGE-10k, with 10k scenes, 50 room types, 50 styles, and 565K uniquely generated 3D objects (Xia et al., 10 Feb 2026). The reported scene-generation results include about 99.9% stability on average in the reported table, and critic ablations report collisions reduced to about 1.9% and stability raised to 99.6% when the physics critic is included.

Scenethesis articulates the same control logic in a training-free pipeline of LLM-based coarse scene planning, vision-guided refinement, physics-aware optimization, and judge-based spatial coherence verification (Ling et al., 5 May 2025). Its optimization objective combines pose alignment, collision avoidance, and stability constraints using RoMa correspondences and Signed Distance Fields, and the judge can reject a scene and force re-planning from Stage 1. On the reported indoor benchmark, Scenethesis achieves CLIP: 30.71, BLIP: 77.17, VQA: 0.8269, Col-O: 0.8%, Col-S: 6%, Inst-O: 3.20%, Inst-S: 16.67%, Reach: 0.94, and Walk: 0.96 (Ling et al., 5 May 2025). A plausible synthesis is that agentic scene generation systems instantiate ASP when they convert scene construction into a closed-loop policy over a stateful spatial representation.

5. Symbolic antecedents and terminological ambiguity

The acronym “ASP” has a longer and distinct history in symbolic AI. In “Assessing Drivers’ Situation Awareness in Semi-Autonomous Vehicles,” ASP stands for Answer Set Programming, not Agentic Scene Policies, and functions as a declarative logic-programming formalism for modeling a driver’s interpretation and projection of a traffic scene (Suchan et al., 2023). The system constructs a mental belief state from scene data and gaze-derived fixation probabilities, updates object positions with a Kalman filter, translates the belief state into symbolic facts, and invokes Clingo to compute an interpretation model $\mathcal{IM}$ and a projection model $\mathcal{PM}$. Its formal domain description is

$$(\mathbf{p} \in \mathbb{R}^2, \theta \in [0, 2\pi))$$0

This work is methodologically adjacent to Agentic Scene Policies because it combines numeric sensing with symbolic scene reasoning, but it is terminologically distinct.

Earlier ASP-based agent architectures also establish relevant precedents for policy reasoning. The UAV architecture of “Towards an ASP-Based Architecture for Autonomous UAVs in Dynamic Environments” combines centralized reasoning, decentralized reasoning, execution monitoring, diagnostic reasoning, and explicit reasoning about network communications in dynamic environments where communications are neither reliable nor free (Balduccini et al., 2014). Its control loop updates a local history, detects unexpected observations, explains them by diagnosis, replans, and executes the next action. That architecture is not scene-centric in the modern 3D sense, but it demonstrates how declarative policies can govern action under partial observability.

The abstraction paper “Towards Abstraction in ASP with an Application on Reasoning about Agent Policies” addresses another precursor issue: scalability (Saribatur et al., 2018). It defines abstraction as an over-approximation mapping from concrete literals and objects to abstract ones such that every answer set of the original program maps to an answer set of the abstract program. The paper’s application is declarative policies for reactive agents, including policy refutation by searching for counterexample trajectories. This suggests that, where Agentic Scene Policies are implemented declaratively, abstraction remains a natural route for reducing state-space complexity.

6. Human-centered, organizational, and normative extensions

A separate line of work asks not only how an agent acts in a scene, but when it should act at all. “When Should an AI Act?” proposes a human-centered model that separates Scene, Context, and Human Behavior Factors, arguing that behavior is an interpretive outcome rather than a direct response to visible cues (Jung et al., 26 Feb 2026). From this model it derives five design principles: behavioral alignment, contextual sensitivity, temporal appropriateness, motivational calibration, and agency preservation. Although the paper does not use the term ASP explicitly, it is directly relevant to any scene policy that conditions intervention on more than geometric or semantic state.

In enterprise workflows, the policy layer becomes explicitly normative. “Towards Enforcing Company Policy Adherence in Agentic Workflows” introduces a deterministic, transparent, and modular framework in two phases: an offline buildtime stage that compiles policy documents into ToolGuards, and a runtime stage in which those guards check compliance before each tool call (Zwerdling et al., 22 Jul 2025). On the $$(\mathbf{p} \in \mathbb{R}^2, \theta \in [0, 2\pi))$$1-bench Airlines domain, the reported end-to-end results improve from pass$$(\mathbf{p} \in \mathbb{R}^2, \theta \in [0, 2\pi))$$2 = 0.450 and pass$$(\mathbf{p} \in \mathbb{R}^2, \theta \in [0, 2\pi))$$3 = 0.227 for the original baseline to pass$$(\mathbf{p} \in \mathbb{R}^2, \theta \in [0, 2\pi))$$4 = 0.685 and pass$$(\mathbf{p} \in \mathbb{R}^2, \theta \in [0, 2\pi))$$5 = 0.500 with ToolGuards.

For web agents, policy becomes both security mechanism and governance question. “Same-Origin Policy for Agentic Browsers” argues that the agent itself can serve as an automated channel for cross-origin data flows, constructs SOPBench, and proposes SOPGuard, which tracks origin labels through agent read, transform, and write behavior (Wang et al., 12 Jun 2026). The paper reports passive SOP violation rate 1.00 for BrowserOS in the specified iframe-based passive setting, and runtime overhead of roughly 2.07% to 5.79%. At a broader normative level, “The Agentic Web Requires New Normative Infrastructure” proposes a triad of Delegation, Transparency, and Proportional Restriction for user-authorized agents acting online (Pattison et al., 9 Jun 2026). A plausible implication is that future ASP-like systems will require not only scene models and affordance maps, but explicit permission, provenance, and authorization layers.

7. Limitations and open directions

The current ASP literature is strong on modularity and interpretability, but its limitations are explicit. The robotics framework notes undersegmentation, depth noise, wrong affordance detection, navigation and localization errors, and the present focus on short-horizon language-conditioned manipulation rather than long-horizon household skills (Morin et al., 23 Sep 2025). SAGE explicitly limits itself to indoor scenes, emphasizes rigid-body physics, and identifies more articulated, deformable, outdoor, and real-robot closed-loop settings as future work (Xia et al., 10 Feb 2026). Scenethesis, although broader in scene type, relies on judge-based re-planning and still inherits errors from segmentation, depth estimation, asset retrieval, and correspondence matching (Ling et al., 5 May 2025).

The human-centered and normative extensions remain incomplete. The Scene–Context–Behavior model is described as conceptual, not yet empirically validated (Jung et al., 26 Feb 2026). SOPGuard acknowledges dependency-inference errors, under-propagation, over-propagation, adaptive attacks, and API cost (Wang et al., 12 Jun 2026). ToolGuard-style enterprise enforcement presently handles “must-not-do” constraints before tool activation, not the complementary problem of ensuring that required actions occur (Zwerdling et al., 22 Jul 2025). Across these strands, the literature suggests that the next stage of ASP research will depend on richer memory, stronger verification, broader embodied skill libraries, and more explicit alignment between scene-level competence and policy-level restraint.