PerceptTwin: Semantic Scene Reconstruction for Iterative LLM Planning and Verification

Abstract: Simulation environments are useful for both robot policy learning and planning verification and validation. Traditionally, the process of creating a simulation was onerous. Creating a bespoke simulation environment for each individual environment that a robot would operate in was simply infeasible. In this work, we introduce PerceptTwin, a fully automatic pipeline that constructs interactive simulations directly from semantic scene representations produced by a robot's perception stack. PerceptTwin combines open-vocabulary object maps with 3D asset generation, affordance prediction, and commonsense condition checking. These interactive simulations can be used to validate and refine plans before they are executed on the robot hardware. Borrowing from the AI alignment literature, we also introduce an LLM judge that verifies plan correctness and alignment with human preferences. Experiments show that PerceptTwin feedback allows LLM planners to refine plans, enhance safety, and resist harmful black-box prompting attacks. In our suite of tasks, PerceptTwin improves plan success by an average of approximately 39% for GPT5, GPT5Mini, and GPT5Nano planners. Additionally, PerceptTwin also improves human plan verification by up to 18% on average for plans that fail due to unfilled skill preconditions. Our results demonstrate the potential of open-vocabulary scene simulation from robot perception as a foundation for safer, more reliable robot planning.

• The paper proposes an automated real2sim pipeline that converts robot perception data into editable, semantically accurate simulations for planning and verification.
• It integrates LLM-guided affordance prediction with iterative planning feedback, significantly boosting plan success rates and safeguarding against adversarial inputs.
• The approach enhances human interpretability in robotic planning by providing simulation-driven insights, improving plan outcome classification accuracy by up to 18%.

PerceptTwin: Semantic Scene Reconstruction for Iterative LLM Planning and Verification

Introduction and Motivation

PerceptTwin addresses the longstanding challenge of converting robot perception data into actionable, verifiable, and interpretable simulation environments for autonomous agents, particularly in open-vocabulary domains. Traditional simulators relied on handcrafted environments, limiting scalability and generalization. Recent advances in SLAM, 3D scene graphs, and foundation models—including CLIP, SAM, and GPT-series LLMs—have enabled rich, open-vocabulary representations, but these remained static and insufficient for planning and feedback-driven reasoning.

PerceptTwin proposes an automated real2sim pipeline that leverages semantic scene graphs output by robot perception stacks to produce interactive, editable simulations. The system reconstructs environments, predicts affordances, and facilitates downstream planning via iterative feedback—including an LLM judge for logical and preference-based plan verification—and addresses vulnerabilities such as LLM jailbreaking.

Semantic Scene Reconstruction Pipeline

The reconstruction pipeline begins with a unified semantic map $M$ containing point clouds, segmented images, and open-vocabulary object descriptions. For each object, PerceptTwin performs asset finding/generation (via CLIP+Objaverse and TRELLIS), spatial localization (alignment of object and asset point clouds), and affordance prediction (LLM-guided assignment of actionable skills, e.g., PickUpObject, SliceObject, OpenObject). Asset placement employs constrained ICP registration and bounding-box analysis to infer relational edges, while affordance assignment uses LLM prompt engineering based on robot and object features.

Figure 2: PerceptTwin recovers semantically accurate and visually coherent scene reconstructions across diverse real-world inputs and asset-generation pipelines.

The pipeline demonstrates robust performance even with single-frame inputs, and adapts to incomplete or noisy perception maps, bolstered by SAM-based segmentation pre-processing for improved asset fidelity.

Figure 4: PerceptTwin reconstructs scenes from single-frame robot observations, maintaining visual and geometric correspondence.

Scene Generation: Dataset Creation and Diversity

PerceptTwin's generative capability extends beyond scene reconstruction for planning; it enables scalable automatic generation of diverse datasets for embodied AI training and evaluation, supporting both symbolic and visual reasoning tasks, and reduces manual engineering required in domains like procedural environment generation.

Figure 1: Diverse automatic scene generation from semantic maps, supporting simulation for perception and planning benchmarks.

Plan Verification and Iterative LLM Planning

A core innovation of PerceptTwin is its feedback-enabled planning loop. After reconstruction, an LLM planner proposes a plan (sequence of skills) to solve a high-level task. The plan is executed in simulation where failures of skill preconditions, logical inconsistencies, or alignment violations are detected and relayed as feedback. Hardcoded error detection leverages skill preconditions, but PerceptTwin augments this via an isolated LLM judge inspired by AI alignment literature, providing stepwise or summary audit reports (e.g., by diffing scene states as key-value pairs after plan execution).

Figure 5: PerceptTwin enables iterative plan refinement via simulated feedback, substantially increasing plan success and mitigating unsafe/adversarial plans.

Quantitatively, PerceptTwin feedback improved plan success rates by $\approx39\%$ across planner variants (GPT5, GPT5Mini, GPT5Nano), with significant gains in task difficulty and horizon length. Notably, the LLM judge reliably blocked adversarially induced unsafe plans (e.g., simulated bomb detonation scenarios analogous to LLM jailbreak attacks), recognizing both logical and value-aligned failures.

Human Interpretability and Planning Transparency

Beyond automated plan verification, PerceptTwin enhances human understanding and prediction of plan feasibility and execution outcome. In a user study ($N=93$), participants tasked with classifying plan outcomes achieved up to $18\%$ higher accuracy (statistically significant for consistency questions) when supplied with PerceptTwin simulation videos as opposed to point cloud baseline visualizations, demonstrating improved transparency in robotic planning and error detection.

Figure 3: Human plan verification accuracy is significantly improved using PerceptTwin-generated simulations, especially for consistency failures.

Discussion and Implications

PerceptTwin marks a substantial increase in the practical utility of open-vocabulary scene representations within robot planning, verification, and interpretability. The pipeline operates fully automatically, supports high-fidelity asset generation, and robustly handles open-set perception. Combination of LLM-based affordance prediction, error checking, and value-aligned judge modules addresses core safety concerns, vital given the demonstrated vulnerabilities of LLM-controlled robots under adversarial prompting (2606.04226).

From a theoretical perspective, the work strengthens the case for integrating real2sim capability in embodied AI, bridging perception, planning, and simulation for trustworthy agent behavior. The modular pipeline supports extension to complex domains—including deformable objects and multimodal environments—with minimal human intervention; and aligns with recent calls for system-level AI safety verification.

Future Prospects

PerceptTwin may serve as a foundation for frameworks combining scalable simulation generation, federated plan verification, and generalist robot policy training. Research directions include extending support for dynamic and articulated assets, scaling simulation integration with differentiable physics engines, enabling cross-task transfer (zero-shot), and tightening feedback granularity for high-stakes planning scenarios. Addressing simulation realism limitations (e.g., stateful asset appearance transformations) could further improve applicability in interactive embodied learning.

Conclusion

PerceptTwin establishes a practical and robust methodology for semantic scene reconstruction, planning, and verification tightly integrated with LLM-based perception and reasoning. Empirical results demonstrate substantial improvement in robot planning correctness, safety, and human interpretability over the prior art. The approach addresses core risks in autonomous agent deployment and offers actionable insights for the development of reliable real-world embodied AI systems.

(2606.04226)