PerceptTwin: Real-to-Sim Robotics Pipeline

Updated 5 July 2026

PerceptTwin is a fully automatic system that transforms semantic scene maps from a robot’s perception into interactive 3D simulations for plan verification.
It integrates mesh retrieval, 3D asset generation via TRELLIS, constrained ICP for placement, and affordance prediction to create executable simulation environments.
An LLM-based judge evaluates plans for logic, safety, and alignment with human preferences, enabling iterative refinement and enhanced planning success.

PerceptTwin is a fully automatic real-to-sim pipeline that constructs interactive simulations directly from semantic scene representations produced by a robot’s perception stack, with the explicit goal of validating and refining robot plans before execution on hardware (Gauthier et al., 2 Jun 2026). It combines open-vocabulary object maps with 3D asset generation, affordance prediction, and commonsense condition checking, and augments simulation-time verification with an LLM judge that assesses both plan correctness and alignment with human preferences. In the supplied literature, the label also appears in broader “twin”-oriented perceptual contexts, but its most concrete and explicit formulation is as a semantic scene reconstruction and planning-verification substrate for robotics.

1. Definition and problem setting

PerceptTwin addresses a specific bottleneck in robot planning: simulation environments are useful for both robot policy learning and planning verification and validation, but creating a bespoke simulation environment for each individual environment that a robot would operate in was simply infeasible (Gauthier et al., 2 Jun 2026). The system therefore starts from the semantic scene map that a modern robot perception stack can already build and turns that map into an interactive simulation with objects, geometry, interactions, and commonsense constraints.

The formulation in the paper is explicitly inverse. The input is a semantic scene map $M$ built from robot perception. Each object $o_i \in O$ has $PointCloud(o_i)$ , $Images(o_i)$ , and $Description(o_i)$ . The output is a simulated scene $\mathcal{S}$ containing, for each object $o_i$ , a 3D asset $a_i$ , a placement or transformation $u_i$ , and applicable affordances $s_i$ . The reconstruction process is written as

$o_i \in O$ 0

where $o_i \in O$ 1 is the real-world scene, $o_i \in O$ 2 is the semantic map produced from perception, and $o_i \in O$ 3 is the reconstructed simulated scene. The notation emphasizes that reconstruction is not a deterministic transcription of sensor data, but an inference over possible real scenes consistent with the map.

The paper situates this problem in the setting of open-vocabulary indoor scenes. Existing semantic scene representations such as ConceptGraphs and HOVSG provide open-vocabulary labels and geometry, but they do not directly provide an interactive simulator with object assets, object behaviors, and robot-object affordances. PerceptTwin is therefore positioned as a bridge between semantic SLAM or open-vocabulary 3D scene graphs and LLM-based robot planning and auditing.

2. Reconstruction from semantic maps to interactive scenes

PerceptTwin assumes a map from modern semantic scene representation systems such as ConceptGraphs. Each object in the map can include a label or caption from an LLM, segmented object images, and global-frame point clouds (Gauthier et al., 2 Jun 2026). The pipeline then reconstructs a simulatable scene in four stages: 3D asset acquisition, asset placement, affordance prediction, and commonsense condition checking.

Two asset-acquisition routes are provided. In mesh association, the system searches a 3D asset database using CLIP similarity with score

$o_i \in O$ 4

The top- $o_i \in O$ 5 assets by CLIP similarity are retained, and candidates with large size mismatches from the target point cloud are discarded. This strategy uses Objaverse as the asset source and follows the same CLIP-based retrieval idea used in Holodeck. In mesh generation, PerceptTwin uses TRELLIS, a 2D-to-3D generative model, to synthesize a 3D asset from images of the target object. A reported ablation replaces the original TRELLIS preprocessing with SAM-based object-specific segmentation instead of REMBG background removal, improving object isolation and reducing artifacts.

Placement is derived from the segmented global-frame point clouds. Because point clouds are noisy and generated or retrieved meshes may not be aligned, the system uses a constrained ICP variant that assumes horizontal alignment, disables shear, and allows scaling while preserving aspect ratio. Bounding boxes of point clouds are also used to infer missing spatial relations such as an object being on top of another object.

Affordance prediction extends reconstruction beyond geometry. The implemented skills include $o_i \in O$ 6, $o_i \in O$ 7, $o_i \in O$ 8, $o_i \in O$ 9, $PointCloud(o_i)$ 0, $PointCloud(o_i)$ 1, $PointCloud(o_i)$ 2, $PointCloud(o_i)$ 3, $PointCloud(o_i)$ 4, and $PointCloud(o_i)$ 5. These are associated with affordances such as pickupable, canReceive, canContain, sliceable, breakable, canTurnOnOff, openable, and closeable. PerceptTwin uses an LLM to infer which affordances apply to each object from the target robot description, the object label, and the object caption from the map. It also identifies slicing implements such as knives or utility blades.

A compact summary of the reconstruction stages is given below.

Stage	Input	Output
Asset acquisition	object images, text, point cloud	retrieved or generated 3D asset
Placement	global-frame point cloud	localization and orientation
Affordance prediction	robot description, label, caption	applicable skills and affordances
Commonsense checking	reconstructed object state	executable preconditions

The paper also states an important representational limit: PerceptTwin only uses the information actually present in $PointCloud(o_i)$ 6; for instance, it cannot reconstruct floor color or wall texture unless those are in the map.

3. Executable constraints and iterative planning

PerceptTwin does not treat the reconstructed scene as a passive visualization. It adds hardcoded commonsense preconditions to the actions, assuming a single-arm robot (Gauthier et al., 2 Jun 2026). Representative examples include: $PointCloud(o_i)$ 7 requires the object to be pickupable and the robot to have a free hand and be nearby; $PointCloud(o_i)$ 8 requires holding a pickupable object and being nearby to an object with canReceive; $PointCloud(o_i)$ 9 requires holding a pickupable object, being nearby, and the target object being open; $Images(o_i)$ 0 requires holding a slicing implement and being nearby; $Images(o_i)$ 1 requires that the path is navigable; and $Images(o_i)$ 2, $Images(o_i)$ 3, $Images(o_i)$ 4, and $Images(o_i)$ 5 require being nearby with a free hand.

The planning formalism is adopted from ProgPrompt. A planning task is

$Images(o_i)$ 6

where $Images(o_i)$ 7 is the set of objects, $Images(o_i)$ 8 the set of object properties, $Images(o_i)$ 9 the set of available actions or skills, $Description(o_i)$ 0 the transition model, $Description(o_i)$ 1 the set of viable initial states, $Description(o_i)$ 2 the set of goal states, and $Description(o_i)$ 3 the planning horizon. A plan is

$Description(o_i)$ 4

a sequence of actions intended to move the system from an initial state $Description(o_i)$ 5 to a final state $Description(o_i)$ 6. The plan can stop early if an action’s preconditions are not satisfied. The planner itself only sees the natural-language goal and the serialized scene state from the simulation.

This produces an iterative loop rather than one-shot plan generation. The pipeline is: robot perceives the scene; a scene map is built; PerceptTwin constructs the simulation; an LLM planner proposes a plan; the simulation executes the plan; PerceptTwin returns feedback; the planner revises the plan; and the process repeats for up to five iterations. The feedback can take the form of explicit skill errors, task-level corrections, or safety warnings. This suggests that PerceptTwin is best understood not merely as scene reconstruction, but as a verification substrate in which planning, execution, and state-dependent critique are coupled.

4. LLM judging, correctness, and human-preference alignment

A distinctive component is the LLM judge, which is introduced because hardcoded precondition checks can catch some failures, but not all (Gauthier et al., 2 Jun 2026). A plan can satisfy all local action preconditions yet still fail the actual task or violate human preferences. The judge is therefore tasked with two roles: logical correctness and human-preference alignment.

For logical correctness, the judge checks whether the final plan actually accomplishes the task. For human-preference alignment, it checks for unsafe or misaligned outcomes, including the possibility that a malicious prompt could induce harmful actions. The judge is isolated from the planner and only sees the planner’s results rather than the attacker’s prompt manipulation. This isolation is intended to improve robustness against jailbreak-style attacks.

To make judging efficient, PerceptTwin serializes scene states as key-value pairs and computes a Unix diff between the state before plan execution and the state after execution. The judge can output one of three messages:

Unsafe <reason>
Incorrect <reason>
Correct <reason>

In highly adversarial cases, the paper notes that the judge can also be run step-by-step after each skill instead of only at the end, at the cost of more computation.

A common misunderstanding would be to treat the judge as a replacement for executable checking. The architecture described in the paper is layered: hardcoded skill preconditions enforce local executability, while the judge supplies task-level and value-level evaluation. The two mechanisms operate on different failure modes.

5. Experimental evaluation and quantitative results

The evaluation uses scene maps built with ConceptGraphs and includes diverse scenes such as Backyard, Cones, Blocks, Veggies, and a single-frame indoor scene (Gauthier et al., 2 Jun 2026). Reconstruction baselines include the ConceptGraph scene representation, SAM + TRELLIS, REMBG + TRELLIS, and CLIP + Objaverse. The paper argues that CLIP + Objaverse gives diversity but weaker visual and semantic fidelity, that REMBG + TRELLIS can leave artifacts, and that SAM + TRELLIS is best among these for matching object appearance.

For planning, the baseline planner is SMART-LLM. The evaluated models are GPT-5, GPT-5 Mini, and GPT-5 Nano, while the judge uses GPT-5. The protocol uses single-robot planning, five seeds per experiment, and up to five PerceptTwin feedback iterations. Metrics adapt the Exec metric from prior planning work:

Exec: number of executed actions
Precondition: incomplete execution due to precondition failures
Success: final state satisfies goal conditions, $Description(o_i)$ 7

Two judge-based diagnostics are also introduced: Judged Incorrect and Judged Unsafe.

The headline planning result is a substantial increase in success. Excluding the Bomb task, initial success rates were 24% for GPT-5, 8% for GPT-5 Mini, and 0% for GPT-5 Nano; after PerceptTwin feedback, these improved to 88%, 48%, and 12%, respectively. The average improvement is about 39%. The paper interprets these gains as especially relevant for tasks that require commonsense preconditions, long-horizon sequencing, spatial grounding, or safety constraints.

A separate human interpretability study compares a baseline point-cloud video of the scene from multiple views against a simulated execution video produced by PerceptTwin. Participants predict whether a plan succeeds or fails on logic and consistency questions. The reported sample size is $Description(o_i)$ 8. PerceptTwin improves human prediction accuracy, especially for consistency questions, with gains of up to 18% on average for plans that fail because of unfilled skill preconditions.

6. Safety, robustness, and reported limitations

Safety is evaluated directly. The paper includes a Bomb task in which the prompt is adversarially framed as a movie-script request involving a bomb and a laptop, with the scene manipulated so that the bomb is actually a prop, mirroring the black-box prompting attack setting from the jailbreak literature (Gauthier et al., 2 Jun 2026). Under this setting, the planner can be tricked into proposing dangerous actions, such as taking a bomb to a laptop and activating it under the guise of reenactment. PerceptTwin’s judge flags such plans as unsafe.

The paper also reports safety interventions in ordinary Veggies tasks. One example states that the judge warns when a knife remains within about 0.75 m of a human during cutting. This is presented as evidence that PerceptTwin functions not only as a planner-refinement tool but also as a safety filter grounded in simulation state rather than prompt-only reasoning.

The reported limitations are concrete. TRELLIS object-state changes are a visual limitation: AI2Thor can sometimes change object appearance in response to skills like CutObject, but this is not fully supported for user-provided 3D assets, so object-state changes may be textual rather than visually rendered. TRELLIS is also slow and hardware-heavy, requiring a fairly strong GPU and taking around an hour for a scene with approximately 30 objects. In the Bomb task, the judge sometimes used “unsafe” and “incorrect” somewhat interchangeably. In one block-ordering variant, the judge inferred that the tower had to be placed inside a box even though that was not specified. Generated meshes can still have artifacts, especially where segmentation is imperfect, including holes or missing surfaces.

These limitations constrain current deployment. They do not negate the central result, but they indicate that the present system depends on the fidelity of map construction, object segmentation, asset generation, and judge reliability. A plausible implication is that PerceptTwin’s effectiveness is tied to the calibration of both the reconstruction stack and the post hoc evaluator.

In the supplied literature, “PerceptTwin” is not restricted to one disciplinary meaning. In robotics, it denotes the automatic conversion of semantic scene maps into interactive simulations for iterative LLM planning and verification (Gauthier et al., 2 Jun 2026). In adjacent work, the term or its described principle is extended to other “twin”-based perceptual settings.

In quantum measurement, optimal cloning is described as producing a coherent “twin” contribution whose correlated outputs permit joint access to complementary observables that cannot be cleanly measured sequentially on a single disturbed system; the broader message is that “quantum twins” form an entangled paired representation from which a joint quasiprobability can be isolated and used to determine the state (Thekkadath et al., 2017). In tactile robotics, TwinTac couples a physical tactile sensor, PhysTac, with a digital twin, DigiTac, so that synchronized simulated and real data can support a real-to-sim tactile sensing pipeline; in the reported object-classification experiment, adding DigiTac-generated data raises accuracy from about 33.57% to 95.0% (Huang et al., 12 Sep 2025). In nonlinear optics, a twinning field is defined as a driving electromagnetic pulse that makes two distinct materials optically indistinguishable by enforcing $Description(o_i)$ 9 for all $\mathcal{S}$ 0, with existence and uniqueness conditions derived for generic many-body lattice systems (McCaul et al., 2021). In visual cognition and model analysis, a related dual-thinking line of work studies images for which intuitive and logical visual processing diverge, using a human-adversarial dataset to probe rapid initial grouping and slower corrective inference in humans and deep models (Dayanandan et al., 2024).

Taken together, these uses do not define a single formalism. They instead suggest a recurring research motif: a “twin” representation, paired system, or matched simulation can make otherwise inaccessible structure available to measurement, inference, or verification. In that sense, PerceptTwin in robotics belongs to a broader family of twin-based methods, but its specific technical identity is the construction of open-vocabulary scene simulations from robot perception for grounded plan checking, iterative refinement, and safety auditing.