LychSim: A Controllable and Interactive Simulation Framework for Vision Research

Abstract: While self-supervised pretraining has reduced vision systems' reliance on synthetic data, simulation remains an indispensable tool for closed-loop optimization and rigorous out-of-distribution (OOD) evaluation. However, modern simulation platforms often present steep technical barriers, requiring extensive expertise in computer graphics and game development. In this work, we present LychSim, a highly controllable and interactive simulation framework built upon Unreal Engine 5 to bridge this gap. LychSim is built around three key designs: (1) a streamlined Python API that abstracts away underlying engine complexities; (2) a procedural data pipeline capable of generating diverse, high-fidelity environments with varying out-of-distribution (OOD) visual challenges, paired with rich 2D and 3D ground truths; and (3) a native integration of the Model Context Protocol (MCP) that transforms the simulator into a dynamic, closed-loop playground for reasoning agentic LLMs. We further annotate scene-level procedural rules and object-level pose alignments to enable semantically aligned 3D ground truths and automated scene modification. We demonstrate LychSim's capability across multiple downstream applications, including serving as a synthetic data engine, powering reinforcement learning-based adversarial examiners, and facilitating interactive, language-driven scene layout generation. To benefit the broader vision community, LychSim will be made publicly available, including full source code and various data annotations.

• The paper introduces LychSim, a simulation framework offering unprecedented control and rich synthetic data generation for vision research.
• It employs a unified Python API and procedural data pipeline to generate annotated 2D/3D scenes with controlled diversity and out-of-distribution challenges.
• The integrated agentic module enables interactive, language-driven scene planning and adversarial evaluation to enhance spatial reasoning models.

LychSim: A Controllable and Interactive Simulation Framework for Vision Research

System Architecture and Core Functionalities

LychSim is introduced as a highly customizable and interactive simulation environment purpose-built for computer vision research. The platform leverages Unreal Engine 5 (UE5) to exploit an extensive library of photorealistic assets, and addresses usability challenges through a streamlined Python interface abstracting engine complexities. The framework is structured around three pivotal components: (1) a unified Python API for scene manipulation and annotation, (2) a procedural data pipeline for generating richly annotated 2D and 3D environments with user-controllable diversity and out-of-distribution (OOD) complexity, and (3) native agentic integration via Model Context Protocol (MCP), enabling autonomous, closed-loop interactions for LLM-based agents.

Figure 1: LychSim features a streamlined Python interface, a rich procedural data pipeline, and native agentic simulation for controlled experimentation.

These elements collectively lower technical barriers for researchers, facilitate scalable synthetic data generation, and empower a broad spectrum of downstream vision tasks spanning spatial VLMs, embodied AI, and interactive reasoning.

Procedural Data Generation and Ground Truth Annotation

A key advantage of LychSim is full control over the granularity and semantics of ground-truth annotation, extending well beyond traditional depth maps, segmentations, and bounding boxes. The procedural pipeline synthesizes 3D environments from curated UE5 asset libraries and supports integration of external scene layouts (e.g., Infinigen, HSSD-200), maximizing domain and asset diversity. Object-level annotations include canonical category, scale, pose alignment, and semantic text captions—enabling precise scene reconstruction and language-guided placement. Scene-level procedural rules are interactively defined and exported, governing navigable spaces, roadways, and trajectory priors critical for realistic layout synthesis. This hybrid approach facilitates domain transfer and the construction of high-density, semantically cluttered, and challenging OOD scene variants.

Rich, pixel-aligned ground truths are extracted at scene, camera, and object levels, including instance-level occlusion metrics and part segmentation driven by direct rendering target customization. LychSim explicitly models object visibility beyond the image plane through instance-level depth buffers and geometric projection, supporting quantitative analysis of occlusion-related model failure modes. Dense point maps and part-level segmentations are generated automatically, enabling supervision for representation learning pipelines reliant on geometric and semantic completeness.

Figure 2: LychSim provides comprehensive 2D and 3D ground truths, enabling rigorous evaluation and training for spatial and compositional reasoning.

Agentic Integration and Interactive Simulation

LychSim introduces native MCP integration, exposing Python tools as standardized agentic APIs for closed-loop scene manipulation by LLM-based agents. The MCP server supports autonomous navigation, object placement, scene state querying, and parallel multi-view rendering for rapid spatial verification. Tool schemas are exposed via both JSON and XML formats, ensuring robust LLM compatibility and minimizing parsing errors during agent execution.

Figure 3: Agentic integration with MCP enables seamless closed-loop reasoning and interaction by language-driven agents.

This integration unlocks advanced applications including adversarial evaluation, interactive scene planning, and multi-turn, natural language-driven spatial reasoning and arrangement. The agentic pipeline can dynamically adjust layouts, verify physical plausibility, and iteratively refine placement based on real-time visual feedback.

Practical Applications: Synthetic Data, Adversarial Examiners, and Scene Planning

LychSim demonstrates utility in three primary application domains:

• Synthetic Data Engine: Procedural pipelines generate post-training datasets tailored to diagnosing spatial reasoning deficiencies in vision-LLMs. The platform supports systematic OOD scenario synthesis—e.g., severe occlusion, clutter—as well as scalable annotation for domain transfer and robustness benchmarks.
• RL-based Adversarial Examiners: The environment powers adversarial policy training for model vulnerability discovery. RL agents explore camera viewpoints and poses to minimize intersection-over-union (IoU) of predictions in segmentation tasks. Consistent with prior findings, examiners identify weaknesses in state-of-the-art models such as Segment Anything, even within simple environments.

Figure 4: RL-driven adversarial examiner exposes weak spots in instance segmentation by actively probing scene parameters.

• Interactive, Language-driven Scene Planning: Integration with Claude Opus 4.6 and Gemma 4 via MCP enables LLMs to parse scene specifications, plan functional group layouts, place anchors, verify spatial relationships, and iterate based on visual feedback. Despite current model limitations in spatial reasoning, LychSim demonstrates multi-turn interaction, navigation, and responsive editing in scene arrangement.

Figure 5: Scene layout generation and planning are performed interactively by agentic LLMs, combining MCP control and natural language instructions.

Annotation Tools and Rule Creation

An interactive annotation toolkit within LychSim allows users to define procedural rules and spatial relationships directly in the Unreal Editor. Users can annotate trajectories, zones, and regions via intuitive clicks, exporting geometric constraints (lines, splines, areas) for automated layout generation. Procedural annotations are critical for guiding object placement, groupings, and OOD scene generation—serving as the backbone for scalable, semantically coherent synthetic benchmarks.

Figure 6: Built-in annotation and rule creation tools enable seamless procedural guidance for layout and navigation.

Implications and Future Directions

The practical and theoretical contributions of LychSim are multifold. As a vision-centric simulation engine, LychSim provides unprecedented controllability, annotation richness, and agentic extensibility. This enables systematic evaluation of spatial and compositional reasoning weaknesses in VLMs, synthetic data-driven robustness enhancement, and interactive embodied agent training under realistic constraints. The explicit modeling of occlusion, part segmentation, and semantic context supports research in 3D representation learning, compositional reasoning, and goal-directed scene synthesis.

Future developments may focus on expanding asset and domain coverage, improving agentic spatial reasoning via compound 3D-informed models, and leveraging LychSim for benchmark creation across spatial aptitude, compositional VQA, causal reasoning, and embodied task planning. The open-source commitment is poised to accelerate community-driven extensions and cross-domain experimentation.

Conclusion

LychSim constitutes a highly accessible yet powerful simulation framework for computer vision research, unifying streamlined scene manipulation, rich procedural annotation, and agentic, interactive simulation. Its integration with agentic protocols and language-driven control fosters advanced applications in adversarial evaluation, synthetic data generation, and collaborative spatial planning. LychSim addresses longstanding technical barriers and enables systematic, fine-grained analysis of vision models under controlled, high-fidelity conditions, providing a robust foundation for next-generation spatial reasoning research and practical deployment (2605.12449).