AgentSGEN: Multi-Agent LLM in the Loop for Semantic Collaboration and GENeration of Synthetic Data (2505.13466v1)

Abstract: The scarcity of data depicting dangerous situations presents a major obstacle to training AI systems for safety-critical applications, such as construction safety, where ethical and logistical barriers hinder real-world data collection. This creates an urgent need for an end-to-end framework to generate synthetic data that can bridge this gap. While existing methods can produce synthetic scenes, they often lack the semantic depth required for scene simulations, limiting their effectiveness. To address this, we propose a novel multi-agent framework that employs an iterative, in-the-loop collaboration between two agents: an Evaluator Agent, acting as an LLM-based judge to enforce semantic consistency and safety-specific constraints, and an Editor Agent, which generates and refines scenes based on this guidance. Powered by LLM's capabilities to reasoning and common-sense knowledge, this collaborative design produces synthetic images tailored to safety-critical scenarios. Our experiments suggest this design can generate useful scenes based on realistic specifications that address the shortcomings of prior approaches, balancing safety requirements with visual semantics. This iterative process holds promise for delivering robust, aesthetically sound simulations, offering a potential solution to the data scarcity challenge in multimedia safety applications.

• The paper introduces a dual-agent LLM framework that decouples semantic planning from low-level scene editing to generate safety-aligned synthetic 3D scenes.
• It leverages context enrichment to transform natural language goals into structured constraints, ensuring precise collision checking and scene realism.
• Both human and automatic evaluations show that AgentSGEN outperforms baseline methods in semantic fidelity and safety compliance.

AgentSGEN: Multi-Agent LLM-in-the-Loop for Semantic 3D Scene Generation and Synthetic Data

Motivation and Problem Setting

The scarcity of real-world data for safety-critical scenarios, such as blocked emergency exits in indoor environments, presents a significant challenge for training and evaluating AI systems in domains like construction safety. Ethical and logistical constraints preclude the collection of such data at scale, necessitating robust synthetic data generation frameworks. Existing 3D scene generation methods, including manual design, procedural algorithms (e.g., Infinigen), and LLM-guided pipelines (e.g., Holodeck), either lack semantic controllability or fine-grained editability, limiting their utility for simulating nuanced safety violations.

Figure 1: Comparison of 3D scene generation methods.

AgentSGEN addresses these limitations by introducing a cognitively inspired, multi-agent system that leverages LLMs for iterative, semantically controlled scene editing. The framework is designed to generate synthetic 3D scenes that are both visually plausible and precisely aligned with user-specified safety constraints, supporting downstream applications in computer vision and robotics.

System Architecture and Methodology

AgentSGEN operationalizes the Dual Process Theory (DPT) by decoupling high-level semantic reasoning from low-level scene manipulation. The architecture comprises two specialized agents:

• Evaluator Agent (System 2): Implements deliberate, reasoning-intensive planning, formulating action sequences to satisfy semantic and safety constraints.
• Editor Agent (System 1): Executes atomic scene modifications with low latency, focusing on precise object placement and manipulation.

The workflow is structured into three stages: context enrichment, semantic planning, and interactive scene editing.

Figure 2: Initial scene from Holodeck with corresponding scene graph and SGRender views.

Context Enrichment and Constraint Generation

Given a high-level natural language goal (e.g., "a bedroom with doors blocked by large objects"), a lightweight LLM generates a structured set of constraints, including collision, spatial, safety, and goal-specific requirements. The initial context bundle comprises the symbolic scene graph, visual projections (2D/3D), and the constraint set, serving as input to both agents.

Figure 3: Multi-modal context components passed to both agents. The context bundle consists of (i) goal-specific constraint requirements, (ii) symbolic scene graph, and (iii) 2D/3D renderings from SGRender.

Semantic Planning (Evaluator Agent)

The Evaluator Agent, instantiated with a high-capacity LLM and chain-of-thought prompting, ingests the context bundle and produces an ordered action plan. Each action specifies an object, an operation (move, rotate, delete), and parameters. The agent performs constraint satisfaction, hypothetical reasoning (e.g., collision forecasting), and global optimization for semantic alignment. The plan is then passed to the Editor Agent.

Interactive Scene Editing (Editor Agent)

The Editor Agent, implemented with a non-reasoning LLM, executes the Evaluator's plan stepwise, updating the scene graph and visualizations after each action. The agent operates under strict constraints, ensuring compliance with collision and spatial rules. The separation of planning and execution enables both semantic depth and operational efficiency.

Figure 4: AgentSGEN architecture showcasing semantic planning by the Evaluator Agent and iterative scene editing by the Editor Agent.

Scene Graph as Interactive Environment

The scene graph functions as both a symbolic and interactive environment, supporting AABB-based collision checking. The system can toggle between collision-aware and collision-disabled modes, allowing controlled violations of physical realism when required by the goal (e.g., intentionally blocking a door). Feedback is provided to the agents for reactive correction.

Final Rendering and Dataset Generation

Upon completion and validation of all constraints, the finalized scene graph is rendered using a high-fidelity engine (e.g., AI2-THOR), producing RGB images, segmentation masks, depth maps, and object-level metadata. This enables the generation of large-scale, annotated synthetic datasets tailored to safety-critical scenarios.

Figure 5: Final rendering and synthetic dataset generation using AI2-THOR. Output includes RGB, depth, segmentation, and annotated metadata.

Evaluation and Results

AgentSGEN was evaluated on 53 indoor scene types from the MIT Indoor Scenes dataset, with the primary goal of generating rooms where doors are blocked by large objects. The evaluation compared three configurations: Holodeck baseline, AgentSGEN with collision checking, and AgentSGEN without collision checking.

Human Evaluation

Two tasks were conducted: binary task completion (preference for goal satisfaction) and goal-oriented Likert-scale scoring (effectiveness, arrangement, scale appropriateness). Inter-annotator agreement (Cohen’s kappa = 0.406) indicated moderate reliability.

• Binary Task Completion: The collision-aware AgentSGEN was preferred in 38 out of 53 trials, while Holodeck was selected only 6 times, demonstrating a strong advantage in goal satisfaction.

  Figure 6: Binary Task Completion for both human evaluation and automatic evaluation setup.

  

  Figure 7: Confusion matrix of binary preference judgments under two collision settings. With collision checking enabled, Cohen’s kappa 0.406 indicates moderate agreement, while disabling collision checking reduces agreement to $\kappa = 0.151$, suggesting that physical plausibility improves annotator consistency.

• Likert-Scale Scores: AgentSGEN with collision checking achieved the highest average scores (>4.5/7) across all semantic criteria, while Holodeck consistently scored near the minimum.

  Figure 8: Average Likert scores (1–7 scale) for three goal-oriented evaluation questions. Our collision-aware method outperforms both the collision-disabled version and the Holodeck baseline across all metrics.

Automatic LLM-Based Evaluation

GPT-4.1 and Gemini 2.5 Pro were used for discrete choice experiments, consistently preferring AgentSGEN-edited scenes, especially with collision checking enabled (38/53 for GPT-4.1). However, LLMs showed a tendency to favor Holodeck in arrangement and scale, likely due to visual regularity biases.

Figure 9: Confusion matrix of LLM judgments under two collision settings.

Figure 10: DCE results from GPT-4.1 and Gemini for the three goal-oriented questions.

Analysis

AgentSGEN demonstrates that embedding semantic reasoning and constraint enforcement within the generation loop yields scenes that are both functionally aligned with safety objectives and visually plausible. The dual-agent design enables fine-grained, context-aware editing, outperforming procedural and one-shot LLM-based methods in both human and LLM-based evaluations. The results highlight that semantic and perceptual objectives can be jointly optimized without trade-off, provided that symbolic reasoning and reactive control are tightly integrated.

Limitations and Future Directions

While AgentSGEN achieves strong performance, the evaluation is limited by the number of human annotators and the scope of scene types. LLM-based evaluation, while promising, exhibits biases toward visual regularity and may not fully capture goal-conditioned semantics. Future work should explore scaling the framework to more complex environments, integrating richer physical simulation, and leveraging multi-modal LLMs for enhanced visual-semantic alignment. The modular architecture supports extension to other domains, such as robotics, urban planning, and healthcare.

Conclusion

AgentSGEN introduces a cognitively grounded, multi-agent LLM-in-the-loop framework for semantically controlled synthetic 3D scene generation. By decoupling high-level reasoning from low-level execution, the system achieves fine-grained spatial control and robust satisfaction of complex safety constraints, outperforming existing procedural and LLM-based baselines. The approach is validated through rigorous human and automatic evaluation, demonstrating its potential as a foundation for scalable, human-aligned synthetic data generation in safety-critical AI applications.