PCG Management Protocol

• PCG Management Protocol is a unified multi-agent orchestration layer that facilitates scalable, controllable, and high-fidelity procedural 3D city generation.
• It incorporates structured plugin registration and dynamic API conversion, which ensure seamless interoperability between LLM-driven agents and Blender Python functions.
• The framework’s extensibility and robust multi-phase feedback loop drive significant improvements in execution reliability and simulation-grade urban environment generation.

The PCG (Procedural Content Generation) Management Protocol in City𝒳 is a unifying, multi-agent orchestration layer that enables scalable, controllable, and high-fidelity 3D city generation. It acts as an abstraction and management interface between LLM-driven agentic workflows and the low-level Blender Python API, ensuring extensibility, heterogeneity support, and robust execution of complex procedural generation pipelines. Through tightly specified plugin encapsulation, dynamic data-format adaptation, and multi-phase agent mediation—including visual feedback via GPT-4V—the protocol undergirds City𝒳’s ability to generate realistic, simulation-grade urban environments from multimodal inputs (Zhang et al., 24 Jul 2024).

1. Protocol Architecture: Layers and Responsibilities

The PCG Management Protocol is structured in three principal logical layers:

• Plugin Registry: Maintains a global registry of all "action functions" representing PCG plugin calls. Each entry, termed an Action Plugin, encapsulates metadata—name, description, input/output formats, and constraints—enabling introspection and compositional planning. The registry ensures that all plugins are discoverable and invocable from a unified interface, decoupling agents and plugins.
• Dynamic API Conversion Interface: Provides a uniform way to resolve data format mismatches between LLM agent outputs and the heterogeneous plugin APIs. This is operationalized as a collection of conversion functions:

$$\mathrm{conv}_{f_i \to f_j}: \mathcal{D}_{f_i} \longrightarrow \mathcal{D}_{f_j}$$

where $f_i, f_j$ span agreed data formats like SceneLayout, Point, Mesh, and others. Chaining of converters is automatic, supporting invocation from any unified interface.

• Infinite Asset Library & Retrieval: Encodes every asset with a text description and a 768-dimensional CLIP embedding. Asset retrieval is performed by computing top-10 cosine similarities between a query embedding and asset embeddings, with a random selection from this set, ensuring scalable and semantically aligned asset acquisition.

A common message pool (simple RPC/message-bus) underpins all layers, facilitating asynchronous, decoupled communication among four primary agents.

2. Structured Plugin Registration and Discovery

Each plugin is "structuredly encapsulated" as a JSON-serializable descriptor paired with a Python callable. The registration schema explicitly declares fields for name, description, input specification, output specification, limitation, and the run function:

class PluginSpec: def __init__(self, name: str, description: str, inputs: Dict[str,TypeSpec], outputs: Dict[str,TypeSpec], limitation: str, run_fn: Callable[...,Any]): self.name = name self.description = description self.inputs = inputs self.outputs = outputs self.limitation = limitation self.run = run_fn

Plugins are registered into a registry with the following interface:

class PluginRegistry: def __init__(self): self.plugins: Dict[str,PluginSpec] = {} def register(self, spec: PluginSpec): self.plugins[spec.name] = spec def discover(self, required_inputs: Set[str], required_outputs: Set[str]) -> List[PluginSpec]: return [p for p in self.plugins.values() if required_inputs.issubset(p.inputs.keys()) and required_outputs.issubset(p.outputs.keys())]

This design enables capability-based querying, allowing agentic planners to compose workflows based on required input/output capabilities, rather than hardcoded tool names.

3. Inter-Plugin Communication and Data-Format Reconciliation

Data and messages between plugins conform to standardized "data-formats," such as SceneLayout, PointCloud, FaceList, Mesh, and Texture. The conversion interface is described by a lookup table:

$$\mathrm{ConvMap} : F\times F \longrightarrow \bigl(\mathcal{D}_{f_i}\to\mathcal{D}_{f_j}\bigr)$$

where $F$ is the set of all supported data formats.

Messages exchanged on the bus follow a canonical schema:

$$M = \{ \texttt{"sender"}: \mathrm{AgentID}, \texttt{"action"}: \text{Name}, \texttt{"args"}: \text{Dict}<\text{Name},\text{Value}>, \tau: \text{timestamp} \}$$

Upon message receipt, the Executor resolves the corresponding PluginSpec, converts each argument as needed via registered converters, executes the wrapped Blender/Python function, applies output conversion, and dispatches the results. This dynamic format reconciliation underpins the seamless integration of heterogeneous plugins.

4. Multi-Agent Orchestration and Choreography

The protocol adopts a four-agent mediation architecture:

• Annotator: Scans all PluginSpec entries, grouping them by conceptual category (e.g., layout tools, material assignment), and publishes a mapping $L$ of "labels → plugin names".
• Planner: Accepts the user goal $I$, label map $L$, and planning guide $D$, and produces a high-level workflow $W = [A_1, A_2, ...]$. At each step, the Planner adapts to changes in Blender state $S_t$, guiding the sequence of actions.
• Executor: Listens for execution requests, invokes the appropriate plugin with dynamically converted arguments, and emits completion messages along with the updated state $S_{t+1}$.
• Evaluator: After each execution step, renders a viewport image $img_{t+1}$, feeds $(img_{t+1}, A_{t+1})$ to GPT-4V, and returns an evaluation label $R \in \{\text{ok}, \text{needs\_fix}, ...\}$. If $R$ is not "ok," the Planner is prompted for refinement.

The overall orchestration is depicted as:

W = Planner.plan_initial(L, I, D) t = 0 while not user_satisfied and t < MAX_STEPS: A = Planner.next_action(L, I, D, W, S_t) publish({sender:'Planner', action:A.name, args:A.args}) S_{t+1} = Executor.execute() R = Evaluator.eval(render(S_{t+1}), A) if R == 'needs_fix': Planner.refine(A, R) else: t += 1

This agentic management loop ensures robust, adaptively refined progress toward user-specified generative goals.

5. Visual Feedback and Inner Refinement Loops

The Evaluator introduces visual feedback through an automated GPT-4V classification mechanism:

$$R = \mathrm{Evaluator}\bigl(\mathrm{GPT4V}(img_{t+1}, A, D_{\mathrm{eval}})\bigr)$$

Should $R$ indicate any inconsistency or required correction, the Planner triggers a subtask loop, possibly inserting intermediary format conversions (e.g., "Point→Face") and rerunning the Executor:

def subtask_loop(A_t): S', img = Executor.execute(A_t) R = Evaluator.eval(img, A_t) if R == 'ok': return S' else: A_t' = Planner.fix(A_t, R) return subtask_loop(A_t')

This recursive structure reinforces precision and correctness, supporting iterative refinement until all visual and semantic criteria are satisfied.

6. Extensibility, Scalability, and Empirical Outcomes

The protocol's architecture facilitates the insertion of new PCG plugins with only minimal wrapper and format specification, requiring no agent or workflow modification. Registry-based discovery eliminates the need for pre-specified tool names in agentic prompts, enabling flexible composition.

The asset retrieval layer—based on CLIP embeddings—permits scaling to unbounded asset collections, with each new asset registered through a simple embedding addition.

Empirical evaluation in (Zhang et al., 24 Jul 2024) demonstrated that structured plugin encapsulation (inclusion of description, input specification, and limitation fields) yielded substantial execution and robustness benefits. Specifically, executability (ER@1) improved from approximately 30% to 94%, and success rate (SR@1) increased from roughly 40% to 83%. A plausible implication is that protocolized registration and data-format control contribute directly to stability and success in LLM-planned PCG workflows.

7. Summary and Significance

The PCG Management Protocol in City𝒳 provides a rigorous, extensible, and modular framework for procedural 3D city generation via LLM-driven agents. Key contributions include: a unified plugin registry with explicit specification, message-bus agent orchestration with clear roles, a comprehensive data-format conversion interface, scalable CLIP-based asset retrieval, and a self-healing execution loop grounded in multimodal visual feedback. Collectively, these design choices enable City𝒳 to support fine-grained, multi-modal, and scalable generation of 3D urban environments suitable for embodied intelligence research (Zhang et al., 24 Jul 2024).