Procedural 3D Synthesis

• Procedural 3D synthesis is a method that employs explicit rules, formal grammars, and parameterized programs to generate complex and editable 3D models and scenes.
• It integrates rule-based systems with neural inversion and hybrid models to efficiently control asset instantiation, texturing, and scene assembly.
• This approach underpins applications such as urban modeling, synthetic data generation, and real-time editing, demonstrating high scalability and semantic alignment.

Procedural 3D synthesis refers to algorithmic methodologies for specifying and generating three-dimensional (3D) content—geometry, structure, textures, and even entire scenes—using explicit procedural rules, formal grammars, parameterized programs, or stochastic models. These approaches enable scalable, editable, and highly variable asset and environment creation, with applications spanning interactive city modeling, content-driven simulation, synthetic data generation, and interactive 3D content frameworks.

1. Foundations and Representations

Procedural 3D synthesis formalizes 3D content generation via explicit rule sets, grammars, or parameterized algorithms that map compact input descriptions (numeric parameters, program tokens, or DSL scripts) to complex spatial assets. Central representations include:

• Shape Grammars and Recursive Rules: Context-free (or more expressive) grammars describe buildings, plants, and objects hierarchically via production rules, e.g., $Building \rightarrow FacadeBlocks_{n_x \times n_y}$ (Tsirikoglou et al., 2017, Dax et al., 28 Jan 2025). L-systems and parametric grammars are used for vegetation and urban street layouts (Wen et al., 8 May 2025).
• Procedural Programs and Compact Graphs: Direct program-like representations (e.g., PCG in Proc3D (Raji et al., 18 Jan 2026)) encode structure as a directed acyclic graph of parameters, primitives, and operators, supporting incremental, interpretable editing.
• Asset Instantiation and Asset Libraries: Reusable sub-assets (windows, doors, façade segments) serve as basic units, instantiated and transformed according to procedural “assembly code” (Li et al., 2024, Dax et al., 28 Jan 2025).
• Graph-based Procedural Abstractions: Attributed graphs or edge-sequentialized tokenizations encode complex assets for neural generation and editing workflows (Zhang et al., 10 Nov 2025).

Procedural synthesis yields families of 3D models parameterized by high-level variables—enabling compact storage and efficient expansion into large, detailed environments (Li et al., 2024, Raji et al., 18 Jan 2026, Wen et al., 8 May 2025).

2. Grammars, Programs, and Rule Integration

A procedural 3D workflow begins with a formal grammar, program, or node-graph description. Key mechanisms include:

• Programmatic Spec: Input as a sequence of tokens, instruction–parameter tuples, or JSON/DAG objects describes asset composition, geometry, and placement (Raji et al., 18 Jan 2026, Dax et al., 28 Jan 2025, Zhang et al., 10 Nov 2025).
• Hierarchical Instantiation: Buildings and cities are assembled by repeated instantiation and transformation of base assets, with explicit rules for floor count, grid/row structure, asset swaps, and symmetry (Li et al., 2024, Dax et al., 28 Jan 2025, Liu et al., 5 Feb 2026).
• Parameter Mapping: Scene-level or asset-level vectors $p$ (continuous/discrete) index over size, count, position, material, or compositional rules (Zhao et al., 2024, Tsirikoglou et al., 2017, Chen et al., 2024).
• Procedural Texture Synthesis: Consistent textures across procedural asset families are achieved by generating texture for a template and transferring it to all parameterized variants via learned UV displacement (Xu et al., 28 Jan 2025).

Procedural grammars constitute an interpretable, modular, and highly controllable interface for 3D content generation. Notably, recent advances exploit LLMs for program synthesis and editing in response to natural language inputs (Raji et al., 18 Jan 2026, Hayashi et al., 6 Oct 2025, Liu et al., 5 Feb 2026).

3. Integration with Differentiable, Neural, and Hybrid Models

Purely rule-based procedural synthesis has been extended with neural and hybrid components to enable higher-level control, sparse supervision, and data-driven generalization:

• Neural Inversion and Decoding: Transformers and diffusion models recover procedural parameters/programs from images or point clouds by minimizing reconstruction loss or via denoising objectives (Zhao et al., 2024, Dax et al., 28 Jan 2025, Zhang et al., 10 Nov 2025).
• Procedural–Neural Fusion: In Proc-GS, procedural code defines building assembly while 3D Gaussian Splatting (3D-GS) is used for high-fidelity rendering and efficient gradient-based learning of shared (base) and instance-specific (variance) components (Li et al., 2024).
• Edge-based and Tokenized Neural Procedural Graphs: For image-to-3D reconstruction, procedural graph abstractions are sequentialized and decoded with transformer priors, with inference augmented by reward-guided search (e.g., MCTS) for alignment to observations (Zhang et al., 10 Nov 2025).
• Self-supervised Learning: Procedural program-driven shape datasets are used for 3D representation learning, often with masked auto-encoding or contrastive objectives, achieving transfer performance rivaling real-world CAD datasets (Chen et al., 2024).

These approaches combine procedural editability with neural priors and learning-based inversion—enabling flexible applications such as inverse procedural content generation and text/image-driven synthesis (Zhao et al., 2024, Dax et al., 28 Jan 2025).

4. Practical Pipelines and Optimization Strategies

Operational pipelines for procedural 3D synthesis include:

• Asset Decomposition and Assembly: Automatic or guided decomposition of captured or designed assets into reusable "base assets," each parameterized and stored compactly for instantiation with optional per-instance variance (Li et al., 2024).
• City Layout and Scene Organization: Multi-agent orchestrations and plugin architectures (e.g., CityX) combine semantic maps, OSM data, and user guidance to assemble unbounded, multi-modal 3D urban scenes with programmatic control and agent-mediated feedback (Zhang et al., 2024).
• Rendering and Annotation: Integrated physically based rendering models, e.g., Monte Carlo path tracing with classic rendering equations, are applied for high-fidelity outputs, with scene graphs supporting automatic ground-truth annotation for downstream tasks (Tsirikoglou et al., 2017).
• Editing and Real-time Feedback: Systems such as Proc3D offer slider/checkbox-based real-time editing of input parameters, with LLM-driven natural language updates and minimal recomputation via incremental graph re-evaluation (Raji et al., 18 Jan 2026, Hayashi et al., 6 Oct 2025).

Optimization and learning employ a range of best practices: distributed/accelerated inference, tile-wise or block-wise GPU parallelization, program-level validity constraints, and reward shaping for spatial and visual alignment (Li et al., 2024, Liu et al., 5 Feb 2026).

5. Evaluation Metrics, Trade-offs, and Comparative Results

Procedural 3D synthesis is evaluated by a spectrum of geometric, visual, and semantic alignment metrics, for both assets and scenes:

• Geometry/Image Fidelity: PSNR, SSIM, LPIPS for rendering; Chamfer and Earth-Mover’s Distance (point clouds); structural correctness and F-scores (Li et al., 2024, Zhao et al., 2024, Chen et al., 2024, Zhang et al., 10 Nov 2025).
• Editability and Parameter Efficiency: Model size (count of stored bases/parameters), editing latency, compile rates, and regeneration speedups (Li et al., 2024, Raji et al., 18 Jan 2026).
• Semantic and Program Validity: ULIP (text–3D alignment), program format accuracy (syntactic and schema pass rates), CLIP/CLIP-Language scores, and user study results (Raji et al., 18 Jan 2026, Liu et al., 5 Feb 2026, Xu et al., 28 Jan 2025).
• Controllability/Scalability: Demonstrated city- or scene-scale assembly; support for sparse view or limited supervision settings (Li et al., 2024, Zhang et al., 2024).
• Comparative performance: Procedural+neural hybrids show higher physical realism, stability, and semantic alignment than pure geometry or unconstrained deep generative models, as demonstrated in ablations and user ratings (Li et al., 2024, Liu et al., 5 Feb 2026, Chen et al., 2024, Zhang et al., 10 Nov 2025).

A summary comparison for model efficiency from Proc-GS:

Model	Gaussians	PSNR	SSIM
3D-GS	1,238k	27.54	0.910
Proc-GS	291k	27.68	0.917

Such summaries expose the compactness and effectiveness of procedural-asset sharing and rule-based model assembly (Li et al., 2024).

6. Applications, Extensions, and Ongoing Challenges

Procedural 3D synthesis underpins applications including:

• Urban and Architectural Modeling: Automated, parameterizable city, building, and infrastructure assembly for simulation and visual effects; e.g., CityGenAgent, CityX, Proc-GS (Liu et al., 5 Feb 2026, Zhang et al., 2024, Li et al., 2024).
• Synthetic Data Generation: Generating diverse, annotated datasets for training computer vision models, with explicit control over scene content and diversity (Tsirikoglou et al., 2017).
• Real-time Content Authoring: Interactive editing and language-based authoring of 3D objects and scenes for graphics, design, and gaming, with frameworks supporting incremental modification (Raji et al., 18 Jan 2026, Hayashi et al., 6 Oct 2025).
• Self-supervised Representation Learning: Procedurally generated datasets for scalability in pretraining, with strong transfer to real 3D analysis tasks (Chen et al., 2024).
• Text-guided and Image-guided Generation: Integration of LLMs and vision-language alignment metrics for NL-driven synthesis and editing of parametric 3D assets (Liu et al., 5 Feb 2026, Hayashi et al., 6 Oct 2025).

Ongoing challenges include expanding grammar/primitive expressivity to better support organic and non-manifold shapes, integrating richer physics or real-world priors into procedural rules, and scaling neural-procedural hybrids for unbounded scenes with tighter semantic control (Zhao et al., 2024, Chen et al., 2024, Zhang et al., 10 Nov 2025, Wen et al., 8 May 2025).

7. Synthesis: Trends and Future Directions

The procedural 3D synthesis landscape is rapidly evolving toward:

• Hybridization: Increasing fusion of explicit procedural representations and neural generative/inverse models for fidelity, editability, and learning efficiency (Li et al., 2024, Zhao et al., 2024, Zhang et al., 10 Nov 2025).
• Interactive and Language-driven Synthesis: LLM-integrated authoring and editing pipelines for 3D content from natural language, coupled with engine-agnostic, interpretable program representations (Raji et al., 18 Jan 2026, Hayashi et al., 6 Oct 2025, Liu et al., 5 Feb 2026).
• Scalability and Real-time Control: Optimized execution engines enabling city-scale or domain-scale scene generation in minutes and sub-second editing responsiveness for asset families (Li et al., 2024, Zhang et al., 2024).
• Evaluation and Benchmarking: Refinement of task- and user-aligned evaluation protocols to measure not just fidelity and efficiency, but also editability, control, and semantic/human alignment (Raji et al., 18 Jan 2026, Liu et al., 5 Feb 2026, Li et al., 2024).

By unifying algorithmic rule systems, neural inversion/generation, and user-facing control, procedural 3D synthesis establishes a robust, modular, and efficient foundation for the vast, editable, and semantically aligned 3D virtual environments demanded by modern applications (Li et al., 2024, Raji et al., 18 Jan 2026, Wen et al., 8 May 2025).