---
title: 'Proc-GS: Modular 3D Gaussian Building Generation'
url: https://www.emergentmind.com/topics/proc-gs
type: topic
---

# Proc-GS: Modular 3D Gaussian Building Generation

Proc-GS is a modular reconstruction-and-assembly framework for procedural building generation with 3D Gaussian Splatting (3D-GS). It treats buildings as hierarchies of reusable facade elements—such as windows, doors, pillars, corners, and floor patterns—rather than as monolithic Gaussian scenes, and represents a building as canonical Gaussian assets plus procedural instantiation code and per-instance variation. The method is presented as the first integration of procedural modeling directly into 3D-GS, with the stated aim of combining procedural modeling’s scalability and control with 3D-GS’s high-fidelity rendering and efficient asset management for city assembly [2412.07660].

## 1. Problem setting and conceptual premise

Proc-GS is motivated by a specific limitation in city-scale 3D content creation. Buildings are primary components of cities and often exhibit strong structural repetition, yet recent city-scale generative methods may lack the geometric precision, visual fidelity, and editability required for high-quality buildings, while traditional procedural pipelines in games and visual effects depend on manually authored mesh assets and hand-designed rules that are expensive to create. Proc-GS addresses this gap by reconstructing a small set of reusable base assets from multi-view imagery and composing them procedurally, instead of reconstructing an entire building as a single flat Gaussian cloud [2412.07660].

The paper’s central premise is that city assembly depends more on asset reuse than on one-off scene reconstruction. In industrial pipelines such as Unreal Engine’s City Sample, artists already build libraries of reusable modules and combine them through procedural rules. Proc-GS aims to automate the asset creation step while preserving the procedural structure that makes assembly scalable. This positions the method between two prior paradigms: standard 3D-GS, which usually models a whole scene as a flat set of Gaussians without object decomposition, and learned or diffusion-based city generation methods, which synthesize broad urban structure from priors but may offer limited control or inconsistent geometry [2412.07660].

A recurrent misconception is to interpret Proc-GS as a monolithic generative prior over buildings or cities. In the reported formulation, it is instead a reconstruction-and-assembly framework built from explicit reusable radiance assets. Its novelty lies in proceduralizing Gaussian splatting: the learned representation is not only a rendering substrate but also an asset library that can be extracted, shared, instantiated, edited, and reassembled.

## 2. Procedural representation and Gaussian asset structure

Proc-GS decomposes a building hierarchically into levels or floors, and each level into base assets such as windows, external corners, pillars, and other facade modules [2412.07660]. The procedural code is effectively a structured string describing which asset appears where and how many times. In the appendix, raw layout descriptions are further converted into more regularized procedural expressions using GPT-4o; parentheses denote repeatable groups, and symbols marked with “\(*\)” denote scalable assets that can stretch to fit building dimensions.

A base asset is parameterized by its bounding box size \((x_e, y_e, z_e)\), a pivot location \((x_c, y_c, z_c)\) in local coordinates, and a set of transformations for its \(K\) world-space instantiations:
$$
\{[R_1, T_1, S_1], [R_2, T_2, S_2], \ldots, [R_K, T_K, S_K]\},
$$
where \(T \in \mathbb{R}^{3 \times 1}\) is translation, \(R \in \mathbb{R}^{3 \times 3}\) is rotation, and \(S \in \mathbb{R}^{3 \times 1}\) is scale. The building is therefore stored as a set of canonical assets plus instantiation transforms described by procedural code, rather than as a single scene-level Gaussian representation [2412.07660].

Within each asset, Proc-GS retains the standard 3D-GS parameterization. Each Gaussian \(i\) has parameters
$$
\{\mu_i, R_i, S_i, \alpha_i, \mathcal{C}_i\},
$$
where \(\mu_i\) is the center, \(R_i\) and \(S_i\) define covariance, \(\alpha_i\) is opacity, and \(\mathcal{C}_i\) are spherical harmonic coefficients for view-dependent color. The Gaussian density is
$$
G_i(x) = e^{-\frac{1}{2} (x-\mu_i)^T \Sigma_i^{-1} (x-\mu_i)}, \qquad
\Sigma_i = R_iS_iS_i^TR_i^T.
$$
Projected 2D Gaussians are then rasterized and alpha-composited in the usual 3D-GS manner.

The distinctive structural feature is transform-based instancing. If \(\mathcal A_i\) is the Gaussian set for base asset \(i\), then for the \(j\)-th instantiation only geometric parameters are transformed:
$$
\mu' = R^{i}_{j} \cdot S^{i}_{j}\cdot \mu + T^{i}_{j}, \qquad
R' = R^{i}_{j} \cdot R, \qquad
S' = S^{i}_{j} \cdot S.
$$
This enables repeated use of the same canonical Gaussian asset at multiple locations. The paper further emphasizes that repeated instances are not assumed to be perfectly identical: each base asset is paired with a variance asset that models local deviations in appearance and geometry. The shared base asset captures common structure; the variance asset restores fidelity that would otherwise be lost under aggressive sharing [2412.07660].

This design has an important optimization consequence. When a window module occurs many times and is visible in multiple views, gradients from all repeated observations flow back to the same canonical asset. The paper interprets this as a form of natural data augmentation and parameter sharing.

## 3. Two-stage pipeline: asset acquisition and optimization

Proc-GS is organized as a two-stage pipeline: **Asset Acquisition** and **Asset Assembly** [2412.07660]. Asset acquisition assumes access to a building’s procedural code. For synthetic data this code is available directly from the City Sample assets used to construct the MatrixBuilding dataset. For real scenes, the paper proposes a semi-automatic pipeline: first use 2D Gaussian Splatting to obtain geometrically accurate point clouds and meshes, then estimate facades, render fronto-parallel facade images, annotate 2D procedural layouts on those facades, and project them back to 3D mesh space to obtain a 3D procedural code.

Initialization depends on data modality. For synthetic scenes, if a building contains assets \(\mathcal M\) and the total initial point budget is \(N\), then asset \(i\) receives
$$
N^i = N * \frac{V^i}{\sum_{j\in\mathcal{M}} V^j}, \qquad
V^i=x^{i}_e \times y^{i}_e \times z^{i}_e.
$$
Larger assets therefore receive more initialization points. For real scenes, initialization is geometry-aware: all instantiation boxes of a given asset are transformed into world coordinates, structure-from-motion points within those boxes are collected, transformed back into the canonical local frame, concatenated, and uniformly downsampled by factor \(K\) to initialize the shared base asset [2412.07660].

Optimization uses the same rendering supervision as 3D-GS, combining \(\mathcal{L}_1\) and \(\mathcal{L}_{\text{SSIM}}\), with \(\lambda_{\text{SSIM}} = 0.2\) in experiments. The paper explicitly notes that it does not introduce bespoke regularization losses for procedural consistency, sharing, or semantics. Instead, structure is enforced operationally through shared asset parameterization, transform-based instancing, and a bounding-box control operator.

That operator is the **Bbox Adaptive Clamp**. Every 100 iterations, for both base and variance assets, Proc-GS applies two steps: if a Gaussian extends beyond a slightly enlarged soft bounding box, its scale is halved; if a Gaussian center goes beyond the hard box, it is pulled back to the boundary. In implementation, the soft box is expanded by 20 cm beyond the asset box. This is not described as a differentiable penalty. Its purpose is structural cleanliness: unconstrained 3D-GS can allow Gaussians to bleed across semantic boundaries, which is acceptable for view synthesis but harmful for modular extraction and editing [2412.07660].

Implementation details reinforce the method’s practical orientation. Proc-GS and the baseline 3D-GS are trained for 30k iterations, with densification up to 15k iterations, on a single RTX 3090. For synthetic data, the initial point budget is \(N=10\text{k}\). The 3D-GS baseline is initialized from the building assembled from randomly initialized base assets according to procedural code, which makes the comparison more aligned.

## 4. Assembly, controllability, and city-scale composition

The procedural representation provides rule-based and parametric control at multiple levels [2412.07660]. Users can manipulate the procedural code directly to change floor repetition, facade module arrangements, scalable asset spans, and building dimensions such as length, width, and height. Because the code can specify repeatable within-floor groups and repeated floor patterns across height, the paper frames the resulting design space as enabling an “infinite variety” of buildings.

Architectural plausibility is preserved by staying within the learned or provided procedural decomposition. Floor-wise organization, repeat groups, corner placements, and scalable wall spans encode regularities of real building design. Proc-GS therefore does not generate arbitrary free-form Gaussian structures; it edits and assembles within a scaffold aligned with architectural patterns. Diversity is further increased by randomly assigning variance assets during generation and by mixing base assets from different architectural sources [2412.07660].

The assembly stage extends from single buildings to cities through two submodules. The **building generator** takes extracted base assets and regular procedural code, then places and scales assets to match a specified building size. The **city layout generator** is rule-based: the user provides city boundary points and primary road endpoints; the system partitions blocks, assigns regional characteristics such as building-size distributions, generates secondary roads perpendicular to primaries, determines building positions, sizes, and topological arrangement, and places additional decorative street assets also represented with 3D-GS. The city-level system is not learned end-to-end. Learning is concentrated on modular building assets, while scalable composition is handled by procedural layout rules [2412.07660].

This division of labor is central to the method’s scope. Proc-GS is not principally a learned urban planner; it is a procedural assembly system grounded in reusable Gaussian assets. Practical applications identified in the paper include game environment creation, visual effects, digital twins, simulation for autonomous driving, embodied AI environments, and interactive urban editing workflows, especially in systems such as Houdini or Unreal Engine.

## 5. Datasets and empirical performance

The paper introduces the **MatrixBuilding** dataset, derived from 17 artist-created buildings from Unreal Engine’s City Sample [2412.07660]. These buildings mimic Chicago, New York, and San Francisco styles and include dense multi-view images, camera poses, and ground-truth procedural codes. The appendix reports substantial variation in modular complexity: some buildings have as few as 8 base assets and others as many as 90, while total instantiated asset counts range from 295 to 2831. Camera trajectories are dense, with about a 5:1 train:test ratio. Real-world validation uses three drone-captured scenes.

The central synthetic and real-scene comparison is summarized below. Values are reported as PSNR / SSIM / LPIPS, followed by Gaussian count.

| Setting | 3D-GS | Proc-GS |
|---|---|---|
| Synthetic MatrixBuilding | 27.54 / 0.910 / 0.108, 1,238k | 27.68 / 0.917 / 0.102, 291k |
| Real scenes | 27.38 / 0.858 / 0.192, 500k | 27.19 / 0.853 / 0.196, 384k |

On synthetic MatrixBuilding scenes, Proc-GS slightly improves image metrics while reducing the Gaussian count by more than \(4\times\) [2412.07660]. On real scenes, compression is smaller and quality drops slightly; the paper attributes this to more complex real-world appearance and imperfect procedural codes, while emphasizing that editability remains.

Sparse-view results are particularly informative about procedural sharing. With only 24 views, 3D-GS scores PSNR 16.93, SSIM 0.542, LPIPS 0.410, whereas Proc-GS reaches 19.70, 0.682, 0.294. With 47 views, 3D-GS obtains 20.65 / 0.688 / 0.283 and Proc-GS 23.11 / 0.795 / 0.196. At full 469 views, Proc-GS remains slightly better than 3D-GS. The paper interprets this as natural data augmentation: repeated facade elements contribute supervision to a shared canonical model [2412.07660].

The ablation study clarifies the role of each component. Vanilla 3D-GS gives 27.54 PSNR with 1238k Gaussians. Adding only procedural code sharing cuts model size to 87k Gaussians but drops PSNR to 25.54. Adding the bbox clamp without variance further reduces quality to 24.40 with the same 87k count. Adding variance assets raises quality to 27.68 while using 291k Gaussians. The reported design logic is therefore explicit: procedural sharing provides compactness, clamping yields clean composable assets, and variance assets recover realism lost by over-sharing [2412.07660].

For city-scale generation, Proc-GS is evaluated with **Camera Error (CE)** and **Depth Error (DE)** against Persistent Nature, SceneDreamer, CityDreamer, and GaussianCity. The reported scores are 86.371 / 0.109 for Persistent Nature, 0.186 / 0.216 for SceneDreamer, 0.060 / 0.096 for CityDreamer, 0.057 / 0.090 for GaussianCity, and **0.049 / 0.032** for Proc-GS, best on both metrics [2412.07660]. The qualitative comparison is described as showing better architectural detail in aerial and street-level views, which the paper attributes to city scenes being assembled from explicit reusable building modules rather than hallucinated monolithically.

## 6. Limitations, scope, and neighboring methods

The paper is explicit about several limitations [2412.07660]. The current pipeline still assumes procedural code is available, and for real-world scenes that code is only semi-automatic: facade extraction is automated, but facade-level procedural annotation remains manual. Extracting high-quality base assets from sparse views or single images remains difficult, although Proc-GS is more data-efficient than standard 3D-GS in the reported sparse-view regime. The city layout generator is rule-based and does not model aesthetics or urban functionality deeply. Another implicit limitation is representational bias: the method is best suited to architecture with strong repetition and modularity, and irregular free-form geometry fits the procedural assumption less naturally.

These limitations clarify the method’s scope. Proc-GS is not a free-form scene synthesizer, nor is it a learned end-to-end city generator. It is a procedural Gaussian asset framework whose core contribution is the decomposition of buildings into canonical modules, instantiation transforms, and variance residuals. The paper suggests future use of LLMs for more realistic and functional layout generation, but that direction is not part of the reported method [2412.07660].

Proc-GS should also be distinguished from similarly named Gaussian-splatting variants. **Proxy-GS** addresses occlusion-aware acceleration for MLP-based Gaussian splatting via a proxy mesh [2509.24421]. **ProtoGS** reduces primitive count through Gaussian prototypes and rendering-guided clustering [2503.17486]. **PCGS** targets progressive compression of anchor-based 3D Gaussian Splatting [2503.08511]. **GS-ProCams** concerns projector-camera systems with 2D Gaussian representations and differentiable physically based rendering [2412.11762]. Proc-GS differs from all of these in that its primary object is procedural building generation and city assembly from reusable Gaussian facade assets.

In that sense, Proc-GS marks a specific shift in how Gaussian splatting is used. Rather than treating 3D-GS as a terminal scene representation, it turns it into an asset representation for modular urban content. The reported contribution is not only memory reduction or rendering fidelity, but the connection of two established but previously separate workflows: procedural modeling’s scalability and control, and Gaussian splatting’s explicit photorealistic rendering [2412.07660].

Source: https://www.emergentmind.com/topics/proc-gs