StructDRAW Prior in GNM Architecture

• The StructDRAW prior is an autoregressive mechanism that models global latent variables to enable density-aware generation of structured scenes.
• It employs ConvLSTM and MLP-based interaction layers to capture long-range dependencies and simulate complex, multi-modal scene characteristics.
• Empirical evaluations show that integrating StructDRAW significantly boosts scene structure accuracy and sample quality compared to traditional Gaussian priors.

StructDRAW is an autoregressive, interaction-enabled prior over the global latent variables in Generative Neurosymbolic Machines (GNM), providing the expressive capacity necessary for density-aware, structured scene generation. Within the GNM architecture, StructDRAW replaces the conventional simple Gaussian prior with a more flexible mechanism that enables both rich inter-object dependencies and effective modeling of world densities—key requirements for effective generative modeling of compositional and structured scenes.

1. Formal Description and Architectural Role

StructDRAW operates within GNM's two-layer latent hierarchy:

• The global distributed latent, denoted $\mathbf{z}^g$ ("scene code"), captures high-level, scene-wide factors.
• The structured symbolic latent $\mathbf{z}^s$ encodes object-level or component-centric representations (presence, location, depth, appearance, etc.).

The generative process is: $$p_\theta(\mathbf{x}) = \int p_\theta(\mathbf{x}\mid \mathbf{z}^s) \, p_\theta(\mathbf{z}^s\mid \mathbf{z}^g) \, p_\theta(\mathbf{z}^g) \, d\mathbf{z}^g \, d\mathbf{z}^s$$

StructDRAW parameterizes the prior $p_\theta(\mathbf{z}^g)$ via an autoregressive mechanism, constructing an abstract latent feature map $\mathbf{f}$ over $L$ DRAW steps. At each step $\ell$:

• A conditional sample $$\mathbf{z}_\ell^g \sim p_\theta(\mathbf{z}_\ell^g \mid \mathbf{z}_{<\ell}^g)$$ is produced.
• The feature map slice $\mathbf{f}_\ell$ is generated as a function of the decoder RNN hidden state (typically ConvLSTM-based), projected by a CNN.
• A global interaction layer (typically an MLP) mixes information across slots, supporting full scene-wide dependencies beyond local convolutional neighborhoods.

The outputs are recursively combined: $\mathbf{f} = \sum_{\ell=1}^L \mathbf{f}_\ell$ Once $\mathbf{f}$ is constructed, it parameterizes the symbolic latent map $\mathbf{z}^s$ from which the image is ultimately generated.

2. Mathematical Formulation and Algorithm

The generative prior over $\mathbf{z}^g$: $$p_\theta(\mathbf{z}^g) = \prod_{\ell=1}^{L} p_\theta(\mathbf{z}_\ell^g \mid \mathbf{z}_{<\ell}^g)$$ with feature map construction: $$\mathbf{f} = \sum_{\ell=1}^L \mathrm{CNN}(\mathbf{h}_{\mathrm{dec},\ell})$$ where $\mathbf{h}_{\mathrm{dec},\ell}$ denotes the decoder RNN hidden state.

Posterior inference for $\mathbf{z}^g$ is performed similarly in an autoregressive, slot-wise manner: $$q_\phi(\mathbf{z}^g \mid \mathbf{x}) = \prod_{\ell=1}^L q_\phi(\mathbf{z}_\ell^g \mid \mathbf{z}_{<\ell}^g, \mathbf{x})$$ Both prior and posterior are parameterized with ConvLSTM cores and MLP-based interaction layers to provide long-range, inter-slot dependencies.

A high-level algorithmic summary is:

Initialize hidden states; For each DRAW step l=1..L: - Prior p(z_l^g) from MLP (on decoder state) - If inference: - Update encoder ConvLSTM with previous decoder state and encodings - Compute posterior q(z_l^g) - Sample z_l^g ~ q(.) - Else (generation): sample z_l^g ~ p(.) - Update decoder ConvLSTM with z_l^g - Update feature map f_l = f_{l-1} + CNN(decoder state)

This process enables the GNM model to produce a feature map $\mathbf{f}$ that captures scene structure in latent space, driving the subsequent symbolic slot-level generation.

3. Hierarchical Modeling and Scene Structure

StructDRAW serves as the prior for the global latent layer in the GNM hierarchy. Its autoregressive, interaction-enabled construction:

• Allows modeling of complex, multi-modal, and correlated distributions over scenes ($\mathbf{z}^g$).
• Translates into structured object layouts, as $\mathbf{f}$ parameterizes the symbolic map $\mathbf{z}^s$ controlling explicit object variables.
• Decouples the number of autoregressive steps $L$ from the number of scene objects, enabling scalability to complex scenes with only a few global steps.

This arrangement provides both highly variable scene-level arrangements (controlled by StructDRAW/global code) and modular, interpretable object representations (via $\mathbf{z}^s$).

4. Comparative Analysis with Other Structured Priors

StructDRAW contrasts sharply with alternative structured priors:

• Models such as Space, AIR, Slot-Attention, and SCALOR utilize fixed, independent symbolic priors for object slots (e.g., independent Bernoulli or Gaussian for presence and location), lacking global latent factors and density-aware sampling.
• GENESIS introduces an autoregressive prior over slots but captures less expressive inter-object dependencies and achieves inferior generative quality.
• ConvDRAW and PixelCNN leverage autoregressive modeling at the pixel level, without structured object representations.

StructDRAW's key differentiators are:

• Explicit modeling of global, inter-object dependencies at the abstract feature level.
• Maintenance of compositional decomposability for object slots.
• Hierarchical architecture with distributed global and structured symbolic layers.
• Scalability due to decoupled DRAW steps and object count.

5. Empirical Evaluation and Performance

GNM equipped with StructDRAW demonstrates quantitative and qualitative superiority over both structured and unstructured baselines. Results across datasets (Arrow room, MNIST-4, MNIST-10) include:

Dataset	Model	Scene Structure Accuracy	D-Steps	Log Likelihood
ARROW	GNM (StructDRAW)	0.976	11099	33809
ARROW	GENESIS	0.092	1900	33241
ARROW	ConvDRAW	0.176	3800	33740
MNIST-10	GNM (StructDRAW)	0.824	2760	10450
MNIST-10	GENESIS	0.000	160	9560
MNIST-10	ConvDRAW	0.000	1200	10544
MNIST-4	GNM (StructDRAW)	0.984	3920	10964

Key findings:

• Substituting StructDRAW with a simple Gaussian prior (“GNM-Gaussian”) results in a marked reduction in structure accuracy (e.g., 0.784 vs. 0.976 on ARROW, 0.096 vs. 0.824 on MNIST-10).
• Ablating the MLP-based interaction layer (“GNM-NoMLP”) degrades results, indicating the importance of global interactions within StructDRAW.
• Addition of an MLP to ConvDRAW does not close the gap, highlighting the structural advantages of GNM's latent hierarchy.

Qualitatively, StructDRAW-based generations preserve scene logic and inter-object relationships, unlike competitors which yield unrealistic or incoherent samples.

6. Flexibility, Scalability, and Interpretability

StructDRAW achieves expressiveness and flexibility via its autoregressive prior and global interaction layers, accommodating dependencies in object counts, positions, and scene-level properties. Density-aware sampling ensures that generated scenes reflect the empirical structure of training data, avoiding the artificial configurations produced by purely symbolic priors.

Efficiency is realized through operating at lower spatial resolution in feature space, requiring significantly fewer steps (e.g., four StructDRAW steps suffice for scenes with ten objects). The compositional slot representations ($\mathbf{z}^s$) remain interpretable and modular, while global latent relationships support complex scene-wide semantics.

Traversals in either $\mathbf{z}^g$ or $\mathbf{z}^s$ space induce consistent, meaningful changes in generated scenes, demonstrating effective disentanglement and controllability. The GNM framework with StructDRAW also preserves high accuracy in detection, localization, and classification, paralleling the structure-focused models in representational tasks.

7. Summary

StructDRAW underpins the GNM model's ability to bridge distributed and symbolic approaches, acting as an autoregressive, interaction-enabled prior over global latent structure. This enables data-driven, density-aware, interpretable scene generation surpassing state-of-the-art structured and unstructured generative models in both accuracy and sample quality. The combination of global scene modeling and compositional symbolic representation provides a robust foundation for further research in object-centric generative modeling and neurosymbolic machine intelligence (Jiang et al., 2020).