DogLayout: Diffusion GAN for Discrete Layouts

• The paper introduces DogLayout, a novel framework that integrates denoising diffusion processes with GAN-style training to overcome challenges in mixed discrete and continuous layout generation.
• The model leverages Transformer-based generators and discriminators to achieve high sample efficiency and robust structural fidelity, as demonstrated on the PubLayNet dataset.
• DogLayout achieves significant performance gains by reducing sampling steps and overlap metrics while maintaining competitive FID and IoU scores, offering practical value for layout synthesis.

DogLayout is a generative framework for discrete and continuous layout generation that integrates denoising diffusion processes with adversarial (GAN-style) training. Its design targets challenges inherent in layout generation tasks where layouts are described by both discrete categorical labels (e.g., object classes) and continuous geometric parameters (e.g., bounding boxes). DogLayout addresses the inefficiency of conventional @@@@1@@@@ in sampling, as well as the limitations of GANs with discrete data, by conditioning GAN training on a denoising diffusion process while maintaining full end-to-end differentiability for both discrete and continuous layout components (Gan et al., 2024).

1. Model Architecture and Design

DogLayout constructs a hybrid generative model, narrowing the gap between the control and sample quality of diffusion models and the sampling efficiency of GANs. The architecture consists of a generator $G_\theta$ and a discriminator $D_\phi$, both built around Transformer encoders and fully connected (FC) projection branches.

• Generator ($G$): Accepts a noisy layout $x_t \in \mathbb{R}^{M \times (N+4)}$, corresponding to $M$ elements, $N$ label logits (pre-softmax), and $4$ box parameters per element. Latent noise $z \sim \mathcal{N}(0,I)$ with shape $(M \times d_z)$ is jointly embedded with $x_t$. The model stacks $L_g$ layers of a multi-head Transformer encoder, producing an output $x'_0 \in \mathbb{R}^{M \times (N+4)}$ — unnormalized logits for discrete labels and continuous coordinates.

h_z = FC_z(z) # (M × d) h_x = FC_x(x_t) # (M × d) h_cat = concat(h_z, h_x) # (2M × d) or interleaved tokens h_out = TransformerEnc_g(h_cat) # L_g layers, H_g heads x0_pred = FC_out(h_out) # (M × (N+4))

• Discriminator and Decoder ($D$ + De): Receives a tuple: either a real pair $(x_{t-1}, x_t)$ or a fake pair $(x'_{t-1}, x_t)$ produced by feeding $x'_0$ through the forward diffusion kernel. Inputs are FC-embedded, concatenated, and processed via $L_d$ Transformer layers with a global special token $h_s$. The discriminator outputs a real/fake score via $p = D(h) \in [0,1]$. An attached decoder head further reconstructs the original $x_0$ from the global token, enforcing structural awareness and preventing trivial solutions.

inp = concat(x_prev, x_t) # (2M × (N+4)) h_in = FC_D(inp) # (2M × d) h_all = TransformerEnc_d([h_s; h_in]) h_glob, h_tok = split(h_all) # h_glob = (1 × d) p_real = sigmoid(FC_logit(h_glob)) x0_rec = De(h_glob)

The model eschews non-differentiable operations (e.g., argmax) during training. Instead, both $G$ and $D$ manipulate real-valued logits, with discrete labels recovered only at inference via $\mathrm{arg\,max}_i\,\mathrm{softmax}(x_0[i])$.

2. Denoising Diffusion Process

DogLayout leverages a Gaussian forward–reverse process inherited from Denoising Diffusion Probabilistic Models (DDPMs), with modifications for adversarial sampling.

• Forward process: For noise schedule $\{\beta_t\}$, set $\alpha_t = 1 - \beta_t$, $\bar \alpha_t = \prod_{s=1}^t \alpha_s$.

$$q(x_t|x_{t-1}) = \mathcal{N}(x_t; \sqrt{\alpha_t} x_{t-1}, \beta_t I)$$

or equivalently,

$$x_t \sim \mathcal{N}(\sqrt{\bar\alpha_t} x_0, (1 - \bar\alpha_t)I)$$

• Reverse process: Standard DDPMs use parameterized Gaussians:

$$p_\theta(x_{t-1}|x_t) = \mathcal{N}(x_{t-1}; \mu_\theta(x_t, t), \Sigma_\theta(x_t, t))$$

DogLayout instead adversarially matches the conditional reverse kernel for small $T$, relying on the closed-form posterior:

$$q(x_{t-1}|x_t, x_0) = \mathcal{N}\left(x_{t-1}; \tilde\mu(x_t, x_0), \tilde\beta_t I\right)$$

with

$$q(x_{t-1}|x_t, x_0) = \frac{q(x_t|x_{t-1},x_0)q(x_{t-1}|x_0)}{q(x_t|x_0)}$$

The diffusion noise is scheduled linearly ($\beta_t \in [10^{-4}, 0.02]$), identically for all channels.

• Discrete label handling: The forward kernel treats label channels as real-valued logits until final argmax extraction, maintaining overall differentiability.

3. Objective Functions and Optimization

DogLayout's loss formulation amalgamates adversarial and denoising objectives. The core losses are:

• Discriminator loss:

$$L_D = \sum_{t=1}^T \mathbb{E}_{q(x_t)}\left[ \mathbb{E}_{q(x_{t-1}|x_t)}[-\log D(x_{t-1},x_t)] + \mathbb{E}_{x'_{t-1}\sim p_\theta(x_{t-1}|x_t)}[-\log(1-D(x'_{t-1},x_t))] + \lambda_\mathrm{rec}\cdot \mathbb{E}_{q(x_0|x_t)}[L_\mathrm{rec}(x_0, De(h))] \right]$$

with $L_\mathrm{rec}$ as an $\ell_2$ or $\ell_1$ loss, $\lambda_\mathrm{rec} \approx 1$.

• Generator loss:

$$L_G = \sum_{t=1}^T \mathbb{E}_{q(x_t),z}[-\log D(x'_{t-1}(x_t, z), x_t)]$$

Optionally, $L_\mathrm{rec}$ is added to stabilize generator predictions.

• Min–max training objective:

$$\min_\theta \max_\phi\ L_\mathrm{GAN}(G_\theta, D_\phi) + L_\mathrm{diff}(G_\theta) + L_\mathrm{rec}(D_\phi)$$

$L_\mathrm{diff}$ is implicit in the sampling of $x_t$ and $x_{t-1}$.

Regularization and architectural features such as the decoder in $D$ are crucial for enforcing nontrivial structure learning, as pure GANs on discrete layouts become trivially degenerate.

4. Training Algorithm and Procedures

Practical training is characterized by short diffusion chains ($T\in\{4,8,12\}$) and high-throughput batch sizes.

for epoch in 1..E: for batch x0 ∼ data: # 1) sample a random t ∈ [1, T] t = Uniform({1,…,T}) # 2) noise up x_{t-1} → x_t ε ∼ N(0, I) x_t = sqrt(α_t) * x_{t-1} + sqrt(β_t) * ε # 3) G predicts clean x'₀ from (x_t, z) z ∼ N(0, I) x0_pred = G(x_t, z) # 4) compute fake x'_{t-1} via q(x_{t-1}|x_t, x0_pred) μ_q, σ_q² = posterior_params(x_t, x0_pred, t) x_prev_fake ∼ N(μ_q, σ_q²) # 5) D update: real vs fake & reconstruction loss_D = – log D(x_{t-1}, x_t) – log(1–D(x_prev_fake, x_t)) + λ_rec · L_rec(x0, De(h)) D.optimizer.zero_grad(); loss_D.backward(); D.optimizer.step() # 6) G update: fool D loss_G = – log D(x_prev_fake, x_t) G.optimizer.zero_grad(); loss_G.backward(); G.optimizer.step()

No explicit time embedding is used; noise scale suffices. Warm-up of $G$ with reconstruction-only loss may aid early stability. The decoder in $D$ inhibits shortcut solutions on discrete channels.

5. Sampling and Inference

DogLayout enables sampling chains up to $175\times$ shorter than standard diffusion models by operating with as few as $T=4$ steps.

x = N(0, I) for t in T..1: # 2) predict clean layout z ∼ N(0, I) x0_pred = G(x, z) # 3) compute posterior mean & var μ_q, σ_q² = posterior_params(x, x0_pred, t) # 4) step to x_{t-1} if t > 1: x = μ_q + sqrt(σ_q²) * ε # ε ∼ N(0, I) else: x = x0_pred # final labels = argmax(softmax(x[:,0:N])) boxes = x[:,N:N+4]

Posterior parameters follow the DDPM closed-form.

6. Empirical Results

DogLayout demonstrates performance improvements in layout quality and efficiency. Quantitative experiments on PubLayNet are summarized below.

Model	Overlap (C→S+P) ↓	FID (C→S+P) ↓	Max IoU ↑	T/sample (ms)
LayoutGAN++	22.8	—	—	0.0327 (T=1)
LayoutDM	16.43	8.96	0.308	23.3 (T=50)
DogLayout	9.59	9.62	0.287	0.133 (T=4)

Additional metrics:

• DogLayout overlap (C+S→P): 12.5
• DogLayout sampling: $T=4: 0.133$ ms, $T=8: 0.255$ ms, $T=12: 0.377$ ms per sample
• GANs without diffusion are unstable on discrete labels (discriminator accuracy saturates).

Ablation for $T$ shows a sweet spot at $T=8$ for unconditional PubLayNet: FID drops from $199$ ($T=2$) to $14.7$ ($T=8$), with alignment improving in parallel.

7. Implementation, Limitations, and Future Extensions

• Hyperparameters: Generator and decoder transformers use $L_g=4$ layers, $H_g=8$ heads; discriminator uses $L_d=8$ layers, $H_d=4$ heads; all with $d=256$, feedforward width $2048$, GELU activations, and LayerNorm. Adam optimizer with $\beta_1=0.5$, $\beta_2=0.999$, learning rate $1e^{-5}$, batch $512$, $\approx$200 epochs.
• Design choices: No explicit time embedding; noise magnitude conveys diffusion step.
• Limitations: Automatic metrics underperform human design, and no image-layout cross-attention is present; content-aware layout generation is a direction for future research. The approach may generalize to other mixed discrete+continuous structured domains.

For reproducibility, precomputing $\bar\alpha_t$ is suggested, and warm-up with pure reconstruction loss stabilizes generator learning in early epochs (Gan et al., 2024).