---
title: 'HiCoGen: Hierarchical Text-to-Image Synthesis'
url: https://www.emergentmind.com/topics/hicogen
type: topic
---

# HiCoGen: Hierarchical Text-to-Image Synthesis

HiCoGen is a Hierarchical Compositional Generative framework for text-to-image synthesis designed to address the compositional and relational failures of conventional diffusion models when confronted with complex prompts involving multiple objects and hierarchical structures. Built upon a novel Chain-of-Synthesis (CoS) paradigm, HiCoGen leverages large language model-driven prompt decomposition, an exploration-enhanced diffusion process, and reinforcement learning (RL) guided by a hierarchical reward, achieving state-of-the-art results in compositional accuracy and concept fidelity [2511.19965].

## 1. Chain-of-Synthesis Paradigm for Hierarchical Generation

HiCoGen departs from traditional single-shot text-to-image generation, which encodes a complex prompt $\mathcal O$ and expects the diffusion model to resolve all objects, attributes, and relationships in one forward pass. Such monolithic approaches degrade in performance as prompt complexity increases, manifesting in concept omission and incorrect relational bindings.

HiCoGen instead applies a "chain-of-thought" strategy by using an LLM to decompose each prompt into $n$ subject-centric subprompts $P^{(i)} = \{c^{(i,1)}, c^{(i,2)}, ..., c^{(i,m_i)}\}$, representing minimal semantic units per subject. The diffusion model then synthesizes each subprompt in an iterative sequence:
- Step 1: Generate $I^{(1)}$, representing the first subject.
- Step 2: Take its denoised latent $z_0^{(1)}$ as a fixed visual context, concatenate with $P^{(2)}$, and generate $I^{(2)}$.
- Repeat for subprompts $1, ..., n$, each time providing all prior denoised latents $\{z_0^{(1)}, ..., z_0^{(i-1)}\}$ as visual anchors.

In the model architecture (e.g., DiT Transformer), this is implemented by replacing prompt tokens with $\text{concat}(P^{(i)}, z_t, z_0^{(1)}, ..., z_0^{(i-1)})$ at each CoS step, allowing previous compositional decisions to be "locked in" visually and reducing text encoder interference.

## 2. Diffusion Process Modification and Decaying Stochasticity

Standard diffusion samplers are inadequately explorative, often collapsing sample diversity early and limiting the effectiveness of RL. HiCoGen introduces stochasticity modulation by embedding a time-dependent term $\eta(t)$ in the reverse-time SDE:
$$
\mathrm{d}z_t = [f(z_t, t) - g(t)^2 \nabla_{z_t} \log p_t(z_t)] \mathrm{d}t + g(t)\, \eta(t)\, \mathrm{d}w_t
$$
where $w_t$ is a Wiener process.

The core theoretical contribution (Theorem 1) establishes that maximizing final-time diversity $\mathrm{Tr}(\mathrm{Cov}[z_0])$, subject to a fixed noise budget, is optimally achieved by concentrating stochasticity $\eta(t)$ at early timesteps (i.e., decaying $\eta(t)$ monotonically as $t \to 0$).
In practice, HiCoGen implements
$$
\eta(t) = \eta_{\min} + \frac{1}{2}(\eta_{\max} - \eta_{\min}) \left[1 + \cos\left(\pi \frac{T_{\max} - t}{T_{\max}}\right)\right]
$$
ensuring maximal exploration during "global-structure" inference and reduced stochasticity during fine-grained refinement. This approach enables exploration required by RL while preserving sample quality.

## 3. Reinforcement Learning with Hierarchical Reward Decomposition

HiCoGen models generation at each CoS step as a Markov decision process (MDP), using a Grouped Proximal Policy Optimization (GRPO) RL algorithm:
- **State** $s_{t,i}$: current noisy latent $z_{t,i}$, plus CoS context $\{P^{(1\ldots i)}, z^{(1\ldots i-1)}_0\}$.
- **Action** $a_{t,i}$: stochastic denoising update (noise injection at $t$).

After $n$ rollouts (full diffusion trajectories), scalar rewards $r_i$ are computed via a three-tier hierarchical reward model:
- **Global reward** ($R_\mathrm{global}$): weighted sum of CLIP-score alignment ($S_\mathrm{clip}$) and learned human-preference score ($S_\mathrm{hps}$, HPSv2).
- **Subject-specific reward** ($R_\mathrm{subject}$): for each detected subject (via GroundingDino), measures cosine similarity between DINOv2 embeddings of the generated and reference crops, and VLM-prompted attribute fidelity.
- **Relationship reward** ($R_\mathrm{relationship}$): VLM-prompted scores verifying spatial/interaction relationships.
Total reward: $R_\mathrm{total} = R_\mathrm{global} + R_\mathrm{subject} + R_\mathrm{relationship}$.

RL updates use standardized advantages and a clipped surrogate objective as in PPO, enabling stable training guided by compositional and relational correctness.

## 4. Architectural Adaptations for HiCoGen

HiCoGen builds upon a Transformer-based diffusion backbone (such as DiT or FLUX) with several core modifications:
- **Multi-stage Conditioning**: At each CoS step, model inputs are augmented to include the current subprompt, noisy latent, and all prior denoised latents via concatenation. This is implemented with prefix-token cross-attention in the DiT decoder and learned positional embeddings signaling previous subject completion.
- **Low-rank Subject-driven LoRA Adapters**: Each subprompt is modeled with a LoRA adapter, which is finetuned per subject and then frozen for subsequent steps, reinforcing per-subject fidelity and preventing catastrophic forgetting.
- **Exploration-Enhanced Sampling**: The decaying stochasticity schedule is integrated into ODE-style Rectified Flow, ensuring sufficient trajectory diversity for RL across each CoS step.

These design strategies enable scalable hierarchical scene construction without the object-omission and relational confusion typical of single-pass architectures.

## 5. HiCoPrompt: Hierarchical-Compositional Benchmark and Empirical Results

To support rigorous evaluation, HiCoGen introduces HiCoPrompt, a dedicated benchmark for hierarchical, compositional, text-to-image tasks:
- **Dataset Composition**: 12,000 RL training prompts and 3,000 test prompts, each describing 4–12 subjects with detailed attributes and explicit inter-subject relations (e.g., "A knight holding a raven perched on his gauntlet next to a squire").
- **Evaluation Metrics** (scored via GPT-4o prompting):
  - **Acc_exist**: Proportion of prompted objects rendered ("concept coverage")
  - **Acc_attribute**: Attribute fidelity—match between object appearances and described details
  - **Acc_relationship**: Compositional accuracy—degree to which spatial/interaction constraints are satisfied
  - CLIP-score and HPSv2 (human preference) are also reported

Performance summary:

| Model                 | Acc_exist | Acc_attr | Acc_rel | CLIP | HPSv2 |
|-----------------------|-----------|----------|---------|------|-------|
| Sdxl/SD3/FLUX         | 0.27–0.52 | 0.08–0.60| 0.08–0.64|~0.28|~0.29 |
| Qwen-Image            | ~0.63     | ~0.69    | ~0.78   | —    | —     |
| **HiCoGen**           | 0.713     | 0.767    | 0.820   | 0.32 | 0.34  |

Empirical evidence shows HiCoGen systematically avoids object omission, assignments are correct by construction, and complex hierarchical bindings are respected. Prior benchmarks, in contrast, address only simple prompts with limited subject count and attribute depth.

## 6. Significance and Implications

HiCoGen demonstrates that decomposing complex generation into an explicit, context-carrying chain of synthesis, when coupled with RL-enabled exploration and hierarchical feedback, can produce images with unprecedented compositional fidelity in complex scenes. The combination of architectural intervention, theoretically-grounded diffusion process modulation, and fine-grained multi-level RL rewards collectively shifts the achievable frontier for text-to-image models under rich, relational prompt settings. A plausible implication is the extensibility of the CoS paradigm to other modalities or generative contexts requiring explicit, decompositional reasoning [2511.19965].

Source: https://www.emergentmind.com/topics/hicogen