KB-DMGen: Pose-Guided Human Image Generation

• KB-DMGen is a pose-guided human image generation method that combines global semantic retrieval via a visual codebook with adaptive local pose control.
• It employs a Knowledge-Based adapter to extract semantic priors from learned image features, guiding overall image realism and alignment with textual prompts.
• The Dynamic pose Masking module adaptively emphasizes pose-relevant regions to balance structural fidelity with semantic completeness during the denoising process.

Searching arXiv for KB-DMGen and closely related pose-guided human image generation papers. KB-DMGen, short for Knowledge-Based Global Guidance and Dynamic pose Masking for human image Generation, is a pose-guided human image generation method built on Stable Diffusion v1.5 for generating human images from a text prompt and a pose prior without requiring a source person image. Its central premise is that sparse pose control alone is insufficient for realistic human synthesis: high pose fidelity must be coupled with global semantic and visual coherence. To this end, the method combines a Knowledge-Based global guidance module (KB), implemented as retrieval from a learned visual semantic codebook, with a Dynamic pose Masking module (DM) that adaptively emphasizes pose-relevant regions during denoising. On the Human-Art benchmark, the method reports the best AP and CAP values among the compared systems (Liu et al., 26 Jul 2025).

1. Research setting and intended scope

KB-DMGen is situated in pose-guided human image generation rather than classical pose transfer. The task input is a text prompt together with a pose prior $c_p$, represented as a pose image or skeleton condition, and the output is a generated human image $\hat{x}$ whose body configuration follows the pose while remaining semantically aligned with the text and globally realistic. The paper explicitly contrasts this setting with older source-image-based HIG / pose transfer methods, which depend on a source image and are therefore less suitable for free-form text-plus-pose generation (Liu et al., 26 Jul 2025).

The motivating claim is that recent pose-guided diffusion systems often optimize primarily for pose fidelity while neglecting global structure, semantic completeness, and whole-image quality. In that framing, sparse skeleton conditions are too weak to guarantee plausible full-body appearance, clothing, scene-level realism, or coherent global semantics. KB-DMGen is designed as a corrective to that imbalance: the KB branch supplies global semantic priors, whereas the DM branch sharpens local structural control.

A common misconception is to read “knowledge base” in this context as a symbolic database or knowledge graph. In KB-DMGen, the “KB” is instead a visual semantic codebook learned from image features. Another possible misunderstanding is to treat the model as a pose-transfer system; the paper defines it as pose-guided human image generation without requiring a source person image (Liu et al., 26 Jul 2025).

2. Diffusion architecture and generation pipeline

The system integrates three parts: a Diffusion Pipeline, a Knowledge-Based Adapter, and a Dynamic pose Mask Adapter. The diffusion backbone is Stable Diffusion, consisting of a VAE encoder $E$, a VAE decoder $D$, and a latent-space denoising U-Net. The image is encoded into latent space as $z_0$, and denoising is conditioned on both text and pose (Liu et al., 26 Jul 2025).

Component	Inputs	Function
Diffusion Pipeline	text prompt, pose condition	latent denoising and image synthesis
Knowledge-Based Adapter	text embedding, visual codebook	global semantic guidance
Dynamic pose Mask Adapter	pose image, timestep embedding	adaptive emphasis on pose-related regions

The pose condition is written as

$c_p \in \mathbb{R}^{H \times W \times C},$

while the text condition $c_t$ is extracted by a CLIP text encoder. The denoising objective is the standard latent diffusion noise-prediction loss: $$\mathcal{L}_d = \mathbb{E}_{\mathbf{z}_t, \epsilon_t \sim \mathcal{N}(\mathbf{0}, \mathbf{I})} \left[ \left\| \epsilon_t - \epsilon_\theta(\mathbf{z}_t, \mathbf{c}_t, \mathbf{c}_p, t) \right\|^2 \right].$$

The pipeline can be summarized as follows. Text is encoded into $c_t$; the text embedding queries the knowledge base and retrieves semantic codebook features; the pose image yields a binary foreground mask; at each denoising step, a timestep embedding modulates that mask into a dynamic soft mask; the U-Net then denoises under the joint influence of text conditioning, pose conditioning, KB guidance, and masked attention. During inference, the method uses DDIM with 50 timesteps (Liu et al., 26 Jul 2025).

This architecture suggests a two-level control strategy: top-down semantic regularization from the KB branch and bottom-up spatial control from the DM branch.

3. Knowledge-Based global guidance

The Knowledge Base is a trainable visual semantic codebook

$$e = \{e_1, e_2, \dots, e_K\}, \quad e \in \mathbb{R}^{K \times C},$$

where $\hat{x}$0 is the number of codebook entries and $\hat{x}$1 is the feature dimension. These entries serve as discrete visual-semantic prototypes. The paper constructs the KB in two stages (Liu et al., 26 Jul 2025).

In Stage 1, a CLIP image encoder extracts dense image features

$\hat{x}$2

with $\hat{x}$3. For each token $\hat{x}$4, squared Euclidean distance to each codebook entry is computed: $\hat{x}$5 The nearest codebook entry is selected by

$\hat{x}$6

yielding the discrete index vector $\hat{x}$7. These assignments are turned into a one-hot matrix $\hat{x}$8, and the quantized feature map is

$\hat{x}$9

The codebook is trained using the reconstruction objective

$E$0

In Stage 2, the paper trains a classifier from text to visual token assignments. A pretrained text encoder produces a global text embedding

$E$1

which is projected into token logits

$E$2

where the paper states $E$3. Supervision comes from the codebook assignments learned in Stage 1, using

$E$4

At inference, text encoding is used to retrieve semantic codebook features from this visual KB, and those retrieved features are injected into Stable Diffusion as global semantic guidance. The intended effect is not only stronger semantic completeness but also better full-image realism, since sparse pose conditions do not adequately constrain clothing, body completeness, or contextual plausibility. In ablations, the paper also reports an additional semantic enhancement denoted D{data}C, described as a decomposition-and-combination procedure over language input, which produces a further gain when added on top of KB and DM (Liu et al., 26 Jul 2025).

4. Dynamic pose masking

The DM module is the local structural control mechanism. From the pose image, the model derives a binary mask

$E$5

where $E$6 denotes foreground or human region and $E$7 denotes background. This is then modulated by diffusion timestep information. Given a timestep embedding

$E$8

a lightweight MLP produces a soft gate

$E$9

The dynamic soft mask is

$D$0

The paper’s intuition is that different denoising stages require different strengths of pose control: early steps organize coarse structure, while later steps refine details. A fixed pose mask would therefore be too rigid or too weak across the full reverse process. The dynamic mask makes the importance of pose-related regions timestep-dependent.

This soft mask is applied to attention logits before softmax. The paper writes

$D$1

with $D$2, $D$3, and $D$4 denoting queries, keys, and values. The intended effect is to bias attention toward pose-related regions and suppress background interference. In the paper’s own interpretation, DM and KB are complementary: KB supplies global semantic guidance, while DM emphasizes local body regions, yielding “unified control of both global semantics and local details” (Liu et al., 26 Jul 2025).

A plausible implication is that DM functions as a lightweight spatial prior rather than a full geometric model of occlusion or limb uncertainty. The paper itself notes no finer-grained masking formulation beyond foreground–background weighting.

5. Training protocol, benchmark, and evaluation criteria

The principal benchmark is Human-Art, described as a dataset of 50,000 high-quality images from 5 real-world and 15 virtual scenarios, annotated with human bounding boxes, keypoints, and textual descriptions. This benchmark is used for both training and evaluation. The train/validation split follows Human-Art (Liu et al., 26 Jul 2025).

The diffusion model is fine-tuned from Stable Diffusion v1.5 using Adam with learning rate $D$5 for 10 epochs. Training uses random text dropping with probability 0.5, following a ControlNet-style classifier-free conditioning strategy. The KB is trained separately in two stages—codebook optimization and classifier fine-tuning—using OpenCLIP ViT-L/14 as the frozen CLIP backbone, AdamW, initial learning rate 0.001, and cosine decay, for 30 epochs in each stage. Inference uses DDIM with 50 timesteps.

The reported baselines are SD, T2I-Adapter, ControlNet, Uni-ControlNet, GLIGEN, HumanSD, GRPose, and Stable-Pose. Evaluation spans three categories. Pose accuracy is measured by AP, CAP, and PCE, using HigherHRNet to estimate poses from generated images. Image quality is measured by FID and KID. Text-image alignment is measured by CLIP-score. The paper notes that KID is multiplied by 100 on Human-Art (Liu et al., 26 Jul 2025).

These choices reveal the paper’s intended contribution: not merely better pose control, but a more balanced optimization across body accuracy, full-image realism, and text consistency.

6. Reported results, ablations, and open issues

On Human-Art, the main reported result for KB-DMGen is:

• AP: 51.71
• CAP: 71.40
• PCE: 1.53
• FID: 10.29
• KID: 2.45
• CLIP-score: 32.45

Among the compared methods, the strongest pose-control baseline in the printed table is GRPose with AP 49.50 and CAP 70.84, so KB-DMGen improves AP by 2.21 and CAP by 0.56. The method therefore achieves the best reported AP and CAP in the table, while keeping FID and KID competitive rather than allowing a severe quality collapse under stronger pose control (Liu et al., 26 Jul 2025).

The ablation study isolates the effects of the two principal modules. Adding KB yields AP 50.73, CAP 71.04, PCE 1.58, FID 11.28, KID 2.52, and CLIP-score 32.47. Adding DM on top of KB improves this to AP 51.40, CAP 71.17, PCE 1.54, FID 10.56, KID 2.54, and CLIP-score 32.41. Adding D{data}C gives the final 51.71 / 71.40 / 1.53 / 10.29 / 2.45 / 32.45 configuration. The progressive pattern supports the paper’s claim that KB primarily restores global semantics and image quality, while DM further sharpens structural fidelity.

Several limitations are also explicit or directly implied. The paper does not report the codebook size $D$6; it does not specify the exact insertion layers of KB features; and it does not fully detail the architecture of the KB Adapter and DM Adapter. Evaluation is concentrated on Human-Art alone. The masking strategy is coarse, being derived from a binary foreground mask rather than a richer articulation-aware uncertainty model. Finally, the paper text claims that KB-DMGen achieved the highest AP and CLIP-score, but the printed table supports only the first part unequivocally: SD has a higher listed CLIP-score of 33.33, so the strongest supported conclusion is that KB-DMGen clearly leads on pose metrics while remaining competitive on text alignment (Liu et al., 26 Jul 2025).

Taken together, KB-DMGen is best understood as a diffusion-based human image generator that introduces a specific division of labor between global semantic retrieval from a learned visual codebook and timestep-adaptive local pose emphasis. This suggests a broader methodological pattern for controllable human synthesis: sparse structural conditions can be strengthened not only by better local control, but also by auxiliary mechanisms that restore full-image priors at the semantic level.