---
title: Human-Guided Conditional Diffusion
url: https://www.emergentmind.com/topics/human-guided-conditional-diffusion
type: topic
---

# Human-Guided Conditional Diffusion

to=arxiv_search.search  全民彩票天天送钱json
{"query":"all:(\"human-guided conditional diffusion\" OR \"score-guided diffusion\" OR \"guided conditional diffusion\")", "max_results": 10, "sort_by": "relevance"}
to=arxiv_search.search  天天中彩票和json
{"query":"id:2306.12169 OR id:2403.09623 OR id:2509.07920 OR id:2507.11025 OR id:2502.03426 OR id:2210.17366 OR id:2411.12773 OR id:2602.05533", "max_results": 20, "sort_by": "relevance"}
Human-guided conditional diffusion denotes a family of diffusion-based generative and inverse-problem methods in which denoising is conditioned or steered by signals that encode human intent, human structure, human preference, or human-related constraints. In the literature, these signals range from guide images and class labels, human perceptual evaluations, clinical metadata, semantic part labels, parsing masks, trajectories, and textual descriptions to explicitly human-centric physical objectives such as contact, support, and collision avoidance. The unifying pattern is a diffusion prior—unconditional, conditional, or bridge-based—whose reverse dynamics are altered so that sampling remains plausible under the learned distribution while moving toward externally specified human criteria [2405.00858][2306.12169][2403.09623][2509.07920][2502.03426][2602.05533].

## 1. Conceptual scope and definitions

The term does not denote a single standardized formalism. Instead, the surveyed literature exhibits several mathematically distinct but conceptually related uses. In "HumanDiffusion" the target distribution is a *human-acceptable distribution*,
$$
p_{\text{human}}(x)=\frac{1}{Z}D(x),
$$
where \(D(x)\) is a human perceptual evaluation of naturalness, so human judgment directly defines the density being sampled [2306.12169]. In "Guided Conditional Diffusion Classifier (ConDiff)" a real guide image \(x_0\) is perturbed and then conditionally denoised under candidate labels, so classification is performed by comparing the original guide image with label-conditioned reconstructions in embedding space [2405.00858]. In "Score-Guided Human Mesh Recovery" and "ScoreHOI", diffusion acts as a conditional prior over human pose or human-object interaction parameters, while denoising is further guided by reprojection, temporal, contact, floor-support, and penetration objectives [2403.09623][2509.07920]. In "TruePose", human parsing is not merely an auxiliary input but a mechanism for reweighting attention during diffusion-based pose transfer [2502.03426].

A second distinction concerns whether the “human” component is a *human feedback signal* or a *human-centered structural prior*. Binary expert preference labels are used as classifier-free guidance conditions in CBCT-to-MDCT translation, where they explicitly steer a Schrödinger Bridge sampler toward clinically preferred artifact suppression [2507.11025]. By contrast, one-shot human image synthesis via 3D priors uses 3D normal maps and color prompts derived from SMPL as geometry and appearance conditions; this is human-centric conditioning rather than human preference feedback [2606.19718]. Guided Motion Diffusion likewise focuses on controllable human motion synthesis conditioned by natural language and spatial constraints such as pre-defined motion trajectories, obstacles, and sparse keyframes, with feature projection, a new imputation formulation, and dense guidance from sparse signals [2305.12577].

This breadth suggests that “human-guided” is best understood as a spectrum. At one end, human evaluation defines the target energy or probability model directly; at the other, human-related structure—body models, parsing maps, trajectories, or metadata—defines the conditional variables through which diffusion operates.

## 2. Mathematical mechanisms of guidance

A recurrent formulation is to decompose a conditional score into a learned prior term and an externally specified guidance term. In ScoreHOI, denoising is guided by physical plausibility constraints \(\mathcal{P}\) through
$$
\nabla_{\mathbf{x}_t}\log p(\mathbf{x}_t\mid \mathbf{c},\mathcal{P})
=
\nabla_{\mathbf{x}_t}\log p(\mathbf{x}_t\mid \mathbf{c})
+
\nabla_{\mathbf{x}_t}\log p(\mathcal{P}\mid \mathbf{c},\mathbf{x}_t),
$$
with the second term approximated on the denoised prediction \(\hat{\mathbf{x}}_0(\mathbf{x}_t)\), and the resulting gradient added to the noise prediction as an energy-guidance correction [2509.07920]. ScoreHMR uses the same logic for inverse problems in 3D human recovery, replacing contact energies with task losses such as 2D reprojection, multi-view consistency, or temporal smoothness [2403.09623].

Another important mechanism is classifier-free guidance. ConDiff trains a conditional diffusion model with label dropout and samples with
$$
\tilde{\epsilon}_\theta(x_t,t,y)
=
(1-\omega)\,\epsilon_\theta(x_t,t)
+
\omega\,\epsilon_\theta(x_t,t,y),
$$
while initializing the reverse process from a noisy version of a specific guide image rather than from pure noise [2405.00858]. The CBCT-to-MDCT Schrödinger Bridge model uses the same principle with binary human feedback labels, combining a conditional score and an unconditional score through
$$
\tilde{\mathbf{s}}(\mathbf{z}_t)
=
(1+w)\,\mathbf{s}_\theta(\mathbf{z}_t\mid \mathbf{z}_0,t,r)
-
w\,\mathbf{s}_\theta(\mathbf{z}_t\mid \varnothing),
$$
so that sampling can be pushed toward the “good” preference class without introducing a separate reward model [2507.11025].

A third family comprises training-free plug-and-play guidance. FreeDoM defines a time-independent energy \(\mathcal{E}(c,x_0)\) using off-the-shelf networks and applies guidance through the gradient of \(\mathcal{E}\) evaluated on the estimated clean sample \(\hat{x}_{0\mid t}\) [2303.09833]. MPGD modifies this pattern by applying guidance in clean space and then performing a shortcut DDIM-style update, with optional manifold-preserving projection through a pretrained autoencoder in data or latent space [2311.16424]. ADMMDiff goes further by decoupling generation and guidance into two variables,
$$
\max_{x,z}\ \log q_\phi(x)+\log c_\theta(z,y)
\quad \text{s.t.}\quad x=z,
$$
and solving the resulting constrained problem with ADMM, treating the diffusion reverse step as an approximate proximal operator of the unconditional prior [2411.12773].

The hard-constraint setting is different again. Under Doob’s \(h\)-transform, conditional generation under an event \(S\) modifies the pretrained reverse dynamics to
$$
dY_t^S
=
\big(\overline{f}(t,Y_t^S)+\overline{g}(t)^2\nabla\log h(t,Y_t^S)\big)\,dt
+
\overline{g}(t)\,d\overline{B}_t,
$$
where \(h(t,y)=\mathbb{P}(Y_T\in S\mid Y_t=y)\). Here the constraint is not a soft preference but a target event that should hold with probability one in the ideal conditional law [2602.05533]. Tempered Guided Diffusion occupies an intermediate position: it keeps the unconditional diffusion prior fixed, defines tempered posteriors
$$
\pi_r(x_0)\propto p(x_0)\,p(y\mid x_0)^{\lambda_r},
$$
and allocates compute across multiple trajectories via sequential Monte Carlo rather than by modifying a single trajectory alone [2605.03712].

## 3. Sources of human guidance

The surveyed works differ primarily in the form of the external signal that enters the diffusion process.

| Guidance source | Role in diffusion | Representative papers |
| --- | --- | --- |
| Human perceptual score or preference label | Defines target density or CFG condition | [2306.12169], [2507.11025] |
| Guide image or reference appearance | Anchors conditional editing to a specific sample | [2405.00858], [2502.03426], [2606.19718] |
| Human-body structure, motion, or physical constraints | Steers denoising toward pose, contact, and feasibility | [2305.12577], [2403.09623], [2509.07920] |
| Clinical metadata or semantic labels | Conditions denoising on structured human context | [2408.07709], [2509.17206] |
| User-authored logical rules or hard events | Defines differentiable robustness or exact conditional law | [2210.17366], [2602.05533] |

At the preference end, HumanDiffusion estimates both \(D(x)\) and \(\nabla_x D(x)\) from human evaluations, using periphery samples and NES-like finite differences to approximate a perceptual score field that is wider than the empirical data distribution [2306.12169]. The CBCT-to-MDCT Schrödinger Bridge model uses binary expert labels only—“good” and “bad”—but couples them to CFG and iterative fine-tuning, so the model internalizes clinically preferred outcomes through tournament-based selection of generated candidates [2507.11025].

At the structural-conditioning end, several methods guide diffusion with human-body or human-scene representations. Guided Motion Diffusion incorporates natural-language descriptions together with pre-defined motion trajectories, obstacles, and sparse keyframes, and explicitly addresses the problem that sparse signals can be ignored during reverse diffusion by turning them into denser guidance signals [2305.12577]. ScoreHMR conditions a diffusion prior on image features and then guides sampling with losses defined on SMPL pose, camera projection, and temporal or multi-view relations [2403.09623]. ScoreHOI extends this to joint human-object parameters, with contact masks, floor support, and SDF-based penetration avoidance as human-centric physical constraints [2509.07920]. TruePose uses human parsing maps to specify which source regions are semantically relevant for the target pose, and uses those masks to reweight attention so that clothing details and facial identity are preserved through pose transfer [2502.03426].

A different line of work treats structured human context as metadata. mbVDiT conditions latent-space diffusion on the observed portion of a microbiome profile and on patient metadata embedded by modality-specific MLPs, then fuses those embeddings through cross-attention in a DiT-like backbone [2408.07709]. In 3D point-cloud generation, guided diffusion holds per-point semantic labels fixed while only diffusing geometry; because those labels can be specified by a human before sampling, the method supports direct semantic composition control over generated object parts [2509.17206]. This suggests that “human guidance” can be instantiated by explicit user constraints, by expert annotations, or by structured human-side covariates, provided those signals are differentiably connected to denoising.

## 4. Architectural and training patterns

A common architectural pattern is to separate a pretrained prior from task-specific conditioning modules. ScoreHOI first trains an affordance-aware regressor and contact predictor, then freezes the image backbone and affordance network while training a diffusion model over the 331-dimensional HOI parameter vector \(\{\theta,\beta,R_o,t_o\}\). Conditioning is injected through an IG-Adapter with an extra cross-attention block over image features and geometry/affordance features, plus timestep-dependent scaling and shifting of the state [2509.07920]. ScoreHMR similarly learns a task-agnostic image-conditional diffusion prior over SMPL pose and then reuses it across single-frame fitting, multi-view refinement, and video smoothing without retraining the prior itself [2403.09623].

Another recurring pattern is two-stage learning. ConDiff first fine-tunes a conditional latent diffusion backbone, then freezes it and trains a separate EfficientNet-B0 embedding network with triplet loss on real and synthetic images; the diffusion model is thus used as a conditional editor, and the embedding model as the discriminative decision rule [2405.00858]. mbVDiT likewise pretrains a VAE on public microbiome datasets, freezes or reuses that latent model, and then trains a metadata-conditional latent diffusion model for imputation [2408.07709]. In both cases, the separation of generative modeling and downstream decision-making reduces overfitting and allows the guidance signal to be modified without relearning the entire pipeline.

Human-guided diffusion also motivates specialized attention design. TruePose employs a human-parsing-aware Siamese architecture with dual identical UNets, a Human-Parsing-Guided Fusion Attention module, and a CLIP-Guided Attention Alignment module. The crucial intervention is not an auxiliary loss but attention reweighting itself: source features are reweighted by target-relevant parsing masks before cross-attending into the target denoising path, and low-attended regions are then refined through CLIP region embeddings [2502.03426]. In one-shot novel-view and pose synthesis, a separate line uses a reference encoder and a noise encoder, injecting 3D normal maps and a target-view color prompt as image-like conditions, with layer-wise cross-attention at \(32\times32\), \(16\times16\), and \(8\times8\) resolutions [2606.19718]. These designs indicate that human-structured conditions are often too localized or too geometry-dependent to be handled well by simple global conditioning.

Training-free methods replace learned conditional modules with optimization layers around pretrained priors. FreeDoM computes guidance through differentiable energies evaluated on \(\hat{x}_{0\mid t}\) and can combine text, segmentation, landmarks, identity, style, or low-pass constraints with no retraining of the diffusion model [2303.09833]. MPGD adds autoencoder-based manifold projection to keep guidance tangent to an approximate data manifold [2311.16424]. ADMMDiff and Tempered Guided Diffusion further replace single-trajectory heuristics with optimization or particle-based outer loops that adaptively balance prior realism and condition satisfaction [2411.12773][2605.03712].

## 5. Representative applications and empirical behavior

Medical imaging provides several of the clearest demonstrations. ConDiff reports an accuracy of \(83\%\) and an \(F1\)-score of \(0.858\) for diabetic foot ulcer infection prediction, outperforming the compared state-of-the-art models by at least \(3\%\) while using guided conditional synthesis plus distance-based classification rather than a conventional discriminative head [2405.00858]. In CBCT-to-MDCT translation, the Schrödinger Bridge model with human feedback achieves RMSE \(0.0030 \pm 0.0010\), SSIM \(0.9971 \pm 0.0013\), LPIPS \(0.0015 \pm 0.0009\), and Dice \(83.95 \pm 4.97\), and does so with only \(10\) sampling steps; on the bad-case subset it reaches ARR \(96.98\) and ARSR \(96.23\) [2507.11025]. In microbiome imputation, mbVDiT attains PCC \(0.634\) on STAD, \(0.704\) on COAD, and \(0.626\) on HNSC, and is reported to be best across PCC, cosine similarity, RMSE, and MAE on all three cancer datasets [2408.07709].

Human-body reconstruction and interaction are another major domain. ScoreHMR improves HMR 2.0 on 3DPW from PA-MPJPE \(54.3\) to \(51.1\) in single-frame model fitting, and in video refinement reduces acceleration error on 3DPW from \(17.3\) to \(11.1\) while also lowering PA-MPJPE to \(50.5\) [2403.09623]. ScoreHOI improves contact plausibility on BEHAVE from F-score \(0.554\) for CONTHO to \(0.609\), while also improving \(\text{CD}_{\text{human}}\) from \(4.99\) to \(4.85\) and \(\text{CD}_{\text{object}}\) from \(8.42\) to \(7.86\); it runs at \(0.2895\) FPS, compared with \(0.0035\) FPS for CHORE, and a reduced-iteration variant reaches \(2.0080\) FPS with only a small drop in accuracy [2509.07920]. These results reflect the value of combining a learned conditional prior with task-specific human-body energies rather than relying on pure regression or pure iterative fitting.

Pose- and image-conditioned synthesis shows a related pattern. TruePose achieves LPIPS \(0.151\), SSIM \(0.727\), and PSNR \(18.123\) on DeepFashion \(256\times176\), and LPIPS \(0.180\), SSIM \(0.744\), and PSNR \(17.627\) on DeepFashion \(512\times352\), with clear gains over CFLD and PCDM; on the in-the-wild WPose benchmark it obtains LPIPS \(0.583\) and PSNR \(11.457\), substantially better than the compared diffusion baselines [2502.03426]. One-shot human image synthesis with 3D priors reports, on RenderPeople novel view, PSNR \(23.29\), SSIM \(0.91\), LPIPS \(0.07\), and FID \(17.05\), and on RenderPeople novel pose, PSNR \(22.36\) and LPIPS \(0.07\), outperforming SHERF, Champ, and PIDM on the listed metrics [2606.19718].

The same logic extends beyond explicitly human imagery. In controllable traffic generation, user-defined STL rules such as speed limits, goal waypoints, collision avoidance, off-road avoidance, and stop-sign compliance act as differentiable logical guidance. Under the speed-limit rule, CTG reports rule violation \(0.019\), realism deviation \(0.359\), and fail \(0.165\), compared with \(0.033\), \(1.190\), and \(0.487\) for BITS+opt, illustrating the controllability–realism tradeoff that rule-guided diffusion can navigate more effectively than purely optimization-based baselines [2210.17366]. This supports a broader interpretation of human-guided diffusion as externally constrained conditional generation rather than only human-image synthesis.

## 6. Limitations, tensions, and emerging directions

A persistent tension is between guidance strength and fidelity to the learned prior. CTG explicitly reports a controllability–realism tradeoff: stronger STL guidance reduces rule violation but can worsen realism deviation and failure rate [2210.17366]. FreeDoM observes that large guidance strength \(\rho_t\) can overshoot and produce artifacts or off-manifold images, especially when multiple conditions conflict [2303.09833]. ADMMDiff reframes this as a balancing problem between a prior term and a guidance term, which it solves with dual variables instead of a fixed scalar weight, but the need to tune \(\rho\), \(\eta\), and the number of inner optimization steps remains [2411.12773]. This suggests that human-guided diffusion is not only a modeling problem but also a control problem over the reverse trajectory.

A second limitation is dependence on external priors and intermediate predictors. ScoreHOI depends on predefined canonical object templates and on the quality of contact masks; under extreme occlusion, wrong masks can misguide optimization [2509.07920]. TruePose depends on accurate human parsing, since parsing masks directly reweight attention [2502.03426]. One-shot human image synthesis is limited by SMPL estimation quality, and performance degrades when ground-truth SMPL parameters are replaced by HMR2.0 estimates [2606.19718]. Hard-constraint guidance under Doob’s \(h\)-transform avoids heuristic weighting, but its guarantees depend on estimating \(h\) and \(\nabla \log h\) accurately, which becomes difficult for rare events or noisy labels [2602.05533].

Sampling cost remains substantial. ConDiff requires about \(12.5\) minutes for \(251\) test images on an A100, approximately \(3\) seconds per image, compared with less than \(4\) seconds total for the compared CNN/ViT baselines [2405.00858]. ScoreHOI is much faster than Adam-based optimizers but still heavier than pure regression [2509.07920]. By contrast, the Schrödinger Bridge CBCT system shows that strong human-guided performance can be obtained with \(10\) sampling steps when the bridge structure is analytically exploited [2507.11025]. A plausible implication is that future work will increasingly combine stronger inductive structure—bridges, manifolds, latent priors, or exact constraints—with preference guidance, in order to reduce the cost of iterative denoising.

Several papers point toward richer forms of interaction. ScoreHOI explicitly suggests modifying contact masks or adding user-specified constraints such as “the right hand must touch the handle,” and also suggests language-guided HOI, multi-person and multi-object interaction, and temporal diffusion over motion sequences [2509.07920]. The structured point-cloud work suggests interactive editing by painting or manipulating per-point labels, and proposes soft labels, continuous guidance strength, and hierarchical semantics as open directions [2509.17206]. ConDiff identifies adversarial diffusion distillation such as SDXL Turbo as a promising route to one-step sampling, which would make guide-image-based decision pipelines much more practical [2405.00858]. Together these works indicate that the field is moving from static conditional generation toward iterative, constraint-rich, and potentially interactive human guidance over diffusion trajectories.

Source: https://www.emergentmind.com/topics/human-guided-conditional-diffusion