Diff-Prior in Diffusion Models

• Diff-prior is a diffusion-derived prior that biases sampling or optimization by replacing standard Gaussian distributions with pretrained or learned latent structures.
• It is employed as a backbone regularizer, a learned latent prior, or a structured initialization to ensure more realistic, semantically coherent outputs.
• Empirical studies demonstrate that diff-prior techniques improve metrics such as FID and registration recall, though they require careful calibration to avoid domain mismatches.

to=arxiv_search.search ુમૈjson {"^^^^^^^^14query14 OR ^^^^^^^^14all:\14 prior\"^^^^^^^^14 arXiv14^^^^^^^^", "^^^^^^^^14max_results14 ^^^^14all:\14query14^^^^, "^^^^^^^^14sort_by14 "^^^^^^^^14submittedDate14 "^^^^^^^^14sort_order14 "^^^^^^^^14descending14 to=arxiv_search.search 彩神争霸可以json {"^^^^^^^^14query14 Diffusion Planning for Offline Reinforcement Learning\" OR ^^^^^^^^14all:\14query14 A Model-Based Score Learning Framework for Inverse Problems\" OR ^^^^^^^^14all:\14query14 OR ^^^^^^^^14all:\14query14 Priors In Variational Autoencoders\"", "^^^^^^^^14max_results14 ^^^^14all:\14query14^^^^, "^^^^^^^^14sort_by14 "relevance", "^^^^^^^^14sort_order14 "^^^^^^^^14descending14 Diff-prior is a literature-dependent term for a diffusion-derived prior used to constrain generation, inference, or latent calibration. Rather than denoting a single formal object, it has been used for a pretrained latent diffusion backbone that regularizes downstream synthesis, a learned prior that replaces the standard Gaussian endpoint of diffusion sampling, a structured initialization distribution tailored to a conditioning signal, and a denoising-style prior over structured latent variables such as graphs. Across these usages, the common role is to bias sampling or optimization toward a distribution that is more realistic, semantically coherent, or better matched to the task than an unstructured reference prior.

^^^^14all:\14^^^^. Terminological scope

Recent papers use “Diff-prior” in several distinct but related senses.

Usage	Mechanism	Representative papers
Backbone prior	Frozen pretrained diffusion model regularizes a downstream model	(&&&^^^^14query14^^^^&&&, &&&^^^^14all:\14^^^^&&&, &&&^^^^14 OR all:\14^^^^&&&)
Learned latent prior	Diffusion or mixture prior replaces PRESERVED_PLACEHOLDER_^^^^14query14^^^^	(&&&^^^^14 arXiv14^^^^&&&, &&&^^^^14max_results14^^^^&&&, &&&^^^^14sort_by14^^^^&&&)
Sampling prior	Structured initialization replaces standard diffusion noise at inference	(&&&^^^^14submittedDate14^^^^&&&, &&&^^^^14sort_order14^^^^&&&)
Inverse-problem prior	Diffusion score or posterior score acts as a regularizer with data consistency	(&&&^^^^14descending14^^^^&&&, &&&^^^^14query14^^^^&&&)
Structured-variable prior	Diffusion defines or calibrates priors over graphs or edge logits	(&&&^^^^14all:\14query14^^^^&&&, &&&^^^^14all:\14all:\14^^^^&&&)

In diffusion-model literature, one canonical meaning of a diffusion prior is a learned probabilistic model that captures the distribution of representations or images and serves as a generative regularizer. A second canonical meaning is a pretrained image diffusion backbone, such as Stable Diffusion, whose latent manifold and cross-attention machinery are reused while most core modules remain frozen (&&&^^^^14query14^^^^&&&). Other papers extend the term to the prior distribution at the endpoint of a diffusion chain, to latent priors embedded in VAEs, or to inference-time initialization priors that better match the training trajectory than a standard normal distribution (&&&^^^^14 arXiv14^^^^&&&).

This plurality is substantive rather than terminological. In some works, the prior is an explicit probability law; in others it is a frozen generative model whose score, manifold, or denoising behavior acts as the effective prior. A plausible implication is that “Diff-prior” is best understood functionally: it identifies the component through which diffusion-derived inductive bias enters the system.

^^^^14 OR all:\14^^^^. Diff-prior as a pretrained generative regularizer

A prominent usage treats a pretrained diffusion model as a fixed prior that anchors outputs to a high-quality latent manifold while trainable branches inject task-specific control. In CCIS-Diff, the prior is Stable Diffusion v^^^^14all:\14^^^^.^^^^14sort_by14^^^^ used as both backbone and regularizer for controlled colonoscopy synthesis. The latent VAE, U-Net, and text encoder are frozen; a replicated U-Net encoder forms a trainable ControlNet-like branch initialized with zero convolutions, together with two trainable mask encoders and a text-aware attention mechanism (&&&^^^^14query14^^^^&&&). Sampling is performed in the SD latent space, while conditioning is PRESERVED_PLACEHOLDER_^^^^14all:\14^^^^, combining a binary mask, its blurred variant, and clinical text. The paper explicitly formulates the epsilon-prediction objective as

PRESERVED_PLACEHOLDER_^^^^14 OR all:\14^^^^

This architecture uses the prior to preserve image realism while allowing spatial and clinical control. Empirically, CCIS-Diff reports FID ^^^^14sort_order14all:\14^^^^.^^^^14sort_order14 arXiv14^^^^, CLIP-score ^^^^14 arXiv14all:\14^^^^.^^^^14query14submittedDate14^^^^, and CLIP-image similarity ^^^^14descending14descending14^^^^.^^^^14sort_order14query14 outperforming ControlNet and Uni-ControlNet trained on the same dataset; expert scores for image fidelity, mask accuracy, and text accuracy are also highest, and downstream PraNet and Polyp-PVT segmentation metrics improve with CCIS-Diff augmentation (&&&^^^^14query14^^^^&&&).

Diff-Mosaic uses a related but domain-adapted pattern. Its second-stage Diff-Prior takes Pixel-Prior outputs, encodes them into a latent diffusion model, resamples them, and decodes more realistic infrared images. The base LDM is pretrained at scale and then fine-tuned for ^^^^14all:\14query14query14^^^^ epochs on SIRST-style infrared imagery; training uses the standard latent diffusion loss together with a latent consistency term

PRESERVED_PLACEHOLDER_^^^^14 arXiv14^^^^

which keeps resampled results faithful to the conditioned input while improving realism and diversity (&&&^^^^14all:\14submittedDate14^^^^&&&). On NUDT-SIRST, adding Diff-Prior on top of Pixel-Prior raises IoU from ^^^^14query14all:\14^^^^.^^^^14query14all:\14^^^^ to ^^^^14query14all:\14^^^^.^^^^14all:\14descending14 raises PRESERVED_PLACEHOLDER_^^^^14max_results14^^^^ from ^^^^14query14descending14^^^^.^^^^14query14all:\14^^^^ to ^^^^14query14query14^^^^.^^^^14max_results14sort_order14 and lowers PRESERVED_PLACEHOLDER_^^^^14sort_by14^^^^ from ^^^^14 OR all:\14^^^^.^^^^14all:\14 arXiv14^^^^ to ^^^^14all:\14^^^^.^^^^14query14all:\14 Diff-Mosaic also reports FID ^^^^14all:\14 OR all:\14submittedDate14^^^^.^^^^14query14all:\14^^^^ and KID ^^^^14query14^^^^.^^^^14query14sort_order14submittedDate14^^^^ on infrared realism metrics (&&&^^^^14all:\14submittedDate14^^^^&&&).

In DiffPRESERVED_PLACEHOLDER_^^^^14submittedDate14^^^^I^^^^14 OR all:\14^^^^P, the diffusion prior is a frozen depth-conditioned latent diffusion model with a ControlNet depth branch, distilled into an image-to-point-cloud registration pipeline through Control-Side Score Distillation. The prior is never fine-tuned; instead, a differentiable depth rendering from the predicted pose is fed to the frozen model, and the denoising residual supervises pose and feature learning (&&&^^^^14all:\14^^^^&&&). This makes the prior a cross-modal geometric regularizer rather than an image generator used at inference. On ^^^^14sort_order14^^^^-Scenes, the method reports mean registration recall ^^^^14descending14 arXiv14^^^^.^^^^14query14^^^^% versus ^^^^14sort_order14sort_by14^^^^.^^^^14descending14 for ^^^^14 OR all:\14^^^^D^^^^14 arXiv14^^^^D-MATR, together with lower RRE and RTE (&&&^^^^14all:\14^^^^&&&).

Diff-ICMH extends the same backbone-prior logic to compression. It uses a Stable-Diffusion-style latent diffusion prior with a modified ControlNet-like module, reconstructs in SD-VAE latent space, and injects image-level tags as prompts through a Tag Guidance Module. Its semantic consistency loss is computed from Stable Diffusion feature mappings, while distortion is optimized in latent space rather than pixel space; the paper reports that latent-space distortion substantially outperforms pixel-space distortion in ablation (&&&^^^^14 OR all:\14^^^^&&&).

^^^^14 arXiv14^^^^. Diff-prior as a learned replacement for simple latent priors

Another major usage replaces a simple prior such as PRESERVED_PLACEHOLDER_^^^^14sort_order14^^^^ with a learned diffusion-based or pseudo-input-based prior. In “Diffusion Priors in Variational Autoencoders,” the VAE prior PRESERVED_PLACEHOLDER_^^^^14descending14^^^^ is modeled by a DDPM over latent variables rather than by a standard Gaussian. Because PRESERVED_PLACEHOLDER_^^^^14query14^^^^ is intractable, the VAE objective replaces it with a diffusion ELBO applied to the latent code PRESERVED_PLACEHOLDER_^^^^14all:\14query14^^^^, yielding a lower bound on the data likelihood that combines the decoder term, encoder entropy term, and diffusion prior bound (&&&^^^^14 arXiv14^^^^&&&). The paper’s practical surrogate objective is

PRESERVED_PLACEHOLDER_^^^^14all:\14all:\14^^^^

On CelebA, the diffusion prior improves over Gaussian priors and is competitive with normalizing-flow priors; for latent size ^^^^14all:\14query14query14^^^^, the reported FID is ^^^^14submittedDate14sort_order14^^^^.^^^^14query14sort_by14^^^^ for the diffusion prior versus ^^^^14all:\14max_results14query14^^^^.^^^^14max_results14^^^^ for the Gaussian prior (&&&^^^^14 arXiv14^^^^&&&).

Residual Prior Diffusion generalizes this idea by coupling a coarse latent-variable prior with a residual diffusion process. The prior predicts a mean and variance,

PRESERVED_PLACEHOLDER_^^^^14all:\14 OR all:\14^^^^

and diffusion operates on normalized residual coordinates around that coarse prediction (&&&^^^^14max_results14^^^^&&&). Its reverse chain starts from PRESERVED_PLACEHOLDER_^^^^14all:\14 arXiv14^^^^ rather than from a featureless standard normal, and the forward terminal is constructed to match the reverse initial distribution. The paper shows that the ELBO again reduces to familiar noise- or velocity-prediction losses, but now with prior-derived auxiliary variables that simplify prediction. Empirically, RPD and RPD-vpred outperform baseline diffusion models on hetero-scale synthetic datasets and remain strong in few-step natural-image generation, with Butterfly KID at ^^^^14sort_by14query14^^^^ steps reported as approximately PRESERVED_PLACEHOLDER_^^^^14all:\14max_results14^^^^ for RPD (&&&^^^^14max_results14^^^^&&&).

Prior-Guided Diffusion Planning adopts the same replacement principle in offline reinforcement learning. A pretrained behavior-cloned denoiser is fixed, but the standard endpoint prior PRESERVED_PLACEHOLDER_^^^^14all:\14sort_by14^^^^ is replaced by a learnable, state-conditional prior PRESERVED_PLACEHOLDER_^^^^14all:\14submittedDate14^^^^ (&&&^^^^14sort_order14^^^^&&&). Under an approximate bijectivity assumption for DDIM, the paper rewrites the behavior-regularized planning objective from trajectory space into prior space:

PRESERVED_PLACEHOLDER_^^^^14all:\14sort_order14^^^^

A latent critic then avoids backpropagation through denoising. This makes the prior itself the optimized object. Reported D^^^^14max_results14^^^^RL results show gains over the baseline planner DV* in Kitchen, AntMaze, and Maze^^^^14 OR all:\14^^^^D, with AntMaze average ^^^^14descending14 arXiv14^^^^.^^^^14max_results14^^^^ for PG versus ^^^^14sort_order14query14^^^^.^^^^14query14^^^^ for DV* (&&&^^^^14sort_order14^^^^&&&).

VAMP-Diff uses a different learned prior family. It combines a temporal encoder, a conditional ^^^^14all:\14^^^^D diffusion decoder, and a VampPrior applied in a compact pooled latent space while the decoder conditions on the full temporal latent during denoising (&&&^^^^14sort_by14^^^^&&&). The joint objective is

PRESERVED_PLACEHOLDER_^^^^14all:\14descending14^^^^

On CapnoBase PPG, the paper reports better heart-rate preservation than Gaussian-prior baselines, with heart-rate absolute error PRESERVED_PLACEHOLDER_^^^^14all:\14query14^^^^ bpm for VAMP-Diff (&&&^^^^14sort_by14^^^^&&&).

^^^^14max_results14^^^^. Diff-prior as posterior score, structural reference, or initialization distribution

In inverse problems, “diffusion prior” often refers to the score of an image distribution used inside a data-consistent solver. Diff-Unfolding states the classical decomposition

PRESERVED_PLACEHOLDER_^^^^14 OR all:\14query14^^^^

where the second term is the diffusion prior and the first enforces measurement fidelity (&&&^^^^14descending14^^^^&&&). Instead of adding an unconditional prior score and a likelihood gradient only at inference time, Diff-Unfolding learns the posterior score directly through a modular deep unfolding architecture. The learned denoiser satisfies

PRESERVED_PLACEHOLDER_^^^^14 OR all:\14all:\14^^^^

and the measurement operator appears only in an explicit data-consistency block, enabling operator swapping at inference under the stated linear-Gaussian assumptions (&&&^^^^14descending14^^^^&&&). The method reports state-of-the-art image restoration and accelerated MRI results, including up to approximately ^^^^14 OR all:\14^^^^ dB PSNR improvement and LPIPS reduction up to ^^^^14 OR all:\14 OR all:\14^^^^.^^^^14sort_order14^^^^%, with ^^^^14max_results14submittedDate14^^^^.^^^^14query14max_results14 parameters and ^^^^14query14^^^^.^^^^14sort_order14 OR all:\14^^^^ s per PRESERVED_PLACEHOLDER_^^^^14 OR all:\14 OR all:\14^^^^ image (&&&^^^^14descending14^^^^&&&).

SCP-Diff shifts attention from the learned score to the starting distribution of the reverse process. For semantic image synthesis with ControlNet, it argues that artifacts arise from a mismatch between the noised training distribution and the standard normal initialization used at inference (&&&^^^^14submittedDate14^^^^&&&). It therefore replaces PRESERVED_PLACEHOLDER_^^^^14 OR all:\14 arXiv14^^^^ with structured Gaussian priors estimated from real latent codes: a spatial prior, a categorical prior, and a spatial-categorical joint prior. Sampling starts from an intermediate noise level PRESERVED_PLACEHOLDER_^^^^14 OR all:\14max_results14^^^^, with token-wise initialization induced by empirical latent statistics conditioned on position and class (&&&^^^^14submittedDate14^^^^&&&). This training-free change produces reported FID ^^^^14all:\14query14^^^^.^^^^14sort_by14 arXiv14^^^^ on Cityscapes and ^^^^14all:\14 OR all:\14^^^^.^^^^14submittedDate14submittedDate14^^^^ on ADE^^^^14 OR all:\14query14^^^^K, while also improving mIoU and accuracy (&&&^^^^14submittedDate14^^^^&&&).

FSP-Diff uses yet another variant: a high-SNR full-spectrum image constructed by fusing multi-energy spectral CT projections becomes a structural prior shared across energy bins. The fusion is

PRESERVED_PLACEHOLDER_^^^^14 OR all:\14sort_by14^^^^

followed by PRESERVED_PLACEHOLDER_^^^^14 OR all:\14submittedDate14^^^^; the resulting full-spectrum image is stacked with direct per-bin reconstructions and projection-domain diffusion reconstructions before a second latent diffusion stage (&&&^^^^14query14^^^^&&&). Here the prior is neither merely an endpoint law nor a frozen backbone, but a cross-energy structural scaffold injected into the latent diffusion pipeline. The paper reports the best PSNR and SSIM across all tested spectral bins at both PRESERVED_PLACEHOLDER_^^^^14 OR all:\14sort_order14^^^^ and PRESERVED_PLACEHOLDER_^^^^14 OR all:\14descending14^^^^ photons, with runtime ^^^^14all:\14^^^^.^^^^14 OR all:\14 OR all:\14^^^^ s per bin (&&&^^^^14query14^^^^&&&).

These works show that diffusion priors can enter a system at three different places: as a posterior score inside an optimizer, as an initialization distribution for reverse sampling, or as an external structural reference that narrows the feasible solution set.

^^^^14sort_by14^^^^. Diff-prior on graphs and other structured variables

In discrete graph diffusion, Diff-prior can mean the convergent prior distribution of the forward Markov chain itself. “Complex Preferences for Different Convergent Priors in Discrete Graph Diffusion” defines asymmetric Bernoulli bit-flip kernels with per-step probabilities PRESERVED_PLACEHOLDER_^^^^14 OR all:\14query14^^^^ and PRESERVED_PLACEHOLDER_^^^^14 arXiv14query14^^^^; if these converge to PRESERVED_PLACEHOLDER_^^^^14 arXiv14all:\14^^^^ and PRESERVED_PLACEHOLDER_^^^^14 arXiv14 OR all:\14^^^^, then the terminal single-bit prior is

PRESERVED_PLACEHOLDER_^^^^14 arXiv14 arXiv14^^^^

On graphs, this induces an Erdős–Rényi prior with edge probability PRESERVED_PLACEHOLDER_^^^^14 arXiv14max_results14^^^^ (&&&^^^^14all:\14query14^^^^&&&). The paper shows that generative performance is sensitive to this prior choice and that the optimal prior does not coincide with the empirical edge density. On Community (small) and SBM datasets, tuning the convergent prior yields competitive or superior MMD ratios against graph generative baselines, while simple heuristics based on edge density fail to predict the best prior (&&&^^^^14all:\14query14^^^^&&&).

“From Uniform to Learned Graph Priors” uses the term differently. There, Diff-prior is a diffusion-parameterized adaptive prior over the full edge-logit tensor in neural relational inference, used for calibration rather than generation (&&&^^^^14all:\14all:\14^^^^&&&). Encoder logits are Gaussianized to PRESERVED_PLACEHOLDER_^^^^14 arXiv14sort_by14^^^^, diffused forward, denoised, and then refined by a one-step residual update before Gumbel-Softmax edge sampling. The diffusion loss is weighted noise regression,

PRESERVED_PLACEHOLDER_^^^^14 arXiv14submittedDate14^^^^

and the calibration step updates the clean logits by

PRESERVED_PLACEHOLDER_^^^^14 arXiv14sort_order14^^^^

The prior is therefore non-factorized and acts directly on joint graph configurations (&&&^^^^14all:\14all:\14^^^^&&&). Across StructInfer benchmarks, the method improves AUROC over uniform or fixed priors, with average gains under Netsims of +^^^^14all:\14^^^^.^^^^14query14descending14^^^^ for NRI, +^^^^14max_results14^^^^.^^^^14query14query14^^^^ for ACD, and +^^^^14submittedDate14^^^^.^^^^14max_results14 arXiv14^^^^ for MPM, while also reducing posterior entropy and ECE (&&&^^^^14all:\14all:\14^^^^&&&).

A plausible synthesis is that graph literature splits Diff-prior into two families: priors defined by the endpoint of a discrete diffusion kernel, and priors learned by denoising continuous surrogates of structured latent variables before discretization.

^^^^14submittedDate14^^^^. Empirical regularities, limitations, and open questions

Across application domains, Diff-prior methods repeatedly improve realism, semantic fidelity, calibration, or sample efficiency. Backbone-based priors increase image realism and downstream utility in colonoscopy synthesis, infrared augmentation, registration, and compression (&&&^^^^14query14^^^^&&&). Learned latent priors improve compatibility between inference and generation in VAEs, residual diffusion models, offline RL planners, and physiological signal models (&&&^^^^14 arXiv14^^^^&&&). Structured priors over initialization or operator-conditioned trajectories correct train-test mismatch or stabilize inverse problems (&&&^^^^14submittedDate14^^^^&&&).

At the same time, the literature consistently reports that prior choice is delicate. CCIS-Diff notes residual domain shift because a Stable Diffusion prior is trained on general images rather than colonoscopy, and noisy text can misguide the text-aware mechanism (&&&^^^^14query14^^^^&&&). Diff-Unfolding requires accurate forward operators and noise specification; mismatch can bias reconstructions (&&&^^^^14descending14^^^^&&&). SCP-Diff reports a small diversity drop from structured initialization (&&&^^^^14submittedDate14^^^^&&&). VAMP-Diff reports under-dispersion in generated amplitude statistics despite realistic mean physiology (&&&^^^^14sort_by14^^^^&&&). The learned graph-calibration prior can over-sharpen uncertain posteriors if the residual strength is too large, and some higher-order graph metrics may degrade even when AUROC improves (&&&^^^^14all:\14all:\14^^^^&&&). The discrete-graph prior-search paper goes further by showing that even apparently natural heuristics, such as matching empirical edge density, do not reliably identify the best convergent prior (&&&^^^^14all:\14query14^^^^&&&).

These results suggest that Diff-prior is not a single algorithmic recipe but a design axis. It determines what baseline distribution, manifold, or structural hypothesis the diffusion system should trust before the task-specific model intervenes. In current work, that trust may be assigned to a frozen foundation model, a learned latent mixture, a model-based posterior score, an inference-time initialization law, or a calibration process over structured latents. The open technical question running through these papers is not whether priors matter, but which prior parameterization best matches a given domain, conditioning regime, and optimization pipeline.