Dynamic VLM-Guided Negative Prompting

• Dynamic VLM-Guided Negative Prompting is an inference-time framework that adaptively generates negative cues using a VLM to steer diffusion models away from undesirable content.
• It employs scheduled VLM queries to probe intermediate denoising states, enabling improved safety filtering, creative generation, and enhanced text–image alignment.
• Empirical evaluations demonstrate reduced attack success rates and maintained perceptual quality while promoting novelty and diversity in generated images.

Dynamic VLM-Guided Negative Prompting is an inference-time framework in which a Vision–LLM (VLM) is used to adaptively generate and inject negative prompts into the sampling loop of text-to-image diffusion models. Rather than relying on static or heuristic hard-coded negatives, the method dynamically probes the evolving intermediate states of the generative model, uses the VLM to identify undesired or “conventional” content as it emerges, and then steers subsequent denoising steps away from those elements via context-dependent negative guidance. This leads to improved safety, enhanced creative generation, and greater text–image alignment with minimal degradation of perceptual quality. The methodology has been empirically validated in multiple research contexts including safety filtering and creative sampling.

1. Motivation and Conceptual Framework

Static negative prompting—where a fixed list of undesirable objects, attributes, or categories is applied uniformly throughout the denoising process—has been shown to have two fundamental flaws: overcorrection, in which benign details are inappropriately erased, and undercorrection, in which late-emerging, context-specific undesired content escapes suppression. Dynamic VLM-Guided Negative Prompting (VL-DNP) addresses these issues by periodically querying a pretrained VLM with the provisional output of the diffusion process, allowing for fine-grained, stepwise identification and subsequent negation of harmful or generic content as it appears (Chang et al., 30 Oct 2025, Golan et al., 12 Oct 2025).

In creative generation settings, this methodology steers generative models away from “typical” predictions by adaptively identifying and forbidding the specific visual concepts most strongly associated (in the VLM embedding space) with the current image, thereby accessing lower-probability regions of the output distribution without sacrificing overall recognizability or category validity (Golan et al., 12 Oct 2025).

2. Dynamic Negative Prompting: Workflow and Algorithmic Realization

VL-DNP is implemented as a wrapper over standard text-to-image diffusion pipelines utilizing classifier-free guidance (CFG). Rather than applying a constant negative prompt vector throughout the full set of denoising steps, VL-DNP selects a finite, scheduled set of timesteps (e.g., $$\mathcal{T} = \{45, 44, 43, 41, 38, 34, 29, 23, 16, 8\}$$ for $T = 50$ steps), where it queries the VLM. The main sampling loop operates as follows (Chang et al., 30 Oct 2025):

1. At each scheduled timestep $t_i$, predict the clean image $\hat{x}_0^{(i)}$ from the current latent.
2. Pass $\hat{x}_0^{(i)}$ (often with a handful of demonstration pairs $\mathcal{D}$ for few-shot steering) to the VLM, which returns a concise, context-specific negative prompt $c^{-}_{t_i}$.
3. For all steps following $t_i$ up to, but not including, the next query in $\mathcal{T}$, incorporate negative guidance using $c^{-}_{t_i}$ in the CFG scoring function.

The classifier-free guidance with dynamic negative prompting is operationalized as:

$$\tilde{s}_{\theta}(x_t, t, c^+, c^-_{t_i}) = \nabla_{x_t} \log p(x_t|c^+) + \omega_{pos}\, (\nabla_{x_t} \log p(x_t|c^+) - \nabla_{x_t} \log p(x_t)) - \omega_{neg} (\nabla_{x_t} \log p(x_t|c^-_{t_i}) - \nabla_{x_t} \log p(x_t))$$

where $c^+$ is the positive prompt, and $\omega_{pos}$, $\omega_{neg}$ are the respective guidance strengths.

The method generalizes to settings where a negative prompt list is accumulated, and the negative embedding grows stepwise—steering generation away from the VLM-identified content, as in creative applications (Golan et al., 12 Oct 2025).

3. VLM Prompting Protocol and Embedding Integration

The VLM is prompted through few-shot templates designed to elicit concise lists of unwanted or conventional visual elements, such as:

“Below is an image. Identify any potentially inappropriate or unwanted visual elements and respond with a comma-separated list of objects or attributes to avoid.”

Demonstration pairs $(\text{image}, \text{negative-concept list})$ are included to bias the output style toward actionable negatives. The VLM response $c^-$ is then tokenized and encoded using the diffusion model’s text encoder (typically CLIP-based), giving an embedding $E(c^-)$ that is directly comparable to the positive embedding in the guidance computation. This mechanism is critical, as both positive and negative guidance are enforced in the same semantic space, and only contextually relevant negative content is targeted at each stage (Chang et al., 30 Oct 2025).

In creative generation, the accumulating negative prompts reflect the VLM’s running assessment of what the image most resembles within the training distribution, systematically pushing sampling toward higher-novelty, lower-probability modes (Golan et al., 12 Oct 2025).

4. Theoretical Insights

Dynamic VLM-guided negative prompting is supported by several empirical and theoretical considerations:

• Momentum Effect: Per-step denoising velocities are highly correlated across timesteps, so a negative prompt introduced early in the process exerts persistent influence even if not recomputed at every step. This property allows the reduction of VLM query frequency with negligible impact on effectiveness (Golan et al., 12 Oct 2025).
• Inducing Effect: Overly generic or excessive negation can paradoxically induce the very visual concepts one intends to suppress. Dynamic, context-specific negatives mitigate this effect by targeting only currently-present, concrete content, thereby reducing unintentional induction (Golan et al., 12 Oct 2025).
• The mechanism may be viewed as a “discrete max-margin” strategy in the negative prompt space, akin to identifying and penalizing the directions of strongest VLM alignment at each step, but without continuous gradient-based optimization (Golan et al., 12 Oct 2025).

5. Empirical Evaluation and Performance Metrics

Dynamic VLM-Guided Negative Prompting has been evaluated across several settings:

• Safety Filtering: On safety-oriented benchmarks (e.g., Ring-a-Bell-16, P4D, Unlearn-Diff), VL-DNP achieves lower Attack-Success Rate (ASR) and Toxic Rate (TR) than static negative prompting at matched guidance strengths, while incurring smaller drops in CLIP alignment and smaller increases in FID. For instance, at $\omega_{neg}=20.0$, static negatives yield ASR=0.025, CLIP drops from 0.313 to 0.277, and FID rises from 107 to 153; VL-DNP achieves ASR=0.011, CLIP=0.311, FID$\approx$15, and reduces inference time by opportunistically halting negative guidance early (Chang et al., 30 Oct 2025).
• Creative Generation: On categories such as “pet,” “plant,” “garment,” and “vehicle”, dynamic negative prompting increases Relative Typicality (novelty metric) from 1.64 (reference prompt) to 2.19, and Vendi Score (diversity metric) from 3.17 to 5.79, with only a marginal decrease in CLIP Score. Human studies corroborate gains in perceived novelty and maintain near-baseline category validity (Golan et al., 12 Oct 2025).

Metrics commonly reported include CLIP score (text-image alignment), FID (perceptual quality), ASR/TR (safety), Relative Typicality, GPT Novelty Score, Vendi Score (diversity), and inference time per image. Limiting VLM queries to 10-15 early denoising steps preserves over 95% of the novelty and safety gains while substantially reducing compute overhead (Chang et al., 30 Oct 2025, Golan et al., 12 Oct 2025).

Empirical Comparison Table

Approach	Safety (ASR ↓, TR ↓)	CLIP Score (↑)	FID (↓)	Novelty (↑)
Static Neg	Moderate/High	Degraded	High	Baseline
VL-DNP	Lower	Near-baseline	Low	Significantly improved

“↑” indicates higher is better; “↓” indicates lower is better.

6. Relationship to Other Dynamic Prompting Paradigms

While VL-DNP is architected for denoising diffusion models, the principle of dynamic, example-driven negative prompting is present in other black-box VLM optimization contexts. “LLMs as Black-Box Optimizers for Vision-LLMs” demonstrates that dynamic negative prompts, incorporated via the inclusion of low-performing template examples (bottom-$k$ prompts) in a prompt ranking pool, enable LLMs to more efficiently explore the prompt space for one-shot image classification and generative tasks (Liu et al., 2023). Inclusion of dynamic negatives provides implicit gradient direction in prompt space, enabling faster and more stable convergence compared to using positive exemplars alone.

7. Limitations and Open Challenges

Dynamic VLM-Guided Negative Prompting introduces several practical and theoretical considerations:

• Computational Overhead: Frequent VLM queries add 10–50 seconds per image depending on VLM size; limiting to select steps or using lightweight VLMs (e.g., ViLT, BLIP-1) can mitigate this (Golan et al., 12 Oct 2025).
• Prompt/Question Design: The choice of VLM prompts and framing critically affects the specificity and utility of negative feedback; automated selection or more sophisticated few-shot templating remain open directions.
• Semantic Completeness: Although the procedure explores away from known or conventional modes, it does not guarantee coverage of all plausible semantic subregions, particularly for rare or underspecified categories.
• VLM Quality: The effectiveness of the approach correlates with VLM capacity; more powerful VLMs (e.g., GPT-4o, Qwen2.5) yield greater safety and novelty gains, but at greater resource cost (Golan et al., 12 Oct 2025).

A plausible implication is that combining VLM-guided negativity with user-in-the-loop supervision or explicit attribute priors may further extend the coverage of semantic space and improve controllability.

---

Dynamic VLM-Guided Negative Prompting represents a generalizable, model-agnostic framework that converts the VLM from a passive decoder or post-hoc critic into an active, adaptive feedback agent during generation. It pushes the state-of-the-art in both safety filtering and creative sampling by tightly integrating multimodal intelligence and context-aware textual steering into diffusion-based pipelines (Chang et al., 30 Oct 2025, Golan et al., 12 Oct 2025, Liu et al., 2023).