Papers
Topics
Authors
Recent
Search
2000 character limit reached

NVQG: Negative Visual Query Guidance

Updated 4 July 2026
  • Negative Visual Query Guidance (NVQG) is a technique that integrates explicit negative constraints into visual model processing to suppress undesired content.
  • It leverages interventions in feature and attention spaces within diffusion models, style prompting, and retrieval to finely control output quality.
  • NVQG methods offer practical benefits such as enhanced diversity, improved retrieval accuracy, and reduced similarity to undesired content, as shown by specific performance metrics.

Negative Visual Query Guidance (NVQG) denotes a family of mechanisms that steer visual models away from unwanted content by injecting explicitly negative constraints into generation, retrieval, or alignment. In recent work, the term has been used for image-based adversarial guidance during diffusion inference, query-swapped negative branches for style prompting, dynamically regenerated negative prompts from intermediate denoising states, representation steering for negated retrieval queries, hard-negative supervision in egocentric localization, and conditional semantic comparison for medical vision-LLMs. The unifying objective is explicit absence control: the system is not only asked what to produce or retrieve, but also what to exclude (Singh et al., 2024, Chang et al., 30 Oct 2025, Jeong et al., 8 Oct 2025, Ko et al., 28 May 2026, Sammani et al., 20 Mar 2026, Kim et al., 30 Jan 2026, Xu et al., 2022).

1. Conceptual scope and representative formulations

NVQG is not a single algorithm. Rather, it is a recurring design pattern in which a negative query modifies the model at the locus where visual semantics are assembled: transformer features, attention outputs, denoising scores, similarity geometry, or candidate comparisons. In diffusion models, this usually means altering the reverse trajectory at inference time; in retrieval and alignment, it means changing ranking or discrimination so that negated constraints become first-class decision variables.

Setting Negative signal Representative formulation
Visual-reference diffusion guidance Reference-image features Omerge=(1+αt)Osrc−αtOtargetO_{\text{merge}} = (1+\alpha_t) O_{\text{src}} - \alpha_t O_{\text{target}}
Style prompting Query-swapped leakage branch ϵ^(xt)=(1+w) ϵ¨θ(xt,Qtext,KVvisual)−w ϵθ(xt,∅)−w′ ϵ¨θ(xt,Qvisual)\hat{\epsilon}(x_t) = (1+w)\,\ddot{\epsilon}_\theta(x_t, Q_{\text{text}}, KV_{\text{visual}}) - w\,\epsilon_\theta(x_t,\emptyset) - w'\,\ddot{\epsilon}_\theta(x_t, Q_{\text{visual}})
Few-step negative prompting Flipped negative values OVSF=APVP−αANVNO_{\text{VSF}} = A_P V_P - \alpha A_N V_N
Orthogonal concept suppression Orthogonal negative attention component O′=Opos−αN⊥O' = O_{\text{pos}} - \alpha N_\perp
Dynamic negative prompting Time-varying negative branch ϵ^t=ϵuncond+ωpos(ϵpos−ϵuncond)−ωneg(t)(ϵneg−ϵuncond)\hat{\epsilon}_t = \epsilon_{\text{uncond}} + \omega_{\text{pos}}(\epsilon_{\text{pos}}-\epsilon_{\text{uncond}}) - \omega_{\text{neg}}(t)(\epsilon_{\text{neg}}-\epsilon_{\text{uncond}})
Evidence-calibrated removal Presence-constrained projection ϵtCRG=ϵθ(xt,t,c)−λ0wtCRGΔϵt\epsilon_t^{\text{CRG}} = \epsilon_\theta(x_t,t,c) - \lambda_0 w_t^{\text{CRG}} \Delta\epsilon_t

A persistent theme is that text-only negation is often too coarse. Several papers explicitly argue that negative prompts alone struggle with complex visual concepts, copyrighted characters, or precise exclusions, which motivates moving the negative signal into feature space or attention space rather than leaving it in prompt text alone (Singh et al., 2024, Guo et al., 11 Aug 2025, Ko et al., 28 May 2026).

2. Feature-space repulsion from visual references

A direct diffusion instantiation of NVQG is "Negative Token Merging" (NegToMe), which performs adversarial guidance using visual features from a reference image or from other images in the same batch. NegToMe is inserted between the attention and MLP branches inside transformer blocks and operates on the attention output tensor O∈RB×N×DO \in \mathbb{R}^{B \times N \times D}. For each source token, it computes cosine similarity against reference tokens, selects the best-matching token if the maximum similarity exceeds a threshold τ\tau, and applies linear extrapolation to repel the source away from the matched target: Omerge=(1+αt)Osrc−αtOtargetO_{\text{merge}} = (1+\alpha_t) O_{\text{src}} - \alpha_t O_{\text{target}}. Tokens without a sufficiently strong match are left unchanged. The same framework supports masked matching through a bias term log⁡(M~ref+ϵ)\log(\tilde{M}_{\text{ref}}+\epsilon) when only part of the reference is undesired (Singh et al., 2024).

This mechanism is explicitly training-free, uses only marginally higher inference time, and is compatible with both latent-diffusion UNets and MM-DiT architectures such as Flux. The paper reports that it adds less than 4%4\% inference time, can be implemented in a few lines of code, and works even for models that do not natively support negative prompts. Its applications include batch-wise diversity enhancement and copyrighted-content mitigation. In the latter setting, masked NegToMe reduces visual similarity to copyrighted content by 34.57%34.57\% while maintaining text-image alignment and quality; in the former, it decreases DreamSim, increases Entropy, and preserves or improves FID and IS across SDXL and Flux (Singh et al., 2024).

The significance of this line of work is that the negative query is no longer an external text string but a visual exemplar encoded at the same layer and timestep as the current latent state. NVQG here is therefore literal visual-query guidance: the model is repelled from a specific reference in its own internal feature geometry.

3. Attention-space NVQG in style prompting and few-step generation

In style-conditioned generation, NVQG has been used to suppress content leakage from visual style prompts. StyleKeeper extends classifier-free guidance to self-attention swapping: the main branch keeps the query from the text-conditioned generation process while injecting keys and values from the style branch, and the negative branch intentionally simulates leakage by swapping in queries from the visual style branch. The operational score is

ϵ^(xt)=(1+w) ϵ¨θ(xt,Qtext,KVvisual)−w ϵθ(xt,∅)−w′ ϵ¨θ(xt,Qvisual).\hat{\epsilon}(x_t)=(1+w)\,\ddot{\epsilon}_\theta(x_t,Q_{\text{text}},KV_{\text{visual}})-w\,\epsilon_\theta(x_t,\emptyset)-w'\,\ddot{\epsilon}_\theta(x_t,Q_{\text{visual}}).

The method is applied only to self-attention in late upblocks, with DDIM sampling for 50 steps and w=7.0w=7.0. For real-image style prompts, the style latent is obtained by stochastic encoding and optional AdaIN-based color calibration over a selected timestep window. Quantitatively, StyleKeeper reports Gram-based style similarity of $0.791$ versus $0.759$ for StyleAligned, $0.768$ for IP-Adapter, $0.759$ for DB-LoRA, and $0.659$ for StyleDrop, and a user-study preference of 58.15%58.15\% against 13.15%13.15\%, 18.47%18.47\%, 7.66%7.66\%, and 2.58%2.58\% respectively (Jeong et al., 8 Oct 2025).

For few-step image and video generation, Value Sign Flip (VSF) implements NVQG by flipping the sign of attention values associated with negative tokens while preserving the keys. With text tokens partitioned into positive and negative subsets, the output becomes

OVSF=APVP−αANVN.O_{\text{VSF}} = A_P V_P - \alpha A_N V_N.

In MMDiT architectures, the negative prompt is duplicated into N(0)N^{(0)} and N(1)N^{(1)}, masks isolate the intended image-to-negative path, and an optional bias −β-\beta is added to the logits for I→N(1)I \rightarrow N^{(1)}. The design remains single-pass per step, unlike CFG-style two-pass subtraction. On SD-3.5 Large Turbo, the reported scores are Positive $0.870$, Negative $0.545$, and Quality $0.952$, compared with NAG at $0.993/0.220/0.968$, NASA at $0.970/0.380/0.867$, and no negative guidance at $0.990/0.195/0.968$; the paper also notes runtime of 3.00 3.00\,s for VSF versus 2.14 2.14\,s baseline on A100 at 1024×10241024\times1024 with 8 steps (Guo et al., 11 Aug 2025).

A third attention-space variant, Orthogonal Negative Guidance, operates in the image-to-text attention outputs of MM-DiT blocks. For each image token and head, it computes positive and negative attention outputs, projects the negative vector onto the positive one, and subtracts only the orthogonal component:

ni⊥=ni−projpi(ni),O′(l,h)[i,:]=Opos(l,h)[i,:]−αni⊥.n_i^\perp = n_i - \mathrm{proj}_{p_i}(n_i), \qquad O'^{(l,h)}[i,:] = O_{\text{pos}}^{(l,h)}[i,:] - \alpha n_i^\perp.

The image-side features are shared between positive and negative branches for spatial alignment. The reported defaults are N=28N=28, α=4\alpha=4, τ=2\tau=2 for FLUX-dev and N=4N=4, α=2\alpha=2, τ=0\tau=0 for FLUX-schnell. On the combined DCS-Bench evaluation, the method reaches 86.88%86.88\% suppression, Alignment $4.64$, and Quality $4.63$ on FLUX-dev, and 85.25%85.25\%, $4.67$, and $4.39$ on FLUX-schnell; human preference is $70.81$ versus $52.03$ for Qwen-Image-Edit and $46.22$ for NAG (Ko et al., 28 May 2026).

4. Dynamic and evidence-calibrated negative prompting

A separate NVQG line keeps the negative signal textual but regenerates or reweights it using intermediate visual evidence. Dynamic VLM-guided negative prompting runs a standard CFG-based latent diffusion model and, at query steps {45,44,43,41,38,34,29,23,16,8}\{45,44,43,41,38,34,29,23,16,8\} out of 50, decodes an intermediate clean image estimate x^0(i)\hat{x}_0^{(i)}, queries Qwen2.5-VL-7B-Instruct for a concise negative prompt, encodes the returned text, and uses a three-branch score

ϵ^t=ϵuncond+ωpos(ϵpos−ϵuncond)−ωneg(t)(ϵneg−ϵuncond).\hat{\epsilon}_t = \epsilon_{\text{uncond}} + \omega_{\text{pos}}(\epsilon_{\text{pos}}-\epsilon_{\text{uncond}}) - \omega_{\text{neg}}(t)(\epsilon_{\text{neg}}-\epsilon_{\text{uncond}}).

The reported system uses Stable Diffusion v1.4, DPM-Solver++ with 50 steps, and ωneg∈{7.5,15,20,25}\omega_{\text{neg}} \in \{7.5,15,20,25\}. At ωneg=20\omega_{\text{neg}}=20, dynamic NVQG attains ASR/TR of $0.011/0.081$ on Ring-a-Bell-16, $0.113/0.163$ on P4D, and $0.085/0.139$ on Unlearn-Diff, with CLIP approximately $0.311$ and FID approximately $15.3$. The trade-off is latency: about 29.7 29.7\,s per image versus about 12 12\,s for static negatives and about 8.0 8.0\,s with no negatives (Chang et al., 30 Oct 2025).

Concept Removal Guidance (CRG) replaces fixed negative weights with an evidence-calibrated projection. It estimates concept presence from the model’s own noise predictions via a step-wise presence signal CPk\mathrm{CP}_k, then solves a minimum-perturbation problem that enforces a threshold τ\tau on the predicted presence of the unwanted concept. When suppression is triggered, the update is

ϵtCRG=ϵθ(xt,t,c)−λ0wtCRGΔϵt,wtCRG=CPk−τk∥Δϵt∥22,\epsilon_t^{\text{CRG}} = \epsilon_\theta(x_t,t,c) - \lambda_0 w_t^{\text{CRG}} \Delta\epsilon_t, \qquad w_t^{\text{CRG}} = \frac{\mathrm{CP}_k - \tau}{k\lVert \Delta\epsilon_t \rVert_2^2},

with Δϵt=ϵθ(xt,t,c−)−ϵθ(xt,t)\Delta\epsilon_t = \epsilon_\theta(x_t,t,c^{-}) - \epsilon_\theta(x_t,t). On SD v1.4 with 50-step DDPM, inference time rises from 3.20 3.20\,s for the base model to 4.80 4.80\,s for CRG. Across six red-teaming suites for nudity removal, the reported Attack Success Rate is as low as $0.011$–$0.164$ depending on the suite, while COCO-1k benign fidelity reaches CLIP approximately $30.85$ and FID approximately $36.05$ (Choi et al., 29 Jun 2026).

These two approaches share the same principle: the negative guidance strength should depend on what the evolving image already contains, not only on a static negative list supplied before denoising begins.

5. NVQG beyond generation: retrieval, localization, and vision-language alignment

Outside diffusion sampling, NVQG has been used to make negated queries operational in ranking and discrimination. In egocentric Visual Query 2D Localization, the baseline detector exhibited high false positives on background frames because training used only clean, stable, labeled frames while evaluation contained noisy, blurry, or unlabeled background frames. The proposed solution explicitly treated proposals from background frames, cross-video samples, and low-IoU proposals as negatives, applied hard-negative mining, and maintained a positive-negative ratio of 1:641{:}64. This reduced training from about 15 days to less than 24 hours, improved stAP to 0.17%0.17\%, which is 31%31\% higher than the baseline, and obtained first place on the public leaderboard (Xu et al., 2022).

In CLIP-like retrieval, negation has been reformulated as a geometric intervention problem. A study on negation in joint vision-language embeddings constructs 4,000 matched affirmative–negated COCO caption pairs, shows that logistic regression on EOS hidden states reaches at least 99%99\% test accuracy at intermediate layers for ViT-B/32, ViT-B/16, and ViT-L/14, and uses the resulting classifier weights as a negation direction. Test-time steering is applied as

h′l=(1−α)hl+αv~negl∥hl∥.h'^l = (1-\alpha) h^l + \alpha \tilde{v}_{\text{neg}}^l \lVert h^l \rVert.

With best α≈0.13\alpha \approx 0.13, steering reaches Top-1/Avg.5/Top-5 scores of $54.3/53.3/98.0$ on SimpleNeg under Qwen-3-VL-32B judging, surpassing the CLIP baseline at $44.9/44.9/84.7$, and improves R@1R@1 on the shifted N-COCO benchmark to $0.80$ versus $0.60$ for CLIP (Sammani et al., 20 Mar 2026).

In medical vision-LLMs, Bi-MCQ reformulates alignment as conditional semantic comparison rather than global InfoNCE-style similarity maximization. It jointly trains Image-to-Text and Text-to-Image multiple-choice tasks over affirmative, negative, and mixed prompts, using direction-specific cross-attention modules and MLP scorers. The inference procedure computes paired scores for “There is [disease].” and “There is no [disease].” and converts them into a Positive–Negative Combined probability via a 2-way softmax. On ChestXray14, Open-I, CheXpert, and PadChest, the reported negation gains reach up to $0.47$ AUC over CARZero zero-shot performance, PNC improves by up to $0.08$, and the affirmative-negative AUC gap is reduced by an average of $0.12$ relative to InfoNCE-based fine-tuning (Kim et al., 30 Jan 2026).

Across these settings, NVQG is not limited to synthesis. It also denotes methods that force the model to adjudicate absence claims, reject confounders, and distinguish semantically plausible but logically incorrect matches.

6. Evaluation regimes, failure modes, and recurrent misconceptions

A central misconception in this literature is that negation can be handled by appending “not,” “no,” or “without” to a prompt. Multiple papers reject that premise in different ways. Visual-reference guidance argues that text-only negative prompts are coarse and often insufficient for complex visual concepts or precise copyrighted elements; few-step guidance argues that conventional negative prompting in distilled models frequently yields mixtures of positive and negative images; negation-aware retrieval argues that cosine ranking does not test logical correctness; and medical alignment work argues that InfoNCE treats negation as a minor linguistic variation rather than a meaning-inverting operator (Singh et al., 2024, Guo et al., 11 Aug 2025, Sammani et al., 20 Mar 2026, Kim et al., 30 Jan 2026).

Evaluation is correspondingly heterogeneous. Generation papers report suppression and fidelity using FID, IS, DreamSim, CLIPScore, VQAScore, Entropy, DINO, Gram-matrix similarity, LPIPS, ASR, and Toxic Rate; retrieval and alignment papers use Recall@K, Top-1/Avg.5/Top-5 under MLLM-as-a-judge, stAP25 and tAP25, and Positive/Negative/PNC AUC. One recurrent methodological point is that retrieval-only metrics can obscure whether negation is actually understood, which motivates semantic-first gating and judge-based verification in the CLIP negation work (Chang et al., 30 Oct 2025, Jeong et al., 8 Oct 2025, Ko et al., 28 May 2026, Xu et al., 2022, Sammani et al., 20 Mar 2026).

Failure modes are also consistent across formulations. Excessive repulsion coefficients or weak similarity thresholds can distort features and degrade fidelity in NegToMe; over-strong negative guidance can introduce artifacts or semantic drift in dynamic VLM prompting; large w′w' can oversuppress style richness in StyleKeeper; high α\alpha can remove adjacent or entangled structures in VSF and orthogonal guidance; single-direction steering in CLIP remains limited for double negation, quantifiers, and ambiguous scope; and batch-level candidate construction in Bi-MCQ can be sensitive to prompt design and class imbalance (Singh et al., 2024, Chang et al., 30 Oct 2025, Jeong et al., 8 Oct 2025, Guo et al., 11 Aug 2025, Ko et al., 28 May 2026, Sammani et al., 20 Mar 2026, Kim et al., 30 Jan 2026).

A plausible implication is that NVQG is becoming a general control principle rather than a narrow prompting trick. The surveyed methods place negative information where the model actually resolves visual semantics: token correspondence, attention aggregation, denoising evidence, candidate competition, or embedding geometry. Their differences lie less in objective—explicit concept suppression—and more in where the negative query enters the computation and how aggressively it is allowed to perturb the positive trajectory.

Topic to Video (Beta)

No one has generated a video about this topic yet.

Whiteboard

No one has generated a whiteboard explanation for this topic yet.

Follow Topic

Get notified by email when new papers are published related to Negative Visual Query Guidance (NVQG).