NVQG: Negative Visual Query Guidance
- Negative Visual Query Guidance (NVQG) is a technique that integrates explicit negative constraints into visual model processing to suppress undesired content.
- It leverages interventions in feature and attention spaces within diffusion models, style prompting, and retrieval to finely control output quality.
- NVQG methods offer practical benefits such as enhanced diversity, improved retrieval accuracy, and reduced similarity to undesired content, as shown by specific performance metrics.
Negative Visual Query Guidance (NVQG) denotes a family of mechanisms that steer visual models away from unwanted content by injecting explicitly negative constraints into generation, retrieval, or alignment. In recent work, the term has been used for image-based adversarial guidance during diffusion inference, query-swapped negative branches for style prompting, dynamically regenerated negative prompts from intermediate denoising states, representation steering for negated retrieval queries, hard-negative supervision in egocentric localization, and conditional semantic comparison for medical vision-LLMs. The unifying objective is explicit absence control: the system is not only asked what to produce or retrieve, but also what to exclude (Singh et al., 2024, Chang et al., 30 Oct 2025, Jeong et al., 8 Oct 2025, Ko et al., 28 May 2026, Sammani et al., 20 Mar 2026, Kim et al., 30 Jan 2026, Xu et al., 2022).
1. Conceptual scope and representative formulations
NVQG is not a single algorithm. Rather, it is a recurring design pattern in which a negative query modifies the model at the locus where visual semantics are assembled: transformer features, attention outputs, denoising scores, similarity geometry, or candidate comparisons. In diffusion models, this usually means altering the reverse trajectory at inference time; in retrieval and alignment, it means changing ranking or discrimination so that negated constraints become first-class decision variables.
| Setting | Negative signal | Representative formulation |
|---|---|---|
| Visual-reference diffusion guidance | Reference-image features | |
| Style prompting | Query-swapped leakage branch | |
| Few-step negative prompting | Flipped negative values | |
| Orthogonal concept suppression | Orthogonal negative attention component | |
| Dynamic negative prompting | Time-varying negative branch | |
| Evidence-calibrated removal | Presence-constrained projection |
A persistent theme is that text-only negation is often too coarse. Several papers explicitly argue that negative prompts alone struggle with complex visual concepts, copyrighted characters, or precise exclusions, which motivates moving the negative signal into feature space or attention space rather than leaving it in prompt text alone (Singh et al., 2024, Guo et al., 11 Aug 2025, Ko et al., 28 May 2026).
2. Feature-space repulsion from visual references
A direct diffusion instantiation of NVQG is "Negative Token Merging" (NegToMe), which performs adversarial guidance using visual features from a reference image or from other images in the same batch. NegToMe is inserted between the attention and MLP branches inside transformer blocks and operates on the attention output tensor . For each source token, it computes cosine similarity against reference tokens, selects the best-matching token if the maximum similarity exceeds a threshold , and applies linear extrapolation to repel the source away from the matched target: . Tokens without a sufficiently strong match are left unchanged. The same framework supports masked matching through a bias term when only part of the reference is undesired (Singh et al., 2024).
This mechanism is explicitly training-free, uses only marginally higher inference time, and is compatible with both latent-diffusion UNets and MM-DiT architectures such as Flux. The paper reports that it adds less than 0 inference time, can be implemented in a few lines of code, and works even for models that do not natively support negative prompts. Its applications include batch-wise diversity enhancement and copyrighted-content mitigation. In the latter setting, masked NegToMe reduces visual similarity to copyrighted content by 1 while maintaining text-image alignment and quality; in the former, it decreases DreamSim, increases Entropy, and preserves or improves FID and IS across SDXL and Flux (Singh et al., 2024).
The significance of this line of work is that the negative query is no longer an external text string but a visual exemplar encoded at the same layer and timestep as the current latent state. NVQG here is therefore literal visual-query guidance: the model is repelled from a specific reference in its own internal feature geometry.
3. Attention-space NVQG in style prompting and few-step generation
In style-conditioned generation, NVQG has been used to suppress content leakage from visual style prompts. StyleKeeper extends classifier-free guidance to self-attention swapping: the main branch keeps the query from the text-conditioned generation process while injecting keys and values from the style branch, and the negative branch intentionally simulates leakage by swapping in queries from the visual style branch. The operational score is
2
The method is applied only to self-attention in late upblocks, with DDIM sampling for 50 steps and 3. For real-image style prompts, the style latent is obtained by stochastic encoding and optional AdaIN-based color calibration over a selected timestep window. Quantitatively, StyleKeeper reports Gram-based style similarity of 4 versus 5 for StyleAligned, 6 for IP-Adapter, 7 for DB-LoRA, and 8 for StyleDrop, and a user-study preference of 9 against 0, 1, 2, and 3 respectively (Jeong et al., 8 Oct 2025).
For few-step image and video generation, Value Sign Flip (VSF) implements NVQG by flipping the sign of attention values associated with negative tokens while preserving the keys. With text tokens partitioned into positive and negative subsets, the output becomes
4
In MMDiT architectures, the negative prompt is duplicated into 5 and 6, masks isolate the intended image-to-negative path, and an optional bias 7 is added to the logits for 8. The design remains single-pass per step, unlike CFG-style two-pass subtraction. On SD-3.5 Large Turbo, the reported scores are Positive 9, Negative 0, and Quality 1, compared with NAG at 2, NASA at 3, and no negative guidance at 4; the paper also notes runtime of 5s for VSF versus 6s baseline on A100 at 7 with 8 steps (Guo et al., 11 Aug 2025).
A third attention-space variant, Orthogonal Negative Guidance, operates in the image-to-text attention outputs of MM-DiT blocks. For each image token and head, it computes positive and negative attention outputs, projects the negative vector onto the positive one, and subtracts only the orthogonal component:
8
The image-side features are shared between positive and negative branches for spatial alignment. The reported defaults are 9, 0, 1 for FLUX-dev and 2, 3, 4 for FLUX-schnell. On the combined DCS-Bench evaluation, the method reaches 5 suppression, Alignment 6, and Quality 7 on FLUX-dev, and 8, 9, and 0 on FLUX-schnell; human preference is 1 versus 2 for Qwen-Image-Edit and 3 for NAG (Ko et al., 28 May 2026).
4. Dynamic and evidence-calibrated negative prompting
A separate NVQG line keeps the negative signal textual but regenerates or reweights it using intermediate visual evidence. Dynamic VLM-guided negative prompting runs a standard CFG-based latent diffusion model and, at query steps 4 out of 50, decodes an intermediate clean image estimate 5, queries Qwen2.5-VL-7B-Instruct for a concise negative prompt, encodes the returned text, and uses a three-branch score
6
The reported system uses Stable Diffusion v1.4, DPM-Solver++ with 50 steps, and 7. At 8, dynamic NVQG attains ASR/TR of 9 on Ring-a-Bell-16, 0 on P4D, and 1 on Unlearn-Diff, with CLIP approximately 2 and FID approximately 3. The trade-off is latency: about 4s per image versus about 5s for static negatives and about 6s with no negatives (Chang et al., 30 Oct 2025).
Concept Removal Guidance (CRG) replaces fixed negative weights with an evidence-calibrated projection. It estimates concept presence from the model’s own noise predictions via a step-wise presence signal 7, then solves a minimum-perturbation problem that enforces a threshold 8 on the predicted presence of the unwanted concept. When suppression is triggered, the update is
9
with 0. On SD v1.4 with 50-step DDPM, inference time rises from 1s for the base model to 2s for CRG. Across six red-teaming suites for nudity removal, the reported Attack Success Rate is as low as 3–4 depending on the suite, while COCO-1k benign fidelity reaches CLIP approximately 5 and FID approximately 6 (Choi et al., 29 Jun 2026).
These two approaches share the same principle: the negative guidance strength should depend on what the evolving image already contains, not only on a static negative list supplied before denoising begins.
5. NVQG beyond generation: retrieval, localization, and vision-language alignment
Outside diffusion sampling, NVQG has been used to make negated queries operational in ranking and discrimination. In egocentric Visual Query 2D Localization, the baseline detector exhibited high false positives on background frames because training used only clean, stable, labeled frames while evaluation contained noisy, blurry, or unlabeled background frames. The proposed solution explicitly treated proposals from background frames, cross-video samples, and low-IoU proposals as negatives, applied hard-negative mining, and maintained a positive-negative ratio of 7. This reduced training from about 15 days to less than 24 hours, improved stAP to 8, which is 9 higher than the baseline, and obtained first place on the public leaderboard (Xu et al., 2022).
In CLIP-like retrieval, negation has been reformulated as a geometric intervention problem. A study on negation in joint vision-language embeddings constructs 4,000 matched affirmative–negated COCO caption pairs, shows that logistic regression on EOS hidden states reaches at least 0 test accuracy at intermediate layers for ViT-B/32, ViT-B/16, and ViT-L/14, and uses the resulting classifier weights as a negation direction. Test-time steering is applied as
1
With best 2, steering reaches Top-1/Avg.5/Top-5 scores of 3 on SimpleNeg under Qwen-3-VL-32B judging, surpassing the CLIP baseline at 4, and improves 5 on the shifted N-COCO benchmark to 6 versus 7 for CLIP (Sammani et al., 20 Mar 2026).
In medical vision-LLMs, Bi-MCQ reformulates alignment as conditional semantic comparison rather than global InfoNCE-style similarity maximization. It jointly trains Image-to-Text and Text-to-Image multiple-choice tasks over affirmative, negative, and mixed prompts, using direction-specific cross-attention modules and MLP scorers. The inference procedure computes paired scores for “There is [disease].” and “There is no [disease].” and converts them into a Positive–Negative Combined probability via a 2-way softmax. On ChestXray14, Open-I, CheXpert, and PadChest, the reported negation gains reach up to 8 AUC over CARZero zero-shot performance, PNC improves by up to 9, and the affirmative-negative AUC gap is reduced by an average of 0 relative to InfoNCE-based fine-tuning (Kim et al., 30 Jan 2026).
Across these settings, NVQG is not limited to synthesis. It also denotes methods that force the model to adjudicate absence claims, reject confounders, and distinguish semantically plausible but logically incorrect matches.
6. Evaluation regimes, failure modes, and recurrent misconceptions
A central misconception in this literature is that negation can be handled by appending “not,” “no,” or “without” to a prompt. Multiple papers reject that premise in different ways. Visual-reference guidance argues that text-only negative prompts are coarse and often insufficient for complex visual concepts or precise copyrighted elements; few-step guidance argues that conventional negative prompting in distilled models frequently yields mixtures of positive and negative images; negation-aware retrieval argues that cosine ranking does not test logical correctness; and medical alignment work argues that InfoNCE treats negation as a minor linguistic variation rather than a meaning-inverting operator (Singh et al., 2024, Guo et al., 11 Aug 2025, Sammani et al., 20 Mar 2026, Kim et al., 30 Jan 2026).
Evaluation is correspondingly heterogeneous. Generation papers report suppression and fidelity using FID, IS, DreamSim, CLIPScore, VQAScore, Entropy, DINO, Gram-matrix similarity, LPIPS, ASR, and Toxic Rate; retrieval and alignment papers use Recall@K, Top-1/Avg.5/Top-5 under MLLM-as-a-judge, stAP25 and tAP25, and Positive/Negative/PNC AUC. One recurrent methodological point is that retrieval-only metrics can obscure whether negation is actually understood, which motivates semantic-first gating and judge-based verification in the CLIP negation work (Chang et al., 30 Oct 2025, Jeong et al., 8 Oct 2025, Ko et al., 28 May 2026, Xu et al., 2022, Sammani et al., 20 Mar 2026).
Failure modes are also consistent across formulations. Excessive repulsion coefficients or weak similarity thresholds can distort features and degrade fidelity in NegToMe; over-strong negative guidance can introduce artifacts or semantic drift in dynamic VLM prompting; large 1 can oversuppress style richness in StyleKeeper; high 2 can remove adjacent or entangled structures in VSF and orthogonal guidance; single-direction steering in CLIP remains limited for double negation, quantifiers, and ambiguous scope; and batch-level candidate construction in Bi-MCQ can be sensitive to prompt design and class imbalance (Singh et al., 2024, Chang et al., 30 Oct 2025, Jeong et al., 8 Oct 2025, Guo et al., 11 Aug 2025, Ko et al., 28 May 2026, Sammani et al., 20 Mar 2026, Kim et al., 30 Jan 2026).
A plausible implication is that NVQG is becoming a general control principle rather than a narrow prompting trick. The surveyed methods place negative information where the model actually resolves visual semantics: token correspondence, attention aggregation, denoising evidence, candidate competition, or embedding geometry. Their differences lie less in objective—explicit concept suppression—and more in where the negative query enters the computation and how aggressively it is allowed to perturb the positive trajectory.