NEAT: Negation-Aware Test-Time Adaptation

• The paper introduces NEAT as a test-time adaptation method that updates only text encoder LayerNorms for improved handling of negated queries in VLMs.
• It employs a composite objective combining refined entropy minimization, semantics reversion, and textual debiasing to address dual-concept distribution shifts.
• Empirical results across retrieval, multiple-choice reasoning, and classification tasks demonstrate significant accuracy gains with minimal data and parameter updates.

Negation-Aware Test-Time Adaptation (NEAT) is a test-time adaptation method for vision-LLMs (VLMs) that targets negation understanding without negation-specific post-training. It is motivated by the observation that pretrained VLMs often mishandle expressions such as “not,” “no,” and “without,” despite otherwise strong zero-shot transfer. Rather than treating this solely as a data-scarcity problem, NEAT frames the failure as a distributional geometry problem induced by negation: negated text drifts away from semantically consistent affirmative text while remaining spuriously close to semantically reversed text. The method therefore adapts only the LayerNorm parameters in the text encoder during inference on unlabeled negated queries, using a composite objective over refined entropy minimization, semantics reversion, and textual debiasing (Han et al., 25 Jul 2025).

1. Problem formulation and motivation

NEAT is defined in the setting of test-time adaptation under concept shift. A pretrained VLM $f_\theta$ is trained on image-text pairs $\mathcal{D}_{train}=\{\mathcal{V},\mathcal{T}\}$, but at inference the textual inputs are negation-conditioned queries $\mathcal{T}_{test}=\{\hat{t}_i\}$, so that $P(\mathcal{T}_{test}) \neq P(\mathcal{T})$. The method is motivated by practical scenarios in which false or excluded content must be represented explicitly, including image and video retrieval under negated captions and medical use cases such as searches that exclude particular findings.

The central diagnosis is the presence of dual-concept shifts. The first is distribution shift within semantic consistency: an affirmative text $t_i$ and its negation-conditioned counterpart $\hat{t}_i$ should remain close because they refer to the same underlying scene semantics with one exclusion, yet their distributions become far apart. The second is distribution consistency within semantic shift: a negation-conditioned text $\hat{t}_i$ and a semantically reversed caption $\bar{t}_i$ should be far apart, yet they are often embedded close to one another. On CIFAR10, the paper visualizes this with UMAP and quantifies it with Maximum Mean Discrepancy, reporting an MMD of $1.57$ between normal text and negation-conditioned text, but only $0.08$ between negation-conditioned and reversed negation-conditioned text.

This framing separates NEAT from earlier data-centric post-training approaches. The paper groups prior methods such as NegBench / NegFull, NegationCLIP, CoNCLIP, and NegCLIP under the view that negation failures arise mainly from insufficient negation-rich supervision. Those methods can be effective, but the paper characterizes them as resource-heavy because they rely on large synthetic corpora, additional training, and substantial compute. NEAT is positioned as a “low-carbon” alternative that exploits the pretrained model’s existing affirmative knowledge and modifies only distribution-sensitive textual parameters at inference. Its efficiency claim is explicit: compared with strong post-training baselines, it uses less than $\mathcal{D}_{train}=\{\mathcal{V},\mathcal{T}\}$0 per-thousand of the data, and $\mathcal{D}_{train}=\{\mathcal{V},\mathcal{T}\}$1 per-thousand of the trainable parameters (Han et al., 25 Jul 2025).

2. Method architecture and optimization objective

NEAT updates only the LayerNorm parameters in the text encoder. The visual encoder is frozen, non-LayerNorm text weights are frozen, the optimizer is AdamW, and the reported batch size for test-time adaptation experiments is $\mathcal{D}_{train}=\{\mathcal{V},\mathcal{T}\}$2. The paper emphasizes that NEAT does not require full-model fine-tuning, supervised negation labels, memory banks, prompt tuning as the primary mechanism, entropy minimization alone, external retrieval databases, or iterative per-sample optimization over all weights.

The pipeline begins with LLM-based parsing. For each negation-conditioned query $\mathcal{D}_{train}=\{\mathcal{V},\mathcal{T}\}$3, an LLM parses the text into a positive component $\mathcal{D}_{train}=\{\mathcal{V},\mathcal{T}\}$4 and a negative component $\mathcal{D}_{train}=\{\mathcal{V},\mathcal{T}\}$5, where the negative component is rephrased affirmatively. For example, “a photo of a dog not on grass” is decomposed into “a photo of a dog” and “a photo of grass.” The same LLM also generates a semantically reversed caption $\mathcal{D}_{train}=\{\mathcal{V},\mathcal{T}\}$6 by swapping the roles of the positive and negative parts, such as “a photo of grass but not of a dog.” The paper uses Llama-3-8B for this decomposition and generation.

NEAT then optimizes three coupled losses. The first is refined entropy minimization $\mathcal{D}_{train}=\{\mathcal{V},\mathcal{T}\}$7, but unlike generic TTA it does not rely on raw text-image similarity for candidate selection. It introduces a negation-separated similarity

$\mathcal{D}_{train}=\{\mathcal{V},\mathcal{T}\}$8

with $\mathcal{D}_{train}=\{\mathcal{V},\mathcal{T}\}$9 and $\mathcal{T}_{test}=\{\hat{t}_i\}$0. The first factor rewards agreement with the affirmative content; the second penalizes similarity to the forbidden concept. Candidate images selected with this score are then used inside the entropy term.

The second loss is semantics reversion loss $\mathcal{T}_{test}=\{\hat{t}_i\}$1. The paper motivates it through a mutual-information argument: the reversed caption $\mathcal{T}_{test}=\{\hat{t}_i\}$2 should have low mutual information with the visual candidate selected for $\mathcal{T}_{test}=\{\hat{t}_i\}$3. Rather than contrasting against all unpaired images, NEAT chooses a single hardest negative visual sample $\mathcal{T}_{test}=\{\hat{t}_i\}$4, defined as the visual sample with minimal similarity to $\mathcal{T}_{test}=\{\hat{t}_i\}$5, and uses a hard-negative objective that the paper interprets as a simple margin-like metric loss. Its purpose is to break the spurious closeness between negated and semantically reversed text.

The third loss is textual debiasing $\mathcal{T}_{test}=\{\hat{t}_i\}$6, which directly regularizes the geometry among the negated caption, its affirmative decomposition, and its reversed counterpart:

$\mathcal{T}_{test}=\{\hat{t}_i\}$7

The first term pulls the negated caption toward its semantically consistent affirmative core; the second pushes it away from the reversed caption. The reported hyperparameters are $\mathcal{T}_{test}=\{\hat{t}_i\}$8 for image/video tasks, $\mathcal{T}_{test}=\{\hat{t}_i\}$9 for image/video tasks, $P(\mathcal{T}_{test}) \neq P(\mathcal{T})$0, and $P(\mathcal{T}_{test}) \neq P(\mathcal{T})$1, with stable performance reported for $P(\mathcal{T}_{test}) \neq P(\mathcal{T})$2.

The overall adaptation objective is

$P(\mathcal{T}_{test}) \neq P(\mathcal{T})$3

where $P(\mathcal{T}_{test}) \neq P(\mathcal{T})$4 denotes the text-encoder normalization parameters. The paper evaluates both offline adaptation, in which all available unlabeled negation data are used before inference, and online adaptation, in which adaptation occurs on the current test batch (Han et al., 25 Jul 2025).

3. Benchmarks, tasks, and empirical performance

The evaluation spans image retrieval, video retrieval, multiple-choice negation reasoning, zero-shot classification under synthetic negation, and medical transfer. The named datasets and task sizes are: MS-COCO Retrieval-Neg ($P(\mathcal{T}_{test}) \neq P(\mathcal{T})$5) and MCQ-Neg ($P(\mathcal{T}_{test}) \neq P(\mathcal{T})$6); VOC2007 MCQ-Neg ($P(\mathcal{T}_{test}) \neq P(\mathcal{T})$7); MSR-VTT Retrieval-Neg ($P(\mathcal{T}_{test}) \neq P(\mathcal{T})$8) and MCQ-Neg ($P(\mathcal{T}_{test}) \neq P(\mathcal{T})$9); and CheXpert Binary Classification-Aff ($t_i$0) and Binary Classification-Neg ($t_i$1). Additional transfer experiments use CIFAR10, CIFAR100, Caltech101, OxfordPets, RESISC45, EuroSAT, STL10, SUN397, and ImageNet1K. Reported metrics include $t_i$2, $t_i$3, $t_i$4, $t_i$5, MCQ accuracy by subtype and overall, top-1 accuracy, error rate, and binary disease accuracy.

On MS-COCO Retrieval-Neg, the base CLIP model has $t_i$6, while CLIP + NEAT reaches $t_i$7, a gain of $t_i$8. On a CC12M-finetuned CLIP, NEAT further improves performance from $t_i$9 to $\hat{t}_i$0, a gain of $\hat{t}_i$1. On MSR-VTT Retrieval-Neg, base CLIP has $\hat{t}_i$2 and CLIP + NEAT reaches $\hat{t}_i$3, a gain of $\hat{t}_i$4; on a CC12M-finetuned CLIP, the score rises from $\hat{t}_i$5 to $\hat{t}_i$6, a gain of $\hat{t}_i$7. NEAT also improves NegCLIP: on COCO, from $\hat{t}_i$8 to $\hat{t}_i$9 or $\hat{t}_i$0 depending on initialization; on MSR-VTT, from $\hat{t}_i$1 to $\hat{t}_i$2 or $\hat{t}_i$3.

The comparison with generic TTA is central. On MS-COCO, the base CLIP model has Retrieval-Neg $\hat{t}_i$4 and MCQ-Neg total accuracy $\hat{t}_i$5. NEAT yields $\hat{t}_i$6 on Retrieval-Neg and $\hat{t}_i$7 on MCQ-Neg, improving both by $\hat{t}_i$8 and $\hat{t}_i$9, respectively. On MSR-VTT, base CLIP has Retrieval-Neg $\bar{t}_i$0 and MCQ-Neg $\bar{t}_i$1, while NEAT achieves $\bar{t}_i$2 and $\bar{t}_i$3, gains of $\bar{t}_i$4 and $\bar{t}_i$5. For NegCLIP on MSR-VTT, baseline MCQ-Neg is $\bar{t}_i$6 and NEAT raises it to $\bar{t}_i$7, a gain of $\bar{t}_i$8. The paper emphasizes that generic entropy-based TTA methods such as TENT and SAR can slightly improve retrieval while sharply hurting MCQ accuracy, and that TCR can improve retrieval while degrading MCQ, which the authors interpret as improved modality alignment without improved negation semantics.

Cross-task transfer is unusually strong. After adapting on unlabeled MS-COCO Retrieval-Neg, NEAT transfers to VOC2007 MCQ-Neg, improving CLIP from $\bar{t}_i$9 to $1.57$0 and NegCLIP from $1.57$1 to $1.57$2. On nine zero-shot image classification datasets with synthetic negation, average top-1 accuracy under negation-conditioned classification rises from $1.57$3 to $1.57$4, while the average error on reversed negation-conditioned classification falls from $1.57$5 to $1.57$6. The paper treats the latter as especially strong evidence that NEAT corrects the false consistency between negated and semantically reversed statements. A further transfer experiment swaps NEAT-adapted textual LayerNorms into the medical VLM PLIP and reports gains on CheXpert, with the strongest improvements on Lung Opacity: $1.57$7 on Binary Classification-Aff and $1.57$8 on Binary Classification-Neg (Han et al., 25 Jul 2025).

4. Ablations and interpretive significance

The ablation study on MS-COCO clarifies the functional role of each loss term. The baseline reports I2T $1.57$9, T2I $0.08$0, and MCQ $0.08$1. Using only $0.08$2 raises I2T $0.08$3 to $0.08$4 and T2I $0.08$5 to $0.08$6, but drops MCQ to $0.08$7. This establishes that entropy refinement alone improves retrieval-style alignment while harming harder negation reasoning. Adding semantics reversion, $0.08$8, yields I2T $0.08$9, T2I $\mathcal{D}_{train}=\{\mathcal{V},\mathcal{T}\}$00, and MCQ $\mathcal{D}_{train}=\{\mathcal{V},\mathcal{T}\}$01, showing that $\mathcal{D}_{train}=\{\mathcal{V},\mathcal{T}\}$02 is crucial for generalizable negation behavior. Using all three losses gives I2T $\mathcal{D}_{train}=\{\mathcal{V},\mathcal{T}\}$03, T2I $\mathcal{D}_{train}=\{\mathcal{V},\mathcal{T}\}$04, and MCQ $\mathcal{D}_{train}=\{\mathcal{V},\mathcal{T}\}$05, indicating that $\mathcal{D}_{train}=\{\mathcal{V},\mathcal{T}\}$06 provides additional text-space regularization.

A second ablation compares the proposed negation-separated candidate selection with plain KNN-style candidate selection and reports a substantial advantage for the former. This is consistent with the paper’s claim that raw nearest-neighbor geometry is unreliable under negation shift: a candidate that is visually similar to the positive content but also strongly similar to the negated concept should not be treated as a high-quality adaptation signal.

The interpretation advanced by the paper is therefore narrower than “VLMs do not know negation.” Its broader claim is that pretrained VLMs already encode much of the required affirmative visual-semantic knowledge, but the relative geometry among affirmative, negated, and reversed-negated texts is miscalibrated. This is why a very small intervention—test-time updates to text-encoder normalization layers—can recover a substantial amount of missing capability. In that sense, NEAT presents negation as a failure of distributional structure rather than an absolute absence of concept knowledge. The paper also notes that some formulas contain notation inconsistencies due to typesetting, but the intended mechanism is clear from the loss definitions, workflow, and ablations (Han et al., 25 Jul 2025).

5. Position within the broader negation-adaptation literature

The surrounding literature distinguishes several regimes that are adjacent to NEAT but not equivalent to it.

Method	Adaptation regime	Relation to negation-aware TTA
NEAT	Unlabeled test-time adaptation; updates text-encoder LayerNorms	Explicitly models negation through decomposition, semantic reversal, and dual-concept shifts
TARA	Offline post-training fine-tuning with text triplets only	Not test-time adaptation; yields negation-aware embeddings as an emergent effect
Panda	Source-free batchwise TTA under image corruptions	Uses “negative” evidence as subtractive feature debiasing, not linguistic negation
NAST	Offline supervised selective fine-tuning with LoRA	Not TTA; provides mechanistic evidence for where negation-sensitive adaptation may be concentrated

TARA, “Time Aware Retrieval Adaptation,” is the most relevant non-TTA comparator because it demonstrates that careful hard-negative design can induce negation-aware retrieval without an explicit negation loss. TARA adapts multimodal LLMs offline using only text triplets, freezes the vision and projection networks, and fine-tunes only the LLM weights; the paper is explicit that the model is “trained once” and then evaluated downstream “without fine-tuning.” Its relevance to NEAT lies in the observation that temporally opposite hard negatives regularize the shared embedding space so that it also performs strongly on NegBench, where Tarsier+TARA reaches COCO $\mathcal{D}_{train}=\{\mathcal{V},\mathcal{T}\}$07 and $\mathcal{D}_{train}=\{\mathcal{V},\mathcal{T}\}$08, and MSRVTT $\mathcal{D}_{train}=\{\mathcal{V},\mathcal{T}\}$09, despite not using any negation-specific objective. This supports the narrower claim that semantically local hard negatives can improve negation sensitivity, but it is not evidence for inference-time adaptation (Bagad et al., 15 Dec 2025).

Panda, “Test-Time Adaptation with Negative Data Augmentation,” is closer in protocol because it is a source-free batchwise TTA method, but its notion of negation is feature-space rather than linguistic. Panda constructs negative visual augmentations by shuffling patches across the test batch, extracts a corruption prototype $\mathcal{D}_{train}=\{\mathcal{V},\mathcal{T}\}$10, and debiases each image feature through $\mathcal{D}_{train}=\{\mathcal{V},\mathcal{T}\}$11. The method is motivated by corruption-induced prediction bias rather than textual polarity. Its conceptual overlap with NEAT lies in the use of counter-semantic negative evidence to cancel spurious support. However, Panda operates on image-space nuisance factors and common corruptions rather than affirmative/negated text geometry. It is therefore better understood as a complementary debiasing primitive than as a linguistic negation method (Deng et al., 13 Nov 2025).

NAST, “Layer-Specific Fine-Tuning for Improved Negation Handling in Medical Vision-LLMs,” addresses medical negation through offline supervised fine-tuning rather than TTA, but it contributes a mechanistic perspective that is directly relevant. NAST uses causal tracing effects to weight LoRA gradient updates by layer and reports that negation-sensitive signal is concentrated in early text-encoder layers $\mathcal{D}_{train}=\{\mathcal{V},\mathcal{T}\}$12–$\mathcal{D}_{train}=\{\mathcal{V},\mathcal{T}\}$13, peaking around layer $\mathcal{D}_{train}=\{\mathcal{V},\mathcal{T}\}$14. On its contextual clinical negation benchmark, NAST reaches $\mathcal{D}_{train}=\{\mathcal{V},\mathcal{T}\}$15, $\mathcal{D}_{train}=\{\mathcal{V},\mathcal{T}\}$16, and claim accuracy $\mathcal{D}_{train}=\{\mathcal{V},\mathcal{T}\}$17, and reduces the affirmative-negation claim-accuracy gap from $\mathcal{D}_{train}=\{\mathcal{V},\mathcal{T}\}$18 for CLIP to $\mathcal{D}_{train}=\{\mathcal{V},\mathcal{T}\}$19. This does not constitute test-time adaptation, but it suggests that selective text-side updates may be preferable to uniform adaptation when negation is the target phenomenon (Abbasi et al., 13 Feb 2026).

6. Limitations, assumptions, and emerging directions

NEAT has several explicit constraints. It depends on an LLM—specifically Llama-3-8B—to decompose negated text into positive and negative components and to generate semantically reversed captions. This keeps adaptation lightweight in parameter count, but the full pipeline is not self-contained. The method is also deliberately text-encoder-focused: it assumes that negation is primarily a text-side concept shift and freezes the visual encoder. The paper notes that this may be less effective if the underlying failure is a deeper multimodal reasoning problem rather than a distributional misalignment localized to the text branch.

As with other batch-based TTA methods, performance depends on the available unlabeled test distribution and on hyperparameter settings such as temperature. The paper’s own interpretation is that generic entropy minimization can sharpen the wrong geometry under negation, which is why candidate construction and semantic reversal are indispensable. This means that NEAT should not be conflated with standard entropy-minimization TTA augmented by negated prompts; its operative assumption is more specific, namely that the geometry among affirmative, negated, and reversed-negated text can be repaired by small normalization updates at test time (Han et al., 25 Jul 2025).

Adjacent work indicates several plausible extensions. TARA suggests that hard-negative synthesis preserving most context while flipping a single critical property can regularize retrieval spaces in a way that transfers to negation-sensitive evaluation. Panda suggests that negative evidence can also be operationalized as a subtractive debiasing signal rather than only as a contrastive label. NAST suggests that negation-sensitive updates may be safest and most effective when concentrated in early text layers, and its medical benchmarks further indicate that clinically relevant negation extends beyond simple presence/absence into location and severity. These observations do not alter NEAT’s original formulation, but they suggest broader design space: structured polarity-opposed candidates, layer-selective online updates, and evaluation protocols that measure not only aggregate accuracy but also the affirmative-negation gap and sensitivity to attribute-level exclusions (Bagad et al., 15 Dec 2025, Deng et al., 13 Nov 2025, Abbasi et al., 13 Feb 2026).

In that broader sense, NEAT occupies a specific position in the evolution of negation-aware multimodal learning. It is neither a large-scale negation post-training recipe nor a generic robustness-oriented TTA baseline. Its defining contribution is to cast negation failure as a dual-concept distribution shift and to show that lightweight inference-time adaptation of textual normalization layers can recover substantial negation competence across retrieval, multiple-choice reasoning, synthetic classification, and transfer settings.