Structure-Targeted Negatives

• Structure-targeted negatives are negative examples anchored to specific structural relations that selectively violate key dependencies while preserving surrounding context.
• They are applied across neural language modeling, knowledge graph embedding, and image-text retrieval to pinpoint syntactic and semantic nuances.
• Techniques such as margin-based losses, local neighborhood sampling, and hard-negative mining yield measurable improvements in model accuracy and robustness.

Structure-targeted negatives are negative examples, negative statements, or negative computational signals that are anchored to a specific structural relation rather than obtained by unrestricted random corruption. In the cited literature, the targeted structure may be a syntactic dependency such as subject–verb agreement or Negative Polarity Item (NPI) licensing, a graph neighborhood in a knowledge graph, a sibling set in a commonsense knowledge base, a local neighborhood in an embedding space, or a set of hard patch correspondences in contrastive image translation (Noji et al., 2020, Kletz et al., 2024, Ahrabian et al., 2020, Arnaout et al., 2022, Gajic et al., 2019, Fan et al., 2021, Lin et al., 2022). Viewed comparatively, the shared design principle is to preserve much of the surrounding structure while selectively violating, probing, or ablating the relation of interest.

1. Conceptual range

The literature uses structurally targeted negatives in several technically distinct ways. In neural language modeling, they appear as minimally altered tokens or sentences that violate a targeted syntactic dependency. In probing work on negation, they appear as tests of whether contextual representations encode the presence of negation and the polarity requirements of a masked item inside or outside scope. In knowledge graph embedding, they appear as corrupted triples drawn from a node’s $k$-hop neighborhood rather than uniformly from the entire entity set. In commonsense knowledge bases, they appear as informative negative statements derived from properties of comparable concepts under a local closed-world assumption. In metric learning and multimodal retrieval, they appear as hard negatives selected from a local embedding neighborhood or generated by constrained edits that preserve most of the caption structure. In unpaired image translation, they appear as a pruned subset of patch negatives selected by feature similarity. A distinct usage concerns negative pre-activations: here “negative” does not denote a sample, but a sign-specific computational regime in which syntax is differentiated (Kletz et al., 2024, Noji et al., 2020, Ahrabian et al., 2020, Arnaout et al., 2022, Gajic et al., 2019, Fan et al., 2021, Lin et al., 2022, Kong et al., 29 Sep 2025).

Domain	Structure being targeted	Mechanism
PLM negation probing	Negation scope; NPI licensing scope	Compare probes inside vs. outside scope
Neural LM training	Agreement; reflexive constraints	Minimal negative tokens or sentences with margin loss
Knowledge graph embedding	$k$-hop neighborhood	Corrupt head or tail within neighborhood
Commonsense KB construction	Comparable sibling concepts	Materialize absent sibling properties as negatives
Siamese retrieval	Local embedding neighborhood	Sample negatives from hash bucket
Image-text retrieval	Scene-graph-aligned caption structure	Masking, refilling, and hard-negative mining
Unpaired image translation	Hard patch correspondences	Prune and rank top-$K$ negatives
LLM internals	Negative pre-activation regime	Sign-specific intervention on Wasserstein neurons

2. Formalization in LLMs

A canonical scope-sensitive formulation is given for negation and NPI licensing. Let $T=(t_1,\dots,t_n)$ be a tokenized sentence, and let $c$ be the index of a negation cue such as “not” or “n’t”. The negation-scope set is defined as

$$S_{\mathrm{neg}}=\{\, i : t_i \text{ is within the subtree of the negated verb, excluding the cue itself} \,\},$$

and the licensing-scope set is defined as a subset

$S_{\mathrm{lic}} \subseteq S_{\mathrm{neg}}$

containing those positions where an NPI may appear under standard licensing patterns, such as a direct object or adjunct of the negated verb. An NPI at position $j$ is licensed iff $j\in S_{\mathrm{lic}}$. In a pretrained LLM, each token $t_i$ yields a contextual representation $k$0 at layer $k$1. Probes can then be defined by logistic-regression-style decision functions such as

$k$2

for the presence of negation, and

$k$3

for the polarity of a masked polarity item (Kletz et al., 2024).

A second formalization targets explicit syntactic errors during training. For a corpus $k$4 of well-formed sentences, target positions are identified at tokens that anchor a phenomenon such as present-tense agreement or reflexive anaphora. Negative tokens are defined by a set $k$5 of ungrammatical alternatives, and negative sentences by

$k$6

The baseline language-model objective

$k$7

is augmented with a margin-based auxiliary loss. The token-level margin variant is

$k$8

with

$k$9

This objective penalizes the model whenever the log-probability gap between the correct token and its negative counterpart falls below the fixed margin $K$0 (Noji et al., 2020).

These two formalisms target different operations. The probing framework asks whether a frozen representation already encodes a scope-sensitive distinction. The margin-loss framework injects a direct learning signal tied to a precisely delimited syntactic dependency. This suggests two complementary roles for structure-targeted negatives in language research: diagnosis and training (Kletz et al., 2024, Noji et al., 2020).

3. Scope-sensitive probing and what it reveals

In probing structural constraints of negation, sentences are extracted from COCA with exactly one uncontracted or contracted “not”, or zero “not” for control, and one non-cue token per sentence is sampled as probe input. For NPI licensing, sentences are matched to syntactic patterns in which “not” modifies a verb and licenses an NPI such as any*, anybody, anyone, anything, anytime, or anywhere. Four structural zones are identified: PRE, PRE-IN, IN, and POST. Frozen BERT-base, BERT-large, RoBERTa-base, and RoBERTa-large are probed by a 2-layer MLP with hidden size 450 and learning rate $K$1; accuracy is the primary metric and micro $K$2 shows similar trends. A distance control buckets probe tokens by relative position $K$3 around the negation cue so that any scope effect can be tested independently of simple proximity to “not” (Kletz et al., 2024).

For negation encoding, contextual representations of tokens inside the negation scope allow better prediction of the presence of “not” than representations of tokens outside the scope. Averaging over $K$4 and $K$5 yields an accuracy gap of approximately $K$6–$K$7 points across the four PLMs; for RoBERTa-large the reported value is $K$8. Fisher–Pitman permutation tests with 5,000 shuffles give $K$9 for nearly every position. For NPI licensing, overall polarity-classifier accuracy on random test data reaches roughly $T=(t_1,\dots,t_n)$0–$T=(t_1,\dots,t_n)$1 against a $T=(t_1,\dots,t_n)$2 chance level, and the average in-vs.-out scope gap ranges from about $T=(t_1,\dots,t_n)$3 points for RoBERTa-base to about $T=(t_1,\dots,t_n)$4 points for RoBERTa-large, again with $T=(t_1,\dots,t_n)$5. The largest gains occur for BERT-base and RoBERTa-large (Kletz et al., 2024).

The same study also provides an explicit control against an over-interpretation of “scope”. Replacing “not” with four ordinary words—often, big, house, and wrote—and labeling probe tokens as in-clause or out-clause yields accuracy gaps of $T=(t_1,\dots,t_n)$6–$T=(t_1,\dots,t_n)$7 points, again with $T=(t_1,\dots,t_n)$8, while distance to the target remains the strongest predictor, with accuracy dropping by about $T=(t_1,\dots,t_n)$9–$c$0 points from $c$1 to $c$2. The paper concludes that PLMs do encode structural constraints on negation and its licensing of NPIs, but that these encodings largely align with a more general sensitivity to syntactic clause boundaries. A common misconception is therefore that better in-scope probing performance directly establishes a specialized negation module; the control experiments show that same-clause structure is a serious alternative explanation (Kletz et al., 2024).

A related diagnostic use of structure-targeted negatives appears in explicit negative-example training for LSTMs. A three-layer LSTM with 1,150 hidden units per layer and 400-dimensional tied embeddings improves markedly when trained with token-level margin negatives for agreement and reflexives. On the Marvin and Linzen targeted evaluation set, simple agreement rises from $c$3 to $c$4, long VP coordination from $c$5 to $c$6, across a subject-relative clause from $c$7 to $c$8, and simple reflexives from $c$9 to $$S_{\mathrm{neg}}=\{\, i : t_i \text{ is within the subtree of the negated verb, excluding the cue itself} \,\},$$0, while Wikipedia-test perplexity moves from $$S_{\mathrm{neg}}=\{\, i : t_i \text{ is within the subtree of the negated verb, excluding the cue itself} \,\},$$1 to $$S_{\mathrm{neg}}=\{\, i : t_i \text{ is within the subtree of the negated verb, excluding the cue itself} \,\},$$2. The persistent outlier is agreement across an object-relative clause, which rises from $$S_{\mathrm{neg}}=\{\, i : t_i \text{ is within the subtree of the negated verb, excluding the cue itself} \,\},$$3 to $$S_{\mathrm{neg}}=\{\, i : t_i \text{ is within the subtree of the negated verb, excluding the cue itself} \,\},$$4 but remains below subject-relative performance (Noji et al., 2020).

4. Hard negatives by preserving local structure

In Siamese retrieval, Bag of Negatives (BoN) targets local structure in embedding space rather than global class labels. A linear auto-encoder projects the embedding $$S_{\mathrm{neg}}=\{\, i : t_i \text{ is within the subtree of the negated verb, excluding the cue itself} \,\},$$5 to a lower-dimensional code $$S_{\mathrm{neg}}=\{\, i : t_i \text{ is within the subtree of the negated verb, excluding the cue itself} \,\},$$6, reconstructs it by $$S_{\mathrm{neg}}=\{\, i : t_i \text{ is within the subtree of the negated verb, excluding the cue itself} \,\},$$7, and binarizes relative to a running threshold vector $$S_{\mathrm{neg}}=\{\, i : t_i \text{ is within the subtree of the negated verb, excluding the cue itself} \,\},$$8 to obtain a hash code

$$S_{\mathrm{neg}}=\{\, i : t_i \text{ is within the subtree of the negated verb, excluding the cue itself} \,\},$$9

A hash table stores image and class identifiers per bucket, and negatives for an anchor are sampled from the current bucket after excluding images with the same identity. The method is loss-independent and can be combined with batch-hard triplet training. On person re-identification, BoN + batch-hard reaches Market mAP $S_{\mathrm{lic}} \subseteq S_{\mathrm{neg}}$0 and Duke mAP $S_{\mathrm{lic}} \subseteq S_{\mathrm{neg}}$1 in 80k steps, compared with batch-hard triplet at $S_{\mathrm{lic}} \subseteq S_{\mathrm{neg}}$2 and $S_{\mathrm{lic}} \subseteq S_{\mathrm{neg}}$3 in 280k steps. On Stanford Online Products, BoN + batch-hard reports Rank-1 $S_{\mathrm{lic}} \subseteq S_{\mathrm{neg}}$4 and Rank-10 $S_{\mathrm{lic}} \subseteq S_{\mathrm{neg}}$5 (Gajic et al., 2019).

In image-text retrieval, TAGS-DC generates synthetic negative sentences by editing only a small, structurally important part of the caption. A shared multimodal Transformer backbone $S_{\mathrm{lic}} \subseteq S_{\mathrm{neg}}$6 supports image-text matching, masked-language modeling, word discrimination, and word correction. The caption is parsed with SPICE, roughly $S_{\mathrm{lic}} \subseteq S_{\mathrm{neg}}$7 of tokens aligned to scene-graph nodes are masked, replacements are sampled from the MLM head with temperature $S_{\mathrm{lic}} \subseteq S_{\mathrm{neg}}$8, false negatives are filtered if all replacement tokens occur verbatim in other human captions of the same image, and the most confusing generated captions are retained by scoring with the current matcher. The total loss combines image-retrieval triplet loss, MLM, synthetic-negative triplet loss, word discrimination, and word correction. With a UNITER-Large backbone, TAGS-DC reports on MS-COCO an $S_{\mathrm{lic}} \subseteq S_{\mathrm{neg}}$9 of $j$0 and an $j$1-Sum of $j$2, which are stated as $j$3 and $j$4 over UNITER; on Flickr30K it reports $j$5 of $j$6 and $j$7-Sum of $j$8, stated as $j$9 and $j\in S_{\mathrm{lic}}$0 (Fan et al., 2021).

In unpaired image-to-image translation, PUT studies whether all contrastive negatives are necessary. Standard PatchNCE treats all out-of-location patches as negatives; PUT instead computes a similarity matrix between translated and source feature patches, masks the self-correspondence, ranks the remaining negatives by dot-product similarity, and keeps only the top $j\in S_{\mathrm{lic}}$1 most informative negatives per anchor. The resulting RankNCE loss is applied across selected layers and combined with the adversarial objective. On Cityscapes, CUT reports semantic mAP $j\in S_{\mathrm{lic}}$2, pixel accuracy $j\in S_{\mathrm{lic}}$3, class accuracy $j\in S_{\mathrm{lic}}$4, and CityFID $j\in S_{\mathrm{lic}}$5, whereas PUT-3 reports $j\in S_{\mathrm{lic}}$6, $j\in S_{\mathrm{lic}}$7, $j\in S_{\mathrm{lic}}$8, and $j\in S_{\mathrm{lic}}$9, and PUT-5 reports $t_i$0, $t_i$1, $t_i$2, and $t_i$3. On Horse$t_i$4Zebra, FID drops from $t_i$5 for CUT to $t_i$6 for PUT-3, with inference speed essentially unchanged at $t_i$7 versus $t_i$8 seconds (Lin et al., 2022).

Across these cases, the negative is made difficult by preserving local organization: a nearby bucket in embedding space, a minimally edited caption that retains scene-graph coherence, or a high-similarity patch that is structurally incongruent. This suggests that the effectiveness of structure-targeted negatives often depends less on maximizing raw quantity than on constraining the negative pool to a structurally plausible neighborhood (Gajic et al., 2019, Fan et al., 2021, Lin et al., 2022).

5. Graph-structured and knowledge-based negative materialization

In knowledge graph embedding, Structure Aware Negative Sampling (SANS) replaces uniform corruption with corruption restricted to a node’s $t_i$9-hop neighborhood. For a knowledge graph $k$00 and a positive triple $k$01, the $k$02-hop reachability set is

$k$03

Given a tail corruption, uniform SANS uses

$k$04

The neighborhood can be computed explicitly from matrix powers or approximated by random walks in RW-SANS. The extra hyperparameters are $k$05, and for RW-SANS also $k$06. On FB15K-237 with TransE, Uniform reports MRR $k$07 and H@10 $k$08, KBGAN $k$09 and $k$10, NSCaching $k$11 and $k$12, Uniform SANS $k$13 and $k$14, and RW-SANS $k$15 and $k$16. The paper states that SANS variants often match or exceed adversarial baselines without extra trainable parameters (Ahrabian et al., 2020).

In commonsense knowledge acquisition, UnCommonSense addresses a different problem: knowledge bases store positive assertions under an open-world assumption, so absence does not imply falsity. The framework first identifies comparable concepts using dense concept embeddings and taxonomic filtering over WebIsALOD. With sibling set $k$17 for a target concept $k$18, positive properties of the target are $k$19 and sibling properties are $k$20. Under the local closed-world assumption on this induced fragment,

$k$21

so the raw negative candidates are $k$22. Candidates are then pruned by KB-based semantic deduplication using SBERT with threshold $k$23, LM-based plausibility filtering with BERT and top-$k$24 predictions, and a generic-phrase filter based on concept frequency. Surviving negatives are ranked by strict or relaxed sibling frequency, and provenance phrases such as “unlike other …” can be generated from the supporting sibling set (Arnaout et al., 2022).

The reported evaluations distinguish informative negatives from indiscriminate closure. On 200 concepts with top-2 negatives and three annotations each, the strict UnCommonSense variant reports a false-negative rate of $k$25, against $k$26–$k$27 for other methods, and an informativeness score of $k$28, versus $k$29 for the next best baseline. On ConceptNet-neg, Strict@10 is $k$30 versus $k$31–$k$32 for baselines, and Relaxed@10 is $k$33 versus $k$34–$k$35. In KB completion, training with UnCommonSense negatives yields $k$36 accuracy, compared with $k$37 for NegatER, $k$38 for COMET, and $k$39 for CWA, with $k$40. The released resource contains approximately $k$41 million negations over about $k$42k concepts (Arnaout et al., 2022).

A recurrent misconception in this area is that any absent statement in a knowledge base can be treated as a negative. The UnCommonSense results directly argue against that view: the open-world setting requires a restricted structural context, here a sibling set under a local closed-world assumption, before absent properties become informative negative knowledge (Arnaout et al., 2022).

6. Negative regions, internal mechanisms, and recurring limitations

A distinct but related line of work locates structure-targeted computation in the negative region of neuron pre-activations. In Transformer MLP blocks, each neuron computes $k$43, and Wasserstein neurons are those whose normalized output distribution has large 1-Wasserstein distance from a standard Gaussian. A mapping-difficulty score measures how far apart the neuron sends locally similar inputs. For smooth activations such as GELU or SiLU, pre-activations are decomposed into $k$44 and $k$45. The reported empirical hallmark is that non-Gaussianity concentrates in the negative tail of $k$46, and among the top-$k$47 pairs ranked by the ratio $k$48, about $k$49–$k$50 are negative–negative pairs in early layers. These pairs disproportionately involve syntactic tokens such as determiners and prepositions (Kong et al., 29 Sep 2025).

Causality is tested by zeroing only the negative pre-activations of the top $k$51 Wasserstein neurons:

$k$52

Clamping only $k$53–$k$54 of Wasserstein neurons’ negatives in Llama 3.1 8B and Mistral 7B roughly doubles perplexity, whereas random ablation has negligible effect. On BLiMP and TSE, the $k$55 Wasserstein ablation drops accuracy by $k$56–$k$57 absolute, versus less than $k$58 for random or perplexity-matched controls. Added surprisal on WikiText2 concentrates on function-word classes, and early-layer ablations have the largest isolated effects; cumulative ablations are roughly additive, especially on long-distance dependencies and negative-polarity items. Over Pythia checkpoints, Wasserstein distance rises sharply within the first $k$59K steps and then plateaus, and mean WD correlates with TSE and BLiMP performance at $k$60 (Kong et al., 29 Sep 2025).

Taken together, the studies identify recurring limitations as well as recurring successes. Negation probing can be confounded by same-clause sensitivity rather than truly negation-specific scope encoding (Kletz et al., 2024). Sequential LSTMs remain notably weaker on object-relative clauses even after direct negative-example supervision and frequency augmentation (Noji et al., 2020). PUT still struggles when very small or rare semantic instances appear in the source image (Lin et al., 2022). Several methods expose explicit hyperparameters that control the structural neighborhood—$k$61 and $k$62 in SANS, $k$63 in BoN, $k$64 in PUT, and $k$65 in UnCommonSense—and the papers treat these as practical tuning knobs rather than universally fixed constants (Ahrabian et al., 2020, Gajic et al., 2019, Lin et al., 2022, Arnaout et al., 2022).

The cumulative implication is not that “negative” examples are uniformly beneficial, but that they are most informative when they are structurally constrained, diagnostically precise, and carefully filtered against triviality or false negativity. In language, this precision exposes both genuine scope effects and clause-boundary confounds. In retrieval and translation, it raises hardness without destroying plausibility. In graphs and knowledge bases, it turns local structure into a principled corruption distribution or a principled source of absent-but-informative statements. And in model internals, negative pre-activation space itself emerges as a structure-targeted substrate for syntactic differentiation (Kletz et al., 2024, Noji et al., 2020, Fan et al., 2021, Lin et al., 2022, Ahrabian et al., 2020, Arnaout et al., 2022, Kong et al., 29 Sep 2025).