---
title: Structure-Targeted Negatives
url: https://www.emergentmind.com/topics/structure-targeted-negatives
type: topic
---

# Structure-Targeted Negatives

Structure-targeted negatives are negative examples, negative statements, or negative computational signals that are anchored to a specific structural relation rather than obtained by unrestricted random corruption. In the cited literature, the targeted structure may be a syntactic dependency such as subject–verb agreement or Negative Polarity Item (NPI) licensing, a graph neighborhood in a knowledge graph, a sibling set in a commonsense knowledge base, a local neighborhood in an embedding space, or a set of hard patch correspondences in contrastive image translation [2004.02451; 2408.03070; 2009.11355; 2208.09292; 1908.02391; 2111.03349; 2204.11018]. Viewed comparatively, the shared design principle is to preserve much of the surrounding structure while selectively violating, probing, or ablating the relation of interest.

## 1. Conceptual range

The literature uses structurally targeted negatives in several technically distinct ways. In neural language modeling, they appear as minimally altered tokens or sentences that violate a targeted syntactic dependency. In probing work on negation, they appear as tests of whether contextual representations encode the presence of negation and the polarity requirements of a masked item inside or outside scope. In knowledge graph embedding, they appear as corrupted triples drawn from a node’s $k$-hop neighborhood rather than uniformly from the entire entity set. In commonsense knowledge bases, they appear as informative negative statements derived from properties of comparable concepts under a local closed-world assumption. In metric learning and multimodal retrieval, they appear as hard negatives selected from a local embedding neighborhood or generated by constrained edits that preserve most of the caption structure. In unpaired image translation, they appear as a pruned subset of patch negatives selected by feature similarity. A distinct usage concerns negative pre-activations: here “negative” does not denote a sample, but a sign-specific computational regime in which syntax is differentiated [2408.03070; 2004.02451; 2009.11355; 2208.09292; 1908.02391; 2111.03349; 2204.11018; 2509.24198].

| Domain | Structure being targeted | Mechanism |
|---|---|---|
| PLM negation probing | Negation scope; NPI licensing scope | Compare probes inside vs. outside scope |
| Neural LM training | Agreement; reflexive constraints | Minimal negative tokens or sentences with margin loss |
| Knowledge graph embedding | $k$-hop neighborhood | Corrupt head or tail within neighborhood |
| Commonsense KB construction | Comparable sibling concepts | Materialize absent sibling properties as negatives |
| Siamese retrieval | Local embedding neighborhood | Sample negatives from hash bucket |
| Image-text retrieval | Scene-graph-aligned caption structure | Masking, refilling, and hard-negative mining |
| Unpaired image translation | Hard patch correspondences | Prune and rank top-$K$ negatives |
| LLM internals | Negative pre-activation regime | Sign-specific intervention on Wasserstein neurons |

## 2. Formalization in language models

A canonical scope-sensitive formulation is given for negation and NPI licensing. Let $T=(t_1,\dots,t_n)$ be a tokenized sentence, and let $c$ be the index of a negation cue such as “not” or “n’t”. The negation-scope set is defined as
$$
S_{\mathrm{neg}}=\{\, i : t_i \text{ is within the subtree of the negated verb, excluding the cue itself} \,\},
$$
and the licensing-scope set is defined as a subset
$$
S_{\mathrm{lic}} \subseteq S_{\mathrm{neg}}
$$
containing those positions where an NPI may appear under standard licensing patterns, such as a direct object or adjunct of the negated verb. An NPI at position $j$ is licensed iff $j\in S_{\mathrm{lic}}$. In a pretrained language model, each token $t_i$ yields a contextual representation $h_i\in\mathbb{R}^d$ at layer $\ell$. Probes can then be defined by logistic-regression-style decision functions such as
$$
P(\mathrm{neg}=1\mid h_i)=\sigma(w_{\mathrm{neg}}^\top h_i+b_{\mathrm{neg}})
$$
for the presence of negation, and
$$
P(\mathrm{pol}=\mathrm{NEG}\mid h_i)=\sigma(w_{\mathrm{pol}}^\top h_i+b_{\mathrm{pol}})
$$
for the polarity of a masked polarity item [2408.03070].

A second formalization targets explicit syntactic errors during training. For a corpus $D$ of well-formed sentences, target positions are identified at tokens that anchor a phenomenon such as present-tense agreement or reflexive anaphora. Negative tokens are defined by a set $\mathrm{neg}_t(x_i)=\{x_i^*\}$ of ungrammatical alternatives, and negative sentences by
$$
\mathrm{neg}_s(x)=\bigcup_{i\in \mathrm{Targets}(x)} \{x_1\ldots x_i^*\ldots x_n\}.
$$
The baseline language-model objective
$$
L_{\mathrm{lm}}=-\sum_{x\in D}\sum_{i=1}^{|x|}\log p(x_i\mid x_{1:i-1})
$$
is augmented with a margin-based auxiliary loss. The token-level margin variant is
$$
L=L_{\mathrm{lm}}+\beta L_{\mathrm{add}},
$$
with
$$
L_{\mathrm{add}}=\sum_{x\in D}\sum_{i\in \mathrm{Targets}(x)}\sum_{x_i^*\in \mathrm{neg}_t(x_i)}
\max\!\bigl(0,\delta-[\log p(x_i\mid x_{1:i-1})-\log p(x_i^*\mid x_{1:i-1})]\bigr).
$$
This objective penalizes the model whenever the log-probability gap between the correct token and its negative counterpart falls below the fixed margin $\delta$ [2004.02451].

These two formalisms target different operations. The probing framework asks whether a frozen representation already encodes a scope-sensitive distinction. The margin-loss framework injects a direct learning signal tied to a precisely delimited syntactic dependency. This suggests two complementary roles for structure-targeted negatives in language research: diagnosis and training [2408.03070; 2004.02451].

## 3. Scope-sensitive probing and what it reveals

In probing structural constraints of negation, sentences are extracted from COCA with exactly one uncontracted or contracted “not”, or zero “not” for control, and one non-cue token per sentence is sampled as probe input. For NPI licensing, sentences are matched to syntactic patterns in which “not” modifies a verb and licenses an NPI such as any*, anybody, anyone, anything, anytime, or anywhere. Four structural zones are identified: PRE, PRE-IN, IN, and POST. Frozen BERT-base, BERT-large, RoBERTa-base, and RoBERTa-large are probed by a 2-layer MLP with hidden size 450 and learning rate $10^{-3}$; accuracy is the primary metric and micro $F_1$ shows similar trends. A distance control buckets probe tokens by relative position $k\in[-8,+8]$ around the negation cue so that any scope effect can be tested independently of simple proximity to “not” [2408.03070].

For negation encoding, contextual representations of tokens inside the negation scope allow better prediction of the presence of “not” than representations of tokens outside the scope. Averaging over $k\in[-8,\dots,-1]$ and $k\in[+3,\dots,+8]$ yields an accuracy gap of approximately $2.6$–$3.5$ points across the four PLMs; for RoBERTa-large the reported value is $2.6\pm1.3\%$. Fisher–Pitman permutation tests with 5,000 shuffles give $p<0.001$ for nearly every position. For NPI licensing, overall polarity-classifier accuracy on random test data reaches roughly $56$–$69\%$ against a $50\%$ chance level, and the average in-vs.-out scope gap ranges from about $1.4$ points for RoBERTa-base to about $8.7$ points for RoBERTa-large, again with $p<0.001$. The largest gains occur for BERT-base and RoBERTa-large [2408.03070].

The same study also provides an explicit control against an over-interpretation of “scope”. Replacing “not” with four ordinary words—often, big, house, and wrote—and labeling probe tokens as in-clause or out-clause yields accuracy gaps of $3.3$–$5.5$ points, again with $p<0.001$, while distance to the target remains the strongest predictor, with accuracy dropping by about $10$–$20$ points from $k=0$ to $k=\pm8$. The paper concludes that PLMs do encode structural constraints on negation and its licensing of NPIs, but that these encodings largely align with a more general sensitivity to syntactic clause boundaries. A common misconception is therefore that better in-scope probing performance directly establishes a specialized negation module; the control experiments show that same-clause structure is a serious alternative explanation [2408.03070].

A related diagnostic use of structure-targeted negatives appears in explicit negative-example training for LSTMs. A three-layer LSTM with 1,150 hidden units per layer and 400-dimensional tied embeddings improves markedly when trained with token-level margin negatives for agreement and reflexives. On the Marvin and Linzen targeted evaluation set, simple agreement rises from $98.1\%$ to $100.0\%$, long VP coordination from $82.2\%$ to $99.0\%$, across a subject-relative clause from $91.5\%$ to $99.8\%$, and simple reflexives from $94.1\%$ to $99.9\%$, while Wikipedia-test perplexity moves from $49.5$ to $50.4$. The persistent outlier is agreement across an object-relative clause, which rises from $84.5\%$ to $93.7\%$ but remains below subject-relative performance [2004.02451].

## 4. Hard negatives by preserving local structure

In Siamese retrieval, Bag of Negatives (BoN) targets local structure in embedding space rather than global class labels. A linear auto-encoder projects the embedding $f(x)\in\mathbb{R}^e$ to a lower-dimensional code $h(x)=W_1f(x)+b_1$, reconstructs it by $\hat y(x)=W_2h(x)+b_2$, and binarizes relative to a running threshold vector $\mu$ to obtain a hash code
$$
c(x)=\mathrm{sign}(h(x)-\mu)\in\{0,1\}^s.
$$
A hash table stores image and class identifiers per bucket, and negatives for an anchor are sampled from the current bucket after excluding images with the same identity. The method is loss-independent and can be combined with batch-hard triplet training. On person re-identification, BoN + batch-hard reaches Market mAP $69.5$ and Duke mAP $62.1$ in 80k steps, compared with batch-hard triplet at $60.8$ and $53.7$ in 280k steps. On Stanford Online Products, BoN + batch-hard reports Rank-1 $75.8$ and Rank-10 $88.6$ [1908.02391].

In image-text retrieval, TAGS-DC generates synthetic negative sentences by editing only a small, structurally important part of the caption. A shared multimodal Transformer backbone $M_\theta$ supports image-text matching, masked-language modeling, word discrimination, and word correction. The caption is parsed with SPICE, roughly $15\%$ of tokens aligned to scene-graph nodes are masked, replacements are sampled from the MLM head with temperature $\tau$, false negatives are filtered if all replacement tokens occur verbatim in other human captions of the same image, and the most confusing generated captions are retained by scoring with the current matcher. The total loss combines image-retrieval triplet loss, MLM, synthetic-negative triplet loss, word discrimination, and word correction. With a UNITER-Large backbone, TAGS-DC reports on MS-COCO an $R@1$ of $67.8$ and an $R$-Sum of $472.9$, which are stated as $+2.1$ and $+4.0$ over UNITER; on Flickr30K it reports $R@1$ of $90.6$ and $R$-Sum of $557.4$, stated as $+3.3$ and $+6.4$ [2111.03349].

In unpaired image-to-image translation, PUT studies whether all contrastive negatives are necessary. Standard PatchNCE treats all out-of-location patches as negatives; PUT instead computes a similarity matrix between translated and source feature patches, masks the self-correspondence, ranks the remaining negatives by dot-product similarity, and keeps only the top $K$ most informative negatives per anchor. The resulting RankNCE loss is applied across selected layers and combined with the adversarial objective. On Cityscapes, CUT reports semantic mAP $22.29$, pixel accuracy $75.22$, class accuracy $29.60$, and CityFID $62.19$, whereas PUT-3 reports $24.31$, $78.28$, $31.70$, and $41.43$, and PUT-5 reports $24.86$, $79.17$, $30.10$, and $42.99$. On Horse$\rightarrow$Zebra, FID drops from $45.51$ for CUT to $33.82$ for PUT-3, with inference speed essentially unchanged at $0.245$ versus $0.245$ seconds [2204.11018].

Across these cases, the negative is made difficult by preserving local organization: a nearby bucket in embedding space, a minimally edited caption that retains scene-graph coherence, or a high-similarity patch that is structurally incongruent. This suggests that the effectiveness of structure-targeted negatives often depends less on maximizing raw quantity than on constraining the negative pool to a structurally plausible neighborhood [1908.02391; 2111.03349; 2204.11018].

## 5. Graph-structured and knowledge-based negative materialization

In knowledge graph embedding, Structure Aware Negative Sampling (SANS) replaces uniform corruption with corruption restricted to a node’s $k$-hop neighborhood. For a knowledge graph $G=(E,R,\mathcal{T}^+)$ and a positive triple $(h,r,t)$, the $k$-hop reachability set is
$$
N_k(s)=\{\, t\in E \mid \text{there exists a path of length }\le k \text{ from } s \text{ to } t \,\}.
$$
Given a tail corruption, uniform SANS uses
$$
p_{\mathrm{SANS}}(t' \mid h,r)=
\begin{cases}
1/|N_k(h)\setminus\{t\}| & \text{if } t'\in N_k(h)\setminus\{t\},\\
0 & \text{otherwise.}
\end{cases}
$$
The neighborhood can be computed explicitly from matrix powers or approximated by random walks in RW-SANS. The extra hyperparameters are $k$, and for RW-SANS also $\omega$. On FB15K-237 with TransE, Uniform reports MRR $0.2927$ and H@10 $48.03\%$, KBGAN $0.2926$ and $46.59\%$, NSCaching $0.2993$ and $47.64\%$, Uniform SANS $0.2962$ and $48.35\%$, and RW-SANS $0.2981$ and $48.50\%$. The paper states that SANS variants often match or exceed adversarial baselines without extra trainable parameters [2009.11355].

In commonsense knowledge acquisition, UnCommonSense addresses a different problem: knowledge bases store positive assertions under an open-world assumption, so absence does not imply falsity. The framework first identifies comparable concepts using dense concept embeddings and taxonomic filtering over WebIsALOD. With sibling set $S(s)$ for a target concept $s$, positive properties of the target are $A=\{f\mid (s,f)\in KB\}$ and sibling properties are $B=\{f\mid \exists x\in S(s):(x,f)\in KB\}$. Under the local closed-world assumption on this induced fragment,
$$
\forall f\in B\setminus A,\quad KB \models \neg(s,f),
$$
so the raw negative candidates are $N=B\setminus A$. Candidates are then pruned by KB-based semantic deduplication using SBERT with threshold $\lambda$, LM-based plausibility filtering with BERT and top-$\tau$ predictions, and a generic-phrase filter based on concept frequency. Surviving negatives are ranked by strict or relaxed sibling frequency, and provenance phrases such as “unlike other …” can be generated from the supporting sibling set [2208.09292].

The reported evaluations distinguish informative negatives from indiscriminate closure. On 200 concepts with top-2 negatives and three annotations each, the strict UnCommonSense variant reports a false-negative rate of $25\%$, against $26\%$–$63\%$ for other methods, and an informativeness score of $50\%$, versus $32\%$ for the next best baseline. On ConceptNet-neg, Strict@10 is $7.6\%$ versus $1.5\%$–$5.2\%$ for baselines, and Relaxed@10 is $26.1\%$ versus $4.4\%$–$9.6\%$. In KB completion, training with UnCommonSense negatives yields $79.56\%$ accuracy, compared with $78.61\%$ for NegatER, $79.06\%$ for COMET, and $75.89\%$ for CWA, with $p<0.01$. The released resource contains approximately $6.2$ million negations over about $8$k concepts [2208.09292].

A recurrent misconception in this area is that any absent statement in a knowledge base can be treated as a negative. The UnCommonSense results directly argue against that view: the open-world setting requires a restricted structural context, here a sibling set under a local closed-world assumption, before absent properties become informative negative knowledge [2208.09292].

## 6. Negative regions, internal mechanisms, and recurring limitations

A distinct but related line of work locates structure-targeted computation in the negative region of neuron pre-activations. In Transformer MLP blocks, each neuron computes $y_i=w_i^\top x$, and Wasserstein neurons are those whose normalized output distribution has large 1-Wasserstein distance from a standard Gaussian. A mapping-difficulty score measures how far apart the neuron sends locally similar inputs. For smooth activations such as GELU or SiLU, pre-activations are decomposed into $z_i^+=\max(z_i,0)$ and $z_i^-=\min(z_i,0)$. The reported empirical hallmark is that non-Gaussianity concentrates in the negative tail of $z$, and among the top-$K$ pairs ranked by the ratio $|z_i-z_j|/\|x_i-x_j\|$, about $30\%$–$60\%$ are negative–negative pairs in early layers. These pairs disproportionately involve syntactic tokens such as determiners and prepositions [2509.24198].

Causality is tested by zeroing only the negative pre-activations of the top $p\%$ Wasserstein neurons:
$$
\tilde z_i=
\begin{cases}
0 & \text{if } i\in S_W \text{ and } z_i<0,\\
z_i & \text{otherwise.}
\end{cases}
$$
Clamping only $1\%$–$2\%$ of Wasserstein neurons’ negatives in Llama 3.1 8B and Mistral 7B roughly doubles perplexity, whereas random ablation has negligible effect. On BLiMP and TSE, the $1\%$ Wasserstein ablation drops accuracy by $20$–$30\%$ absolute, versus less than $5\%$ for random or perplexity-matched controls. Added surprisal on WikiText2 concentrates on function-word classes, and early-layer ablations have the largest isolated effects; cumulative ablations are roughly additive, especially on long-distance dependencies and negative-polarity items. Over Pythia checkpoints, Wasserstein distance rises sharply within the first $\sim25$K steps and then plateaus, and mean WD correlates with TSE and BLiMP performance at $r>0.9$ [2509.24198].

Taken together, the studies identify recurring limitations as well as recurring successes. Negation probing can be confounded by same-clause sensitivity rather than truly negation-specific scope encoding [2408.03070]. Sequential LSTMs remain notably weaker on object-relative clauses even after direct negative-example supervision and frequency augmentation [2004.02451]. PUT still struggles when very small or rare semantic instances appear in the source image [2204.11018]. Several methods expose explicit hyperparameters that control the structural neighborhood—$k$ and $\omega$ in SANS, $s$ in BoN, $K$ in PUT, and $\gamma$ in UnCommonSense—and the papers treat these as practical tuning knobs rather than universally fixed constants [2009.11355; 1908.02391; 2204.11018; 2208.09292].

The cumulative implication is not that “negative” examples are uniformly beneficial, but that they are most informative when they are structurally constrained, diagnostically precise, and carefully filtered against triviality or false negativity. In language, this precision exposes both genuine scope effects and clause-boundary confounds. In retrieval and translation, it raises hardness without destroying plausibility. In graphs and knowledge bases, it turns local structure into a principled corruption distribution or a principled source of absent-but-informative statements. And in model internals, negative pre-activation space itself emerges as a structure-targeted substrate for syntactic differentiation [2408.03070; 2004.02451; 2111.03349; 2204.11018; 2009.11355; 2208.09292; 2509.24198].

Source: https://www.emergentmind.com/topics/structure-targeted-negatives