Adaptive Negative Sampling (ANS) Method

• Adaptive Negative Sampling (ANS) is a method that dynamically masks easy negatives in training to reduce harmful gradients and improve rare token representation.
• The approach employs a threshold mechanism to exclude non-informative negatives, thereby focusing updates on hard negatives and yielding lower perplexity and higher accuracy.
• Empirical results show that ANS can reduce perplexity by over 55% and significantly boost accuracy for rare tokens in low-resource language modeling settings.

Adaptive Negative Sampling (ANS) Method

Adaptive Negative Sampling (ANS) encompasses a range of training methodologies that dynamically select or mask negative (non-target) samples in order to optimize gradient flow, statistical efficiency, and downstream performance in learning settings where negatives greatly outnumber positives. By making the negative-sample selection dependent on the current model’s state—and, in some cases, on dynamic thresholds or adaptive scoring—ANS aims to focus computation and parameter updates on informative mis-rankings, particularly benefiting underrepresented or rare categories. ANS techniques are increasingly favored in language modeling, recommendation, information retrieval, and specialized domains such as low-resource language modeling, where static negative sampling schemes fail to provide sufficient discriminative gradient for rare classes (Turumtaev, 30 Jan 2026).

1. Motivation: Marginalization and Limitations of Standard Negative Sampling

The primary challenge addressed by ANS in training neural LLMs is the marginalization of rare tokens under standard cross-entropy loss. In traditional formulations, for a vocabulary $V$ of size $N$, the loss function updates every non-target (non-$x_t$) token embedding with a small negative gradient at each forward pass. While this negative “push” is negligible for frequent tokens (which receive ample positive updates from direct context), it is proportionally large and accumulates for rare tokens; this is especially problematic in low-resource language settings, where positive supervision is sparse. Empirical analysis reveals that for the rarest tokens, the total harmful (marginalization) gradient exceeds the useful (alignment) gradient, leading to representational collapse of these classes. Such imbalance degrades both the effective perplexity and accuracy on validation data for low-resource tokens. ANS methods seek to eliminate or sharply reduce the marginalization signal for tokens deemed irrelevant by the model in the current context (Turumtaev, 30 Jan 2026).

2. Methodological Formulation and Algorithmic Integration

ANS introduces a simple yet effective thresholding mechanism into the standard cross-entropy workflow. For each time step $t$, define:

• $V=\{v_1, ..., v_N\}$: the vocabulary,
• $g_\theta(\cdot)$: transformer body producing last hidden state $h_t$,
• $\text{logits}_{t,i} = \langle h_t, w_i \rangle$: model scores for token $i$.

Let $x_t$ be the correct next token, so the standard softmax is

$$P_\theta(v_i \mid \text{context}) = \frac{\exp(\text{logits}_{t,i})}{\sum_{j=1}^N \exp(\text{logits}_{t,j})}.$$

ANS introduces a fixed margin $\tau > 0$ and constructs a threshold

$\tau_t = \text{logits}_{t,x_t} - \tau.$

Before the softmax, all logits below $\tau_t$ are set to $-\infty$:

$$\text{logits}'_{t,i} = \begin{cases} \text{logits}_{t,i} & \text{if } \text{logits}_{t,i} \geq \tau_t \ -\infty & \text{otherwise} \end{cases}$$

yielding a restricted softmax where only “hard” negatives—tokens within $\tau$ of the current target—receive nonzero probability. The cross-entropy is computed

$$L^\text{ANS}_\theta(x_t) = -\log\,P^\text{ANS}_\theta(x_t \mid x_{<t}),$$

where zero-gradient updates are assigned to all masked (irrelevant) non-targets. Sample integration into the PyTorch training step is as follows (Turumtaev, 30 Jan 2026):

for each batch of sequences: h = TransformerBody(inputs) # hidden states logits = h @ W.T # shape [B, T, N] target_logits = gather(logits, dim=2, index=targets) threshold = target_logits.unsqueeze(-1) - tau logits = where(logits < threshold, -inf, logits) loss = CrossEntropyLoss(logits, targets) loss.backward() optimizer.step()

3. Comparison with Uniform and Static Sampling Approaches

Uniform Negative Sampling (UNS) randomly selects a fixed number of negative classes for each positive, independent of the model’s current confidence or uncertainty. Static “frequency-based” or “importance-weighted” approaches may skew the selection toward or away from popular classes, but remain fixed throughout training and are insensitive to context. In contrast, ANS adaptively “samples” by masking out negatives whose logits are more than $\tau$ below the target. As a result, only the most potentially confusing misclassifications in the model’s current view receive gradient updates. This adaptivity focuses optimization on resolving genuinely ambiguous “hard negatives,” ceasing update expenditure on negatives that the model already deems highly implausible (Turumtaev, 30 Jan 2026).

4. Empirical Findings: Impact on Rare Tokens and Quantitative Results

Experiments on a character-level language modeling task provide both qualitative and quantitative validations. The setup consists of:

• Data: A modified Shakespeare corpus with 65 high-resource and 65 low-resource (rare) tokens constructed via sentence remapping.
• Model: GPT-2–style decoder, 4 layers, 4 heads, $d_\text{model} = 128$, ~800K parameters.
• Metrics: Character-level perplexity (PPL), optimal PPL after temperature scaling ($\text{PPL}_\text{best}$ at $T_\text{best}$), Accuracy, Recall@5, Mean Reciprocal Rank (MRR), and embedding isotropy $I(W)$.

Key results for low-resource language:

Condition	PPL	$\text{PPL}_\text{best}$	Accuracy
Baseline	10.65	10.63 ($T$=0.95)	0.3147
ANS ($\tau$=0.6)	—	6.17 ($T$=0.32)	0.4868
ANS + SE ($\tau$=1)	—	6.90	0.5478

This indicates over 55% relative improvement in effective perplexity and a 54% gain in accuracy for rare characters. All ranking metrics (Accuracy, Recall@5, MRR) and isotropy $I(W)$ improved monotonically as $\tau$ decreased from 8 to 0.6. The separated-embedding variant, which zeroes out gradients for masked tokens, achieves even higher gains in rare-token accuracy, demonstrating the effect of eliminating marginalization for irrelevant negatives (Turumtaev, 30 Jan 2026).

5. Theoretical Significance and Broader Implications

By dynamically focusing the marginalization gradient on hard negatives, ANS both aligns token representation learning more closely with the actual uncertainty structure faced by the model and prevents excessive embedding displacement for rare, underrepresented types. This is especially relevant in multilingual or low-resource scenarios, where standard cross-entropy will otherwise disadvantage rare tokens by overwhelming them with negative gradients uncounterbalanced by positive learning signals. The methodology generalizes to other architectures and domains where rare class coverage and specificity are critical, and where the number of possible negatives is orders of magnitude greater than positives relative to each example.

6. Limitations and Prospects for Extension

Current findings are restricted to relatively small, synthetic character-level models; the scalability of ANS to large subword-vocabulary LLMs, diverse scripts, or truly multilingual corpora remains undemonstrated. The method requires user-specified margin $\tau$, which determines the trade-off between neglecting tail tokens and increasing raw perplexity due to long-tailed near-threshold negatives. No automation or learning of $\tau$ per class is included, though the authors note this as a potential avenue for future work. Training with dynamic temperature or nucleus-sampling–style criteria, incorporation into contrastive learning settings, and deployment to downstream tasks such as tagging or translation are proposed for further study (Turumtaev, 30 Jan 2026).

7. Summary and Field Position

ANS in language modeling, as formalized in (Turumtaev, 30 Jan 2026), provides a lightweight, implementation-agnostic modification to cross-entropy, requiring only masking of “easy” negative logits per context. This substantially improves the learning of rare or under-sampled token representations, mitigating the central problem of marginalization-driven collapse. Its plug-in simplicity and demonstrated quantitative gains over uniform or static negative sampling support its broad applicability for training models on data distributions with significant class imbalance or tail distributions.