Cross-Batch Negative Sampling

• CBNS is a negative-sampling strategy that extends in-batch negatives using a FIFO memory bank of recently computed item embeddings.
• It leverages the stability of encoder outputs after a warm-up period to increase negative diversity without enlarging the batch size.
• Empirical evaluations on datasets like Amazon-Books show CBNS improves Recall and NDCG metrics while reducing computational costs.

Searching arXiv for the cited CBNS and related negative sampling papers. Search query: "Cross-Batch Negative Sampling for Training Two-Tower Recommenders arXiv (Wang et al., 2021)" Cross-Batch Negative Sampling (CBNS) is a negative-sampling strategy for training two-tower recommenders in which the negative set is expanded beyond the current mini-batch by reusing encoded item embeddings from recent mini-batches. In its canonical formulation, CBNS augments standard in-batch negatives with a first-in-first-out memory bank of cached item embeddings, motivated by the observation that, after a warm-up period, neural encoders can output relatively stable features for the same input. In the broader negative-sampling literature, CBNS is not uniformly treated as a distinct top-level category; a closer taxonomic reading places it as an extension of in-batch negative sampling and memory-based selection, often implemented through a queue, memory bank, or batch-reuse mechanism (Wang et al., 2021, Yang et al., 2024).

1. Problem formulation and motivation

CBNS was introduced for large-scale, content-aware recommendation with a two-tower architecture consisting of a user encoder $f_u$ and an item encoder $g_v$. Given user features $U$ and item features $I$, the model maps them into a shared embedding space,

$$u = f_u(U) \in \mathbb{R}^d, \qquad v = g_v(I) \in \mathbb{R}^d,$$

and scores a user–item pair by

$s(U, I) = f_u(U)^\top g_v(I) = u^\top v.$

Training is cast as sampled softmax or multi-class classification, where a positive pair $(U,I)$ is contrasted against a set of negatives $N$: $$p(I \mid U; \Theta) = \frac{e^{u^\top v}}{e^{u^\top v} + \sum_{I^- \in N} e^{u^\top v^-}},$$ with cross-entropy objective

$$L_{\text{CE}} = -\frac{1}{|B|}\sum_{i \in [|B|]} \log p(I_i \mid U_i; \Theta).$$

The immediate baseline is in-batch negative sampling, where items that appear as positives for other users in the same mini-batch are reused as negatives. This is computationally attractive because it avoids encoding extra candidate items. Its limitation is structural: the number of negatives is tied to batch size. Small batches produce too few negatives and often insufficient diversity; large batches increase memory usage and encoder time because the user and item contents are “heavy” and must still be encoded. CBNS addresses this effectiveness–efficiency trade-off by increasing the negative pool without requiring a correspondingly larger encoded batch (Wang et al., 2021).

2. Mechanism of CBNS

The defining mechanism of CBNS is a first-in-first-out memory bank that stores recently computed item embeddings together with their sampling probabilities. The memory is written as

$g_v$0

where each entry contains an item embedding $g_v$1 and its sampling probability $g_v$2 under the unigram distribution. The scoring function is bias-corrected as

$g_v$3

The workflow has four steps. First, there is a warm-up phase, in which training proceeds with naive in-batch negatives for about $g_v$4 iterations. Second, after warm-up, the current mini-batch is encoded as usual. Third, for each positive item in the batch, the negative set is formed by combining current in-batch negatives with cached negatives from the memory bank: $g_v$5 The resulting CBNS softmax is

$g_v$6

Fourth, at the end of the iteration, the current batch’s item embeddings and their $g_v$7 values are enqueued, and the oldest entries are dequeued to keep the memory size fixed.

A central property of this design is that cached embeddings are reused without re-encoding the underlying items. CBNS therefore does not change the model architecture or the form of the loss; it changes only the composition of the negative set. This is why the method is presented as simple and effective rather than as an architectural modification (Wang et al., 2021).

3. Stability-based justification

The theoretical basis of CBNS is the observation that, after an initial phase of training, encoder outputs for the same item become relatively stable. The paper quantifies this phenomenon through a feature-drift measure over iteration gap $g_v$8: $g_v$9 Empirically, on YoutubeDNN trained on Amazon-Books, the drift is large early in training but becomes much smaller after about $U$0 iterations. This motivates the warm-up requirement: early embeddings are too unstable to reuse safely, whereas recent post-warm-up embeddings can approximate current embeddings closely enough to function as negatives.

The formal argument connects approximation error in the cached item embedding to approximation error in the gradient. Under a Lipschitz assumption on the user encoder, if

$U$1

then the gradient deviation satisfies

$U$2

where $U$3 is a Lipschitz constant. In effect, if the cached embedding $U$4 is close to the current embedding $U$5, then using the cached vector induces only a bounded gradient error. This suggests that CBNS is justified precisely in regimes where embedding freshness is preserved over the memory horizon (Wang et al., 2021).

4. Position within the negative-sampling taxonomy

The 2024 review of negative sampling does not explicitly name “Cross-Batch Negative Sampling” as a distinct method. Instead, the closest placement is within the broader family of in-batch, mini-batch, and memory-based negative sampling. In its main taxonomy, current negative-sampling methods are divided into five categories: static NS, hard NS, GAN-based NS, Auxiliary-based NS, and In-batch NS. In its candidate-selection framework, the review further distinguishes global, local, mini-batch, hop, and memory-based selection (Yang et al., 2024).

Selection method	Candidate pool
Global Selection	all possible negatives in the dataset
Local Selection	a selected subset of negatives
Mini-batch Selection	other samples in the current mini-batch
Hop Selection	$U$6-hop graph neighbors
Memory-based Selection	a memory bank or cache from previous iterations or batches

Within this framework, CBNS aligns most naturally with mini-batch selection when the negative pool is enlarged across batches, and with memory-based selection when negatives are stored and reused from prior batches. The review states that “Mini-batch Selection uses other samples in the current mini-batch as the pool without the additional process of choosing,” whereas “Memory-based Selection maintains a memory bank or a cache as a pool to store the pool of negative sample candidates.” CBNS can therefore be understood as a batch-expansion strategy that grows the candidate pool beyond the current mini-batch.

The same review makes clear why this matters for contrastive objectives. For InfoNCE,

$U$7

the number of negatives depends directly on batch size $U$8. The review notes that in dense retrieval $U$9, and in contrastive learning $I$0. It also states that “the experimental results in RocketQA demonstrated that it is beneficial to increase the number of negatives by introducing cross-batch negatives.” At the same time, memory-based and queue-based methods such as MoCo enlarge the pool of negatives but introduce update overhead, memory or caching complexity, and the possibility of stale negatives (Yang et al., 2024).

5. Bias, negative quality, and related methods

Batch-based negative sampling can be analyzed as a special case of a more general sampling framework. In the large-output-space analysis of negative sampling, within-batch negatives are defined as

$I$1

and, in expectation, this is equivalent to sampling negatives from the training label distribution,

$I$2

The framework is parameterized by a sampling distribution $I$3 and a weighting scheme $I$4. Within-batch sampling with constant weighting,

$I$5

induces an implicit softmax loss

$I$6

which tends to help tail labels. Importance weighting,

$I$7

recovers the standard softmax cross-entropy, while relative weighting,

$I$8

emphasizes head labels. This analysis is directly relevant to CBNS because it shows that enlarging the effective negative pool does not by itself determine the optimized loss; weighting still governs the trade-off between dominant and rare labels (Rawat et al., 2021).

A recurrent misconception is that more negatives, or harder negatives, are necessarily better. Evidence from contrastive audio–text retrieval argues against that simplification. In a purely mini-batch regime, full-mini-batch negative sampling produced little improvement over random negative sampling, while cross-modality semi-hard negatives achieved the strongest retrieval performance: text-to-audio mAP $I$9, R@5 $$u = f_u(U) \in \mathbb{R}^d, \qquad v = g_v(I) \in \mathbb{R}^d,$$0, R@10 $$u = f_u(U) \in \mathbb{R}^d, \qquad v = g_v(I) \in \mathbb{R}^d,$$1, and audio-to-text mAP $$u = f_u(U) \in \mathbb{R}^d, \qquad v = g_v(I) \in \mathbb{R}^d,$$2, R@5 $$u = f_u(U) \in \mathbb{R}^d, \qquad v = g_v(I) \in \mathbb{R}^d,$$3, R@10 $$u = f_u(U) \in \mathbb{R}^d, \qquad v = g_v(I) \in \mathbb{R}^d,$$4. By contrast, cross-modality hard negatives were reported as disastrous, with feature collapse causing the audio encoder to output zero vectors as acoustic embeddings. This indicates that negative quality and stability can matter more than quantity alone, and that the hardest negatives may destabilize optimization (Xie et al., 2022).

Related recommender work on Collaborative Metric Learning reached a similar conclusion through a different mechanism. A two-stage sampler based on popularity-weighted candidate generation and similarity-aware informative selection was proposed to make small batches effective, but it did not use a memory bank and was therefore not a CBNS method. Its relevance lies in motivation: like CBNS, it sought to make small-batch training practical by improving the informativeness of sampled negatives rather than relying on uniformly random sampling (Tran et al., 2019).

6. Empirical performance, operating regime, and open questions

The principal empirical evaluation of CBNS was conducted on Amazon-Books, with 8,898,041 user–item interactions, 459,133 users, and 313,966 items, using an 8:1:1 train/validation/test split by user. Three representative two-tower recommenders were tested: Youtube DNN, GRU4REC, and MIND. CBNS was compared with Uniform sampling, In-batch negative sampling, and Mixed Negative Sampling (MNS), under Recall@20, Recall@50, NDCG@20, NDCG@50, Convergence Time, and Average Training Time per 1000 mini-batches. At Metrics@50, CBNS achieved Recall $$u = f_u(U) \in \mathbb{R}^d, \qquad v = g_v(I) \in \mathbb{R}^d,$$5, NDCG $$u = f_u(U) \in \mathbb{R}^d, \qquad v = g_v(I) \in \mathbb{R}^d,$$6 for Youtube DNN; Recall $$u = f_u(U) \in \mathbb{R}^d, \qquad v = g_v(I) \in \mathbb{R}^d,$$7, NDCG $$u = f_u(U) \in \mathbb{R}^d, \qquad v = g_v(I) \in \mathbb{R}^d,$$8 for GRU4REC; and Recall $$u = f_u(U) \in \mathbb{R}^d, \qquad v = g_v(I) \in \mathbb{R}^d,$$9, NDCG $s(U, I) = f_u(U)^\top g_v(I) = u^\top v.$0 for MIND. The reported NDCG@50 improvements over in-batch sampling were $s(U, I) = f_u(U)^\top g_v(I) = u^\top v.$1, $s(U, I) = f_u(U)^\top g_v(I) = u^\top v.$2, and $s(U, I) = f_u(U)^\top g_v(I) = u^\top v.$3, respectively (Wang et al., 2021).

These gains were accompanied by a specific systems profile. In-batch sampling remained fastest per batch, and CBNS was somewhat slower because it computed similarities with cached embeddings. However, CBNS was still far cheaper than approaches that require more encoding work or larger negative pools through full global sampling. In this sense, its principal systems-level advantage is that it decouples the number of negatives from the batch size. The experimental configuration used batch size $s(U, I) = f_u(U)^\top g_v(I) = u^\top v.$4, embedding dimension $s(U, I) = f_u(U)^\top g_v(I) = u^\top v.$5, and default memory size $s(U, I) = f_u(U)^\top g_v(I) = u^\top v.$6. Memory-size ablations showed that performance improved as memory grew from $s(U, I) = f_u(U)^\top g_v(I) = u^\top v.$7 to a moderate size, but then saturated or dropped: Youtube DNN performed best around $s(U, I) = f_u(U)^\top g_v(I) = u^\top v.$8–$s(U, I) = f_u(U)^\top g_v(I) = u^\top v.$9, GRU4REC around $(U,I)$0, and MIND around $(U,I)$1. If the memory bank becomes too large, embeddings become stale and may hurt performance (Wang et al., 2021).

The best-supported operating regime for CBNS is therefore narrow but important: a two-tower retriever with expensive encoders, GPU-constrained batch size, a need for more negatives than in-batch sampling can provide, and an encoder whose outputs become stable after warm-up. Beyond that regime, the broader review literature identifies unresolved issues that bear directly on CBNS: how to develop better In-batch NS to mine hard negatives and mitigate false negatives; how many negatives are needed; how to balance efficiency and negative quality; how to scale negative sampling without excessive memory or time cost; how to reduce sampling bias in In-batch NS; how to integrate hard negative mining into batch-based methods efficiently; how to design faster cache update mechanisms for memory-based methods; and how to dynamically add easy negatives into hard negative sampling. These questions indicate that CBNS is best understood not as a closed recipe, but as one point in a broader design space of batch-expanded negative sampling (Yang et al., 2024).