Memory-Bank Contrastive Loss

• Memory-bank contrastive loss is a family of objectives that uses stored representations to enrich negative samples beyond current mini-batch comparisons.
• It employs mechanisms like FIFO queues and momentum updates to maintain stable, smooth representation evolution across iterations.
• Applications span self-supervised learning, deep metric learning, dense retrieval, and re-identification to improve training efficiency.

Memory-bank contrastive loss denotes a family of contrastive objectives in which the comparison set is expanded beyond the current mini-batch by a persistent store of previously computed representations. In current usage, this includes momentum queues in self-supervised learning, embedding memories in deep metric learning, dual banks in dense retrieval, and prototype or cluster memories in re-identification. The common purpose is to decouple the number of effective negatives from the instantaneous batch size, preserve cross-mini-batch structure, and stabilize optimization when purely in-batch contrast is inadequate (Xu et al., 2020, Zhang et al., 2021, Kim et al., 2024).

1. Formal structure

A canonical memory-bank contrastive objective follows the MoCo-style InfoNCE template. For an image $x$, a query encoder $f_q$ produces $q=f_q(c(x))$, a key encoder $f_k$ produces a positive key $k_+=f_k(c'(x))$, and negatives are drawn from a memory bank $\mathcal M$. The standard objective is

$$\mathcal{L}_{\mathrm{NCE}(f_q(c(x)),f_k(c'(x)), \mathcal{M})} = -\log \frac{\exp(q \cdot k_+ / T)} {\exp(q \cdot k_+ / T)+\sum_{k_- \in \mathcal{M}}\exp(q \cdot k_- / T)}.$$

In this formulation, the bank is not an auxiliary cache but part of the denominator itself: it supplies the negative pool against which the current query is discriminated (Xu et al., 2020).

The formal role of the bank depends on the task. In self-supervised instance discrimination it typically stores historical keys; in deep metric learning it stores recent embeddings and labels so that current anchors can form positives and negatives against $B \cup M$; in dense retrieval it can store both query and passage representations; and in prototype-based settings it may store cluster centroids rather than raw instances. What remains invariant is that the loss is defined against stored representations rather than only the current batch (Zhang et al., 2021, Kim et al., 2024, Shen et al., 2023).

2. Stored representations and update dynamics

The defining mechanism of a memory-bank contrastive loss is the persistence of representations across iterations. In MoCo-style systems, the bank is a FIFO queue: minibatch keys are enqueued and the oldest are dequeued. To keep queued representations consistent enough for contrastive use, the key encoder is updated by momentum rather than direct backpropagation,

$\theta_k \leftarrow m\theta_k + (1-m)\theta_q,$

so that the bank evolves more smoothly than the query encoder alone would permit. AMOC adopts this exact mechanism and stresses that such stabilization is especially important when adversarial examples already introduce additional instability (Xu et al., 2020).

Deep metric learning papers reframe the same issue as one of pair availability rather than only loss design. With an embedding memory, each current anchor can contrast not only with current-batch examples but with stored historical embeddings, effectively turning a small-batch pairwise loss into a much larger candidate set. The key observation reported in this line of work is that, in memory-based DML, it is critical to mine hard negatives and discard easy negatives which are less informative and redundant, whereas weighting on positive pairs is not helpful; this yields an efficient design rule in which memory expansion matters more than elaborate in-batch pair weighting (Zhang et al., 2021).

The same persistence mechanism becomes structurally more delicate in dual-encoder retrieval. ContAccum stores detached query representations in $M_{\mathbf q}$ and detached passage representations in $f_q$0, then computes InfoNCE over current local-batch representations concatenated with both banks. Its analysis shows that one-sided banking is not merely stale-negative reuse: it changes the gradient structure of the two towers asymmetrically. A plausible implication is that, in dual encoders, memory-bank design must be treated as an optimization-geometry issue rather than only a negative-sampling issue (Kim et al., 2024).

3. Multi-bank and distribution-matched designs

A major development is the move from a single bank to multiple banks that correspond to distinct representation distributions. AMOC extends MoCo to adversarial self-supervised pre-training by introducing two separate memory banks, $f_q$1 and $f_q$2, together with dual batch normalization. Its two core losses are

$f_q$3

and

$f_q$4

combined as

$f_q$5

The design choice is precise: the adversarial query is aligned to a clean positive key, while adversarial negatives are drawn from a separate adversarial bank rather than from the clean bank. AMOC reports $f_q$6, $f_q$7, and memory bank size $f_q$8, and further reports that on CIFAR-10 it outperforms ACL with batch size $f_q$9 and $q=f_q(c(x))$0 epochs using batch size $q=f_q(c(x))$1 and $q=f_q(c(x))$2 epochs, while batch size $q=f_q(c(x))$3 and $q=f_q(c(x))$4 epochs achieves performance comparable to ACL with batch size $q=f_q(c(x))$5 and $q=f_q(c(x))$6 epochs (Xu et al., 2020).

The same multi-bank idea appears in re-identification, but with semantic rather than perturbation-based separation. TCRL introduces three banks: a part memory bank $q=f_q(c(x))$7, a cluster memory bank $q=f_q(c(x))$8, and a global memory bank $q=f_q(c(x))$9. The cluster bank is initialized by

$f_k$0

and then updated by momentum. Its central claim is that directly contrasting part and global features is unstable, whereas a cluster bank can act as an intermediate proxy so that part-cluster and cluster-global relations are learned as adjacent-bank associations. This produces three distinct bank-related losses: Proxy Contrastive Loss, Hybrid Contrastive Loss, and Weighted Regularization Cluster Contrastive Loss (Shen et al., 2023).

A related two-scale design appears in unsupervised person re-identification. TCMM alternates full-dataset feature extraction, DBSCAN clustering, construction of an instance memory $f_k$1 and a prototype memory $f_k$2, and mini-batch training with

$f_k$3

This suggests a recurrent pattern in memory-bank contrastive loss design: fine-grained instance memory preserves local discrimination, while prototype or cluster memory supplies more stable global structure (Zhu et al., 15 Jan 2025).

4. Negative quality, false negatives, and weighting

Once the negative set is enlarged, the central problem shifts from quantity to quality. HeroCon addresses this directly by modifying the contrastive objective itself rather than the storage mechanism. In its weighted unsupervised loss, negatives are multiplied by a representation-dependent weight $f_k$4, so likely false negatives receive smaller contribution. The paper’s theoretical analysis states that vanilla contrastive learning becomes sub-optimal if even one false negative is present in the batch, whereas the weighted loss can automatically adjust the weight based on learned similarity. In multi-label settings the same logic is extended to supervised contrastive learning by weighting positives with $f_k$5 and negatives with $f_k$6 derived from label-vector distance (Zheng et al., 2021).

A MoCo-style modification of the same concern appears in "Momentum Contrastive Learning with Enhanced Negative Sampling and Hard Negative Filtering" (Hoang et al., 20 Jan 2025). That work adds a dual-view loss and replaces one denominator by a filtered subset $f_k$7 chosen by cosine-similarity ranking. The paper describes this both as enhanced negative sampling and as a way to avoid noisy false negatives. The operational emphasis is therefore not simply on maximizing hardness, but on controlling the hardness-versus-noise tradeoff within the queue (Hoang et al., 20 Jan 2025).

The embedding-memory perspective in deep metric learning reaches a similar conclusion by a different route. Its reported result is that hard negatives are crucial, easy negatives are redundant, and sophisticated positive weighting brings little benefit once an embedding memory is available. This suggests that memory-bank contrastive loss has two separable design axes: bank construction determines which candidates are available, while weighting or filtering determines which of those candidates actually shape the gradient (Zhang et al., 2021).

5. Task-specific generalizations

Dense retrieval provides one of the clearest task-specific reinterpretations. ContAccum uses two FIFO memory banks of equal size, one for queries and one for passages, and computes InfoNCE over current local-batch representations plus detached cached representations. The method reports that removing the query bank degrades NQ Top@20 from $f_k$8 to $f_k$9, and its gradient-norm analysis shows that with only passage banking the ratio $k_+=f_k(c'(x))$0 can rise up to $k_+=f_k(c'(x))$1, whereas the dual-bank design keeps the ratio near $k_+=f_k(c'(x))$2. In this setting, the memory bank is not only a source of more negatives; it is an explicit device for balancing the two towers of a dual encoder (Kim et al., 2024).

Generative modeling extends the concept further. Repulsor stores a large FIFO queue

$k_+=f_k(c'(x))$3

of projected latent representations and replaces positive-pair InfoNCE by a repulsion-only memory-bank loss,

$k_+=f_k(c'(x))$4

The total training objective is

$k_+=f_k(c'(x))$5

There are no positives in the auxiliary loss; the denoising objective supplies alignment, while the memory bank supplies only repulsion. On ImageNet-256, the paper reports that bank size $k_+=f_k(c'(x))$6 gives FID $k_+=f_k(c'(x))$7, improving over $k_+=f_k(c'(x))$8 at $k_+=f_k(c'(x))$9, whereas an even larger bank $\mathcal M$0 worsens to $\mathcal M$1, making bank size a direct quality-control parameter rather than a monotonic scaling factor (Zhang et al., 9 Dec 2025).

Re-identification illustrates yet another pattern: the bank may store proxies at several semantic levels. TCRL’s HCL uses all positive instances and all negative instances in the corresponding bank rather than a single positive pair, while WRCCL reweights pseudo-cluster supervision by intra-cluster similarity. This is a distinct usage from classical MoCo: the bank is not merely a negative dictionary, but a structured repository that mediates between part-level, instance-level, and cluster-level semantics (Shen et al., 2023).

6. Exact large-batch alternatives and boundary cases

Not every memory-efficient contrastive method is a memory-bank method. DisCo-CLIP keeps the exact CLIP objective over the current global batch and reformulates the distributed loss so that each GPU computes only local-row terms and receives remote gradient contributions by all_reduce. It reduces per-GPU contrastive-loss memory from $\mathcal M$2 to $\mathcal M$3, is mathematically equivalent to the original non-distributed loss, and enables ViT-B/32 training with batch size $\mathcal M$4K or $\mathcal M$5K using $\mathcal M$6 or $\mathcal M$7 A100 40GB GPUs, whereas the original CLIP solution reportedly requires $\mathcal M$8 A100 40GB GPUs for batch size $\mathcal M$9K. Inf-CL pushes the same idea further with tile-based exact computation, reporting CLIP-ViT-L/14 batch sizes of $$\mathcal{L}_{\mathrm{NCE}(f_q(c(x)),f_k(c'(x)), \mathcal{M})} = -\log \frac{\exp(q \cdot k_+ / T)} {\exp(q \cdot k_+ / T)+\sum_{k_- \in \mathcal{M}}\exp(q \cdot k_- / T)}.$$0M or $$\mathcal{L}_{\mathrm{NCE}(f_q(c(x)),f_k(c'(x)), \mathcal{M})} = -\log \frac{\exp(q \cdot k_+ / T)} {\exp(q \cdot k_+ / T)+\sum_{k_- \in \mathcal{M}}\exp(q \cdot k_- / T)}.$$1M on $$\mathcal{L}_{\mathrm{NCE}(f_q(c(x)),f_k(c'(x)), \mathcal{M})} = -\log \frac{\exp(q \cdot k_+ / T)} {\exp(q \cdot k_+ / T)+\sum_{k_- \in \mathcal{M}}\exp(q \cdot k_- / T)}.$$2 or $$\mathcal{L}_{\mathrm{NCE}(f_q(c(x)),f_k(c'(x)), \mathcal{M})} = -\log \frac{\exp(q \cdot k_+ / T)} {\exp(q \cdot k_+ / T)+\sum_{k_- \in \mathcal{M}}\exp(q \cdot k_- / T)}.$$3 A800 80GB GPUs without sacrificing accuracy (Chen et al., 2023, Cheng et al., 2024).

A second boundary case is that some methods contain both a memory bank and a contrastive component but not a strict memory-bank contrastive loss. ResiTTA uses a memory bank of samples $$\mathcal{L}_{\mathrm{NCE}(f_q(c(x)),f_k(c'(x)), \mathcal{M})} = -\log \frac{\exp(q \cdot k_+ / T)} {\exp(q \cdot k_+ / T)+\sum_{k_- \in \mathcal{M}}\exp(q \cdot k_- / T)}.$$4 for teacher-student self-training rather than InfoNCE over stored embeddings (Zhou et al., 2024). MemCollab constructs a shared bank of textual reasoning constraints $$\mathcal{L}_{\mathrm{NCE}(f_q(c(x)),f_k(c'(x)), \mathcal{M})} = -\log \frac{\exp(q \cdot k_+ / T)} {\exp(q \cdot k_+ / T)+\sum_{k_- \in \mathcal{M}}\exp(q \cdot k_- / T)}.$$5 by contrasting trajectory pairs, but its appendix-only MemNCE is explicitly not directly optimized (Chang et al., 24 Mar 2026). DRE-SLCL stores tile embeddings for all WSIs in a dataset-wide bank and then applies a slide-report CLIP-style loss only over current-batch slides; the bank affects representation construction, not the denominator of the contrastive loss (Jin et al., 7 Nov 2025). MTRMB likewise combines a task memory bank with separate structural and cross-modal contrastive objectives, but the bank itself is used for task retrieval and anomaly scoring rather than as a queue of contrastive keys (Zhou et al., 10 Feb 2025).

The resulting taxonomy is therefore narrow in one sense and broad in another. In the strict sense, memory-bank contrastive loss refers to objectives in which stored representations directly participate in the contrastive logits, denominators, or proxy comparisons. In the broader sense, it names a design space organized around persistent representation stores, bank update rules, and negative-set control. Current work suggests that the decisive questions are no longer simply whether to use a bank, but which distributions the bank should represent, how stale or asymmetric representations are handled, and how the bank’s enlarged comparison set is weighted, filtered, or matched to the task structure (Xu et al., 2020, Kim et al., 2024, Zheng et al., 2021).