ContAccum: Contrastive Accumulation

• ContAccum is a contrastive accumulation method that uses gradient accumulation with dual memory banks to enrich the negative set for enhanced dense retrieval.
• It employs symmetric query and passage memory banks to maintain balance in gradient dynamics, mitigating instability common in naive memory-bank approaches.
• The method efficiently simulates large-batch behavior on limited GPU resources by combining micro-batches without storing extra activations, resulting in superior retrieval performance.

ContAccum, short for Contrastive Accumulation, is a memory-reduction and gradient accumulation method tailored to InfoNCE-based dense retrieval. It is designed for dual-encoder training under tight GPU constraints, and combines gradient accumulation with a dual memory bank of query and passage representations so that each optimization step can see many more negatives than the physical batch would otherwise allow, while avoiding the instabilities seen in naive memory-bank approaches (Kim et al., 2024).

1. Problem setting and conceptual scope

Dense retrievers trained with InfoNCE typically rely on in-batch negatives: for a batch of $N$ query–passage pairs $(q_i,p_i)$, each $p_j$ for $j \neq i$ is treated as a negative for $q_i$. The loss is written as

$$\mathcal{L}(S) = -\frac{1}{N} \sum_{i=1}^{N} \log \frac{\exp(S_{(i,i)} / \tau)} {\sum_{j=1}^{N} \exp (S_{(i,j)} / \tau)},$$

with $S = \mathrm{Softmax}(\mathbf{Q}\mathbf{P}^{\top})$. Larger $N$ yields more negatives per query, and the paper states that both theory and practice show this is crucial: the mutual-information lower bound of InfoNCE tightens with more negatives, and dense retrievers often use batch sizes 128–8192 and multi-GPU setups with 8–32 accelerators to obtain strong retrieval performance (Kim et al., 2024).

This creates the central hardware bottleneck that ContAccum addresses. On single commodity GPUs such as 11GB or 24GB devices, true large batches do not fit. The paper contrasts three standard workarounds. Naïve Gradient Accumulation (GradAccum) splits a logical batch into micro-batches and averages losses across them, but each query still sees only $N_{\text{local}}-1$ negatives rather than $N_{\text{total}}-1$, so it is not the same InfoNCE gradient as a genuine large batch. Gradient Cache (GradCache) reconstructs the total-batch InfoNCE by multiple forward and backward passes and cached gradients with respect to representations, but it incurs substantial compute and wall-clock overhead and cannot exceed the performance of a high-resource setting that uses the same $(q_i,p_i)$0. Naïve memory banks / pre-batch negatives reuse cached passage representations as additional negatives, but the paper identifies training instability and a severe gradient norm imbalance between query and passage encoders when only one side is cached (Kim et al., 2024).

Within that landscape, ContAccum is positioned as a method that attempts to retain the batch-size benefits of large negative sets while remaining practical under memory constraints. A common misconception is that ordinary gradient accumulation already provides large-batch InfoNCE. The paper explicitly rejects that equivalence: gradient averaging over micro-batches preserves the amount of data, but not the number of negatives seen by each query.

2. Method architecture: dual memory banks and enlarged contrastive sets

The core of ContAccum is a dual memory bank. It maintains two FIFO queues:

• a query memory bank $(q_i,p_i)$1, storing $(q_i,p_i)$2 past query vectors;
• a passage memory bank $(q_i,p_i)$3, storing $(q_i,p_i)$4 past passage vectors.

For accumulation step $(q_i,p_i)$5, the current micro-batch representations are $(q_i,p_i)$6 and $(q_i,p_i)$7. ContAccum augments them with cached, stop-gradient representations: $(q_i,p_i)$8 and computes the similarity matrix

$(q_i,p_i)$9

Here $p_j$0 denotes stop-gradient; in the implementation it is realized with .detach(), so gradients propagate only through representations from the current micro-batch (Kim et al., 2024).

The immediate effect is an enlarged negative set. For the current batch, the number of negatives per query becomes

$p_j$1

The paper notes that if $p_j$2, ContAccum can provide more negatives per query than a high-resource baseline with true batch size $p_j$3. This is the central mechanism by which the method can match or surpass true large-batch baselines without storing activations for such batches (Kim et al., 2024).

The defining design choice is symmetry. The query and passage memory banks are used symmetrically, and the analysis strongly recommends

$p_j$4

That symmetry is not incidental; it is the mechanism used to stabilize dual-encoder optimization. In the paper’s formulation, ContAccum does not alter the form of the InfoNCE objective so much as enlarge the representation sets appearing in the similarity sums, while keeping the enlargement balanced across the two encoders.

3. Gradient dynamics, stability, and the role of symmetry

The paper’s theoretical contribution centers on the gradient structure induced by the dual memory bank. For accumulation step $p_j$5, the gradients with respect to encoder parameters depend on partial derivatives of the loss with respect to current query and passage vectors. The paper derives that the query encoder’s gradient depends on a weighted combination of all passage vectors, current plus memory, while the passage encoder’s gradient depends on a weighted combination of all query vectors, current plus memory (Kim et al., 2024).

That symmetry matters because one-sided memory breaks it. If only a passage memory bank is used, then the query gradient aggregates over $p_j$6 passages, while the passage gradient aggregates over only $p_j$7 queries. The paper states that this leads systematically to

$p_j$8

which it terms the gradient norm imbalance problem. In the empirical study, $p_j$9 can be up to 30× larger than $j \neq i$0 in a pre-batch-like setting with no query memory bank (Kim et al., 2024).

To quantify this, the authors define

$j \neq i$1

On NQ, the DPR baseline with no memory bank keeps this ratio near 1. A pre-batch-like setting with no query memory bank causes the ratio to increase steadily and exceed 30. ContAccum, with a dual memory bank, keeps the ratio close to 1, similar to DPR, across training. Appendix D further shows that if training begins with dual memory banks and the query memory bank is later omitted at epochs 10, 20, or 30, the ratio immediately becomes unstable and grows, indicating that the instability is not restricted to early training (Kim et al., 2024).

The paper also addresses the separate concern that memory-bank representations may be too stale to remain useful. It defines a similarity mass

$j \neq i$2

and reports that passage representations from up to six previous steps have similar similarity mass to current-step passages. Since gradients are proportional to similarity in the derived expressions, this is taken as evidence that past representations remain informative negatives even early in training (Kim et al., 2024).

A second misconception is therefore addressed by the paper: the instability of naive memory-bank methods is not explained solely by representational staleness. The analysis identifies asymmetric gradient aggregation as an additional critical mechanism.

4. Training algorithm and implementation profile

The paper does not present explicit pseudocode, but it provides a stepwise training procedure. The training state consists of query encoder $j \neq i$3, passage encoder $j \neq i$4, two FIFO memory banks $j \neq i$5 and $j \neq i$6, local batch size $j \neq i$7, accumulation steps $j \neq i$8, temperature $j \neq i$9, and an optimizer. In the experiments, the optimizer is AdamW with learning rate $q_i$0, weight decay 0, gradient clipping 2.0, and temperature $q_i$1 (Kim et al., 2024).

For each optimizer step, gradients are zeroed, and then each accumulation step proceeds as follows. A micro-batch of $q_i$2 query–passage pairs is sampled. The current representations $q_i$3 and $q_i$4 are encoded. The extended sets $q_i$5 and $q_i$6 are formed by concatenating the current micro-batch with detached memory-bank entries. The similarity matrix and InfoNCE loss are computed on that enlarged set. The loss is scaled by $q_i$7, and backward() is applied so that the total accumulated gradient is comparable to a single large-batch gradient. The current micro-batch representations are then detached and enqueued into the corresponding FIFO banks; if a bank exceeds capacity $q_i$8, the oldest entries are dequeued. After all accumulation steps, gradients are clipped, optimizer.step() is executed, and the process repeats (Kim et al., 2024).

The paper emphasizes several implementation nuances. Balanced memory sizes are strongly recommended. ContAccum works with $q_i$9, meaning only a memory bank and no gradient accumulation, but combining both mechanisms is better. The experiments are limited to a single GPU; the paper states that multi-GPU extensions would need synchronization of memory banks, for example by merging cross-device embeddings, but this is not covered (Kim et al., 2024).

In memory terms, ContAccum adds detached representation banks of size $$\mathcal{L}(S) = -\frac{1}{N} \sum_{i=1}^{N} \log \frac{\exp(S_{(i,i)} / \tau)} {\sum_{j=1}^{N} \exp (S_{(i,j)} / \tau)},$$0 on each side, but it does not store extra activations beyond the current micro-batch. In compute terms, the main additional cost over GradAccum is the enlarged similarity matrix multiplication and softmax; relative to full encoder forward and backward passes, the paper characterizes this as modest. This design is the reason ContAccum is presented as materially simpler than GradCache while still increasing the effective contrastive set (Kim et al., 2024).

5. Experimental setup, quantitative behavior, and ablations

The empirical evaluation spans five IR datasets: Natural Questions (NQ), TriviaQA, Curated TREC (TREC), WebQuestions (WebQ), and MS MARCO. Metrics are Top@20 and Top@100 for NQ, TriviaQA, TREC, and WebQ, and NDCG@20/100 and Recall@20/100 for MS MARCO. The backbone is BERT-base-uncased in a dual-encoder DPR architecture, with FAISS exact search used at retrieval time (Kim et al., 2024).

Resource constraints are simulated on a single A100 (80GB physical) using torch.cuda.set_per_process_memory_fraction, emulating 11GB, 24GB, and 80GB settings. Example batch settings are:

• 11GB: DPR uses $$\mathcal{L}(S) = -\frac{1}{N} \sum_{i=1}^{N} \log \frac{\exp(S_{(i,i)} / \tau)} {\sum_{j=1}^{N} \exp (S_{(i,j)} / \tau)},$$1; GradAccum, GradCache, and ContAccum use $$\mathcal{L}(S) = -\frac{1}{N} \sum_{i=1}^{N} \log \frac{\exp(S_{(i,i)} / \tau)} {\sum_{j=1}^{N} \exp (S_{(i,j)} / \tau)},$$2.
• 24GB: DPR uses $$\mathcal{L}(S) = -\frac{1}{N} \sum_{i=1}^{N} \log \frac{\exp(S_{(i,i)} / \tau)} {\sum_{j=1}^{N} \exp (S_{(i,j)} / \tau)},$$3; the others use $$\mathcal{L}(S) = -\frac{1}{N} \sum_{i=1}^{N} \log \frac{\exp(S_{(i,i)} / \tau)} {\sum_{j=1}^{N} \exp (S_{(i,j)} / \tau)},$$4.
• 80GB: DPR uses $$\mathcal{L}(S) = -\frac{1}{N} \sum_{i=1}^{N} \log \frac{\exp(S_{(i,i)} / \tau)} {\sum_{j=1}^{N} \exp (S_{(i,j)} / \tau)},$$5 (Kim et al., 2024).

Memory-bank sizes are tuned on validation for NQ and TriviaQA from candidates [128, 512, 2048], yielding 2048 for NQ and 512 for TriviaQA. For datasets without separate validation, the chosen sizes are 1024 for MS MARCO and 128 for WebQ and TREC (Kim et al., 2024).

The main result is that under 11GB and 24GB settings, ContAccum consistently outperforms DPR and GradAccum, and also outperforms GradCache across most metrics. The most striking statement in the paper is that ContAccum with only 11GB VRAM surpasses the high-resource 80GB DPR baseline on 18 out of 24 metrics, with improvements up to ~4.9 points (Kim et al., 2024).

Two examples are highlighted explicitly. On MS MARCO NDCG@20 under 11GB, the scores are 27.9 for DPR, 31.1 for GradAccum, 34.9 for GradCache, and 39.1 for ContAccum. On NQ Top@20 under 11GB, the scores are 72.2 for DPR, 77.1 for GradAccum, 79.5 for GradCache, 79.4 for the 80GB DPR baseline, and 80.1 for ContAccum (Kim et al., 2024).

The ablation study on NQ sharpens the role of each component. With hard negatives, DPR (BSZ=8) obtains 70.9, DPR (BSZ=128) obtains 78.4, and ContAccum obtains 78.8. Removing the query memory bank $$\mathcal{L}(S) = -\frac{1}{N} \sum_{i=1}^{N} \log \frac{\exp(S_{(i,i)} / \tau)} {\sum_{j=1}^{N} \exp (S_{(i,j)} / \tau)},$$6 drops performance to 70.8, essentially back to the small-batch baseline. Removing past encoder representations yields 76.5. Removing both $$\mathcal{L}(S) = -\frac{1}{N} \sum_{i=1}^{N} \log \frac{\exp(S_{(i,i)} / \tau)} {\sum_{j=1}^{N} \exp (S_{(i,j)} / \tau)},$$7 and past encodings yields 67.8. Removing GradAccum while keeping the memory-bank mechanism yields 76.7 (Kim et al., 2024). These results support three factual conclusions stated in the paper: the query memory bank is pivotal, past representations are beneficial rather than merely approximate, and gradient accumulation remains helpful even when a memory bank is present.

The paper also evaluates scaling with memory-bank size and accumulation step. Under the low-resource NQ setting with local batch size 8, ContAccum always outperforms GradAccum and DPR across tested combinations. Even at $$\mathcal{L}(S) = -\frac{1}{N} \sum_{i=1}^{N} \log \frac{\exp(S_{(i,i)} / \tau)} {\sum_{j=1}^{N} \exp (S_{(i,j)} / \tau)},$$8, increasing $$\mathcal{L}(S) = -\frac{1}{N} \sum_{i=1}^{N} \log \frac{\exp(S_{(i,i)} / \tau)} {\sum_{j=1}^{N} \exp (S_{(i,j)} / \tau)},$$9 improves performance, and with moderate settings such as $S = \mathrm{Softmax}(\mathbf{Q}\mathbf{P}^{\top})$0, $S = \mathrm{Softmax}(\mathbf{Q}\mathbf{P}^{\top})$1, and $S = \mathrm{Softmax}(\mathbf{Q}\mathbf{P}^{\top})$2, ContAccum already surpasses high-resource DPR with batch size 128. Gains saturate as $S = \mathrm{Softmax}(\mathbf{Q}\mathbf{P}^{\top})$3 and $S = \mathrm{Softmax}(\mathbf{Q}\mathbf{P}^{\top})$4 continue to increase, indicating diminishing returns (Kim et al., 2024).

Training-speed comparisons further distinguish ContAccum from GradCache. At $S = \mathrm{Softmax}(\mathbf{Q}\mathbf{P}^{\top})$5 under 11GB, GradCache is ~93% slower per update than GradAccum, whereas ContAccum with $S = \mathrm{Softmax}(\mathbf{Q}\mathbf{P}^{\top})$6 is only ~26% slower than GradAccum. The paper summarizes this as a ~34% speed advantage over GradCache at that scale (Kim et al., 2024).

6. Limitations, extensions, and adjacent uses of the term

The authors delimit the method’s current scope. The experiments focus on supervised fine-tuning of dual-encoder dense retrievers and do not test ContAccum on unsupervised or contrastive pre-training for retrieval, nor on uni-encoders, cross-encoders, or multi-vector models such as ColBERT. The method still relies on a full softmax over a large set of negatives, so when $S = \mathrm{Softmax}(\mathbf{Q}\mathbf{P}^{\top})$7 becomes large, softmax itself may become a computational bottleneck. The paper identifies two future directions: efficient softmax alternatives and applying the method to pre-training to examine whether the same gradient norm imbalance arises there. It also cautions against using ContAccum-trained systems directly in high-stakes domains such as medical or legal settings without thorough validation (Kim et al., 2024).

The term “ContAccum” also appears in adjacent arXiv discussions with a broader systems meaning. In “Budgeted Dynamic Trace Structures for Token-Efficient Sequential Computation”, BDTS is described as “exactly the kind of mechanism a ‘ContAccum’ system needs” when maintaining an ever-growing trace under a hard context limit. In that setting, the relevant machinery is not contrastive learning but summary-plus-suffix compaction, explicit byte or token budgets, rooted trace graphs, delta overlays, and bounded cost caches for maintaining a compact recent view of a long-running computation (Alpay et al., 20 May 2026). A different systems-oriented usage appears in “Scalable Eventually Consistent Counters over Unreliable Networks”, which presents implications for a system like “ContAccum” through Eventually Consistent Distributed Counters and Handoff Counters, emphasizing no over-counting, local monotonicity, and eventual accounting under message loss, duplication, reordering, partitions, and crash–recovery (Almeida et al., 2013).

This suggests that “ContAccum” functions in two distinct but related ways in current arXiv discourse. In the precise, method-specific sense, it denotes Contrastive Accumulation for dense retriever training under memory constraint. In adjacent systems discussions, it serves as a broader label for continuous or cumulative accumulation mechanisms, particularly where long-lived state must be maintained under budgetary or distributed-systems constraints.