Dictionary-Free Contrastive Learning

• Dictionary-free contrastive learning is defined by eliminating external dictionaries and constructing positive and negative pairs using only in-batch or local data signals.
• This approach has enabled significant improvements in bitext mining, semantic concept embedding, and vision self-supervised learning with measurable performance gains.
• Practical implementations optimize negative sampling with strategies like window negatives and dual-temperature loss functions to mitigate noise and improve robustness.

Dictionary-free contrastive learning refers to a class of contrastive learning techniques that dispense with any form of external negative sample "dictionary" (queue or memory bank) or linguistic lexicon during training. These frameworks construct positive and negative pairs internally—using in-batch or data-driven statistical heuristics—rather than relying on explicit bilingual dictionaries in NLP or the frequently used negative-dictionary queues in computer vision contrastive learning. This approach enables more flexible, scalable, and often more robust representation learning, particularly under constraints such as low-resource languages or limited hardware.

1. Core Principles of Dictionary-Free Contrastive Learning

Dictionary-free contrastive learning eliminates the need for two different types of "dictionaries":

• In natural language processing, it avoids reliance on external bilingual lexica or translation tables for mining positive or negative sample pairs.
• In computer vision and general representation learning, it omits momentum-encoded or other external queues for storing previous negative samples.

At its core, these methods maintain the InfoNCE or similar contrastive objective, but construct positives and negatives using only local or batch-based information. This is either accomplished by sampling from document alignments, mining structural similarities in embedding space, or using only mini-batch samples as negatives. This paradigm underpins recent advances in bitext mining for low-resource languages (Tan et al., 2022), unsupervised concept embedding distillation (Li et al., 2023), and simplified visual self-supervised learning architectures (Zhang et al., 2022).

2. Methodological Implementations and Framework Variants

(a) Bitext Mining (Low-Resource Languages)

The approach of (Tan et al., 2022) leverages a "Multiple Negatives Ranking" (MNR) loss without any external dictionary. Sentence-BERT (SBERT) bi-encoders are trained to maximize the cosine similarity between noisy, document-aligned sentence pairs and minimize it for negatives chosen from local (window-based) and random (global) sources:

• Positives: Any parallel sentence pair from weakly aligned document pairs (using Hunalign or LASER).
• Negatives: For each source sentence $s_i$, negatives are drawn from the surrounding $W$ sentences ("window" negatives) and $R$ random target sentences ("global" negatives) without reference to lexica or translation tables.

The resulting 768-dimensional representations are aligned using dynamic programming (Vecalign), scored with a margin-based nearest-neighbor metric, and filtered to produce high-quality mined bitext. All mining and filtering is done without any custom lexicon.

(b) Semantic Concept Embedding (Unsupervised Lexicon-Free Strategies)

(Li et al., 2023) describes a fully dictionary-free method for learning semantic concept embeddings. The approach constructs mention vectors φ(s,c) by masking the target concept in sentence $s$, then encoding with a frozen or fine-tuned BERT model. Positive pairs are inferred by comparing neighbor sets of mention vectors—if two mention vectors have similar neighbor-set distributions over masked concept mentions, they are treated as reflecting similar semantic properties.

The InfoNCE-style supervised contrastive loss is then applied, with all other examples in the minibatch acting as negatives. All positive and negative sampling processes are performed without any external dictionary or precompiled semantic lexicon.

(c) Visual Representation Learning Without Momentum Queues

The MoCo paradigm originally introduced large momentum-based negative dictionaries to overcome mini-batch size constraints (Zhang et al., 2022). By analyzing the gradient structure of InfoNCE, (Zhang et al., 2022) demonstrates that the need for a dictionary can be obviated by reparameterizing the loss with two separate temperature hyperparameters: one for "intra-anchor" negative hardness (τₐ) and one for "inter-anchor" scaling (τ_b). In their SimMoCo and SimCo frameworks, all negatives are sourced from the current mini-batch, using a dual-temperature InfoNCE loss. This recovers and surpasses performance of dictionary-based MoCo variants across multiple vision benchmarks.

3. Loss Formulations and Negative Pair Construction

Dictionary-Free InfoNCE and MNR Losses

Contrastive losses operate over pairs of representations $(q_i, k_j)$, with minimization encouraging paired positives and separation from negatives. Typical dictionary-based methods introduce external memory banks for negatives; dictionary-free approaches leverage:

• In-batch negatives: All other samples in the mini-batch.
• Window negatives: For sequential/text data, use temporal, spatial, or document-local proximity for hard negative selection.
• Data-driven positives: For NLP, structural compatibility in embedding neighborhoods or document alignment is used.

MNR Loss (for sentence alignment) (Tan et al., 2022)

$$J(s, t; \theta) = -\frac{1}{K} \sum_{i=1}^K \left[ d(s_i, t_i) - \log \sum_{j=1}^{2W+R} \exp(d(s_i, t_j)) \right]$$

with cosine similarity $d(s, t) = \cos(r_s, r_t)$, where $r_s$ and $r_t$ are sentence embeddings.

Unsupervised Concept Pairing and InfoNCE (Li et al., 2023)

Positive pairs are constructed as those where mention vectors have similar neighbor distributions. The supervised contrastive loss:

$$L = \sum_{(s,c)\in B} -\frac{1}{|Pos_B(s,c)|} \sum_{(s',c')\in Pos_B(s,c)} \log \frac{\exp(\cos(u(s,c), u(s',c'))/\tau)}{ \sum_{(s'',c'')\in B \setminus \{(s,c)\}} \exp(\cos(u(s,c), u(s'',c''))/\tau )}$$

where all non-positive pairs in the batch serve as negatives.

Dual-Temperature InfoNCE for Vision (Zhang et al., 2022)

The SimMoCo and SimCo loss is structured to decouple inter- and intra-anchor contrastive effects:

$W$0

where $W$1 and $W$2 are computed from softmaxes with $W$3 and $W$4 respectively.

4. Architectures, Optimization, and Sampling Strategies

Dictionary-free methods require careful design of network architectures and data pipelines to maximize contrastive signal without the crutch of large external negative sets.

• Bitext mining: Utilizes pretrained SBERT bi-encoders run separately on source/target, mean-pooling, and Vecalign-based alignment. Positives from ParaCrawl, negatives from in-batch/document context, no lexicon (Tan et al., 2022).
• Concept embeddings: Mask-invariant BERT (frozen or fine-tuned), shallow linear projector (ConProj), or full fine-tuning (ConFT/ConCN), no external concept dictionary required (Li et al., 2023).
• Visual SSL: Standard ResNet-MLP backbone, batch negatives only, dual temperature loss, symmetric or asymmetric computation depending on variant (Zhang et al., 2022).

Optimization typically employs AdamW (NLP) or SGD with momentum (vision), standard learning rates and decay, and moderate batch sizes (ranging from 32 to 1024). Distributed and hardware acceleration (single or multi-GPU) is used for scalable embedding computation and nearest neighbor search.

5. Evaluation Benchmarks and Empirical Insights

Quantitative Gains

• Bitext mining: SBERT-ALIGN outperforms prior state-of-the-art bitext mining systems on English–Pashto and English–Khmer, with BLEU improvements of +1.1 to +2.0 over strong baselines using dictionary-free contrastive learning (Tan et al., 2022).
• Semantic concept embeddings: Dictionary-free contrastive representations surpass skip-gram, GloVe, Masked-BERT, and specialized models in macro F₁ for semantic property classification and cluster purity for concept grouping. Example uplifts: X-McRae property prediction F₁ rises from 64.1% to 68.3% (fully unsupervised) or up to 73.7% with distant supervision (Li et al., 2023).
• Vision SSL: SimMoCo and SimCo, without any momentum queue, achieve or exceed MoCo v2 performance (absolute top-1 accuracy gain of ∼4–5% on CIFAR, STL, and other image datasets), with gains robust across architectures and long training regimes (Zhang et al., 2022).

Qualitative Observations

Dictionary-free frameworks demonstrate:

• Robustness to spurious matches (boilerplate, mis-language) via local-context negatives (Tan et al., 2022).
• Automatic pruning of "hub" vectors in high dimensions using margin-based scoring (Tan et al., 2022).
• Stronger semantic generalization in concept embedding tasks, with property-level semantics emerging purely from sentence-structure statistics (Li et al., 2023).
• Performance stability across batch sizes and architectures when inter-anchor scaling is controlled (SimCo/SimMoCo) (Zhang et al., 2022).

6. Limitations and Open Challenges

Dictionary-free contrastive learning reveals specific challenges:

• Semantic drift for short/ambiguous sentences: Very short input segments remain challenging without dictionary or contextual cues (Tan et al., 2022).
• Noise sensitivity: Non-parallel or noisy aligned corpora introduce false positives, mitigated partially but not entirely by robust contrastive training (Tan et al., 2022).
• Negative sample dilution: Vision models with only in-batch negatives were previously believed to underperform; dual-temperature modeling partially resolves this but optimal hyperparameter selection remains nontrivial (Zhang et al., 2022).
• Noisily inferred positives: Reliance on neighbor compatibility structures for identifying positives may warrant additional cleaning or multi-step curriculum in concept embedding (Li et al., 2023).
• Generalization to truly low-resource/data-scarce regimes: Success relies on the availability of at least weakly aligned data or sufficiently diverse minibatch negatives.

7. Theoretical and Practical Implications

Dictionary-free contrastive learning unifies and generalizes representation learning frameworks by illustrating how negative sampling, semantic alignment, and batch-based optimization can together substitute for external dictionaries or lexica. In vision, this renders self-supervised pipelines more accessible for resource-constrained environments and clarifies the theoretical basis for the efficacy of InfoNCE versus non-contrastive losses (Zhang et al., 2022). In natural language, dictionary-free mining and embedding yield practical advances for marginalized languages and robust semantic induction (Tan et al., 2022, Li et al., 2023).

A plausible implication is that as batch sizes and data diversity increase, dictionary-free frameworks may continue to subsume traditional dictionary-based approaches, provided negative sampling is sufficiently informative and scaling factors (temperatures) are correctly tuned. Extension to new modalities or settings will likely revolve around efficiently constructing positives/negatives from unlabeled or weakly labeled structure, obviating domain-specific lexicon engineering.