DropMix: Hard Negative Synthesis in GCL

• DropMix is a graph contrastive learning method that synthesizes hard negative samples using a principled two-view selection and partial-dimension mixing to preserve key features.
• It leverages both local adjacency and global diffusion similarities to rank negatives, ensuring that only informative negatives contribute to the InfoNCE loss.
• Empirical evaluations demonstrate that DropMix consistently improves node classification accuracy across benchmarks, outperforming traditional Mixup, CutMix, and ProGCL methods.

DropMix is a technique designed to improve graph contrastive learning (GCL) by synthesizing harder negative samples through partial-dimension mixing of node representations selected based on a principled two-view "hardness" criterion. It addresses the limitations of existing negative sample generation methods for graph-structured data within self-supervised settings and achieves consistent performance gains across multiple benchmarks (Ma et al., 2023).

1. Background: Contrastive Learning on Graphs and the Role of Hard Negatives

Contrastive learning (CL) seeks to learn expressive encodings by attracting representations of positive pairs (semantic equivalents) and repelling negatives. In GCL, this is operationalized by generating two "views" (via augmentation or graph diffusion), encoding node representations with a graph neural network (GNN), and applying the InfoNCE loss:

$$L_{i,j} = -\log \frac{\exp(\mathrm{sim}(h_i, h_j)/\tau)}{\sum_{k=1}^N \exp(\mathrm{sim}(h_i, h_k)/\tau)}$$

where $h_i$ and $h_j$ are embeddings of augmented samples of node $v_i$, temperature $\tau$, and similarity function (cosine). Recent advances highlight that "hard" negatives—those semantically or structurally close to the anchor—contribute most valuable gradients, but can be challenging to generate robustly in graphs due to label scarcity and structural nuances.

2. Limitations of Existing Mixup Methods and Motivation for DropMix

Standard Mixup interpolates between pairs of data and their labels, generating synthetic points in the representation space. In unsupervised graph settings, label mixing is infeasible. Naïve Mixup on whole graph representations is problematic:

• Label absence: No label supervision impedes correct blending of representations;
• Full-dimension mixing: Directly interpolating all embedding dimensions can eradicate features that make negatives "hard," reducing their informativeness.

Previous methods such as CutMix (patch-level mixing) and ProGCL (local hardness with Mixup) neglect global graph structure and employ whole-embedding Mixup, leading to suboptimal preservation of discriminative features. DropMix introduces:

• Two-view hardness selection: Using both local adjacency and global diffusion similarity for negative selection;
• Partial-dimension Mixup: Only mixing on a proportion $\gamma$ of representation dimensions to contain information loss.

3. The DropMix Method: Hard Negative Synthesis with Partial Mixing

3.1 Graph Encoding and Contrastive Objective

Given $G=(V,E)$, node features $X \in \mathbb{R}^{N \times d}$, and normalized adjacency $\hat{A}$, a standard GNN encoder computes representations via

$$h_i^{(\ell)} = \mathrm{COMBINE}\bigl(h_i^{(\ell-1)},\; \mathrm{AGGREGATE}(\{ h_u^{(\ell-1)} : u \in \mathcal{N}(i) \})\bigr)$$

Two views are constructed:

• Local view: direct adjacency $\hat{A}$,
• Global view: diffusion matrix $S$, commonly by Personalized PageRank (PPR): $S = \alpha(I - (1-\alpha)\hat{A})^{-1}$.

The loss is the sum of InfoNCE computed on both views.

3.2 Hard Negative Selection (Multi-View)

For each anchor-positive pair, DropMix computes cosine similarity for each negative between:

• The anchor and negatives in local view, $\Phi_\ell$;
• The anchor and negatives in global view, $\Phi_g$;

$$\Phi_\ell(v_p, v_n) = \frac{h_p^\top h_n}{\|h_p\|\|h_n\|}, \quad \Phi_g(v_p, v_n) = \frac{\tilde{h}_p^\top \tilde{h}_n}{\|\tilde{h}_p\|\|\tilde{h}_n\|}$$

$\Phi(v_p, v_n) = \Phi_\ell + \Phi_g$

Negatives are ranked by $\Phi$; only those within percentiles $[\alpha,\beta]$—neither too easy nor extreme—are retained to form a hard negative pool.

3.3 Partial-Dimension Mixup (DropMix Operator)

Given two negatives $h_1, h_2$ from the pool, the following procedure generates a synthetic hard negative:

1. Compute a convex combination, $h_{\text{mix}} = \lambda h_1 + (1-\lambda) h_2$, $\lambda \sim \mathrm{Beta}(a, a)$ or uniform.
2. Generate binary mask $M \in \{0,1\}^d$ with proportion $\gamma$ of ones.
3. Compute

$$h_{\text{new}} = M \odot h_1 + (1-M) \odot h_{\text{mix}}$$

Only $(1-\gamma)d$ dimensions are affected by Mixup, while $\gamma d$ dimensions remain from the base hard negative. This partial mixing mitigates the dilution of critical features. The new $h_{\text{new}}$ replaces a negative in the InfoNCE denominator.

3.4 Algorithmic Summary

A single epoch proceeds as:

• Encode both local and global views
• For each node, select hard negatives using multi-view similarity
• From the hard negative pool, use DropMix to synthesize new negatives
• Replace a negative in the contrastive loss with $h_{\text{new}}$
• Backpropagate total contrastive loss (sum of both views)

4. Experimental Framework and Benchmarks

DropMix was evaluated on six prominent graph benchmarks:

Dataset	Nodes	Edges	Features	Classes
Cora	2,708	5,429	1,433	7
Citeseer	3,327	4,732	3,703	6
Pubmed	19,717	44,338	500	3
Wiki-CS	11,701	216,123	300	10
Amazon-Photo	7,650	119,081	745	8
Coauthor-CS	18,333	81,894	6,805	15

The base GNN is MVGRL, utilizing a shared GCN encoder for both local and global views. Hyperparameters such as $\lambda$, $\gamma$, $\alpha$, and $\beta$ are chosen via grid search. Evaluation is carried out via a downstream node classification task using a linear classifier on frozen embeddings, with test accuracy reported as mean $\pm$ standard deviation over 10 runs.

5. Comparative Results and Performance Analysis

Test accuracy comparisons are summarized below:

Method	Cora	Citeseer	Pubmed	Wiki-CS	Amazon-Photo	Coauthor-CS
MVGRL+DropMix	87.17±0.31	74.74±0.23	81.29±0.26	78.82±0.16	94.46±0.33	96.66±0.52
MVGRL+Mixup	86.92±0.44	74.08±0.28	80.62±0.52	78.23±0.46	93.55±0.63	94.66±0.21
MVGRL+CutMix	86.87±0.66	74.24±0.52	80.68±0.40	78.42±0.64	93.47±0.39	95.21±0.50
ProGCL	83.50±0.22	74.19±0.13	80.77±0.15	78.45±0.04	93.64±0.13	93.67±0.12
MVGRL	86.80±0.50	73.30±0.50	80.10±0.70	77.43±0.17	92.08±0.01	92.18±0.05

DropMix produces consistent improvements of 0.4–1.2 points over MVGRL and outperforms ProGCL by 1.5–3.0 points. The superior performance compared to standard Mixup and CutMix indicates the significance of partial-dimension mixing and multi-view negative selection.

6. Ablation Studies and Sensitivity

Detailed ablation studies dissect the contributions of core components:

• Multi-View Hardness vs. Single View: Combining both local and global hardness yields higher accuracy than either alone, verifying their complementarity.
• Mixing Method: DropMix on embeddings surpasses both Mixup and CutMix, showing that partial-dimension mixing better maintains hard negative informativeness.
• Application Layer: DropMix applied to learned embeddings offers larger gains over input features, highlighting the importance of representation-level augmentation.
• Hyperparameters:
  • Thresholds ($\alpha$, $\beta$): Both too lenient and too strict filtering reduces performance; mid-range thresholds are optimal.
  • Mask keep-rate ($\gamma$): Best values in $[0.2, 0.4]$. Too high ($\gamma > 0.5$) weakens mixing; too low ($\gamma \ll 0.1$) increases information loss.
  • Mixup coefficient ($\lambda$): Values in $[0.2, 0.5]$ are best; extremes revert to originals or overly dilute features.

7. Theoretical Insights and Practical Implications

DropMix's improvements are attributed to several design principles:

• Balanced Hardness: By leveraging both local and global similarities, DropMix focuses on negatives that are sufficiently hard but not likely mislabeled positives.
• Controlled Information Blending: Partial-dimension mixing protects critical structural or semantic features that define hard negatives, preventing representation collapse.
• Unsupervised Suitability: The method is fully label-free, relying solely on embedding-space operations, thus naturally extending to unsupervised GCL.
• Enhanced Gradient Signal: Focusing optimization on informative negatives theoretically and empirically leads to richer node representations.

DropMix is a lightweight, general augmentation compatible with any two-view GCL framework, requiring no label information or complex auxiliary structures, and delivers consistent accuracy improvements across diverse graph domains (Ma et al., 2023).