Structure-Based Contrastive Learning

• Structure-Based Contrastive Learning is a method that leverages explicit structures such as label hierarchies, graph connections, and invariances to form disentangled and interpretable latent representations.
• It refines standard contrastive objectives by partitioning embeddings into invariant, variant, and free subspaces, thereby boosting model robustness against nuisance variations.
• Empirical results demonstrate that SCL improves performance in diverse tasks including aspect-based sentiment analysis, graph classification, and power grid stability prediction through tailored structural priors.

Structure-Based Contrastive Learning (SCL) refers to a set of contrastive learning methods in which structural information—whether defined by explicit data attributes, label hierarchies, graph structure, or transformation invariance—is leveraged to supervise or regularize the representation space. These approaches operate across modalities, including text, vision, time series, and graphs, and may be supervised or unsupervised. Central to SCL is the explicit encoding of inductive biases or structural priors into contrastive objectives, thereby producing latent representations that are disentangled, interpretable, and robust to nuisance factors.

1. Theoretical Foundations and Motivation

SCL arises as a response to limitations in both laissez-faire representation learning and standard (instance-based) contrastive methods. In typical deep networks, latent spaces are organized solely to minimize task loss, often resulting in representations that are brittle to semantically irrelevant transformations and fail to reflect structural semantics (such as label hierarchies, graph topologies, or global attributes) (Shen et al., 18 Nov 2025).

Standard contrastive learning, especially self-supervised variants (e.g., SimCLR, MoCo), enforces invariance of representations under augmentations but does not distinguish which aspects of the latent code should remain invariant versus those encoding explicit structural variation. This uniform treatment produces representations that may underfit global structure and offer limited interpretability or robustness under distributional shifts.

Structure-Based Contrastive Learning explicitly partitions, weights, or regularizes contrastive objectives using task-relevant structure:

• Semantic attributes (e.g., sentiment, aspect polarity)
• Label hierarchies or similarities
• Graph connectivity, multi-scale or cluster semantics
• Nuisance transformations (e.g., rotation, phase shift)

The SCL paradigm leads to improved generalization, greater interpretability, and enhanced downstream task performance, especially in challenging or underdetermined regimes (Shen et al., 18 Nov 2025, Peper et al., 2022).

2. Formalism: Loss Functions and Structural Pair Construction

SCL modifies the standard contrastive loss to account for structural attributes present in the data. Key formal components include:

• Pair Construction: Structural attributes define which examples should be considered "positives" (to pull together), and which should be "negatives" (to push apart), beyond simple instance identity.
  • In text, sentiment, aspect, and opinion labels define groups for positive coupling (Peper et al., 2022).
  • In graph data, structural views (different propagation depths) or semantic cluster assignments provide pos/neg structure (Ding et al., 2022).
  • In label hierarchies, class-proximity matrices weight the push-pull interaction (Lian et al., 2024).
• Loss Formulation: For each structural attribute $c$, and anchor $x_i$, positives $P(i)$ are those examples (and augmented views) with the same $y_{c,i}$ label; negatives are the complement.

For example, the supervised structural contrastive loss in GEN-SCL (Peper et al., 2022) is: $$L_{i}^{c} = - \frac{1}{|P(i)|} \sum_{p\in P(i)} \log \frac{\exp( \mathrm{sim}(h_{c,i}, h_{c,p}) / \tau )}{\sum_{b\in B(i)} \exp( \mathrm{sim}(h_{c,i}, h_{c,b}) / \tau )}$$ where $h_{c,i}$ is a projected representation for characteristic $c$ of $x_i$.

Label-aware SCL (LASCL) (Lian et al., 2024) introduces a scaled temperature for negatives that reflects class proximity $s_{ik}$: $\ell_{sii}(x_i,y_i) = \mathbb{E}_{j\in\Pcal(y_i)} \left[ -\log \frac{\exp( \mathrm{sim}(x_i,x_j)/\tau )}{\sum_{k\in\Ncal(y_i)} \exp(\mathrm{sim}(x_i,x_k)/(\tau s_{ik}))} \right]$

• Partitioned Representations: In the framework of (Shen et al., 18 Nov 2025), latent representations are partitioned into subspaces—invariant, variant, and free—with a corresponding loss promoting invariance in one, variance in another, and task regularization in the remainder.

3. Methodological Variants and Architectures

SCL is realized through multiple architectural and algorithmic interventions:

• Projection Head Partitioning: Characteristic-specific projection heads or latent splits for different structure types (e.g., in GEN-SCL and (Shen et al., 18 Nov 2025)), ensuring disentanglement in the contrastive space.
• Prototype-Based Objectives: SCL models may instantiate learnable or fixed prototypes representing semantic centers (e.g., classes or clusters). These proxies are incorporated as extra points in the embedding space to drive geometry towards desired configurations, such as equiangular tight frames (ETF) or more general Gram matrices (Gill et al., 2023, Ding et al., 2022, Lian et al., 2024).
• Graph Structural Encoding: On graphs, SCL uses propagation-based augmentations and explicit cross-view alignment losses (e.g., between one-hop and multi-hop views in S³-CL), or semantic clustering to form positive sets at the cluster or prototype level (Ding et al., 2022).
• Domain-Specific Input Features: In power systems, SCL is applied to graph embedding dynamic features (GEDF) that encode transient structural dynamics of grid topology (Lv et al., 2023).
• Loss Aggregation: Multi-objective L_total is typically a weighted sum of task loss and multiple SCL losses, with hyperparameters tuned for optimal balance (Peper et al., 2022, Shen et al., 18 Nov 2025).

4. Interpretability and Inductive Bias

A central justification for SCL is the production of interpretable latent spaces—often termed "glass-box embeddings" (Shen et al., 18 Nov 2025). Through explicit partitioning and targeted contrastive regularization, different subspaces are compelled to encode distinct semantic, structural, or nuisance factors.

t-SNE analyses and ablation studies demonstrate that:

• Invariant subspaces capture task-relevant, transformation-agnostic content, enabling invariance to noise and irrelevant variations.
• Variant subspaces organize data according to nuisance transformations (such as rotation angle or phase shift), endowing the representation with transparency regarding what factors contribute to output variation.

This disentanglement facilitates glass-box analysis: one can inspect the invariant subspace to determine semantic classification, the variant subspace to infer applied transformations, and the free subspace for auxiliary task information.

Ablation studies further confirm that removing structure-aware SCL terms predominantly degrades performance on those aspects most correlated with the suppressed loss (e.g., implicit opinions or aspects in sentiment analysis) (Peper et al., 2022).

5. Empirical Results and Applications

SCL methods achieve measurable improvements across diverse domains. Highlights include:

• Aspect-based Sentiment Analysis: GEN-SCL-NAT achieves +1.08 to +1.73 F1 improvement over baselines on ACOS extraction, with the largest gains on implicit aspect/opinion splits (Peper et al., 2022). The model demonstrates substantial improvements in clustering and discrimination among latent representations associated with subtle structural cues.
• Robust Representations under Transformations: In ECG phase-invariance, SCL increases latent similarity from 0.25 (baseline) to 0.91 under phase shifts; in IMU activity data, SCL improves accuracy to 86.65% with 95.38% rotation consistency, outperforming both standard contrastive and data augmentation (Shen et al., 18 Nov 2025).
• Graph Representation Learning: S³-CL achieves state-of-the-art unsupervised node classification and clustering across six benchmarks, outperforming baselines by 1–2 accuracy points and yielding representations robust under label sparsity (Ding et al., 2022).
• Long-Tail and Class-Imbalanced Regimes: Prototype-augmented SCL drives embedding spaces toward target geometries (ETF or custom angular constraints), which improves classification accuracy and minority-class recall under severe imbalance (Gill et al., 2023).
• Power Grid Stability Prediction: In transient stability prediction, GEDF-SCL provides 2–5% gains over end-to-end CNNs and 15–25% over GNNs, and generalizes effectively to previously unseen grid topologies (Lv et al., 2023).

Detailed empirical ablations establish that structural contrastive terms directly improve representation geometry and downstream predictive performance—particularly in regimes requiring generalization to unseen transformations, topologies, or rare classes.

6. Extensions, Practical Considerations, and Limitations

Practical integration of SCL requires selection of structural attributes, partitioning schema, and hyperparameters (temperature, weighting coefficients, prototype geometries). Prototypical SCL methods offer plug-and-play geometry control with minimal computational cost—achievable with small prototype sets, normalized embeddings, and standard optimization schemes (Gill et al., 2023).

Extensions of SCL include:

• Incorporation of hierarchical or side information via learnable label embeddings and custom similarity metrics (Lian et al., 2024)
• Augmentation with semantic clustering and label propagation for unsupervised global knowledge extraction (Ding et al., 2022)
• Application to dynamically structured domains (e.g., time-evolving power grids) using graph-embedded dynamic features (Lv et al., 2023)

Known limitations include the need for tuning structural hyperparameters (e.g., partition dimensions, number of prototypes), potential sensitivity of clustering-based methods to imbalanced or noisy data, and the requirement that structure-defining attributes be available or reliably inferred.

7. Representative Table: Domains and SCL Methodologies

Domain	SCL Mechanism	Reference
ABSA (text)	Characteristic-based SCL (sentiment, aspect)	(Peper et al., 2022)
Time-series / IMU	Invariant/variant-free partition & variant loss	(Shen et al., 18 Nov 2025)
Graphs	Structural/semantic contrastive via propagation	(Ding et al., 2022)
Class-imbalanced vision	Fixed prototypes, ETF geometry	(Gill et al., 2023)
Hierarchical labels	Label-aware instance/center SCL	(Lian et al., 2024)
Power grid modeling	Graph-embedding dynamic feature SCL	(Lv et al., 2023)

These works collectively establish SCL as an extensible paradigm for encoding structure, improving robustness, interpretability, and performance in complex, real-world tasks.