Subject-Aware Contrastive Learning

• Subject-Aware Contrastive Learning is a family of techniques that adjust contrastive objectives to account for subject-specific variability, enhancing model generalization across different individuals.
• It refines pair constructions and loss weighting within the InfoNCE framework to balance intra-subject consistency and inter-subject distinctiveness in applications like EEG, HAR, and gaze estimation.
• Empirical results demonstrate notable improvements in classification accuracy and few-shot transfer, while addressing challenges in scalability, invariance balance, and multimodal integration.

Subject-aware contrastive learning encompasses a family of methods that modify contrastive objectives, architectures, and sampling strategies to explicitly account for subject (or instance, identity) variability during representation learning. These strategies are designed to overcome inter-subject differences in domains where data are structured around distinct entities—such as individuals in biosignal acquisition, gaze estimation, HAR, or uniquely specified visual/textual “subjects” in generative models. By leveraging knowledge of subject identity, these methods improve the alignment, disentanglement, and generalization of learned features across diverse, subject-driven data distributions.

1. Core Principles and Motivation

The principal aim of subject-aware contrastive learning is to develop representations that are invariant to idiosyncratic, subject-specific variation while retaining sensitivity to task-relevant distinctions. This approach is particularly motivated by domains such as EEG, ECG, HAR, gaze estimation, and subject-driven T2I (text-to-image) generative modeling, where cross-subject variability otherwise limits generalization and transferability. Empirically, subject-aware contrastive methods have been shown to substantially improve cross-subject classification, few-shot transfer, interpretability, and text/image controllability metrics over standard instance-discriminative or self-supervised contrastive objectives (Zhang et al., 2023, Lee et al., 2022, Yarici et al., 4 Jul 2025, Du et al., 2023, Chen et al., 2024).

The theoretical foundation often derives from neuroscientific inter-subject correlation (ISC) theory, which posits that homologous brain states evoke shared neural patterns across individuals, as well as from the observation that naive contrastive learning can inadvertently cluster on subject identity rather than relevant semantic or physiological classes (Zhang et al., 2023, Shen et al., 2021).

2. Mathematical Formulations and Loss Design

Subject-aware contrastive learning redefines positive and negative pair assignments, loss weighting, or projection architectures to factor in subject identity. Typical contrastive objectives adopt the InfoNCE framework, but with the following subject-driven modifications:

• Subject-wise positive/negative construction: For an anchor sample $z_i$ from subject $s_i$ and label $y_i$, positives may be restricted to (a) samples with the same class but other subjects, or (b) same subject, different augmentations. Negatives are selected to enforce invariance or distinctiveness at the subject level (Lee et al., 2022, Yarici et al., 4 Jul 2025).
• Weighted denominators: Some frameworks, such as SICL, upweight or downweight the contribution of same-subject negatives via a scalar $Q_{Si}$, increasing the penalty for subject-specific variability (Yarici et al., 4 Jul 2025).
• Multi-level contrastive losses: In subject-driven generative frameworks, e.g., for T2I customization, multilevel contrastive losses are formulated to simultaneously optimize (i) intra-subject consistency across diverse contexts (pulling together embeddings of the same subject across views), (ii) inter-subject distinctiveness (pushing apart distinct subjects), and (iii) alignment across modalities (textual/visual) using multi-level InfoNCE-based terms (Chen et al., 2024).
• Projection modules and subject-conditional heads: Architectural modifications such as per-subject projection heads decouple identity encoding from shared backbone features, as implemented in ConGaze (Du et al., 2023).

A representative formalism for inter-subject contrastive loss (e.g., for EEG) is: $$\mathcal L_{\rm contrast} = -\,\frac{1}{|\mathcal B|}\sum_{i\in\mathcal B} \log \frac{ \sum_{j\in \mathcal P'(i)} \exp(z_i\cdot z_j/\tau) }{ \sum_{k\in \mathcal A'(i)} \exp(z_i\cdot z_k/\tau) }$$ where the anchor set construction $\mathcal A'(i)$ and positive set $\mathcal P'(i)$ are explicitly subject-aware (Lee et al., 2022).

3. Pairing, Sampling, and Augmentation Strategies

Success in subject-aware contrastive learning depends critically on principled construction of positive and negative pairs. Key sampling strategies include:

• Inter-subject positives: Pairs are constructed from different subjects but matching on stimulus/class, removing easy intra-subject matches to avoid trivial alignment and better enforce subject invariance (Lee et al., 2022, Shen et al., 2021, Zhang et al., 2023).
• Dedicated negatives and exclusion criteria: Pairs which would encourage spurious invariance or non-discriminativity, such as same-class/same-subject or different-class/different-subject, may be systematically excluded for focus and efficiency (Lee et al., 2022).
• Domain-specific augmentations: Time-series augmentations (temporal cutout, delay, bandstop, mixing) are employed for biosignals (Cheng et al., 2020), spatial cropping or landmark-conditioned augmentations preserve task-relevant attributes for gaze (Du et al., 2023), and multiscale patches or subsegment permutation are used in sleep staging (Zhang et al., 2023).
• Multimodal and multiview constructions: For crossmodal subject-driven problems, distinct crossmodal and multilevel contrastive losses may be defined at several hierarchy layers (e.g., using “CSCL” and “MACL” in CustomContrast (Chen et al., 2024)).

4. Architectures and Training Procedures

Subject-aware contrastive methods leverage both conventional and novel architectures, adapting them for subject-driven structure:

• Backbone encoders: MobileViT-derived (MViTime), GRU, CNNs, and Transformer architectures are adopted for time-series, EEG, and HAR (Zhang et al., 2023, Lee et al., 2022, Yarici et al., 4 Jul 2025).
• Projection heads: Standard and subject-conditional MLPs serve to map to the contrastive space, sometimes instantiated per subject (Du et al., 2023).
• Additional modules: Subject-invariance may be amplified by adversarial subject classifiers, as in the adversarial minimax regime for biosignals (Cheng et al., 2020).
• Multi-head and multilevel feature injection: For crossmodal generative modeling (e.g., text-to-image customization), systems employ parallel Qformers, PerceiverAttention, and special crossmodal integration modules for effectively fusing subject information at multiple semantic and appearance levels (Chen et al., 2024).

Training typically alternates between contrastive initialization and task-specific fine-tuning. Hyperparameters such as temperature ($\tau$), negative set weighting ($\lambda$, $Q_{Si}$), and batch construction are carefully cross-validated per dataset and application.

5. Applications and Empirical Results

Subject-aware contrastive learning has yielded state-of-the-art or substantially improved performance across multiple modalities:

• Sleep staging: Cross-subject pretraining with MViTime achieves overall accuracy of 87.8% and macro-F1 of 82.7% on the EDF-20 benchmark, outperforming prior self-supervised and supervised methods (Zhang et al., 2023).
• EEG-based visual and emotion recognition: Inter-subject contrastive objectives significantly boost accuracy in few-shot EEG-based visual recognition (top-1: 72.6% with 5 shots per class vs. 69.5% vanilla; 1-shot regime: absolute +5.5% gain) (Lee et al., 2022). For cross-subject emotion recognition, CLISA improves multi-class accuracy by nearly 10 points (45.7% vs. 35% for baselines) on 9-way classification and generalizes both to unseen subjects and stimuli (Shen et al., 2021).
• Human Activity Recognition: SICL provides up to an 11% relative top-1 accuracy gain on the DARai inertial dataset and consistently improves cross-subject adaptation over SimCLR, CMC, SupCon, and other instance-level SSL frameworks (Yarici et al., 4 Jul 2025).
• Gaze estimation: Subject-conditional representation learning in ConGaze reduces within-dataset angular error by up to 55% relative to global-contrastive SimCLR, and outperforms supervised pretraining in cross-dataset evaluation by 15–25% (Du et al., 2023).
• Subject-driven generative modeling: Multilevel contrastive learning in CustomContrast achieves highest reported scores on DreamBench (e.g., CLIP-T 0.325, Edit-DI 0.591), with ablations confirming that the dual constraints of intra-consistency and inter-distinctiveness are crucial for quality subject-driven image synthesis (Chen et al., 2024).

6. Generalization, Extensions, and Practical Considerations

Subject-aware contrastive learning methods have demonstrated, across domains:

• Superior cross-subject generalization: By aligning feature spaces across subjects, such methods yield robust performance on entirely unseen individuals or out-of-domain scenarios (Zhang et al., 2023, Shen et al., 2021, Yarici et al., 4 Jul 2025).
• Scalability and modularity: The underlying principles readily integrate with standard SSL pipelines such as SimCLR, Barlow Twins, VICReg, supervised contrastive frameworks, and multimodal contrastive setups (Yarici et al., 4 Jul 2025).
• Ablations and parameter sensitivity: The effectiveness of subject-aware strategies depends on meticulous pairing logic, choice of negative weighting, augmentation strength, and (for adversarial approaches) tuning the regularization parameters to avoid erasing relevant subject attributes (Lee et al., 2022, Cheng et al., 2020).
• Applications to bias mitigation: Context-enriched contrastive losses (ConTeX) extend the subject-aware concept to broader nuisance axes (class, bias, etc.), providing robustness against spurious correlations in structured datasets (Deng et al., 1 Dec 2025).
• Potential for extension: The negative-pair reweighting, pair construction, and architectural decoupling principles may be generalized to other axes of sample-level variability (scene, sensor, context), suggesting a plausible direction for further generalization (Yarici et al., 4 Jul 2025).

7. Open Challenges and Future Directions

While subject-aware contrastive learning has delivered marked gains, several challenges remain:

• Balance between invariance and discriminativity: Excessive removal of subject cues can degrade performance in tasks where subject-specific information is relevant; conversely, weak alignment may not offset subject-related distribution shifts (Cheng et al., 2020).
• Scalability to many subjects or under extreme low-data: Small subject pools can limit the effectiveness of subject-specific heads or per-subject negative decompositions—expanding unlabeled cohorts for SSL pretraining is a recommended mitigation (Du et al., 2023).
• Integration with more complex data modalities: Multimodal, hierarchical, and crossmodal subject-driven tasks (e.g., T2I, HAR with multi-sensor fusion) are promising frontiers needing further methodological unification and scaling (Chen et al., 2024, Yarici et al., 4 Jul 2025).

Ongoing research continues to improve both the theoretical understanding and practical instantiations of subject-aware and subject-invariant contrastive learning across increasingly diverse and challenging domains.