Participant-Invariant Representation Learning

• PiRL is a framework that suppresses individual variability by enforcing invariance to participant identity using methods such as MMD, adversarial losses, NMF, and inverse contrastive loss.
• It is applied in temporal networks, healthcare, and computer vision to mitigate covariate shift and label leakage, thereby enhancing model generalization.
• Empirical results demonstrate that PiRL frameworks can boost classification accuracy by around 5% while yielding scalable and interpretable representations for downstream tasks.

Participant-invariant Representation Learning (PiRL) refers to a class of machine learning frameworks designed to extract representations from multi-participant data in which inter-individual variability is explicitly suppressed while retaining features relevant to a downstream predictive or generative task. The defining property of PiRL is enforced invariance to the participant ("subject" or "identity" in other literature) that generated each observation, producing embeddings aligned across users and robust to heterogeneity, while still being discriminative for relevant labels or capable of reconstructing salient temporal/network dynamics. PiRL has emerged as a solution to the pervasive challenge of user or participant heterogeneity in temporal networks, healthcare, and computer vision, enabling models to generalize beyond individual-specific idiosyncrasies (Shaojie, 2023, Cao et al., 2023, Cao et al., 2022, Chen et al., 2019, Akash et al., 2021).

1. Motivation and Conceptual Foundations

PiRL addresses persistent gaps observed between generic ("one-size-fits-all") models and person-specific models, particularly in temporal networks and health applications involving behavioral or physiological signals (Cao et al., 2022, Cao et al., 2023). In such domains, inter-participant variability in signal structure, network position, interaction rates, or behavior can lead to substantial performance degradation when deploying pooled models. However, person-specific models are impractical at scale due to requirements for per-user data collection, calibration, and system complexity. PiRL offers a principled approach for closing this gap by learning representations that are functionally blind to participant identity, thereby mitigating covariate shift, label leakage, and overfitting to subject-specific artifacts. In network science, PiRL formalizes participant-invariant influence as a latent signature that governs edge-formation or interaction rates identically across all nodes, independent of role or degree (Shaojie, 2023).

2. Formal Methodologies

PiRL frameworks generally comprise a core feature encoder trained with explicit participant-invariance constraints. The methodologies can be grouped by the application domain and architecture:

• Temporal Networks: PiRL posits the hypothesis that for each node $i$ in a temporal network $G_t=(V_t,E_t)$, one can model the local degree (or centrality) increment trajectory $M^*_{i,\cdot}$ as a nonnegative scaling $W_i$ of a shared, network-specific time-varying "influence signature" $H(t)$ (Shaojie, 2023). This is operationalized as a nonnegative matrix factorization (NMF) problem:

$$M^* \approx W H, \qquad W\in\mathbb R_+^{n\times 1},\; H\in\mathbb R_+^{1\times t_{\max}},$$

solved with multiplicative updates, Frobenius-regularization, or alternative NMF algorithms.

• Healthcare and Time Series: Here, PiRL is typically an autoencoder-based pipeline with additional losses: maximum mean discrepancy (MMD) is used to align the distributions of latent representations across participants, a domain-adversarial loss blurs identity information by having an auxiliary classifier tried to recover participant ID (optimized against the encoder via a gradient reversal layer), and a triplet loss constrains the latent space to remain label-discriminative (Cao et al., 2023, Cao et al., 2022). The combined objective for unsupervised and supervised phases is of the form:

$$L_{unsup} = L_{ae} + \lambda_{mmd} L_{mmd} + \mu L_{domain}, \quad L_{sup} = L_{cls} + \gamma L_{triplet}.$$

• Computer Vision (Adversarial and Cyclic Networks): PiRL-inspired methods use encoders trained with adversarial classifiers to remove specified (e.g., identity) information from latent codes, sometimes employing cyclic consistency in latent and image spaces to reinforce invariance (Chen et al., 2019). Losses combine adversarial, reconstruction, KL, and cycle-consistency terms.
• Inverse Contrastive Loss (ICL): An alternative, analytically-motivated approach inverts classical contrastive learning: intra-participant sample pairs are explicitly repelled, and inter-participant pairs are attracted, yielding pairwise convex losses and a guarantee of participant-invariance in the latent embedding (Akash et al., 2021).

PiRL Variant	Key Invariance Method	Core Architecture
Temporal Networks (Shaojie, 2023)	NMF on degree/centrality increments	Nonnegative factorization
Healthcare (Cao et al., 2023, Cao et al., 2022)	MMD/Adversarial/Triplet loss	1D CNN autoencoder/classifier
Computer Vision (Chen et al., 2019)	Adversarial, cyclic losses	VAE-GAN + cyclic reconstruction
ICL (Akash et al., 2021)	Inverse contrastive loss	Generic encoder + ICL regularizer

3. Loss Functions and Optimization

PiRL frameworks enforce participant-invariance via multiple strategies:

• Maximum Mean Discrepancy (MMD): Minimizes distances between participant-specific embedding distributions in RKHS, empirically via kernelized pairwise comparisons. For participants $i$ and $j$ with embedding distributions $P_i$ and $G_t=(V_t,E_t)$0, the MMD term is:

$G_t=(V_t,E_t)$1

(Cao et al., 2023, Cao et al., 2022)

• Domain-Adversarial Loss: Adopts a domain discriminator to classify participant ID; encoder is optimized to maximize discriminator loss (via gradient reversal), enforcing identity-invariance. Empirical results indicate marginal gains when participant count is high and batch sizes are limited (Cao et al., 2023, Cao et al., 2022).
• Triplet Loss: Ensures inter-class (label) separation in latent space while preserving intra-class compactness; essential to prevent trivial or collapsed representations during alignment (Cao et al., 2023, Cao et al., 2022).
• Nonnegative Matrix Factorization (NMF): In temporal networks, NMF is used to distill a low-rank, participant-invariant signature $G_t=(V_t,E_t)$2 by minimizing $G_t=(V_t,E_t)$3, subject to nonnegativity (Shaojie, 2023).
• Inverse Contrastive Loss: Defines a loss function such that intra-participant pairs contribute a repulsive (exponential) term and inter-participant pairs contribute an attractive (quadratic) term; for binary participant labels, this is equivalent to regularized MMD (Akash et al., 2021).
• Adversarial and Cyclic Losses: In image applications, adversarial classifiers and cyclic consistency losses enforce invariance to specified factors (e.g., identity), ensuring the latent variable encodes only unspecific variation (Chen et al., 2019).

4. Interpretation, Applications, and Empirical Results

PiRL yields representations—or, in temporal networks, network-level signatures—used for multiple downstream purposes:

• Temporal Networks: The extracted $G_t=(V_t,E_t)$4 is a compact time series describing the participant-invariant dynamics of the network, which can be used for cross-network comparison, domain clustering, or anomaly detection (Shaojie, 2023). Pairwise dynamic time warping (DTW) distances between network signatures $G_t=(V_t,E_t)$5 enable accurate hierarchical clustering by domain, with silhouette scores exceeding 0.75 and cluster purity over 85%.
• Healthcare: PiRL-trained models improve generic classification accuracy for tasks such as stress detection (CLAS dataset) and sleep apnea detection (Apnea-ECG), with combined MMD + triplet loss yielding a consistent 5% absolute gain over baseline autoencoder/classifier pipelines (e.g., from 64.3% to 70.6% on CLAS, and from 75.2% to 79.9% on Apnea-ECG; all $G_t=(V_t,E_t)$6 vs. baseline) (Cao et al., 2023, Cao et al., 2022). Person-specific models remain upper-bound (86.8% and 95.7%, respectively), but PiRL narrows the gap while maintaining scalability and adaptability to new participants.
• Invariant Generative Models: In image domains, cyclic, adversarial PiRL frameworks achieve latent codes free of identity while preserving relevant factors (pose, illumination, etc.), with state-of-the-art performance on both latent invariance (e.g., chair or face identity classification rates near chance) and conditional generation metrics (Chen et al., 2019).
• ICL Approach: Demonstrates superior subject-invariance and competitive downstream accuracy compared to classical MMD, information bottleneck, or adversarial approaches, as measured by adversary accuracy or ROC-AUC for subject inference remaining close to chance (Akash et al., 2021).

5. Limitations and Challenges

Multiple limitations are recognized across PiRL formulations:

• Non-uniqueness in NMF: Factorization of the degree/centrality increment matrices is nonconvex, with multiple local minima (Shaojie, 2023). This is partly alleviated by the strong participant-invariance assumption, but care is needed (e.g., multi-start strategies).
• Incomplete Higher-Order Alignment: MMD and DANN losses primarily control first-moment alignment across participants. Residual participant or domain leakage, observable via t-SNE or adversarial probes, can persist (Cao et al., 2022). Additional alignment losses (e.g., ICL or Wasserstein) may be necessary.
• Label Discriminability: Excessive enforcement of participant-invariance can collapse label structure or remove predictive information, necessitating explicit discriminative losses (e.g., triplet, supervised cross-entropy) (Cao et al., 2023).
• Batch Size and Participant Count: Domain-adversarial losses may be ineffective when participant count is large relative to batch size, as the discriminator receives too few samples per domain per update (Cao et al., 2022).
• Scalability and Modalities: Current frameworks primarily address unimodal, moderate-scale data. Extensions to multimodal, streaming, or highly imbalanced regimes remain an open direction (Cao et al., 2022).

6. Extensions and Future Directions

Several extensions to PiRL have been proposed:

• Alternate Invariance Losses: Incorporation of contrastive-based (e.g., inverse contrastive loss), Wasserstein, or higher-moment domain-alignment penalties to capture subtle participant-specific differences (Cao et al., 2022, Akash et al., 2021).
• Multi-Modal PiRL: Application to fused signals (ECG + EDA + PPG) for richer multimodal health or behavior modeling (Cao et al., 2022).
• Meta-Learning and Rapid Adaptation: Leveraging prototypical networks or meta-learning strategies for accelerated adaptation to unseen participants (Cao et al., 2022).
• Generalization to Other Domains: Extension of PiRL to speech emotion, activity recognition, recommendation systems, and network science, wherever participant/user heterogeneity is a primary confound (Shaojie, 2023, Cao et al., 2022, Cao et al., 2023).
• Network Roles and Multi-Rank NMF: Allowing for multiple roles or modalities by increasing NMF rank, yielding mixtures of participant-invariant signatures and finer-grained clustering (Shaojie, 2023).

7. Summary and Significance

Participant-invariant Representation Learning (PiRL) provides a structured solution to modeling under participant/user heterogeneity by systematically aligning representations, removing identity information, and retaining task-relevant signal. Whether by kernel-based alignment, adversarial training, matrix factorization, or pairwise convex losses, PiRL frameworks yield scalable, flexible models that avoid the system complexity of fully person-specific approaches, improve out-of-participant generalization, and enable compact, interpretable signatures—both for predictive modeling and unsupervised analysis. Continued advances are focused on capturing higher-order invariance, accommodating multi-modal and multi-role interactions, and extending to broader application domains (Shaojie, 2023, Cao et al., 2023, Cao et al., 2022, Akash et al., 2021, Chen et al., 2019).