Contrastive Self-Supervised CAE

• CS-CAE is a hybrid model that integrates convolutional autoencoder reconstruction with contrastive self-supervision to enforce noise invariance in the latent space.
• It jointly optimizes reconstruction, noise-centered latent regularization, and contrastive alignment to enhance detection performance in gravitational-wave data.
• The architecture leverages a convolutional backbone and a dedicated projection head to improve feature separation, generalization, and anomaly detection compared to standard CAEs.

Searching arXiv for the cited CS-CAE paper and closely related contrastive/autoencoder references. Contrastive self-supervised convolutional autoencoder (CS-CAE) denotes a class of models that jointly optimize a convolutional autoencoder objective and a contrastive self-supervised objective in order to shape the latent space toward noise-invariant, task-relevant structure while retaining reconstructive capacity. In the formulation introduced for core-collapse supernova (CCSNe) gravitational-wave detection, CS-CAE combines a convolutional autoencoder (CAE), a noise-centered latent regularizer, and a projection head trained with a contrastive objective, so that independent noisy realizations of the same underlying signal are mapped to nearby latent representations (Sun et al., 20 May 2026). More generally, the design is intelligible within the modular contrastive self-supervised learning framework that decomposes methods into data augmentation, encoder selection, representation extraction, similarity measure, and loss function (Falcon et al., 2020). In this sense, CS-CAE is not a single canonical architecture but a hybrid design pattern in which reconstruction and contrastive alignment are trained jointly, with the latent geometry often playing a central methodological role (Sun et al., 20 May 2026, Arpit et al., 2021).

1. Conceptual definition and scope

CS-CAE occupies an intermediate position between conventional autoencoders and contrastive self-supervised representation learners. A conventional CAE is trained only with reconstruction and therefore has no explicit mechanism that enforces invariance across different noisy realizations of the same underlying example; in the CCSNe setting, this leaves the latent representation too sensitive to incidental noise fluctuations (Sun et al., 20 May 2026). By contrast, a pure contrastive method emphasizes agreement between positive pairs and separation from negatives, but does not by itself provide a decoder or a reconstruction-based anomaly score (Falcon et al., 2020).

The core CS-CAE construction described for CCSNe gravitational-wave detection preserves the base encoder-decoder architecture of a CAE and adds two components: a projection head for contrastive learning and a noise-centered latent regularizer (Sun et al., 20 May 2026). The resulting detector is not merely a contrastive pretraining scheme followed by a separate downstream model. Instead, the contrastive term, reconstruction term, and latent regularization term are optimized jointly, and the trained model is used directly at inference through a hybrid detection statistic that combines latent distance from the noise center with reconstruction error (Sun et al., 20 May 2026).

Within the broader literature, related autoencoder-contrastive hybrids differ in purpose. The momentum contrastive autoencoder (MoCA) is a Wasserstein autoencoder whose latent regularizer is replaced by a momentum-contrastive objective, with the aim of matching the aggregated posterior to a uniform hyperspherical prior rather than primarily improving downstream semantic representation quality (Arpit et al., 2021). This suggests that “CS-CAE” can refer to more than one research lineage: one centered on self-supervised invariant representation learning with an auxiliary decoder, and another centered on generative latent-space regularization via contrastive machinery. A plausible implication is that the term is best understood functionally—joint reconstruction plus contrastive latent shaping—rather than as a single standardized architecture.

2. Contrastive self-supervision as a modular design basis

A general conceptual framework for contrastive self-supervised learning characterizes such methods by five aspects: data augmentation pipeline, encoder selection, representation extraction, similarity measure, and loss function (Falcon et al., 2020). This decomposition is especially useful for CS-CAE because it isolates which parts of a contrastive pipeline can be grafted onto an autoencoder and which parts remain external design choices.

The augmentation pipeline constructs anchor, positive, and negative samples from unlabeled data. Let $D=\{x_1,x_2,\ldots,x_N\}$ be an unlabeled dataset, let $a_n$ be a stochastic augmentation, and let $A=(a_1,\ldots,a_N)$ be a sequential pipeline. Two common positive-pair constructions are

$v^a \sim A(x), \qquad v^+ \sim A(x),$

or two subsets drawn from the same feature representation,

$v^a, v^+ \subseteq v_x.$

Negatives are drawn from another sample,

$v^- \sim A(x'), \qquad x' \neq x .$

The same framework defines an encoder $f_\theta$, a representation extraction rule, a scalar similarity function $\Phi(r_a,r_b)$, and a contrastive loss based on positive and negative similarity scores (Falcon et al., 2020).

For CS-CAE, these five aspects map naturally onto the encoder branch of an autoencoder. The convolutional encoder corresponds directly to $f_\theta$. Representation extraction determines whether contrastive alignment is applied to a pooled global code or to a spatial latent tensor. Similarity is typically computed in a projection space rather than directly on decoder outputs. The reconstruction path itself is an extension beyond the contrastive framework, but the decomposition clarifies where that extension can be inserted without obscuring the contrastive design (Falcon et al., 2020).

Three canonical contrastive families illustrate the available design space. AMDIM uses multiscale intermediate feature maps and local NCE-style comparisons; CPC uses context-based prediction of spatially separated features; SimCLR uses strong stochastic augmentations, a projection head, and temperature-scaled contrastive loss on a flattened final representation (Falcon et al., 2020). YADIM, introduced within the same framework, simplifies representation extraction by using the last feature map only with dot-product similarity and NCE, and is reported as more robust to encoder choice and representation extraction strategy (Falcon et al., 2020). This supports a common CS-CAE design choice: maintain a simple final-latent contrastive branch while leaving the decoder tied to the same latent tensor or its pooled derivative.

3. Architectural structure of CS-CAE

In the CCSNe instantiation, the input is a three-channel whitened time series

$x \in \mathbb{R}^{3\times N},$

corresponding to the three Einstein Telescope interferometer channels $a_n$0 (Sun et al., 20 May 2026). The model compares three architectures built around a common 1D convolutional backbone: a supervised CNN, a CAE baseline, and the proposed CS-CAE (Sun et al., 20 May 2026). This shared backbone is methodologically important because it localizes the performance differences to the objective and latent-space organization rather than to gross architectural disparities.

The CAE encoder uses four 1D convolutional stages with channels $a_n$1, kernel sizes $a_n$2, stride $a_n$3 throughout, and paddings $a_n$4. Unlike the supervised CNN, the CAE encoder does not include max-pooling layers. After adaptive average pooling, a linear embedding layer maps the encoder output to a 128-dimensional normalized latent feature (Sun et al., 20 May 2026). The decoder reconstructs the input through four linear upsampling stages, each with scale factor $a_n$5, followed by 1D convolutional layers with kernel size $a_n$6 and padding $a_n$7, with decoder channel widths $a_n$8, and a final output back to 3 channels; the first three decoder blocks use batch normalization and ReLU (Sun et al., 20 May 2026).

CS-CAE retains this encoder-decoder backbone and augments it with a normalized latent embedding and a contrastive projection head. The latent vector is

$a_n$9

so the bottleneck is a unit-normalized 128-dimensional embedding (Sun et al., 20 May 2026). The projection head $A=(a_1,\ldots,a_N)$0 is a two-layer MLP consisting of linear $A=(a_1,\ldots,a_N)$1, ReLU, and linear $A=(a_1,\ldots,a_N)$2, with normalized output

$A=(a_1,\ldots,a_N)$3

Its stated role is standard in contrastive learning: it provides a space specialized for the contrastive objective, allowing the encoder’s latent $A=(a_1,\ldots,a_N)$4 to remain more useful for downstream detection statistics (Sun et al., 20 May 2026).

This separation between decoder-facing latent space and contrastive projection space parallels the SimCLR-style distinction between encoder representation and projection head output (Falcon et al., 2020). It also differs from MoCA, where the latent code itself is constrained to lie on the unit hypersphere and is used directly for both decoding and latent prior matching (Arpit et al., 2021). The comparison indicates that CS-CAE architectures can vary in whether the contrastive geometry is imposed on the decoder bottleneck itself or on a dedicated projection branch.

4. Objectives, latent geometry, and inference statistics

The CAE baseline reconstructs

$A=(a_1,\ldots,a_N)$5

with mean-squared reconstruction loss

$A=(a_1,\ldots,a_N)$6

At inference, the CAE detection statistic is the normalized mean-squared reconstruction error

$A=(a_1,\ldots,a_N)$7

This makes the baseline a purely reconstruction-based self-supervised detector (Sun et al., 20 May 2026).

CS-CAE adds a noise-center regularizer,

$A=(a_1,\ldots,a_N)$8

which pulls pure-noise embeddings toward a latent center $A=(a_1,\ldots,a_N)$9, maintained as an exponential moving average (Sun et al., 20 May 2026). The purpose is explicitly one-class-like: pure noise is concentrated near a compact central region, while signal-containing examples are expected to lie farther away. This gives the latent space a more interpretable geometry and makes the distance $v^a \sim A(x), \qquad v^+ \sim A(x),$0 a usable detection component (Sun et al., 20 May 2026).

The contrastive term is built from positive pairs consisting of two independent noisy realizations of the same clean CCSNe waveform: $v^a \sim A(x), \qquad v^+ \sim A(x),$1 In a minibatch containing $v^a \sim A(x), \qquad v^+ \sim A(x),$2 clean signals, there are $v^a \sim A(x), \qquad v^+ \sim A(x),$3 noisy views. For each anchor $v^a \sim A(x), \qquad v^+ \sim A(x),$4, the matching view $v^a \sim A(x), \qquad v^+ \sim A(x),$5 is the positive, and the remaining $v^a \sim A(x), \qquad v^+ \sim A(x),$6 views are negatives. The loss is an InfoNCE / NT-Xent-style objective,

$v^a \sim A(x), \qquad v^+ \sim A(x),$7

with $v^a \sim A(x), \qquad v^+ \sim A(x),$8 and $v^a \sim A(x), \qquad v^+ \sim A(x),$9 (Sun et al., 20 May 2026). Because the embeddings are normalized, dot product equals cosine similarity (Sun et al., 20 May 2026).

The total training objective is

$v^a, v^+ \subseteq v_x.$0

with

$v^a, v^+ \subseteq v_x.$1

At inference, CS-CAE uses a hybrid score

$v^a, v^+ \subseteq v_x.$2

with

$v^a, v^+ \subseteq v_x.$3

Thus the contrastive loss is only a training-time term, but it shapes the inference-time latent anomaly score through the geometry of $v^a, v^+ \subseteq v_x.$4 relative to the learned noise center (Sun et al., 20 May 2026).

A broader methodological comparison is useful here. In the WAE-based MoCA formulation, the combined loss is also additive—reconstruction plus contrastive regularization—but the contrastive term is justified as an entropy-maximizing latent prior matcher over the unit hypersphere rather than as a noise-invariance mechanism (Arpit et al., 2021). This suggests two distinct functions for contrastive regularization in autoencoders: latent distribution matching for generation, and invariance induction for detection or representation learning.

5. Empirical performance in CCSNe gravitational-wave detection

The CS-CAE study evaluates the method on simulated Einstein Telescope data under three test settings: an in-distribution phenomenological CCSNe test set, a cross-distribution test set consisting of unseen numerical waveform catalogs, and a glitch-contaminated robustness test generated with gengli (Sun et al., 20 May 2026). Training uses Gaussian stationary ET-D noise, a sampling rate of 4096 Hz, 1.5 s segments, 1 s intrinsic CCSNe waveform duration, and randomized injection start time in $v^a, v^+ \subseteq v_x.$5 s (Sun et al., 20 May 2026).

On the matched phenomenological test set, reported AUC values are $v^a, v^+ \subseteq v_x.$6 for the supervised CNN, $v^a, v^+ \subseteq v_x.$7 for CS-CAE, and $v^a, v^+ \subseteq v_x.$8 for the CAE baseline (Sun et al., 20 May 2026). On unseen numerical CCSNe waveform families outside the training distribution, the ordering reverses slightly: CS-CAE attains $v^a, v^+ \subseteq v_x.$9, the supervised CNN $v^- \sim A(x'), \qquad x' \neq x .$0, and the CAE $v^- \sim A(x'), \qquad x' \neq x .$1 (Sun et al., 20 May 2026). The paper interprets this as evidence that contrastive self-supervision learns features more invariant to detector-noise realization and less over-specialized to the training waveform family (Sun et al., 20 May 2026).

Distance-dependent sensitivity is summarized operationally by the distance at which the detection efficiency reaches about $v^- \sim A(x'), \qquad x' \neq x .$2 at the adopted operating threshold. Under the Einstein Telescope configuration, the reported effective sensitive distance for CS-CAE is approximately $v^- \sim A(x'), \qquad x' \neq x .$3 (Sun et al., 20 May 2026). The study also reports ROC behavior at 10 kpc, 100 kpc, and 500 kpc. At 10 kpc all methods perform strongly; at 100 kpc CS-CAE shows the clearest advantage in the low-FPR region; and at 500 kpc overall TPR drops for all methods, but CS-CAE retains a relative advantage at low FPR (Sun et al., 20 May 2026).

Latent-space organization is another empirical emphasis. UMAP projections show that CNN embeddings exhibit substantial overlap between noise, glitches, and CCSNe; CAE improves separation somewhat; and CS-CAE yields the clearest organization, with CCSNe forming a compact cluster and glitches splitting into separate isolated clusters (Sun et al., 20 May 2026). Because glitches are injected in one ET channel at a time, the glitch clusters correspond to channel-localized artifacts, which is physically meaningful under a three-channel detector geometry (Sun et al., 20 May 2026).

The study does not present a dedicated ablation section. It therefore does not disentangle the contributions of the projection head, the contrastive loss, the noise-center regularizer, and the hybrid inference statistic. The closest aggregate comparison is CAE versus CS-CAE, where CAE can be viewed as CS-CAE without the contrastive head/objective and without latent center regularization (Sun et al., 20 May 2026). This implies that the reported gains establish the value of the added components jointly rather than individually.

6. Theoretical interpretation and relation to adjacent autoencoder paradigms

A central theoretical argument for adding contrastive learning to an autoencoder is that pure reconstruction may preserve nuisance variation rather than recover the latent signal subspace. In a linear representation setting under a spiked covariance model,

$v^- \sim A(x'), \qquad x' \neq x .$4

“The Power of Contrast for Feature Learning” shows that standard autoencoders and linear Wasserstein GANs reduce to PCA-like estimators, whereas contrastive learning with masking behaves like a covariance estimator that suppresses diagonal noise (Ji et al., 2021). Under the stated assumptions, the expected feature recovery error of the autoencoder satisfies a constant-order lower bound,

$v^- \sim A(x'), \qquad x' \neq x .$5

while contrastive learning satisfies

$v^- \sim A(x'), \qquad x' \neq x .$6

The paper also derives downstream excess-risk guarantees favoring contrastive learning over standard autoencoders in regression and classification (Ji et al., 2021).

These results are not direct theorems about deep convolutional CS-CAE architectures, but they provide a principled rationale for hybridization: reconstruction alone may bias the latent space toward high-variance nuisance detail, whereas contrastive alignment can emphasize signal-consistent structure (Ji et al., 2021). The same theory includes a remark that denoising autoencoders and masked autoencoders can behave similarly to contrastive learning in the linear framework, which suggests that the advantages of CS-CAE may depend not only on contrastive alignment but also on how the reconstruction pathway is formulated (Ji et al., 2021).

This observation resonates with the medical-imaging comparison between contrastive pretraining and convolutional masked autoencoding. In CT classification with scarce labels, MoCo v2, SwAV, and BYOL are strong baselines, but the convolutional masked autoencoder SparK is reported to be more robust as downstream labeled data decrease (Wolf et al., 2023). The authors propose SparK for CT classification tasks with fewer than about 150 samples per class and note that approximately 60 images per class are needed to achieve “decent” results, defined as $v^- \sim A(x'), \qquad x' \neq x .$7, while explicitly cautioning that these thresholds are task-dependent (Wolf et al., 2023). This suggests that a CS-CAE for small-data medical imaging might benefit from a reconstruction branch that captures local structural relations, alongside a contrastive or alignment-based branch that organizes feature space globally.

A different but related neighboring paradigm is MoCA, where contrastive learning regularizes the latent space of a Wasserstein autoencoder toward a uniform distribution on the unit hypersphere. There, the crucial regularization term is interpreted as maximizing latent entropy rather than aligning multiple noisy realizations of the same semantic object (Arpit et al., 2021). The contrastive mechanism is MoCo-style, using a momentum encoder and a queue, and the method is explicitly generative in emphasis, with evaluations on FID, reconstructions, and interpolations rather than detection (Arpit et al., 2021). The contrast with the CCSNe detector clarifies that CS-CAE can target either discriminative robustness, generative latent matching, or both, depending on the role assigned to the contrastive branch.

7. Design issues, limitations, and research directions

Several recurring design tensions define the CS-CAE problem. Reconstruction encourages preservation of all information needed to reproduce the input, whereas contrastive self-supervision encourages invariance across positive views and de-emphasis of nuisance variation (Falcon et al., 2020, Sun et al., 20 May 2026). In the CCSNe setting, the contrastive objective is intentionally constructed from the same astrophysical waveform corrupted by independent noise realizations, so that the learned invariance is specifically noise-invariance rather than augmentation-invariance in the image-SSL sense (Sun et al., 20 May 2026). A plausible implication is that the choice of positive-pair construction is the central problem-specific ingredient in any CS-CAE: it determines what variability is to be suppressed and what structure is to be preserved.

The available evidence also advises against assuming that every contrastive formulation transfers equally well into an autoencoder. The contrastive framework paper reports that representation extraction strategy can be a major source of sensitivity, especially for AMDIM, while YADIM is more robust to encoder choice and representation extraction strategy (Falcon et al., 2020). This suggests that CS-CAE designs using a simple final latent map or pooled final representation may be preferable to elaborate multiscale handcrafted pairings when a decoder already imposes architectural constraints.

For small labeled medical datasets, the CT study indicates that reconstruction-based self-supervision can degrade more slowly than pure contrastive pretraining as downstream labels are reduced (Wolf et al., 2023). This does not establish that a joint CS-CAE would outperform either component alone, because no true combined contrastive-plus-reconstruction model is tested there (Wolf et al., 2023). Still, the comparison provides a clear hypothesis: contrastive learning may contribute inter-image invariance and feature-space organization, while masked reconstruction may contribute local anatomy-sensitive structure.

The CCSNe CS-CAE paper explicitly acknowledges several limitations. Training and core evaluation are conducted in stationary Gaussian ET-D noise, whereas real detector noise is more complex and nonstationary. The model uses a convolutional-only architecture, with future work suggested on LSTM-based self-supervised models. Although the three ET channels are processed jointly, there is no explicit cross-channel coherence constraint. Glitch handling is only partial, based on simulated blip-like glitches rather than full real-detector glitch populations. Sensitive distance remains modest at approximately $v^- \sim A(x'), \qquad x' \neq x .$8, indicating sensitivity mainly to Galactic and nearby Local Group-scale CCSNe rather than cosmological distances (Sun et al., 20 May 2026).

Proposed future directions include detector-wise feature extraction, explicit cross-channel coherence constraints, glitch-aware contrastive training, and LSTM-based self-supervised architectures (Sun et al., 20 May 2026). Read together with the broader contrastive and masked-autoencoding literature, these directions suggest that the next phase of CS-CAE research is likely to focus less on the mere coexistence of reconstruction and contrastive loss, and more on the precise geometry of the latent space, the semantics of positive-pair construction, and the interaction between local reconstructive fidelity and invariance-inducing alignment (Falcon et al., 2020, Wolf et al., 2023, Arpit et al., 2021).

CS-CAE is therefore best understood as a hybrid latent-learning framework rather than a fixed recipe. Its defining feature is the joint use of a convolutional autoencoder backbone and a contrastive self-supervised mechanism to impose structure on the latent representation. In the CCSNe application, this yields near-supervised in-distribution performance, better generalization to unseen waveform families than a supervised CNN, improved low-FPR behavior, and a latent organization that separates signals, noise, and glitches more clearly than a conventional CAE (Sun et al., 20 May 2026). Across adjacent literatures, the same hybrid principle appears in other guises—as latent prior matching in Wasserstein autoencoders, as a theoretically motivated alternative to pure reconstruction, and as a prospective bridge between contrastive invariance and masked reconstructive robustness (Arpit et al., 2021, Ji et al., 2021, Wolf et al., 2023).