LeNEPA: No-Augmentation Next-Latent Prediction for Time-Series Representation Learning

Abstract: Time series are central to modern data mining applications, from industrial telemetry and server metrics to finance and physiology, yet time-series self-supervised learning often depends on view and augmentation choices that encode domain-specific invariances. We study how an SSL recipe behaves when its method-specific configuration is reused unchanged after the pretraining signal family changes, framing this as a fixed-recipe stress test rather than a comparison against optimally tuned methods. We introduce Latent Euclidean Next-Embedding Prediction Architecture (LeNEPA), a no-augmentation next-latent-token objective with a causal backbone. LeNEPA replaces the stop-gradient/EMA stabilization used by vanilla NEPA with SIGReg-based isotropy regularization and computes the predictive loss in a lightweight projected space that is discarded for evaluation. We compare LeNEPA with an ECG-tuned JEPA recipe under a fixed-horizon frozen-probe protocol on PTB-XL and Diag, a synthetic diagnostic corpus generated with Aionoscope. Both methods are retrained independently on each dataset while keeping their method-specific recipes unchanged. In this protocol, the ECG-tuned JEPA recipe is strong in-domain on PTB-XL but weaker when reused unchanged on Diag, whereas LeNEPA preserves useful frozen-probe gains on both datasets. Learning curves suggest faster early representation acquisition: LeNEPA reaches 80% of its final AUROC/AUPRC gain after 2--5k updates, compared with 5--10k updates for the faster JEPA readout. As a separate external frozen-encoder check, a CauKer-pretrained LeNEPA variant reaches 77.65% mean UCR-128 Random-Forest accuracy in a single-seed, best-checkpoint run, within 1.16 points of Mantis and within 0.24 points of MOMENT (77.89%). Overall, the results support no-augmentation latent prediction as a useful candidate recipe for low-retuning time-series SSL.

• The paper presents a novel no-augmentation method that uses next-latent prediction with temporal isotropy regularization to learn transferable time-series representations.
• It employs a convolutional patch embedder, causal Vision Transformer, and a disposable projector head to stabilize training without gradient stopping.
• Experimental results show that LeNEPA achieves competitive performance across diverse datasets, highlighting its robustness compared to augmentation-dependent methods.

LeNEPA: No-Augmentation Next-Latent Prediction for Time-Series Representation Learning

Introduction

Self-supervised learning (SSL) in time series remains deeply reliant on view and augmentation engineering, with each new data modality often necessitating customized processor pipelines. While vision and language SSL objectives—such as contrastive learning, masked language modeling, and joint embedding architectures—have converged on powerful and somewhat standardized recipes, porting these approaches to time-series data is hampered by the data-type dependency of augmentation schemes. "LeNEPA: No-Augmentation Next-Latent Prediction for Time-Series Representation Learning" (2607.00958) addresses this challenge by eliminating handcrafted augmentations, proposing a no-augmentation, next-latent-token prediction architecture that achieves strong, transferable representations across distinct time-series domains with minimal per-dataset tuning.

LeNEPA Architecture and Training Objective

LeNEPA (Latent Euclidean Next-Embedding Prediction Architecture) extends NEPA-style [Xu et al., 2025] architectures by focusing on an augmentation-free design and stabilization. The pipeline consists of three main stages: a convolutional patch embedder, a causal Vision Transformer (ViT) backbone, and a lightweight projector head, the latter used only for training and then discarded—aligning with the Guillotine Regularization principle [Bordes et al., 2022].

At each time step, LeNEPA predicts the next latent token and minimizes the mean squared error (MSE) between the predicted and target embeddings in the projected space. Unlike NEPA, which stabilizes training via stop-gradient or EMA on targets, LeNEPA introduces Sketched Isotropic Gaussian Regularization (SIGReg) [Balestriero & LeCun, 2025] over the temporal axis. This encourages per-sample isotropy in projected embeddings, preventing feature collapse within each sequence. Losses are computed as:

• Projected prediction loss over adjacent patch embeddings.
• Temporal SIGReg loss, ensuring isotropy across all tokens of a sample.

A key implementation detail is that no gradients are stopped for the prediction target; stability emanates entirely from the regularization. Projector weights are removed after training, and evaluation leverages the main backbone, promoting generalization.

(Figure 1)

Figure 1: The LeNEPA pipeline: input series are patch-embedded, processed through a causal ViT, and a next-latent prediction loss plus SIGReg is applied in the projected space, which is discarded after training.

Evaluation Protocol and Datasets

The study targets pretraining recipe reuse rather than shared checkpoint transfer, operationalizing robustness as the ability of a fixed SSL configuration to generalize across structural changes in the pretraining signal family. Three instantiations of LeNEPA are compared:

• LeNEPA-PTBXL: Trained and evaluated on PTB-XL (12-lead ECG).
• LeNEPA-Diag: Trained and evaluated on Diag, a synthetic diagnostic time-series corpus designed to stress structural variance.
• LeNEPA-CauKer: Trained on CauKer (synthetic), evaluated on the UCR-128 archive.

All configurations use the same architectural and loss hyperparameters, permitting a controlled comparison of recipe transferability. As a baseline, a JEPA variant [Assran et al., 2023] tailored for ECG and using a fixed masking/augmentation recipe is retrained on both PTB-XL and Diag, with no recipe retuning.

Probing is performed via frozen linear or Random Forest classifiers, using representations extracted from all layers due to the observation that next-latent prediction methods often peak at intermediate depths.

Experimental Results

Fixed-Recipe Portability (PTB-XL/Diag)

Empirical results focus on whether SSL recipes maintain their efficacy when transferred without retuning. On PTB-XL, the ECG-tuned JEPA and LeNEPA reach comparable AUROC/AUPRC, confirming that leveraging strong augmentations or none at all can produce similarly useful features when objectives are matched to the data.

On Diag, the differences become stark: the ECG-tuned JEPA recipe—strong in-domain—drops sharply in downstream probe metrics when transferred to the synthetic diagnostic domain. In contrast, LeNEPA maintains high AUROC, AUPRC, and regression metrics, highlighting its portability:

Figure 2: PTB-XL/Diag fixed-recipe reuse: LeNEPA maintains strong AUROC/AUPRC on both domains, while JEPA falters when the augmentation recipe is reused unchanged on Diag.

Learning curves further indicate that LeNEPA reaches 80% of its probe performance within 2–5k updates, outperforming JEPA in terms of early representation acquisition under the fixed-recipe regime. This is notable given that Diag intentionally introduces multiple rare latent events and a wide diversity of generative styles, which amplify the cost of rigid augmentation design.

Frozen-Encoder Generalization (UCR-128)

The frozen-encoder experiment addresses whether LeNEPA-trained representations on a synthetic corpus (CauKer) serve as competitive frozen features for external datasets (UCR-128, 128 diverse univariate classification tasks). Without using handcrafted augmentations or bidirectional attention, LeNEPA achieves 77.65% mean accuracy with a Random Forest probe—within 1.16 and 0.24 percentage points of Mantis [Feofanov et al., 2026] and MOMENT [Goswami et al., 2024] protocol-matched results, respectively, both of which rely on larger models and meticulous augmentation chains.

Figure 3: UCR-128 layer-profile diagnostic: LeNEPA achieves peak probe accuracy at intermediate backbone layers, slightly outperforming the final layer.

Layer-Wise Probing and Optimization Insights

Intermediate backbone layers frequently achieve the best probe results for LeNEPA and NEPA, in contrast to JEPA and contrastive objectives, which often favor deeper layers. The layer-sensitivity analysis confirms that fixing the probe to a mid-layer (e.g., $L4$) achieves nearly all benefits of oracle layer selection, greatly reducing the cost of per-dataset tuning.

Figure 4: Last-step ($20,000$ updates) per-layer probe performance: LeNEPA's and NEPA's optimal classification layers are consistently in the middle of the transformer stack, not the final block.

Methodological Ablations

Ablations confirm that:

• Applying projective losses via a disposable training head (Guillotine Regularization) robustly improves probe performance.
• Temporal SIGReg is the only regularization variant (vs. batch-wise, pooled, or innovation-based placements) that yields sustained gains across all tested datasets.
• No significant gain is observed from increasing projector depth or width beyond a standard configuration, indicating that the main effect is decoupling loss space from representation space.
• JEPA's performance is notably sensitive to the masking recipe; a 2–3x shift in the masking keep-ratio substantially degrades downstream probe metrics, corroborating the risk of domain-mismatched augmentations.

Implications and Future Directions

This work demonstrates that next-latent prediction with robust isotropy regularization constitutes a strong and highly portable foundation for time-series SSL even in the absence of expert-crafted augmentations. The architectural and objective design choices enable rapid, low-friction reuse of pretrained SSL recipes across disparate domains. The results provide evidence that augmentation-free SSL targets a distinct regime: robustness to signal-family shifts where view engineering is impractical.

The main limitations revolve around dependence on tokenizer/patch sizes and regularization scale, sparse multivariate evaluation, and the synthetic nature of some test suites (e.g., Diag). Extension to irregularly sampled, multivariate, and more complex event-driven time series, as well as establishing principled scaling rules for SIGReg and convolutional tokenizers, remain open research topics.

Conclusion

LeNEPA presents a principled approach to domain-agnostic self-supervised time-series representation learning by eschewing handcrafted augmentations in favor of next-latent prediction stabilized by temporal isotropy regularization and computed in a disposable projected space. Fixed-recipe experiments demonstrate that this configuration yields robust, transferable features for both in-domain and out-of-domain tasks, as confirmed by controlled benchmarks and strong protocol-matched performance across representative datasets. Consequently, LeNEPA defines a viable design direction for practitioners seeking to minimize the operational burden of SSL recipe tuning across heterogeneous time-series regimes.